[GUI Plugin] QuarantineAndScrub

[DaltonST]

Version 3.3.5- 30 December 2014 Database Table and View Changes; Revised User Guide; Official Q&S Database to Download [for new users only] Upgraded to DB Version 3.5.0.


As always, please execute the "After Version Upgrade" job so your Q&S database will be updated to reflect changes in several SQLite database objects resulting in DB version 3.5.0.




DaltonST

[arogl]

I have upgraded to this version and am running through some 7000 books that were imported and it crashes with the following:

It seems it cannot handle utf8 characters in the series/working series column.

I tried to PM but I have insufficient privileges.


calibre, version 2.15.0 (win32, isfrozen: True)
Failed to Scrub Selected Books in Foreground/GUI: Failed: Quarantine And Scrub

Starting job: Quarantine And Scrub
H:/Portable/Calibre/QuarantineAndScrub/metadata.db
SQLite version is: 3008004
PRAGMA main.locking_mode=EXCLUSIVE;
Beginning Scrubbing
26 Book(s) Being Scrubbed
Title as a Tag Has Been Removed: London
Series Name as a Tag Has Been Removed: Murder
Title as a Tag Has Been Removed: Mystery
Title as a Tag Has Been Removed: Sisters
New Work Tag(s) derived from book: Daniel Defoe the Urther Adventures of Crusoe
Series Name as a Tag Has Been Removed: Adventure
New Work Tag(s) derived from book: Danielle Steel : At a Time: A Novel
Title as a Tag Has Been Removed: Sisters
Tag with Series Name Has Been Removed: Women's Fiction
Series Name as a Tag Has Been Removed: Adventure
Title as a Tag Has Been Removed: Mystery
Series Name as a Tag Has Been Removed: Adventure
Title as a Tag Has Been Removed: Mystery
Series Name as a Tag Has Been Removed: Adventure
Title as a Tag Has Been Removed: Mystery
Job: "Quarantine And Scrub" failed with error:
Traceback (most recent call last):
File "site-packages\calibre\gui2\threaded_jobs.py", line 83, in start_work
File "calibre_plugins.quarantine_and_scrub.bookleve l", line 135, in main_scrub_book_level
File "calibre_plugins.quarantine_and_scrub.bookleve l", line 234, in Scrub_Control
File "calibre_plugins.quarantine_and_scrub.bookleve l", line 6145, in scrub_work_series_with_numerics
File "calibre_plugins.quarantine_and_scrub.bookleve l", line 5959, in __generic_update_work_series_seriesindex
File "calibre_plugins.quarantine_and_scrub.bookleve l", line 7526, in __add_missing_spaces_to_camelback_strings
File "calibre_plugins.quarantine_and_scrub.bookleve l", line 7595, in __change_tuple_to_bytestring
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 11-12: invalid continuation byte

Called with args: (<calibre_plugins.quarantine_and_scrub.ui.Quaranti neAndScrub object at 0x0C66D8A0>, <calibre.db.legacy.LibraryDatabase object at 0x068A2810>, ['9189', '6751', '4370', '7308', '7791', '8863', '9045', '3366', '7240', '3733', '5911', '6559', '8286', '5418', '8405', '8038', '8039', '8040', '8041', '3678', '6293', '6294', '5808', '4972', '4973', '6670'], [], u'H:\\Portable\\Calibre\\Calibre Settings\\plugins\\QuarantineAndScrub.zip') {u'log': <calibre.utils.logging.GUILog object at 0x0A23A050>, u'notifications': <Queue.Queue instance at 0x09B5F850>, u'abort': <threading._Event object at 0x0A23A030>}

Rgds Andrew.

Originally Posted by DaltonST

Version 3.3.5- 30 December 2014 Database Table and View Changes; Revised User Guide; Official Q&S Database to Download [for new users only] Upgraded to DB Version 3.5.0.


As always, please execute the "After Version Upgrade" job so your Q&S database will be updated to reflect changes in several SQLite database objects resulting in DB version 3.5.0.




DaltonST

[DaltonST]

Arogl:

Please take a screen snip of the gui showing the possible books that were being scrubbed when the job failed. It was scrubbing 27, but it was a specific book causing this error. I need to see what its gui row looked like in Q&S. A screen print of the entire Q&S gui is fine as long as all of the work columns are shown. You can edit your post and attach the snips to it if you wish. I will check back.

Temporarily, just skip that particular book when selecting books to scrub.


DaltonST

Subsequent note: Apparently some of the metadata for that particular book was well and truly malformed. I have bulletproofed the area that caused the UTF8 failure in newly released Version 3.3.7, but I still would like to see that book's Q&S metadata, and add it to my testcase library.

[arogl]

OK,

With Book 1 - The plugin now handles the data.
With Book 2 - the plugin turns the series into a "corrupt metatdata" tag.

I'm not sire how to edit my posts as I can not find a button to do so.

Andrew.

Originally Posted by DaltonST

Arogl:

Please take a screen snip of the gui showing the possible books that were being scrubbed when the job failed. It was scrubbing 27, but it was a specific book causing this error. I need to see what its gui row looked like in Q&S. A screen print of the entire Q&S gui is fine as long as all of the work columns are shown. You can edit your post and attach the snips to it if you wish. I will check back.

Temporarily, just skip that particular book when selecting books to scrub.


DaltonST

Subsequent note: Apparently some of the metadata for that particular book was well and truly malformed. I have bulletproofed the area that caused the UTF8 failure in newly released Version 3.3.7, but I still would like to see that book's Q&S metadata, and add it to my testcase library.

[DaltonST]

Version 3.3.8 - 04 January 2015 New Scrubbing Rules for Work Title, Work Series and Tags; UTF8 bug fixes.

Q&S has now been tested with an Eastern European language, Polish, using metadata originally created with Windows-1250 or some other encoding that supports it. Previously it was only tested with Western European languages (English, French, Spanish, Portuguese, German) using metadata originally created via Windows-1252 encoding.



DaltonST

[DaltonST]

Version 3.4.0- 07 January 2015 New User-Maintained "Series Rules Table" that uses regular expressions to change/delete words in Work Series (analogous to the Title Rules Table); Revised User Instructions.

See page 45 of the revised User Instructions. This table does for Work Series what the _title_rules table does for Work Title.

As always, immediately run the "After Version Upgrade" job so that the new table can be created before running Book Level Scrubbing.

The new table comes with 16 seed entries, mostly capitalization regular expression rules. See the attached image.



DaltonST

[DaltonST]

Version 3.4.2- 09 January 2015 New User-Maintained "Tags by Comment Table" to derive Work Tags from Book Comments; Revised User Guide.

See Page 46 of the revised User Guide regarding the new "Tags by Comment" table, _tags_by_comment. This new feature has the ability to create an enormous number of new Work Tags for each book, so the table for it is delivered empty, with no seed entries. You must "opt in" by adding rows to this table. See the attached image for examples.

As always, please run the "After Version Upgrade" job first. Otherwise, the new table will not exist when Book Level Scrubbing runs, causing a SQL operational error.


DaltonST

[DaltonST]

Version 3.5.0- New: Derive Genres core functionality has been integrated with Q&S, using Q&S database tables instead of any .csv files like the generic Derive Genres plugin uses. The User Guide has been revised.

The Q&S Derive Genres does not use any .csv files. The equivalent four (4) tables are maintained by the user within the Q&S library metadata.db using the same tools that are used for the other user-maintained tables, such as Tag Rules, Title Rules, Tag Author Defaults, Tag Priorities, Tags by Comment, etc.

The new Q&S Derive Genres is extremely fast compared to the original, standalone Derive Genres Plugin.

As always, please run the "After Version Upgrade" job since this new functionality brings 8 new tables and 3 new views to the Q&S database.


DaltonST

[DaltonST]

Version 3.5.4- New database version 3.9.0


Please run the "After Version Upgrade" job at the top of the menu first thing to create new tables and views, and to seed 4 new Q&S Derive Genres Rules tables.


If you have not run the "Copy Real Data to Work Data for All Books" job lately, then your Q&S library metadata.db has not been vacuumed/defragmented automatically by Q&S. If so, it would be a good idea to do a Calibre > Library > Library Maintenance > Check Library, which also does a vacuum, after running the "After Version Upgrade" job.



DaltonST

[DaltonST]

Version 3.5.8-New Menu Item: Auto-Populate Tags-by-Comment Rule Table using Tag Rules Table plus Comments of All Books; Revised User Guide.

This time-saving feature runs the Tag Rules Table “oldtags” through the Calibre Comments Table, searching for simple matches. When if finds a match, it uses the “oldtag” as the Comment and the “newtag” as the Tag for new entries it puts into table _tags_by_comment_candidates.

After you review those candidate entries (and delete or change those you don't like), it then moves them to the real _tags_by_comment table. The Book Level Tag Scrubber uses that table, as does Book Level Scrubbing itself.

This makes it easy to derive new Work Tags using your own Tag Rules with your own library’s book Comments.

As always, please first run the "After Version Upgrade" job, since this new feature adds one new table and one new view that have to be created before you can use them.



DaltonST

[DaltonST]

Version 3.5.10 Major enhancements to Author Level Scrubbing; New Book Level Scrubbing Rules for Title and Author; New Database Version 4.1.0.

Please run the "After Version Upgrade" job first thing, as there have been database changes for this release.

Also, it is very important that "good", "valid" Real Authors in Q&S be copied to the Pristine Author validation table using the menu selection at the bottom of the main Q&S menu. Pristine and Global Authors are used to sniff out Authors not where they belong, so as you acquire new books with new Authors, and they are "valid" Real Authors even before you scrub the books, it is best to copy them to the validation tables before scrubbing in order to achieve the benefits of having them available. Of course, garbage/malformed Real Authors are not really Authors, so don't copy them to the validation table; that would make Q&S treat garbage as valid, the opposite of what we want.


DaltonST

[DaltonST]

New "Tag Combinations Rule Table", which Tag Scrubbing uses to derive new Work Tags based on combinations of up to three Work Tag Keywords per Rule; Revised User Guide.

Please run the "After Version Upgrade" job first thing so that this new table may be created before you try to use it.

The User Guide has been revised for this new functionality; see Page 52.



DaltonST

[Knuckles McGinty]

Before I installed quarantine and scrub Calibre would convert my doc.x files to the MOBI format with a working table of contents in both files. After I installed quarantine and scrub my table of contents worked in the doc.x files but not at all in the MOBI files. Clicking on any entry in the TOC in the MOBI file took me back to the first page of my document. Clicking did not take me to my chapter of choice.

What's the problem? I am not knowing enough about HTML to hazard an opinion.

[eschwartz]

Does it start working again if you uninstall Q&S PI?

[theducks]

Q&S has very rigid installation and use rules.
Did you do them precisely?