[GUI Plugin] QuarantineAndScrub

[DaltonST]

@g25:

Yes, there seems to have been a change in the usage of a Standard Calibre User-defined SQLite Function, which is what is being referenced. No biggie. I will fix Q&S and upload a new release of Q&S shortly.

Thanks for letting me know.


DaltonST


n.b.: Version 3.6.92 has been uploaded.

[g25]

I notice during the "Scrub Work Data at Book Level [Selected Books]" it tends to get stuck at 99% doing miscellaneous scrubbing (or whatever).

I then have to manually stop the job. The books don't update. But then I exit out of Calibre (Database locked error) and start it back up and the books have been scrubbed with a "book_ok" status and they look fine. Is the miscellaneous scrubbing supposed to take so long? Is it because I am doing a huge 6,000 book library? Even though it hangs for several minutes at the 99% mark even when book level scrubbing only a couple of books? It seems that if I was able to purge the work data of all my *selected books* I could just transfer the 3000 book_ok I had finished over to pristine. But I don't want to do just 100 at a time for 3000 books! I am afraid if I purge *ALL* work data it would also purge the 3000 dirty ones I had already done the author scrubbing for :/

Anyway for now I am book level scrubbing the last 3,000 books and after that purge all work data and transfer the entire 6,000 to the pristine library.

[DaltonST]

@g25:

Several comments that you should heed:

I notice during the "Scrub Work Data at Book Level [Selected Books]" it tends to get stuck at 99% doing miscellaneous scrubbing (or whatever)

No. It is not "stuck". At all. It is doing important, miscellaneous things in pure SQL that is "above" Python, so there is no way to communicate its progress back to Calibre's job notification. It does what it does at the Library Level for all books that have a status of any type. When it says "doing miscellaneous scrubbing", that is exactly what it is doing.

I then have to manually stop the job. The books don't update. But then I exit out of Calibre (Database locked error) and start it back up and the books have been scrubbed with a "book_ok" status and they look fine.

Of course they have a "book_ok" status, because you will note from the Job Log (that you must watch as the job runs), Book Level Scrubbing finished. The total count of books scrubbed at that point is identical to the original unscrubbed number. You wrongly stopped the job after Book Level Scrubbing finished but before Miscellaneous Scrubbing finished.

I then have to manually stop the job. The books don't update. But then I exit out of Calibre (Database locked error) and start it back up and the books have been scrubbed with a "book_ok" status and they look fine.

No, you do not "have to" manually stop the job. That was a personal decision on your part. Who knows what did not get fixed as a result. Why not just unplug your pc? Same result.

Is it because I am doing a huge 6,000 book library? Even though it hangs for several minutes at the 99% mark even when book level scrubbing only a couple of books?

It does not "hang". It is doing many different things as shown below. Those marked "Entirely in SQL" mean that it does not "report back" its progress. That does not mean it is "hung".

• add_tag_combinations - * Entirely in SQL
• apply_tag_string_replacement_rules - * Entirely in SQL
• __delete_unused_values
• __set_null_seriesname_index_to_zero
• _insert_single_tags_into_work_tags_single
• _convert_identifiers_isbn_from_10_to_13
• change_work_title_to_web_title_for_previously_vali dated_work_series - * Entirely in SQL
• miscellany_change_work_index_to_web_index_standalo ne
• apply_author_mapping_rules - * Entirely in SQL

As the User Guide says, scrub your books from start-to-finish in Batches. That means there should be no books whatsoever in the "current" Q&S library that you do not want to be scrubbed simultaneously, or, if that is impractical, all but a batch-size should not have any Work Data whatsoever. Q&S jobs ignore all books with no Work Data. It also says that in the User Guide.

Author Level Scrubbing is for all books that have a status of "dirty". Period.

Miscellaneous Scrubbing is for all books in the current Q&S that have any non-null status.

You should not have copied "real to work" for all 6000. That is the root cause of your problem.


I have 5 "production" Q&S Libraries, which you would see in the Configuration screen, the User Guide, and some of the .txt attachments in the Original Post.

• QuarantineAndScrub_prefilter - where I "Add Books", convert them to epubs, add ISBN, modify EPUBs, count pages, and so forth. Then, they are moved to _prescrub.
• QuarantineAndScrub_prescrub - where I hold them after _prefilter, and then move them to the next library when I feel like working on one or some of them. Some I work on in this library because I am impatient to read them. Obviously I copy all of my "rules" from my "best" Q&S library prior to that. You will note there is a special job just for that under "Special Tools".
• QuarantineAndScrub - where all Author Level and Book Level Scrubbing is done. I then "copy to work to real" metadata for the entire Batch. Afterwards, moved to _postscrub.
• QuarantineAndScrub_postscrub - where scrubbed books go for final spot-cleaning. I very likely will (again) copy "real to work", but this time the "real" is the "green" metadata from the prior library. Here I will do Tag Scrubbing and Tag Minimization, Consolidating Series Names, Deriving Genres, Deriving Library of Congress and Dewey Decimal Codes, Pseudonyms, and so forth. When I have finished with a book, I move it to my "pristine" library.
• QuarantineAndScrub_CalibreMain - my "pristine" Calibre Library. I work on it in batches of ~200 or so to tweak the metadata of books that I had prior to my creating Q&S for myself to speed things up. Q&S was for my personal use in the beginning. New books start at _prefilter, but my "original" books are worked-on in situ.

At this point, you of course do not want to lose what scrubbing you have done so far. Off the top of my head, this is what I think I would do if I were in your bind (thankfully I am not). Create a copy of your single Q&S Library, books and all, and then paste that into a new Q&S Library called "QuarantineAndScrub_prescrub". Then, switch back to the original one you have been working in, and delete the 3,000 books that you have not yet started. Then, finish the good ones and move them forward. Then, switch back to the _prescrub library and delete the ones that you just moved forward. Finish them there, and move them to the end, bypassing the "real" Q&S library. There are variations on this that I would do, such as doing them in batches of 500 instead of one batch of 3000. That is up to you.

Good luck.

DaltonST


[Additional Comments:]

But I don't want to do just 100 at a time for 3000 books!

Who said you had to do just 100 at a time for 3000 books? You can execute the "copy real to work for "selected books only" as many times as you wish. I prefer batches of 200-500, myself.

[g25]

No I meant PURGING work data for only 100 books at a time or the entire library. But I see that I should not bee doing more than about 500 at a time anyway :P

I can see my primary mistake was attempting to do all 6,000 books at once. You see the author level scrub I initially did, I did for ALL 6000 books at once (overnight while sleeping). It left almost half of them still dirty and not author_ok (or whatever). BUT! The authors HAD been corrected. In any case it is now doing book level scrubbing for the last 3000 as I just now see your reply :/ Its almost done so I don't see the point in stopping it :/

I suppose I could copy the entire 6,000 lot to a prescrub library when finished and do the entire process over again in batches of 500 to make sure everything is pristine?

My primary use for this plugin was to correct bad author and title names which were horribly messed up in my pdf folder. The series function is also great, but goodreads worked just fine for my series in my other library. I still have to download the tags, comments, and book covers (I only use goodreads). I have the goodreads plugin configured to tag exactly how I want it so I don't need the tagging functionality of Q&S except maybe for isbn if isbn plugin fails to extract isbn.

[DaltonST]

Originally Posted by g25

........ You see the author level scrub I initially did, I did for ALL 6000 books at once (overnight while sleeping). It left almost half of them still dirty and not author_ok (or whatever). BUT! The authors HAD been corrected. ......

As the User Guide says on Page 11 in item (5), the Status of books will be changed to "auth_ok" only if Author Level Scrubbing changes the Work Author itself. It will change Work Title to remove bits and pieces of the Work Author name regardless of whether it had to change the Work Author itself. The Job Log will have shown what was changed (From:To) for each of the many scenarios for which it finds metadata related (directly or indirectly) to Work Author that needs fixing.

The vast majority of your books will retain their Status of "dirty" at the end of Author Level Scrubbing. The minority will have been changed to "auth_ok".

I refer you to the following May 16, 2015 post in this thread that has a picture of what is 100% normal to see at the end of Author Level Scrubbing: https://www.mobileread.com/forums/sho...64#post3103064

Book Level Scrubbing only looks at books with a Status of "dirty" or "auth_ok". The so-called "at: 99% Doing Miscellaneous Scrubbing", which rides on the coat-tails of Book Level Scrubbing to ensure it gets executed at the right time and in the right place (which is why it is not its very own job to begin with), will look at any non-null Status: "dirty"; "auth_ok", "book_ok".

Pay much less attention to Status and a whole lot more attention to what the Job Log says. I refer you to what the Original Post says about that:

Q&S requires that you closely review every Q&S Job Log, preferably while it is executing. This is for many reasons, all of which are for your benefit. If you do not wish or intend to review each and every Q&S Job Log, and follow any instructions or take note of any warning or error messages they may provide, then Q&S will not work properly for you.

The User Guide also says to review every Q&S Job Log.

Given what you have been doing, I suggest that you run the Standard Calibre "Library > Library Maintenance > Check Library" function, which will automatically defragment/compress the metadata.db file. It will have grown enormously by now.


DaltonST

[DaltonST]

@NewUsersOfQ&S:

Please do yourself a huge favor and read these Q&S OP attachments before you begin to do anything with Q&S. They will save you a great deal of time, energy and frustration:


https://www.mobileread.com/forums/att...6&d=1424120599


https://www.mobileread.com/forums/att...3&d=1435073179


Also, it would be a great idea to peruse these posts after you have thoroughly analyzed and digested the above 2 attachments to the OP:

https://www.mobileread.com/forums/sho...65602#poststop

https://www.mobileread.com/forums/sho...64#post3103064

It goes without saying that you must read the User Guide after you have read the above.

You should also reread the Original Post, since it has very important tips embedded within it. https://www.mobileread.com/forums/sho...40048#poststop


Afterwards, please "learn the Q&S process" with your new Q&S library when it has only a few (~ 10, not 100 or 1,000 or 5,000 or 10,000...) copies of random or representative books in it, each with a different author. Take them through the "process" from start to finish. Along the way, you will have questions. Please reread the attachments and links listed above. Then, add a few more copies of your books that have different types of issues that you want Q&S to solve, and run them through the process too. Again, you will have questions, and should again reread the User Guide and other information.

Taking "baby steps" when first learning Q&S pays huge dividends later.


Thank you in advance.


DaltonST

[axewood]

i dont understand a bit of how to install this. Just the first instructions:

"QuarantineAndScrub_official_library_to_extract_re n ame_to_zip.jar file, place it in the parent directory of your Calibre libraries, rename the extension from .jar to .zip, and "Extract (it) Here" automatically into its special Calibre Library subdirectory, QuarantineAndScrub ."

Where the Hell do i get this jar file. The jar file has a space in the name? Why the hell is rename spelled "ren ame"? No clue. Then we get a whole page of instructions in different letter types and colors. I'll be honest, i'm trying to help my girl friend who wants this add-on, she asks my help because i'm an it guy for 30 years, but i can't help her. Incomprehensible instructions are all i get.
Sorry girlfriend!

[PeterT]

Go to the first post of this thread. ALL the files are attached to it. Unfortunately, the standard calibre plugin installation will only download the main plugin file; not the additional ones.

[eschwartz]

I would guess that:

QuarantineAndScrub_official_library_to_extract_rename_to_zip.jar

is actually:

QuarantineAndScrub_Official_Library_To_Extract_v496b_with_qs_service_pack_incl__rename_to_zip.jar

But the instructions haven't been updated. (Never mind that the filename need not have changed???)


@PeterT -- I suggested to the author before, that those could be contained and self-extracted by the plugin (or else downloaded *by* the plugin) which would make it less awkward.
Apparently barriers to entry are desirable for weeding out the non-technical and uncommitted.

[DaltonST]

The Q&S OP is required reading, and comprises an important part of the User Guide. Automatically downloading anything whatsoever for potential Q&S users would undermine their absolute need to educate themselves before clicking one of the 63 menu choices on the Q&S main menu.

The Standard Calibre "Plugin Updater" icon that opens to a modal dialog named "User Plugins" has a hyperlink to the "Plugin Homepage" in the lower right corner in plain view.

The Q&S Original Post clearly states the following:

• "This add-on is intended for use only by "Advanced Calibre Users".
  
• "Q&S is designed for users who have already been using Calibre for quite some time, so the Q&S User Guide assumes the reader is already familiar with Calibre."
  
• "99.9% of Calibre users do not need Q&S, and should not install it."
  
• "Percentage of Total Calibre Users to Whom This Add-On is Targeted: <0.001%"

I do not want anyone who cannot even get past the entire OP (much less the first few paragraphs) without becoming frustrated to even try to install Q&S, because I am the person they will inevitably contact. Indeed, that is what has just now occurred.


DaltonST

[axewood]

Not sure that is what has happened. When the first line is :
"Easy to install (but only if you follow the directions). Download the QuarantineAndScrub_official_library_to_extract_ren ame_to_zip.jar", again a filename that probably is not correct, then you can claim that i am the correct person trying to use this tool. I'm rather of the opinion that a programmer is not the person per se to write an instructions manual.

[davidfor]

Originally Posted by DaltonST

The Q&S OP is required reading, and comprises an important part of the User Guide. Automatically downloading anything whatsoever for potential Q&S users would undermine their absolute need to educate themselves before clicking one of the 63 menu choices on the Q&S main menu.

The Standard Calibre "Plugin Updater" icon that opens to a modal dialog named "User Plugins" has a hyperlink to the "Plugin Homepage" in the lower right corner in plain view.

The Q&S Original Post clearly states the following:

• "This add-on is intended for use only by "Advanced Calibre Users".
  
• "Q&S is designed for users who have already been using Calibre for quite some time, so the Q&S User Guide assumes the reader is already familiar with Calibre."
  
• "99.9% of Calibre users do not need Q&S, and should not install it."
  
• "Percentage of Total Calibre Users to Whom This Add-On is Targeted: <0.001%"

I do not want anyone who cannot even get past the entire OP (much less the first few paragraphs) without becoming frustrated to even try to install Q&S, because I am the person they will inevitably contact. Indeed, that is what has just now occurred.

I think eschwartz was referring to the fact that the first post has instructions to download a filed called "QuarantineAndScrub_official_library_to_extract_re name_to_zip.jar" but that file doesn't exist in the first post. The filename is mentioned twice. When I looked at the list of attachments to that first post, I might have guessed it was the other file that eschwartz referred to, but I still couldn't be sure.

You make a big fuss about reading the user guide, but I can't find them. Well, there is a reference to using the menu options "Read Q&S Instructions", but I have to install it first. That means I have to do a bit of work that will go to waste if I read the first paragraph those instructions and realise I don't want to use the plugin. With the number of attachments to the first post, I'm surprised you don't have it there. Or am I being confused by the fact you refer to a "User Guide" several times and "User Instructions" elsewhere. As a title, they sort of sound the same, but I'm sure that my grammar and English language loving wife would point out that they should be very different things. And if the attachment "quarantine_instructions.pdf" is the user guide Q&S instructions, how about naming it consistently with everything else.

The first post of this thread is disorganised, has erroneous and conflicting info and doesn't clearly state what this plugin can do. If the intention of the first post is to repel people who would find this plugin useful and can actually follow clearly worded instructions, then it is probably doing a good job. But, if you want those people, then you should clean up the first post. A simple "Who this is for/not for" statement near the start. And removing insults to potential users ("emulate one Einstein") is probably a good idea. Or a least a polite one.

[DaltonST]

The following quotes are spot on, correct, well-articulated, accurately reflect facts, and are valid corollaries to: "Percentage of Total Calibre Users to Whom This Add-On is Targeted: <0.001%".

Apparently barriers to entry are desirable for weeding out the non-technical and uncommitted.

If the intention of the first post is to repel people who would find this plugin useful and can actually follow clearly worded instructions, then it is probably doing a good job.

DaltonST

[eschwartz]

... by which you mean, of course, byzantine instructions liberally overdecorated with garish colors, containing false (and occasionally condescending or simply insulting) instructions?

Yes, I suppose that would be an effective way of limiting your users.



But silly me thought that people tend to create free tools for the sake of anyone who wants to use them -- and can RTFM.
But if TFM is full of lies designed to "test" people for worthiness, then there seems no rhyme or reason To Whom This Addon-On Is Targeted.

Maybe the target users are "people who enjoy playing games with the developer"?

[axewood]

hear hear