[GUI Plugin] QuarantineAndScrub

[DaltonST]

Version 3.6.12 New DB Version 4.7.0; Increase in performance of Series Consolidation/Renaming.


Please run the "After Version Upgrade" job near the top of the Q&S menu to update your database to version 4.7.0.


This version vastly increases the speed of the Series Consolidation/Renaming job in huge Calibre libraries (30,000+ books).




DaltonST

[DaltonST]

Version 3.6.16 New DB Version 4.8.0


Please run first-thing the "After Version Upgrade" job near the top of the Q&S main menu in order to upgrade your Q&S SQLite database(s) to Q&S DB version 4.8.0. This job must be run separately for each Q&S library (although most folks have only one), since each library has its own metadata.db file that must be upgraded.



DaltonST

[DaltonST]

Per user request, I have attached a .pdf file, series_derivation_sample_patterns_for_manual_usage .pdf, to the OP at https://www.mobileread.com/forums/sho...48#post2940048 that shows some basic patterns for someone to use if they wish to "manually force" a particular series derivation from just the title (i.e., series is blank).

Q&S was explicitly developed to avoid the need for any such special action by the users for ebooks obtained "in the wild". However, some folks have acquired or created some sketchy metadata that could benefit from a little human help. Examples: organizing paper book libraries or cataloging music files within Calibre.

This is totally optional, and not required for use by anyone.

I certainly don't manually specify anything in Q&S other than via the table maintenance of Tag Rules, Tag Priorities, etc. etc. However, if you want to use it, the example patterns should work fine for you if the punctuation is followed.


DaltonST

[DaltonST]

Version 3.6.19 -16 February 2015 Many New Work Title Scrubbing Rules; Improved Author Level Scrubbing; Revised User Guide; New DB Version 4.9.0.

Please run the "After Version Update" job near the top of the Q&S main menu so your metadata.db database can be upgraded to version 4.9.0 which supports two new views created for the Author Level Scrubbing job.



DaltonST

[DaltonST]

Version 3.6.20 - Author Level Scrubbing Enhanced; New Work Title Scrubbing Rules; Jobnames Improved; New DB Version 4.91.0.

Please run the "After Version Update" job near the top of the Q&S main menu to upgrade your metadata.db to version 4.91.0 for each and all of your Q&S libraries.


DaltonST

[Fiat_Lux]

A) Can this extension change semi-colons in tags to commas, deleting duplicate tags in the process, without user intervention?

Example # 1:
From: «Andrew Gih; Revival; God the Father; Jesus; Son of God; Holy Ghost; Holy Spirit; Revival; train; wind; fire; baptism; filling; Bible; salvation; sin; forgiveness; hell; heaven; New Testament; Acts; Old Testaments; Revelation; School of Prayer; Peter-John Parisis, Son of God, heaven, Revival, salvation, New Testament, Acts, filling, Old Testaments, train, Revelation, forgiveness, Holy Ghost, baptism, Andrew Gih, hell, Holy Spirit, God the Father, Jesus, School of Prayer, sin, bible, Peter-John Parisis, wind, fire»
To: «Andrew Gih, Revival, God the Father,Jesus, Son of God, Holy Ghost,Holy Spirit; Revival, train, wind, fire, baptism, filling, Bible, salvation, sin,forgiveness, hell, heaven, New Testament, Acts, Old Testaments,Revelation, School of Prayer, Peter-John Parisis, Son of God, heaven, Revival, salvation, New Testament, Acts, filling, Old Testaments, train, Revelation, forgiveness, Holy Ghost, baptism, Andrew Gih, hell, Holy Spirit, God the Father, Jesus, School of Prayer, sin, bible, Peter-John Parisis, wind, fire»


B) Can this extension delete, clean up, or otherwise make sense of tags like the following?
Maybe an option to just delete everything here.

Example # 2: «mysticism the path of the initiate the life of the master jesus, interpretation of the gospels, The Annunciations, The Immaculate Conception, The Birth of the Christ, The Wise Men and the Star, the shephards and the sheep, The Manger and the Swaddling Clothes, the gifts of the wise men, Herod and the Wise Men, The Flight and Massacre, Twelve Years of Age, the baptism, the fast, the marriage feast in cana, the betrayal, the crucifixion, three days in the tomb, the resurrection, the ascension, astronomical events, symbolical interpretation, birth dates, inaccuracies and contradictions, The Universal Solar Myth; Basis of all world religions; The Annunciations; To Elizabeth and Zacharias; To Mary and Joseph; Why made by Gabriel; The Immaculate Conception; Why disputed for ages; Joseph the real father of Jesus;The Birth of the Christ; Actual date unknown; Why Dec 25th was chosen; Its meaning in our daily lives; The Wise Men and the Star; How could a star stand still over a certain spot and then turn South? What it really was; Who were the Wise Men;The Shepherds and the Sheep; Revelation made to humble shepherds; Not to the high priests;The Manger and the Swaddling Clothes; Why was Christ born in a stable; Meaning of swaddling clothes; Gifts of the Wise Men; Symbology of gold; frankincense and myrrh; Their meaning; Herod and the Wise Men; Key to unex¬plained experiences in our lives; The Flight and Massacre; Were any children actually slain; How many; Why denied by St Luke; Twelve Years of Age; Where was Jesus during His early years; Why 12 is an important age; The Baptism John; Why did Jesus, Basis of all world religions, The Shepherds and the Sheep, Their meaning, How many, To Mary and Joseph, Its meaning in our daily lives, Where was Jesus during His early years, Were any children actually slain, Revelation made to humble shepherds, Symbology of gold, frankincense and myrrh, Not to the high priests, Gifts of the Wise Men, Meaning of swaddling clothes, The Baptism John, Actual date unknown, Why did Jesus, Why denied by St Luke, Who were the Wise Men, Why was Christ born in a stable, Why disputed for ages, Why Dec 25th was chosen, Why made by Gabriel, Key to unex¬plained experiences in our lives, To Elizabeth and Zacharias, The Universal Solar Myth, How could a star stand still over a certain spot and then turn South? What it really was, Joseph the real father of Jesus, Why 12 is an important age»

C) Can it automatically remove tag(s) like the following:

Example # 3: «The Teachings of the Order of Christian Mystics Set the Inner Spiritual Pattern for Life. Every seeker and initiate is destined to walk the same well trodden path of those that have come before. There is but one universal story of the initiate. The One Pattern Life we are all in the process of unfolding. Based on the Life of the Master Jesus. The most astonishing yet revealing interpretation of the Gospel story. Gives it a new spiritual yet logical and scientific appeal. The result of Dr. Curtiss' 20 years of painstaking and scholarly research in the fields of comparative religion, philosophy, psychology and symbolism. The results are astonishing if not sensational in the religious field. Not dry scholastic argument, but applied to the daily experiences of our lives. Gives the only satisfactory explanation of the many inaccuracies and flat contradictions in the Gospels. Accepts the historical facts recently discovered, but definitely refutes the materialistic conclusions of the "higher critics" and mythologists. A great reconciler of conflicting opinions and theories. Will cause as many happily changed lives as any other of The Curtiss Books. Illustrated with a hitherto unpublished picture of the Master Jesus.»

I don't want to install this extension, and learn how to use it, if it isn't capable of that type of cleanup.

Why, o why does Google Books/Internet Archive/Project Gutenberg not stick to the library standard of three or four subject headings as tags in their books?

Amber

[DaltonST]

@Fiat_Lux:

Yes. See the attached.

Obviously it is up to you to specify what is garbage/trash/undesirable and hence should be immediately purged, and also to specify what each Work Tag that is not purged should be changed into via a Work Tag to a New Work Tag mapping rule. For each, the rule can be a simple 1:1, or can include an automatically created regular expression.

All of these rules remain in Q&S after you have moved your books to your "real" Calibre library after Q&S has finished with them. That is where the incomplete Fiction:Science mapping choice originated.

After scrubbing, a Work Tag Minimization function that uses Tag Priorities (that you define in the Tag Priorities Table ) will reduce the number of Work Tags to the maximum number that you also specify (in Customization).


DaltonST

[camelx]

here's a screen shot after author scrub. Originally I had JRR Tolkien, then I searched and replaced in bulk metadata to J.R.R. Tolkien and I'm still getting all these dirtys.

[matthewc100]

This link goes nowhere.

[DaltonST]

Originally Posted by camelx

here's a screen shot after author scrub. Originally I had JRR Tolkien, then I searched and replaced in bulk metadata to J.R.R. Tolkien and I'm still getting all these dirtys.

Your screen print looks 100% correct to me. Please read the User Guide. It has the steps to follow. All you have done is to perform Author Level Scrubbing. Now, as the next step of many, you must run Book Level Scrubbing. "Dirty" means "ready for Scrubbing".


DaltonST

[DaltonST]

Originally Posted by matthewc100

This link goes nowhere.

Because it points to a very old, obsolete attachment number that no longer exists. That is why it goes nowhere.

You should use the Original Post as your source of downloads.
DaltonST

[camelx]

I did read it, but maybe missed something. I started everything from scratch this morning. I was pretty sure, when reading, that it said - do the author scrub, the status should change to author_ok, then move on to title.

When I didn't see author okay for most of my records, but on a couple, I thought that might have been a problem.

step 8: 5) Scrub for Author at Author Level. The Status of changes will be set to “auth_ok”.

Like I Mentioned, I originally had those authors like: JRR Tolkien, after the author scrub it didn't make any changes to it, was expecting J.R.R. Tolkien.

(I know you clearly mentioned that 1) set author sort and 2) change initials to X.Y.)
that's why I changed prior to clearing out and recopying to J.R.R.

-----

Anyway, why does it set a few entries to author_ok and most are left dirty? (even, as you say, they appear to be fine)

Thanks in advance, I'm gonna love this tool!!

[DaltonST]

Originally Posted by camelx

I did read it, but maybe missed something. I started everything from scratch this morning. I was pretty sure, when reading, that it said - do the author scrub, the status should change to author_ok, then move on to title.

When I didn't see author okay for most of my records, but on a couple, I thought that might have been a problem.

step 8: 5) Scrub for Author at Author Level. The Status of changes will be set to “auth_ok”.

Like I Mentioned, I originally had those authors like: JRR Tolkien, after the author scrub it didn't make any changes to it, was expecting J.R.R. Tolkien.

(I know you clearly mentioned that 1) set author sort and 2) change initials to X.Y.)
that's why I changed prior to clearing out and recopying to J.R.R.

-----

Anyway, why does it set a few entries to author_ok and most are left dirty? (even, as you say, they appear to be fine)

Thanks in advance, I'm gonna love this tool!!

@camelx:

The job log for Author Level Scrubbing (and indeed, all job logs in Q&S) explain what they did and why. Sometimes they also explain what they did not do and why they did not do it. They also provide helpful advice and other information when useful.

You must review each and every job log in Q&S to understand your results, preferably while the job is still running.

If you examine your top 3 books with a status of "auth_ok", you will see that their original titles had the Author name commingled with the real title. Author-Level Scrubbing removed the Author from the Work Title. None of the other books had that issue. The job log will have printed the Old Work Author, Old Work Title, New Work Author, and New Work Title as of the end of the job. Next, when you run Book Level Scrubbing for those books, it will pick up where the former job left off and complete the scrubbing of the New Work Author and New Work Title, as well as Work Series, Work Series Index, and Work Tags.

The very bottom of the Author Level Scrubbing job log also shows the total number of books that had a Work Author and/or Work Title changed. That count can be reconciled to the details of what was changed in the previous lines of the job log.

Only books that are changed by Author Level Scrubbing have their Status changed to "auth_ok". If it does not change anything, it does not change the Status. That is why the User Guide says: "The Status of changes will be set to “auth_ok”".



DaltonST

[DaltonST]

QuarantineAndScrub: A Calibre Add-on

Application Summary

1. Calibre's “Plugin Updater” summary description of Q&S:
   
   
   • [For Advanced Calibre Users Only: Must Download Special Database From mobileread.com]
     
   • Clean Malformed, Misplaced or Missing Author, Title, Series, Index, ISBN;
     
   • Tag Scrubber, Deriver, Propagator, Defaulter & Minimizer;
     
   • Series Deriver & Consolidator;
     
   • Series & Index Validator/Corrector;
     
   • Genre Deriver;
     
   • Library of Congress Code & Dewey Decimal Code Deriver;
     
   • Book Awards;
     
   • & Much More:
     
   • Author Pseudonyms
   • Automatic Author Renaming
   • Derive User-Defined Custom Columns from Title
   • Derive User-Defined Custom Columns from Tags
   • Special Manga Metadata Processing
   • Special Graphic Novel Metadata Processing
     
   
   
2. Total pages in its two-volume User Guide: 81
   
   
3. Number of Q&S-specific, non-Calibre SQLite Tables: 64
   
   
   • Number of User Maintained Q&S Tables: 18
     
     
   • Number of Q&S Maintained Q&S Tables: 46
   
   
4. Number of Q&S-specific, non-Calibre SQLite Views: 55
   
   
5. Number of Menu Choices: 63
   
   
6. Number of Foreground GUI Update Transactions: 34
   
   
7. Number of Background Update Jobs: 19
   
   
8. Number of Current Users: approximately 589
   
   
9. Percentage of Successful Users Who Read and Followed at Least the 1st Volume of the User Guide: 100%
   
   
10. Percentage of Total Calibre Users to Whom This Add-On is Targeted: : <0.001%
    
    

DaltonST

[g25]

Hi, I just tried your plugin with the latest version of calibre - Version: 2.37.1 that just came out. I followed the user manual exactly with the plugin and the Quarantine library, etc. and follw the the steps in "Basic Steps in Using Q&S" and get to step 7:

"Copy All Q&S Dirty Data to Work Data. The Status will be set to “dirty”."

It copies the data over but DOES NOT set any status. Status remains empty.
I get this error in the log:


Starting job: Quarantine And Scrub: Copy Real to Work
C:/Users/J/Documents/QuarantineAndScrub_OfficialLibrary/metadata.db
Fri Sep 04 20:41:05 2015
Python: Windows CPython 2.7.9
SQLite Version: 2.6.0
SQLite DB Locking Mode: EXCLUSIVE [GUI Is Locked]

══════════════════════════════════════════════════ ══════════════════════════════════════════════════ ═════════════════════════════════════════════
Beginning Copying Real to Work Data
══════════════════════════════════════════════════ ══════════════════════════════════════════════════ ═════════════════════════════════════════════
Erasing All Work Custom Column Data
Populating All Work Custom Columns From Real Metadata
Job: "Quarantine And Scrub: Copy Real to Work" failed with error:
Traceback (most recent call last):
File "site-packages\calibre\gui2\threaded_jobs.py", line 83, in start_work
File "calibre_plugins.quarantine_and_scrub.copywork ", line 617, in util_copy_original_metadata
OperationalError: no such function: title_sort

Called with args: (<calibre_plugins.quarantine_and_scrub.ui.Quaranti neAndScrub object at 0x08A2A030>, <calibre.db.legacy.LibraryDatabase object at 0x06F05A70>) {u'abort': <threading._Event object at 0x0CA7E2F0>, u'log': <calibre.utils.logging.GUILog object at 0x0CA7E3B0>, u'notifications': <Queue.Queue instance at 0x09125F58>}


I tried purging all work data and doing it again but same result.
Was there a change to Calibre with the latest Calibre update? Please help, thanks!