[GUI Plugin] Quality Check

[paulfiera]

Originally Posted by kiwidude

@paulfiera - no, obviously not yet. Thousands of lines of code related to Modify ePub has soaked up most of the last few weeks, and there have also been multiple releases of Count Pages, Generate Cover, & Reading List. Then add to that Extract ISBN got broken with calibre 0.8.53... oh yeah, I do have a day job too...

No problem at all

[kiwidude]

Originally Posted by cecdc7731

Does the ASIN Check look at EXTH 113, 504 or both? Is there a way to designate which on is checked as I have been editting with mobi meta editor and putting matching numbers from Amazon in both as I go along. I would like to be able to do more edits as I have free time than just for books I am fixing to put on my Fire. Is there a way to add a quick fix that would make 113 and 504 the same?

I don't have a Fire and what I know come from what others have told me, mostly on this thread. I was asked only to check EXTH 113 for the ASIN check. It is my understanding that 504 is only of interest when you are trying to "fix" the problem. If others confirm that actually the ASIN check should be looking for the presence of an ASIN in *both* 113 and 504 then I will change the plugin to do so.

Also, can the ASIN be extracted directly from Amazon Kindle versions in the metadata editor rather than or in conjunction with the ISBN number?

No real idea what you are asking for here, but it doesn't sound anything to do with this plugin.

[kiwidude]

There's a few things in this one that may be of interest to some of you...

For Kindle Fire users - two new options:

• Check MOBI -> Check Twitter/Facebook disabled - looks for an identical ASIN present in both EXTH 113 & 504, if not found displays the book.
• Fix -> Fix MOBI ASIN for Kindle Fire - gets whatever asin/amazon identifier you have associated with that book and writes it into EXTH 113 & 504 (uses a random id if no identifier found). Also sets cdetype in 501 to EBOK. Should save users faffing around with external MOBI editors (or doing a conversion) provided it works of course (I don't have a Fire to test it with so please feedback to me).

@paulfiera, two options of interest:

• Check ePub -> Check calibre SVG cover
• Check ePub -> Check no calibre SVG cover

There is also an important bug fix to the "Check Adobe inline .xpgt links" feature to match a fix I did for Modify ePub recently. And a few other things got renamed/cleaned up along the way which will be in the official change release notes.

Let me know how you get on, thanks.

[kiwidude]

I've updated the beta in the post above to increase performance of a couple of the checks. Also another new option:

• Check javascript <script> - looks for ePubs with either .js script files or inline <script type="text/javascript"> blocks

I've also added a new search option to the "Search ePubs..." dialog to allow searching the filenames in the zip (to allow finding an unmanifested file by name).

[ElizabethN]

Originally Posted by kiwidude

There's a few things in this one that may be of interest to some of you...

For Kindle Fire users - two new options:

• Check MOBI -> Check Twitter/Facebook disabled - looks for an identical ASIN present in both EXTH 113 & 504, if not found displays the book.
• Fix -> Fix MOBI ASIN for Kindle Fire - gets whatever asin/amazon identifier you have associated with that book and writes it into EXTH 113 & 504 (uses a random id if no identifier found). Also sets cdetype in 501 to EBOK. Should save users faffing around with external MOBI editors (or doing a conversion) provided it works of course (I don't have a Fire to test it with so please feedback to me).

my work schedule got changed at the last minute so I can't play with this until tomorrow evening. although sleep is overrated, so maybe I can manipulate a few files tonight...

[paulfiera]

Originally Posted by kiwidude

@paulfiera, two options of interest:

• Check ePub -> Check calibre SVG cover
• Check ePub -> Check no calibre SVG cover

Let me know how you get on, thanks.

Working great. It found about 1000 epubs with no calibre SVG cover.

I didn't know that the option "Insert or replace cover" in the Modify epub plugin would also update covers to SVG covers without doing a calibre conversion. Very cool indeed

[kiwidude]

Changes in this release:

• Fix bug in "Check Adobe inline .xpgt links" due to ordering of attributes dependency
• Add a "Check calibre SVG cover" and "Check no calibre SVG cover" options to Check ePub Structure menu
• Add a "Zip filenames" scope to the "Search ePub" dialog to allow searching for files of a specific name
• Add a "Check javascript <script>" to Check ePub Style menu
• Add a "Check Twitter/Facebook disabled" option to Check Mobi menu to look for an equal ASIN in EXTH 113/504
• Add a "Fix MOBI ASIN for Kindle Fire" option to the "Fix" menu to set the EXTH 113/504 fields to your asin/amazon identifier, or a random identifier if not present, plus set cdetype to EBOK
• Rename the various epub "TOC" checks to include the word "NCX" to reflect that TOC they refer to
• Ensure the various NCX TOC checks do not attempt to run on DRM encrypted books
• Do not show error message if unable to parse metadata/ncx files due to incorrect encoding

Those of you who tried the beta, please force an update of the latest version via the plugin updater - there are no code changes but the help file has been updated.

[pwr]

Hi kiwidude,

Love your plugin, very useful. One small suggestion for the 'Fix MOBI ASIN' feature -- I think it could also apply to AZW and AZW3 files as well, not just MOBI. I think the basic formats are the same (at least as far as the EXTH header), and even on encrypted files these should be readable. Also, rather than generating a new UUID ASIN for the book, why not use the Calibre internal UUID?

Cheers,
-pwr

[kiwidude]

@pwr - as I have posted previously I don't have a Fire so those check and fix options related to MOBI have been made "blind" based on other people's suggestions, and the question of other formats like AZW and AZW3 hadn't come up yet. I don't use those formats personally so I know nothing about them.

What QC options would such a change apply to - does that mean all of those "Check Mobi" menu options as well as the "Fix MOBI ASIN" one? And how sure are you about the EXTH headers being the same - can anyone else confirm it?

I am also wondering if as per this post by Kovid I should add another Fix option to allow *removing* the ASIN/resetting the cdetype? Because it sounds like for *some* books due to bugs in the Kindle Fire then running the Fix MOBI ASIN option is going to give you a book that will crash? And that in such a case Kovid is recommending a conversion to effectively reverse those EXTH values? I really haven't bothered paying attention to any of this, the 4G Kindles were of no interest to me personally so if anyone has an opinion feel free to share it, I am not motivated to go hunting through loads of threads myself about it. Perhaps it may just be a short term thing and Amazon pull finger with a firmware release to fix it making it not worth worrying about.

As for the UUID - to be brutally honest I had forgotten that a calibre book even had one . That is certainly an easy change to make.

[pwr]

Originally Posted by kiwidude

@pwr - as I have posted previously I don't have a Fire so those check and fix options related to MOBI have been made "blind" based on other people's suggestions, and the question of other formats like AZW and AZW3 hadn't come up yet. I don't use those formats personally so I know nothing about them.

Actually, the AZW is basically MOBI with (optional) encryption of the book payload (not metadata). AZW3 is also mostly the same, but the payload is in Amazon's KF8 format. And all the book samples from Amazon, and all the books that Calibre generates (MOBI, MOBI+KF8 hybrid, KF8) have the same MOBI headers and the EXTH header.

I'm not really sure how far the 'mostly the same' part goes for these formats, but before knowing about Quality Check I've written a small function to check the ASIN for a project of mine. It works fine on all my books, no matter the extension, whether they were generated by Calibre or imported from elsewhere, etc.

Btw, KF8 is supported by the latest software updates on Kindle 4 and newer.

Originally Posted by kiwidude

What QC options would such a change apply to - does that mean all of those "Check Mobi" menu options as well as the "Fix MOBI ASIN" one? And how sure are you about the EXTH headers being the same - can anyone else confirm it?

As long as the file has the EXTH block, I guess all would apply. But my knowledge of EXTH is based on the wiki, so if Amazon gets 'creative' with it... .

Cheers,
-pwr

[kiwidude]

Changes in this release:

• Now requires calibre 0.8.57
• Store configuration in the calibre database rather than a json file, to allow reuse from different computers (not simultaneously!)
• Add a support option to the configuration dialog allowing viewing the plugin data stored in the database
• Fix bug which intimated it was possible to exclude books from "Check missing" type checks
• Rename "Fix MOBI ASIN for Kindle Fire" to "Fix ASIN for Kindle Fire"
• Enhance "Fix MOBI ASIN for Kindle Fire" to now handles MOBI, AZW and AZW3 formats, iterating all if multiple for a book
• Fix "Fix ASIN for Kindle Fire" so in scenario of not having an ASIN uses calibre's uuid for the book rather than generating a random one
• Change all the "Check Mobi" checks to now work with MOBI, AZW or AZW3 formats

This is the last of today's plugin updates based around calibre 0.8.57, taking advantage of a new feature to store plugin settings inside the calibre library. The benefit of this is for users that store their library in a shared network location and use calibre from multiple machines (though never at the same time - that limitation hasn't changed!). Your plugin configuration settings which are specific to each library (such as custom column names, book exemptions, reading lists etc) will be automatically kept in sync when calibre is next opened on each machine.

Specifically for this plugin it means that exemptions to checks you configure do not have to be repeated on each machine you open the library on.

The other main change was for all of the MOBI related Check xxx & Fix ASIN features to also be run/performed on AZW/AZW3 files. Note that I am doing this "at your own risk" based on the suggestion of pwr above - if anyone reports an issue I can easily disable it.

[GeckoFriend]

I updated this plugin today, ran the check for excess tags, and received the following error message:

Spoiler:

calibre, version 0.8.57
ERROR: Unhandled exception: <b>AttributeError</b>:'module' object has no attribute 'STORE_NAME'

Traceback (most recent call last):
File "calibre_plugins.quality_check.action", line 182, in perform_check
File "calibre_plugins.quality_check.check_metadata" , line 42, in perform_check
File "calibre_plugins.quality_check.check_metadata" , line 340, in check_tags_count
AttributeError: 'module' object has no attribute 'STORE_NAME'

Is this a bug or something I need to fix?

[kiwidude]

Changes in this release:

• Bug fix for check tags count.

[GeckoFriend]

Small issue:

Running "check javascript" returns positives with the comment "marked:epub_address_tags"

[kiwidude]

Sigh, copy/paste coding strikes again . It will be fixed whenever I release the next version, whenever I next work on it again, thanks.