[GUI Plugin] Quality Check

[ElizabethN]

Originally Posted by kiwidude

Thx Wallcraft. Sounds like I should include them in the "missing" results then, which they are not currently.

Which brings me onto some more questions:

1. Is there a need for separate ASIN and EBOK checks? What would people prefer - as it is now or a single check which had the details of what was missing in the log? I don't know what I would call a single check though. If I called it something device specific like "Check invalid for Kindle Fire" then it causes a headache when Amazon releases its next generation of devices...
2. ElizabethN found an interesting case - catalogs generated by calibre. If you generate in MOBI, it sets the cdetag to be NWPR. What happens to these on a Kindle Fire? Is that desirable? Should I exclude books in your library with the "Catalogue" tag so they don't end up in the results, or should they be included?
3. How are users going to "remedy" mobi files that need changing? Obviously a MOBI->MOBI conversion is one way, though potentially extreme. I see ElizabethN is mentioning a manual process of a third party tool. It seems to me that perhaps there could either be an addition to the "Fix" menu, or maybe even a "Modify MOBI" plugin. I am hesitant to suggest the latter because it opens up all sorts of cans of worms for the future (like people wanting to then be able to do similar things they can do with Modify ePub such as updating metadata, covers, smarten punctuation etc). That is way more than I want to think about (and since I always keep ePub and my MOBI files are converted from that master ePub I personally would never use it which makes me even less enthusiastic about the time spent on it). So "maybe" an entry on the Fix menu of this plugin is something I would consider... maybe...
4. If automating such a "fix", replacing the cdetag with EBOK should be straightforward. But what about a missing asin? Where does that come from? Does the value matter?

(1)I would prefer keeping the 2 checks separate - cdetype and asin. The info that I get from each I can use in different ways when updating files for the fire or for the keyboard.

(2)"catalogue" tags could flag in the "not ebok" tag result, looking over the log gives more specific results like the NWPR tag. Surprisingly I didn't have flags in the report as PDOC or EBSP (sample) and I thought that I had a few of each in my library. So I probably wouldn't want non-ebok tags excluded in the results as long as I can review the log. Can the log list results in groups of cedtype - none, pdoc, nwpr - or does it do that already?

(3)fixing the problems - if I have my epub and/or mobi master file, I'd just do an AZW3 conversion as that has resulted in fire "book" files each time (so far). for the few books that can't be converted yet to AZW3 (at least one error in the original that crashes the conversion per calibre), either a mobi-mobi conversion or manual update is usually sufficient.

If there was an easy (for you) addition to the results that would enable the file(s) to be fixed, that would be great. Otherwise, I have no need to modify most mobi files other than the EXTH header, annoyances within documents I usually fix in an epub file then convert. 1 vote for NO modify mobi plugin.

(4)cdetype could be a simple change to EBOK (or PDOC depending on need).

ASIN is not as simple a fix depending upon your desired result. If you just want something in field 113 so that you can send the file to books on the fire or sync on the keyboard, "any" value seems to work. I think calibre conversions result in some type of uuid (??) in field 113.

If you want to "share" on facebook or twitter at the end of the book using the keyboard, then field 113 has to be the ASIN specific for the kindle store ebook, plus the same ASIN has to be in field 504 also.

Bottom-line, I'm happy to go with the decisions of the plugin creator. The current additions have been very helpful.

My very long-winded thoughts, now off to work.

[kiwidude]

@ElizabethN - thx very much for that, just the info I needed.

And no it doesn't sort in the log, but since it is writing out as it processes its not particularly "convenient" to do so. As I mentioned in a previous post you can exclude books from the results, so if you know there are specific books that you don't want in the results just add them to your exclusion list.

So it looks like the only change I need to make in the version you have is to include books that have no EXTH headers in the results. The ability to "fix" mobis I shall leave as an exercise to the reader at this point...

Does anyone have any further feedback on the "Search ePub" feature? Functionally complete/happy with it?

[CRussel]

Originally Posted by kiwidude

"GR currently-reading - calibre" is an ebook in your library, presumably a catalog of some kind? The author being calibre.

Did you find any books in your log that did not have EXTH headers at all (they show as a red error in the log results). I wasn't sure what to do about those (or what that means). Does a MOBI without an EXTH work on the Fire, or should I include it in the output results of the check as a file the user must do something about?

Yes, it found some of those. But in all cases, they were also ones without EBOK tags or ASIN numbers. So the MOBI to MOBI conversion fixes the problem.

What I found interesting is that it identified books with no ASIN, even though I had edited my metadata for the books to include ASIN. But they were all books bought from third party sources, AS MOBI, so while Calibre had the ASIN, it hasn't become an actual part of the book. A conversion fixed the file and all is well.

[CRussel]

Originally Posted by kiwidude

Ok, here is a new beta version to replace the one I posted earlier. This has my first attempt at the checking of MOBI files for EBOK and ASIN, which was requested recently on this thread by CRussel, for users who have a Kindle Fire.

I don't have such a device myself so I have had to guess at a few things in conjunction with Kovid's tips in his post, but I *think* it is right. Hopefully someone (CRussel?) can give it a whirl and let me know if the results you see gel with what you know about books that do and don't place correctly on your Fire.

Works, and results correspond to which books don't show right or sync right on the Fire. (No ASIN := no sync, phoney ASIN := No rating/twitter.)

[nynaevelan]

Originally Posted by kiwidude

So you are saying you have docs which are NOT of type EBOK, but you don't want to change them, right?

Just run the check, select the books you want to not see again and use the "Exclude from check..." option

Thanks, don't know how I missed that.

[kiwidude]

Here is a new version to replace that I posted a few days ago and includes the tweak to the Check Mobi options so that mobi files with invalid headers also show up in the QC results.

This version also has a new check as per the screenshot of "Check broken image links". This will allow you to find epubs that contain <img> or <image> tags where the image being pointed to is not present in the ePub.

This can happen quite often, such as:
(1) You do calibre epub conversion, and calibre does not "properly" replace the existing cover. So instead you end up with an inner html cover page that contains a broken link, because calibre has removed the old image but not the associated html. When you open the epub in the viewer, you see a big blue question-mark showing where the image should have been.
(2) Your original source prior to the conversion contained incorrect links - someone renamed the images without fixing the tags
(3) The images were not present when you added the html to calibre

From a Modify ePub perspective, I am considering adding an option which will allow you to automate removing the html pages for situation (1) above.

While calibre does have a remove second cover option in the conversion dialog, this both (a) requires another conversion, (b) I don't trust it to always do what I want since it cannot discriminate whether an image is a front cover or not, and (c) IIRC does not remove the html page, just the img tag. So you end up with an extra blank page in your epub which isn't desirable either imo.

Instead such a Modify ePub option will look for html pages whose body tags contain only an image link which is broken and no other text. If so it will remove the entire html page. Or so the theory goes anyways...

As always, feedback welcomed...

[kiwidude]

Changes in this release:

• Add a "Search ePub" dialog allowing the user to perform a search for any content matching a regular expression
• Fix cosmetic issue of marked text incorrect for search results of "Check inline xpgt" links
• Add a "Check Mobi" submenu with "Check missing EBOK cdetype" and "Check missing ASIN" options
• Add a "Check broken image links" option to the Check ePub Structure menu, to find epubs with html pages that have broken image links
• Add a "Check TOC with broken links" option to the Check ePub Structure menu, to find epubs with NCX links to pages that do not exist

Anyone reading the last page or so will know about most of these changes - the two biggies being a new ability to search for anything you like in ePubs, as well as some checks for Mobi files of interest to Amazon Fire users.

A late breaking addition today is the check for TOC with broken links. Ever done an ePub conversion and ended up with a broken cover link? This check will find them. As well as the various other reasons why your TOC links are broken, such as the result of editing/tweaking and forgetting to update the ncx file.

Enjoy...

[JimmXinu]

Originally Posted by kiwidude

Changes in this release:

• Add a "Search ePub" dialog allowing the user to perform a search for any content matching a regular expression

Very nice.

Wish I'd been following this thread earlier, because now I feel bad for speaking up after the release instead of when it was still in beta.

Have you considered making a separate "Search ePub" plugin? I can see a lot of casual users wanting to search inside books, and Quality Check is not necessarily an intuitive place for it. (I don't understand what half of Quality Check does and I write epubs. )

The detail/log view after searching; if that could show some context on either side of the found text, it would be helpful. Perhaps try and report finds by chapter instead of file?

Include a 'text only' search that ignores HTML perhaps? So that the search 'We can only hope' will still find 'We can <i>only</i> hope'?

Jim

[kiwidude]

Originally Posted by JimmXinu

Have you considered making a separate "Search ePub" plugin? I can see a lot of casual users wanting to search inside books, and Quality Check is not necessarily an intuitive place for it. (I don't understand what half of Quality Check does and I write epubs. )

Not seriously considered it, no. Momentarily, yes. It would be one more plugin people would be finding space on the toolbar for, one more plugin to install and update, and contain a lot of replicated code from this plugin that I have to maintain. People using it for its intended purpose of searching for some specific characteristics will think of this plugin as being the right place. People wanting a general search content tool will not, but as implemented it won't fulfill their needs. Your suggestions if implemented are focused on the latter which sways it towards being available separately as an option. I think perhaps will see how it evolves and stabilises and then consider duplicating it, so non QC users can have a standalone option.

Your comments about not understanding some of the features QC has - have you looked at the help file? I have tried to explain what they all are and when you might use them. They are very granular which is why so many, but some things matter more to some people than others. I may one day attempt a single "fix my epubs" option, which combines some QC checks with modify ePub features so all the submenus get left to advanced users, but it is a maybe one day thing for now.

Originally Posted by JimmXinu

The detail/log view after searching; if that could show some context on either side of the found text, it would be helpful. Perhaps try and report finds by chapter instead of file?

Finding by chapter will never happen, an ePub has no concept of chapters. They could be a simple number, a character name, an image, non existent at all, multiple in a file, split across files.... It just isn't feasible.

I agree text either side would be nice sometimes, but not trivial to implement. Users won't want giant log files if their search expression covers a large content match, so it would have to be an option. And rendering output on the log window is problematic since it is HTML itself. I think this option should only apply during a text only search you suggest below.

Originally Posted by JimmXinu

Include a 'text only' search that ignores HTML perhaps? So that the search 'We can only hope' will still find 'We can <i>only</i> hope'?

It sounds nice in theory but not sure how well it would work. Stripping the text of all HTML tags is fine but will cause paragraphs to run together, which means words will run together if your ePub is a crappy PDF conversion. And you have stuff like if it has an &nbsp; inserted in the middle rather than a space. Or double spaces causing a lack of match. Or smart quotes failing a match of apostrophes and quotes the user types. So lots of stuff that will impact general search success.

It is good food for thought, appreciate the feedback.

[JimmXinu]

Fair enough. I knew when I was writing that post that I was suggesting a general search tool for casual users instead of the specialist's tool that it currently is. And that it would be a significant project to make it that.

Your comments about not understanding some of the features QC has - have you looked at the help file?

Honestly, no, I haven't really tried to understand all the features. So please don't take my ignorance as any sort of statement about your documentation. I just meant to illustrate that QC has a lot of very technical options.

[kiwidude]

I've looked back through the thread and can't find a post which covers exactly why I added this check. It has the effect of giving you a set of books that have calibre covers in them, which by association means that if you export them using calibre (or use Modify ePub "Update metadata") that their covers are replaceable.

However this is only a *subset* of books that "could" have their covers replaced when exporting/updating metadata. An epub doesn't always have to have been converted by calibre previously for this to work.

I think it would be much more useful to have a check which can tell you which ebooks need calibre conversions or not to replace the cover (well, until my new Modify ePub cover insertion feature appears of course).

So I would like to add a "Check replaceable cover" option in Quality check (along with a "Check not replaceable cover" option), and kill off the existing two inline cover checks.

Any objections? If you happen to use the feature as it exists currently I'd love to hear from you. Otherwise I'm going to assume the answer is good idea

[GeckoFriend]

It's a wonderful idea!

[paulfiera]

Originally Posted by GeckoFriend

It's a wonderful idea!

+ 1

[jackie_w]

+1 for the 'check not replaceable cover'

[theducks]

A Calibre cover is a :

1: Cover generated by clicking the Generate Cover button?
2: A cover Generated from a Calibre Plugin (the books image cover) ?
3: Added from Metadata, not 'as delivered' from the publisher?
Just what is a Calibre Cover and how would it be detected programatically?