[GUI Plugin] Quality Check

[kiwidude]

Originally Posted by JSWolf

Well, what I think is to run whatever is selected against each book and then if any issue is found, present it to the user for that one book. Then when that is over, move on until the next book has an error. That would do in the short term until a better way can be figured out.

Unfortunately that isn't quite as simple as it sounds to me at least. How does the user "progress" to the next book? That implies there is some sort of state being kept of the search being made, with all the implications that carries of a UI for controlling it and the ability to cancel, see where you are up to etc. And you still need a way of seeing what is "wrong" with that particular book to do something with it.

And while there are times where you might work book by book, there are other users/times where doing the fix in bulk makes sense across a batch of books. After all it is easier to select a bunch of books and use modify ePub once against them with the common settings checked, than it is to book by book do various combinations.

It may be that the ultimate solution is going to have to involve another window (modeless), allowing the user to view the results in various ways such as by issue or by book etc. I've been trying to avoid doing that for various reasons but it might be the most appropriate solution here. In Find Duplicates I took the approach of using highlighting groups, but that had the advantage of the context being identical between and across groups. Another possibility would be to store the data about what is wrong with a book in the database with that book, but a user would still need to choose a menu item to actually see that info which is a bit clumsy to use. So my gut feel at the moment is that some sort of navigational window allowing the filtering of results in various ways and displaying of the associated log data for that result set is going to be the way to go. But alternative ideas welcomed...

Sure, the ePub in question may have some file that's not UTF-8, but to the user, it looks like it's crashed. The OPF & CSS were encoded in ANSI and when I converted them to UTF-8 without BOM, the crash stopped. Another check you can add in is for proper UTF-8 without BOM encoding. Also, to not have it blow up on ANSI.

This is a handled situation where I chose to put the entire error message into the log file to help the user understand why a particular ePub could not have that check run against it. It isn't what I call a crash - that would be if the plugin stopped working. Had I just left the error message at "Unable to open ePub", then you wouldn't know why, and I couldn't tell you either without you sending it to me. I could have added some wordy message in there to that affect of telling the user not to panic, but (a) users are generally lazy and would still post about it anyway to double check what it meant, and (b) it is such a rare occurrence it hasn't been a priority.

There is however probably something I can do with the xml parser to have it avoid that error situation, which I will experiment with. Thanks for posting the exact details of what triggered it/solved it.

As for having a check for UTF-8 without BOM, I'm not quite sure how feasible that is. From what I know and have googled in the past, it isn't possible to tell from a file what encoding it actually has. You can attempt to open it (like with an xml parser) and see if it errors (like has happened here) but I'm not sure what else you can do to identify it. Perhaps Kovid or someone may have a suggestion...

[JSWolf]

Originally Posted by kiwidude

As for having a check for UTF-8 without BOM, I'm not quite sure how feasible that is. From what I know and have googled in the past, it isn't possible to tell from a file what encoding it actually has. You can attempt to open it (like with an xml parser) and see if it errors (like has happened here) but I'm not sure what else you can do to identify it. Perhaps Kovid or someone may have a suggestion...

Notepad++ can tell what the encoding is and the source for it is available via Sourceforge if you are interested in having a look at it.

[DiapDealer]

Originally Posted by JSWolf

Notepad++ can tell what the encoding is and the source for it is available via Sourceforge if you are interested in having a look at it.

Like everything else, Notepad++ does its best to determine what the encoding might be, but it's certainly not fail-safe.

[JSWolf]

Originally Posted by DiapDealer

Like everything else, Notepad++ does its best to determine what the encoding might be, but it's certainly not fail-safe.

But I've not seen it get UTF-8 without BOM wrong. So at least that code is working properly. And if you say there are issues determining the encoding, then the puglin should stop as maybe it could get it wrong (non-UTF-8).

[paulfiera]

@kiwidude

Did you ever implement checking for SVG covers in the calibre generated titlepage.xhtml?

I'd love to update all my ePubs to have an SVG cover.

Many thanks !

[JSWolf]

I cannot seem to find an option to check for broken links. I see options for checking broken image and ToC links. Also, does checking for broken ToC links get the ncx and the internal ToC? If it only does the ncx file, then a rename of the option is in order.

[kiwidude]

@paulfiera - no, obviously not yet. Thousands of lines of code related to Modify ePub has soaked up most of the last few weeks, and there have also been multiple releases of Count Pages, Generate Cover, & Reading List. Then add to that Extract ISBN got broken with calibre 0.8.53... oh yeah, I do have a day job too...

@JSWolf - correct, there is no option for general broken link checking. With recent releases of this plugin I have started tumbling down the slippery slope of all sorts of potential internal ePub content validation. My initial additions to this plugin were focused around things that calibre had responsibility for breaking (until recent fixes), like broken image links and broken NCX links. Having opened the door a crack with those, it is not unreasonable to consider other things eventually. Of course it is pretty much replicating some of FlightCrew validation, the difference being that this plugin would allow you the 50,000 foot view of which books have issues across your library, that you can then open in isolation to fix.

As for the TOC/NCX naming, that was deliberate on my part for historical consistency but it is a fair point. The issue is that the very first option I had for QC was "Check TOC with < 3 entries". That was named that way so as to not bamboozle users with the term NCX, as I would suggest only a very small subset of users would understand what an NCX is. However you are 100% technically correct that all three of those options should state NCX. Perhaps a good compromise is "Check NCX TOC with < 3 entries", and then if we ever have checks related to an inline TOC we say "Check inline TOC ..."

Oh look, 3000 posts...

[theducks]

Originally Posted by kiwidude

...
Having opened the door a crack with those, it is not unreasonable to consider other things eventually. Of course it is pretty much replicating some of FlightCrew validation, the difference being that this plugin would allow you the 50,000 foot view of which books have issues across your library, that you can then open in isolation to fix.

I talked to Valloric about getting a CLI interface opened on the Standalone Flightcrew so I could use your 'Open With' PI to do a Pass/Fail check and return the status to be flagged (similar to what the QC check does)

Wishing for Cake, It would be nice if it returned a Count of errors (that I coud save in a custom column) so I could prioritize repairs

[JSWolf]

Originally Posted by kiwidude

@JSWolf - correct, there is no option for general broken link checking. With recent releases of this plugin I have started tumbling down the slippery slope of all sorts of potential internal ePub content validation. My initial additions to this plugin were focused around things that calibre had responsibility for breaking (until recent fixes), like broken image links and broken NCX links. Having opened the door a crack with those, it is not unreasonable to consider other things eventually. Of course it is pretty much replicating some of FlightCrew validation, the difference being that this plugin would allow you the 50,000 foot view of which books have issues across your library, that you can then open in isolation to fix.

One thing I was thinking is if you have a general check for broken links, you could do away with all the separate link checking as this one would be a catchall for all links that are broken be they on the ncx or some internal link and it would also catch broken links from an internal ToC and image links that are broken. That would make link checking a one option only option that does it all.

As for the TOC/NCX naming, that was deliberate on my part for historical consistency but it is a fair point. The issue is that the very first option I had for QC was "Check TOC with < 3 entries". That was named that way so as to not bamboozle users with the term NCX, as I would suggest only a very small subset of users would understand what an NCX is. However you are 100% technically correct that all three of those options should state NCX. Perhaps a good compromise is "Check NCX TOC with < 3 entries", and then if we ever have checks related to an inline TOC we say "Check inline TOC ..."

Oh look, 3000 posts...[/QUOTE]

I agree to renaming the remaining ncx checks (if you implement my suggestion above) to NCX ToC.

[JSWolf]

I've noticed one other thing left out that's a constant issue especially with a lot of publisher ePub. Incorrect guide types. A lot of guide entries are made with invalid types.

[theducks]

Originally Posted by JSWolf

I've noticed one other thing left out that's a constant issue especially with a lot of publisher ePub. Incorrect guide types. A lot of guide entries are made with invalid types.

IMHO this is flightcrews realm. to flag incorrect use, when the proper way is to precede with other. (took me a bit to realize I needed the dot ) .

[JSWolf]

Originally Posted by theducks

IMHO this is flightcrews realm. to flag incorrect use, when the proper way is to precede with other. (took me a bit to realize I needed the dot ) .

But other features of this plugin are FC's domain as well.

[theducks]

Originally Posted by JSWolf

But other features of this plugin are FC's domain as well.

but many are just superficial (yellow items) in Flight crew.

Flight crew could care less if there are 3 or 30 TOC items, as long as they are done correctly

so there is only a few that really point to a malformed EPUB... That is no good reason to proliferate the overlap.

BTW I am in favor of a fully populated NCX, not just that which is needed to pass the check.

[cecdc7731]

Does the ASIN Check look at EXTH 113, 504 or both? Is there a way to designate which on is checked as I have been editting with mobi meta editor and putting matching numbers from Amazon in both as I go along. I would like to be able to do more edits as I have free time than just for books I am fixing to put on my Fire. Is there a way to add a quick fix that would make 113 and 504 the same? Also, can the ASIN be extracted directly from Amazon Kindle versions in the metadata editor rather than or in conjunction with the ISBN number?

[morantis]

Nice job, I am working on a web app that will search the internet based on a photo or image and find a person using the data in it. It would be nice to be able to do the same thing with cover data from a book, but harder to accomplish in the end.