[GUI Plugin] Quality Check

[jackie_w]

I agree totally with theducks. Nothing would persuade me to do any kind of "bulk automatic repair" on a set of epubs. I'm all for a set of magic buttons which each do a particular epub repair, but I want to look at the result afterwards to make sure I haven't wrecked anything.

Mind you I do tend to be over-cautious. I don't do bulk conversions or bulk metadata downloads either

[DoctorOhh]

For jacket removal and zeroing of page-template I would be willing to throw the dice once I verified it worked on a few books. These two items aren't likely to blow up any book.

Looking forward to trying these features out.

[kiwidude]

I'm not far away from releasing the next Quality Check version (got one more "Check" I am going to add related to navpoints in ncx files).

You can see something else in the top right that is also in progress...

@jackie/theducks - obviously the choice is entirely yours as to whether to do stuff manually or use a tool. Quality Check will allow you to "find" such cases without touching the files. Obviously my goal with the plugin that "fixes" the ePubs is that any changes it makes are done in the most conservative and safest way possible. It has to be a "no-brainer" decision for any feature that gets added to it.

[ldolse]

Originally Posted by kiwidude

So i guess what I could do is write a plugin that a bit like book conversions will run in the background on your ePub books to apply various transformations. It would work on your selected books, and give you a very simple dialog of checkboxes of stuff you want it to do with select none and all buttons. Then I would make sure each of those functions is coded to work independently. Then if it is deemed appropriate in future this whole plugin could just become an item on the tweak submenu. And the existing tweak dialog could offer similar functionality reusing that part of the code.

Then it will be up to Quality Check as to how many of these things it offers "Check" functions for. It may only be a subset, the two plugins can grow independently.

How does that sound for a plan?

Sounds like a good plan to me. If you're willing to take on the initial effort of creation that's great, as I don't envision myself having much time the next few weeks to jump on this one. However if you've put the basic framework in place I can look into tying other 'safe' pieces of the conversion engine into the plugin.

@JackieW, part of the thinking behind this is to address a feature request that you opened a while back, your 'epub conversion lite' idea:
http://oldbugs.calibre-ebook.com/ticket/8252

The idea is to be able to automatically do any individual function that Calibre's conversion pipeline allows while avoiding doing a 'full' conversion. This isn't really possible with any other ebook formats, but since epub is zipped OEB and OEB is Calibre's intermediate format for all conversions it's an option in this case.

[jackie_w]

Originally Posted by ldolse

@JackieW, part of the thinking behind this is to address a feature request that you opened a while back, your 'epub conversion lite' idea:
http://oldbugs.calibre-ebook.com/ticket/8252

The idea is to be able to automatically do any individual function that Calibre's conversion pipeline allows while avoiding doing a 'full' conversion. This isn't really possible with any other ebook formats, but since epub is zipped OEB and OEB is Calibre's intermediate format for all conversions it's an option in this case.

Excellent! I've been trying out some of my own beginners Python code to tackle some of these myself, as I wasn't sure my request would ever make it to the top of anyone's list. I don't think my code would be robust/generic enough to release to the public but it's been fun trying. I cobbled together an xgpt tweaker, a css tweaker and an embedded font stripper. I even managed an epub to html utility, then a week later user_none brought out the official convert to HTMLZ format. Still I guess no experience is wasted

I look forward to whatever you come up with. I'm sure it will be better than anything I write.

[jackie_w]

Originally Posted by kiwidude

@jackie/theducks - obviously the choice is entirely yours as to whether to do stuff manually or use a tool.

Just to clarify, I will be more than happy to use your automatic fix tools. I just want to apply them one at a time not in bulk. I hope no offense was taken, I love your plugins.

[kiwidude]

@jackie_w - I didn't see it as offensive in the slightest. There are certain things I don't do in bulk either. Conversions is top of my list - I cringe when I see people posting about doing thousands at a time or whatever. I suppose that is fine if you don't ever intend to actually read them. No wonder there is so many crap conversions floating around out there with lost quotes, bad encodings, embedded page numbers, etc.

The functions I had in mind for the Clean ePub plugin would never need to be applied to a book that I have converted, as I know it is already in a state I am happy with. However I do have a big backlog of stuff that hasn't yet had my TLC to make it into my "master" library - so if I can run a bunch of things against that mess in bulk then provided as I say I have confidence I am not destroying the ePub it should save me a lot of work. And others who use it of course...

[kiwidude]

Changes in this release:

• Move all the ePub checks onto a Check ePub submenu
• Add epub check for legacy jackets only
• Add epub check for missing container.xml files
• Add epub check for files listed in manifest missing from epub
• Add epub check for unmanifested files
• Add epub check for iTunes plist files
• Add epub check for Calibre bookmarks files
• Add epub check for .xpgt margins
• Add epub check for TOC with < 3 entries
• Add epub check for inline Calibre cover
• Add epub check for no inline Calibre cover
• Add epub check for Calibre conversion
• Add epub check for not Calibre converted
• Add check for duplicate series index
• Improve having jacket and multiple jacket checks to include legacy jackets
• Restructure code internally for ease of future expansion

This started off as a minor release and become rather more.

Hopefully those of you who are concerned with the quality of their ePubs will find a few things to like out of that lot. Identifying is one thing - fixing will be another. As some will have gathered from the last few pages I have started another plugin which will allow automating the remedy for many of these items without requiring you to do a conversion.

Enjoy, I look forward to your feedback.

[theducks]

Do you know how much work you caused

I just did about 10 of the checks. (Using Natty Narwhal(32) today)
I opened a number of each type of error in Sigil.

Unfortunately, the report was correct

Kiwidude does it again

[Perkin]

Wonderful work. Loving it. Thanks kiwidude.

[kiwidude]

Thanks guys.

A post release thought I have had is that in the next release I will change the check for unmanifested files. By definition currently it is a superset of the bookmarks and plist files, plus any other "stray" files you may have hiding in your epubs.

Instead I think I should make it a check for "other" files excluding plist and bookmarks ones. That way if you decide to for instance retain Calibre's ability to open an epub at the last page read in its ebook viewer, you could still detect other stray files lurking within with this check.

@Idolse - I think that is two of the four items off your checklist you posted here. The other two items you mentioned were checking for embedded fonts in the opf file, and checking for svg files. If either of these is simple to describe in a PM let me know. Or if you want to code them yourself to send to me then take a look at check_epub.py where all the ePub checks are, and the top of config.py where you define a menu item.

[drMerry]

I still love it.
Yet another option could be to check if date == pubdate (I do not know the value used for the date column...)

If those are the same, You have added the book but apparently not (yet) set the publication date.

[kiwidude]

Nice idea drMerry, I'll add it to the list for the next version.

It would be nice if Calibre had the option to set an empty publication date by default but that has never made it very high up Kovid's list of issues to address it seems. And given the spasms the Qt datepickers have with empty dates it's likely not a trivial thing to fix.

[drMerry]

Originally Posted by kiwidude

Nice idea drMerry, I'll add it to the list for the next version.

It would be nice if Calibre had the option to set an empty publication date by default but that has never made it very high up Kovid's list of issues to address it seems. And given the spasms the Qt datepickers have with empty dates it's likely not a trivial thing to fix.

should be no problem.
If you mass-edit, you can remove the update without any problems.
This would be the best way because you could search on pubdate:false

[nynaevelan]

Kiwidude:

I just had a thought, since I was thinking about the no formats problem which interfered with my goodreads syncing, would it be possible to add an option to this script to check for books with no formats??