[GUI Plugin] Quality Check

[ldolse]

I think I'd have to leave the full illumination on cover handling to Kovid, I don't actually know that code. I also generally avoid epub->epub conversions for the same reasons you do and manually update covers when so motivated (which is actually infrequent).

Based on the behavior I see when adding books it seems like Calibre will grab the jpg file that represents the cover if the OPF/Guide actually specifies it as a cover - other times it just renders the html of the first page to an image, which seems to indicate that there aren't any significant heuristics going on to test if the file is a cover when the manifest is silent.

As far as the things that happen during conversion, it's relatively easy to disable things that happen in the structure detection phase, which is what causes new split points, TOC changes, etc, but disabling stuff that happens later in the pipeline is pretty difficult, in particular css flattening, font size rescaling, etc. This is the bigger reason I avoid it as I don't want to change the css wholesale when the original css was good/easy to tweak in Sigil.

Kovid's mentioned a framework he started called epub-fix, it's discussed a bit in this bug:
http://oldbugs.calibre-ebook.com/ticket/8252#comment:5

It might offer a way to hook into some of this.

Other features/functions that exist that might be able to help from a plugin are:

• The save-to-disk feature - this will insert new covers/update metadata without doing an epub->epub conversion. I don't believe Calibre offers a way to do this directly to the books inside the library, which would be nice.
• The 'remove first image' option in the conversion pipeline might also be useful. You might be able to write a plugin that optionally removes the first image with that feature and inserts a new cover a'la save-to-disk without actually saving the file to a new location.

[Perkin]

Originally Posted by kiwidude

@Perkin,

The whole issue of covers in epubs is a murky one, if my understanding from previous threads on this is correct. This is again perhaps something that Idolse could answer way better than me. Certainly it would be nice to be able to check for books with a cover and books without a cover. I think the problem is the definition of what is a "cover", as commercial epubs could have some filename that doesn't look like any of those you suggested.

I do 100% agree that the ability to remove (or in my case add if not present) a cover would be useful without having to go down the conversion route. There are occasions where I get concerned that an epub -> epub conversion may "harm" the file in some other way particularly if it has never been subjected to such a conversion previously. For example making sure that it doesn't resplit pages on headings etc. I just don't know enough about the voodoo that happens when you hit convert to know whether with default options set it is anything to be concerned about or not. Undoubtedly there are people who have posted in the past about Calibre "messing" with their epub in some bad way in some particular situation but whether that is a now historical bug issue, user error or something else I have no idea. The whole "Tweak Epub" feature allows you to (manually) make such adjustments or fiddle with the css of course.

Yes, I do the removing using sigil, makes it slightly easier as it takes care of the opf etc. I know it also changes the structure of the epub but that isn't that much of a problem unless there's a 'proper' ncx.

Doing so many by hand is quite time consuming. I'd like it (if it is possible) to be only conducted on selected books, that way If I mess it up (by removing ones I shouldn't) then it would be my fault (as most thing are).

I'd hope to use it on ones I know that have been calibre'd (as I said before - for articles/short-stories)

I think checking similar to the way you do (in the plugin code) for the jacket, adjusted for cover would work for my purposes, it's just the extra 'remove files and update opf' that would then need implemented in some way.

Yes, adding a cover (when wanted) without conversion would also be a great addition.

[Dopedangel]

Originally Posted by kiwidude

@Perkin,

The whole issue of covers in epubs is a murky one, if my understanding from previous threads on this is correct. This is again perhaps something that Idolse could answer way better than me. Certainly it would be nice to be able to check for books with a cover and books without a cover. I think the problem is the definition of what is a "cover", as commercial epubs could have some filename that doesn't look like any of those you suggested.

I do 100% agree that the ability to remove (or in my case add if not present) a cover would be useful without having to go down the conversion route. There are occasions where I get concerned that an epub -> epub conversion may "harm" the file in some other way particularly if it has never been subjected to such a conversion previously. For example making sure that it doesn't resplit pages on headings etc. I just don't know enough about the voodoo that happens when you hit convert to know whether with default options set it is anything to be concerned about or not. Undoubtedly there are people who have posted in the past about Calibre "messing" with their epub in some bad way in some particular situation but whether that is a now historical bug issue, user error or something else I have no idea. The whole "Tweak Epub" feature allows you to (manually) make such adjustments or fiddle with the css of course.

I would also like to check if the cover has svg code or not and add if it doesn't here is the sample of the svg code. Also the ability to disable\enable aspect ratio. Another thing if you do a epub to epub conversion and it already has a cover.jpg the new cover is named cover1.jpg even if you use remove the first image option.

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="100%" height="100%" viewBox="0 0 500 749" preserveAspectRatio="none">
                <image width="500" height="749" xlink:href="cover1.jpeg"/>
            </svg>

[kiwidude]

Originally Posted by dwanthny

The split jackets happened before Kovid put the jacket in the metadata. It occurred when you converted a book with a Jacket in it and your preferences had pagebreaks set to insert before all h1 tags? Which I think might have been default so split jackets as theducks describes were fairly common. You ended up with the title on one page and the author/series/comments on the next.

I PM'd you an example.

Thanks for the PM, just what I needed. Ick, what a mess in that book

And yeah it was the <h2> below the <h1> which caused the split as I expected. I'm sure I can come up with something to detect this.

One further question - this whole thing started because you asked for the ability to remove the "extra" jackets. Would it be safer just to have a single menu option of just remove "all" jackets? And then you regenerate to add an up to date one? Or would you want to have two menu options - one to remove all, and one that removes all except for one that has the calibre metadata tag on it?

I just wonder if you might get yourself in a mess not knowing what type of jacket is in what epub. For instance, if I expand the "check jacket" and "check multiple jackets" tests to cater for these extra scenarios, you aren't going to know whether your multiple jacket book has two old ones or an old one and a new one. Or indeed whether a book with a jacket has the new one or the old style one

Or should we instead have a check for "legacy" jackets?

Have a think about the permutations and tell me what you would like. I hate jackets myself, and remove *any* I find so I'm not representative of understanding the requirements of those who want to have one in one shape or form...

[DoctorOhh]

Originally Posted by kiwidude

One further question - this whole thing started because you asked for the ability to remove the "extra" jackets. Would it be safer just to have a single menu option of just remove "all" jackets? And then you regenerate to add an up to date one? Or would you want to have two menu options - one to remove all, and one that removes all except for one that has the calibre metadata tag on it?

Both options would be nice. Remove all jackets would be nice for many folks that decide that they don't like the jackets, but the one I would use would be the one that removes all jackets except for the one that has the calibre metadata tag on it.

The style of the existing jacket doesn't matter just as long as the existing jacket is identified in the metadata then if at any time in the future I decide to do a epub to epub conversion I know that the jacket will be replaced with the current information and format.

Originally Posted by kiwidude

I hate jackets myself, and remove *any* I find so I'm not representative of understanding the requirements of those who want to have one in one shape or form...

So you would definitely like a remove all feature.

[kiwidude]

Originally Posted by dwanthny

The style of the existing jacket doesn't matter just as long as the existing jacket is identified in the metadata then if at any time in the future I decide to do a epub to epub conversion I know that the jacket will be replaced with the current information and format.

So to summarise, this is what we would have:

• Check for jacket - scans library for books that have at least one Calibre jacket, be it old style or new style
• Check for missing jacket - scans library for books that have no Calibre jacket of either style
• Check for multiple jackets - scans library for books that have more than one Calibre jacket (could be multiple of old style, or mix of new and old style)
• Remove selected jackets - works on the selected books to remove any Calibre jackets of either style
• Remove selected excess jackets - works on the selected books to remove all but the new Calibre jacket style

Do you see the holes? And are you happy with them or do we need to tweak the above...

• I find a book that has multiple jackets of the old style (only). How do I remove them? Should "Remove selected excess" in this situation try to keep one old style jacket? If so which one should it keep?
• I use check jackets and it tells me a book has a jacket (but it happens to be a legacy one). I later at some point do a conversion. Now the book will have two jackets. So I need to go back and rerun the check multiple jackets check, then run the remove selected excess jackets?

[DoctorOhh]

Originally Posted by kiwidude

So to summarise, this is what we would have:

• Check for jacket - scans library for books that have at least one Calibre jacket, be it old style or new style
• Check for missing jacket - scans library for books that have no Calibre jacket of either style
• Check for multiple jackets - scans library for books that have more than one Calibre jacket (could be multiple of old style, or mix of new and old style)
• Remove selected jackets - works on the selected books to remove any Calibre jackets of either style
• Remove selected excess jackets - works on the selected books to remove all but the new Calibre jacket style

Do you see the holes? And are you happy with them or do we need to tweak the above...

I like the above.

Originally Posted by kiwidude

• I find a book that has multiple jackets of the old style (only). How do I remove them? Should "Remove selected excess" in this situation try to keep one old style jacket? If so which one should it keep?

A good question. Now that I added the new jacket to the 1000+ you originally identified I want everything not identified in metadata to go away, but I can see where others would disagree.

It may or may not be practicable to differentiate between old jackets. The option to remove excess might be limited to jacket listed in metadata versus all others.

Originally Posted by kiwidude

• I use check jackets and it tells me a book has a jacket (but it happens to be a legacy one). I later at some point do a conversion. Now the book will have two jackets. So I need to go back and rerun the check multiple jackets check, then run the remove selected excess jackets?

If you kept an old jacket then this is the outcome.

[kiwidude]

Cool. I think based on what you have said that "Remove selected excess jackets" could instead be "Remove selected legacy jackets". This will give you what you want for when a book has multiple jackets - it will keep the new one.

I think for the sake of any ongoing maintenance in a library it would be better to encourage the legacy jackets without the tags to be removed. That way if at any point you decide to reconvert a book, you aren't immediately creating a problem for yourself of a double jacket.

So I would see the workflow as:
Check having jackets
Select all those epubs and run Remove selected legacy jackets
Then run Check missing jackets
Then do your conversions to add a new jacket.

Under that workflow, you should never have a need to check multiple jackets.

If your library is large with a lot of legacy jackets that you really don't ever intend to reconvert, then you can use Check multiple jackets. However at least this way the Remove selected legacy jackets wil be useful to you without having to do extra check/remove steps after a conversion.

How does that sound?

[DoctorOhh]

Sounds like a plan to me.

[Dopedangel]

Originally Posted by kiwidude

Cool. I think based on what you have said that "Remove selected excess jackets" could instead be "Remove selected legacy jackets". This will give you what you want for when a book has multiple jackets - it will keep the new one.

I think for the sake of any ongoing maintenance in a library it would be better to encourage the legacy jackets without the tags to be removed. That way if at any point you decide to reconvert a book, you aren't immediately creating a problem for yourself of a double jacket.

So I would see the workflow as:
Check having jackets
Select all those epubs and run Remove selected legacy jackets
Then run Check missing jackets
Then do your conversions to add a new jacket.

Under that workflow, you should never have a need to check multiple jackets.

If your library is large with a lot of legacy jackets that you really don't ever intend to reconvert, then you can use Check multiple jackets. However at least this way the Remove selected legacy jackets wil be useful to you without having to do extra check/remove steps after a conversion.

How does that sound?

Would it be possible to replace old jacket with new one without converting
As I hesitate to reconvert as that might change the formatting.

[kiwidude]

Originally Posted by Dopedangel

Would it be possible to replace old jacket with new one without converting
As I hesitate to reconvert as that might change the formatting.

Not in my plans to support this currently. It falls into the same category as replacing a cover without reconverting. It sounds doable in theory but requires a lot more knowledge of the conversion pipeline than I have or want to deal with currently.

I think reconverting a book that has had a jacket is less of an issue, as you know that has been converted by calibre previously. Well at least if as Idolse has said above your defaults are with all the things like heuristics and splits turned off.etc.

But I agree that in the longer term it should be possible to change covers and jacket without touching the rest of the book.

[mehetabelo]

kiwi> just wanted to note that the reason I didn't post the problem I had on this thread was because I was ignoring it, figuring it was something I could figure out myself later, and it wasn't necessary at the time. I probably (by accident) placed it in my right click menu when I was half asleep and couldn't remember how to get rid of the damned thing.

Sometimes it seems calibre has too many options and I can't find what I know is there because I think too hard about it. And no, that's not in anyway meant to say that calibre is bad. having options is nice, but if you never use them sometimes you forget where to make changes.

[kiwidude]

@mehetabelo - no problem, it is a very large and complex program and I can't remember where to find stuff at times either. When you install the plugin (with relatively recent Calibre versions) you are given the choice of where to put it in terms of menus and toolbars so that would be likely how you ended up with it on your right-click menu. And as I said on the other thread Preferences -> Toolbars and changing the dropdown within that screen allows you to customise where else (or not) to put it at any point.

So is it that the only issue that annoyed you was the inconvenience of having a giant menu appear when you moused past it in the context menu? There wasn't a functional issue with it?

[mehetabelo]

@kiwi. No, no functional issue. My problem is the massive quality problems in my dump DB. It's slowly being whittled down, and I'm sure I'll find things to suggest or possibly bugs, and I'll bring them up as I see them.

As for installing it, I don't think that was the problem. It's possible, but I think I was just screwing around with the toolbar and somehow switched the option when I wasn't paying enough attention usually when I sit down to work on my database it's late, and I'm tired, so sometime I miss things that should be obvious.

The real annoyance was because everytime during cleanup I was trying to delete a specific format after converting/cleaning it up (often times for older books that have serious need of physical cleanup, I convert to rtf and I have about a dozen macros in word that do some standardized cleanup for me and add a navigation/toc then export as stripped html) I would be on the delete menu and then it would jump to the QC menu. This is because I have my mouse set pretty sensitive and the menu's where right next to one another. Drove me nuts for about an hour and then I just dumped QC when I couldn't remember how to get stuff out of the context menu.

[drMerry]

Hi Kiwidude

2 questions.
1. Is it possible to add some series tests? like:
-Series Missing Parts

Spoiler:

Assume last continuous index#

-Series Having Duplicates

Spoiler:

Series with same title more than once (no index-check)

-Series Having Indexing Errors

Spoiler:

Duplicate index# or 'strange' index-numbers (like all numbers are 'integers' and 1 or two are 'Doubles'

-Magazine Items Missing

Spoiler:

Assume magazines are series with 'Doubles', where some fractions are skipped - Maybe with an option to tell the interval (week, month, 4 weeks, 2 months, 2 weeks, daily - This would give the possibility to calculate missing numbers (52 or 53 weeks, 13 (4wk) or 12 (month), etc))

-Magazine-Series Having Index Errors

Spoiler:

For example all magazines have fractions and some have .00

2. Where do you get the 'Title Case Information'?
I have some problems with it, All books having the word macht have to be written as MacHt (capital H). I think I made a mistake somewhere but I do not know where.
I also would like change this behaviour so only the first word and names get capitals, but I think I have to change a regex than.