I managed to trigger a very interesting bug... :)

[davidfor]

Originally Posted by compurandom

I claim that if the epub has a cover image but not a (marked) cover page, and the extended driver doesn't generate a wrapper cover page during translation, it is a bug, as this is a clear difference between epubs and kepubs that it should fix.

Well your claim is WRONG! How many times must I say this: There is nothing in the kepub specs that say this is mandatory. Please read them and tell me where it says something different.

Everything else I agree with you, it's a feature request.

Some preliminary results from my checks...
I used quality check to search for books without a replacable cover.

Any particular reason you ignored the other check that I suggested? The one that I actually suggested first and then added the "replaceable cover" check. And in fact, suggested using the two together?

It found 500 books as follows:
399 either are marked as or are likely to contain only a single xhtml file (will check these later)
101 were left that needed checking

A lot had unmarked covers and after marking the covers, quality check now says they have replacable covers. So, to answer davidfor's question directly: Yes, I found one piece of software at least that uses the cover page: quality check.

Oh, wow, you found a validation routine that came close to proving I was wrong. Clumsy me for asking a bad question. How about answering the question I was really asking: Tell me about a reading application or device that uses the "cover" type on an XHTML file. Or even something that doesn't run in calibre context.

I've found two books (I think) so far that even after marking their cover page, quality check says it isn't removable. Perhaps worthy of investigation later, but it's a small number so far so perhaps not.

Some of the epubs with unmarked cover pages were probably originally single html files that calibre split when converting to an epub, and (coincidentally?) split the cover page as a separate file.

And if you go back and read my suggestion, you will see that I was in no way suggesting the suggested checks would be accurate. All I was suggesting was that they might be a way to reduce the number to check. But, if you want accuracy, than anything with a calibre SVG will do what you want. Anything else is suspect.

[davidfor]

Originally Posted by Terisa de morgan

To remove automatically the cover from the ToC when they appear. And yes, NCX ToC.

Funnily enough, I tend to add the cover to the ToC when it is missing.

[davidfor]

Originally Posted by compurandom

Most of the professional computer industry might not agree with you.

No, the rest of the professional computer industry will agree with DNSB. If the code doesn't do what the specs say it should do, there is a bug. If the application doesn't do what you want, it isn't necessarily a bug. Just you misunderstanding what the application is supposed to do.

[Terisa de morgan]

I totally agree with DNSB and davidfor, mainly because those have been my words to some customers along the years

[compurandom]

Originally Posted by davidfor

Any particular reason you ignored the other check that I suggested?

I didn't ignore it, just that quality check doesn't provide a convenient way to do them both at once, so I did them one at a time. Also, the non-replacable cover test caught only another ~6% and most of those it caught only after they had already been repaired.

Yes, I agree, they both need to be done. Please realize that I did it that way not to prove a point, but to actually clean up problems I missed on the last pass through my library.

I did understand that you don't think the checks are accurate, and I appreciate anything that can reduce the number of books I have to check by hand. Actually, I think the replaceable/non-replaceable checks probably are accurate enough, between them I didn't find any false positives of books needing fixing.

I just checked, and the calibre SVG cover test is not helpful for this purpose. Too many false positives; about 250 books fail this test, many with hand generated covers made in calibre. (No false negatives -- it found ~600 books all with automatically generated covers.)

[milady133]

Originally Posted by compurandom

Most of the professional computer industry might not agree with you.

As DNSB, davidfor and Terisa have stated, no, you are wrong, is the marketing people deciding to change programs against specifications on how it should work that generate a lot of problems professional computer people have to trouble shoot because some genius decided against their recommendations. (Says someone who's been expending years explaining that the tool I'm using is NOT Excel and everything I do to make it work like Excel generates a lot of other problems)
Yes, there's some people who's going to want that functionality added, but there's also people who is not going to want it, and should be a plugin apart from the kepub converter, or at least a check in the plugin you can activate or deactivate,given that it's going to produce random results in some cases.

[davidfor]

Originally Posted by compurandom

I didn't ignore it, just that quality check doesn't provide a convenient way to do them both at once, so I did them one at a time. Also, the non-replacable cover test caught only another ~6% and most of those it caught only after they had already been repaired.

Yes, I agree, they both need to be done. Please realize that I did it that way not to prove a point, but to actually clean up problems I missed on the last pass through my library.

I did understand that you don't think the checks are accurate, and I appreciate anything that can reduce the number of books I have to check by hand. Actually, I think the replaceable/non-replaceable checks probably are accurate enough, between them I didn't find any false positives of books needing fixing.

Than you need to go back, read your post and think about how you can be clearer in the future. The complete lack of mention of the SVG check makes it appear that you ignored it. The way you discussed then non-replaceable cover test makes it look like you were expecting better results.

And no, you can't run them both at the same time. But, you can run one for all the books and the other on the results of the first. Or tag them results of each and look at the combinations.

I just checked, and the calibre SVG cover test is not helpful for this purpose. Too many false positives; about 250 books fail this test, many with hand generated covers made in calibre. (No false negatives -- it found ~600 books all with automatically generated covers.)

Of course, there is an extremely simple solution to your problem. Select all the books in your library and run Polish books to update/add the covers. Or Modify ePub. You can exclude the ones that already have an SVG cover. But, doing the rest mean you have consistency across your library.

[compurandom]

Originally Posted by davidfor

TThe complete lack of mention of the SVG check makes it appear that you ignored it.

I didn't ignore the SVG test. It wasn't useful.
It flags a significant portion of my library and almost everything it flagged didn't need fixing. The removable/non-removable tests found 101 books. The svg test found 250 *different* books which didn't have problems.

Of course, there is an extremely simple solution to your problem. Select all the books in your library and run Polish books to update/add the covers.

That's not really a solution either, as some of the books had unmarked covers and polish books would have added a second cover.

Now, if I could search for books without a cover page but with a cover image, then all of *those* I could have just run polish on. This is really the only case that is trivial to fix.

[davidfor]

Originally Posted by compurandom

I didn't ignore the SVG test. It wasn't useful.
It flags a significant portion of my library and almost everything it flagged didn't need fixing. The removable/non-removable tests found 101 books. The svg test found 250 *different* books which didn't have problems.

And my issue was that you didn't bother to mention that. From your reply, you had completely ignored this suggestion. All you needed to do was include the above statement and I wouldn't have needed to waste my time wondering why you were concentrating on the part of my suggestion that I honestly thought would be the most useless.

That's not really a solution either, as some of the books had unmarked covers and polish books would have added a second cover.

Yes, that can happen. But, my experience is that it is rare.

Now, if I could search for books without a cover page but with a cover image, then all of *those* I could have just run polish on. This is really the only case that is trivial to fix.

At the moment, I feel like you are wasting your time worrying about this. Unless you feel like writing the update to Quality Check to do the detection, my suggestion would be to bite the bullet and do the Polish for all the potentially problem books. Yes, you will process some books that are OK, but the cover will be the calibre version and you know this will not cause issues. And you will get two covers on a few. But, you won't see any books with the messed up first file after sending the book as kepub to the device. Overall, your reading experience will be improved. And on those (I think) rare books that do get two covers, they can be fixed with a lot less trouble than what you are doing now.

But, I suppose that you could do the Polish and compare the file size of the original with the new. If they are similar, the extra cover probably wasn't added. If there is a bigger difference, it probably was. Of course, there are probably so many problems with this method that it isn't worth considering.

[JSWolf]

Originally Posted by DNSB

I have to almost agree with you there though I tend to validate on opening before I even start editing. A surprising number of commercial ebooks fail to validate. Once I have modified and validated, the number of issues with reading the ebooks is pretty minimal. I don't bother with trying to losslessly compress the graphics since very few graphics source supply lossless originals.

Yes, it is interesting how few problems there are when you know the eBooks sent to the Reader are validated.As for losslessly compressing the graphics, it doesn't matter if the ones in the book are lossless originals. What lossless compression does is get rid of any crud in the graphics container without changing them. So you end up with the same graphics at a smaller size.

One fun item were the sheer number of ebooks where the cover image was repeated. Attributor protected ebooks were a fine example where the first cover image had their logo and the second did not but there were enough others that bulked up the ebook with two cover images. Then there were those ebooks where the scene and/or chapter break was a graphic and every break had it's own copy of the image.

And there sometimes are eBooks where you get cover thumbnails that have no use at all. I've seen some eBooks where one cover was JPG and the other was PNG. And when you get duplicated section break graphics, they are slightly different. So it's a matter of figuring out which version you want, deleting the rest and fixing the code to use just that one copy.

[JSWolf]

Originally Posted by compurandom

Most of the professional computer industry might not agree with you.

I believe the old hacker's description of that is "broken as designed".

Now, if you document how broken it is, it becomes a misfeature.

If I wrote a program to show eBooks with green text and a blue background and you want black text and white background, the green and blue colorsare not a bug. They are as programmed.

A bug is when a program should do something a certain way and doesn't do it. If you expect it to do something different then it does, that does not mean it's a bug. It might or might not be a bug.

[compurandom]

Originally Posted by davidfor

And my issue was that [...]

I feel this thread has too much in it that is borderline personal attack with pointless arguments over weather it is a bug or a misfeature or whatever, and I don't need to (or should not) respond to it. I want to keep (my) discussion here constructive.

I just want my epubs fixed or converted in a way that makes them pleasant to read on my kobo while making the smallest number of changes to them possible with the least amount of work on my part. Discussion over weather it is a waste of time or not is pointless, as I've already (regrettably) spent the time, and (gratefully) most of my library is now fixed so I don't have any more test cases left.

So, until I either do another bulk import, or find another formatting issue common in a lot of books, or find something else to automate, or find more cases of this problem that were missed somehow, I'm done discussing that side of things.

I'd like to keep this constructive. I'd like to continue discussing what features / changes could be made to modify/polish epub and quality check. I just went through the entire thread and made a list of possible features that have been mentioned. I'll keep updating that list until someone has a use for it.

If I see anyone step forward to update these plugins, or better, I find time to start looking into it, I'll post my list of mentioned feature requests. (Or I suppose I could start a new thread and post the list at the top of the thread.)

Right now, the list is a bit disorganized and should be cleaned up and compared against actual existing features, and I'll add to it if I see anything new mentioned.

My next personal frustration is that I have so many epubs and pdfs with missing or badly formed TOCs.

The toc editor built into calibre only works on epubs, and for a lot of poorly formed epubs, does not do a good job of autogenerating ToCs -- I have several books (converted to epub) that the autogenerated pdf is about 50% bad links.

I've partially written code that will extract metadata from a PDF, import a hand generated TOC, and re-insert that into the pdf. Unfortunately, creating the toc is a bit of a laborious manual process, even if you can 90% copy and paste it from the text of the pdf. It might be interesting to somehow leverage the UI of the existing TOC editor in calibre to assist with this process or design something better. It's an idea I've been kicking around for a while.

Also, someone mentioned leveraging existing calibre library features that didn't exist when modify epub / quality check were created. I'd be interested in looking at that, is there a specific API reference somewhere that focuses on epub (or pdf) manipulation? If I do dig into this, that would be a good place to start.

[jackie_w]

Originally Posted by compurandom

Also, someone mentioned leveraging existing calibre library features that didn't exist when modify epub / quality check were created. I'd be interested in looking at that, is there a specific API reference somewhere that focuses on epub (or pdf) manipulation? If I do dig into this, that would be a good place to start.

I think you probably mean me. The calibre container classes, in calibre.ebooks.oeb.polish.container.py, handle the safe manipulation of epub/azw3 innards and can also be used for kepubs. I think new methods get added as needed. These classes don't handle PDF and I know almost nothing about PDF editing.

I know from personal experience there are calibre source .py files full of utilities regarding covers, TOCs, OPFs and much more. You'll find some of them in the same directory as container.py.

I think there is a logical structure to the way the calibre source is arranged but unfortunately it doesn't always help me zero in on what I'm looking for. I suspect the fault is all mine A good search tool gets you there in the end.

[compurandom]

Originally Posted by jackie_w

The calibre container classes, in calibre.ebooks.oeb.polish.container.py, handle the safe manipulation of epub/azw3 innards and can also be used for kepubs.

Thanks. Added to the top of my list.

These classes don't handle PDF and I know almost nothing about PDF editing.

Right now, I'm using a command line tool to do the PDF heavy lifting, what I've written just massages the metadata to make it sanely editable with a regular editor. I know I'll have to do it differently if I integrate with calibre.

I'm sure the pdf tools aren't too far from there.

[davidfor]

Originally Posted by compurandom

I feel this thread has too much in it that is borderline personal attack with pointless arguments over weather it is a bug or a misfeature or whatever, and I don't need to (or should not) respond to it. I want to keep (my) discussion here constructive.

Yes, my last response was a bit of an attack at you. But, you triggered one of my pet peeves in all support environments. That is the complete utter inability to actually supply information or answer questions. Do you know how often it appears the person reporting a problem seems to think we can read minds? Or how often I see, "I thought of mentioning that, but didn't think it was important".

In this case, I made a suggestion that you apparently tried and decided didn't work for, but you didn't bother to mention that fact. But, you went on about the other part of my suggestion, the part that I didn't think would be useful, and basically complained about how useless it was.

Put yourself in my shoes for a moment. Can you see why I might have been peeved?