Modify ePub plugin dev thread

[JSWolf]

Originally Posted by kiwidude

I'm going to release it as 1.2.0 when the final build is done, which will make sure that anyone who has been trying the beta will get notified via plugin updater. I could have called these beta versions 1.1.9, 1.1.10 etc before then but I hadn't expected there to be quite so many iterations required...

Well, you didn't have me testing things before recently.

[kiwidude]

True. And while I expected "some" issues from the major changes I made underneath, completely forgetting about one of the code files even existing created half of them. Python for all its positives is a caveman's language in terms of lack of proper refactoring tools and code analysis. Which makes development a case of "code & fix", which I hate. And if as I have been you jump in and out in spare moments at work or while watching TV at home, it is easy to forget/miss something. Then without any type or compile checking the only way of catching it is to test every of code. Without any unit testing support like I would have in .NET... sigh...

Look at the whats new with every calibre releases and there are frequently regression fixes, even an uber experienced full-time Python guru like Kovid gets tripped up - so that doesn't leave much chance for the rest of us without even more time/effort.

We are definitely close on this release though...

[JSWolf]

One thing I thought I may have scene once was the original cover XML left behind. But my last test looking for it did not find that had happened. So I'll do some more testing and see what I can break next.

[kiwidude]

Cool, let me know. The conditions on which it will keep an existing cover xml page are based around (a) whether it has any *other* images on that page, and (b) whether there is any text content on that page. If either of those conditions are true, then rather than removing the whole page it just removes the link to the cover image and leaves the rest of the page as is.

So if you have an ePub that doesn't have a "real" cover page, but incorrectly (Sigil can cause this issue) has marked the first page semantically as a cover, you don't lose any text content from the book. I have seen this happen with calibre conversions in the past, I believe it has been fixed now with the same sort of logic.

Edit: I should add, this is all assuming the plugin is able to "identify" the existing cover page. If there are no guide entries indicating a cover, or no meta tag entries pointing to an image, then it won't know what page the existing cover is. So it will just add a new page at the front, just like a calibre conversion does. These are the really badly published epubs, that hopefully are in the minority.

[theducks]

The latest run against a EPUB that I Sigil edited and passed FC

Spoiler:

calibre, version 0.8.53
ERROR: Modify ePub failed: No ePub files were updated

Modify ePubs
Logfile for book ID 3416 (Skirting the Grave / Annette Blair)
3416
Modifying: C:\DOCUME~1\shop1\LOCALS~1\Temp\calibre_0.8.53_tmp _9qukir\hh0ucu_modify_epub\3416.epub
Looking for files to remove: [u'META-INF/calibre_bookmarks.txt']
Looking for Adobe xpgt files to remove
Looking for Adobe xpgt xhtml links to remove
Looking for Adobe DRM meta tags to remove
Looking for legacy jackets
Skirting the Grave - ERROR: Traceback (most recent call last):
File "calibre_plugins.modify_epub.modify", line 69, in process_book
File "calibre_plugins.modify_epub.modify", line 154, in _process_book
File "calibre_plugins.modify_epub.jacket", line 54, in remove_legacy_jackets
AttributeError: 'ExtendedContainer' object has no attribute 'get'

ePub not changed after 1.14 seconds

[kiwidude]

@theducks - you are running an older beta, I fixed this issue yesterday. Try again with the new beta in post #182. If you think you upgraded, did you remember to restart calibre?

[theducks]

Originally Posted by kiwidude

@theducks - you are running an older beta, I fixed this issue yesterday. Try again with the new beta in post #182. If you think you upgraded, did you remember to restart calibre?

I removed all copies of the PI from my download dir.
I Uninstalled the PI
I restarted Calibre (the Modify Icon is gone)
I downloaded the file from post 181
installed the PI
Restarted Calibre (the Icon now shows)

Same book, same error (3 seconds longer)

Spoiler:

calibre, version 0.8.53
ERROR: Modify ePub failed: No ePub files were updated

Modify ePubs
Logfile for book ID 3416 (Skirting the Grave / Annette Blair)
3416
Modifying: C:\DOCUME~1\shop1\LOCALS~1\Temp\calibre_0.8.53_tmp _pigaag\lug_nt_modify_epub\3416.epub
Looking for files to remove: [u'META-INF/calibre_bookmarks.txt']
Looking for Adobe xpgt files to remove
Looking for Adobe xpgt xhtml links to remove
Looking for Adobe DRM meta tags to remove
Looking for legacy jackets
Adding or updating jacket
Current jacket removed: OEBPS/jacket.xhtml
Manifest item removed: jacket.xhtml (calibre_jacket)
Spine itemref removed: calibre_jacket
Skirting the Grave - ERROR: Traceback (most recent call last):
File "calibre_plugins.modify_epub.modify", line 69, in process_book
File "calibre_plugins.modify_epub.modify", line 158, in _process_book
File "calibre_plugins.modify_epub.jacket", line 16, in add_replace_jacket
File "site-packages\calibre\ebooks\oeb\transforms\jacket.py", line 142, in render_jacket
AttributeError: 'str' object has no attribute 'ratings_char'

ePub not changed after 1.19 seconds

Version 1.1.8 - 27 May 2012 (Beta)
Change to require minimum calibre version 0.8.53 in order to utilise some calibre bug fixes/changes
Change to calibre API for deprecated dialog in 0.8.49 which caused issues that intermittently crashed calibre on Mac OS
Add a "Insert or replace cover" option to attempt to insert or replace a cover without doing a conversion
Add a "Remove cover" option to attempt to completely remove an identified cover from the ePub.
Rewrite "Removed unused image files" and "Remove broken cover images" features to use lxml rather than regex for better accuracy
Add protection for numerous options against trying to apply them to a DRM encrypted book
Better handle ebooks where the ncx file is not in same directory as opf manifest

[kiwidude]

Ok, thats a different error - the one you had previously was most definitely from the old code (I can tell by the error messages).

However the latest error you just posted is also something I fixed yesterday. But in transferring the files between work and home this morning it seems I missed one and so that fix got reverted, the joys of ad hoc "source control"...

New version posted above, I'm going to bed, clearly I need sleep.

[Firedancer885]

I just updated to your latest beta, restarted calibre, tried to insert covers into books and got the following error. (With an FFDL updated epub ebook.)

Spoiler:

Modify ePubs
Logfile for book ID 17 (Back Again, Harry? / Jedibuttercup)
17
Modifying: /tmp/calibre_0.8.53_tmp_ihTKGX/FIrPHr_modify_epub/17.epub
Insert or replace cover
...Looking for guide cover reference
...Looking for meta cover
...Looking for other.ms-* guide references to clean out
...Writing new cover image and titlepage html
New cover image written to: cover.jpeg
New titlepage html written to: OEBPS/titlepage.xhtml
...Updating manifest and TOC for the new cover
Manifest item added: OEBPS/titlepage.xhtml (titlepage)
Manifest item added: cover.jpeg (cover)
Meta item inserted: cover:cover
Spine item inserted: titlepage at pos: 0
No guide parent element found - inserting one
Guide item inserted: OEBPS/titlepage.xhtml:Cover:cover
Back Again, Harry? - ERROR: Traceback (most recent call last):
File "calibre_plugins.modify_epub.modify", line 69, in process_book
File "calibre_plugins.modify_epub.modify", line 180, in _process_book
File "calibre_plugins.modify_epub.modify", line 672, in _insert_replace_cover
File "calibre_plugins.modify_epub.covers", line 64, in insert_or_replace_cover
AttributeError: CoverUpdater instance has no attribute 'removed_html_name'

ePub not changed after 0.18 seconds

[kiwidude]

@Firedancer885 - thx for reporting that. New build on the beta post above.

[JSWolf]

Originally Posted by kiwidude

@Firedancer885 - thx for reporting that. New build on the beta post above.

Did you remember to update the version number?

[kiwidude]

@JSWolf - I think I did, though the change notes won't reflect it, I don't bother with updating those for bug-fixes related to new features, it will all get cleaned up when I officially release 1.2.0.

Anyone who has installed one of the beta versions along the way should make sure they are using that version posted above, with I think is set to 1.1.10. I'm going to officially release it tomorrow night in all probability unless anyone comes up with an example epub cover situation I haven't correctly handled before then.

[JSWolf]

I've just found one bug that's in need of fixing. The link to the cover in the internal ToC was not updated.

One feature request... When there already exists a cover XML file, it would be good to have the plugin keep the same file name as the existing cover XML.

[Firedancer885]

Originally Posted by kiwidude

@Firedancer885 - thx for reporting that. New build on the beta post above.

Thanks It solved the problem.

[kiwidude]

Originally Posted by JSWolf

I've just found one bug that's in need of fixing. The link to the cover in the internal ToC was not updated.

Please PM me a link to the original ePub to try to replicate it and understand what happened...

One feature request... When there already exists a cover XML file, it would be good to have the plugin keep the same file name as the existing cover XML.

Hmmm... will think about that one. I based my code around what calibre does during a conversion which is why it "does what it does" currently, but we have already started deviating from that with locations of images/xml files and some optimisations. The only potential issues I can think of is that it will still need a "new" cover name if either (a) no existing cover is detected, or (b) the existing cover has other stuff on it and so cannot be safely removed...