[GUI Plugin] Modify ePub

[Barb-B]

Originally Posted by kiwidude

@JSWolf - well to be fair to Barb-B, she doesn't really care what epubcheck says, she just wants to be open the book in Sigil after doing a metadata update without having the opf corrupted. It was you that asked her to try the epubcheck route

kiwidude - appreciate the defense. Indeed I do not feel that I "have to" get epubcheck 3.0 up and running, for the very reason you say. Sorry JS - I DO appreciate your take on things...

While it sounds like there may be some improvement to Sigil when it looks for well formed opf, I gather I could solve the immediate problem and still be able to use the Update metadata selection if I simply turn on the Calibre feature you mention. So, thanks for that!

[user_none]

Kiwidude sent me the two files to look at. Here is what EPUBCheck Says:

• element "metadata" incomplete; missing required element "dc:identifier"
• unique-identifier attribute in package element must reference an existing identifier element id

So the Modify Plugin is stripping the dc:identifier which is means the OPF is now invalid. Sigil is detecting this and fixing the OPF. However, it does appear that Sigil's correction needs a bit of work since it should use the spine, manifest and any individual metadata elements when it creates the new (correct) OPF.

Where the date issue came from I don't know because I don't get that with EPUBCheck...

[JSWolf]

delete post

[DavidReed]

In doing a "modify ePub" of a batch of books to remove known artifacts and pro-actively remove margins from Adobe .xpgt files, I had several books "fail" when looking at the log file.

The "root cause" is that the epub's I was processing had empty (zero length/zero size) .xpgt files.

This *only* causes a problem when a user of Modify ePub selects several different options (like removing iTunes files or OS artifact files) AND remove margins. In that case the processing will fail. What really should happen is everything else "goes like normal" but the margins aren't removed (since there aren't any.) So maybe change the code to check for a zero length file and "skip it" if it is empty.

The error message in the log was:
File "calibre_plugins.modify_epub.modify", line 72, in process_book
File "calibre_plugins.modify_epub.modify", line 142, in _process_book
File "calibre_plugins.modify_epub.modify", line 313, in _zero_xpgt_margins
File "calibre_plugins.modify_epub.container", line 229, in get_parsed_etree
ParseError: Failed to parse: ops/styles/page-template.xpgt with error: None
ePub not changed after 1.52 seconds

From a feature question standpoint - is there a way to remove .xpgt files only if they are empty? I suppose it doesn't really matter if they exist if they are empty, but just curious.

[JSWolf]

What it looks like is that the Modify ePub sees that there is a page-template.xpgt file and also expects there to be an entry for it in the OPF. So it failed because there was a page-template file but no entry in the OPF. It has nothing to do with the size/contents of the page-template file.

[DavidReed]

@JSWolf - respectfully disagree. From the OPF (of a copy that never went through Modify ePub):

<item id="pt" href="page-template.xpgt" media-type="application/vnd.adobe.page-template+xml" />

And page-template.xpgt is of size zero.

So, removing margins of an xpgt fails when the xpgt is in the OPF but the file is zero length.

[JSWolf]

Originally Posted by DavidReed

@JSWolf - respectfully disagree. From the OPF (of a copy that never went through Modify ePub):

<item id="pt" href="page-template.xpgt" media-type="application/vnd.adobe.page-template+xml" />

And page-template.xpgt is of size zero.

So, removing margins of an xpgt fails when the xpgt is in the OPF but the file is zero length.

The best way to remove margins from page-template is to actually delete it. Unless it's being used for columns, it's 100% worthless. margins should be set in the CSS.

But, I did misread the error message. But if you are removing the margins, set the plugin to delete the file. It's not needed.

[DavidReed]

Originally Posted by JSWolf

The best way to remove margins from page-template is to actually delete it. Unless it's being used for columns, it's 100% worthless. margins should be set in the CSS.

But, I did misread the error message. But if you are removing the margins, set the plugin to delete the file. It's not needed.

I totally agree about deleting but... part of the "problem" in processing a library of books is not always being 100% sure if there are columns anywhere in a book so I don't want to delete the .xpgt unless it's only used for margins. So my "backup" plan was to just "remove the margins". The work around is to do margin removal as a separate step from any other processing of known artifacts / metadata. If an xpgt is "empty" the margin removal will error out but since it's a separate step, no harm done.

The code should still not error on an XML file that's empty.

[JSWolf]

Originally Posted by DavidReed

I totally agree about deleting but... part of the "problem" in processing a library of books is not always being 100% sure if there are columns anywhere in a book so I don't want to delete the .xpgt unless it's only used for margins. So my "backup" plan was to just "remove the margins". The work around is to do margin removal as a separate step from any other processing of known artifacts / metadata. If an xpgt is "empty" the margin removal will error out but since it's a separate step, no harm done.

The code should still not error on an XML file that's empty.

Of all the ePub I've seen with a page-template, not a single one was ever used to have columns. The problem with columns is that if you are using a small screen such as a cell phone in portrait, columns don't work.

But if you do remove margins and the page-template is empty, it should be deleted (IMHO).

[kiwidude]

Originally Posted by user_none

Kiwidude sent me the two files to look at. Here is what EPUBCheck Says:

• element "metadata" incomplete; missing required element "dc:identifier"
• unique-identifier attribute in package element must reference an existing identifier element id

So the Modify Plugin is stripping the dc:identifier which is means the OPF is now invalid. Sigil is detecting this and fixing the OPF. However, it does appear that Sigil's correction needs a bit of work since it should use the spine, manifest and any individual metadata elements when it creates the new (correct) OPF.

My thanks to John and Dave for taking the time to track this down, that date thing appears to be a red herring though there are multiple things going on here.

Firstly, Sigil does need to handle these ePubs better so as to not just silently generate a "random" (alphabetical) opf and muck up the book. The guys are working on that one which is brilliant.

Secondly it seems to be that with the combination of settings Barb B chose, the dc:identifier node is completely missing from the resulting ePub. I need to look into what caused that to happen.

Thirdly, the calibre save to disk logic (which is what update metadata calls) is producing an invalid identifier node. The original book goes from this:

<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="PrimaryID" version="2.0">
...
<dc:identifier id="PrimaryID" opf:scheme="ISBN">978-0-440-42334-8</dc:identifier>

to this:

<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="PrimaryID" version="2.0">
...
<dc:identifier opf:scheme="calibre">7f516111-8b74-4a4a-bf8c-79742d3bec9c</dc:identifier>

And if I understand correctly it is the dc:identifier node not having an id attribute to match the package declaration which is currently causing Sigil to throw a wobbly. This one I believe falls into Kovid's camp to comment on, as to why calibre does not preserve or generate the id attribute on that element...

[kovidgoyal]

@kiwidude: The metadata object you are passing to set_metadata probably has mi.uuid != None or mi.application_id != None. Dont do that if you dont want to change the book uuid.

[kiwidude]

@Kovid, thx I will check into that.

However I am specifically using the example of adding the book to calibre and then just doing save to disk on it, not using the Modify ePub plugin at all? Those steps are what produces the identifier with the missing id attribute? And I *think* (testing my memory here) that this only occurred when I turned off "get metadata from file contents" when adding the book to calibre just before the save to disk. With the "get metadata" option turned on, save to disk worked ok (or at least the book opened in Sigil ok, which I assume to mean the id was present). But I would need to recheck that - certainly what I described above is what happens with get metadata turned off...

[kovidgoyal]

http://bazaar.launchpad.net/~kovid/c...revision/14527

[JimmXinu]

@kiwidude,

FYI, if you're not following it, we discovered in this thread that the CLI instructions you included in the zip for Modify ePub are out of date.

(I'd also suggest adding CRs to the line ends in the readme.txt and example.cmd files since Windoze notepad still inexplicably continues to ignore unix style newlines.)

[Barb-B]

@Kiwidude,

How hard would it be to add an option that performs a regex search/replace in the opf, to get rid of the 'linear="no"' instructions from the spine? I'm doing it manually now using linear="[a-z]*" as my find value, and it would be ever so handy to get it in the same pass with all the other great stuff this plugin does. Do you think it fits conceptually with the overall intent of the tool?