[GUI Plugin] Modify ePub

[theducks]

Originally Posted by icallaci

<What the ducks says is spot on. However I am curious that you believe that runnin some Modify ePub functions is creating the issue and that it does not exist in the original ePub. If you ever hit this issue again, please PM me a link to the original ePub and the modified one along with as above what options you had ticked.>

Kiwidude, I didn't know what was causing the problem and didn't know where to start looking. I just noticed that it ONLY happened after I ran Modify ePub, but not when I ran Calibre's conversion program. Next time it happens, I'll look at the before and after OPFs to see what the difference is. TheDucks' answer makes complete sense to me. I'll post my findings (or if I can't figure it out, I'll PM you with links). Thanks. And thanks for Modify ePub, too. It makes cleaning up epubs less of a chore.

There is something else going on. I have used the Modify EPUB PI thousands of times. Never, has it made a mess of the books order.
I suspect GiGo . The PI only changes what it was told and does not validate the rest.

[JSWolf]

I've used Modify ePub extensively and find it works very well. Next time you have a problem, take the ePub (the one before the plugin was used) and validate it with ePubcheck and see what errors there are. Fix the errors and then when it validates, run the plugin again and see what happens.

[icallaci]

<I've used Modify ePub extensively and find it works very well. Next time you have a problem, take the ePub (the one before the plugin was used) and validate it with ePubcheck and see what errors there are. Fix the errors and then when it validates, run the plugin again and see what happens.>

Thanks, that's a good idea. I'll validate it with ePub Check, fix the errors, and run it through Modify ePub. If I see the problem pop up after that, I'll compare the OPFs. That gives me several things to try. Thanks to everyone who made suggestions.

[Barb-B]

I've had the same difficulty that icallaci describes with the plugin, but went a little further in trying to figure out how to work around it. I was able to isolate the result to the "Update Metadata" option. For files that will experience the problem, running the plugin with that option checked will cause the opf to fail and Sigil will create a new one in alpha order. If I re-do the process with the same book, and leave that option unchecked, the resulting opf will be acceptable.

Kiwidude - I have pm'd you links to before and after versions of a file that experienced this. Didn't want to post publicly.

[JSWolf]

Originally Posted by Barb-B

I've had the same difficulty that icallaci describes with the plugin, but went a little further in trying to figure out how to work around it. I was able to isolate the result to the "Update Metadata" option. For files that will experience the problem, running the plugin with that option checked will cause the opf to fail and Sigil will create a new one in alpha order. If I re-do the process with the same book, and leave that option unchecked, the resulting opf will be acceptable.

Kiwidude - I have pm'd you links to before and after versions of a file that experienced this. Didn't want to post publicly.

Does the OPF (that fails) validate using epubcheck 3.0?

[Barb-B]

I was unable to get 3.0 working on my machine set up as it is, and got tired of futzing around with it. However, was able to use an older version that some nice local folks had put into a GUI wrapper - and it did pass that with no errors.

Oops! (later edit) I just realized I slightly misread your question. The original version of the file passed with no errors. After running through the Modify Epub plugin, the after version produced the following errors:

date value '0101-01-01T00:00:00+00:00' is not valid, YYYY[-MM[-DD]] expected

unfinished element

unique-identifier attribute in package element must reference an existing identifier element id

[icallaci]

Barb B -- Thank you, thank you, thank you!

[kiwidude]

Barb B - I just dragged the epub you sent into calibre, ticked *only* the update metadata option in Modify ePub, and it all works perfectly fine.

As you say in the ePub the date field in that metadata looks a bit screwed up. I would suspect that this comes somehow from the metadata associated with the book as it exists in your calibre library. In my updated ePub this has:

<dc:date>2011-09-26T14:00:00+00:00</dc:date>

This plugin is calling standard calibre code for this feature. So I would expect you to get exactly the same result if you were to use the Save to Disk feature of calibre, where it applies the same updating of metadata.

Try that save to disk and see if you get the result I predicted. Also make sure you are only checking the Update metadata option, in case in fact it is some other option that is somehow disrupting the data.

[icallaci]

It seems to happen randomly for me, which is why I am having so much trouble tracking it down. The last time it happened (when I posted to ask about it), I reimported the original epub and ran it through Modify ePub, ticking just one box each time, and had no problems. It's not a big deal for me to check the order of files against the original, since it happens so rarely, but it would be nice to figure it out.

[Barb-B]

Originally Posted by kiwidude

...
This plugin is calling standard calibre code for this feature. So I would expect you to get exactly the same result if you were to use the Save to Disk feature of calibre, where it applies the same updating of metadata.

Try that save to disk and see if you get the result I predicted. Also make sure you are only checking the Update metadata option, in case in fact it is some other option that is somehow disrupting the data.

KiwiDude - given your result with checking only "Update metadata", I'm guessing the problem may be related to a combination of selected options, and the sequence in which they occur. Here are all the other standard options I routinely use:

Remove calibre bookmark files
Remove unused image files
Remove Adobe .xpgt files and links
Remove Adobe resource DRM meta tags
Encode HTML in UTF-8
Remove all metadata jackets
Remove legacy metadata jackets
Remove broken image pages
Remove non dc: metadata elements

After running these others, and using the Calibre Save to Disk feature, the saved files don't exhibit any trouble with their opf's.

Since I care less about the "Update metadata" option than any of the others, I'd just as soon let it go, and get the other results that matter to me.

Hope that's helpful if you want to continue pursuing this issue. I'm perfectly happy to continue working around it. Either way, it's still a fantastic tool, and much appreciated!

[kiwidude]

Barb-B - ok, I think I know what is happening. On my calibre I have read file metadata turned on when adding books, and I am betting you do not. So when I add to calibre, the date field calibre uses to put into the opf file is set, however I am betting you have not updated the metadata for the book, and hence that is why it is storing an invalid date in there.

This is not a bug in Modify ePub, it is really a flaw in calibre in my opinion, though Kovid may not see it that way, either that or Sigil should handle it better. As I predicted above this is the same behaviour you will see if you use the calibre Save to Disk feature to export the book, you get exactly the same "invalid" epub resulting.

So... I suggest you raise a bug request for calibre for Kovid and that way it will be fixed for all scenarios, he just needs to store a better "null date" than his current approach I believe. Of course if he disagrees then you should raise a bug request to Sigil, and ask the devs to handle that date value more effectively. Though the fact that epubcheck shows it as an error would indicate to me at least that really calibre should not be putting such a date value in there.

So for the record, the issue is occurring when the original book has this in the opf:

<dc:date opf:event="publication">2011-09-27</dc:date>

Which if calibre is not reading metadata from the book when adding, will then on save to disk (or update metadata in Modify ePub) turn it into this:

<dc:date>0101-01-01T00:00:00+00:00</dc:date>

[kovidgoyal]

<dc:date>0101-01-01T00:00:00+00:00</dc:date> is a perfectly valid date as per the EPUB spec. If epubcheck is complaining about it, epubcheck is, as usual, wrong. http://idpf.org/epub/20/spec/OPF_2.0...m#Section2.2.7

[JSWolf]

Originally Posted by kovidgoyal

<dc:date>0101-01-01T00:00:00+00:00</dc:date> is a perfectly valid date as per the EPUB spec. If epubcheck is complaining about it, epubcheck is, as usual, wrong. http://idpf.org/epub/20/spec/OPF_2.0...m#Section2.2.7

I have just taken a look a that link and yes, <dc:date>0101-01-01T00:00:00+00:00</dc:date> is correct.

[JSWolf]

Originally Posted by Barb-B

I was unable to get 3.0 working on my machine set up as it is, and got tired of futzing around with it. However, was able to use an older version that some nice local folks had put into a GUI wrapper - and it did pass that with no errors.

Oops! (later edit) I just realized I slightly misread your question. The original version of the file passed with no errors. After running through the Modify Epub plugin, the after version produced the following errors:

date value '0101-01-01T00:00:00+00:00' is not valid, YYYY[-MM[-DD]] expected

unfinished element

unique-identifier attribute in package element must reference an existing identifier element id

You will need to get epubcheck 3.0 working as this is not an error but an error is whatever version of epubcheck you used. epubcheck 3.0 says all is well. So there could be other errors in epubcheck prior to 3.0 that have been fixed in 3.0.

[kiwidude]

@JSWolf - well to be fair to Barb-B, she doesn't really care what epubcheck says, she just wants to be open the book in Sigil after doing a metadata update without having the opf corrupted. It was you that asked her to try the epubcheck route

Thanks for the reply Kovid. Given you have checked the spec and what calibre generates is correct, then that would appear to only leave Sigil as the culprit. I will drop an email to the other (i.e. active) Sigil devs and see what they have to say about it. Certainly since I see the exact same behaviour from a simple Save to Disk operation from calibre I don't feel there is anything that needs to be changed in the Modify ePub plugin concerning it.