Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 03-04-2013, 05:38 AM   #406
Barb-B
programmer wannabe
Barb-B began at the beginning.
 
Posts: 21
Karma: 10
Join Date: Jul 2012
Device: Nook Simple Touch Glowlight; Sony PRS-T1; Kindle
Quote:
Originally Posted by kiwidude View Post
@JSWolf - well to be fair to Barb-B, she doesn't really care what epubcheck says, she just wants to be open the book in Sigil after doing a metadata update without having the opf corrupted. It was you that asked her to try the epubcheck route
kiwidude - appreciate the defense. Indeed I do not feel that I "have to" get epubcheck 3.0 up and running, for the very reason you say. Sorry JS - I DO appreciate your take on things...

While it sounds like there may be some improvement to Sigil when it looks for well formed opf, I gather I could solve the immediate problem and still be able to use the Update metadata selection if I simply turn on the Calibre feature you mention. So, thanks for that!
Barb-B is offline   Reply With Quote
Old 03-04-2013, 08:23 AM   #407
user_none
Sigil & calibre developer
user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.
 
user_none's Avatar
 
Posts: 2,473
Karma: 1053245
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
Kiwidude sent me the two files to look at. Here is what EPUBCheck Says:
  • element "metadata" incomplete; missing required element "dc:identifier"
  • unique-identifier attribute in package element must reference an existing identifier element id

So the Modify Plugin is stripping the dc:identifier which is means the OPF is now invalid. Sigil is detecting this and fixing the OPF. However, it does appear that Sigil's correction needs a bit of work since it should use the spine, manifest and any individual metadata elements when it creates the new (correct) OPF.

Where the date issue came from I don't know because I don't get that with EPUBCheck...
user_none is offline   Reply With Quote
 
Advertisement
Old 03-04-2013, 11:23 AM   #408
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 38,536
Karma: 19637653
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 1, iPhone 5
delete post
JSWolf is online now   Reply With Quote
Old 03-04-2013, 01:18 PM   #409
DavidReed
Junior Member
DavidReed began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Mar 2013
Device: iPad /
XPGT parsing "bug"

In doing a "modify ePub" of a batch of books to remove known artifacts and pro-actively remove margins from Adobe .xpgt files, I had several books "fail" when looking at the log file.

The "root cause" is that the epub's I was processing had empty (zero length/zero size) .xpgt files.

This *only* causes a problem when a user of Modify ePub selects several different options (like removing iTunes files or OS artifact files) AND remove margins. In that case the processing will fail. What really should happen is everything else "goes like normal" but the margins aren't removed (since there aren't any.) So maybe change the code to check for a zero length file and "skip it" if it is empty.

The error message in the log was:
File "calibre_plugins.modify_epub.modify", line 72, in process_book
File "calibre_plugins.modify_epub.modify", line 142, in _process_book
File "calibre_plugins.modify_epub.modify", line 313, in _zero_xpgt_margins
File "calibre_plugins.modify_epub.container", line 229, in get_parsed_etree
ParseError: Failed to parse: ops/styles/page-template.xpgt with error: None
ePub not changed after 1.52 seconds

From a feature question standpoint - is there a way to remove .xpgt files only if they are empty? I suppose it doesn't really matter if they exist if they are empty, but just curious.
DavidReed is offline   Reply With Quote
Old 03-04-2013, 02:27 PM   #410
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 38,536
Karma: 19637653
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 1, iPhone 5
What it looks like is that the Modify ePub sees that there is a page-template.xpgt file and also expects there to be an entry for it in the OPF. So it failed because there was a page-template file but no entry in the OPF. It has nothing to do with the size/contents of the page-template file.
JSWolf is online now   Reply With Quote
Old 03-04-2013, 02:41 PM   #411
DavidReed
Junior Member
DavidReed began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Mar 2013
Device: iPad /
@JSWolf - respectfully disagree. From the OPF (of a copy that never went through Modify ePub):

Code:
<item id="pt" href="page-template.xpgt" media-type="application/vnd.adobe.page-template+xml" />
And page-template.xpgt is of size zero.

So, removing margins of an xpgt fails when the xpgt is in the OPF but the file is zero length.
DavidReed is offline   Reply With Quote
Old 03-04-2013, 03:00 PM   #412
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 38,536
Karma: 19637653
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 1, iPhone 5
Quote:
Originally Posted by DavidReed View Post
@JSWolf - respectfully disagree. From the OPF (of a copy that never went through Modify ePub):

Code:
<item id="pt" href="page-template.xpgt" media-type="application/vnd.adobe.page-template+xml" />
And page-template.xpgt is of size zero.

So, removing margins of an xpgt fails when the xpgt is in the OPF but the file is zero length.
The best way to remove margins from page-template is to actually delete it. Unless it's being used for columns, it's 100% worthless. margins should be set in the CSS.

But, I did misread the error message. But if you are removing the margins, set the plugin to delete the file. It's not needed.
JSWolf is online now   Reply With Quote
Old 03-04-2013, 03:11 PM   #413
DavidReed
Junior Member
DavidReed began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Mar 2013
Device: iPad /
Quote:
Originally Posted by JSWolf View Post
The best way to remove margins from page-template is to actually delete it. Unless it's being used for columns, it's 100% worthless. margins should be set in the CSS.

But, I did misread the error message. But if you are removing the margins, set the plugin to delete the file. It's not needed.
I totally agree about deleting but... part of the "problem" in processing a library of books is not always being 100% sure if there are columns anywhere in a book so I don't want to delete the .xpgt unless it's only used for margins. So my "backup" plan was to just "remove the margins". The work around is to do margin removal as a separate step from any other processing of known artifacts / metadata. If an xpgt is "empty" the margin removal will error out but since it's a separate step, no harm done.

The code should still not error on an XML file that's empty.
DavidReed is offline   Reply With Quote
Old 03-04-2013, 03:14 PM   #414
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 38,536
Karma: 19637653
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 1, iPhone 5
Quote:
Originally Posted by DavidReed View Post
I totally agree about deleting but... part of the "problem" in processing a library of books is not always being 100% sure if there are columns anywhere in a book so I don't want to delete the .xpgt unless it's only used for margins. So my "backup" plan was to just "remove the margins". The work around is to do margin removal as a separate step from any other processing of known artifacts / metadata. If an xpgt is "empty" the margin removal will error out but since it's a separate step, no harm done.

The code should still not error on an XML file that's empty.
Of all the ePub I've seen with a page-template, not a single one was ever used to have columns. The problem with columns is that if you are using a small screen such as a cell phone in portrait, columns don't work.

But if you do remove margins and the page-template is empty, it should be deleted (IMHO).
JSWolf is online now   Reply With Quote
Old 03-04-2013, 09:13 PM   #415
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by user_none View Post
Kiwidude sent me the two files to look at. Here is what EPUBCheck Says:
  • element "metadata" incomplete; missing required element "dc:identifier"
  • unique-identifier attribute in package element must reference an existing identifier element id

So the Modify Plugin is stripping the dc:identifier which is means the OPF is now invalid. Sigil is detecting this and fixing the OPF. However, it does appear that Sigil's correction needs a bit of work since it should use the spine, manifest and any individual metadata elements when it creates the new (correct) OPF.
My thanks to John and Dave for taking the time to track this down, that date thing appears to be a red herring though there are multiple things going on here.

Firstly, Sigil does need to handle these ePubs better so as to not just silently generate a "random" (alphabetical) opf and muck up the book. The guys are working on that one which is brilliant.

Secondly it seems to be that with the combination of settings Barb B chose, the dc:identifier node is completely missing from the resulting ePub. I need to look into what caused that to happen.

Thirdly, the calibre save to disk logic (which is what update metadata calls) is producing an invalid identifier node. The original book goes from this:
Code:
<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="PrimaryID" version="2.0">
...
<dc:identifier id="PrimaryID" opf:scheme="ISBN">978-0-440-42334-8</dc:identifier>
to this:
Code:
<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="PrimaryID" version="2.0">
...
<dc:identifier opf:scheme="calibre">7f516111-8b74-4a4a-bf8c-79742d3bec9c</dc:identifier>
And if I understand correctly it is the dc:identifier node not having an id attribute to match the package declaration which is currently causing Sigil to throw a wobbly. This one I believe falls into Kovid's camp to comment on, as to why calibre does not preserve or generate the id attribute on that element...
kiwidude is offline   Reply With Quote
Old 03-05-2013, 12:00 AM   #416
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,436
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@kiwidude: The metadata object you are passing to set_metadata probably has mi.uuid != None or mi.application_id != None. Dont do that if you dont want to change the book uuid.

Last edited by kovidgoyal; 03-05-2013 at 12:06 AM.
kovidgoyal is online now   Reply With Quote
Old 03-05-2013, 01:32 AM   #417
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@Kovid, thx I will check into that.

However I am specifically using the example of adding the book to calibre and then just doing save to disk on it, not using the Modify ePub plugin at all? Those steps are what produces the identifier with the missing id attribute? And I *think* (testing my memory here) that this only occurred when I turned off "get metadata from file contents" when adding the book to calibre just before the save to disk. With the "get metadata" option turned on, save to disk worked ok (or at least the book opened in Sigil ok, which I assume to mean the id was present). But I would need to recheck that - certainly what I described above is what happens with get metadata turned off...
kiwidude is offline   Reply With Quote
Old 03-05-2013, 01:59 AM   #418
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,436
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
http://bazaar.launchpad.net/~kovid/c...revision/14527
kovidgoyal is online now   Reply With Quote
Old 03-07-2013, 11:55 PM   #419
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,821
Karma: 510215
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
@kiwidude,

FYI, if you're not following it, we discovered in this thread that the CLI instructions you included in the zip for Modify ePub are out of date.

(I'd also suggest adding CRs to the line ends in the readme.txt and example.cmd files since Windoze notepad still inexplicably continues to ignore unix style newlines.)
JimmXinu is offline   Reply With Quote
Old 03-16-2013, 05:12 PM   #420
Barb-B
programmer wannabe
Barb-B began at the beginning.
 
Posts: 21
Karma: 10
Join Date: Jul 2012
Device: Nook Simple Touch Glowlight; Sony PRS-T1; Kindle
@Kiwidude,

How hard would it be to add an option that performs a regex search/replace in the opf, to get rid of the 'linear="no"' instructions from the spine? I'm doing it manually now using linear="[a-z]*" as my find value, and it would be ever so handy to get it in the same pass with all the other great stuff this plugin does. Do you think it fits conceptually with the overall intent of the tool?

Last edited by Barb-B; 03-29-2013 at 03:36 AM.
Barb-B is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Quality Check kiwidude Plugins 797 12-16-2014 01:01 PM
[GUI Plugin] Open With kiwidude Plugins 251 12-15-2014 01:07 PM
[GUI Plugin] Manage Series kiwidude Plugins 113 08-01-2014 09:18 PM
Modify ePub plugin dev thread kiwidude Development 346 09-02-2013 06:14 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 01:27 PM


All times are GMT -4. The time now is 11:37 PM.


MobileRead.com is a privately owned, operated and funded community.