Modify ePub plugin dev thread - Page 14

kiwidude · 10-01-2011, 02:47 PM

I'm saying that unless you do not use the Save to disk/Send to device function, your cover is going to (most of the time) get overwritten in your ePubs anyway when it ends up on your device/folder. So using this plugin to try to just update the metadata but not the cover is "futile" if your intent is to 100% guarantee that when the ePub is exported from calibre it still has your original large cover that is embedded. The only way I know of to ensure the cover remains at its large size, is to extract the cover from the ePub and make it your calibre cover. In which case using the update metadata command of the plugin is going to not negatively impact your cover at all since it will replace it with the same one.

drMerry · 10-01-2011, 06:32 PM

Quote:

Originally Posted by kiwidude

I'm saying that unless you do not use the Save to disk/Send to device function, your cover is going to (most of the time) get overwritten in your ePubs anyway when it ends up on your device/folder. So using this plugin to try to just update the metadata but not the cover is "futile" if your intent is to 100% guarantee that when the ePub is exported from calibre it still has your original large cover that is embedded. The only way I know of to ensure the cover remains at its large size, is to extract the cover from the ePub and make it your calibre cover. In which case using the update metadata command of the plugin is going to not negatively impact your cover at all since it will replace it with the same one.

In that case, I did understand the way the plugin works.

.
In my opinion there is a big advantage if you should provide the option as separated functions.
You've created a plugin that can change parts of the content of an epub without editing other files.
E.g. if I use the calibre explode and repack function, or use sigil, they all change additional files. e.g. files not in the opf are removed. This is not always what you want.
In my case, I often want to change just some info. E.g. a lot of files do have the author swapped (ln, fn) or have the language set to UND (or worse, EN while it's not).

In that case, this is the only info I want to update. At this moment I unpack the file, change the data in the opf and repack the file (not with use of calibre because of the additional changes).

In my opinion it would be great to skip this part and just use the plugin to update the data.

imaginary workflow:

Import epub in calibre (including available metadata)
Edit metadata in metadata window (e.g. swap ln, fn and change lang)
Update metadata using the plugin (cover is not touched).

This way, I keep most of the original data and only change some info or add some info (lang, pubdate, ....)

While I do not need to convert the file, there are no other changes made, it is the fastest way I can think of to update or change some metadata without loosing or changing other files.

I was thinking this was one of the main purposes of this plugin (But it seems not to be, I'll have to hack it

).

drMerry · 10-02-2011, 08:37 AM

FYI, I made an export of a subset of ebooks in my library.
These are the stats of 7072 epub-files.

Maybe you can find some interesting info in it.
A very curious one was the pdf file I found in one of the books.
The file is manifested on a original epub.
content of the pdf: the book cover (to print a paper cover to wrap around a hard-cover book). Including the offset information for the printing office. Curious because this book is sold as e-book this way.
Another curious thing was the fact of empty folders and empty files.
So, no request this time, just some stats you can maybe use to add extra clean-up functions or just to exclude some files of being removed.

The examined files all where cleaned first by using the plugin. That's why e.g. no itunes files are shown.

The xpgt files shown are files located in epubs with a bad opf-files causing the plugin to skip this epub while cleaning.

Spoiler:

Edit 1:
the <none> is the mimetype file.
As you can see, some books have more than one opf file.

Edit 2:
While I did a recursive scan, all zips and rars possible inside epubs are extracted also so they are not shown here, there content is.

Edit 3: Distribution of filesize

Spoiler:

SCION · 10-16-2011, 04:53 PM

Pardon my ignorance. Where can I find the plugin that's being referenced in this thread? Thank you.

Duh...never mind.

DoctorOhh · 10-16-2011, 08:05 PM

Quote:

Originally Posted by SCION

Pardon my ignorance. Where can I find the plugin that's being referenced in this thread? Thank you.

Duh...never mind.

For others the latest is Modify ePub 0.3.5 Beta and is found in this post.

kiwidude · 10-22-2011, 12:40 PM

This plugin is now officially released and can be found at this thread in the plugins forum.

kiwidude · 05-02-2012, 11:29 AM

@agama and @paulfiera...

I've got a new version of Quality Check (ready to test) and Modify ePub (in progress) as per the screenshots below... yell if you spot anything obvious I have forgotten.

I am intentionally not removing @font-face stuff at this point but if they are replaceable with a regex (rather than requiring css/html parsing) I will add support for it. Perhaps one of the css gurus out there may care to comment on whether some simple regex like "@font\-face\s+{[^}]+?}" would cover sufficient use-cases or not...

The other thing I want to investigate is cover insertion at some point though I have no doubt there are gremlins in there. I have seen people mention ePubFixer as something they use currently - is its approach recommended? I could steal ideas from that, but the brief read of its website said it produces svg covers, and I vaguely recall Kovid saying something about those and calibre having options not to produce them which presumably are for a reason...

Any chance Kovid or someone who knows about these things could enlighten me with some hints? Idolse "kind of" volunteered to add support for this feature a very long time ago but he seems to have been distracted by other things.

paulfiera · 05-02-2012, 01:17 PM

kiwidude: thanks, thanks and thanks

Just being able to identify which epubs have inline font declarations it's a huge time saver. I can take it from there and edit these epubs in Sigil and remove the inline font declarations.

Right now I'm opening every single epub in Sigil and checking for inline font declarations. When you take into account that this is a 4.000+ epubs library, well...I'd probably have to quit my day time job to be able to finish this quest in my lifetime

Regarding covers: I'd just want to put my 2 cents. More info here.

Many, many thanks, again.

kovidgoyal · 05-02-2012, 01:22 PM

Quote:

Originally Posted by paulfiera

Thanks, Walt.

Just tried it. Same results. Font declarations are moved from the css file to every html/xhtml file.

I really wonder why calibre does that.

Because unlike most other css rules, @fontface rules apply globally, not via a selector. Therefore they can, in principle, be different for every HTML file.

@kiwidude: Cover insertion is possible, (and you dont need to use SVG) provided you are willing to take the chance that your epub will end up with two, possibly duplicated, covers, in the case where the cover in the original file is not identified correctly. Or the case where the first html file is identified as the cover, but it contains non cover content which will get deleted. EPUB files in the wild are mess.

paulfiera · 05-02-2012, 01:31 PM

Quote:

Originally Posted by kovidgoyal

Because unlike most other css rules, @fontface rules apply globally, not via a selector. Therefore they can, in principle, be different for every HTML file.

Many thanks, Kovid.

I may be wrong, but when you have only one css applied to an epub, all these declarations apply to every html/xhtml file. Unless, of course, you have inline declarations, which will have higher specificity.

I'm aware that there can be several css files in a epub - especially in omnibus editions - and that several font declarations may apply to the different titles contained in an epub.

But...when those fonts no longer exist because they have been removed before the conversion, wouldn't it make sense to remove the declarations altogether?

kovidgoyal · 05-02-2012, 02:20 PM

All rules in the css are scoped inside selectors, which means they apply only to the elements matched by the selectors. @font-face rules do not have selectors.

Quote:

Originally Posted by paulfiera

But...when those fonts no longer exist because they have been removed before the conversion, wouldn't it make sense to remove the declarations altogether?

Certainly, however, it just isn't worth the effort. There are all sorts of redundancies in the markup of a typical book. This is just one example. I reserve my cleanup efforts to the cases where the cleanup makes some measurable impact on the performance of the resulting epub. But, that's not to say that I will refuse a patch for it.

kiwidude · 05-02-2012, 03:58 PM

I'm still wrapping my head around the font-face in every file thing

. So you are saying calibre does it because they "could" be different in each file, since each html file "might" link to a different css file in the original? But in the situation where you have a single css file, and the font-face declarations are contained within it, there is no need to replicate them into the html files, is that correct?

The case where I particularly notice it is when I edit in Sigil. And I have to scroll through hundreds of lines of font declarations just to get to the content (since code view is the only view worth doing most things in Sigil). So I end up writing a regex to blitz the whole inline style section across the epub (which is usually fine unless someone had edited the epub previously in Sigil and used inline styles from its horrible sgc-x Tidy crap).

As for the covers, thx for the info Kovid. I wasn't sure if there was some special reason why epubFixer goes the SVG route. If I can avoid that (I don't really know the "why" of SVG in an epub) then all the better.

So as I understand it the potential issues are:
- always inserting a cover will sometimes duplicate an existing cover.
- choosing to replace an existing cover is not good sometimes either, since as you say some epubs mark a content page as the cover.
- handling various issues of naming conflicts of a titlepage html, cover.jpg, manifest ids etc
- setting the guide to point to the new cover

I've certainly seen the impact of calibre's conversions sometimes blitzing content due to poorly defined covers in the original. And the missing image question mark icons from the standard logic getting tripped up. And often an extra blank page if you enable the "remove extra cover" option. There ain't no magic beans with this.

I guess the question is what do people want the plugin to do about it. Should the plugin just be completely "dumb" and offer a number of options, like "Replace existing cover" and "Insert cover", leaving it to the user to decide.

Or should it try to make an effort to make an "intelligent" guess. For instance...
- If a cover is defined in the guide, look for any text inside body html tags. If any text is found, treat the page as *not* being a replacable cover page, and insert a new cover instead. If no text is found, replace the content of that page with a new titlepage like the calibre one.
- If no cover is defined, always insert a new titlepage. There might be duplicates from the original not defining the first page as a "cover", but tough luck.

Any thoughts welcomed...

DoctorOhh · 05-02-2012, 09:09 PM

Quote:

Originally Posted by kiwidude

As for the covers, thx for the info Kovid. I wasn't sure if there was some special reason why epubFixer goes the SVG route. If I can avoid that (I don't really know the "why" of SVG in an epub) then all the better.

Epubfixer appears to say that they do exactly what calibre does with covers by default whenever a book is converted. SVG is calibre's default method of formatting covers during a conversion.

Quote:

Change the Cover without having to reconvert your file (Like calibre does)(Note that using this will convert your covers to SVG).

I haven't looked into any details so my assumption may be wrong.

kovidgoyal · 05-02-2012, 11:35 PM

Quote:

Originally Posted by kiwidude

I'm still wrapping my head around the font-face in every file thing

. So you are saying calibre does it because they "could" be different in each file, since each html file "might" link to a different css file in the original? But in the situation where you have a single css file, and the font-face declarations are contained within it, there is no need to replicate them into the html files, is that correct?

If the font face declarations for all html files are the same, then you can put them into the single css file. But note that this is almost never the case. For example, the cover html page and the jacket page typically do not share font-face declarations with the rest of the book, since they are inserted afterwards. If the font face declarations really bother you, the solution is to put them into eternal stylesheets, and link to them from each html file. Every html file that has a different set of font-faces will link to a different stylesheet. I've never bothered to do this, as IMO it isn't worth the effort, but a patch is welcome.

Also, as a quick and dirty fix, you can convert the epub to kf8 and then convert the kf8 back to epub. (KF8 requires all inline style info to be extracted into stylesheets, so calibre does it automatically in that case.

Quote:

As for the covers, thx for the info Kovid. I wasn't sure if there was some special reason why epubFixer goes the SVG route. If I can avoid that (I don't really know the "why" of SVG in an epub) then all the better.

SVG allows you to specify that the cover be fullscreen. That is why calibre (and presumably, epub fixer) use it.

Quote:

So as I understand it the potential issues are:
- always inserting a cover will sometimes duplicate an existing cover.
- choosing to replace an existing cover is not good sometimes either, since as you say some epubs mark a content page as the cover.
- handling various issues of naming conflicts of a titlepage html, cover.jpg, manifest ids etc
- setting the guide to point to the new cover

You will also have to replace/insert the <meta name="cover"> tag in the opf. And note that the id of the cover in the manifest must be "cover" and nothing else (which means you have to handle the case fo the manifest having something else with id+'cover" already) and it must be close to the top of the manifest, otherwise it will not work in some devices. And the <meta name="cover"> tag must have the name attribute before the content attribute.

kiwidude · 05-03-2012, 03:17 AM

Thx Kovid. What I am finding for the particular books that I am converting (like rtf files that I export to filtered html and then convert) is that there only ever was one "stylesheet" in the first place. Since I personally don't use metadata jackets, and the calibre cover page only has an image on it I'm trying to figure out if there is any reason for the font-face declarations to be replicated in that scenario? They really are a pain in Sigil, if there could be a conversion option somewhere to turn that off it would be wonderful...

10-02-2011, 08:37 AM	#198
drMerry Addict Posts: 293 Karma: 21022 Join Date: Mar 2011 Location: NL Device: Sony PRS-650	FYI, I made an export of a subset of ebooks in my library. These are the stats of 7072 epub-files. Maybe you can find some interesting info in it. A very curious one was the pdf file I found in one of the books. The file is manifested on a original epub. content of the pdf: the book cover (to print a paper cover to wrap around a hard-cover book). Including the offset information for the printing office. Curious because this book is sold as e-book this way. Another curious thing was the fact of empty folders and empty files. So, no request this time, just some stats you can maybe use to add extra clean-up functions or just to exclude some files of being removed. The examined files all where cleaned first by using the plugin. That's why e.g. no itunes files are shown. The xpgt files shown are files located in epubs with a bad opf-files causing the plugin to skip this epub while cleaning. Spoiler: Empty Folders: OEBPS/Images OEBPS/Fonts OEBPS/Styles Empty files (0 bytes): OEBPS/Styles/style.css OEBPS/Styles/style0001.css OEBPS/Fonts/Arial OEBPS/Fonts/AGaramondPro-Regular.otf Code: Extension File Sizes (KB) % of Total Files % of Files xhtml 3.534.981 45,1% 120.584 46,7% html 946.861 12,1% 39.115 15,2% jpg 1.592.273 20,3% 28.180 10,9% xml 155.732 2,0% 17.058 6,6% css 23.285 0,3% 8.885 3,4% png 321.662 4,1% 7.845 3,0% ncx 42.827 0,5% 7.073 2,7% <None> 144 0,0% 7.072 2,7% opf 37.060 0,5% 7.069 2,7% jpeg 210.567 2,7% 5.839 2,3% otf 649.969 8,3% 3.238 1,3% htm 187.503 2,4% 3.014 1,2% gif 73.818 0,9% 1.457 0,6% xpgt 2.640 0,0% 1.366 0,5% ttf 39.879 0,5% 135 0,1% thmx 58 0,0% 19 0,0% txt 3 0,0% 7 0,0% epub 476 0,0% 6 0,0% dfont 7.180 0,1% 5 0,0% bmp 1.449 0,0% 4 0,0% svg 1.016 0,0% 4 0,0% dtd 33 0,0% 3 0,0% bak 9 0,0% 2 0,0% dat 1 0,0% 2 0,0% db 32 0,0% 2 0,0% exe 1.281 0,0% 2 0,0% ds_store 1 0,0% 1 0,0% pdf 2.018 0,0% 1 0,0% Edit 1: the <none> is the mimetype file. As you can see, some books have more than one opf file. Edit 2: While I did a recursive scan, all zips and rars possible inside epubs are extracted also so they are not shown here, there content is. Edit 3: Distribution of filesize Spoiler: Distribution of sizes in H:\ebooks\Calibre Library Code: Size Interval Sum of File Sizes (KB) % of Total Files % of Files Over 1 GB 0 0,0% 0 0,0% 256 MB - 1 GB 0 0,0% 0 0,0% 64 MB - 256 MB 0 0,0% 0 0,0% 16 MB - 64 MB 0 0,0% 0 0,0% 4 MB - 16 MB 0 0,0% 0 0,0% 1 MB - 4 MB 129.282 1,7% 80 0,0% 256 KB - 1 MB 1.089.428 13,9% 2.292 0,9% 64 KB - 256 KB 3.094.362 39,5% 24.428 9,5% 16 KB - 64 KB 2.730.763 34,9% 85.201 33,0% 4 KB - 16 KB 696.930 8,9% 75.826 29,4% 1 KB - 4 KB 77.415 1,0% 32.676 12,7% 0 KB - 1 KB 14.570 0,2% 37.485 14,5% Last edited by drMerry; 10-02-2011 at 08:42 AM. Reason: extra info

10-16-2011, 04:53 PM	#199
SCION Séduisant Posts: 4,706 Karma: 2107018 Join Date: Jan 2010 Location: Texas, USA Device: Boox Note Air2+; Kobo Libra2; Kindle Scribe, Oasis3; iPad Mini6	Pardon my ignorance. Where can I find the plugin that's being referenced in this thread? Thank you. Duh...never mind. Last edited by SCION; 10-16-2011 at 04:58 PM.

05-02-2012, 11:29 AM	#202
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@agama and @paulfiera... I've got a new version of Quality Check (ready to test) and Modify ePub (in progress) as per the screenshots below... yell if you spot anything obvious I have forgotten. I am intentionally not removing @font-face stuff at this point but if they are replaceable with a regex (rather than requiring css/html parsing) I will add support for it. Perhaps one of the css gurus out there may care to comment on whether some simple regex like "@font\-face\s+{[^}]+?}" would cover sufficient use-cases or not... The other thing I want to investigate is cover insertion at some point though I have no doubt there are gremlins in there. I have seen people mention ePubFixer as something they use currently - is its approach recommended? I could steal ideas from that, but the brief read of its website said it produces svg covers, and I vaguely recall Kovid saying something about those and calibre having options not to produce them which presumably are for a reason... Any chance Kovid or someone who knows about these things could enlighten me with some hints? Idolse "kind of" volunteered to add support for this feature a very long time ago but he seems to have been distracted by other things. Attached Thumbnails

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Any web-to-epub plugin for internet browser?	bthoven	ePub	7	07-10-2011 05:14 AM
[Old Thread] Reading epub on viewer inexplicably changes the time stamp of epub	greenapple	Library Management	20	03-19-2011 10:18 PM
Easy way to modify thread subscription emails in bulk?	snipenekkid	Feedback	11	02-06-2011 03:47 AM
Another plugin dev question	DiapDealer	Plugins	2	12-11-2010 01:46 PM
Epub plugin dev	DiapDealer	Plugins	15	11-12-2010 09:36 AM

10-01-2011, 02:47 PM	#196
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	I'm saying that unless you do not use the Save to disk/Send to device function, your cover is going to (most of the time) get overwritten in your ePubs anyway when it ends up on your device/folder. So using this plugin to try to just update the metadata but not the cover is "futile" if your intent is to 100% guarantee that when the ePub is exported from calibre it still has your original large cover that is embedded. The only way I know of to ensure the cover remains at its large size, is to extract the cover from the ePub and make it your calibre cover. In which case using the update metadata command of the plugin is going to not negatively impact your cover at all since it will replace it with the same one.

10-22-2011, 12:40 PM	#201
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	This plugin is now officially released and can be found at this thread in the plugins forum.

05-02-2012, 01:17 PM	#203
paulfiera Addict Posts: 378 Karma: 3102 Join Date: Dec 2010 Location: EU Device: Kobo Aura ONE, Kobo Libra H20	kiwidude: thanks, thanks and thanks Just being able to identify which epubs have inline font declarations it's a huge time saver. I can take it from there and edit these epubs in Sigil and remove the inline font declarations. Right now I'm opening every single epub in Sigil and checking for inline font declarations. When you take into account that this is a 4.000+ epubs library, well...I'd probably have to quit my day time job to be able to finish this quest in my lifetime Regarding covers: I'd just want to put my 2 cents. More info here. Many, many thanks, again.

05-02-2012, 03:58 PM	#207
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	I'm still wrapping my head around the font-face in every file thing . So you are saying calibre does it because they "could" be different in each file, since each html file "might" link to a different css file in the original? But in the situation where you have a single css file, and the font-face declarations are contained within it, there is no need to replicate them into the html files, is that correct? The case where I particularly notice it is when I edit in Sigil. And I have to scroll through hundreds of lines of font declarations just to get to the content (since code view is the only view worth doing most things in Sigil). So I end up writing a regex to blitz the whole inline style section across the epub (which is usually fine unless someone had edited the epub previously in Sigil and used inline styles from its horrible sgc-x Tidy crap). As for the covers, thx for the info Kovid. I wasn't sure if there was some special reason why epubFixer goes the SVG route. If I can avoid that (I don't really know the "why" of SVG in an epub) then all the better. So as I understand it the potential issues are: - always inserting a cover will sometimes duplicate an existing cover. - choosing to replace an existing cover is not good sometimes either, since as you say some epubs mark a content page as the cover. - handling various issues of naming conflicts of a titlepage html, cover.jpg, manifest ids etc - setting the guide to point to the new cover I've certainly seen the impact of calibre's conversions sometimes blitzing content due to poorly defined covers in the original. And the missing image question mark icons from the standard logic getting tripped up. And often an extra blank page if you enable the "remove extra cover" option. There ain't no magic beans with this. I guess the question is what do people want the plugin to do about it. Should the plugin just be completely "dumb" and offer a number of options, like "Replace existing cover" and "Insert cover", leaving it to the user to decide. Or should it try to make an effort to make an "intelligent" guess. For instance... - If a cover is defined in the guide, look for any text inside body html tags. If any text is found, treat the page as not being a replacable cover page, and insert a new cover instead. If no text is found, replace the content of that page with a new titlepage like the calibre one. - If no cover is defined, always insert a new titlepage. There might be duplicates from the original not defining the first page as a "cover", but tough luck. Any thoughts welcomed...

05-03-2012, 03:17 AM	#210
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Thx Kovid. What I am finding for the particular books that I am converting (like rtf files that I export to filtered html and then convert) is that there only ever was one "stylesheet" in the first place. Since I personally don't use metadata jackets, and the calibre cover page only has an image on it I'm trying to figure out if there is any reason for the font-face declarations to be replicated in that scenario? They really are a pain in Sigil, if there could be a conversion option somewhere to turn that off it would be wonderful...

Advert

Advert