Modify ePub plugin dev thread

[Dopedangel]

About updating covers I did not use svg covers earlier as my reader did not support them will this plugin be able to check if the cover is svg compatible if not maybe add the svg code. Also the abilty to remove svg code would be good for people that don't like or their reader is unable to show svg covers.

[anamardoll]

This is genius -- thank you! I was also wanting to strip embedded fonts from my epubs and was dreading doing it by hand. Seems to have worked perfectly so far. THANK YOU THANK YOU THANK YOU!

[Japes]

Thanks for this wonderful plugin but, I had a quick question. What EXACTLY is an embedded font? I understood it to mean any Epubs with fonts referenced at all. For example, I have the LexiDaMa font installed on my Sony reader and, for MOST epubs, when I convert using Calibre, it works, but, every now and then, LexiaDaMa is NOT displayed, which I took to mean that the epub had embedded fonts, however, after using this plugin, for some reason, my selected font (LexiaDaMa) is still not displaying on the problematic Epubs.

Am I doing something wrong? I will cut and paste the "Extra CSS" that I use on every conversion.




@font-face {
font-family: "LexiaDaMa";
font-weight: normal;
font-style: normal;
src: url(res:///Data/fonts/LexiaDaMa.otf);
}

@font-face {
font-family: "LexiaDaMa";
font-weight: bold;
font-style: normal;
src: url(res:///Data/fonts/LexiaDaMa-Bold.otf);
}

@font-face {
font-family: "LexiaDaMa";
font-weight: normal;
font-style: italic;
src: url(res:///Data/fonts/LexiaDaMa-Italic.otf);
}

@font-face {
font-family: "LexiaDaMa";
font-weight: bold;
font-style: italic;
src: url(res:///Data/fonts/LexiaDaMa-BoldItalic.otf);
}

body {
font-family: "LexiaDaMa";
}

[kiwidude]

Hi Japes, welcome to MobileRead.

I can't comment on the Sony Reader behaviour, I leave that to others.

I can tell you what an embedded font is as far as this plugin is concerned. An embedded font is one or more font files (.ttf or .otf) embedded inside the epub. In addition to the physical font files, there will usually be references to these fonts from the .opf manifest file inside the ePub, indicating they are a part of the ePub contents. Finally, there will be css declarations of @font-face, pointing to the embedded font files.

This plugin can remove the physical font files and any links to them from the opf manifest. What it does not do is touch the css file in any way. So as Kovid said in a few posts earlier, if the ereader processing the stylesheet does not find the font at the location pointed to in the @font-face declaration, then it is ignored.

So to me what you are doing with pointing to a font on your Sony is not what is classed as an embedded font, because the font is not contained inside the ePub and instead you are hard-coding to a font on your reader.

[Japes]

Gotcha. Thanks for the clarification. So, now I understand why it's not working for me. There are STILL (even after running this plugin), Font-Family references (for example...Times New Roman), in the CSS file, which screws things up for me.

[kiwidude]

@Japes - yes, this plugin doesn't have any CSS removal features. Perhaps one day it might have if others think it is sensible/possible - as a Kindle owner I likewise remove all font-family references from my ePubs. Though I don't need any @font-face declarations, I just leave that to the Kindle to use whatever it's default font that is configured. I am surprised the Sony does not work the same way, but as I said I am not a Sony owner.

[ldolse]

My 2 cents on the css removal features - I think there are a couple things that could be both feasible and recommended. This plugin has led me to a crusade to remove all the garbage margins from my epubs, and I'm learned that in some cases margins I've been blaming on adobe xpgt are actually css margins. In the examples I've looked at these are easy to remove programmatically - basically just parse the css and look for any margins on body or @page, and delete/zero these. Margins on any other element could potentially be specific, but margins on body and @page are safe to delete wholesale.

To keep the UI simple perhaps the Modify Epub plugin should just have a checkbox to 'remove book level' margins or something along those lines and then delete both css and Adobe types of margins.

A preference to add back an @page css margin of the user's specification could also be useful, as it seems like the best practice from the epub sub-forums (and Calibre's default) is to use @page to specify margins across the book.


On the Adobe xpgt front, I've found a couple useful things. Number one is a mimetype is missing - I had to add a third mimetype to the function to successfully modify the margins on some files:

            if (mt.lower() in ('application/vnd.adobe-page-template+xml',
                    'application/adobe-page-template+xml', 'application/vnd.adobe.page-template+xml') and
                    hasattr(data, 'xpath')):

Oddly enough Quality Check did find xpgt files with that mimetype....

Second, I've discovered another anomaly between the way Quality Check and Modify Epub work. Quality check is discovering xpgt files which have margins, but then Modify Epub was unable to remove them. Further investigation showed that the reason was the xpgt files weren't in the OPF manifest. Now it turns out that Adobe DE ignores unmanifested xpgt files, so that leaves some options on how to handle them. I'd like to fix things so that these books don't show up in future Quality Check searches, so that leaves two options - either Quality Check recognizes that unmanifested xpgt files don't return 'True' for the xpgt margins check, or the Modify ePub plugin deletes the offending xpgt files completely.

[kiwidude]

@Idolse - yes you are correct in there is inconsistency between the two plugins.

The Quality Check plugin does not check the manifest, it just looks for files of that name. As you have found above the Modify ePub plugin does check the manifest. I agree they should be made consistent. I appreciate your investigation that unmanifested xpgt files have no effect, I had no idea of that. So I will change Quality Check to also check the manifest (with your extra mimetype). That way any unmanifested xpgt files will be identified in the unmanifested files check and can be removed using the remove unmanifested files option in this plugin.

The css file stuff I will keep mulling over. I confess to rarely changing body/page margins myself, as the ePubs I have edited haven't needed it. There is the standard 5pt left/right margin that Calibre puts in place, but for my own purposes that hasn't been an issue. Perhaps as a Kindle owner and Calibre's own "margin removal" option around mobi I have that taken care of elsewhere, I do not know. Far more often I see epubs where it is either the xpgt file or other non-body/page styles that have nasty large margins set on them like top/bottom or large left/right. That tends to be very bespoke to fix though of course, as you do not want to affect indented paragraphs versus whatever is impacting the whole page when a badly edited book is not as you say relying on @page.

[anamardoll]

My 2 cents on the css removal features - I think there are a couple things that could be both feasible and recommended. This plugin has led me to a crusade to remove all the garbage margins from my epubs, and I'm learned that in some cases margins I've been blaming on adobe xpgt are actually css margins. In the examples I've looked at these are easy to remove programmatically - basically just parse the css and look for any margins on body or @page, and delete/zero these. Margins on any other element could potentially be specific, but margins on body and @page are safe to delete wholesale.

I would LOVE this, although it would be even better if the margins could be customized (i.e., set all book margins to X).

I just opened an epub the other day where the indie author or whoever formatted their book for them set the margins at what seems to be half an inch on either side. A whole inch of white space running down my reader for the entire book! I like a little margin so that the shade of the bezel isn't obstructing the text, but this is nonsense.

[kiwidude]

Changes in this release:

• No longer look in manifest for NCX file, look for physical file instead to get around media-type variant issues
• If cancel updating the ePubs, remove the temp directory
• Additional mime type for xpgt files as supplied by Idolse

This requires Calibre 0.8.5.

No new functionality, just some tweaks to improve the coverage of what is there already.

[capnm]

Stumbled across this:

Modifying:  e:\_tempf~1\calibre_0.8.6_tmp_m8saka\calibre_0.8.6_f6jxn1_modify_epub\128.epub
	Looking for files to remove: [u'.DS_Store', u'thumbs.db']
	  Found file to remove: OPS/Thumbs.db
	  Manifest item removed: Thumbs.db (x_9780061757068_88)
Magyk - ERROR: Traceback (most recent call last):
  File "calibre_plugins.modify_epub.modify", line 71, in process_book
  File "calibre_plugins.modify_epub.modify", line 139, in _process_book
  File "calibre_plugins.modify_epub.modify", line 256, in _remove_files_if_exist
  File "calibre_plugins.modify_epub.container", line 129, in delete_from_manifest
  File "calibre_plugins.modify_epub.container", line 175, in delete_from_toc
AttributeError: 'str' object has no attribute 'xpath'

I'll second (third?) Idolse's comment on body & @page margins.
I've been just living with the annoying margins (it hardly seems worth hand fixing a thriller I'm going to read in a matter of hours & discard) but the xpgt removal is so easy and delightful now I want to fix the rest
And they're almost all as Idolse described. I suppose arguably Calibre's 5pt margins in the text block are 'book level' margins too, but they don't bother me (and are easy to control at conversion time)

A preference to add back an @page css margin of the user's specification could also be useful, as it seems like the best practice from the epub sub-forums (and Calibre's default) is to use @page to specify margins across the book.

I think Calibre uses @page for top/bottom but .text class for right/left ....




Oh, I just realized this can get annoyingly messy... while the retail epubs I've been reading have one @page in the style sheet, Calibre puts an @page in each html block ...

[paulfiera]

Originally Posted by capnm

I'll second (third?) Idolse's comment on body & @page margins.

Me too.

[kiwidude]

@capnm - yes, it is the fact that it can get "annoyingly messy" as you say as to why I haven't looked into the page/body margins. I'm not an HTML/CSS guru by any means but as you have found there are multiple ways these things can be declared. Now if someone is prepared to spend the time posting what the variations are that the plugin should handle then I can take a look. However without that info it would involve me spending an awful lot of time figuring it out for myself (given my CSS ignorance) which I lack the enthusiasm for.

With regards to your other error, that looks interesting. It seems to be a problem with your NCX file. Any chance you could PM me a link to that ePub that had the issue?

[capnm]

Originally Posted by kiwidude

Now if someone is prepared to spend the time posting what the variations are that the plugin should handle then I can take a look.

I would just fix .body & @page in the css. (And, of course, zero the xpgt)

I did some keyword searching across a large bunch of epubs and concluded that those were the only commonplace 'book level' margins found in commercial epubs. If that doesn't solve it, it's too complex for this plugin

The next level would be to look at Calibre generated epubs, but then it gets messy fast.
I think the older versions of Calibre put all 4 'book level' margins in .calibre in the css, but the newer Calibre puts the top/bottom margins in each split, rather than the css, so that's not readily fixable. The left/right margins are in .text in the css.

On the other hand, it seems to me a Calibre generated epub has a really good chance of surviving a Calibre epub->epub conversion intact, so maybe that's the easiest way to adjust margins in that case.

Thus just using this plugin to tweak epubs you don't want to risk doing a full conversion on.

The other special case is books with multiple css files. I'd suggest they're too complex for this plugin too.

So while I'd just modify the simplest cases (which covers the vast majority), it might make sense for Check Epub to also identify some more complex cases.

I like the idea of this being able to add @page margins to the css too.

Maybe somebody who actually knows what they're talking about should revisit how Calibre handles 'book level' margins

[ldolse]

I agree with Capnm - when I suggested this initially I wasn't even thinking about all the potential places a css style could be specified. Generally when I see offensive ( i.e. publishing house created) margins like this they're actually in the .css file, not used in-line. A best effort approach which only modified the actual css files should be sufficient for the vast majority of cases.