Modify ePub plugin dev thread

[paulfiera]

That was fast !!!. Currently installing 0.8.50

Many thanks again.

[jackie_w]

Originally Posted by JSWolf

IMHO, I'd prefer to have the aspect ratio kept with no option not to keep it. Those 510x680 resolution covers bother me because they aren't aspect correct.

...and I hope kiwidude ignores your request. Why are you constantly banging on about this, insisting that everyone else should want to see covers the way you want them? Surely the idea of using every user's personal calibre default for cover aspect ratio should keep everyone happy.

[DoctorOhh]

Originally Posted by jackie_w

Why are you constantly banging on about this, insisting that everyone else should want to see covers the way you want them?

While it is irritating, I was thinking the opposite, this is the first I heard him speak of this obsession in quite a while.

Originally Posted by jackie_w

Surely the idea of using every user's personal calibre default for cover aspect ratio should keep everyone happy.

I agree, as far as calibre is concerned this has already been discussed and decided.

[kovidgoyal]

@kiwidude: I was bored while waiting for today's calibre release to upload, so I implemented the stylesheet consolidation. This patch is without any testing at all, but it should be a good starting point for you:

=== modified file 'src/calibre/ebooks/oeb/transforms/flatcss.py'
--- src/calibre/ebooks/oeb/transforms/flatcss.py	2012-05-03 13:58:28 +0000
+++ src/calibre/ebooks/oeb/transforms/flatcss.py	2012-05-04 09:38:10 +0000
@@ -378,7 +378,7 @@
         for child in node:
             self.flatten_node(child, stylizer, names, styles, psize, item_id, left)
 
-    def flatten_head(self, item, stylizer, href):
+    def flatten_head(self, item, href, global_href):
         html = item.data
         head = html.find(XHTML('head'))
         for node in head:
@@ -390,24 +390,13 @@
                  and node.get('type', CSS_MIME) in OEB_STYLES:
                 head.remove(node)
         href = item.relhref(href)
-        etree.SubElement(head, XHTML('link'),
-            rel='stylesheet', type=CSS_MIME, href=href)
-        stylizer.page_rule['margin-top'] = '%gpt'%\
-                float(self.context.margin_top)
-        stylizer.page_rule['margin-bottom'] = '%gpt'%\
-                float(self.context.margin_bottom)
-
-        items = stylizer.page_rule.items()
-        items.sort()
-        css = '; '.join("%s: %s" % (key, val) for key, val in items)
-        style = etree.SubElement(head, XHTML('style'), type=CSS_MIME)
-        style.text = "\n\t\t@page { %s; }" % css
-        rules = [r.cssText for r in stylizer.font_face_rules]
-        raw = '\n\n'.join(rules)
-        # Make URLs referring to fonts relative to this item
-        sheet = cssutils.parseString(raw, validate=False)
-        cssutils.replaceUrls(sheet, item.relhref, ignoreImportRules=True)
-        style.text += '\n' + sheet.cssText
+        l = etree.SubElement(head, XHTML('link'),
+            rel='stylesheet', type=CSS_MIME, href=href)
+        l.tail='\n'
+        href = item.relhref(global_href)
+        l = etree.SubElement(head, XHTML('link'),
+            rel='stylesheet', type=CSS_MIME, href=href)
+        l.tail = '\n'
 
     def replace_css(self, css):
         manifest = self.oeb.manifest
@@ -418,6 +407,37 @@
         item = manifest.add(id, href, CSS_MIME, data=css)
         return href
 
+    def collect_global_css(self):
+        global_css = defaultdict(list)
+        for item in self.oeb.spine:
+            stylizer = self.stylizers[item]
+            stylizer.page_rule['margin-top'] = '%gpt'%\
+                    float(self.context.margin_top)
+            stylizer.page_rule['margin-bottom'] = '%gpt'%\
+                    float(self.context.margin_bottom)
+            items = stylizer.page_rule.items()
+            items.sort()
+            css = ';\n'.join("%s: %s" % (key, val) for key, val in items)
+            css = '@page {\n%s\n}\n'%css
+            rules = [r.cssText for r in stylizer.font_face_rules]
+            raw = '\n\n'.join(rules)
+            css += '\n\n' + raw
+            global_css[css].append(item)
+
+        gc_map = {}
+        manifest = self.oeb.manifest
+        for css in global_css:
+            id_, href = manifest.generate('page_css', 'page_styles.css')
+            manifest.add(id_, href, CSS_MIME, data=cssutils.parseString(css,
+                validate=False))
+            gc_map[css] = href
+
+        ans = {}
+        for css, items in global_css.iteritems():
+            for item in items:
+                ans[item] = gc_map[css]
+        return ans
+
     def flatten_spine(self):
         names = defaultdict(int)
         styles = {}
@@ -433,7 +453,8 @@
         items.sort()
         css = ''.join(".%s {\n%s;\n}\n\n" % (key, val) for key, val in items)
         href = self.replace_css(css)
+        global_css = self.collect_global_css()
         for item in self.oeb.spine:
             stylizer = self.stylizers[item]
-            self.flatten_head(item, stylizer, href)
+            self.flatten_head(item, href, global_css[item])

[kiwidude]

Haha Kovid, very cool thx. I wonder if I keep procrastinating long enough you can do the rest of it too...

I'm going to get my QC & Modify ePub releases sorted out and then can start to look into this. Though of course if you happened to have it done by then... well, we wouldn't mind, honest!

[kovidgoyal]

I went ahead and committed it, along with some changes to the rest of the conversion pipeline to deal with the fact that there is no longer a single external stylesheet. It still needs more testing, so let me know if you spot anything that breaks.

[kiwidude]

Thx Kovid, I will give it a look at home on the weekend. Sadly I can't get through the work firewall to the calibre repository so have to rely on downloading the zip files which of course won't contain that code until next week (or I make my own zip at home to download from).

Actually while on that subject - any chance of a calibre-debug command line option which asks calibre not to connect to the internet at all? It can never succeed with the firewalls here, and when I do my plugin testing all my debug output gets shoved around by the two error messages which get output over the opening minute. I *could* find the code and blitz it but right now I am not running from source and I would have to keep reapplying the change when I update etc (I don't want to install bzr on my work machine particularly given I can't reach the repo anyway).

Finally (to anyone) - I'm thinking of adding an option to the plugin to remove line-height declarations from css files. They cause me nothing but grief - I don't understand why the publisher should dictate that particularly when it just goes badly wrong and text either appears squashed or too spaced out. Usually I manually blitz them in Sigil. I "presume" they have some purpose for some device somewhere, I guess my question is can anyone tell me what that purpose is, and whether they might find such an option useful?

[jackie_w]

Originally Posted by dwanthny

While it is irritating, I was thinking the opposite, this is the first I heard him speak of this obsession in quite a while.

Lucky you... perhaps you don't frequent Nigol's ePubFixer thread, there was a skirmish quite recently

[kovidgoyal]

@kiwidude: calibre should support a simple authenticating http proxy. So unless you use a SOCKS proxy, it should not have any errors. I dont want to add an option to suppress internet connections as there's no simple way to do that. I can probably have the update check code print out a single line error instead of a full traceback, if that will help.

Here's the complete patch:

=== modified file 'src/calibre/ebooks/conversion/plugins/epub_output.py'
--- src/calibre/ebooks/conversion/plugins/epub_output.py	2012-03-17 13:35:07 +0000
+++ src/calibre/ebooks/conversion/plugins/epub_output.py	2012-05-04 10:07:10 +0000
@@ -312,13 +312,9 @@
         Perform various markup transforms to get the output to render correctly
         in the quirky ADE.
         '''
-        from calibre.ebooks.oeb.base import XPath, XHTML, OEB_STYLES, barename, urlunquote
+        from calibre.ebooks.oeb.base import XPath, XHTML, barename, urlunquote
 
-        stylesheet = None
-        for item in self.oeb.manifest:
-            if item.media_type.lower() in OEB_STYLES:
-                stylesheet = item
-                break
+        stylesheet = self.oeb.manifest.main_stylesheet
 
         # ADE cries big wet tears when it encounters an invalid fragment
         # identifier in the NCX toc.

=== modified file 'src/calibre/ebooks/conversion/plugins/pdf_output.py'
--- src/calibre/ebooks/conversion/plugins/pdf_output.py	2012-02-05 18:14:19 +0000
+++ src/calibre/ebooks/conversion/plugins/pdf_output.py	2012-05-04 10:07:10 +0000
@@ -99,12 +99,8 @@
 
         # Remove page-break-before on <body> element as it causes
         # blank pages in PDF Output
-        from calibre.ebooks.oeb.base import OEB_STYLES, XPath
-        stylesheet = None
-        for item in self.oeb.manifest:
-            if item.media_type.lower() in OEB_STYLES:
-                stylesheet = item
-                break
+        from calibre.ebooks.oeb.base import XPath
+        stylesheet = self.oeb.manifest.main_stylesheet
         if stylesheet is not None:
             from cssutils.css import CSSRule
             classes = set(['.calibre'])

=== modified file 'src/calibre/ebooks/oeb/base.py'
--- src/calibre/ebooks/oeb/base.py	2012-04-26 10:28:18 +0000
+++ src/calibre/ebooks/oeb/base.py	2012-05-04 10:07:10 +0000
@@ -1142,6 +1142,19 @@
             element(elem, OPF('item'), attrib=attrib)
         return elem
 
+    @dynamic_property
+    def main_stylesheet(self):
+        def fget(self):
+            ans = getattr(self, '_main_stylesheet', None)
+            if ans is None:
+                for item in self:
+                    if item.media_type.lower() in OEB_STYLES:
+                        ans = item
+                        break
+            return ans
+        def fset(self, item):
+            self._main_stylesheet = item
+        return property(fget=fget, fset=fset)
 
 class Spine(object):
     """Collection of manifest items composing an OEB data model book's main

=== modified file 'src/calibre/ebooks/oeb/transforms/flatcss.py'
--- src/calibre/ebooks/oeb/transforms/flatcss.py	2012-05-03 13:58:28 +0000
+++ src/calibre/ebooks/oeb/transforms/flatcss.py	2012-05-04 10:07:10 +0000
@@ -378,7 +378,7 @@
         for child in node:
             self.flatten_node(child, stylizer, names, styles, psize, item_id, left)
 
-    def flatten_head(self, item, stylizer, href):
+    def flatten_head(self, item, href, global_href):
         html = item.data
         head = html.find(XHTML('head'))
         for node in head:
@@ -390,34 +390,56 @@
                  and node.get('type', CSS_MIME) in OEB_STYLES:
                 head.remove(node)
         href = item.relhref(href)
-        etree.SubElement(head, XHTML('link'),
-            rel='stylesheet', type=CSS_MIME, href=href)
-        stylizer.page_rule['margin-top'] = '%gpt'%\
-                float(self.context.margin_top)
-        stylizer.page_rule['margin-bottom'] = '%gpt'%\
-                float(self.context.margin_bottom)
-
-        items = stylizer.page_rule.items()
-        items.sort()
-        css = '; '.join("%s: %s" % (key, val) for key, val in items)
-        style = etree.SubElement(head, XHTML('style'), type=CSS_MIME)
-        style.text = "\n\t\t@page { %s; }" % css
-        rules = [r.cssText for r in stylizer.font_face_rules]
-        raw = '\n\n'.join(rules)
-        # Make URLs referring to fonts relative to this item
-        sheet = cssutils.parseString(raw, validate=False)
-        cssutils.replaceUrls(sheet, item.relhref, ignoreImportRules=True)
-        style.text += '\n' + sheet.cssText
+        l = etree.SubElement(head, XHTML('link'),
+            rel='stylesheet', type=CSS_MIME, href=href)
+        l.tail='\n'
+        href = item.relhref(global_href)
+        l = etree.SubElement(head, XHTML('link'),
+            rel='stylesheet', type=CSS_MIME, href=href)
+        l.tail = '\n'
 
     def replace_css(self, css):
         manifest = self.oeb.manifest
-        id, href = manifest.generate('css', 'stylesheet.css')
         for item in manifest.values():
             if item.media_type in OEB_STYLES:
                 manifest.remove(item)
-        item = manifest.add(id, href, CSS_MIME, data=css)
+        id, href = manifest.generate('css', 'stylesheet.css')
+        item = manifest.add(id, href, CSS_MIME, data=cssutils.parseString(css,
+            validate=False))
+        self.oeb.manifest.main_stylesheet = item
         return href
 
+    def collect_global_css(self):
+        global_css = defaultdict(list)
+        for item in self.oeb.spine:
+            stylizer = self.stylizers[item]
+            stylizer.page_rule['margin-top'] = '%gpt'%\
+                    float(self.context.margin_top)
+            stylizer.page_rule['margin-bottom'] = '%gpt'%\
+                    float(self.context.margin_bottom)
+            items = stylizer.page_rule.items()
+            items.sort()
+            css = ';\n'.join("%s: %s" % (key, val) for key, val in items)
+            css = '@page {\n%s\n}\n'%css
+            rules = [r.cssText for r in stylizer.font_face_rules]
+            raw = '\n\n'.join(rules)
+            css += '\n\n' + raw
+            global_css[css].append(item)
+
+        gc_map = {}
+        manifest = self.oeb.manifest
+        for css in global_css:
+            id_, href = manifest.generate('page_css', 'page_styles.css')
+            manifest.add(id_, href, CSS_MIME, data=cssutils.parseString(css,
+                validate=False))
+            gc_map[css] = href
+
+        ans = {}
+        for css, items in global_css.iteritems():
+            for item in items:
+                ans[item] = gc_map[css]
+        return ans
+
     def flatten_spine(self):
         names = defaultdict(int)
         styles = {}
@@ -433,7 +455,8 @@
         items.sort()
         css = ''.join(".%s {\n%s;\n}\n\n" % (key, val) for key, val in items)
         href = self.replace_css(css)
+        global_css = self.collect_global_css()
         for item in self.oeb.spine:
             stylizer = self.stylizers[item]
-            self.flatten_head(item, stylizer, href)
+            self.flatten_head(item, href, global_css[item])
 

=== modified file 'src/calibre/ebooks/oeb/transforms/page_margin.py'
--- src/calibre/ebooks/oeb/transforms/page_margin.py	2011-04-14 14:59:27 +0000
+++ src/calibre/ebooks/oeb/transforms/page_margin.py	2012-05-04 10:07:10 +0000
@@ -9,7 +9,7 @@
 
 from collections import Counter
 
-from calibre.ebooks.oeb.base import OEB_STYLES, barename, XPath
+from calibre.ebooks.oeb.base import barename, XPath
 
 class RemoveAdobeMargins(object):
     '''
@@ -51,10 +51,7 @@
         self.stats = {}
         self.selector_map = {}
 
-        for item in self.oeb.manifest:
-            if item.media_type.lower() in OEB_STYLES:
-                stylesheet = item
-                break
+        stylesheet = self.oeb.manifest.main_stylesheet
         if stylesheet is None:
             return

[kovidgoyal]

Also, just FYI, you can get a tarball that contains the full calibre source code at any given revision from launchpad, by clicking the download tarball link, for example, here: http://bazaar.launchpad.net/~kovid/c...revision/12098

[kiwidude]

Thx Kovid. I don't know what sort of proxy they use here, I just know calibre can't cope with it . So nothing will work, be it the callback to the calibre mothership at startup, checks for updates, metadata downloads, plugin installs etc. When I looked at the calibre command line in the Help it seems there was an argument to prevent calibre looking for updates but only for the calibre.exe, not calibre-debug.exe. Not dumping the whole stack trace would certainly be a nice improvement thanks - after all it is an "expected" exception in this scenario so really could just have a user friendly debug message.

Thx also for that second revision stuff link btw - shall give that a whirl!

[kovidgoyal]

Committed.

[jackie_w]

@kiwidude,

A small observation with v1.03...

On the few epubs I've tested to remove the inline @font-faces, they are correctly removed but a blank line remains. This may be considered quite minor but if the epub has been created from html generated by MSWord it may amount to a lot of blank lines. My particular epub was 400 superfluous lines per epub html file. This is because Word-generated CSS seems to insist on creating a useless @font-face for every font found on your PC system, even if none of them are relevant to your particular document. I've never found a way of preventing this other than manually removing them from the source html file before importing to calibre. I'd think most users don't bother doing that.

[jackie_w]

... a second observation...

After modifying an epub, the calibre last_modified date column does not change. Is this deliberate? It may have always worked like this but I never noticed before.

[paulfiera]

Originally Posted by jackie_w

@kiwidude,

A small observation with v1.03...

On the few epubs I've tested to remove the inline @font-faces, they are correctly removed but a blank line remains. This may be considered quite minor but if the epub has been created from html generated by MSWord it may amount to a lot of blank lines.

I was going to post exactly that

It's quite noticeable with MS Word generated epubs.