ePUB Optimizer

[Toxaris]

This is an edit plugin that should work on Windows, Linux and OSX.

It is a wrapper plugin around my ePUB Optimizer program. It will subset the included fonts according to usage and optimize JPG, PNG and GIF images.

If a font is in the ePUB, but not actually used, it will be removed from the ePUB . The references will also be removed from the stylesheet(s). Usage of glyphs is determined by actually looking at the formatting of all elements, taking heredity into account.

Prereqs: Sigil 0.8.2 (or higher when it gets there) and Mono (for Linux/OSX). Also, on Linux the programs jpegtran/optipng/gifsicle are required. For Windows and OSX these are included. The program jpegtran is included in libjpeg-progs on Linux.
I have tested the program/plugin on Ubuntu 14.04 (64-bit) and 12.04 (32-bit). The 12.04 is actually Linux Mint 3. I have tested it with Mono 3.x only.

The plugin can be found here.

There are two options, subsetting based on fontfamily (so all fonts in the same family is treated equally) or based on fontdefinition. By default the second option will be used. It can be turned off by opening the ePUBOptimizer.ini and changing the 'usefontfamily' option to 'true'.

Current Version: "0.7.2"

[user_none]

Interesting that you're using the Python plugin interface to call a .Net application. It's just funny that Sigil is C++ which can call Python code which is now calling .Net code...

@Toxaris don't forget we designed the plugin system to be flexible to be able to call/use other languages than Python so if you're going to to make more .Net plugins we can look into a native .Net plugin interface so you don't have to go though the Python one.

[Toxaris]

Oh, it was a fun exercise. It is my first Python program and the examples and other plugins helped a lot. It is just that I am much more used to program in .Net. It also encouraged me to program in such a way that Mono could be used.

There is a small issue if there are media queries in the stylesheet, but that will be fixed later today. Bugs always tend to pop up after release...

[DiapDealer]

Just playing a bit, but I noticed that in one of my epubs, the font00345.ttf ("Kings-things") file disappears from the epub (and the css)--even though the font is used in several places. Any idea what might be going on?

Attaching a rather bloated version of "The Well at the World's End" to demonstrate.

[Toxaris]

I found the issue with the ePUB. The program is 'correct'. The 'King-things' font file is only used as bold. However, in the font-face it is not specified that the font is bold. If it is not specified, it will take the default, which is normal for both font-weight and font-style.
It has to do this, because a font-family can have multiple members and the used glyphs can change per member. For the defined member here, there are no glyphs in use.

If you would add 'font-weight: bold;' in your font-face declaration it will work as it should.

I hope I made it clear...

[Toxaris]

BTW, version 0.2 is placed. No change in the plugin itself, but in some cases media-queries in the stylesheet caused crashes.

[Doitsu]

Originally Posted by Toxaris

I found the issue with the ePUB. The program is 'correct'. The 'King-things' font file is only used as bold. However, in the font-face it is not specified that the font is bold.

AFAIK, only the font-family parameter is required. epubcheck doesn't report a warning or error message if the font-weight and font-style parameters are missing and epubs are correctly rendered by ADE without these parameters.

You may want to update your plugin so that it only removes fonts if the embedded fonts are either not used in the stylesheet or stylesheet definitions that reference the embedded fonts aren't actually used.

BTW, the plugin doesn't work with my Debian Linux 3.2.0-4-amd64 x86_64 distro, even though Debian mono 2.10.8.1-8 is installed. What mono version did you test it with?

It fails with the following error message:

Traceback (most recent call last):
  File "/usr/local/share/sigil/plugin_launchers//python/launcher.py", line 135, in launch
    self.exitcode = target_script.run(container)
  File "/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/plugin.py", line 58, in run
    with open(actionlist) as f:
IOError: [Errno 2] No such file or directory: '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/actionlist.txt'
Error: [Errno 2] No such file or directory: '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/actionlist.txt'

I assume that one of the binaries failed to generate actionlist.txt. To be on the safe side, you may want to check that actionlist.txt was actually generated, before you try to open it.

The Windows version of the plugin also failed with some books, for example this book, which crutledge uploaded a couple of days ago.

I'm getting the following error message:

Processing C:\Users\User\AppData\Local\sigil-ebook\sigil\plugins\ePUBOptimizer\dummy.epub
Get unique characters per font

[...]

Unhandled exception: System.Xml.XPath.XPathException: Namespace Manager or XsltContext needed. This query has a prefix, variable, or user-defined function.

[Toxaris]

Originally Posted by Doitsu

AFAIK, only the font-family parameter is required. epubcheck doesn't report a warning or error message if the font-weight and font-style parameters are missing and epubs are correctly rendered by ADE without these parameters.

You may want to update your plugin so that it only removes fonts if the embedded fonts are either not used in the stylesheet or stylesheet definitions that reference the embedded fonts aren't actually used.

If it is not specified, the default values of 'font-weight:normal' and 'font-style: normal' apply (see also here or here). In this case the font is used only as bold. There is no font in this font-family with that specifics. Therefore the font is considered as not being used. I would not be surprised at all if some readers or reading programs would not show the font due to this.
A font-family can have multiple members and the used glyphs per usage is determined. Fonts that are not used in text or stylesheet will be removed.

Originally Posted by Doitsu

BTW, the plugin doesn't work with my Debian Linux 3.2.0-4-amd64 x86_64 distro, even though Debian mono 2.10.8.1-8 is installed. What mono version did you test it with?

It fails with the following error message:

Traceback (most recent call last):
  File "/usr/local/share/sigil/plugin_launchers//python/launcher.py", line 135, in launch
    self.exitcode = target_script.run(container)
  File "/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/plugin.py", line 58, in run
    with open(actionlist) as f:
IOError: [Errno 2] No such file or directory: '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/actionlist.txt'
Error: [Errno 2] No such file or directory: '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/actionlist.txt'

I assume that one of the binaries failed to generate actionlist.txt. To be on the safe side, you may want to check that actionlist.txt was actually generated, before you try to open it.

Thanks, I was going to adapt the plugin to check for the actionlist already. It is strange that the actionlist is not written. I have tested it on Linux Mint. I will check my mono version, although I think that is not the issue here.

Originally Posted by Doitsu

The Windows version of the plugin also failed with some books, for example this book, which crutledge uploaded a couple of days ago.

I'm getting the following error message:

Processing C:\Users\User\AppData\Local\sigil-ebook\sigil\plugins\ePUBOptimizer\dummy.epub
Get unique characters per font

[...]

Unhandled exception: System.Xml.XPath.XPathException: Namespace Manager or XsltContext needed. This query has a prefix, variable, or user-defined function.

I will run his book through the debugger to see what is going on and where. So far I cannot reproduce this.

[Doitsu]

Originally Posted by Toxaris

If it is not specified, the default values of 'font-weight:normal' and 'font-style: normal' apply (see also here or here).

Even though epubs use XHTML, they're not websites and what works for websites often doesn't work for ebooks and vice versa.

Originally Posted by Toxaris

In this case the font is used only as bold. There is no font in this font-family with that specifics. Therefore the font is considered as not being used. I would not be surprised at all if some readers or reading programs would not show the font due to this.

DiapDealer's example book displays fine even in ADE 1.7. It might not display OK with other readers, but those readers usually also have other rendering problems.

As for the Windows issue, are there any minimum version requirements for the binaries/system libraries that your tool uses?

[DiapDealer]

I'm not going to beleaguer the point (I'm not one to do much font embedding in the first place), but it does seem a little drastic to throw a font out on such a technical definition of "being used." It renders in ADE, Adobe's RMSDK, Sigil and calibre, so the font is clearly "being used" (even if it's being used improperly).

It just seems odd to me that doing nothing other than removing "font-weight: bold" from the "first-words" class, makes your plugin believe the font is suddenly being used (with no change to the @font-face declaration) ... and that adding the line back in renders the font "unused" once again.

I do understand the technical distinction, but it doesn't seem appropriate to me to delete font files (and their accompanying css references) that are, in fact, being used. Inappropriately perhaps, but "used" nonetheless.

If I were doing it, "used" would mean: a font-face declaration; the font-family being used in a class selector; and that class being applied to elements in the (x)html that contain text. But in the end ... it's your plugin.

[Doitsu]

Originally Posted by DiapDealer

I'm not going to beleaguer the point (I'm not one to do much font embedding in the first place), but it does seem a little drastic to throw a font out on such a technical definition of "being used." It renders in ADE, Adobe's RMSDK, Sigil and calibre, so the font is clearly "being used" (even if it's being used improperly).

+1

@Toxaris: If you strongly believe in this feature, you may want to make it at least an optional feature, because I for one, won't use a plugin that might delete declarations and content, just because they're not 100% perfect. (If you don't want to create a GUI for it, you can easily define plugin settings with a user-editable .ini file. For an example, have a look at my very simple GUI-less Kindlegen plugin.)

[Toxaris]

Originally Posted by Doitsu

BTW, the plugin doesn't work with my Debian Linux 3.2.0-4-amd64 x86_64 distro, even though Debian mono 2.10.8.1-8 is installed. What mono version did you test it with?

It fails with the following error message:

Traceback (most recent call last):
  File "/usr/local/share/sigil/plugin_launchers//python/launcher.py", line 135, in launch
    self.exitcode = target_script.run(container)
  File "/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/plugin.py", line 58, in run
    with open(actionlist) as f:
IOError: [Errno 2] No such file or directory: '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/actionlist.txt'
Error: [Errno 2] No such file or directory: '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/actionlist.txt'

I assume that one of the binaries failed to generate actionlist.txt. To be on the safe side, you may want to check that actionlist.txt was actually generated, before you try to open it.

The Windows version of the plugin also failed with some books, for example this book, which crutledge uploaded a couple of days ago.

I'm getting the following error message:

Processing C:\Users\User\AppData\Local\sigil-ebook\sigil\plugins\ePUBOptimizer\dummy.epub
Get unique characters per font

[...]

Unhandled exception: System.Xml.XPath.XPathException: Namespace Manager or XsltContext needed. This query has a prefix, variable, or user-defined function.

The ePUB used pseudo classes like hover and visited. That is what caused the barfing. This is now fixed on my local version. The failure with the actionlist does now not happen with my local version either.

[Toxaris]

Originally Posted by DiapDealer

I'm not going to beleaguer the point (I'm not one to do much font embedding in the first place), but it does seem a little drastic to throw a font out on such a technical definition of "being used." It renders in ADE, Adobe's RMSDK, Sigil and calibre, so the font is clearly "being used" (even if it's being used improperly).

It just seems odd to me that doing nothing other than removing "font-weight: bold" from the "first-words" class, makes your plugin believe the font is suddenly being used (with no change to the @font-face declaration) ... and that adding the line back in renders the font "unused" once again.

I do understand the technical distinction, but it doesn't seem appropriate to me to delete font files (and their accompanying css references) that are, in fact, being used. Inappropriately perhaps, but "used" nonetheless.

If I were doing it, "used" would mean: a font-face declaration; the font-family being used in a class selector; and that class being applied to elements in the (x)html that contain text. But in the end ... it's your plugin.

The point is, it is not a technical definition. Let me try to explain in more detail what my problem is. The purpose to remove as many glyphs as possible to get the smallest font files. Lets give an example. Lets say I implement Charis Sil as base font. That would mean that I would need 4 fonts to implement this correctly: regular, bold, italic and bolditalic. I cannot just do with one, because otherwise there would be no italic and such. All those 4 fonts have the same font-family of course.
Now, if I use certain glyphs in italic but not in bold, they could be removed from bold.
The way to discriminate between these fonts, are the attributes font-weight and font-style. I have to tell in the stylesheet which is which. If I don't add those, the reading application should fall back to the default value which is normal for both. If I would not use it, the wrong font would be used.
The same applies for all the other default values for all classes/tags. Headers are bold by default for example.

If I would ignore the font-weight and font-style, I would not know which glyphs could be deleted from the fonts in the same family. I could combine of course all used characters for that font-family, but then the font is not optimized fully. There is of course a difference between a font and a font-family.

I am actually quite surprised that ADE shows the font when the usage is actually not correct. Does it also render on ADE 1.7?

Perhaps I will add a feature that only discriminates on font-family used instead of actually used fonts as it is now.

[Toxaris]

Originally Posted by user_none

@Toxaris don't forget we designed the plugin system to be flexible to be able to call/use other languages than Python so if you're going to to make more .Net plugins we can look into a native .Net plugin interface so you don't have to go though the Python one.

I would really like a .Net plugin interface, but I assume that would be quite some work to do.

[DiapDealer]

Originally Posted by Toxaris

The point is, it is not a technical definition. Let me try to explain in more detail what my problem is. The purpose to remove as many glyphs as possible to get the smallest font files. Lets give an example. Lets say I implement Charis Sil as base font. That would mean that I would need 4 fonts to implement this correctly: regular, bold, italic and bolditalic. I cannot just do with one, because otherwise there would be no italic and such. All those 4 fonts have the same font-family of course.
Now, if I use certain glyphs in italic but not in bold, they could be removed from bold.
The way to discriminate between these fonts, are the attributes font-weight and font-style. I have to tell in the stylesheet which is which. If I don't add those, the reading application should fall back to the default value which is normal for both. If I would not use it, the wrong font would be used.
The same applies for all the other default values for all classes/tags. Headers are bold by default for example.

If I would ignore the font-weight and font-style, I would not know which glyphs could be deleted from the fonts in the same family. I could combine of course all used characters for that font-family, but then the font is not optimized fully. There is of course a difference between a font and a font-family.

I am actually quite surprised that ADE shows the font when the usage is actually not correct. Does it also render on ADE 1.7?

Perhaps I will add a feature that only discriminates on font-family used instead of actually used fonts as it is now.

As I said, I understand your reasoning perfectly: I just don't agree with it. It's a perfectly reasonable approach to take with a font that is intended to be used as a complete replacement of the body text (multiple @font-face declarations, multiple font-files to represent the entire font "family"), but I imagine that that same approach is going to remove a lot of ornamental fonts that are only used for chapter headers, drop/raised caps and the like. Situations where it would be perfectly sensible to base the subsetting on overall character/glyph usage (a per-family basis). Because weight/style just aren't usually all that relevant in a one-font-file, one @font-face, ornamental-only embedded font. *shrug*

And yes, ADE 1.7 renders the embedded font in my sample epub. I've yet to find a device/app that doesn't (not to say there aren't any).

Discrimination based on font-family would certainly be a useful feature for me. Otherwise, I have to remember that even though my epub passes validation with flying colors, there may be situations where my ornamental fonts disappear when using this plugin.

But enough!! Glad to see you contributing to the plugin cause. The more the merrier.