ePUB Optimizer
 

This is an edit plugin that should work on Windows, Linux and OSX.

It is a wrapper plugin around my ePUB Optimizer program. It will subset the included fonts according to usage and optimize JPG, PNG and GIF images.

If a font is in the ePUB, but not actually used, it will be removed from the ePUB . The references will also be removed from the stylesheet(s). Usage of glyphs is determined by actually looking at the formatting of all elements, taking heredity into account.

Prereqs: Sigil 0.8.2 (or higher when it gets there) and Mono (for Linux/OSX). Also, on Linux the programs jpegtran/optipng/gifsicle are required. For Windows and OSX these are included. The program jpegtran is included in libjpeg-progs on Linux.
I have tested the program/plugin on Ubuntu 14.04 (64-bit) and 12.04 (32-bit). The 12.04 is actually Linux Mint 3. I have tested it with Mono 3.x only.

The plugin can be found here.

There are two options, subsetting based on fontfamily (so all fonts in the same family is treated equally) or based on fontdefinition. By default the second option will be used. It can be turned off by opening the ePUBOptimizer.ini and changing the 'usefontfamily' option to 'true'.

Current Version: "0.7.2"

Interesting that you're using the Python plugin interface to call a .Net application. It's just funny that Sigil is C++ which can call Python code which is now calling .Net code...

@Toxaris don't forget we designed the plugin system to be flexible to be able to call/use other languages than Python so if you're going to to make more .Net plugins we can look into a native .Net plugin interface so you don't have to go though the Python one.

Oh, it was a fun exercise. It is my first Python program and the examples and other plugins helped a lot. It is just that I am much more used to program in .Net. It also encouraged me to program in such a way that Mono could be used.

There is a small issue if there are media queries in the stylesheet, but that will be fixed later today. Bugs always tend to pop up after release...

Just playing a bit, but I noticed that in one of my epubs, the font00345.ttf ("Kings-things") file disappears from the epub (and the css)--even though the font is used in several places. Any idea what might be going on?

Attaching a rather bloated version of "The Well at the World's End" to demonstrate.

I found the issue with the ePUB. The program is 'correct'. The 'King-things' font file is only used as bold. However, in the font-face it is not specified that the font is bold. If it is not specified, it will take the default, which is normal for both font-weight and font-style.
It has to do this, because a font-family can have multiple members and the used glyphs can change per member. For the defined member here, there are no glyphs in use.

If you would add 'font-weight: bold;' in your font-face declaration it will work as it should.

I hope I made it clear...

BTW, version 0.2 is placed. No change in the plugin itself, but in some cases media-queries in the stylesheet caused crashes.

AFAIK, only the font-family parameter is required. epubcheck doesn't report a warning or error message if the font-weight and font-style parameters are missing and epubs are correctly rendered by ADE without these parameters.

You may want to update your plugin so that it only removes fonts if the embedded fonts are either not used in the stylesheet or stylesheet definitions that reference the embedded fonts aren't actually used.

BTW, the plugin doesn't work with my Debian Linux 3.2.0-4-amd64 x86_64 distro, even though Debian mono 2.10.8.1-8 is installed. What mono version did you test it with?

It fails with the following error message:

I assume that one of the binaries failed to generate actionlist.txt. To be on the safe side, you may want to check that actionlist.txt was actually generated, before you try to open it.

The Windows version of the plugin also failed with some books, for example this book, which crutledge uploaded a couple of days ago.

I'm getting the following error message:

If it is not specified, the default values of 'font-weight:normal' and 'font-style: normal' apply (see also here or here). In this case the font is used only as bold. There is no font in this font-family with that specifics. Therefore the font is considered as not being used. I would not be surprised at all if some readers or reading programs would not show the font due to this.
A font-family can have multiple members and the used glyphs per usage is determined. Fonts that are not used in text or stylesheet will be removed.

Thanks, I was going to adapt the plugin to check for the actionlist already. It is strange that the actionlist is not written. I have tested it on Linux Mint. I will check my mono version, although I think that is not the issue here.

I will run his book through the debugger to see what is going on and where. So far I cannot reproduce this.

Even though epubs use XHTML, they're not websites and what works for websites often doesn't work for ebooks and vice versa.

DiapDealer's example book displays fine even in ADE 1.7. It might not display OK with other readers, but those readers usually also have other rendering problems.

As for the Windows issue, are there any minimum version requirements for the binaries/system libraries that your tool uses?

I'm not going to beleaguer the point (I'm not one to do much font embedding in the first place), but it does seem a little drastic to throw a font out on such a technical definition of "being used." It renders in ADE, Adobe's RMSDK, Sigil and calibre, so the font is clearly "being used" (even if it's being used improperly).

It just seems odd to me that doing nothing other than removing "font-weight: bold" from the "first-words" class, makes your plugin believe the font is suddenly being used (with no change to the @font-face declaration) ... and that adding the line back in renders the font "unused" once again.

I do understand the technical distinction, but it doesn't seem appropriate to me to delete font files (and their accompanying css references) that are, in fact, being used. Inappropriately perhaps, but "used" nonetheless.

If I were doing it, "used" would mean: a font-face declaration; the font-family being used in a class selector; and that class being applied to elements in the (x)html that contain text. But in the end ... it's your plugin. :)

+1

@Toxaris: If you strongly believe in this feature, you may want to make it at least an optional feature, because I for one, won't use a plugin that might delete declarations and content, just because they're not 100% perfect. (If you don't want to create a GUI for it, you can easily define plugin settings with a user-editable .ini file. For an example, have a look at my very simple GUI-less Kindlegen plugin.)

The ePUB used pseudo classes like hover and visited. That is what caused the barfing. This is now fixed on my local version. The failure with the actionlist does now not happen with my local version either.

The point is, it is not a technical definition. Let me try to explain in more detail what my problem is. The purpose to remove as many glyphs as possible to get the smallest font files. Lets give an example. Lets say I implement Charis Sil as base font. That would mean that I would need 4 fonts to implement this correctly: regular, bold, italic and bolditalic. I cannot just do with one, because otherwise there would be no italic and such. All those 4 fonts have the same font-family of course.
Now, if I use certain glyphs in italic but not in bold, they could be removed from bold.
The way to discriminate between these fonts, are the attributes font-weight and font-style. I have to tell in the stylesheet which is which. If I don't add those, the reading application should fall back to the default value which is normal for both. If I would not use it, the wrong font would be used.
The same applies for all the other default values for all classes/tags. Headers are bold by default for example.

If I would ignore the font-weight and font-style, I would not know which glyphs could be deleted from the fonts in the same family. I could combine of course all used characters for that font-family, but then the font is not optimized fully. There is of course a difference between a font and a font-family.

I am actually quite surprised that ADE shows the font when the usage is actually not correct. Does it also render on ADE 1.7?

Perhaps I will add a feature that only discriminates on font-family used instead of actually used fonts as it is now.

I would really like a .Net plugin interface, but I assume that would be quite some work to do.

As I said, I understand your reasoning perfectly: I just don't agree with it. It's a perfectly reasonable approach to take with a font that is intended to be used as a complete replacement of the body text (multiple @font-face declarations, multiple font-files to represent the entire font "family"), but I imagine that that same approach is going to remove a lot of ornamental fonts that are only used for chapter headers, drop/raised caps and the like. Situations where it would be perfectly sensible to base the subsetting on overall character/glyph usage (a per-family basis). Because weight/style just aren't usually all that relevant in a one-font-file, one @font-face, ornamental-only embedded font. *shrug*

And yes, ADE 1.7 renders the embedded font in my sample epub. I've yet to find a device/app that doesn't (not to say there aren't any).

Discrimination based on font-family would certainly be a useful feature for me. Otherwise, I have to remember that even though my epub passes validation with flying colors, there may be situations where my ornamental fonts disappear when using this plugin.

But enough!! Glad to see you contributing to the plugin cause. The more the merrier. :)

Hi Toxaris,

That really depends on how much hand holding you want to do for the .net plugin developer.

At its minimum, a plugin interface gets sent the the plugin type, the path to the plugin, the internal Sigil directory where the ebook files are stores (unzipped), and a temp directory that is created and owned by Sigil.

"Something" then has to protect the internal Sigil files, and allow direct reading of internal Sigil ebook files but that does a file-copy-on-write/change to the temp directory, keeping track of what has been changed by the plugin (modified, deleted, added). And then build a result xml stream that is returned back to Sigil with the list of changed, modified, added, deleted files.

Sigil takes this result xml stream and parses it and updates its own internal Sigil ebook files based on the result xml file and the modified files in the temp directory.

For python plugins, I wanted this to be as simple as possible, so I wrote a wrapper.py and *container.py classes that implement this "Something" for every python plugin. The wrapper.py will automatically parses the opf, and keeps track of copying the files upon write/change, which files you want to delete, which files you want to add, and it automatically creates the result xml file for the plugin developer. So the python launcher.py and wrapper.py code do a lot of hand holding and interface building for the python plugin developer so that they don't have to worry about the interface and result xml and copy-on-modify/write approach used to protect the ebook files in Sigil.

So if you wanted you could create a quite bare bones .NET launcher without the equivalent of the wrapper.py and *container.py and simply allow the plugin developer to have direct access to files and just tell them they need to write any changes to the temp directory, and make tell them to create their own result.xml instead of automating that, etc.

Then if interest develops, you could add some of the more powerful .net xml processing tools and develop a full-blown launcher/wrapper for .net

Take care,

KevinH

It is possible to define a font as "normal" in the @font-face, and the use it as bold and/or italic in other font-family rules, expecting the renderer to artificially embolden and/or slant the font (if no other appropriate bold/italic @font-face was defined).

Whether the renderer will actually do this, or just use the normal font, or use any other bold/italic font available, that's a different matter. And how easy it is to detect this situation is yet another different matter :D

Although I still feel it is wrong, I am nothing but flexible (although some might disagree there...:rofl:).

I have adapted the program to accept an additional startup parameter so that usage is calculated per font-family only. Now I have to adapt the python script and it should work as some of you would like. :D

Oh, you didn't fight nearly long enough before giving in to external pressure. ;)

I would like it to work as it originally worked as you are correct and the others are incorrect.

It works both ways now (after I release it...). By default 'my way' and by setting an option the other way.

Version 0.3 with the option for other subset calculation is released.

Thanks for implementing my suggestion.

BTW, the new version worked fine with DiapDealer's and crutledge's books on my Windows machine, but on my Linux machine I'm getting the following error message (regardless of the book):

No problem, you are aware that you have not activated it yet? I can see that in the log you placed below...

Hmm, strange. It works fine with me. My testing environment (32-bit):
- Linux Mint 13 Maya (basically Ubuntu 12.04)
- Mono 3.10
- Sigil 0.8.2 (although this doesn't matter at all)

I have no idea which Mono version is actually required. When I wanted to test the program, I just downloaded and installed Mono to be honest. I have also tested the program on Ubuntu 14.01 (64-bit) and Mono 3.10.
As the program is built against .Net Framework 4 (client profile even) and does not use 'exotic' programming, I would not expect the Mono version actually be a problem.
However, it may be that you are missing a Mono package. I found some online reports that say that monodevelop may be needed. I installed the full package myself... So, I think it is a missing mono library.

It is activated. I.e. the error must occur before the .ini file is being loaded.

I'm using Debian Wheezy, which came with Mono 2.10.8.1-8.

I installed monodevelop, but it didn't make a difference. Most likely my mono version is too old, however, since a manual upgrade might lead to an unstable system, I'm going to use the Windows version of Sigil 0.8.2 instead.

Maybe you can mention in the first post, that Mono 3.x and/or a current Ubuntu-based distro might be required for the Linux version.

Nope, ini file is loaded from the start. I can see it because I haven't cleaned up printing the start command which I used for backup. It says:
but it should say:
Remember, the value of the parameter 'usefontfamily' must be set to 'True' to make it work like you requested.

Will do.

Hi,

I am trying to test things under Mac OS X. I was able to find and download the Mono for Mac OS X 3.10._19 but it is broken and won't even run the WinForms example. Filed a bug and found you basically had to download a new MDK version just to get it to work at all. This is no where on the download page and they have not bothered to update their release build.

So then tried to download MonoMac and MonoDevelop but again, there are no direct links from the download page and when I found it in the Docs I had to instead download Xamarin Studio.app as the other links are gone. So their website seems to be horribly out of date.

Next I tried to run the Gtk# example but of course it fails. I looked and it said Gtk# requires the Mono Gtk# package but no download links point to anything of the sort. In one spot on the website it says it only runs on Unix and Win but on another spot it says if runs on Mac OS X as well but again no download links, so I have given up on Gtk#.

At least I can now run the Console and WinForms versions of "Hello World".

So now I need some of the other support programs you list. Unfortunately, none of the standard linux rpm/deb source packages are open-able under stock Mac OS X.

So can anyone point me to the proper source archives (tar.gz) and versions needed for any of these required items:

jpegtran - part of libjpeg-progs
optipng
gifsicle


I will build them and make them available for other Mac OS X users.

Thanks,

KevinH

OptiPNG: http://optipng.sourceforge.net/

For jpegran (older version, but I think same logic applies):
jpegtran is part of a package of few tools known as libjpeg

1) Get the source code from here. It's the file called jpegsrc.v6b.tar.gz. Using cURL you can download like:
curl http://www.ijg.org/files/jpegsrc.v6b.tar.gz > /tmp/libjpeg.tar.gz
2) Uncompres the package, e.g. tar -xzvf /tmp/libjpeg.tar.gz
3) go to the directory that contains the uncompressed code, e.g. cd /tmp/jpeg-6b
4) ./configure
5) sudo make install

Via http://www.lcdf.org/gifsicle/ the source for Gifsicle can be found and also an OSX port I believe.

Hi Toxaris,

Thanks for the links.

On Mac OS X 10.10 with the command line tools installed, jpeg6src.v6b and gifsicle build right out of the box with the standard ./configure; make sequence

Optipng does not build out of the box as it does not properly protect a call to utimensat that does not exist on Mac OS X but which does define the sys/fcntl.h AT_FDCWD.

So I have attached a patch for that.

Hopefully, when I get some time tonight, I will try your plugin. If it works, I would be happy to zip up those 3 command line programs and post them for you (or other Mac OS X users).

Thanks!

Kevin

Hi Toxaris,

Not much luck. Optimizing the jpegs failed. I notice your "windows" files are named jpegtran.exe whereas on a Mac they do not have the .exe file extensions. Should I rename them to match your names? I tried that and it does not help.

Here is what the output says when run on the Sigil_Plugin_Framework_rev4.epub:

And here is what it looks like if I run it directly from the command line. It seems to be missing the jpeg input files:


To test jpegtran I ran the following and it worked just fine:

./jpegtran -optimize ~/Desktop/Manage_Plugins.jpg > ~/Desktop/output.jpg

So jpegtran does seem to work.


I did notice one other windows executable called pngout.exe that does not seem to be part of Mono nor part of optipng. Is there a source archive for pngout as well I should have built?

Any ideas on how to debug further?

Thanks,

KevinH

If you need source packages, try checking http://archlinux.org/packages which will provide the Upstream Urls instead of repackaging the sources.

Hi eschwartz,

Thanks for the link. Unfortunately, I found out that pngout.exe is proprietary code that no source has been released for. Someone else has built a binary of it for older Macs running OS X 10.4, but I gave up running binaries I find on the web from unknown/untrusted sources long ago. I am surprised it runs on Linux.

Thanks,

Kevin

Oh, that seems to be priceless. :rolleyes:
I really love closed-source linux programs, too... I am sure it runs, it just won't be packaged by anybody.

Is pngout actually being used? It probably isn't necessary, how many different png optimizers does one program need?
But assuming we want to squeeze every possible byte out, a noteworthy goal, perhaps it is a better idea to use zopflipng which is part of google's open-source zopfli compression engine (alternative to deflate, it takes 2-3 times longer in order to find the smallest compression path). It even claims to be marginally better than pngout, by a whole 0.5%
https://code.google.com/p/zopfli/sou...ADME.zopflipng

You can ignore PNGOUT. It is an old leftover I apparently forgot to delete. The reason I kicked it out, is that there is no Linux/OSX version and it is closed source (no way to rebuild if the need arises).
The program is checking what the OS is to determine how to call the image optimizing programs. If I recall correctly (will look it up later today), on Windows systems it will use the executables in the directory and for Linux (and also OSX) it should use the ones in the path. Based on your error report, it seems that it does not starts fine, it tries to open 'open'. I will get back to you.

As it is now, I am detecting if it is running on Windows or other. If it is running on other, it will not give a pathname, but just execute the command. For jpegtran the command would be like:

jpegtran -optimize -progressive -copy none -outfile <filename> <filename>

Could you check if that is working on OSX?

Hi Toxaris,

Will check that and get back to you. One approch to making a cross-platform plugin that has binaries might be to create your own winbin\, osxbin/, linuxbin/ directories in the plugin folder and store the small required binaries in those and then invoke them with the correct full path depending on what sys you are running on. The platform can be easily detected by python and passed in if need be.

That way, no one has to build those binaries on OSX as there are no simple packages for Mac users.

I have tried setting the path env var to point to the binaries but I am not sure since you use subprocess to invoke it if those environment vars are properly picked up in the child process or not.

KevinH

You can actually optimize the images even better as long as you first figure out if they are grayscale or RGB and apply the appropriate profile.

For RGB
jpegtran.exe" -optimize -progressive -copy none -scans jpeg_scan_rgb.txt <filename> <filename>

For grayscale
jpegtran.exe" -optimize -progressive -copy none -scans jpeg_scan_bw.txt <filename> <filename>

For PNG, there is an open source optimizer called OptiPNG that you could use. http://optipng.sourceforge.net/
There is also another free PNG optimizer called PNG Optimizer. http://sourceforge.net/projects/pngoptimizer/

Hi,

I think your trying to write over the same file you read from is the issue on Mac OS X.

It successfully creates junk.jpg with no errors.

But if I try to overwrite the same file in-place ...

I get back the same "Empty input file" error I saw previously.
So at least on Mac OS X you can't overwrite in place.

Hope this helps,

KevinH

Tbis plugin already uses optipng.

You have a point there. The only catch would be that additional libraries would be needed, but I should be able to test that.