ePUB Optimizer

[KevinH]

Hi Toxaris,

Originally Posted by Toxaris

I would really like a .Net plugin interface, but I assume that would be quite some work to do.

That really depends on how much hand holding you want to do for the .net plugin developer.

At its minimum, a plugin interface gets sent the the plugin type, the path to the plugin, the internal Sigil directory where the ebook files are stores (unzipped), and a temp directory that is created and owned by Sigil.

"Something" then has to protect the internal Sigil files, and allow direct reading of internal Sigil ebook files but that does a file-copy-on-write/change to the temp directory, keeping track of what has been changed by the plugin (modified, deleted, added). And then build a result xml stream that is returned back to Sigil with the list of changed, modified, added, deleted files.

Sigil takes this result xml stream and parses it and updates its own internal Sigil ebook files based on the result xml file and the modified files in the temp directory.

For python plugins, I wanted this to be as simple as possible, so I wrote a wrapper.py and *container.py classes that implement this "Something" for every python plugin. The wrapper.py will automatically parses the opf, and keeps track of copying the files upon write/change, which files you want to delete, which files you want to add, and it automatically creates the result xml file for the plugin developer. So the python launcher.py and wrapper.py code do a lot of hand holding and interface building for the python plugin developer so that they don't have to worry about the interface and result xml and copy-on-modify/write approach used to protect the ebook files in Sigil.

So if you wanted you could create a quite bare bones .NET launcher without the equivalent of the wrapper.py and *container.py and simply allow the plugin developer to have direct access to files and just tell them they need to write any changes to the temp directory, and make tell them to create their own result.xml instead of automating that, etc.

Then if interest develops, you could add some of the more powerful .net xml processing tools and develop a full-blown launcher/wrapper for .net

Take care,

KevinH

[Jellby]

It is possible to define a font as "normal" in the @font-face, and the use it as bold and/or italic in other font-family rules, expecting the renderer to artificially embolden and/or slant the font (if no other appropriate bold/italic @font-face was defined).

Whether the renderer will actually do this, or just use the normal font, or use any other bold/italic font available, that's a different matter. And how easy it is to detect this situation is yet another different matter

[Toxaris]

Although I still feel it is wrong, I am nothing but flexible (although some might disagree there...).

I have adapted the program to accept an additional startup parameter so that usage is calculated per font-family only. Now I have to adapt the python script and it should work as some of you would like.

[DiapDealer]

Oh, you didn't fight nearly long enough before giving in to external pressure.

[JSWolf]

Originally Posted by Toxaris

Although I still feel it is wrong, I am nothing but flexible (although some might disagree there...).

I have adapted the program to accept an additional startup parameter so that usage is calculated per font-family only. Now I have to adapt the python script and it should work as some of you would like.

I would like it to work as it originally worked as you are correct and the others are incorrect.

[Toxaris]

Originally Posted by JSWolf

I would like it to work as it originally worked as you are correct and the others are incorrect.

It works both ways now (after I release it...). By default 'my way' and by setting an option the other way.

[Toxaris]

Version 0.3 with the option for other subset calculation is released.

[Doitsu]

Originally Posted by Toxaris

Version 0.3 with the option for other subset calculation is released.

Thanks for implementing my suggestion.

BTW, the new version worked fine with DiapDealer's and crutledge's books on my Windows machine, but on my Linux machine I'm getting the following error message (regardless of the book):

Status: success

['mono', '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/ePUBOptimizer.exe', '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/dummy.epub']

Unhandled Exception: System.TypeLoadException: A type load exception has occurred.
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeLoadException: A type load exception has occurred.

[Toxaris]

Originally Posted by Doitsu

Thanks for implementing my suggestion.

No problem, you are aware that you have not activated it yet? I can see that in the log you placed below...

Originally Posted by Doitsu

BTW, the new version worked fine with DiapDealer's and crutledge's books on my Windows machine, but on my Linux machine I'm getting the following error message (regardless of the book):

Status: success

['mono', '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/ePUBOptimizer.exe', '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/dummy.epub']

Unhandled Exception: System.TypeLoadException: A type load exception has occurred.
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeLoadException: A type load exception has occurred.

Hmm, strange. It works fine with me. My testing environment (32-bit):
- Linux Mint 13 Maya (basically Ubuntu 12.04)
- Mono 3.10
- Sigil 0.8.2 (although this doesn't matter at all)

I have no idea which Mono version is actually required. When I wanted to test the program, I just downloaded and installed Mono to be honest. I have also tested the program on Ubuntu 14.01 (64-bit) and Mono 3.10.
As the program is built against .Net Framework 4 (client profile even) and does not use 'exotic' programming, I would not expect the Mono version actually be a problem.
However, it may be that you are missing a Mono package. I found some online reports that say that monodevelop may be needed. I installed the full package myself... So, I think it is a missing mono library.

[Doitsu]

Originally Posted by Toxaris

No problem, you are aware that you have not activated it yet? I can see that in the log you placed below...

It is activated. I.e. the error must occur before the .ini file is being loaded.

Originally Posted by Toxaris

Hmm, strange. It works fine with me. My testing environment (32-bit):
- Linux Mint 13 Maya (basically Ubuntu 12.04)
- Mono 3.10
- Sigil 0.8.2 (although this doesn't matter at all)

I'm using Debian Wheezy, which came with Mono 2.10.8.1-8.

Originally Posted by Toxaris

However, it may be that you are missing a Mono package. I found some online reports that say that monodevelop may be needed.

I installed monodevelop, but it didn't make a difference. Most likely my mono version is too old, however, since a manual upgrade might lead to an unstable system, I'm going to use the Windows version of Sigil 0.8.2 instead.

Maybe you can mention in the first post, that Mono 3.x and/or a current Ubuntu-based distro might be required for the Linux version.

[Toxaris]

Originally Posted by Doitsu

It is activated. I.e. the error must occur before the .ini file is being loaded.

Nope, ini file is loaded from the start. I can see it because I haven't cleaned up printing the start command which I used for backup. It says:

['mono', '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/ePUBOptimizer.exe', '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/dummy.epub']

but it should say:

['mono', '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/ePUBOptimizer.exe', '-f', '/home/user/.local/share/sigil-ebook/sigil/plugins/ePUBOptimizer/dummy.epub']

Remember, the value of the parameter 'usefontfamily' must be set to 'True' to make it work like you requested.

Originally Posted by Doitsu

I'm using Debian Wheezy, which came with Mono 2.10.8.1-8.

I installed monodevelop, but it didn't make a difference. Most likely my mono version is too old, however, since a manual upgrade might lead to an unstable system, I'm going to use the Windows version of Sigil 0.8.2 instead.

Maybe you can mention in the first post, that Mono 3.x and/or a current Ubuntu-based distro might be required for the Linux version.

Will do.

[KevinH]

Hi,

I am trying to test things under Mac OS X. I was able to find and download the Mono for Mac OS X 3.10._19 but it is broken and won't even run the WinForms example. Filed a bug and found you basically had to download a new MDK version just to get it to work at all. This is no where on the download page and they have not bothered to update their release build.

So then tried to download MonoMac and MonoDevelop but again, there are no direct links from the download page and when I found it in the Docs I had to instead download Xamarin Studio.app as the other links are gone. So their website seems to be horribly out of date.

Next I tried to run the Gtk# example but of course it fails. I looked and it said Gtk# requires the Mono Gtk# package but no download links point to anything of the sort. In one spot on the website it says it only runs on Unix and Win but on another spot it says if runs on Mac OS X as well but again no download links, so I have given up on Gtk#.

At least I can now run the Console and WinForms versions of "Hello World".

So now I need some of the other support programs you list. Unfortunately, none of the standard linux rpm/deb source packages are open-able under stock Mac OS X.

So can anyone point me to the proper source archives (tar.gz) and versions needed for any of these required items:

jpegtran - part of libjpeg-progs
optipng
gifsicle


I will build them and make them available for other Mac OS X users.

Thanks,

KevinH

[Toxaris]

OptiPNG: http://optipng.sourceforge.net/

For jpegran (older version, but I think same logic applies):
jpegtran is part of a package of few tools known as libjpeg

1) Get the source code from here. It's the file called jpegsrc.v6b.tar.gz. Using cURL you can download like:
curl http://www.ijg.org/files/jpegsrc.v6b.tar.gz > /tmp/libjpeg.tar.gz
2) Uncompres the package, e.g. tar -xzvf /tmp/libjpeg.tar.gz
3) go to the directory that contains the uncompressed code, e.g. cd /tmp/jpeg-6b
4) ./configure
5) sudo make install

Via http://www.lcdf.org/gifsicle/ the source for Gifsicle can be found and also an OSX port I believe.

[KevinH]

Hi Toxaris,

Thanks for the links.

On Mac OS X 10.10 with the command line tools installed, jpeg6src.v6b and gifsicle build right out of the box with the standard ./configure; make sequence

Optipng does not build out of the box as it does not properly protect a call to utimensat that does not exist on Mac OS X but which does define the sys/fcntl.h AT_FDCWD.

So I have attached a patch for that.

Hopefully, when I get some time tonight, I will try your plugin. If it works, I would be happy to zip up those 3 command line programs and post them for you (or other Mac OS X users).

Thanks!

Kevin

Originally Posted by Toxaris

OptiPNG: http://optipng.sourceforge.net/

For jpegran (older version, but I think same logic applies):
jpegtran is part of a package of few tools known as libjpeg

1) Get the source code from here. It's the file called jpegsrc.v6b.tar.gz. Using cURL you can download like:
curl http://www.ijg.org/files/jpegsrc.v6b.tar.gz > /tmp/libjpeg.tar.gz
2) Uncompres the package, e.g. tar -xzvf /tmp/libjpeg.tar.gz
3) go to the directory that contains the uncompressed code, e.g. cd /tmp/jpeg-6b
4) ./configure
5) sudo make install

Via http://www.lcdf.org/gifsicle/ the source for Gifsicle can be found and also an OSX port I believe.

[KevinH]

Hi Toxaris,

Not much luck. Optimizing the jpegs failed. I notice your "windows" files are named jpegtran.exe whereas on a Mac they do not have the .exe file extensions. Should I rename them to match your names? I tried that and it does not help.

Here is what the output says when run on the Sigil_Plugin_Framework_rev4.epub:

Status: success

['mono', '/Users/kbhend/Library/Application Support/sigil-ebook/sigil/plugins/ePUBOptimizer/ePUBOptimizer.exe', '/Users/kbhend/Library/Application Support/sigil-ebook/sigil/plugins/ePUBOptimizer/dummy.epub']
Processing /Users/kbhend/Library/Application Support/sigil-ebook/sigil/plugins/ePUBOptimizer/dummy.epub
Get unique characters per font
Extract ePUB
Subset fonts
Optimizing OEBPS/Images/Manage_Plugins.jpg
Optimizing failed.
Optimizing OEBPS/Images/PluginRunner.jpg
Optimizing failed.
Optimizing OEBPS/Images/Sigil_Plugins_Menu.jpg
Optimizing failed.
Rebuild ePUB
Cleanup
open: invalid option -- o
Usage: open [-e] [-t] [-f] [-W] [-R] [-n] [-g] [-h] [-b <bundle identifier>] [-a <application>] [filenames] [--args arguments]
Help: Open opens files from a shell.
      By default, opens each file using the default application for that file.  
      If the file is in the form of a URL, the file will be opened as a URL.
Options: 
      -a                Opens with the specified application.
      -b                Opens with the specified application bundle identifier.
      -e                Opens with TextEdit.
      -t                Opens with default text editor.
      -f                Reads input from standard input and opens with TextEdit.
      -F  --fresh       Launches the app fresh, that is, without restoring windows. Saved persistent state is lost, excluding Untitled documents.
      -R, --reveal      Selects in the Finder instead of opening.
      -W, --wait-apps   Blocks until the used applications are closed (even if they were already running).
          --args        All remaining arguments are passed in argv to the application's main() function instead of opened.
      -n, --new         Open a new instance of the application even if one is already running.
      -j, --hide        Launches the app hidden.
      -g, --background  Does not bring the application to the foreground.
      -h, --header      Searches header file locations for headers matching the given filenames, and opens them.
open: invalid option -- o
Usage: open [-e] [-t] [-f] [-W] [-R] [-n] [-g] [-h] [-b <bundle identifier>] [-a <application>] [filenames] [--args arguments]
Help: Open opens files from a shell.
      By default, opens each file using the default application for that file.  
      If the file is in the form of a URL, the file will be opened as a URL.
Options: 
      -a                Opens with the specified application.
      -b                Opens with the specified application bundle identifier.
      -e                Opens with TextEdit.
      -t                Opens with default text editor.
      -f                Reads input from standard input and opens with TextEdit.
      -F  --fresh       Launches the app fresh, that is, without restoring windows. Saved persistent state is lost, excluding Untitled documents.
      -R, --reveal      Selects in the Finder instead of opening.
      -W, --wait-apps   Blocks until the used applications are closed (even if they were already running).
          --args        All remaining arguments are passed in argv to the application's main() function instead of opened.
      -n, --new         Open a new instance of the application even if one is already running.
      -j, --hide        Launches the app hidden.
      -g, --background  Does not bring the application to the foreground.
      -h, --header      Searches header file locations for headers matching the given filenames, and opens them.
open: invalid option -- o
Usage: open [-e] [-t] [-f] [-W] [-R] [-n] [-g] [-h] [-b <bundle identifier>] [-a <application>] [filenames] [--args arguments]
Help: Open opens files from a shell.
      By default, opens each file using the default application for that file.  
      If the file is in the form of a URL, the file will be opened as a URL.
Options: 
      -a                Opens with the specified application.
      -b                Opens with the specified application bundle identifier.
      -e                Opens with TextEdit.
      -t                Opens with default text editor.
      -f                Reads input from standard input and opens with TextEdit.
      -F  --fresh       Launches the app fresh, that is, without restoring windows. Saved persistent state is lost, excluding Untitled documents.
      -R, --reveal      Selects in the Finder instead of opening.
      -W, --wait-apps   Blocks until the used applications are closed (even if they were already running).
          --args        All remaining arguments are passed in argv to the application's main() function instead of opened.
      -n, --new         Open a new instance of the application even if one is already running.
      -j, --hide        Launches the app hidden.
      -g, --background  Does not bring the application to the foreground.
      -h, --header      Searches header file locations for headers matching the given filenames, and opens them.
Optimization process complete

And here is what it looks like if I run it directly from the command line. It seems to be missing the jpeg input files:

KevinsiMac:ePUBOptimizer kbhend$ mono /Users/kbhend/Desktop/ePUBOptimizer/ePUBOptimizer.exe ./dummy.epub
Processing ./dummy.epub
Get unique characters per font
Extract ePUB
Subset fonts
Optimizing OEBPS/Images/Manage_Plugins.jpg
Empty input file
Optimizing OEBPS/Images/PluginRunner.jpg
Empty input file
Optimizing OEBPS/Images/Sigil_Plugins_Menu.jpg
Empty input file
Rebuild ePUB
Cleanup
KevinsiMac:ePUBOptimizer kbhend$

To test jpegtran I ran the following and it worked just fine:

./jpegtran -optimize ~/Desktop/Manage_Plugins.jpg > ~/Desktop/output.jpg

So jpegtran does seem to work.


I did notice one other windows executable called pngout.exe that does not seem to be part of Mono nor part of optipng. Is there a source archive for pngout as well I should have built?

Any ideas on how to debug further?

Thanks,

KevinH