Sigil 0.8.0 Released

[KevinH]

Hi,

Originally Posted by mrmikel

Would the path be calibre2/dlls in Windows?

I don't think you can actually use calibre's embedded python to be a general python interpreter release. Calibre's python strips out any libraries it doesn't need including the Tk gui libraries you would need to run DiapDealer's KindleImport and SmartenPunctuation plugins for Sigil.

In addition, I could not find any executable actually named python anyplace in the calibre.app distribution (at least on a Mac). I did find pre-compiled .pyo files and shared libraries (dlls on Windows) but no actual entry point called python.

There may be a way to trick it to work, but I don't know it. And even if you did, you would not be able to run any gui plugins. You would simply be better off installing a real Python interpreter. On Windows, I recommend the ActiveState ActivePython 2.7 community edition. It is free, comes with a nice installer, and can easily be uninstalled if you decide you do not want it.

Hope this helps,

KevinH

[eschwartz]

Originally Posted by mrmikel

Would the path be calibre2/dlls in Windows?

You can use calibre-debug.exe -- it spawns a python shell from the version of python that calibre uses and is intended for the use case of importing calibre.

But as KevinH said, it will probably be missing components that calibre does not need but the plugin does.

[kovidgoyal]

You can run any python script with calibre like this:

calibre-debug path/to/script.py

As for GUI, since DiapDealer has calibre versions of his plugins, presumably, it should not be hard to adapt to using PyQt instead of Tk.

On a personal note, I recommend you guys bundle python, at least on windows. It will make life a lot easier for your users.

And even on linux and os x there are no guarantees of the python environment available. What python libraries are present and what are not. For instance, on my Gentoo system I dont typically have the python Tk bindings installed. Not to mention the whole problem of python 2 vs 3. Different linux distros may or may not have either python 2 or 3. And since Sigil already distributes the Qt DLLs, it should be relatively easy to also distribute the PyQt bindings.

Then you can provide a consistent, platform independent python environment for your plugin developers and ease of use for your users.

While the goal of having the plugins work with any scripting language is laudable, that doesn't mean you cannot have a single, favored language.

Anyway, that is my opinion.

[KevinH]

Hi Kovid,
According to user-none we may end up packaging Python at some point. But doing things this way we can support both Python 2 and Python 3 as the same launcher code will now run on both. If we package in Python, we would have to choose one as adding in both would make the download huge.

Given Python 2.7 is in maintenance mode, we would probably go with Python 3, and I would then need to convert KindleUnpack to work with both versions of Python just as the launcher code does now.

But as you said, we could then control exactly what packages are included.

I am not too worried about Linux users figuring out which Python packages to add. And even Mac OS X packages all the way back to 10.5 have pretty full versions of Python with Tk. So it is really the Windows users that seem to need the help and seem to be the most hesitant to download and install Python. ActiveStates Active Python community edition for Windows is quite full featured, is totally free, is well supported., comes with Tk, and has its own easy to use installer. I am simply astounded by how many people seem to fear installing it.

I will talk it over with user-none to see what he thinks. If we do decide to package Python 3, we could add lxml and pyqt and even use Python internally to add new Sigil features.

Thanks,

KevinH

[kovidgoyal]

I strongly suggest you stick with python 2.x, if you are choosing a single python version. The python developers would really, really like python 2.x to die, but unfortunately for them, they did not introduce any sufficiently important new features to python 3 to make it worth the effort of migrating a large existing codebase. So the vast majority of python code will remain python 2, certainly for several years, probably forever. Furthermore, it is pretty easy to write python 2 code in a way that it can be easily ported to python 3 in the future.
Simply stick the following at the start of all the .py files:

from __future__ import (unicode_literals, division, absolute_import, print_function)

Then, in the future migrating to python 3 is mostly just a matter of changing a few names for standard library modules/functions, which the 2to3 tool does automatically.

And note that python 2.8 will be released soon.

[crutledge]

Although I spent many years at Software Engineering, I am now asking for something I know nothing about. I have scanned through the Calibre files and dll's and can't make heads or tails of them.

So as I did when needed, I call in the experts.

Would it be possible to translate the Convertbooks into a plugin? This would give a one stop tool. Convertbooks does a fine job of translation. No need to reinvent the wheel.

[DiapDealer]

Originally Posted by crutledge

Would it be possible to translate the Convertbooks into a plugin? This would give a one stop tool. Convertbooks does a fine job of translation. No need to reinvent the wheel.

You mentioned something about this before. I must admit I'm a little confused by some of your terminology ("Convertbooks" and "translation"), though. If you're talking about calibre's "normal" ebook conversion process, then yes... it should be quite possible to create an output plugin that would take Sigil's current content and farm it out to calibre's command-line conversion tools. If you're talking about something else, then I'm afraid I don't quite follow.

[KevinH]

Hi Kovid,
Unfortunately, Python 2 has broken windows sys.argv Unicode commandline parsing, multiprocessing is also severely broken because they used a naked fork without an exec on Linux, and MacOS X so that they would not have to serialized objects/state and pass it to the newly started process, etc. All of these problems have been around for years but never fixed in Python 2. Hell the naked fork problem was only fixed in Python 3.4! Tk askdirectory does not return Unicode on Windows, etc, etc. We have found ways to work around most of these but broken multiprocessing is a hard one to fix as it severely limits the new process to only pure ansi functions to prevent crashes. Yes we could move to threads but the big lock kind of defeats the whole purpose.

If I thought some group would truly fork Python 2.7/2.8 and actually start fixing things properly and backporting these critical fixes then I would stay with Python 2. But somehow all of these fixes get called "new features" by the Python developers and never get added to the 2.X stream.

So right now, we use your from future import line and then add our little compatibility lib and try to write code that runs as is on both Python 2 and Python 3. That is what the launcher does and what a future version of KindleUnpack will do once I get some free time.

Take care,
KevinH

[mrmikel]

Is the idea to go from the edited epub to any output format calibre can do?

[mrmikel]

Though your post is over my head, Kevin, it does sound very much like epub3 vs epub2. Lots of whizbang without fixing actual and important issues in epub2. Hmmm are the same people involved in both????

[kovidgoyal]

@KevinH: Fixing sys.argv is trivial, if you want I can send you some code for it. Or just use this years old recipe: http://code.activestate.com/recipes/572200/

As for multiprocessing, just dont use it. There is no need to to use it. All of calibre, which makes extensive use of worker processes never uses it. You can simply use the Listender and Client classes from the multiprocessing module to implement IPC between processes you launch with subprocess.Popen

[KevinH]

Hi Kovid,

Yes, I know we can fix all of these things and work around them. I have code to do all of this and more (although using subprocess.Popen and Listener and client classes was a new one for me). It is just reaching the point of silliness. I need to carry around a library of python 2 fixes that simply should not be needed. Especially after 5 to 10 years or more! These fixes should have been incorporated into the official python 2.X years and years ago but never made it and never will.

Also, I have looked all over the python developer's mailing list archive and can not find anything that states python 2.8 will happen. In fact, all I can see if Guido clearly stating it will never happen (see the May 2014 Python conference) but he did add that support for python 2.X will continue until 2020. So where can I find a link that talks about Python 2.8 actually happening and what if any changes it includes?

Unlike 5 years ago, most of the main libraries and even scientific libraries (like numpy) now support Python 3. And python 3 seems to be the only platform getting any of these old problems fixed.

So unless I see some large group step forward to incorporate all of these fixes into a forked python 2.X series, I personally am strongly leaning towards using Python 3.4 and later when (and if) we are forced to choose one. But of course, the final call belongs to user-none on that. It is his project.

Right now, using external interpreters luckily means we do not have to choose and can support both. But then we get hit by Windows users who are reluctant to install either version!

Originally Posted by kovidgoyal

@KevinH: Fixing sys.argv is trivial, if you want I can send you some code for it. Or just use this years old recipe: http://code.activestate.com/recipes/572200/

As for multiprocessing, just dont use it. There is no need to to use it. All of calibre, which makes extensive use of worker processes never uses it. You can simply use the Listender and Client classes from the multiprocessing module to implement IPC between processes you launch with subprocess.Popen

Take care,

KevinH

[mrmikel]

I bit the bullet and installed Active python. I will have to figure out the plug in stuff on Sigil now, but there isn't anything too big for me yet. Tho there is a lot of promise in doing things you just can't do through regex.

[KevinH]

Hi mrmikel,

Originally Posted by mrmikel

I bit the bullet and installed Active python. I will have to figure out the plug in stuff on Sigil now, but there isn't anything too big for me yet. Tho there is a lot of promise in doing things you just can't do through regex.

Glad to hear it. Yes python 2.7 (and Python 3.4) both have great support for regular expressions and can of course do many many things that are impossible to do with regular expressions alone.

If you are interested in developing your own plugins, grab the documentation on plugins from the download site for 0.8.0 and also grab the testme_v0.1.0beta.zip (for python 2) and unzip it and look at the plugin.py code to see examples of using some of the features described in the documentation. If you have any questions, just ask. Happy to help.

Of course, if you just want to use plugins, no need for any of that. Just grab DiapDealer's latest plugin versions and start playing around. We have lots more ideas for plugins and they will come as soon as I get some free time.

Take care,

KevinH

[crutledge]

Originally Posted by DiapDealer

You mentioned something about this before. I must admit I'm a little confused by some of your terminology ("Convertbooks" and "translation"), though. If you're talking about calibre's "normal" ebook conversion process, then yes... it should be quite possible to create an output plugin that would take Sigil's current content and farm it out to calibre's command-line conversion tools. If you're talking about something else, then I'm afraid I don't quite follow.

The calibre icon says Convertbooks and can build lrh, mobi, etc from the loaded epub.