Writing a first plugin

[Agama]

I want to write my own plugin to automate some ePub post-conversion processing. I have read the calibre manual on writing plugins but there is still quite a learning curve to go through, so I would like to "cheat".

My plugin will work along the following lines:

1) Select an ePub book in the Library (GUI).
2) Explode the selected ePub.
3) Do some stuff to the xhtml files.
4) Rebuild the ePub.

Can any recommend a simple plugin that I can use as a basis for this type of processing? I can then reverse engineer it to, hopefully, get a better understanding of how plugins work.

I did start looking at kiwidude's Modify ePub plugin, (since it's one I already use), but it's quite a substantial plugin and does not appear to be simple!

[kiwidude]

The challenge with using Modify ePub as a starting point is it is designed to (a) work with multiple ePubs at once, and (b) to run as a background job in calibre. The latter in particular is a requirement if you need (a) or else your calibre will freeze up while the plugin runs. In your case if you are just doing an ad hoc single epub at a time you could get away without all that complexity to do with threading, queuing books and passing data around between processes which Modify ePub has to do.

However you should be able to still steal parts of the code. The modify.py class contains the BookModifier class, which if you look at the process_book function has responsibility for extracting the epub into a temporary directory, giving you an "ExtendedContainer" object (from container.py) to work with which lets you manipulate the opf manifest and get access to the html files, writing any modified files back to the zip when done. It also has the various tasks that Modify ePub implements so plenty of examples of how to manipulate the ePub using my container object. You could copy/strip those two python files, strip down the action.py to just directly call your version of the BookModifier class from the modify_epub function (rather than launching dialogs to get options like it does now), and then use some code to put the modified ePub back into your library (self.db.add_format).

Or you could start with something really basic which is where I originally started for Modify ePub - looking at the ePubFixer plugin code in the calibre source, in calibre/ebooks/epub/fix/ which has a very basic container object etc. That plugin does a very simple extraction of an epub for some modifications. I found a number of bugs/limitations with that particular container object in container.py which is why I ended up writing my own version of it for Modify ePub, but it might be ok for your needs or to start with. If you bolted that onto one of Kovid's plugin examples in the help tutorial that might be a simpler starting point.

[Agama]

Thanks for the pointers, maybe this is not so daunting as it first appeared. The BookModifier class does look a good starting point. I had wondered why all the job queuing was necessary, but I was really thinking of my own workflow which would take a single book at a time and run pretty quickly - therefore of no concern that it would lock up the GUI. I can see that you need more sophistication when potentially processing many ePubs with many options selected.

I should be able to replace the options dialogue with a config widget to store preferences for the plugin.

If I want to use regex processing do I need to import a class from somewhere or is this built-in?

[kiwidude]

For an example of a regex replacement, just look at the _remove_drm_meta_tags() or _remove_embedded_fonts() functions in BookModifier. When you work with my container there are two key options when you ask the container for a file:

container.get_raw(name) - gives you the file data as a raw text string. So if you are intending to use regex to parse that text, this is the one you want.

container.get_parsed_etree(name) - gives you the data as an etree object to allow you to perform xpath based lookups and replacements on xml/xhtml files.

If you make changes to that data, then call container.set(name, obj) which will update all its internal caches etc. So if you were to ask for that same object again using either of the functions above you will be working with the latest modified version. Then at the very end of the process in container.write() all the updated cached data is physically written into the files before rezipping.

[Agama]

container.get_raw sounds ideal. There are quite a few regexes in modify.py and the ones you mention use re.compile whereas others use re.sub. What's the difference?

[kiwidude]

re.compile() just precompiles the regular expression, it is more efficient if you intend to use it in a loop. I suggest you take a look at the python docs for re.

[Agama]

The Python documentation is very good. Is there anything similar available for calibre which documents all the classes, properties, methods etc.?

[kiwidude]

The Python docs are pretty good, though a few more examples sprinkled around would help, and the giant long pages covering a number of classes annoy me as if you google for something you then have to type a local browser find again on the page to find it.

As for calibre's API, Kovid used to build such help and include it in the online calibre manual. It doesn't seem to appear there any more - to be honest that is something I am happy about because I didn't believe an end user searching for a key word should be given search results showing low level coding API stuff. No problem with it being available online, just didn't think it belonged in the same search scope as the user manual. Though now (or at least here at work on IE8) there seems to be no search capability whatsoever in the calibre manual for anything - leaving it up to google I guess?

I think you are better off just loading the calibre source code up in the IDE you are developing your plugin in such as Eclipse, and using its search/navigation capabilities. I find it orders of magnitude more useful to see the actual underlying code in a class/function - particularly in case the documentation is out of date, missing or terse. With Eclipse/PyDev you get the preview in the tooltip and hyperlink navigation.

The biggest challenge I have with calibre's source code is that having a long history from multiple developers, there is a lot of legacy code hanging around and in many cases there are lots of namespaces and classes that are very similar to each other but for different purposes. Two years on and I still don't find the namespace hierarchy intuitive enough to be confident to be able to browse to a class file beyond the few I am forever referencing (like database2.py). Instead I tend to start from the calibre plugin code for a feature and work my way back through the code links to get to where the "real" code lives. That doesn't always work though due to the limitations of IDEs with Pythons dynamic and multi-inheritance nature with lack of interface contracts - once you hit something like self.gui.xxx or self.db.xxx then it becomes time to dig out the workspace search for xxx again to figure out where that code lives... it is a labour of love at times!

The stuff like the plugins API has excellent code documentation, plus you have dozens of examples in the forums to cater for "how would I do that?" along with the built in plugins in calibre's code. Beyond that it gets deep in the pool very quickly

[kovidgoyal]

The API documentation and search are still both in the user manual, the search is provided by google, however, which makes it prioritize end user stuff in the results automatically.

For example: http://manual.calibre-ebook.com/news...ml#news-recipe
or
http://manual.calibre-ebook.com/plugins.html#plugins

But API documentation in calibre is only available for the parts that have well defined external APIs like the plugin system or the news download system. For the rest, you have to go spelunking in the code.

[kiwidude]

Ahhh, thx Kovid, I didn't see it in the index and my quick glance down a few pages didn't see obvious links either. Did there not use to be some top level contents link to the API docs, or am I imagining that?

I've since found a link about 80% of the way down this page to the link you gave above:
http://manual.calibre-ebook.com/creating_plugins.html

Perhaps something more explicit could be put in the contents of that page? Rather than just a link within "The different types of plugins" section?

[kovidgoyal]

I renamed the section to "The plugin API"

[Agama]

I'm making some progress, (and learning more about plugins), by taking apart the built-in Tweak ePub plugin. This is making sense but has rasied another question:

There is a line of code:

from calibre.gui2.dialogs.tweak_epub_ui import Ui_Dialog

but there is no such file as tweak_epub_ui.py. However there is a tweak_epub.ui. Is this what actually gets used? The contents of this .ui file appear to be some sort of dialogue definition in xml.

[kovidgoyal]

Yes, the .ui files automatically generate corresponding _ui.py files.

[chaley]

Hmmm....

In my local copy of trunk there is no file tweak_epub.ui. I have gui2/actions.tweak_epub.py and gui2/dialogs/tweak_epub_ui.py. I then rebuilt the source tree from the repository, and the file gui2/dialogs/tweak_epub_ui.py went away. And in any event, I don't see that import in the plugin source (in gui2/actions.tweak_epub.py).

I confess to confusion. @Agama: what version of the source are you using?

[Agama]

Due to my lack of knowledge I saw:

calibre.gui2.dialogs.tweak_epub_ui import Ui_Dialog

and I wrongly assumed that this refered to gui2/dialogs/tweak_epub_ui.py, which I couldn't find!
Kovid confirmed that this actually refers to gui2/dialogs/tweak_epub.ui which I do have.
(I don't know exactly how these .ui files work but I guess there is a help document somewhere.)

I downloaded calibre source some months ago so I'm not sure which version it is. How do I find out and should I update it? I am not running from source but simply using it as a means of learning calibre/Python programming.