Developing plugin to process notes from Sony PRS-T1: no module named

[sjvs]

Hi all,

I'm developing a Calibe plugin to import/export notes from a Sony PRS-T1 device. My first goal is to be able to "merge" handwritten annotations with the original pdf, using the "cairo" and "rsvg" libraries.

The user interface and SQLite bits are in place, but I can't seem to do "import cairo" or "import rsvg" in my plugin, because Calibre will tell me:

Traceback (most recent call last):
File "site-packages/calibre/gui2/ui.py", line 123, in __init__
File "site-packages/calibre/gui2/ui.py", line 137, in init_iaction
File "site-packages/calibre/customize/__init__.py", line 543, in load_actual_plugin
File "importlib/__init__.py", line 37, in import_module
File "site-packages/calibre/customize/zipplugin.py", line 150, in load_module
File "calibre_plugins.prst1_notes.ui", line 18, in <module>
File "site-packages/calibre/customize/zipplugin.py", line 150, in load_module
File "calibre_plugins.prst1_notes.main", line 27, in <module>
File "site-packages/calibre/customize/zipplugin.py", line 150, in load_module
File "calibre_plugins.prst1_notes.PRST1NotesPDFMergeInt erface", line 6, in <module>
ImportError: No module named cairo

However, the cairo and rsvg python libraries are in place on my system (Ubuntu 11.10), and working for other (standalone) Python applications.

I tried calibre-debug, same result:

$ calibre-debug -c "import cairo"
Traceback (most recent call last):
File "site.py", line 58, in main
File "site-packages/calibre/debug.py", line 216, in main
File "<string>", line 1, in <module>
ImportError: No module named cairo

Am I overlooking something? I'm running Python 2.7.2.

Many thanks for your help!

[kovidgoyal]

calibre has its own private copy of python. You will need to include those packages in the zip file, or add the path to them explicitly to sys.path in your calibre plugin before trying to import. Note that if the packages contain sompiled code, then they cannot be imported from zip files, in which case you will have to add the path to them to sys.path.

[kovidgoyal]

Also, rather than using rsvg to render svg you can simply use QSvgRenderer from PyQt.

[sjvs]

I'm not sure whether distributing cairo and all of its dependencies with the plugin is a good idea...

On the other hand, including hard-coded paths in the plugin will limit portability to other distributions and platforms. Is there are more widely accepted and standardised way of doing this?

I'm new to Calibre plugin development and wouldn't want to invent the wheel twice!

[kovidgoyal]

It most definitely isn't a good idea WHat exactly are you trying to use cairo for? You might me able to do whatever it is with pypdf, poppler or podofo, all of which are distributed with calibre. And use Qt to render svg instead of rsvg, which is also distributed with calibre.

[sjvs]

Excellent tips, I'll look into using those. Many thanks!

[sjvs]

Hmm... Seems like Poppler is not exactly what I'd call well-documented. I'm trying to get it to render on a QPainter instance, no luck so far... I could rasterise the SVG first, but that would be a waste of time, space and quality.

A hint anyone?

[kovidgoyal]

IIRC okular uses the combination of poppler and Qt to render PDFs. Try a quick look through its source code.

[sjvs]

The plugin (using QSvgRenderer and PyPdf) works on my machine, I'll be testing it on other platforms over the next couple of days. If all goes well, I'll publish the plugin early next week.

Currently, it's only capable of processing notes from (unscaled!) PDF books. I'm not quite sure what to do with EPUBs. First of all, I'd have to find out how EPUB scaling works on the reader. After that, I'd have to hook the plugin up to Calibre's EPUB conversion code in order to produce an output PDF.

Before I start digging through the Calibre source code looking for hints - which approach would you prefer:
1) convert both ebook and handwriting to PDF, merge into single PDF.
2) directly render handwriting on EPUB. I have no clue whatsoever how EPUB is structured, any clues?

Thanks!

[kovidgoyal]

EPUB is just HTML. you can insert the SVG into the HTML directly. No need to render, and especially, no need to convert to PDF.

[sjvs]

For some reason I was expecting something much more complicated than HTML. However, wouldn't the rendering of the EPUB heavily rely on the device (browser)? Especially when merging with an SVG of handwriting, it's important to have the text at a consistent position.

I'll look into this, but if you have any hints, tips or trips - feel free to let me know.

[kovidgoyal]

Pretty much all epub viewers (barring very few in marginal devices) are based on (somewhat modified) webkit engines, so there shouldn't be any problem with rendering them. Also SVG support is IIRC mandated by the EPUB spec. As for positioning them, that is a bigger issue. You can always just insert the svg has an inline <svg> tag at the start of the closest element to the annotation position. Anything more sophisticated will probably require support from the viewing application.

[sjvs]

I like annotating books on my PRS-T1 with handwriting: underlining sentences, circling words, scribbling things in the margin. Based on your post, I think it will be virtually impossible to (a) reproduce the exact same rendering (zoom factor!) outside the ereader and (b) render handwriting at the correct location.

I'll give it a shot, but no high hopes...

[sjvs]

Oh, and I forgot to ask: it seems that the Poppler Python libraries are not shipped with Calibre, is this right? The source code only seems to contains references to Poppler in C++ parts (reflow.cpp and reflow.h), but I can't do "import poppler" in Python.

PyPDF is capable of processing (reading/writing) PDF files, but as far as I can see doesn't support rendering in a GUI. Although it's not essential, it would be nice to show the user a preview of what he/she is about to get.

Any hints on this? Or would you mind shipping the Poppler Python libraries with Calibre in future releases?

[kovidgoyal]

poppler is shipped with calibre, though not its GUI bindings. If you want to show the suer a PDF, simply save it as a file on the filesystem and use local_open(). Most computers have PDF reading software on them.