DjVu plugin for DR1000 (testers wanted)

[owl123]

Originally Posted by owl123

Does removing the mobipocket plugin mean there's no dictionary support anymore for PDFs? (DR1000S supports only mobipocket dictionaries.)

Just checked it. The dictionary lookup function is greyed out and inaccessible.

I'll test the plugin in the coming hours.

[Mackx]

Originally Posted by badbob001

Hopefully when irex brings out the update to the dr1000, they will have an on-device method to merge scribbles to pdf and that method can be used for other document types.

Merging scribbles with a format is very format specific, so if it works for one format, it can not easily be copied for another format. Pdf allows to add an extra 'segment' that contains extra data for a page, this can be used to 'merge' the annotations with the pdf (like done by xournal) as a sequence of line segments. For image one would need to create an extra image with the annotations and merge both. For every format a specific method needs to be created to view the annotations stand-alone.
Maybe iRex can also publish the code for the PC companion application, and add a framework for merging annotations so new mergers could easily be added by the user community.

[luite]

Originally Posted by Mackx

Merging scribbles with a format is very format specific, so if it works for one format, it can not easily be copied for another format. Pdf allows to add an extra 'segment' that contains extra data for a page, this can be used to 'merge' the annotations with the pdf (like done by xournal) as a sequence of line segments.

DjVu also supports annotations in text format, but those are mostly meant for creating hyperlinks ( see http://djvu.sourceforge.net/doc/man/djvused.html under 'Annotation Syntax' ). The type (line x0 y0 x1 y1) would probably be most useful, it has a width and a color. Maybe these hyperlinks can be abused for merging scribbles.

I don't do anything with those annotations at the moment, so unless djvulibre automatically renders the borders, this kind of annotations will not show up in the viewer. Does anyone know if any of the common djvu viewers for the pc show these annotations?

[tjdean256]

Mackx's port of Xournal to the DR1000 allows merging scribbles to PDF now.

td

[luite]

I have tested a few viewers and found out that both WinDjView 1.0.3 and DjView 4.6 support line annotations. The UDS plugin does not show them (yet?).

If you want to experiment with annotations, try this:
djvused myfile.djvu -e 'output-all' > annotations.dsed

open annotations.dsed, if you have a hidden text layer, you will see things like: select 'pagename.djvu' ... set-txt ... . Choose your favourite page and add the following lines below the line "select 'pagename.djvu'":
set-ant
(maparea "" "" (line 400 400 800 800))
.

now run:
djvused myfile.djvu -f annotations.dsed -s

and view your file in DjView 4.6

[owl123]

Originally Posted by luite

Hmm, that is a problem. I'll look into this, possibly the next update.


It is a problem on both the actual device and the emulator. Opening a djvu file results in an error message: "'filename.djvu' cannot be opened"

Looking at the debug output of the emulator, these are the relevant error messages:
(ERROR)uds_plugin_manager.cpp:150,get_plugin_path( ) No matched plugin for .djvu
com.mobipocket.common.library.reader.InvalidBookEx ception: Corrupted
at com.mobipocket.common.library.reader.BookItem.<ini t>(libuds-plugin-mobipocket.so)
(ERROR)../plugin_impl/reader_impl.cpp:2302,get_error_code() This book is corrupted.
(ERROR)uds_doc_manager.cpp:149,open_document() Error opening document for /media/mmcblk0p1/test.djvu

After the first error, the plugin is queried (through IPluginLibrary) to check whether the document is supported. Even though the plugin tells UDS that the file is supported if its extension is .djvu or .djv, it still queries the mobipocket plugin afterwards.

I'm not sure what goes wrong, as both UDS and the mobipocket plugin are closed source. Perhaps I didn't implement some interface correctly.

I'm not into debugging or programming at all but maybe the mobipocket plugin is queried because both ebook formats supported natively by UDS (.mobi and .pdf) work with mobipocket dictionaries.

Maybe the mobipocket plugin returns "The book is corrupted error" because the dictionary function can't recognize the djvu format.

[luite]

Originally Posted by owl123

I'm not into debugging or programming at all but maybe the mobipocket plugin is queried because both ebook formats supported natively by UDS (.mobi and .pdf) work with mobipocket dictionaries.

Maybe the mobipocket plugin returns "The book is corrupted error" because the dictionary function can't recognize the djvu format.

It seems to be extension dependent. If I remove libuds-plugin-pdf.so (the default PDF plugin) and rename my plugin to libuds-plugin-pdf.so and then open a renamed DjVu file (with .pdf extension), it will load just fine. Even when the mobipocket plugin is installed! It does not even ask the plugin whether the file format is supported.

I did some digging around in the uds executable (source code not available, unfortunately), and it seems to have a hardcoded map of extensions and plugins ( uds::PluginMgr::initialize_ext_table ).

The method uds::PluginMgr::create_document first calls uds::PluginMgr::get_plugin_path to check this map for one of the hardcoded extensions. Unknown extensions are handled by loading all plugins and checking them to see if the format is supported.

There may be something that goes wrong there.

[owl123]

Is there a plugin for HTML or TXT that you could rename? Both of these formats aren't really usable with the native reader.

[Gertjan]

Originally Posted by luite

The method uds::PluginMgr::create_document first calls uds::PluginMgr::get_plugin_path to check this map for one of the hardcoded extensions. Unknown extensions are handled by loading all plugins and checking them to see if the format is supported.

There may be something that goes wrong there.

That would be my guess too. I haven't checked, but I think the mobipocket plugin for some reason reports it can open the djvu file. When it actually opens the file, the plugin finds that it is not a valid Mobipocket.

You might be able to work around it by renaming the Mobipocket plugin so it is probed _after_ the DJVU plugin. In that case, opening Mobipocket files will take a bit longer as UDS has to check all plugins first.

[thomega]

Originally Posted by luite

If you can render the scribbles for a single page to an image with the correct resolution, it should be fairly easy to do using imagemagick and djvulibre. [...] render scribbles for the page to an image file (own tool, does one already exist?)

Since the scribbles are most likely stored in the same formalt as for PDF files, it should be really easy to use my PDF scribble merger for rendering them on a blank page. The rest is a matter of GhostScript, ImageMagick and djvulibre. This could be slow, but should be straightforward.

[luite]

Originally Posted by Gertjan

That would be my guess too. I haven't checked, but I think the mobipocket plugin for some reason reports it can open the djvu file. When it actually opens the file, the plugin finds that it is not a valid Mobipocket.

You might be able to work around it by renaming the Mobipocket plugin so it is probed _after_ the DJVU plugin. In that case, opening Mobipocket files will take a bit longer as UDS has to check all plugins first.

Unfortunately, that doesn't work. If I rename the mobipocket plugin, it will not load DjVu (same problem, caused by the mobipocket plugin) or MobiPocket (I think because it tries to use the hardcoded file name)

[luite]

Ok, I have found something. If you change the first 'if(x)' to 'if(!x)' in the loop after the load_all_plugins call in uds::PluginMgr::create_document you can load all files (tested mobipocket, pdf, djvu) correctly. (I don't have the source code, so it might look a little different of course )

I'm not sure if this is the real problem or that it's just a coincidence that this change just makes it work.

[luite]

I have tested the plugin for a few more days on my reader and haven't encountered any problems. I'd like to make a 0.2.0 update with a new installation method (no more schema overwriting) and that doesn't require removal of mobipocket support, but I need to know a few things for that. I hope someone (from irex?) can answer them:

- Is the fix I found for UDS 'correct' (see previous post, it's about uds in firmware version 1.7.1)? Or is there a better way to make it work?
- I noticed that the virtual machine does not load the content browser when uds is removed. If something goes wrong overwriting uds or other system files, is there still a way for end users to recover their firmware, without sending in the device?

after the basics are in place, the plugin could be improved/extended:
- support for table of contents, hyperlinks, text search
- better image rendering (libdjvulibre does a very simple bilinear resample to rescale the high resolution bitonal (black/white) images to a relatively low resolution screen size grayscale image, which makes the images slighly blurry. this could be improved, but obviously shouldn't take too much time. does anyone know a suitable algorithm that preserve edge sharpness but don't destroy readability of small characters?)

any other requests/suggestions?

[Gertjan]

Originally Posted by luite

- Is the fix I found for UDS 'correct' (see previous post, it's about uds in firmware version 1.7.1)? Or is there a better way to make it work?

I don't know what happens there but it looks like your plugin reports that it uses a "high level" interface which is incorrect. If you provide source code of the plugin we can see what's going on.

- If something goes wrong overwriting uds or other system files, is there still a way for end users to recover their firmware, without sending in the device?

The end user is always able to recover the device by re-installing a firmware update. In case the device does not start in the normal way, the update can be copied to the SD card directly. A hard reset of the device triggers the update process.

[luite]

Thanks for the reply

Originally Posted by Gertjan

I don't know what happens there but it looks like your plugin reports that it uses a "high level" interface which is incorrect. If you provide source code of the plugin we can see what's going on.

I will clean up the source a little to make it ready for distribution. Like I said in the other thread, it is based on the pdf plugin (1.7.1 svnr4447), and the plugin_impl directory is almost unchanged. I only modified the file library_impl.cpp, to change the ".pdf" extension in g_ext_table to ".djvu".

The end user is always able to recover the device by re-installing a firmware update. In case the device does not start in the normal way, the update can be copied to the SD card directly. A hard reset of the device triggers the update process.

Thanks, that makes me worry a little less about releasing things that copy files to system directories

By the way, I saw that the DR800's plugin interfaces have been (slightly) changed, will the new DR1000 firmware also use the new interfaces?