KPV: a PDF reader for Kindle, based on muPDF, GPLv3

[hawhill]

Hi,

(hidden OP text, since it has only historic value now

Spoiler:

as they say: release early, release often. So I'm presenting my first approach towards creating a better PDF reading experience - mainly for myself, that is. It still has a long way to go and as it is, it is only a proof-of-concept.

I was interested in how fast a PDF reader could actually be. I used the muPDF library (http://www.mupdf.com/) before and it has a good reputation for having a small memory footprint. On the other hand, I have made some bad experiences (mainly in terms of rendering speed) using poppler-qt4 binding.

Another choice I did make was to implement the muPDF binding (and some other stuff) with a Lua API. So the user experience is created by a Lua script, which calls into C for doing actual work. This is to make quick changes to user interface related things and should make for a very adaptable reader application later.

Note that currently the focus is still very much on development. The reader has no graphical UI and only displays in fit-to-page mode. A lot of things are hardwired. And you will need a terminal on the Kindle or USB networking to start it in the first place - it doesn't have a file chooser. And last but not least, documentation could certainly be much more extensive. However, it is pretty fast in my first tests. And by the way, it should do JPEG2k and JBIG2 streams, too.

This is a document reading application for (jailbreaked) Kindles. It will probably be useless for now on the Kindle 4 (Non-Touch) version. It is basically directed at Kindle 3 (or Kindle Keyboard), Kindle DX(G) and maybe (untested) Kindle 2. For touch based devices, have a look at the successor, KOReader: https://www.mobileread.com/forums/sho...d.php?t=209276

It brings its own file browser and comes bundled with a script for the "launchpad" utility.

Despite its name kindlepdfviewer, it supports a lot of document formats.

Features: PDF and DJVU reading with support for table of contents, arbitrary zooming, fit to page content, gamma correction plus a bunch of other features. Also supported are XPS, CBZ (but not CBR, sorry), and thanks to CREngine also text-based formats FB2, TXT, HTML, RTF, CHM, MOBI, EPUB.

The software is still under development. Nevertheless, please report bugs you may encounter (but check the latest discussion here in the thread to avoid duplicates). Also, you're very much invited to get your own hands dirty on the code. It's open source and the development happens mainly on github. After I started it in a humble proof-of-concept version, dpavlin, dave2008, germanc, traycold, NuPogodi and others made great contributions and it's a reader that's now really useful.

For more information, relate to this thread - and the overview in the wiki: https://github.com/koreader/kindlepdfviewer/wiki

So for the brave, here it is:

• sources are available at http://github.com/koreader/kindlepdfviewer
• compiled binaries are available via the Download page in the Wiki: https://github.com/koreader/kindlepd.../wiki/Download (also has the Changelog)

Basic installation / update (with launchpad):

• You need to jailbreak and install launchpad first
• Create a folder "customupdates" on the exported Kindle drive, if not yet present
• Copy a release ZIP file from https://github.com/koreader/kindlepd.../wiki/Download into the folder "customupdates"
• Press Shift-Shift-I sequence on the Kindle for installing (if this is the first time you use launchpad: you only have 0.7 secs for the sequence!)
• Press Shift-Shift-Space sequence on the Kindle to reload launchpad
• done! now you can run the viewer via a Shift-P-D sequence.

Basic installation / update (with Kite)

• make sure Kite is installed. You can find the software and instructions here: https://www.mobileread.com/forums/sho...d.php?t=168270
• unzip the contents of the installation ZIP file in the root directory of the volume the Kindle presents via USB
• Kite should now present two new pseudo-documents for starting KPDFviewer

Uninstallation:
Please note that uninstallation is not needed when updating the application. Just install a newer release ZIP as described above (unless otherwise mentioned). But if you want to remove it for good, this is the procedure:

• Just delete "kindlepdfviewer" folder.
• Also, delete "kpdf.sh" and "kpdf.ini" from the "launchpad" folder
• Delete KPDFviewer... files from "kite" folder"
• You might want to scan the drive for files "*.kpdfview.lua", in which document specific settings got stored - and delete them too.

Known problems:

• Unfortunately, there might always be bugs. You can look at the project's bug tracker to see if we are already aware: https://github.com/koreader/kindlepdfviewer/issues - also, discussion there is always welcome!
• If you happen to get a cluttered screen, please try to restart Amazon framework by issuing launchpad command Shift-P-R (for "Restart framework")
• Kindle DX(G): The device hasn't that much RAM. Leaving Amazon's framework running, you may experience funny restarts and similar stuff. We added a launchpad sequence for you that will shut down Amazon's framework (fast) and restart it when you quit kindlepdfviewer (slow). Press launchpad sequence Shift-P-K (for "Kill framework") in order to do that.

For the current release, have a look at the Download page: https://github.com/koreader/kindlepd.../wiki/Download

[vdp]

Hi,

that's great news !

This weekend I made some quick experiments with muPDF and poppler to measure how fast they render the pages. I've used pdfdraw(mupdf) and pdftoppm(poppler). In both cases modifications were made in order to measure only the rendering time and to exclude the time needed to write the image to the flash drive. It looks like muPDF is twice as fast as poppler. About the mupdf rendering problem that I've told you about in a previous(OT) post, it turns out to be introduced in a relatively recent commit and reverting to an older revision(28e88083d782eb) does work as it should. The people at Freenode's #ghostscript channel seem to be quite responsive too. So I completely agree with your choice of PDF rendering library.

I admit I know nothing about Lua, except what I am hearing about it - that it is minimalistic and fast(for a scripting language). Are you planning to use it just for a quick prototyping or you think it should be part of the end product too?

What library you plan to use for the UI? I still think Qt would be a great choice if we can make it fast enough. I still haven't have the time to play with it, but I intend to do it. In my opinion using a tested and mature UI toolkit could be a big advantage. I don't have a previous experience with Qt but I think it can be quicker and more pleasant if we are able to develop the application on a desktop using the virtual framebuffer features of Qt(qvfb).

[hawhill]

Yes, I did only use Lua for prototyping right now. Your characterization is quite right: it's lightweight. I would not say "fast", though, but that's a relative term anyway. It's easy to interface from C code (I would even go as far as saying it's fun to do so).

I think you can get a quick impression of it by looking at the "reader.lua" file.

As for the UI, I always found Qt to be a bit sluggish. So I was even thinking of implementing my own UI. I'm not too sure about that, however, since it would definitely be more work than just relying on Qt standard dialogues. Qt/qvfb would also be indeed a great help.

Thanks for the tipp regarding the muPdf problem in newer revisions. I'll have a look at it.

Edit/Addendum: Hm, maybe integrating it into CoolReader would be an option. I'll have a look at it. However, there doesn't seem to be an up to date source code dump available right now. It's based on Qt.

[vdp]

Today I've had a little time to try your viewer. It works very well and is fast indeed. Even the gamma calculation, which as I understand from the commit log implies some performance penalty, doesn't seem to cause a noticeble slowdown. Are you working on zooming and scrolling already? Say something like shift+FiveWay_UP to scale up the document with 30% and alt+FW_UP to scale up with 5% ...?
Perhaps I can help with this, but I will need some time to familiarize myself with Lua and the codebase as a whole and I don't know if I will have enough free time this week.

About the GUI I still haven't made any experiments with Qt but I think that there should be a way to draw on the framebuffer without performance loss for example using QDirectPainter or even a more direct(hacky) way if needed. After all we are providing the screen driver plugin. What I want to try is to use some kind of direct rendering for the pages, and return the control to Qt to visualize the dialogs. This way even if the painting of the dialogs is somewhat sluggish it won't affect the user experience most of the time.

[hawhill]

Well, a lot of its speed it currently gains from a simple caching strategy. It will always render the next page just after displaying the current one.

I think starting on zooming and navigation will be the next thing I'll tackle. I will have to revise the caching mechanism and policy for that, I think. Also I'd like to include support for muPDFs "bbox" device that will calculate a "real usage" dependent bounding box for a given page which would allow for cut-to-content feature (the efficiency of which would depend on how the PDF is created, though).

Note that I just added an "emulator" mode that will use SDL for rendering/input, so development on a typical linux PC is possible. This allows for rapid prototyping.

Don't feel pressed to start on this. It's still a "pet project" for me, I'm doing it just for the joy of doing it. BTW, dpavlin already had a look at it and played with it. Github is a great help for integration work, so if you start playing with it, I suggest to just clone the Github repo.

[Kindlevarazs]

Hi,
I think a lot of Kindle users would be very very happy for the solution of better PDF handling on Kindle3... So good luck for You!

[h1uke]

Originally Posted by hawhill

However, there doesn't seem to be an up to date source code dump available right now.

The latest cr3_kindle source code is available here.

[hawhill]

Thanks, that's great! Now I'm even more tempted to make a Qt integration at some point...

[hawhill]

I've just updated the release version (see link in OP for "compiled version"). It now comes in a ZIP file. You are supposed to copy the contents into a folder "test" (sorry, still didn't manage to come up with a clever storage point...) in your document partition. Except for the contents of the "launchpad" subfolder, which you can copy into the launchpad config folder on your document partition if you want to use this integration.

The software now has Gamma adjustment (volume keys), beginnings of flexible Zoom including fit-to-content and now stores last read page and last used gamma for each document viewed. Oh, well, and it comes with a simple file chooser now.

I think it's coming towards a point where the main development will focus on the Lua scripts and the C code part is almost stable (although still not complete, things like metadata or text extraction are still missing). I still like the lightweight approach, and it's reasonable fast to script a UI in Lua. I've added access points to font rendering as freetype is included anyway (and some fonts are also always there).

It might also be a base for scripters who look for a base that can interface with the framebuffer, the input and also brings sqlite. Lots of little apps come to my mind. So if you like Lua scripting, this might be for you, even if you don't use it to read PDFs.

[geekmaster]

@hawhill: Do you know of a way to "pause" the framework, so it does not mess with the framebuffer, but can be resumed quickly (not needing a full framework start)?

I tried to kill -STOP the /usr/java/bin/cvm process, which certainly works but seems rather extreme. Even the power switch stops. I can still run launchpad apps though.

kill -CONT resumes the cvm process, but when it wakes up it wierds-out, asking if I want to "Delete Collection"... Hmm...

There *must* be a better way...

P.S. I really do like your PDF viewer. The double-buffered (pre-cached) rendering is great. Keep up the great work. The updates might feel faster if the next-page does a mode 2 (fast) update on button press, followed by a full flash mode 1 update on button release. Mode 21 (negative image) updates could be a useful option.

[h1uke]

Originally Posted by geekmaster

kill -CONT resumes the cvm process, but when it wakes up it wierds-out, asking if I want to "Delete Collection"... Hmm...

you have to grab ( ioctl(fd, EVIOCGRAB, on) ) all input devices in order to avoid
key presses to be put into CVM's input queue. See capture_input() and release_input() in launchpad/src/main.c

This technique is used in Qt keyboard driver for kindle. I even do not send kill -STOP / kill -CONT to CVM,
but grab all input devices if I want to run Qt application in a foreground.

[Iznogold]

how do I install this?

[geekmaster]

Originally Posted by h1uke

...
This technique is used in Qt keyboard driver for kindle. I even do not send kill -STOP / kill -CONT to CVM,
but grab all input devices if I want to run Qt application in a foreground.

I want to stop the framework from overwriting the top and bottom of the screen (3G signal, battery, etc.). So I think I still need to pause he CVM in that case. That is less severe than a full framework stop and restart like some fullscreen apps do.

[hawhill]

Originally Posted by Iznogold

how do I install this?

Grab the ZIP from the page, unpack most of its contents to a folder "test" (another one would work, too, but that matches the launchpad scripts) on your Kindle's document partition, except for the contents of the folder "launchpad" which go into the existing launchpad folder (that step would require you installed launchpad beforehand). Reload launchpad config, then you could run it with Shift-P-D (for the file browser mode).

Note that it is still more or less a preview of things to come. I publish it to inspire people to play with it - and that would also mean to change its code.

[geekmaster]

This looks like a useful feature to add. It could be integrated into your PDF viewer (one page at a time), activated by a menu item or button.

The script that splits the 2-column PDF:
http://www.cs.toronto.edu/~hector/1colpdf

A sample PDF. Original:
http://www.cs.toronto.edu/~hector/jcan.pdf

Converted:
http://www.cs.toronto.edu/~hector/1col-jcan.pdf

Much better than the "zoom box" in the webkit browser, IMHO.