PDFRead 1.7 released

[gdxf]

Originally Posted by ashkulz

I guess we'll have to live with the current quality then. Beyond a certain point, the quality of the tool matters less than the quality of the input document ... The only thing I can think of is to add 1/4 mode, but that would work only for documents which have 2 column text.

ashkulz, please consider adding a 1/3 mode too. From my experiments, I find that if you reduce the width to 2/3 of its original 784 pixels (which is 523, perhaps the overlap should be counted here, then another 15 pixels added to 523 =538) and keep the length at the same 1012, the result would be 3 evenly split parts, very legible because it is magnified in each part as a result of adjusted aspect ratio, actually much more readable than two halves.

[ashkulz]

Originally Posted by gdxf

ashkulz, please consider adding a 1/3 mode too. From my experiments, I find that if you reduce the width to 2/3 of its original 784 pixels (which is 523, perhaps the overlap should be counted here, then another 15 pixels added to 523 =538) and keep the length at the same 1012, the result would be 3 evenly split parts, very legible because it is magnified in each part as a result of adjusted aspect ratio, actually much more readable than two halves.

That would mean in the Reader landscape mode, right? I'll be busy for a few days, so probably will work on it this weekend...

[gdxf]

Originally Posted by ashkulz

That would mean in the Reader landscape mode, right? I'll be busy for a few days, so probably will work on it this weekend...

It should work the same way as when you did landscape-half mode, where you use "-p prs500-l -m landscape-half". You can simply modify it a little bit to become "-p prs500-l -m landscape-one third". In my view, whether the width/length is 1012/784 or 754/584 does not matter much, since the aspect ratio is about the same 1.29. In one third mode, the aspect ratio may be approximately 1.88. You may test it on the Connect Software yourself on these figures. Thanks.

[alex_d]

ashuklz, the samples page for pngnq shows amazing rgb->256 quantizations. However, I think its magic is about finding the right palette and it doesn't let you specify your own. If so, then it's the wrong tool for the job because its optimized palette would degrade quality because the reader cannot display it. The only tool i've found so far that does color reduction properly is pnmquant.

Anyway. It looks like you are creating an extremely powerful backend and pdfread may soon obviate rasterfarian. Before that happens, though, you need to add a few more screens to your windows gui. One screen has to let you set up how the page looks. This means rotation, splitting, etc. Optionally in addition, you should have a page that shows an in-window preview (which is important with all your options for autocropping, etc.) Another page, in the beginning, should let you collect multiple source (e.g. multiple images or chapters from a book). Also, some form of GUI batching is also needed (maybe it can go on that first page). What i did in rasterfarian was to use the filesystem to effect semaphores. Very dirty, but better than nothing. I don't know too much about the limitations of the way you do your gui, but at least some things should be implemented.

But anyway, now that we have a good backend I really hope someone who knows C# or something can step up and make an excellent front end too. It upsets me how few people know of these rasterization tools (eg on sites that talk about the reader or in reviews). That means something very big is still missing.

[kovidgoyal]

I am working on a GUI right now. Just have to make my current GUI multi-threaded.

[ashkulz]

Originally Posted by kovidgoyal

I am working on a GUI right now. Just have to make my current GUI multi-threaded.

That's very good news! So libprs500 will hopefully be the official GUI going forward

BTW, I think the implementation of the Device class for REB1100 should be done in a day or two, so you'll also have to refactor the GUI to take that into account too...

[ashkulz]

Originally Posted by alex_d

ashuklz, the samples page for pngnq shows amazing rgb->256 quantizations. However, I think its magic is about finding the right palette and it doesn't let you specify your own. If so, then it's the wrong tool for the job because its optimized palette would degrade quality because the reader cannot display it. The only tool i've found so far that does color reduction properly is pnmquant.

Well, I've done conversions using both pngnq and pnmquant and I've found that the pngnq conversions look best on the desktop. All the other devices I have access to either support monochrome (so it doesn't matter) or at least 16 colors (where there the palette difference would be small). Also, most of the PDFs I tested were mostly text where there aren't too many colors involved -- results may be different if images are involved. If you (or someone else) can verify that there is significant degradation then I'll implement pnmquant side by side with pngnq and use pnmquant for the reader profiles.

Anyway. It looks like you are creating an extremely powerful backend and pdfread may soon obviate rasterfarian. Before that happens, though, you need to add a few more screens to your windows gui. One screen has to let you set up how the page looks. This means rotation, splitting, etc. Optionally in addition, you should have a page that shows an in-window preview (which is important with all your options for autocropping, etc.) Another page, in the beginning, should let you collect multiple source (e.g. multiple images or chapters from a book). Also, some form of GUI batching is also needed (maybe it can go on that first page). What i did in rasterfarian was to use the filesystem to effect semaphores. Very dirty, but better than nothing. I don't know too much about the limitations of the way you do your gui, but at least some things should be implemented.

But anyway, now that we have a good backend I really hope someone who knows C# or something can step up and make an excellent front end too. It upsets me how few people know of these rasterization tools (eg on sites that talk about the reader or in reviews). That means something very big is still missing.

Thanks! I still have plans to improve PDFRead backend, I will prefer to leave the frontend to kovidgoyal Some things I am planning to add:

1. the 1/3 mode requested by gdxf
2. 1/4 mode, similiar to what you have in RasterFarian
3. color support throughout (only useful for the 1200 atm)
4. "reflow" mode

The "reflow" mode is where I am going to put in the most effort. That will take the dilated image at 300dpi, use page segmentation techniques for figuring out the individual lines and words (taking inspiration from OCR) and then cutting/pasting image segments so that the aspect ratio of the image is changed, while keeping the same text. This is important as sometimes even when using landscape mode the text is too small because of the page size and/or the aspect ratio. This will effectively reflow documents having mainly text content. I'll be using the ideas/code I've already implemented for aggressive cropping and taking it quite a bit further

[gdxf]

Originally Posted by ashkulz

[*] the 1/3 mode requested by gdxf[*] 1/4 mode, similiar to what you have in RasterFarian[*] color support throughout (only useful for the 1200 atm)[*] "reflow" mode[/list]The "reflow" mode is where I am going to put in the most effort. That will take the dilated image at 300dpi, use page segmentation techniques for figuring out the individual lines and words (taking inspiration from OCR) and then cutting/pasting image segments so that the aspect ratio of the image is changed, while keeping the same text. This is important as sometimes even when using landscape mode the text is too small because of the page size and/or the aspect ratio. This will effectively reflow documents having mainly text content. I'll be using the ideas/code I've already implemented for aggressive cropping and taking it quite a bit further

Those improvements are all great news! I am eagerly looking forward to the new release. Ever since the release of the Sony Reader last year, I think one of the most crucial issues for effective reading on it is converting large-sized pdf pages into mutiple smaller pages without losing much legibility.

[alex_d]

ashkulz, that page reflowing sounds pretty cool. will it only take out big spaces, or could you do something subtle like decrease line spacing? Actually, for that, you'd probably be better off trying something drastic like modifying the pdf itself. (It's nearly impossible to increase boldness that way... believe me, I've tried... but font kerning should be easier.)

Another thing you should put in the backend is the ability to render any subsection of a page. JAP used this feature as the basis of the tool, and for a suitably powerful gui it would allow the greatest flexibility.

My next project I think will be to start writing a new gui for the Sony Reader. (The knowledge for doing this is a bit scattered right now, but I've talked with some people who collectively seem to have all the pieces.) At first i'll focus on the core rendering to speed up page-turns (very important for nonfiction/textbooks that I read) and improve quality. I'll do this by implementing page caching and change the the refresh policy in a way that lowers battery life (i.e. no more black-white flashing but rather multiple refreshes). Cold page turns should go from the 2s they are now (for rastered .lrf) to 0.25s or less. Also I'll implement page-number entry and folder-based navigation (and maybe panning, zooming, and even searching... although searching won't be straightforward seeing as I'll still be using an image-based book format). I know none of the above things are an issue for you, ashkulz, with your fancy 1100, but hopefully it'll be the start of a good project.

I've also given some thought to a better font-enhancement algorithm vs edge enhancement that would essentially do autohinting on bitmaps. I'll try experimenting a bit with that (but I have a feeling it'll be incredibly processor-intensive).

[alex_d]

P.S. For 16-color devices, the dithering won't make a big difference. For the 4-color sony, it does. White backgrounds get a light gray snow and black text gets a dark gray one. The end result is decreased contrast on a device which has poor contrast to begin with. I made a mistake, however. The right tool is NOT pnmquant (which does the whole find-the-best-palette bs). Rather, it is pnmremap which lets you specify a custom palette. The command line for it is 'pnmremap mapfile=palette.pnm -floyd' where palette.pnm is a file that contains:

P2
4 1
255
0  85  170  255

or

P2
16 1
255
0  17  34  41  68  85  102  119  136  153  170  187  194  211  238  255

Also, in regards to further tuning, I suggest you tweak your default settings a little. This may vary by device, but on my reader the best edge enhancement seems to be 7 not five, and you may want to change your default dilation factor. In fact, how exactly are you doing it right now? By setting a dpi and then multiplying by page size? That would give very inconsistent results if the size of the page is different from A4 (a good example is if one does manual cropping as one should, but also most pdf books are not a4).

[kovidgoyal]

Originally Posted by ashkulz

That's very good news! So libprs500 will hopefully be the official GUI going forward

BTW, I think the implementation of the Device class for REB1100 should be done in a day or two, so you'll also have to refactor the GUI to take that into account too...

That's great I'm going to refactor the GUI anyway to support multithreaded device operations as I dont like having the GUI freeze while transferring stuff to the device. There may be a couple of small additions to the device interface to support multithreading...nothing that would require any major rewrites though.

[ashkulz]

Originally Posted by alex_d

ashkulz, that page reflowing sounds pretty cool. will it only take out big spaces, or could you do something subtle like decrease line spacing? Actually, for that, you'd probably be better off trying something drastic like modifying the pdf itself. (It's nearly impossible to increase boldness that way... believe me, I've tried... but font kerning should be easier.)

alex_d, PDFRead already supports "taking out big spaces" and "decreasing line spacing" -- it's done by the cropping mechanism. If you reduce it from 2.0 to something like 1.0 or 0.5, it will decrease the inter-line spacing drastically. An example of removing big spaces is present in the demos at the start of the thread (the "romans" example).

What I want to do with reflow is to break up indiviudal lines, and move part of the words to the next line (and so on for the rest of the lines). Something like what happens when you reduce the browser window size, the text automatically reflows. I've got a very simplistic algorithm in mind, but I'll document it after I've actually tested it out

Originally Posted by alex_d

Another thing you should put in the backend is the ability to render any subsection of a page. JAP used this feature as the basis of the tool, and for a suitably powerful gui it would allow the greatest flexibility.

I can't think how to implement this properly, as specifying "which" subsection of the page is quite tricky (considering I have to support multiple devices and dilation at variable DPI). It'd be better to crop it properly in Acrobat or whatever, and then render it through PDFRead (as it respects the CropBox).

My next project I think will be to start writing a new gui for the Sony Reader. (The knowledge for doing this is a bit scattered right now, but I've talked with some people who collectively seem to have all the pieces.) At first i'll focus on the core rendering to speed up page-turns (very important for nonfiction/textbooks that I read) and improve quality. I'll do this by implementing page caching and change the the refresh policy in a way that lowers battery life (i.e. no more black-white flashing but rather multiple refreshes). Cold page turns should go from the 2s they are now (for rastered .lrf) to 0.25s or less. Also I'll implement page-number entry and folder-based navigation (and maybe panning, zooming, and even searching... although searching won't be straightforward seeing as I'll still be using an image-based book format). I know none of the above things are an issue for you, ashkulz, with your fancy 1100, but hopefully it'll be the start of a good project.

Do you mean something that will actually run on the Sony Reader itself? That'd be absolutely great, as I don't think anyone has gotten/wanted to run apps on the Sony yet (as compared to the Librie). Wish you the best of luck, and hats off to you

I've also given some thought to a better font-enhancement algorithm vs edge enhancement that would essentially do autohinting on bitmaps. I'll try experimenting a bit with that (but I have a feeling it'll be incredibly processor-intensive).

It's done by all desktop PDF Readers I've seen, and I think it would be rather processor intensive, even if you do it in C.

[ashkulz]

Originally Posted by alex_d

P.S. For 16-color devices, the dithering won't make a big difference. For the 4-color sony, it does. White backgrounds get a light gray snow and black text gets a dark gray one. The end result is decreased contrast on a device which has poor contrast to begin with.

Hmm, I think that I'll go the image -> pngnq -> pnmremap route. What I'll do is add another command-line option, --grayscale-remap which will take a comma seperated list of values for the allowed pixel intensities. Then, during conversion I'll create the map dynamically and call pnmremap with it. I'll set it up so that the reader profiles have the proper values set properly. Do you think that's a good approach?

Originally Posted by alex_d

Also, in regards to further tuning, I suggest you tweak your default settings a little. This may vary by device, but on my reader the best edge enhancement seems to be 7 not five, and you may want to change your default dilation factor. In fact, how exactly are you doing it right now? By setting a dpi and then multiplying by page size? That would give very inconsistent results if the size of the page is different from A4 (a good example is if one does manual cropping as one should, but also most pdf books are not a4).

Hmm, I think that 7 does a little too much edge-enhancement, but I'm willing to change it. Actually, what would be nice if someone could volunteer to test various parameters and report what looks the best. Any volunteers?

I also do dilation and resizing differently from RasterFarian. I render the PDF/DJVU at the dilation DPI, without specifying a page size. So Ghostscript or DJVU automatically create a image with size appropriate for that resolution. I don't know what the size is up front at all, and in fact it varies from book to book. I perform cropping/dilation at that DPI, and then depending on the mode I resize it down and split it up if necessary while maintaining the aspect ratio. This ensures that whatever may be the input page size it really doesn't matter to PDFRead -- what matters is the DPI, aspect ratio of cropped content and the mode.

BTW, I've already implemented everything except the "reflow" mode in SVN. I hope to have at least a rudimentary implementation done by tomorrow (I'll optimize for speed later).

[ashkulz]

Originally Posted by kovidgoyal

That's great I'm going to refactor the GUI anyway to support multithreaded device operations as I dont like having the GUI freeze while transferring stuff to the device. There may be a couple of small additions to the device interface to support multithreading...nothing that would require any major rewrites though.

Instead of trying to make the GUI multithreaded, wouldn't it be a bit simpler to add a callback to display the progress during bulk writes, which is where it would freeze most of the time? So you could display a progressbar showing [[[ xyz bytes transferred ]]]. If you don't specify the callback then you'd simply won't call it. Don't know whether calling it directly and updating the Qt control will work, as I recall that you have to change control values only from the event thread for Win32, there must be someway of enqueuing things to run in the event thread. I assume you know what'll work in Qt

[kovidgoyal]

The callback way is how its done now. When I said the GUI freezes, I meant only that you cant perform other tasks while waiting for bulk writes to finish.
The reason I'm switching to threads is that I will want to make a bunch of processes multithreaded...converting from TXT,HTML,PDF->LRF as well as actually copying files to the device. In the future there will probably be more functionality as well. Making it multi-threaded will make implementing a queueing system easy. Besides I don't have much experience with multi-threaded programming and I'm looking at this as a good way to learn ;-)