A4 PDFs - tips, tricks, and a call to the community

[alex_d]

I've been playing with the reader for a bit, and it seems to me that the problem with A4 pdf display isn't so much the screen size or the paper size or all that benefit-of-the-doubt stuff... it's just bad font rendering. It's not bold enough and the antialiasing does more harm than good. Even plain, blocky, aliased letters look much more readable on the Reader at small font sizes.

I think what happens is that the Reader's pdf viewer tries to use antaliasing to render the fonts most 'faithfully'. But at small font sizes it ends up turning a blocky letter with sharp contrast into a little mess of grayscale. On the reader's 4-bit screen it, first of all, works poorly. But then the real kick in the pants comes when within seconds the eInk begins to unsettle and those very subtle antialiased letters lose their subtleness. Seriously, you can see the text looking half-decent after you just flip a page and then pretty quickly lose its lustre as the white in the page also loses a bit of clarity. (Sony... cut the "look...no electricity!" crap and start giving us the option to actively maintain the image!)

I hate sony. I love the eInk, but sony always royally messes up the software. It tries to pull an Apple by not giving you any options and only grandma-approved features, but then can't even do that right. The only hope I have for the reader is that the modding community will pick up the ball.

Possible solutions:
Hack to fix font rendering on the Reader
Hack to actively refresh the Reader display (useful in combo with other solutions)
Converter to rasterize PDFs (and do the text-cleanup on the PC... plus it can get rid of whitespace and of course speed up viewing)


Right now, I'm investigating that last option. I've tried several utilities but none do it right. FINALLY I had some success with imagemagick (you also need to install ghostscript). Right now I got it to put out .pngs without any antialiasing, and it is MUCH more readable. Still ugly... but it doesn't make your eyes pop out. I have to tune it a lot more and create some script that turns the images back into a pdf. It would also be nice to substitute a custom font (one of those DOS fonts optimized for consoles). Imagemagick does, though, seem to have a huge wealth of options and potential. Custom programs accessing ghostscript or another pdf rasterizer library directly would probably be even more powerful.

I'll update as I play with this some more, but it seems the holy grail of reading A4 pages on the reader is a step closer.

Also, I'll try to post some example files ASAP

Does anyone have any other advice on the matter?

[alex_d]

Here is what i'm talking about:

PDF page
PNG page

Copy these to your Reader and compare. Opening it on the PC is a bit meanigless.

The png was made using gsview, using "convert" to pnggray at 82 dpi and with options -dTextAlphaBits=2 -dGraphicsAlphaBits=2 (these set antialiasing bitdepth).

[alex_d]

P.S. To prevent rescaling, I formatted the PNG to 600 pixels width and had to just cut off the bottom. What the hell is the vertical resolution of the viewing area (ie, minus the status bar at the bottom)????

[alex_d]

Here's a much more powerful example:

Almost totally unreadable:
PDF
Merely somewhat illegible:
PNG

Obviously there's still a long way to go and the 800x600 resolution does make it inherently difficult to display a full A4 page.

But the point here is that Sony's pdf viewer can be improved TREMENDOUSLY. It can't even render a half-A4 page legibly, and that's inexcusable.

[alex_d]

I made a 2nd version of example 2:
PNG - example 2b

I rasterized with 4-bit antialialising instead of 2-bit. Although the Reader's screen is 2-bit, 4-bit antialising seems to give better results. I don't know why, but there it is.

[NatCh]

Originally Posted by alex_d

What the hell is the vertical resolution of the viewing area (ie, minus the status bar at the bottom)????

768 pixels. The status bar is 32 pixels high, near as we can figure.

This might be of interest.

[Sponge_Bath]

I've tried a lot of experimenting with PDFs and fonts:
GhostScript produces PDFs that use Type 3 fonts and results in blocky/bitmapped look
Acrobat, I find is slightly nicer, because it creates PDFs that use Type 1 fonts but it's still faint as most people are complaining about.

The best font that I have found for the reader when creating PDFs is ComicSans (around 14 to 18 point on A4 size with a very small margin all around say about .1mm to .25 mm) because of its uniform thickness.

Does anybody know how to switch the rendering from 2bit(4 shades) to 1 bit(b&w) that might help the rendering?

Cheers.

[alex_d]

natch, thanks regarding resolution.
sponge, yeah, switching to 1bit rendering (ie, turning off antialiasing) might help if its possible.

Hmm... I think I'm pretty satisfied with the PNG files I'm producing (eg example2b), but now i'm stuck on the question of what to do with them.

Man, I just wish I could use the PNGs directly, but there's no way to organize the pictures into folders! @!#$#$@# sony! Converting to pdf seems to mess up the images... maybe someone knows how to fix that? Or maybe converting to BBeB would be smoother?

If not that, then maybe someone knows a source of really cheap SD cards, lol?

[alex_d]

UPDATE: Readability seems further improved when you invert the colors and make it White text on Black background.

PNG example 2c


Also, I should for reference say which command I used to output to PNG:

"C:\Program Files\gs\gs8.54\bin\gswin32c.exe" -q -dBATCH -dSAFER -dNOPAUSE -sDEVICE=pnggray -dTextAlphaBits=4 -dGraphicsAlphaBits=4 -g600x768 -r72x72 -dUseCropBox -sOutputFile=out%03d.png -fin.pdf

gswin32c.exe is a binary you get when you install Ghostscript
-f specified input files
%03d inside the output filename tells it to output one PNG per page, replacing "%03d" with 001, 002, etc.
-r72x72 specifies raster size. 72 is good for A4. If you use a different page size, change this or the output will be either cropped or won't fit the whole image (which is always 600x768). You can also use this to stretch the produced image by using differing numbers.
-dUseCropBox crops PDFs, but only if they have defined a crop box. It kinda really sucks that ghostscript can't autocrop files itself (ie, what the Reader tries to do).

now, I just have to find a commandline program to invert PNG colors.

also, there are probably utilities which'll manipulate PDFs to autocrop them or add cropboxes before they're sent to ghostscript. I'm calling it a day for now, but maybe someone already knows which ones those are? It would be extremely useful.


Now, the UBER hack would be to compile ghostscript for the Reader, load it, and use it as the rastering engine. Earlier I mentioned that fixing the Reader would probably be better than converting, and it now seems there's a [relatively] easy path to doing it. Now why the hell couldn't Sony have done it itself? Can I curse at Sony some more? This hack also presents an opportunity to do stuff like pre-rastering the next page or, for godssake, make navigating books easier with an improved UI (like one where you can, say, USE THE NUMBER PAD TO ENTER PAGE NUMBERS).

[kovidgoyal]

pdfcrop is part of the tetex distribution

[alex_d]

hmm... i should've realized that autocropping won't really work. Because of the manual way I tell ghostscript how big to render the pdf, all the pages of the document have to be the same size.

Besides this, autocropping leaves behind a lot of stuff that's better off cropped. It does bother me that this adds manual steps, but the best result is to just go into Acrobat and set a cropbox for all the pages. For the Scientific American I'm using as an example, this worked really well. This magazine always sticks the actual text within a 7.75"x9.75" box, but usually has various other bits of crap going all the way to the edge.

After I edited in Acrobat, I then used the -dUseCropBox flag in ghostscript and changed resolution to -r82x81 (to enlarge and stretch a little horizontally).

To batch invert colors, I issued:
convert "out*.png" -negate "out%03d.png"
convert is a command you get after you install ImageMagick.
the weird characters in the filenames are there to get the numbering correct.


Result:
PNG - example 2d - Final Result
Compared to the original:
PDF - example 2
It is simply night and day. (pun intended)
I can now comfortably read full A4 pages on the Reader (in portrait mode!), justifying the purchase for me. I think others will feel the same way.


What's left is to figure out how to organize the PNGs ($#@#$ sony) and to simplify all this with a nice interactive batch-script wizard (or better yet a VB gui) and a howto. (I know the command lines must have really scared some people.) Lastly, this all needs to be extended to landscape mode which many people will likely prefer. (just render at 800x1128 or so, and split in half)

Also, to edit the cropbox in Acrobat I had to remove PDF security using PDF Password Remover v2.5. (The same thing can also be accomplished by creating a new pdf in Ghostscript and copying in the old one.)

P.S. I'm really not sure about the 768 vertical height. Just to be on the safe side, I'm using 764 so I can know that there is no resizing since I can see the extra borders.

[SoCal Bob]

Call me simple...

I just copy and paste into a "Sony Reader" sized document in Word, upsize the text, and print in "Sony Reader' sized page in PDF for this result. I trade the publication format for legibility. Works for me. I have also attached an article from the Economist formated for my old eyes.

[kovidgoyal]

organizing the images should be possible. One way is to use the html to bbeb converter (serch the forums). generate a dumb html file which links to all the images in sequence and then convert it to bbeb.

[henkvdg]

For fun I looked how the files are readable on my iRex iLiad.
The Scientific American page is well readable directly because of the larger screen, I can read it all, although my eyes have become weaker (at 53) and I prefer larger letters. Color and grayscale images look better because of the 16 hues of grey compared to the Reader's 4. The iLiad can zoom and rotate PDF's in a lot of ways.

Probably the Sony Reader can do that as well.

[jæd]

There's a utility by Scotty which dices'n'slices A4 pdfs into A5 from back before the Iliad had zzoom... Perhaps the Sony Reader people should take a look into that...