k2pdfopt: optimizes PDFs for viewing on e-readers

[japinder]

Hi guys,

I need some help in converting a mixed color text+photo PDF to 16 shade grayscale & to trim the borders, headers & footers as much as possible, for viewing on my Onyx Boox M96.

The text has a color background, so the re-flow option creates grey colored strips of black text over a white background.

I gave up on re-flow & tried to do a simple grayscale + trim job, but although using the GUI, it shows the preview as black & white, the output is still color.

I switched to command line with the shortcut method, dropping the files on the shortcut already updated with the options. When I ONLY use the option -c-, then the output is grayscale, but re-flow occurs, with grey color strips of text. When I omit -c-, the output is still color. When I use any other option with -c-, like -wrap-, or -t, the output is again color.

PM'd the link to the file to Willus.

Please help. I'm at my wits end.

[willus]

Originally Posted by japinder

Hi guys,

I need some help in converting a mixed color text+photo PDF to 16 shade grayscale & to trim the borders, headers & footers as much as possible, for viewing on my Onyx Boox M96.

The text has a color background, so the re-flow option creates grey colored strips of black text over a white background.

I gave up on re-flow & tried to do a simple grayscale + trim job, but although using the GUI, it shows the preview as black & white, the output is still color.

I switched to command line with the shortcut method, dropping the files on the shortcut already updated with the options. When I ONLY use the option -c-, then the output is grayscale, but re-flow occurs, with grey color strips of text. When I omit -c-, the output is still color. When I use any other option with -c-, like -wrap-, or -t, the output is again color.

PM'd the link to the file to Willus.

Please help. I'm at my wits end.

@Japinder--I couldn't get your PDF file. See my PM back to you.

It will help if you show exact options that you used for various cases. If you used -mode tm (or selected "trim" from the Conversion Mode menu in the MS Windows GUI), this uses native output mode, and in native output mode, things like color, contrast, sharpness, and gamma factor cannot be adjusted, because the output file is identical to the source file except that it is cropped and scaled. So if you want to combine grayscale output with the trim mode, you need to 1. select "Trim" from the Conversion Mode menu, 2. be sure color output is unchecked, and 3. uncheck "Native PDF output." Unfortunately, as you have seen, the preview mode incorrectly shows the grayscale output (if you don't have color output checked) even when you have "Native PDF output" checked. This is a bug which I've added to my list to fix.

You may wish to experiment with the option which sets the white threshold: -wt. You can use this to convert your background color to white assuming it is light enough (use -wt+ to do this). For example, -wp+ 128 will convert to white all colors that have an equivalent grayscale pixel shade (0-255) lighter than 128 (you can put this in the "Additional Options" box). Again, this feature only works if native output is turned off.

Finally, if you need higher resolution output (with native output mode turned off), bump up the "Document Resolution Factor" to a value higher than 1.0. E.g. 2.0 will double the output resolution.

[markom]

Originally Posted by japinder

...
PM'd the link to the file to Willus.

Please help. I'm at my wits end.

Could you send me the link too, or just the name and size of pdf if it is available somewhere online ? so that I can try some other pdf tools also.

[japinder]

Originally Posted by willus

@Japinder--I couldn't get your PDF file. See my PM back to you.

It will help if you show exact options that you used for various cases. If you used -mode tm (or selected "trim" from the Conversion Mode menu in the MS Windows GUI), this uses native output mode, and in native output mode, things like color, contrast, sharpness, and gamma factor cannot be adjusted, because the output file is identical to the source file except that it is cropped and scaled. So if you want to combine grayscale output with the trim mode, you need to 1. select "Trim" from the Conversion Mode menu, 2. be sure color output is unchecked, and 3. uncheck "Native PDF output." Unfortunately, as you have seen, the preview mode incorrectly shows the grayscale output (if you don't have color output checked) even when you have "Native PDF output" checked. This is a bug which I've added to my list to fix.

You may wish to experiment with the option which sets the white threshold: -wt. You can use this to convert your background color to white assuming it is light enough (use -wt+ to do this). For example, -wp+ 128 will convert to white all colors that have an equivalent grayscale pixel shade (0-255) lighter than 128 (you can put this in the "Additional Options" box). Again, this feature only works if native output is turned off.

Finally, if you need higher resolution output (with native output mode turned off), bump up the "Document Resolution Factor" to a value higher than 1.0. E.g. 2.0 will double the output resolution.

Originally Posted by markom

Could you send me the link too, or just the name and size of pdf if it is available somewhere online ? so that I can try some other pdf tools also.

Thanks guys.

Sent the correct link/file to both of you.

I believe I was struggling since I'm still a newbie to k2pdfopt & was indeed trying out these incompatible options. I've been trying the following options:

GUI:
-mode tm -c-
-mode tm -c- -ws 0.2

CMD line:
-gui- -wrap- -c-
-gui- -c- -n

& a lot of others I don't even recall.

The PDF actually contains a lot of colorful tables/diagrams etc, so I'm afraid, by trying to remove the background from the text, I would affect those tables/diagrams as well. So I've just contented myself with trying to do a normal grayscale + trim/crop.

Another thing, when I manually specify the crop margin & play around with the device selection, I've seen a white margin being 'added' by k2pdfopt, even if I input an already cropped PDF through briss. Could you advise what would be the correct device settings for Onyx Boox M96, 825x1200px, 150dpi. And should I try to change from the default of Kindle 1-5, 560x735px, 167dpi.

Thanks,
Jp

[markom]

-wt+ 128 as Wilus suggested was good enough for me and I just then cropped left and right margins with briss, although I can crop it in Kindle DX too (with 2-point crop) and turn it to landscape whenever letters are problematic to read in portraite e.g. those small white letters in black boxes.

I can also use night mode on DX to convert those white letters to black on pages like that.

If some diagrams/tables are affected by too much I would rather use tablet for them, as I always do when color is important while reading on e-ink majority of the time.

[willus]

Originally Posted by japinder

Thanks guys.

Sent the correct link/file to both of you.

I believe I was struggling since I'm still a newbie to k2pdfopt & was indeed trying out these incompatible options. I've been trying the following options:

GUI:
-mode tm -c-
-mode tm -c- -ws 0.2

CMD line:
-gui- -wrap- -c-
-gui- -c- -n

& a lot of others I don't even recall.

The PDF actually contains a lot of colorful tables/diagrams etc, so I'm afraid, by trying to remove the background from the text, I would affect those tables/diagrams as well. So I've just contented myself with trying to do a normal grayscale + trim/crop.

Another thing, when I manually specify the crop margin & play around with the device selection, I've seen a white margin being 'added' by k2pdfopt, even if I input an already cropped PDF through briss. Could you advise what would be the correct device settings for Onyx Boox M96, 825x1200px, 150dpi. And should I try to change from the default of Kindle 1-5, 560x735px, 167dpi.

Thanks,
Jp

Yikes. That PDF has so many different colored texts and backgrounds and tables and graphics that it is really outside of what k2pdfopt was designed to process. The one saving grace is that the margins are very regular, so you can use the -cbox option to crop all the pages the same way (or you could just use Briss).

To crop each page and include the header and footer:

k2pdfopt -rt 0 -cbox .95,.48,6.63,10 -mode crop -c- -n- yourfile.pdf

This will take each cropped source page and turn that page into a single output page. The -c- and -n- are only if you really need grayscale output--I'm not sure you need this since your e-reader should convert to grayscale for you(?), and then you can just stay with native output mode. The -rt 0 just prevents k2pdfopt from checking the document orientation and thus saves processing time.

To crop each page and not include the header/footer;

k2pdfopt -rt 0 -cbox .95,.95,6.63,9.1 -mode crop -c- -n- yourfile.pdf

You probably don't need to do this since the Onyx Book M96 has such a large screen, but to fit the cropped source pages to landscape mode on your e-reader and break them across multiple pages:

k2pdfopt -rt 0 -cbox .95,.95,6.63,9.1 -mode fw -c- -n- -wt 128 yourfile.pdf

(You might try different values for -wt -- that will affect where it breaks the pages.)

You should be able to add your Onyx Book device settings to any of the above commands, e.g.

-w 825 -h 1200 -odpi 150

Of course, if you stay in native mode, I don't think it matters--hopefully your Onyx Book will fit the pages to its display.

[markom]

Orange background shall make pages appear too dark for someone's liking for reading on e-ink reader, so in this case I preferred b/w option without applied threshold (copy mode; native output disabled and color also disabled) but with contrast -cmax on 3 or 4, because there is no contrast command in Kindle DX(KPV reader), just gamma option to make pdf text darker and night mode.

This way, using contrast command, background was whiter than with color pdf, words of different colors appear darker or visually different and table nuances are preserved better than with white-threshold command.

Pdf then just quickly cropped using pdfscissors, briss or k2pdfopt and then if we are not satisfied with the flipping speed (e.g. at landscape mode), we can quickly optimize it in k2pdfopt for landscape(fit width mode, the same values for M92 as for Kindle DX).

[japinder]

Yikes doesn't even remotely express my take when, on my new M96, this was the first PDF I tried & imagine my frustration when it took ages to render each page & even then rendering it horribly. I thought I had purchased the biggest mistake of my life.

It was only later, when I tried some 'normal' PDFs, that I was able to appreciate my latest acquisition.

Thanks a ton for your advice Will & Mark. I was just stumbling along in the 'k2pdfopt' dark, till you shone some light to show me the way...

Update:
Now I'm using the combined option:
-rt 0 -cbox .95,.95,6.63,9.1 -mode crop -c- -n- -idpi 400 -cmax 4

And the results are perfectly usable. Using the cmax option also brings down the size of the generated grayscale PDF from 12.5 to 10.5 MB (from the original 4.2 MB).

A new problem, I don't know whether I like '-cmax 3' or '-cmax 4' more. The black text really pops on an almost white background with 4, but the diagrams look washed out. 3 gives somewhat more realistic diagrams, but I miss that pop. Looks like I have a problem of plenty.

[markom]

Originally Posted by japinder

...
A new problem, I don't know whether I like '-cmax 3' or '-cmax 4' more. The black text really pops on an almost white background with 4, but the diagrams look washed out. 3 gives somewhat more realistic diagrams, but I miss that pop. Looks like I have a problem of plenty.

You can try adding different gamma e.g. -g 0.55 or -g 0.6 etc. at -cmax 3, it will make background whiter than with -cmax 3 only, at the same time probably not reducing table's nuances by too much.

As it says up there behind Help menu, under command-line options:

-g <gamma> Set gamma value of output bitmaps. Default is 0.5.
A value less than 1.0 makes the page darker and may make the font more readable.

BTW, to reduce the time needed for similar testings I would usually export several problematic pdf pages as new pdf and use it instead of complete pdf, but we can of course use field "pages to convert" typing in there e.g. something like 7-9,33 to process just pages 7,8,9 and 33.

[willus]

Originally Posted by japinder

Update:
Now I'm using the combined option:
-rt 0 -cbox .95,.95,6.63,9.1 -mode crop -c- -n- -idpi 400 -cmax 4

The -idpi 400 is unnecessary if you are using 200 dpi for your device (e.g. -odpi 200) since by default the source file will be rendered at twice the output dpi. Markom beat me to the punch on explaining the -g option, which is exactly what I would have done for your -cmax 3 or 4 issue. BTW, you have to use a negative value for the -cmax argument to force that contrast value (e.g. -cmax -3). Otherwise it just specifies a maximum limit for k2pdfopt, but k2pdfopt might still choose a lower value if it determines it to be "optimum." In your case, k2pdfopt is probably using that maximum limit, so it likely won't change anything.

[markom]

Originally Posted by willus

The -idpi 400 is unnecessary if you are using 200 dpi for your device (e.g. -odpi 200) since by default the source file will be rendered at twice the output dpi. Markom beat me to the punch on explaining the -g option, which is exactly what I would have done for your -cmax 3 or 4 issue. BTW, you have to use a negative value for the -cmax argument to force that contrast value (e.g. -cmax -3). Otherwise it just specifies a maximum limit for k2pdfopt, but k2pdfopt might still choose a lower value if it determines it to be "optimum." In your case, k2pdfopt is probably using that maximum limit, so it likely won't change anything.

As for the gamma, I've tried gamma 1.9 and 2 without threshold and contrast commands, and guess what

The background with gamma 1.9 is certainly not as white as it gets but is good enough for me on Kindle DX even in portraite with smaller letters (than original ones on the paper), letters are dark enough for my liking and details and nuances in tables and graphs are better preserved.
If needed we could also use gamma correction in KPV reader to darken letters a bit.
Reading in landscape (2 screens per page) shall yield bigger letters than on paper, more convenient for those small letters in graphs/tables.

Some "crazy" ideas:

Those on (multitasking) M92 if they wanted, could open two pdfs simultaneously, I guess, and read textual parts in pdf with perfectly white background, jumping to the other pdf for better preserved tables and graphs.

Those who don't have multitasking on their e-ink or maybe do have it but would like the best of the two worlds in the very same pdf (i.e. as white background and as dark letters as it gets and the best preserved tables as possible) could simply merge those two pdfs together e.g. pages with the same number placed next to each other, 1,1,2,2...90,90... or one complete pdf behind another and then use goto-page for jumping to the table on twin page (by always adding the number of pages contained in the second pdf to the current page) by making the bookmarks beforehand or just by paginating it in a few seconds e.g. those in second pdf in Arabic numerals and those in first in Roman or not numbered at all.

To get some idea what do the original and optimized pages (using just gamma 1.9) look like here are thumbnails of similar pdf:




So, to summarize it for those who might encounter similar problems with colored backgrounds and graphs/tables, we can adjust such pdfs using k2pdfopt's commands -wt, -g and -cmax (white threshold, gamma and contrast), as alone or in combination with each other.

-cmax <max> Set max contrast increase on source pages. 1.0 keeps
contrast from being adjusted. Use a negative value to
specify a fixed contrast adjustment. Default = 2.0.

-g <gamma> Set gamma value of output bitmaps. A value less than 1.0
makes the page darker and may make the font more readable.
Default is 0.5.

-wt[+] <thresh> Any pixels whiter than <thresh> (0-255) are treated as
"white". Setting this lower can help k2pdfopt better process
some poorly-quality scanned pages or pages with watermarks.
Note that the pixels which are above this threshold value and
therefore are treated as white are not actually changed to
pure white (255) unless the '+' is also included. Otherwise,
this only sets a threshold.
The default value is -1, which tells k2pdfopt to pick the
optimum value. See also -cmax.

[japinder]

Thanks guys. Just playing around with the options & testing. Will be back with feedback & questions.

[japinder]

Hi Will/Mark,

kpdfopt does a remarkable job to convert to landscape mode & never breaks the text into half, diagrams are a different kettle of fish though.

How do I convert a page into exactly two halves, preserving the text if it falls on the break boundary & if needed, a bit overlap, among the parts. Can this option be combined with other options like crop (cbox) etc?


Thanks,
Jp

[willus]

Originally Posted by japinder

Hi Will/Mark,

kpdfopt does a remarkable job to convert to landscape mode & never breaks the text into half, diagrams are a different kettle of fish though.

How do I convert a page into exactly two halves, preserving the text if it falls on the break boundary & if needed, a bit overlap, among the parts. Can this option be combined with other options like crop (cbox) etc?

If the diagrams have a box around them or are otherwise contiguous, you can adjust whether k2pdfopt will break them by adjusting the -gtr option. But if the diagrams have clean breaks in them, then yes, there is no fuzzy logic in k2pdfopt to try to tell lines of text from diagrams by some other method, so it may break them across pages.

You can use the -grid option to break pages in half with some overlap, e.g.

k2pdfopt -grid 1x2x5 ...

... breaks the source file into two halves (1 column x 2 rows) with 5% overlap. This is virtually identical to this:

k2pdfopt -cbox 0,0,1s,.525s -cbox 0,.475s,1s,.525s ...

The above method makes two crop-boxes on each source page. The "s" qualifier means the values are fractions of the source page size.

Neither of these methods guarantee a clean break, i.e. they don't try to avoid cutting a line of text in half. If you want that, then you should use the fit-width mode:

k2pdfopt -mode fw ...

It won't guarantee to cut each page exactly in two because it does look for clean line breaks, but if you size your device settings correctly (e.g. set your device width to half the source page height and set your device height to the source page width using -w and -h), you can get approximately two output pages for each source page.

You can get more detail on all these options at the help page (or select Help | Command-line Options... from the menus in the MS Windows GUI).

[japinder]

Originally Posted by willus

If the diagrams have a box around them or are otherwise contiguous, you can adjust whether k2pdfopt will break them by adjusting the -gtr option. But if the diagrams have clean breaks in them, then yes, there is no fuzzy logic in k2pdfopt to try to tell lines of text from diagrams by some other method, so it may break them across pages.

You can use the -grid option to break pages in half with some overlap, e.g.

k2pdfopt -grid 1x2x5 ...

... breaks the source file into two halves (1 column x 2 rows) with 5% overlap. This is virtually identical to this:

k2pdfopt -cbox 0,0,1s,.525s -cbox 0,.475s,1s,.525s ...

The above method makes two crop-boxes on each source page. The "s" qualifier means the values are fractions of the source page size.

Neither of these methods guarantee a clean break, i.e. they don't try to avoid cutting a line of text in half. If you want that, then you should use the fit-width mode:

k2pdfopt -mode fw ...

It won't guarantee to cut each page exactly in two because it does look for clean line breaks, but if you size your device settings correctly (e.g. set your device width to half the source page height and set your device height to the source page width using -w and -h), you can get approximately two output pages for each source page.

You can get more detail on all these options at the help page (or select Help | Command-line Options... from the menus in the MS Windows GUI).

Thanks Will.

The 'grid' option, as well as 'cbox' (with s qualifier), are not playing well with the -rt 90 option to rotate the output to landscape mode. The text lines seem to get cut-off & redistributed among adjacent pages. If I remove the -rt 90 option to rotate to landscape, hoping to do it in another pass, the output is somewhat squarish, with the same length as well as width, not a good candidate for landscape, will need another split, or will leave huge margins on the side when rotated.

Sending you the original (already cropped in briss), as well as the output files via PM. Apologies for the trouble in advance.

EDIT: I'm pretty happy with the default fit width conversion with k2pdfopt. The split into two pages exactly is just something extra to have, so that I can be aware of the page boundaries, since I've already removed the margins including page numbers in briss.

I also saw the option -bp & instead of fit width, tried to specify '-bp -f2p 2', '-bp -f2p 2 -col 1', along with '-rt 90' but its creating even more issues (horizontal text split across adjacent pages, or only rotation with the original page being moved to the right in landscape keeping the left half empty) rather than solving the existing ones.

EDIT 2: PM sent.


Jp