k2pdfopt: optimizes PDFs for viewing on e-readers

[willus]

Originally Posted by Sajjad76

Hi,Thanks so much for your answer.
The converted PDFs (with native PDf output option) are shown problematic on kindle paperwhite with duokan operating system.
The words are mixed and the device restarts.
On amazon kindle operating system they are showing fine ,but very very slow in turning pages.

How are you turning on native PDF output? Are you using one of the selectable conversion modes (upper-left pull down menu), like "fit width" or "2-column"? That is the best way to select native PDF output. If that still doesn't help, try putting this into the "Additional options" box:

-bp m

[nickq]

I appreciate the welcome, and thanks for the advice!

@willus, thanks! And I sent you a pm.

@markom, thanks! I'll check out the programs. Thought that fiddling with the margin controls in k2 might be enough, so I wouldn't have to edit the files beforehand. But yeah, I have seen worse scans, and it would certainly be useful to have a program with which I can easily edit .pdf files.

@Sajjad76, I noticed that too when I tried converting a .pdf processed via k2 (set "native PDF" via the check-box), then converting with Calibre to .epub. Completely messed up the text. Settled for just reading the .pdf set-up just right for my e-book reader with k2. I realize that the situation is different from that which you described, but I thought that there might be a connection.

Regards,
nick

[willus]

Originally Posted by nickq

... I noticed that too when I tried converting a .pdf processed via k2 (set "native PDF" via the check-box), then converting with Calibre to .epub. Completely messed up the text. Settled for just reading the .pdf set-up just right for my e-book reader with k2. I realize that the situation is different from that which you described, but I thought that there might be a connection.

@nickq -- This likely happens because using native PDF output in k2pdfopt reproduces the entire source page but with cropping directives for each k2 output page so that only certain parts of the source page are shown. Calibre's conversion to .epub is likely completely ignoring these cropping directives and just blindly copying the all of the text from the entire original source page (not just the cropped part), which would result in a lot of repeated text in the .epub file. Does that describe what you see?

[willus]

Originally Posted by nickq

@willus, thanks! And I sent you a pm.

@nickq -- try these settings with the document you PM'd to me:

k2pdfopt -ml 0.32 -mb 0.6 -mt 0.6 -mode 2col -col 4 -bp m -gtc .02 -crgh .1 -sm

The first three options set the crop margins (these can actually be set by page using the new -cbox option, but you don't need that for your specific file--you can also use Briss for this, as markom pointed out).

The -bp m option will start a new output page whenever a new region is put on an output page. This improves text highlighting.

The last two parameters (besides -sm) do most of the magic of getting k2pdfopt to correctly split the three columns. The -gtc .02 gets it to be more tolerant of the non-clean scan, which created some smudges between the columns that were causing problems. The -crgh 0.1 keeps it from breaking up the column continuity.

The -sm option generates the marked up file so you can see how k2pdfopt is parsing the source file. You can leave that off for the final conversion.

If using the GUI, you can put the -bp m -gtc .02 -crgh .1 in the "Additional options" box since you can't set them anywhere else in the GUI.

[nickq]

Wow. Those settings worked great. Thanks!

Is there any particular reason it was in 2-column mode rather than default mode though?

I have to admit that I don't completely understand the reasoning behind all of the settings specified the way they were, but it works, so they're saved under "High Tolerance," and I'll look them over again once I have more time.

@nickq -- This likely happens because using native PDF output in k2pdfopt reproduces the entire source page but with cropping directives for each k2 output page so that only certain parts of the source page are shown. Calibre's conversion to .epub is likely completely ignoring these cropping directives and just blindly copying the all of the text from the entire original source page (not just the cropped part), which would result in a lot of repeated text in the .epub file. Does that describe what you see?

Yes, fragments of text repeated multiple times -- sometimes single words, sometimes fragments of words. Occasionally a whole paragraph, but in no way usable. Is there a way to generate decent .epub files from .pdf's via k2 -> calibre? With "Re-flow Text," Calibre doesn't seem to detect the Text as all and seems to treat all of the pages as images. (bitmaps?)

[willus]

Originally Posted by nickq

Is there any particular reason it was in 2-column mode rather than default mode though?

I have to admit that I don't completely understand the reasoning behind all of the settings specified the way they were, but it works, so they're saved under "High Tolerance," and I'll look them over again once I have more time.

2-column mode uses native output and puts a block of each column onto each output page without re-flowing the text, which is usually the best way to go if you don't need text re-flow--it retains the source file's text, scaleability, and size. My only caution against saving those settings is that the margin crop settings were very specific to that one document. The options that resulted in the "high tolerance" were really the -gtc and -crgh options.

Originally Posted by nickq

Is there a way to generate decent .epub files from .pdf's via k2 -> calibre? With "Re-flow Text," Calibre doesn't seem to detect the Text as all and seems to treat all of the pages as images. (bitmaps?)

I'm sure this depends on the specific type of source document. Are you referring to the one that you sent to me? For that one, I'd try the settings I recommended above, except remove the "-mode 2col". That will put out a bitmap with an invisible text layer (the text being copied from the source file's OCR layer) for each k2pdfopt output page, and only that text will be on the output page (no cropping directive used on the source file), so it should convert better in calibre--no repeated words or paragraphs, though I don't use calibre much, so I can't say for sure.

[nickq]

Originally Posted by willus

For that one, I'd try the settings I recommended above, except remove the "-mode 2col". That will put out a bitmap with an invisible text layer (the text being copied from the source file's OCR layer) for each k2pdfopt output page, and only that text will be on the output page (no cropping directive used on the source file), so it should convert better in calibre--no repeated words or paragraphs, though I don't use calibre much, so I can't say for sure.

That's the thing. I'm assuming that removing "-mode 2col" would result in the default mode with re-flowed text? Converting it via Calibre results in the pages being converted into a series of .png images. Can't re-flow or re-scale the text in my e-reader then; and since that seems to be the only point in converting it to .epub in the first place, it seems rather pointless. Then again, it is very likely due to conversion settings I don't understand yet, or tools I don't have yet, since I just started all of this a couple of weeks ago. Not a big problem, since the best solution -- to use k2 to set the .pdf just right for my screen, and run with that -- is not at all a bad one.

[willus]

Originally Posted by nickq

That's the thing. I'm assuming that removing "-mode 2col" would result in the default mode with re-flowed text? Converting it via Calibre results in the pages being converted into a series of .png images. Can't re-flow or re-scale the text in my e-reader then; and since that seems to be the only point in converting it to .epub in the first place, it seems rather pointless. Then again, it is very likely due to conversion settings I don't understand yet, or tools I don't have yet, since I just started all of this a couple of weeks ago. Not a big problem, since the best solution -- to use k2 to set the .pdf just right for my screen, and run with that -- is not at all a bad one.

You can turn off re-flow if you want with -wrap-. Since the columns are so narrow, it doesn't re-flow your particular document by default, though. By default, k2pdfopt will also include an OCR text layer over the bitmaps (the same OCR text in the original source file), which I was thinking calibre would pick up. If calibre just sees the bitmaps, you can turn them off with -ocrvis t (this will show only the OCR text layer). Or you can just dump the OCR text to a file as UTF-8 text with -ocrout <filename>, but for your document, cutting and pasting from a PDF viewer (I used Sumatra) to Word actually does a much better job of this--it even correctly navigates the columns.

[Sajjad76]

Thanks.I used default mode.I think the problem comes from that.
But there is still a blank space after each page that covers the text and it is unreadable(just in duokan).
in amazon operating system it is just ok.

[willus]

Originally Posted by Sajjad76

Thanks.I used default mode.I think the problem comes from that.
But there is still a blank space after each page that covers the text and it is unreadable(just in duokan).
in amazon operating system it is just ok.

I'm not sure what you mean by a "blank space after each page that covers the text". Maybe you could post some attachments showing what you are talking about (including the source file)? Did you end up using the "2-col" or "fitwidth" modes? Did you try also adding the -bp m in the "Additional Options"?

[iceburn]

Hello everybody, i'm just register but i read lots of pages.Frst of all thank you @willus for this program.But i have some problem while converting to pdf files.I try to 2 coulmns pages but inside some math formulas and character.That is ok no problem but i see vertical line on pages.But it just shown just left sides. that means i don't see right side becosue of i cant find hof to fix midle of the pages.i'm useing defutl mode. itry the change margins right-left-up-or bottom but icant fix it.i will send pm if it possibe.Could you give me options aboute that.sory me english not so good and i cant explane deeply my problem.i hope i can explane.Thnky you so much.Have nice weekand

[willus]

Originally Posted by iceburn

Hello everybody, i'm just register but i read lots of pages.Frst of all thank you @willus for this program.But i have some problem while converting to pdf files.I try to 2 coulmns pages but inside some math formulas and character.That is ok no problem but i see vertical line on pages.But it just shown just left sides. that means i don't see right side becosue of i cant find hof to fix midle of the pages.i'm useing defutl mode. itry the change margins right-left-up-or bottom but icant fix it.i will send pm if it possibe.Could you give me options aboute that.sory me english not so good and i cant explane deeply my problem.i hope i can explane.Thnky you so much.Have nice weekand

@iceburn -- Welcome to MR forums! The documents you sent me (thank you!) have the advantageous features that they are very clean (machine generated) and that the column divide is always right in the middle, so assuming you mostly just care about the 2-column section of the document, this is a perfect use for a new k2pdfopt feature--crop boxes. Put the following in the "Additional options" box:

-cbox 0,0,.49s,1s -cbox .51s,0 -bp m

Also, select "fitwidth" from the "Conversion mode" menu and uncheck "rotate output to landscape".

This sets up two one-column-wide crop boxes on each page so that each column is processed separately and then processes each one in "fitwidth" mode. The "-bp m" will start a new output page after processing each column.

As an alternate conversion technique, you can then uncheck "native mode" and then add -vb -1 to the "Additional options" box. That will remove some of the large gaps in the columns, but it will also prevent the output from being in native mode, so it will be bitmapped.

[iceburn]

@willus Thank you for fast answer.I just did what you say and everything is looks fine. Right and left is perfect. But i realize one thing. Up and bottom freespace is to much on my documents. I know pages has too many empty area between the question (excamples) , i accept them but i don't want to see bottom and up empty area.How can arrange them.I try something before your answer "-dev kpw -evl 1 -wrap- -mb 1.4 -mt 1.4 -ml 0.2 -mr 0.2" and it was look good but i couldn't fix sides.When i read your answer i try your code with bottom 1.30 up 1.30 or difrent number but i cant fix it.What should i decide bottom and up empty area.Witch method is good for them. Thank you so much

[willus]

Originally Posted by iceburn

@willus Thank you for fast answer.I just did what you say and everything is looks fine. Right and left is perfect. But i realize one thing. Up and bottom freespace is to much on my documents. I know pages has too many empty area between the question (excamples) , i accept them but i don't want to see bottom and up empty area.How can arrange them.I try something before your answer "-dev kpw -evl 1 -wrap- -mb 1.4 -mt 1.4 -ml 0.2 -mr 0.2" and it was look good but i couldn't fix sides.When i read your answer i try your code with bottom 1.30 up 1.30 or difrent number but i cant fix it.What should i decide bottom and up empty area.Witch method is good for them. Thank you so much

@iceburn -- If you want to minimize the blank gaps and crop off the top and bottom 1.4 inches:

-dev kpw -cbox 0,1.4in,.49s,8.2in -cbox .51s,1.4in,.49s,8.2in -mode fitwidth -ls- -n- -vb -1

To see more about the -cbox option, see my command-line options help page.

[iceburn]

Thank you so much. i will look and try to learn more.
Take it easy.Have nice work.
Edit:That program realy good. it detect space area and fix it. Thank yo again.