k2pdfopt: optimizes PDFs for viewing on e-readers

[loviedovie]

Originally Posted by willus

I have added a link to this post on my contrib page under the Wine section.

Good idea.

As far as I can tell, the idea works. I am able to select a pdf file in Calibre and run it by using open with plugin. The file is saved in the same folder as the original pdf.

[loviedovie]

Hi

How do I avoid page numbers, reperating chapter names (at the top or at the bottom) mingling with reflow? it makes it confusing to read when they mixup with the actual page reflow.

thanks

[willus]

Originally Posted by loviedovie

Hi

How do I avoid page numbers, reperating chapter names (at the top or at the bottom) mingling with reflow? it makes it confusing to read when they mixup with the actual page reflow.

thanks

If using the MS Windows GUI, use the crop regions--select a crop region and only include the main body of the page in the crop region. If command-line, you can use the -m option (margins), e.g.

k2pdfopt -m 0.5,1.0,0.75,0.6 ...

... will crop out the left 0.5 inches, the top 1 inch, the right 0.75 inches, and the bottom 0.6 inches from the processed area of the PDF. The -cbox option is similar.

[loviedovie]

willus thanks for the reply.

I actually tried that but the issue was a usability thing for me, because I am not sure if I was getting the dimensions right.

The other issue is that sometimes the placement of those can be minus plus with scanned pdfs, especially if the pdfs are scanned with a camera device so putting hard coded numbers will most likely cut off usable parts as well.

The last issue related to it is that I personally like to have them as part of the page just in proper place, not part of the last line.

I am not sure if there is a way to achieve this maybe K2Pdf needs some kind of pattern recognition and training mode

In anycase I like this software as a desktop tool and as it being part of the Koreader.

thanks

[willus]

Originally Posted by loviedovie

willus thanks for the reply.

I actually tried that but the issue was a usability thing for me, because I am not sure if I was getting the dimensions right.

The other issue is that sometimes the placement of those can be minus plus with scanned pdfs, especially if the pdfs are scanned with a camera device so putting hard coded numbers will most likely cut off usable parts as well.

There is an app called ScanTailor that helps you clean up scanned PDFs. Or you could do a first pass where you simply have k2pdfopt crop and autostraighten your doc:

k2pdfopt -ac -as -mode trim -n source.pdf -o source_cropped.pdf

...and then use the technique I described in the previous post.

Originally Posted by loviedovie

The last issue related to it is that I personally like to have them as part of the page just in proper place, not part of the last line.

I am not sure if there is a way to achieve this maybe K2Pdf needs some kind of pattern recognition and training mode

Don't expect a feature like this from me anytime soon. The heuristics involved with auto-detecting a page number / header / footer are likely fraught with false positives and false negatives. The goal of k2pdfopt is to make your document comfortably readable and intelligible on an e-reader within reason, but not necessarily to achieve perfection.

Originally Posted by loviedovie

In anycase I like this software as a desktop tool and as it being part of the Koreader.

Thank you.

[loviedovie]

@willus

Thanks for all the info. I will checkout scantailor.

Do you have plans for multitheading? Especially for the preview window, sometimes it is a bit slow.

thanks

[willus]

Originally Posted by loviedovie

Do you have plans for multitheading? Especially for the preview window, sometimes it is a bit slow.

I do have plans for multithreading, but not for the preview window at the moment, though I suppose it may get impacted. I haven't entirely thought it through. The multithreading plan is to break the source file into N sections, process each section in parallel, and then join the output together with a separate tool like jpdftweak or pdfsam (MuPDF doesn't have an API for joining PDF files yet). Or, if multiple docs are selected for conversion, to simply convert each of them in parallel (even easier). The issue with the preview window, if you preview a high page number, is that the entire conversion has to be done up to that preview page number each time. There's not really an easy way to get around that. To multithread that, I'd have to parallelize at a micro-level, which isn't as trivial, rather than the macro-level I described above.

You can always put a limited source page range into the GUI's "Pages to Convert" box to preview how a specific source page (or set of pages) will convert.

[loviedovie]

@willus

thanks for the heads up. I wish that you continue developing this tool. It is a tremendous help in Koreader.

[ephestione]

Originally Posted by willus

The multithreading plan is to break the source file into N sections, process each section in parallel, and then join the output together with a separate tool like jpdftweak or pdfsam

A further suggestion regarding that: you may want to leave the "join together" part as an optional checkbox function, as I found that the joined version of the manually multithreaded conversion I did performed really bad on my paperwhite 3, as it was painfully slow to manage, both with turning pages and selecting text (but then again the pdf I'm talking about is 1000+ pages originally, go figure how big it became after conversion).
Now that on my kindle I put the 8 smaller pdf's, I can turn pages instantaneously and highlight text pretty much as fast as I can do that for proper ebooks, so leaving out separate pdf's for bigger documents is the way to go.

[willus]

Originally Posted by ephestione

A further suggestion regarding that: you may want to leave the "join together" part as an optional checkbox function, as I found that the joined version of the manually multithreaded conversion I did performed really bad on my paperwhite 3, as it was painfully slow to manage, both with turning pages and selecting text (but then again the pdf I'm talking about is 1000+ pages originally, go figure how big it became after conversion).
Now that on my kindle I put the 8 smaller pdf's, I can turn pages instantaneously and highlight text pretty much as fast as I can do that for proper ebooks, so leaving out separate pdf's for bigger documents is the way to go.

Interesting. Do you know if it is the joining technique or just the sheer size of the PDF that reduces performance? You used pdfsam to join the files? Could you try jpdftweak also? Or could you try an overnight conversion that does it all in one piece (though as I recall this segfaulted k2pfdopt--so maybe just half of it?) to see how that compares? What would be best would be an A-B comparison between something produced only with k2pdfopt vs. something put together with various joining tools.

[capink]

Originally Posted by kleinjar

Hey Willus, thank you so very much for your k2pdfopt. Reading pdf's on my ereader is so much better now.


So: when using k2pdfopt with wrapping, I wish there were a way to directly control the size of text (like setting a median text height, which will scale everything proportionately). -odpi controls text size indirectly but text size can get inconsistent across files of course. Whereas i'm most interested in batch-converting and getting the text to a consistent not-too-large, not-too-small size on the ereader.

(or is this already possible?)

I second that. Batch conversion is not working for me as I have to test each file by converting one page while fiddling with the odpi setting untill it produces acceptable result. A feature like this would make batch conversion much easier.

[willus]

Originally Posted by capink

I second that. Batch conversion is not working for me as I have to test each file by converting one page while fiddling with the odpi setting untill it produces acceptable result. A feature like this would make batch conversion much easier.

While I do plan to add this feature eventually, I am a bit puzzled. I would think that most PDF documents would use a relatively standard font size, somewhere between 10 and 12 points, so that if you use a consistent odpi, you would get a consistent output font size. What is the font size range in your source PDFs? Can you (or kleinjar) post an example of two extremes (source PDFs)?

[capink]

My books are in Arabic. I don't know if they will be of any benefit to you. I attached them to this post anyway. The first attachment is for a pdf with standard font size which is converted fine without the need to play with the odpi setting. The second attachment is for a smaller font which needs an odpi setting of 250 to be readable.

[thatworkshop]

Originally Posted by capink

Weirdzzz

On my DXG, small.pdf opens perfectly fine with built-in PDF viewer. Weirdly enough, standard.pdf gives an error saying it has embedded features not yet supported by Kindle!

Both open nicely in KOreader and KindlePDFViewer.

[willus]

Originally Posted by capink

...The first attachment is for a pdf with standard font size which is converted fine without the need to play with the odpi setting. The second attachment is for a smaller font which needs an odpi setting of 250 to be readable.

Okay, thanks. The font sizes are approximately 9-point and 18-point in your two examples.