k2pdfopt: optimizes PDFs for viewing on e-readers

[ephestione]

Hi willus
I am using latest version in x64 compile on an i7 laptop, and since I am converting a 1100+ pages PDF (first time aroung it crashed slightly after half file, but maybe it was something else than k2pdfopt itself and I am on my second -slow- pass, will let you know how it goes), I found myself whishing there was multithreading, as my octacore is quite underloaded at about 18%, for a single core output.
Found a GUI which did just that, but its website is not updated since a couple of years, so I did not dare to try it right off the bat, would it be feasible to implement on k2pdfopt itself? Ideally in the executable or even as a workaroung in the builtin GUI by processing multiple pages intervals and then merging at the end?

[willus]

Originally Posted by ephestione

Hi willus
I am using latest version in x64 compile on an i7 laptop, and since I am converting a 1100+ pages PDF (first time aroung it crashed slightly after half file, but maybe it was something else than k2pdfopt itself and I am on my second -slow- pass, will let you know how it goes), I found myself whishing there was multithreading, as my octacore is quite underloaded at about 18%, for a single core output.
Found a GUI which did just that, but its website is not updated since a couple of years, so I did not dare to try it right off the bat, would it be feasible to implement on k2pdfopt itself? Ideally in the executable or even as a workaroung in the builtin GUI by processing multiple pages intervals and then merging at the end?

I believe the GUI you were looking at (Jens Wallauer's?) only does multitasking by running multiple instances of k2pdfopt at the same time, on different source files. Multithreaded operation has been requested before, but I don't really have the time or energy to work on that right now. I'm just trying to chase through my list of bugs and reasonably-easy-to-implement feature requests.

All I can advise is that you do it manually--it's not very hard to write a batch script that launches multiple instances of k2pdfopt, each converting 100 pages at a time (or whatever you want), and then joins the outputs together using PDFSAM or jpdftweak. E.g.

start k2pdfopt -gui- -ui- -x -o out01.pdf srcfile.pdf -p 1-100
start k2pdfopt -gui- -ui- -x -o out02.pdf srcfile.pdf -p 101-200
start k2pdfopt -gui- -ui- -x -o out03.pdf srcfile.pdf -p 201-300
start k2pdfopt -gui- -ui- -x -o out04.pdf srcfile.pdf -p 301-400
...
rem When all conversions are done:
jpdftweak -i out01.pdf -i out02.pdf -i out03.pdf ... -o finaloutput.pdf
del out01.pdf out02.pdf out03.pdf ...

I will think about maybe automating such a script--but it will require that you have jpdftweak installed.

[ephestione]

Yes, I realized that GUI did exactly that, doing the same on yours was the least effort workaround idea, but that was a lazy request on my part given I could very well doit myself as you said.
Regarding this 1100+ ages PDF I just fed to k2pdfopt, this time it reported the conversion to have gone ok, but the output file was less than 16mb against an original 200mb, and couldn't be opened by sumatra... so I guess something went wrong even if the gui didn't realize it. Maybe 1000+ pages is just too much, and multithreading it over several instances with lesser amount would be the only viable option.

[willus]

Originally Posted by ephestione

Yes, I realized that GUI did exactly that, doing the same on yours was the least effort workaround idea, but that was a lazy request on my part given I could very well doit myself as you said.
Regarding this 1100+ ages PDF I just fed to k2pdfopt, this time it reported the conversion to have gone ok, but the output file was less than 16mb against an original 200mb, and couldn't be opened by sumatra... so I guess something went wrong even if the gui didn't realize it. Maybe 1000+ pages is just too much, and multithreading it over several instances with lesser amount would be the only viable option.

If you want to PM me a link to the file, I'll take a look, otherwise it's hard to tell. My recommendation, as you guessed, would be to piece-meal it, converting part of it at a time.

[ephestione]

It's awful kind of you to be willing to test a 200+MB pdf file to check for errors, seriously, but then it wouldn't be using your time wisely as given the size of the document the only decent way of converting it, even if it wasn't for k2pdfopt crashing by the size, is to multi-instance, as I just did. Also the command line interface (following your syntax) seemed faster than what the gui achieved. Converted the document in 8 separate files in a whoosh, now to merge them and test on kindle
Thank you!

EDIT: on a side note, an automated script would be best in that it will be able to evenly distribute the pages number between instances, and automatically merge them with almost no intervention for the user, but that's only if you really have spare time for that; I understand those constraints, being a hobbist coder myself.

EDIT 2: I tried jpdftweak, and either it's not working for me, or it is so counterintuitive I am not doing the correct thing, but it's been working for a long while now and I have still no output (had to increase java maximum useable ram as well).
PDFSAM tho, is definitely straightforward and in a nice graphical interface told me the percentage of processing and had the merged file ready in a short while, I am reading on my kindle now.

[willus]

Originally Posted by ephestione

It's awful kind of you to be willing to test a 200+MB pdf file to check for errors...

...PDFSAM tho, is definitely straightforward and in a nice graphical interface told me the percentage of processing and had the merged file ready in a short while, I am reading on my kindle now.

Thank you for the feedback. I will consider PDFSAM.

[jwbales]

The download page for k2pdfopt appears to be broken. When one enters the captcha and clicks on the download link it takes you, after about 10 seconds, to a page that says "Download Expired." Evidently one has only microseconds before the captcha expires. Is there any other place where k2pdfopt can be downloaded? I need the latest 32 bit linux version. I am a retired mathematics professor and have hundreds of pdf math study guides, exercise sets, exams + keys, etc that were originally formatted for letter size paper and would like to convert them to a smaller format for ebooks and make them available for free for undergraduate students. It appears that k2pdfopt may be exactly what I am looking for if only I could obtain a copy.

[willus]

Originally Posted by jwbales

The download page for k2pdfopt appears to be broken. When one enters the captcha and clicks on the download link it takes you, after about 10 seconds, to a page that says "Download Expired." Evidently one has only microseconds before the captcha expires. Is there any other place where k2pdfopt can be downloaded? I need the latest 32 bit linux version. I am a retired mathematics professor and have hundreds of pdf math study guides, exercise sets, exams + keys, etc that were originally formatted for letter size paper and would like to convert them to a smaller format for ebooks and make them available for free for undergraduate students. It appears that k2pdfopt may be exactly what I am looking for if only I could obtain a copy.

Welcome to MR. I have PM'd you a link to the linux32 version that doesn't require a capcha code, though I just tried my site from my Windows PC using Firefox and it worked fine--I downloaded several different binaries without any issues. What OS/browser are you using to attempt the download?

Hi! I am using k2pdfopt on Mac OS X (Yosemite). I am trying to convert an eBook I bought so I can read it from my Kindle (Voyage). Unfortunately, I keep getting "Segmentation fault: 11". What settings should I try altering?

[willus]

Originally Posted by alex2407

Hi! I am using k2pdfopt on Mac OS X (Yosemite). I am trying to convert an eBook I bought so I can read it from my Kindle (Voyage). Unfortunately, I keep getting "Segmentation fault: 11". What settings should I try altering?

Are you sure it's a PDF ebook? Most ebooks that you buy are not PDF. If it is a PDF file, can you post a link to it or PM me a link? And can you post a screenshot of your conversion that shows your settings? That would help.

Originally Posted by willus

Are you sure it's a PDF ebook? Most ebooks that you buy are not PDF. If it is a PDF file, can you post a link to it or PM me a link? And can you post a screenshot of your conversion that shows your settings? That would help.

I have sent you a PM with link to the eBook. I didn't save the settings unfortunately, but I can briefly tell you what I used.

device -> amazon voyage
co -> 2 columns, everything else to default

Edit: I have a weird suspicious that it is simply a scanned text book and not an actual digital document :S

[willus]

Originally Posted by alex2407

I have sent you a PM with link to the eBook. I didn't save the settings unfortunately, but I can briefly tell you what I used.

device -> amazon voyage
co -> 2 columns, everything else to default

Edit: I have a weird suspicious that it is simply a scanned text book and not an actual digital document :S

Thanks for posting the link. That helps. Several comments:
1. It's a very large book--900 pages, so you may want to try converting a subset of it--a couple chapters at a time or something, or an even smaller number to start out with to see that you are getting a conversion you like before you convert the whole thing. The sheer size may be causing the seg fault, or just one particular page may be causing it, and you should then let me know which page that is.
2. It's not scanned. You can tell just by zooming way in on the text. It doesn't pixelate even at extreme magnifications.
3. Try turning on the "show marked" option: Add -sm to the command options. Then look at the ..._marked.pdf to see how k2pdfopt is parsing your pages.
4. This is a tough PDF to convert--the formatting varies quite a bit page to page. K2pdfopt really started as something to convert journal papers which have consistent formatting throughout. I've tried to adapt it to other types of PDFs as much as possible, but there will likely be some pages that don't convert well.
5. Most pages have notes in the margins, so you should use the "notes" options by adding these to the command-line options:
-nle .2,.4 -nlo .6,.8

I recommend starting with something like this:

k2pdfopt -nle .2,.4 -nro .6,.8 -mb 1 -dev kv psych.pdf -p 100-120 -sm

(page numbers from 100 to 120 optional--that was my test range. Take out -sm when you're satisfied that the conversion is acceptable--it will speed things up to remove it.) The -mb 1 will cut off the page numbers at the bottom of every page, which helps k2pdfopt analyze the rest of the page better. You might also start without setting your device to the Voyage since the high resolution of the Voyage makes everything take longer.

Originally Posted by willus

Thanks for posting the link. That helps. Several comments:
1. It's a very large book--900 pages, so you may want to try converting a subset of it--a couple chapters at a time or something, or an even smaller number to start out with to see that you are getting a conversion you like before you convert the whole thing. The sheer size may be causing the seg fault, or just one particular page may be causing it, and you should then let me know which page that is.
2. It's not scanned. You can tell just by zooming way in on the text. It doesn't pixelate even at extreme magnifications.
3. Try turning on the "show marked" option: Add -sm to the command options. Then look at the ..._marked.pdf to see how k2pdfopt is parsing your pages.
4. This is a tough PDF to convert--the formatting varies quite a bit page to page. K2pdfopt really started as something to convert journal papers which have consistent formatting throughout. I've tried to adapt it to other types of PDFs as much as possible, but there will likely be some pages that don't convert well.
5. Most pages have notes in the margins, so you should use the "notes" options by adding these to the command-line options:
-nle .2,.4 -nlo .6,.8

I recommend starting with something like this:

k2pdfopt -nle .2,.4 -nro .6,.8 -mb 1 -dev kv psych.pdf -p 100-120 -sm

(page numbers from 100 to 120 optional--that was my test range. Take out -sm when you're satisfied that the conversion is acceptable--it will speed things up to remove it.) The -mb 1 will cut off the page numbers at the bottom of every page, which helps k2pdfopt analyze the rest of the page better. You might also start without setting your device to the Voyage since the high resolution of the Voyage makes everything take longer.

Thank you. I will try these settings and will see how it will go.

[kleinjar]

Hey Willus, thank you so very much for your k2pdfopt. Reading pdf's on my ereader is so much better now.

A little wish: it would be great, when batch-converting many PDFs with wrapping, to have a way to get all the output to have about the same-sized body text.

So: when using k2pdfopt with wrapping, I wish there were a way to directly control the size of text (like setting a median text height, which will scale everything proportionately). -odpi controls text size indirectly but text size can get inconsistent across files of course. Whereas i'm most interested in batch-converting and getting the text to a consistent not-too-large, not-too-small size on the ereader.

(or is this already possible?)

Sorry for asking two questions at the same time (but gotta read the chapters for the lectures =/ ). So here's my question. I cannot enlarge the text on another PDF book (PMed) the link. I tried -odpi from 150 to 560 but the text size remains the same. What gives?