k2pdfopt: optimizes PDFs for viewing on e-readers

[willus]

Originally Posted by fullspeedin2sun

The options I'm using are in the attached screencap. I was hoping to make this PDF easier to read on my Kindle Paperwhite 3 without making it much bigger as it looks like it was natively created.

Also, is there an option to add margins in the GUI?

You really need to turn off native mode output and allow text wrapping to be able to magnify the text. Just try it out. I think you'll like the result (is file size really more important to you than readability?). To add quarter inch margins, put this in the "Additional Options" field:

-om 0.25

P.S. You can convert a few pages at a time (e.g. A chapter or two) to keep the converted files to reasonable sizes if that helps.

[fullspeedin2sun]

Originally Posted by willus

You really need to turn off native mode output and allow text wrapping to be able to magnify the text. Just try it out. I think you'll like the result (is file size really more important to you than readability?). To add quarter inch margins, put this in the "Additional Options" field:

-om 0.25

P.S. You can convert a few pages at a time (e.g. A chapter or two) to keep the converted files to reasonable sizes if that helps.

Ultimately, I need an easier time reading this than I need to preserve space on the Kindle, so I will do this. Thank you for your help, this program is a fantastically useful piece of software.

When you convert a PDF in smaller sets, do they add up to a total smaller size? As in, I could convert them in smaller packets and combine them at the end?

[willus]

Originally Posted by fullspeedin2sun

[...] When you convert a PDF in smaller sets, do they add up to a total smaller size? As in, I could convert them in smaller packets and combine them at the end?

No such luck. The total size will still end up the same. See this page for tips on how to reduce file size for bitmapped output.

[fullspeedin2sun]

Originally Posted by willus

No such luck. The total size will still end up the same. See this page for tips on how to reduce file size for bitmapped output.

I had some cool but curious results when I played with those options. When I used the default settings for Kindle 1-5 this same file was 45mb, when I used the default settings for a Kindle Voyage/Paperwhite 3 the file is 78mb, and when I modified the settings to alter the bpc, the jpg quality, and the dithering [as well changing the dpi a little, screencap of settings attached], the file was 65mb.

Why is there such a difference in sizes between the Kindle 1-5 and the Voyage? Shouldn't the file size be smaller when I change the settings, maybe even a quarter of the size like the page on bpc and jpg quality management said?

[willus]

Originally Posted by fullspeedin2sun

I had some cool but curious results when I played with those options. When I used the default settings for Kindle 1-5 this same file was 45mb, when I used the default settings for a Kindle Voyage/Paperwhite 3 the file is 78mb, and when I modified the settings to alter the bpc, the jpg quality, and the dithering [as well changing the dpi a little, screencap of settings attached], the file was 65mb.

Why is there such a difference in sizes between the Kindle 1-5 and the Voyage? Shouldn't the file size be smaller when I change the settings, maybe even a quarter of the size like the page on bpc and jpg quality management said?

The Kindle Voyage has more pixels (300 dpi) than the Kindle 1-5 (167 dpi), so each bitmapped page saved by k2pdfopt for the Voyage has a much higher pixel count.

[fullspeedin2sun]

Originally Posted by willus

The Kindle Voyage has more pixels (300 dpi) than the Kindle 1-5 (167 dpi), so each bitmapped page saved by k2pdfopt for the Voyage has a much higher pixel count.

Makes sense. So I'm using the optimization options correctly and I've hit the limit on how much I can shrink this pdf?

[willus]

Originally Posted by fullspeedin2sun

Makes sense. So I'm using the optimization options correctly and I've hit the limit on how much I can shrink this pdf?

A couple things:

1. Your use of the -bpc option needs a space between "-bpc" and the value, e.g. "-bpc 1" instead of "-bpc1".

2. The -bpc option is not compatible with the -jpg option. If you use the -jpg option to save the bitmaps as JPEG, the -bpc option is ignored. So you need to pick one or the other. I need to make this more clear in the documentation.

A JPEG quality factor of 10 seems pretty low, but as long as you find it tolerable, then that's fine.

[timofonic]

@willus

Is it possible to user other OCR engines? Even commercial ones.

I tried to find a repository where k2pdfopt is stored, but it was impossible to find.

Do you use some source code versioning control system? It would be ideal to have it in GitHub.

Here's an outdated mirror, but obviously all the history is lost.
https://github.com/JohannesBuchner/k2pdfopt

Are you planning to update your software to latest Tesseract?


3.04.00 released on Jul 11, 2015
https://github.com/tesseract-ocr/tes...901f361ecd7e90

Here's the langdata
https://github.com/tesseract-ocr/lan...f3bf238ee8903d

Latest MuPDF is 1.8.1-ios from 15 days ago, I'm not sure if you updated it:
https://github.com/ArtifexSoftware/m...61f3815a92a375

Latest Leptonica is 1.7.2

Here's a GitHub unofficial mirror:
https://github.com/egorpugin/leptonica
Version notes: http://leptonica.com/source/version-notes.html

Official version isn't hosted in a repository, the code is here:
http://leptonica.com/source/leptonica-1.72.tar.gz


I wrote this because KOreader project got unable to use newer Tesseract and Leptonica, you need to update it in your project:
https://github.com/koreader/koreader-base/issues/361

They seem to use a wrapper around k2pdfopt to make it a library, or something like that:
https://github.com/koreader/libk2pdfopt


NOTE: I'm not part of KOreader Team and not a developer at all. Just an user of the software.


Here's an historical reference why using GitHub would make other projects to deploy k2pdfopt a lot easier, plus potential contributions from other developers.

Originally Posted by chrox

Yes. The koreader team would always like to keep up with the latest k2pdfopt code. On each k2pdfopt version release we will make a fresh codebase override in libk2pdfopt with the lattest k2pdfopt source. The outcome commit in libk2pdfopt should be something like "fresh k2pdfopt v2.12"(d988c98ae7ec889fecc0bc8b5cae051f7b573988). Then we use this branch to merge with our latest branch. Conflicts, if there will have, will be fixed in the immediate commit like "update k2pdfopt to version v2.12"(e36230b59ba92e9da5ed261253acd805eba5baa9). So patches will be obtained with a command:

git clone https://github.com/koreader/libk2pdfopt.git
cd libk2pdfopt
git diff d988c98ae7ec889fecc0bc8b5cae051f7b573988 k2pdfoptlib/

It will diff all changes in k2pdfoptlib since the fresh k2pdfopt v2.12 source.

[willus]

Originally Posted by timofonic

Is it possible to user other OCR engines? Even commercial ones.

I have no idea how I'd integrate commercial source into a GPL project like k2pdfopt. Tesseract is the best open source OCR library I know of.

Originally Posted by timofonic

I tried to find a repository where k2pdfopt is stored, but it was impossible to find.

My official releases are always on my web site.

Originally Posted by timofonic

Do you use some source code versioning control system? It would be ideal to have it in GitHub.

I have very limited time to spend on k2pdfopt. I prefer to use it adding features / fixing bugs rather than learning how to use github.

Originally Posted by timofonic

Are you planning to update your software to latest Tesseract?

Tesseract v3.04.00 now depends on the ICU library, which has hundreds of source files. I'll have to wade through that at some point and see if it is worth the effort.

Originally Posted by timofonic

Latest MuPDF is 1.8.1-ios from 15 days ago...

I'll update to the latest MuPDF on my next k2pdfopt bug-fix release, but I'm not sure when that will be.

Originally Posted by timofonic

I wrote this because KOreader project got unable to use newer Tesseract and Leptonica, you need to update it in your project...

It used to be that k2pdfopt was compiled for KOreader with the MuPDF and Tesseract dependencies turned off (i.e. with macros set so that MuPDF and Tesseract library functions were not used). I'm not sure when that all changed, but apparently it has. Either way, it looks like the only reason KOreader uses Tesseract or Leptonica is so that it can compile them into the k2pdfopt library, so I'm not sure why you find it so urgent to update to bleeding edge versions of MuPDF and Tesseract. Is some other part of the KOreader source making use of the newer versions of MuPDF / Tesseract independently of the k2pdfopt source code?

Originally Posted by timofonic

Here's an historical reference why using GitHub would make other projects to deploy k2pdfopt a lot easier, plus potential contributions from other developers.

It probably wouldn't be that hard for some clever person, perhaps yourself, to write a script which would pull down my latest official release zip file and dump it to a github release.

[eschwartz]

It has occurred to me before, the idea of hosting k2pdfopt on GitHub, but I was loath to suggest it for precisely that reason -- that if you weren't using it already, it's probably because you aren't familiar with VCS.


Really, uploading release snapshots for each commit doesn't do a lot... the main reason people use VCS is to track the evolution of software line by line and change by change, not simply version by version. Seeing not just how stuff changed, but why it changed (commit messages). Experimenting with changes and rolling them back, mixing and matching (the branching model). Stepping through the commits to see where "x" went wrong (git bisect).

Running a `diff` on consecutive releases is not significantly different from committing each release tarball.

If the dev isn't familiar enough to do any of that, there is little point, I guess.
(And I don't blame you for not having the time to spare.)


It doesn't even lower the barrier to entry for contributing. (Easy and familiar to open a pull request on GitHub, but who knows the best way to contribute to a tarball?)

That requires the repo be maintained by the dev.

[willus]

Originally Posted by eschwartz

It has occurred to me before, the idea of hosting k2pdfopt on GitHub, but I was loath to suggest it for precisely that reason -- that if you weren't using it already, it's probably because you aren't familiar with VCS....

I don't know if I'd quite phrase it that way (that I'm not familiar with VCS). I know what VCS is, and I've learned enough about git to install it on my PC and try it out, but my old habits die hard. I am admittedly set in my ways and very comfortable and efficient with my own brand of primitive/manual VCS, which mostly involves zip to archive snapshots of code. I've also never had to work closely with a lot of other people on a large software project, which is where VCS is truly a necessity.

[timofonic]

@willus


No problem! Keep up your great work!

Is this a cowboy project? If so, I hope someday you get some devs to help you. This project is growing a lot!

What about vectorization support? Maybe this could be good enough in hi-res scanned books, maybe even detecting patterns (letters) and make a custom font library.

I failed to find a vectorization library/program at least as good as the old VectorMagic:

- potrace lacks color
* color trace does some trick I fail to understand it.
- Autotrace has color, but seems worse.
- imagetracerjs is a javascript library that vectorices raster images.

[b]Misc[/url]
http://xmailserver.org/davide.html
https://en.wikipedia.org/wiki/Image_tracing
https://en.wikipedia.org/wiki/Compar...rsion_software
http://www.scale-a-vector.de/svg-test4-e.htm#cr2v
http://paperjs.org
https://github.com/schicks/pdarff

[willus]

Originally Posted by willus

Tesseract v3.04.00 now depends on the ICU library, which has hundreds of source files. I'll have to wade through that at some point and see if it is worth the effort.

I was wrong about Tesseract v3.04. It does not have any new dependencies compared to v3.02. It only has the extra dependencies if you want to compile the training tools. It's been so long since I've compiled it that I'd forgotten. I have successfully compiled Tesseract v3.04 into my k2pdfopt main branch (no noticeable change in performance). It will be in the next release. Next up is MuPDF v1.8.

[drjd]

Many thanks willus, for this great tool! Finally I could read that damn three column pdf on my Kobo after conversion.

Keep up your great work!

[gibletpie]

Thanks so much for the awesome program, willus!

I'm having one small problem. I like using this with my Nook Glowlight Plus. Same resolution as the Kindle Voyager. But on the nook, it keeps a header and footer at the top and bottom. Which ends up overlapping some of the PDF text. What I want to do is fit to width, but leave space at the top and bottom to accommodate the header and footer. I've figured out I can leave space for the footer by adjusting the height in the device settings, but I can't figure out how to leave space for the header.

Is this possible?