k2pdfopt: optimizes PDFs for viewing on e-readers

[MarjaE]

Sorry.

[xilopaint]

Hello willus!

I have been trying to follow the instructions to build k2pdfopt in macOS, but I'm getting these errors:

$ gcc -Ofast -Wall -m64 -o k2pdfopt.o -c k2pdfopt.cg
clang: error: no such file or directory: 'k2pdfopt.cg'
clang: error: no input files

My intention is trying to build a lightweight version of k2pdfopt for my Alfred workflow. I need only four command line options: "-as", "-mode copy", "-dpi" and "-o".

[willus]

Originally Posted by xilopaint

Hello willus!

I have been trying to follow the instructions to build k2pdfopt in macOS, but I'm getting these errors:



My intention is trying to build a lightweight version of k2pdfopt for my Alfred workflow. I need only four command line options: "-as", "-mode copy", "-dpi" and "-o".

What instructions are you trying to follow? The instructions I include with the source code say this:

Build Steps on OS/X (64-bit, gcc 6.2.0, compiled on OSX 10.12 Sierra)
----------------------------------------------------------------------
1. gcc -Ofast -Wall -m64 -o k2pdfopt.o -c k2pdfopt.c

2. g++ -Ofast -m64 -o k2pdfopt k2pdfopt.o -static-libgcc -static-libstdc++ -lk2pdfopt -lwillus -lgocr -ltesseract -lleptonica -ldjvu -lmupdf -lfreetype -ljbig2 -ljpeglib -lopenjpeg -lpng -lzlib -lpthread

The trick is going to be getting all of the libraries to compile. Honestly, I don't think this is worth your effort--you would have to substantially modify the code to really strip out everything but those options. Why not just use the OSX binary? Is the extra size really a problem?

[xilopaint]

Originally Posted by willus

What instructions are you trying to follow? The instructions I include with the source code say this:

Build Steps on OS/X (64-bit, gcc 6.2.0, compiled on OSX 10.12 Sierra)
----------------------------------------------------------------------
1. gcc -Ofast -Wall -m64 -o k2pdfopt.o -c k2pdfopt.c

2. g++ -Ofast -m64 -o k2pdfopt k2pdfopt.o -static-libgcc -static-libstdc++ -lk2pdfopt -lwillus -lgocr -ltesseract -lleptonica -ldjvu -lmupdf -lfreetype -ljbig2 -ljpeglib -lopenjpeg -lpng -lzlib -lpthread

The trick is going to be getting all of the libraries to compile. Honestly, I don't think this is worth your effort--you would have to substantially modify the code to really strip out everything but those options. Why not just use the OSX binary? Is the extra size really a problem?

Oh, sorry! I didn't realise I had included a letter "g" at the end of the command. Now I'm getting other error:

$ gcc -Ofast -Wall -m64 -o k2pdfopt.o -c k2pdfopt.c
k2pdfopt.c:76:10: fatal error: 'k2pdfopt.h' file not found
#include <k2pdfopt.h>
^~~~~~~~~~~~
1 error generated.

k2pdfopt.h is not in the same path of k2pdfopt.c.

[pgodz]

Hey! I'm trying to preserve the original layout but bitmap the image and the text. Is it possible to disable scaling?

Thanks!

[willus]

Originally Posted by pgodz

Hey! I'm trying to preserve the original layout but bitmap the image and the text. Is it possible to disable scaling?

Thanks!

This is exactly what "-mode copy" is for. Set the output page dpi with -odpi. E.g.
k2pdfopt -mode copy -odpi 300 myfile.pdf
...will bitmap each page at 300 dpi and store in the output file.

[Ramo]

My Kobo Touch C is having problems reading a very big pdf file - 1466 pages - that also has hidden text. I think that just removing the hidden text would solve my problem. I was able to do this once, but only with a subset of the file that I set u to test it but now can not repeat the results.

Could some one help me show me the options to mantaing everything - size, dpi, color etc. - but just get rid of the hidden text?

[willus]

Originally Posted by Ramo

My Kobo Touch C is having problems reading a very big pdf file - 1466 pages - that also has hidden text. I think that just removing the hidden text would solve my problem. I was able to do this once, but only with a subset of the file that I set u to test it but now can not repeat the results.

Could some one help me show me the options to mantaing everything - size, dpi, color etc. - but just get rid of the hidden text?

There's not a simple option to do this--to just strip hidden text--with k2pdfopt. What you have to do is use -mode copy, but you'll want to set your DPI and color depth to try and match what you have now. You can see information about how the PDF you have is encoded with the -i option, e.g.

k2pdfopt -i myfile.pdf

You can then select options to match, e.g.

k2pdfopt -mode copy -odpi 200 -c -g 1 -sh- -cmax 1 -ocr- myfile.pdf

This copies the dimensions, sets bitmap DPI to 200, turn on color output, sets gamma to 1 (no change), turns off sharpening, turns off contrast adjust, and turns off OCR (no hidden layer). You can try just a few pages of conversion by adding: -p 1-10 (convert the first 10 pages only).

[Ramo]

Originally Posted by willus

k2pdfopt -mode copy -odpi 200 -c -g 1 -sh- -cmax 1 -ocr- myfile.pdf

This copies the dimensions, sets bitmap DPI to 200, turn on color output, sets gamma to 1 (no change), turns off sharpening, turns off contrast adjust, and turns off OCR (no hidden layer). You can try just a few pages of conversion by adding: -p 1-10 (convert the first 10 pages only).

Thank You!

It worked!

I don't know if you are still developing the the software, although I can see that you are very active in the forum, but maybe this is a feature worth implementing: deleten the hidden text layer.

In my case the Kobo just couldnt cope with the hidden text. Whenever I moved th image to reposition it I would get the image of the hidden text instead of the actual image layer. It would freeze there and the only way I was able to circunvent it was to put the device to sleep, when waken back it would show me the image like nothing had happend...

Anyways great piece of software! Just the learning curve that is a bit steep.

[DaleDe]

Removing the text layer has its downsides. Search for text will no longer work. All you have is a set of images.

[Ramo]

Originally Posted by DaleDe

Removing the text layer has its downsides. Search for text will no longer work. All you have is a set of images.

I know, but right now it is unreadable for me, it takes 2 minutes to turn each page. And the hidden text layer isn't good at all, it was made through ORC of images set um in Garamond, a lot of it italics and in French. A recepy for disaster.

I might as well get rid of the whole thing. It is just impressing me how hard it has been to find a tool to such a simple job. At least I think it is simple.

[willus]

Originally Posted by Ramo

I know, but rigjt now it is unredable for me, it takes 2 minutes to turn each page. And the hidden text layer isn't good at all, it was made through ORC of images of garamond, a lot of it italics and in French. A recepy for disaster. I might as well get rid of the whole thing. It si just impressing me how hard it has been to find a tool to do the job.

Is there any chance you can post a few pages from your PDF file? I'd like to experiment with it.

[Ramo]

Originally Posted by willus

Is there any chance you can post a few pages from your PDF file? I'd like to experiment with it.

Thank you, willus.
Just sent it via PM.

[willus]

Originally Posted by Ramo

Thank you, willus.
Just sent it via PM.

As I suspected, the images are stored in JPEG 2000 format (you can see this when you use the k2pdfopt -i option), which taxes most PDF readers significantly more than JPEG or PNG. Moreover, they are 600 dpi--very high res. That is probably why your reader does not like displaying the file--not because of the hidden text. The default k2pdfopt output is PNG ("Flate"), which is much faster to display, but, as you noted, balloons the file size considerably depending on your chosen resolution and color depth. You might try leaving OCR selected (-ocr m) rather than disabling it. I'll bet it will still work fine and you'll then be able to search the document.

There is not a trivial way to simply remove hidden text from a PDF and leave everything else exactly the way it is. I could maybe make it easier to use the method I showed you with a single command-line option to try to intelligently choose the parameters, but in terms of leaving all of the bitmaps in exactly their original format (highly compressed JPEG 2000), I don't have a way to do that.

[Ramo]

Originally Posted by willus

As I suspected, the images are stored in JPEG 2000 format (you can see this when you use the k2pdfopt -i option), which taxes most PDF readers significantly more than JPEG or PNG. Moreover, they are 600 dpi--very high res. That is probably why your reader does not like displaying the file--not because of the hidden text. The default k2pdfopt output is PNG ("Flate"), which is much faster to display, but, as you noted, balloons the file size considerably depending on your chosen resolution and color depth. You might try leaving OCR selected (-ocr m) rather than disabling it. I'll bet it will still work fine and you'll then be able to search the document.

There is not a trivial way to simply remove hidden text from a PDF and leave everything else exactly the way it is. I could maybe make it easier to use the method I showed you with a single command-line option to try to intelligently choose the parameters, but in terms of leaving all of the bitmaps in exactly their original format (highly compressed JPEG 2000), I don't have a way to do that.

Thank you! It is not the perfect one-button-solution for all my problems, but now I understand what is happening!

I learned about JPEG 2000 just 2 minutes ago when downloading a set of scanned images from archive.org and failing to make scantailor work on them. Talk about Sincronicity!

Way better suport that I've ever had from any company! You're awesome!

Just out of curiosity, do you have a guess of if KOreader would do a better job with this kind of pdf instead of the Nikel standart software on my Kobo TouchC? And how did you found out about the resolution of the images on the PDF, is there a option to do that on K2PDFopt? I Couldn't find it. And the JPX & JBIG2 on brackets on -i are the file formats of the imagens than?