k2pdfopt: optimizes PDFs for viewing on e-readers

[ew3244]

Hi all,

I used this tool to convert a PDF and use it in my Kindle keyboard. Everything seemed fine, but the problem is that when I highlight text it is captured with all the words together in the "My clippings" file with no whitespaces. For example: "Thisisasentencecomplete" instead of "This is a sentence complete". It is strange because if I copy and paste from Acrobat in PC the text does not get intermingled.

Questions:

-Is this normal when using k2optpdf? Can this problem be avoided? I executed the tool with the default options but I did other tests changing them with no luck.

-Can I do something to recover my notes? I have a lot of them and I would like to avoid having to separate manually all the words...

-I noticed that if I generate a new PDF file with Acrobat using the OCR then the notes are captured correctly in the Kindle. However, I did not find a way to recover my previous notes with the right word separation.

-I tried also with kinde-annotations tool with no luck.

Thank you for any advice

[willus]

Originally Posted by ew3244

Hi all,

I used this tool to convert a PDF and use it in my Kindle keyboard. Everything seemed fine, but the problem is that when I highlight text it is captured with all the words together in the "My clippings" file with no whitespaces. For example: "Thisisasentencecomplete" instead of "This is a sentence complete". It is strange because if I copy and paste from Acrobat in PC the text does not get intermingled.

Questions:

-Is this normal when using k2optpdf? Can this problem be avoided? I executed the tool with the default options but I did other tests changing them with no luck.

-Can I do something to recover my notes? I have a lot of them and I would like to avoid having to separate manually all the words...

-I noticed that if I generate a new PDF file with Acrobat using the OCR then the notes are captured correctly in the Kindle. However, I did not find a way to recover my previous notes with the right word separation.

-I tried also with kinde-annotations tool with no luck.

Thank you for any advice

@ew3244 -- Welcome to MR forums. It is a concern to me that this is happening on such a popular device as the Kindle Keyboard (one that I don't have available for testing). Try using the -ocrsp option (or -ocrsp+) and see what you get (if you are using the MS Windows GUI, type -ocrsp into the "Additional Options" box). Is there any chance you can post the source PDF (or a couple pages of it)? If you'd rather not post it, you can PM it to me.

[willus]

Originally Posted by ImABadGuyOk?

Hi, i dunno if i can post here... Anyway, i've used K2pdfopt to optimize an ebook for Kobo Glo and it worked great then i've tried to convert that same PDF_k2opt to epub using Calibre but i can't see anyt text content on the .epub file, i can only see the couverture and the table of contents. Does Anyone have the same problem and know how to resolve it ? Thx and sorry for bad english.

Welcome to MR forums. Did you convert with default settings (bitmapped output pages), or with native PDF output? Does your source PDF have natively rendered text, or scanned pages? This may do better posted in the epub forum. I do not have experience with calibre conversions. Moderators?

[ew3244e]

For example, you can try with the free book:

http://statweb.stanford.edu/~tibs/El...earn/main.html
http://statweb.stanford.edu/~tibs/El...II_print10.pdf

It seems to happen with many other similar examples. I understand that these are native PDF, no scan.

I tried the options provided. They change the notes captured and sometimes some whitespaces are preserved but the contents of My Clippings.txt is bad in the three cases (no options, -ocrsp, -ocrsp+).

Thanks.

[willus]

Originally Posted by ew3244e

For example, you can try with the free book:

http://statweb.stanford.edu/~tibs/El...earn/main.html
http://statweb.stanford.edu/~tibs/El...II_print10.pdf

It seems to happen with many other similar examples. I understand that these are native PDF, no scan.

I tried the options provided. They change the notes captured and sometimes some whitespaces are preserved but the contents of My Clippings.txt is bad in the three cases (no options, -ocrsp, -ocrsp+).

Thanks.

Actually, it looks like -ocrsp+ doesn't work correctly--it doesn't seem to have any effect. I'll have to fix that. Is there any way you can maybe post a photo of what "My Clippings.txt" looks like for the "bad" cases, or just post the contents directly? Do the source PDF files that you linked to copy and paste correctly to "My Clippings.txt" on your kindle? If not, can you post an example of a PDF file which does copy and paste correctly to "My Clippings.txt"? Also, did you try a native PDF conversion, e.g. using -mode fw -ls- ? Does it copy and paste correctly?

[kyzcreig]

Fantastic. I only wish I Calibre could further convert it to kindle formats without distortion.

[krogank9]

Hey, I have pdf which is mostly text with some images. When I use k2pdfopt to convert it for my e reader, it crops based on white space as it says, but it splits up some images and doesn't preserve their orientation to each other. for example i have a table like:

x        y
___      ___
1         5
2        10
3        15
4        20

but it splits up x and y from each other (both tables are in same image) one goes before a paragraph, the other goes after. I want to keep them next to each other. I understand this is probably useful for pdfs with no text and that are printed, but for me I need to preserve the tables orientation to each other. Is there an option to preserve all images in a pdf?

[willus]

Originally Posted by krogank9

... I understand this is probably useful for pdfs with no text and that are printed, but for me I need to preserve the tables orientation to each other. Is there an option to preserve all images in a pdf?

That's a good idea for an option. I think it should be do-able with the MuPDF library, but I'm not certain yet--I'll make a note to check. Anyway, there's presently no such option. You can try turning off text re-flow and just use -mode fw instead (select "fitwidth" from the conversion mode menu if using the MS Windows GUI), but it may make the text too small to read on your device. If you can, post a couple example pages of the source PDF (or send to me via a private message) and I might be able to recommend some options.

[krogank9]

Originally Posted by willus

-snip- , but it may make the text too small to read on your device. If you can, post a couple example pages of the source PDF (or send to me via a private message) and I might be able to recommend some options.

sure. i dont mind posting the pdf. here it is:

https://384ceda309d543db8121e87d97c2...g/Unit%201.pdf

also, it originally came in .odt open office format, i converted it to pdf from that:

https://googledrive.com/host/0BwfCKx...1oazg/Unit.odt

the other option i was thinking but have no idea how to do is to put a black border around all the images in my document to try to make sure it doesnt autocrop as whitespace. not sure how that would work though...

but yeah anyway it totally messes up the formatting of the tables & images and makes it unusable

[willus]

Originally Posted by krogank9

sure. i dont mind posting the pdf. here it is:

https://384ceda309d543db8121e87d97c2...g/Unit%201.pdf

also, it originally came in .odt open office format, i converted it to pdf from that:

https://googledrive.com/host/0BwfCKx...1oazg/Unit.odt

the other option i was thinking but have no idea how to do is to put a black border around all the images in my document to try to make sure it doesnt autocrop as whitespace. not sure how that would work though...

but yeah anyway it totally messes up the formatting of the tables & images and makes it unusable

You have several options since you have the source file. For one thing, you could try setting the page size of your .odt file to your device page size so that you generate the correctly-sized PDF right from your word processing program. I think that would be your best option. Like you said, you could also try to box all the figures--that would help, too. But even on the PDF you generated, you can help the conversion by setting the white threshold value very high since it is generated rather than scanned:

k2pdfopt -wt 254 -c -gtr .002 -col 1 unit1.pdf

The high white threshold will help see the light pink and light gray and light blue colors in the figures as "black" rather than "white" (i.e. as content)--and will thus make it harder for k2pdfopt to split them. I've also set the -gtr value lower than normal to try and prevent splitting figures, but I don't think it did much. And I used -col 1 since it's not a 2-column document.

But again, I'd definitely try working with the .odt file directly. I'll try it later myself if I get a chance.

[mikalto]

I'm not used to coding at all, so have run into a dead end on this. I have k2pdfopt installed on my macbook pro using OS10.9.4 and that is working well. I have been able to use it successfully to optimize files to read on my new kindle paperwhite. My problem is that I have a lot of PDFs from photocopies and want to set up tesseract to convert them. I downloaded and installed the Tesseract files and also RCenvironment to set the environment variables.

I think I just don't know enough about this to figure out the correct value to go with the variable: TESSDATA_PREFIX

I thought I needed to put in a path map like: /Library/Java/tesseract-ocr/
but obviously this isn't working. When I try it, k2pdfopt says: Could not find Tesseract data (env var TESSDATA_PREFIX = (not assigned)).
Using GOCR v0.49.

Can someone help me figure out what I need to do?
I apologize if this has been answered in earlier posts, as I just have not had the time to comb through everything to problem solve this.

Thank you

[willus]

Originally Posted by mikalto

I'm not used to coding at all, so have run into a dead end on this. I have k2pdfopt installed on my macbook pro using OS10.9.4 and that is working well. I have been able to use it successfully to optimize files to read on my new kindle paperwhite. My problem is that I have a lot of PDFs from photocopies and want to set up tesseract to convert them. I downloaded and installed the Tesseract files and also RCenvironment to set the environment variables.
[...]
When I try it, k2pdfopt says: Could not find Tesseract data (env var TESSDATA_PREFIX = (not assigned)).
Using GOCR v0.49.
[...]

I'll try and find some time to try out RCenvironment soon. I've never actually used it--in my test set up, I set the environment variable through the terminal shell. I'll let you know what I find out. Any other Mac experts that can help?

[markom]

Originally Posted by krogank9

Hey, I have pdf which is mostly text with some images. When I use k2pdfopt to convert it for my e reader, it crops based on white space as it says, but it splits up some images and doesn't preserve their orientation to each other. for example i have a table like:

x        y
___      ___
1         5
2        10
3        15
4        20

but it splits up x and y from each other (both tables are in same image) one goes before a paragraph, the other goes after. I want to keep them next to each other. I understand this is probably useful for pdfs with no text and that are printed, but for me I need to preserve the tables orientation to each other. Is there an option to preserve all images in a pdf?

Has Max. Columns value been set to 1 or 2 ?

Here is attached reflowed file in landscape mode Max. Columns 1, with table on page 2 left intact i.e. x and y columns left next to each other on the same page (using 2 as a value for Max. Columns would split the columns on separate pages.)

You can also use another e-ink or tablet(color recognition) simultaneously, just for zooming-in on the pictures and tables in the original file and checking an original layout, while reading reflowed text on another. e.g. I can use Sony Prs T1 for scribbling, higlighting, dictionary, annotation(by keyboard or scribling) etc. while simultaneously reading the same pdf enlarged on Kindle DX that lacks those same functions for pdfs.

Better idea for such letter sized and A4 pdfs with very small letters is using 10" e-ink though e.g. I can even read it in portraite mode on Kindle DX (140 mm screen width) zoomed-in without margins.

In landscape mode zoomed without margins (202 mm) I can get magnification compared to 185 mm text width of that letter sized pdf on paper.

Also, if you have a lot of such letter sized pdf files and want small e-ink then you can better try some 6.8" e-ink like Kobo Aura HD that has got around 140 mm screen width in landscape i.e. about 20 mm more than 120 mm of 6" e-ink.

If you have Abbyy Finereader you can transform every recognized table into a picture by selecting it and choosing Change-Area-Type.

In Adobe Acrobat you can select a picture and choose Edit in Paint and then draw rectangle around a picture.

[mikalto]

Originally Posted by willus

I'll try and find some time to try out RCenvironment soon. I've never actually used it--in my test set up, I set the environment variable through the terminal shell. I'll let you know what I find out. Any other Mac experts that can help?

Thanks for the reply.
I'd be happy to do this through the terminal shell if I knew how. I found out about RCenvironment through an earlier posting and thought that I had enough step by step info to be successful. I just don't have enough background (read: any at all) in code to be able to even figure out where I need to look to figure out the steps I need to take to at this point. I really would love to use OCR with k2pdfopt (which otherwise has been really helpful) to better use my Kindle for coursework. Thanks for any feedback. I really appreciate it!

[willus]

Originally Posted by mikalto

Thanks for the reply.
I'd be happy to do this through the terminal shell if I knew how. I found out about RCenvironment through an earlier posting and thought that I had enough step by step info to be successful. I just don't have enough background (read: any at all) in code to be able to even figure out where I need to look to figure out the steps I need to take to at this point. I really would love to use OCR with k2pdfopt (which otherwise has been really helpful) to better use my Kindle for coursework. Thanks for any feedback. I really appreciate it!

Okay, so I downloaded RCEnvironment and played around with it some. I presume you figured out how to install it and use it (to use it you have to go to the Apple menu and select "System Preferences..." and then click on the "Environment Variables" icon in the "Other" section at the bottom of the System Preferences window). I then clicked on the "+" icon at the bottom of the dialog box and added my TESSDATA_PREFIX variable and then clicked the "Save" button. A screenshot of that is attached. Once you do that, you have to logout and then login again. Then, assuming there are no typos and you picked the right folder, it should work.

Interestingly, if you delete an environment variable with RCEnvironment, you have to completely re-start to get rid of it for good (you can't just logout and login again).

What this all does is to save the environment variable info to a file in the .MacOSX folder underneath your home folder. There is another file, /etc/launchd.conf, which I believe stores system-wide environment variables. So you can also store the TESSDATA_PREFIX environment variable in /etc/launchd.conf by adding a line to it like so:

setenv TESSDATA_PREFIX /usr/local/Cellar/tesseract/3.02.02/share/

You'll need sudo privileges to edit /etc/launchd.conf, but hopefully RCEnvironment will do the trick.

Let me know if you get it working.