k2pdfopt: optimizes PDFs for viewing on e-readers

[willus]

K2pdfopt v2.21 is released. It is now compiled with MuPDF v1.5. See the web site for details.

[tigerjack89]

Hey willus, here again
I'd like to convert another pdf book. It has some lists like the ones attached.
Which options can I use to have better results? Now, I have very bad results using very different option, but I acted only on the "Max. columns" option.

Thanks in advance

[willus]

Originally Posted by tigerjack89

Hey willus, here again
I'd like to convert another pdf book. It has some lists like the ones attached.
Which options can I use to have better results? Now, I have very bad results using very different option, but I acted only on the "Max. columns" option.

Thanks in advance

Can you give me more than a few images? It really helps to see several pages of the source file. You can send to me in a private message if you don't want to post it. Generally, though, the best option for tables like this is to use -mode fitwidth (select "fitwidth" from the mode conversion menu in the upper right part of the MS Windows GUI), but that may make the text too small for you.

[ittiandro]

Cannot convert k2PdfOpt to Mobi with Calibre. I get the following conversion error message

Put long messages in spoiler tags! - MODERATOR]

Spoiler:

QUOTE
Convert book 1 of 1 (Lysias Collected Works)
Resolved conversion options
calibre version: 1.47.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0.0,
'book_producer': None,
'change_justification': u'original',
'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., '\\s*((chapter|book|section|part)\\s+)|((prolog|pr ologue|epilogue)(\\s+|$))', 'i')) or @class = 'chapter']",
'chapter_mark': u'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': u'',
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x03594970>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': u'both',
'mobi_ignore_margins': False,
'mobi_keep_original_images': True,
'mobi_toc_at_start': False,
'new_pdf_engine': False,
'no_chapters_in_toc': False,
'no_images': True,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindleFireOutput object at 0x03594CF0>,
'page_breaks_before': u"//*[name()='h1' or name()='h2']",
'personal_doc': u'[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': u'C:\\Users\\franco\\AppData\\Local\\Temp\\calibre _ydpc9p\\ixouwp.opf',
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': u'',
'search_replace': '[]',
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': None,
'sr1_search': None,
'sr2_replace': None,
'sr2_search': None,
'sr3_replace': None,
'sr3_search': None,
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_factor': 0.45,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: PDF Input running
on C:\Users\franco\AppData\Local\Temp\calibre_ydpc9p\ wuqx0a.pdf
Converting file to html...
Python function terminated unexpectedly
[Error 5] Access is denied (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 195, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_override
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 1040, in run
File "site-packages\calibre\customize\conversion.py", line 241, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\pdf_inp ut.py", line 50, in convert
File "site-packages\calibre\ebooks\pdf\pdftohtml.py", line 67, in pdftohtml
File "subprocess.py", line 711, in __init__
File "subprocess.py", line 948, in _execute_child
WindowsError: [Error 5] Access is denied
UNQUOTE

The book was originally converted with k2PdfOpt because it was not readable in my Android device in its original PDF version( scanned, I suppose?). It is perfectly readable now but I wanted to convert it to Mobi, because the interface is much nicer.Any suggestions?

Thanks

Ittiandro

[willus]

Originally Posted by ittiandro

Cannot convert k2PdfOpt to Mobi with Calibre. I get the following conversion error message...

...The book was originally converted with k2PdfOpt because it was not readable in my Android device in its original PDF version( scanned, I suppose?). It is perfectly readable now but I wanted to convert it to Mobi, because the interface is much nicer.Any suggestions?

Thanks

Ittiandro

@Ittiandro -- can you post or PM me 1. the source PDF, 2. the k2pdfopt output, and 3. the k2pdfopt options you used? I would think that if the input is just a scanned article then the conversion to mobi should be pretty straightforward--just a sequence of bitmapped pages to convert, but the options you chose could be important--for instance, if you did a native PDF conversion, try turning that off (unchecking "Native PDF Output" in the MS Windows GUI). The standard bitmapped output (i.e. non-native conversion) from k2pdfopt, I would guess, would likely be easier for Calibre to digest than the native-conversion output.

[ittiandro]

Originally Posted by willus

@Ittiandro -- can you post or PM me 1. the source PDF, 2. the k2pdfopt output, and 3. the k2pdfopt options you used? I would think that if the input is just a scanned article then the conversion to mobi should be pretty straightforward--just a sequence of bitmapped pages to convert, but the options you chose could be important--for instance, if you did a native PDF conversion, try turning that off (unchecking "Native PDF Output" in the MS Windows GUI). The standard bitmapped output (i.e. non-native conversion) from k2pdfopt, I would guess, would likely be easier for Calibre to digest than the native-conversion output.

Willus

I am trying to send both files you requested ( the original PDF and the converted k2PdfOpt, but I get a message that these two files cannot be uploaded due to something wrong with the " security'(?).
THese files are big : the original PDF is 25 Mb and the k2PdfOpt conversion 50 Mb!. May be this is the problem! Maybe it is better that I e-mail you but I forgot your address. Mine is vifra@videotron.ca.
About the settings I used to convert to k2PdfOpt, I cannot dig them back, but I can tell you that I never use the the native PDF setting. Actually , I never check any of the boxes, except, sometimes, the Autostraighten box and the Colour Output. I use the fit-to-page setting and I seldom play with the margins.

Thank you so much

Ittiandro

[willus]

Originally Posted by ittiandro

Willus

I am trying to send both files you requested ( the original PDF and the converted k2PdfOpt, but I get a message that these two files cannot be uploaded due to something wrong with the " security'(?).
THese files are big : the original PDF is 25 Mb and the k2PdfOpt conversion 50 Mb!. May be this is the problem! Maybe it is better that I e-mail you but I forgot your address. Mine is vifra@videotron.ca.
About the settings I used to convert to k2PdfOpt, I cannot dig them back, but I can tell you that I never use the the native PDF setting. Actually , I never check any of the boxes, except, sometimes, the Autostraighten box and the Colour Output. I use the fit-to-page setting and I seldom play with the margins.

Thank you so much

Ittiandro

I will send you a private message. If you select "fitwidth" or "fitpage" from the "Conversion Mode" menu, this also automatically selects native PDF output, so after selecting either of those modes, you need to then uncheck the "Native PDF Output" box to turn off native PDF output. Although if the k2pdfopt output file is twice as large as your source file, like you say, then you probably have native PDF output turned off.

[pmarty]

Greetings to everyone, this is my first post to the forum.

k2pdfopt -col 1 -cbox 0,1 -de 1.5 -x -ui- 43.pdf

• When joining lines the baselines are not properly aligned (see warehouses on the second line of output). The line being attached appears to be systematically lower.
• Indented blocks like headings (1.2 Overview of Database Management System) and ordered lists (1. Conventional users (...)) are not joined at all.

I'm asking for your help on tweaking k2pdfopt parameters to resolve above issues (if possible). See my attached files. The output is already pretty decent, just needs a few polishing touches

Best regards,
pmarty

[willus]

Originally Posted by pmarty

Greetings to everyone, this is my first post to the forum.

k2pdfopt -col 1 -cbox 0,1 -de 1.5 -x -ui- 43.pdf

• When joining lines the baselines are not properly aligned (see warehouses on the second line of output). The line being attached appears to be systematically lower.
• Indented blocks like headings (1.2 Overview of Database Management System) and ordered lists (1. Conventional users (...)) are not joined at all.

I'm asking for your help on tweaking k2pdfopt parameters to resolve above issues (if possible). See my attached files. The output is already pretty decent, just needs a few polishing touches

Best regards,
pmarty

@pmarty -- Welcome to the MR forums. Thank you for your detailed post and for including your source document. I believe the joining problem with "warehouses" is because your source doc is slightly skewed. Try using -as to autostraighten it. It may not work because there is a threshold amount which your doc has to be rotated by for k2pdfopt to apply the straightening procedure (since it has a cpu cost). It will echo confirmation to the screen if it straightens a page. The second issue is because of the indentation of the following lines. K2pdfopt uses some fuzzy logic to decide when to join lines, and it is hard to make it foolproof. There currently are not any options that affect that behavior.

[pmarty]

willus, thank you for your reply.

Originally Posted by willus

Try using -as to autostraighten it.

(Straightening page: rotating cc by -0.13 deg.)

It did the trick, now the lines are aligned properly.

K2pdfopt uses some fuzzy logic to decide when to join lines, and it is hard to make it foolproof. There currently are not any options that affect that behavior.

I can imagine that changing it to suit one document might easily break others.

[pmarty]

Here are my results with another file.

k2pdfopt -col 1 -cbox 0,0,21cm,26.7cm -odpi 100 -sm 2.pdf

k2pdfopt is apparently confused with the text box in the middle. It detects 3 red regions there which are oddly numbered (58, 99, 99) in the marked source. I tried using -evl 2 but it didn't help.

Any ideas?

[willus]

Originally Posted by pmarty

Here are my results with another file.

k2pdfopt -col 1 -cbox 0,0,21cm,26.7cm -odpi 100 -sm 2.pdf

k2pdfopt is apparently confused with the text box in the middle. It detects 3 red regions there which are oddly numbered (58, 99, 99) in the marked source. I tried using -evl 2 but it didn't help.

Any ideas?

Interesting case. Thanks again for the thorough information and attachments. The 58 and 99 regions are not actually "red"--they're orange (the color I use for diagnostics involving finding rows of text), and IIRC the values are related to a row-splitting figure of merit. k2pdfopt initially doesn't split the center box into text rows but then goes back and looks for where it might have missed row splits and this is where the issue is. See the -gtr and -rsf options and maybe try tweaking them. I'm surprised -evl 2 didn't at least change the behavior. I'll investigate further when I have more time.

[pmarty]

Unfortunately, I have found yet another two issues with my previous conversion:

k2pdfopt -col 1 -cbox 0,1 -de 1.5 -x -ui- 43.pdf

Attachments: 43.pdf, 43_k2pdfopt.pdf

• The "size" of resulting PDF doesn't fit Kindle 4 screen. It has to be viewed in 'fit-to-screen' mode instead of 'actual size' which is bigger and requires panning. The result is bitmap quality loss and jagged fonts. Default values (560x735) for width and height don't seem to work out.
• Kindle dictionary lookup becomes broken for most words (cf. popular on the first line in both files).

[willus]

Originally Posted by pmarty

Unfortunately, I have found yet another two issues with my previous conversion:

k2pdfopt -col 1 -cbox 0,1 -de 1.5 -x -ui- 43.pdf

Attachments: 43.pdf, 43_k2pdfopt.pdf

• The "size" of resulting PDF doesn't fit Kindle 4 screen. It has to be viewed in 'fit-to-screen' mode instead of 'actual size' which is bigger and requires panning. The result is bitmap quality loss and jagged fonts. Default values (560x735) for width and height don't seem to work out.
• Kindle dictionary lookup becomes broken for most words (cf. popular on the first line in both files).

I unfortunately do not have access to a kindle 4, but this is the first I've heard of this issue. All the command-line options are available to optimize size and dpi for any device (-w, -h, -odpi, -pad, -mc). I optimized the output for a kindle 2 a long time ago by repeatedly taking screen captures from the kindle and iterating the command-line values to get a 1-to-1 pixel mapping between the bitmap saved in the PDF file and the bitmap displayed on the kindle 2 screen. This minimizes artifacts by not forcing the kindle to re-size the bitmap. You might try using something like -dr 1.5. It may improve your output.

Both PDF files allowed me to highlight and define "popular" on my (original) iPad PDF reader. So I would suggest that it is a firmware issue. Were you able to highlight and define other words within that same doc? Either way, if you can't highlight the word in the source PDF, don't expect to be able to highlight it in the converted PDF unless you turn on OCR with the -ocr switch (not counting -ocr m, which is on by default). There is a relatively new option which affects the selection of words in the converted file, though: -ocrsp. I know that on my kindle 2 with native firmware, PDF highlighting is not possible.

[ImABadGuyOk?]

Hi, i dunno if i can post here... Anyway, i've used K2pdfopt to optimize an ebook for Kobo Glo and it worked great then i've tried to convert that same PDF_k2opt to epub using Calibre but i can't see anyt text content on the .epub file, i can only see the couverture and the table of contents. Does Anyone have the same problem and know how to resolve it ? Thx and sorry for bad english.