Conversion from two-column PDF

[Borodin]

I have read the sticky regarding this and suspect the answer to my question is that there is no current solution. I've just come here to make sure.

I have a PDF document that has two columns of text per page with a central gutter. I pushed it through the document converter using all default settings, and the output contains text reading from left to right acros the page, ignoring the gutter.

Is there any solution to this, or has anyone begun to work on a fix? I am happy to roll up my sleeves and write a little Pythin if that's what it takes.

[theducks]

Moderator Notice
Did you bother to read this sticky? https://www.mobileread.com/forums/sho...d.php?t=118605

[PeterT]

One option might be to use a tool like Briss to crop the pages.

See http://sourceforge.net/projects/briss/

[ldolse]

Last paragraph in the sticky if you want to contribute or experiment - the new engine already handles two column, but overall it does a worse job at laying out the html to match the pdf than the existing engine.

[Borodin]

Originally Posted by theducks

Did you bother to read this sticky? https://www.mobileread.com/forums/sho...d.php?t=118605

Yes I 'bothered' to read it as I said in my post. The sticky doesn't make it clear that this specific problem in unsolved and I thought it prudent to ask before I launched into writing code to resolve something that was already fixed.

Is this the quality of moderation to expect on this forum?

[Borodin]

Originally Posted by ldolse

Last paragraph in the sticky if you want to contribute or experiment - the new engine already handles two column, but overall it does a worse job at laying out the html to match the pdf than the existing engine.

Thanks Idolse. Does 'the new engine' mean that which comes with the latest release, which I already have? I am really not too interested in layout as my input is simply two-column streamed text.

[Borodin]

Originally Posted by PeterT

One option might be to use a tool like Briss to crop the pages.

See http://sourceforge.net/projects/briss/

Nice. I will give this a try.

[ldolse]

The new engine is in every release, but it's disabled by default - instructions on how to use it are in the sticky - basically you need to do the conversion from the command line with two specific arguments:
--new-pdf-engine
--debug-pipeline "/some/directory"

For all the cli arguments and examples refer to the manual:
http://manual.calibre-ebook.com/cli/...#ebook-convert
http://manual.calibre-ebook.com/conv...ormatting-demo

The above won't generate an actual ebook for you, it will just dump an html version of the pdf into the debug folder. You can convert the html to an ebook by re-importing it to Calibre and converting from zip to whatever.

You'll still wish the layout was better after you see the output, but for two column it will probably give you manually salvageable results, which the production version won't do. By layout I mean many paragraphs will probably need to be manually fixed - the sentences will be correct, but it's not good at detecting appropriate paragraph breaks/page breaks as I recall.

[kiwidude]

@Borodin - obviously theducks just missed that part of your post. You have to appreciate that there are an enormous number of repeated posts every week from people who *don't* bother to read the pdf sticky or think that somehow their case is different. You happened to get hit by theducks on autopilot. Birdstrike

If you are wanting to continue the technical discussion along the lines of you adding the feature yourself this thread would be better placed in the Development forum.

Whether that new PDF engine is going to give you much joy in this regard though is another matter. Kovid recently upgraded a number of the pdf components and the latest version that ships with calibre has some terminal issues I found with my Extract ISBN plugin (causing unmanaged code crashes). I changed my plugin to use some different pdf components as did Kovid for calibre as he has other priorities to work on. I am sure he can give you a more definitive answer.

[Borodin]

Originally Posted by kiwidude

If you are wanting to continue the technical discussion along the lines of you adding the feature yourself this thread would be better placed in the Development forum.

Thanks Kiwi I will do that I haven't looked at how the conversion process pipes through HTML but I am fluent with Python, and as long as the HTML is XHTML I am sure XSLT will be useful.

It sounds like the PDF engine is a work in progress and I am anxious to contibute where I can without creating an unmergeable branch.

[kiwidude]

I've moved the thread.

As I mentioned above the initial problem you will have is that the engine is written in C++ to produce the (x?)html output. And last time I tried it (via my plugins) it was regularly falling over with nasty crashes. So unless you are prepared to roll up your sleeves on the C++ side and figure out why that is you may not get past the starting gate.

[ldolse]

I didn't realize the C++ portion had been updated and was buggy now - assuming you get past that part you will get perfect xhtml. Basically what poppler does (when it works) is create a nice xhtml file which will render exactly like the original pdf. But aside from rendering perfectly and providing lots of info about the layout it's still no good because it still has all the limitations of pdf. The reflow code then tries to turn all the sentence fragments into paragraphs. So the debug output will actually have the original xhtml from Poppler along with the reflowed version.

[kovidgoyal]

The C++ part was removed, it now uses

pdftohtml -xml


to generate the layout XML. However, I haven't gotten around to migrating the python code that reads the XML and converts it to HTML. That should be fairly simple to do. The python code still expects the old version of the XML, so you will need to change it slightly.

[avantman42]

If you have access to pdftohtml & sed, I use the following:

pdftohtml -c -s -i -xml INPUT_FILE.pdf
sed -e s/"<[^>]*>"//g INPUT_FILE.xml > OUTPUT_FILE.txt

That usually gives a reasonable text file, which can then be worked on if needed and converted to whatever format you wish using ebook-convert.

[theducks]

Originally Posted by Borodin

Yes I 'bothered' to read it as I said in my post. The sticky doesn't make it clear that this specific problem in unsolved and I thought it prudent to ask before I launched into writing code to resolve something that was already fixed.

Is this the quality of moderation to expect on this forum?

Sorry, missed the first paragraph.