Converting PDF/HTML to ereader formats

[HappyChris]

I have downloaded a free pdf book and wish to convert it to be read via my Kindle or my epub readers. I used Calibre to convert it to AZW3 and HTMLZ, both of which display similarly, that is, with hard-coded line lengths and page headers, numbers etc., displaying clumsy line formatting on small 'pages' with paragraph, not line- spacing between lines and chapter titles, etc. inappropriate to the page size.

I tried one or two online services but, while they got the ToC lines down to one line each with fewer dots between the chapter title and the page number, they choked when faced with the many diacritics in the text! Calibre handled the diacritics without fault.

I would rather not spend days removing '</p>'s and repeating page header and footer texts! Is there a program or a script that could speed this process up or even do it all for me?

[Turtle91]

That is the million dollar question... PDF is probably the worst possible source document to do a conversion from. In my opinion, it is a far better use of your time to search for a different source format than try and convert a pdf. Sorry

[BetterRed]

If it's a multi-column PDF you might be able to make use of ==>> k2pdfopt: optimizes PDFs for viewing on e-readers

Or, if you can wrangle the PDF into Word (maybe via this ==>> Edit PDF content in Word), then you could use Toxaris' add in ==>> e-Book Tools - a Word add-in to create an ePUB that you can finalise in an ebook editor such as Sigil or Caiibre.

Otherwise - what Turtle91 said.

BR

[JSWolf]

Forget PDF exists. Problem with converting PDF solved.

[Hitch]

Originally Posted by HappyChris

I have downloaded a free pdf book and wish to convert it to be read via my Kindle or my epub readers. I used Calibre to convert it to AZW3 and HTMLZ, both of which display similarly, that is, with hard-coded line lengths and page headers, numbers etc., displaying clumsy line formatting on small 'pages' with paragraph, not line- spacing between lines and chapter titles, etc. inappropriate to the page size.

I tried one or two online services but, while they got the ToC lines down to one line each with fewer dots between the chapter title and the page number, they choked when faced with the many diacritics in the text! Calibre handled the diacritics without fault.

I would rather not spend days removing '</p>'s and repeating page header and footer texts! Is there a program or a script that could speed this process up or even do it all for me?

Sorry Chris:

My business does this every single day, and there's no magic, fast way. The only way to get there from here is to a) use Abbyy Fine Reader, which at least will remove the running heads/footers, and b) do the rest by hand. I wish there were a faster/better way, but there isn't.

Having said that, yes, you can write regex to clean up some portion of the line-ending </p>'s, and all that, but...it's still all human labor, eyes and hands.


Hitch

[HarryT]

Yes, as Hitch says, a decent OCR program such as Abbyy, followed by hand clean-up of the resulting code, really is as good as it gets. PDF is a "terminal format": that is, you can convert to it easily enough, but you can't convert from it.

[RobertDDL]

Instead of Abbyy, which costs money, you can try pdftotext, which is open source - it's a little Windows command line utility, which you can download from www.xpdfreader.com (it's included in the Xpdf tools). With the -layout option, it gives you a plain text file with line breaks that correspond to the lines in the PDF (doesn't work with all PDF files, though).

You'll still have a lot of editing to do, but with a decent editor and a little knowledge of regular expressions and a tool to convert plain text to epub you should be able to get a reasonably readable epub version in an hour or so. (I have a tool that helps with the process, but like pdftotext it's Windows command line stuff, so I don't know if you're interested.)

[Hitch]

Originally Posted by RobertDDL

Instead of Abbyy, which costs money, you can try pdftotext, which is open source - it's a little Windows command line utility, which you can download from www.xpdfreader.com (it's included in the Xpdf tools). With the -layout option, it gives you a plain text file with line breaks that correspond to the lines in the PDF (doesn't work with all PDF files, though).

You'll still have a lot of editing to do, but with a decent editor and a little knowledge of regular expressions and a tool to convert plain text to epub you should be able to get a reasonably readable epub version in an hour or so. (I have a tool that helps with the process, but like pdftotext it's Windows command line stuff, so I don't know if you're interested.)

Robert:

Doesn't that give you, as you said, a plain text file? So that all formatting is lost? For some folks, that would be a crapload more work.

Abbyy, if I am not mistaken, has a website where you can convert a single document for free. There are limits on it, and all that, but you might try searching for that.

Hitch

[HarryT]

You can also buy older releases of Abbyy at much lower prices than the current version, and they are more than adequate for the job - you just lose the spelling dictionary for Outer Mongolia (or whatever ).

[RobertDDL]

Originally Posted by Hitch

Robert:

Doesn't that give you, as you said, a plain text file? So that all formatting is lost? For some folks, that would be a crapload more work.

Abbyy, if I am not mistaken, has a website where you can convert a single document for free. There are limits on it, and all that, but you might try searching for that.

Hitch

Yes, you are right about all formatting being lost. For novels, in most cases that would be headings and italics. It doesn't take much time to manually restore headings, in a plain text file (word-wrap disabled) you can usually spot them quite easily, even if they don't start with numbers or "chapter" etc. The loss of italics can be quite unfortunate, if the writer relies on them - not all of them do, though.

Unless I've missed something, even with a good OCR software, though, it's not trivial to retain italics, while getting rid of all the excess formatting that the OCR'd output file is burdened with (even Abbyy is wont to hallucinate dozens of different page and paragraph formats, font sizes, etc.). I do have a fairly recent version of Abbyy, but, for instance, when I had to convert a series of 26 PDF books for a friend (being blind, she couldn't read the PDFs), I got it done much more quickly with the pdftotext tool. (There were no italics in those PDFs, which we had bought, but which seem to have been produced from carelessly OCR'd print editions. No idea if the paper originals had them, but I've often come across public-domain books on the Internet without italics, when the originals have them.) (Which is not meant to say it's ok to lose them.)

[Hitch]

Originally Posted by RobertDDL

Yes, you are right about all formatting being lost. For novels, in most cases that would be headings and italics. It doesn't take much time to manually restore headings, in a plain text file (word-wrap disabled) you can usually spot them quite easily, even if they don't start with numbers or "chapter" etc. The loss of italics can be quite unfortunate, if the writer relies on them - not all of them do, though.

Unless I've missed something, even with a good OCR software, though, it's not trivial to retain italics, while getting rid of all the excess formatting that the OCR'd output file is burdened with (even Abbyy is wont to hallucinate dozens of different page and paragraph formats, font sizes, etc.). I do have a fairly recent version of Abbyy, but, for instance, when I had to convert a series of 26 PDF books for a friend (being blind, she couldn't read the PDFs), I got it done much more quickly with the pdftotext tool. (There were no italics in those PDFs, which we had bought, but which seem to have been produced from carelessly OCR'd print editions. No idea if the paper originals had them, but I've often come across public-domain books on the Internet without italics, when the originals have them.) (Which is not meant to say it's ok to lose them.)

Honestly, I've never noticed any sturm und drang retaining italics with Abbyy. Using Abbyy is, as you rightly say, a PITA, overall, but in my opinion, compared to all the other methods to get a PDF to an editable form, it's the best way to go. Maybe I'm just lucky that way, that we get a lot of books that are bold/italics laden, but we do.

I'm sure that everyone has their own preferred way of working. I have a ton of expertise in Word, so frankly, it's super-easy for me to do the cleanup on Abbyy output into a Word file, or, of course, regex it to the nth, in HTML. If there ARE italics and bold, in large numbers, I'll use Toxaris' superb "ePUB Tools" Word plug-in, first--as that makes marking/retaining both of those character markups simplicity itself--and I'll "clean" the styles from the rest, and then restyle them. I find that the fastest route.

Offered solely FWIW.

Hitch

[Tex2002ans]

Originally Posted by Hitch

My business does this every single day, and there's no magic, fast way. The only way to get there from here is to a) use Abbyy Fine Reader, which at least will remove the running heads/footers, and b) do the rest by hand. I wish there were a faster/better way, but there isn't.

Yep, I agree completely. PDF is an awful input source, and there is no real good way besides a lot of human elbow grease.

Originally Posted by Hitch

Honestly, I've never noticed any sturm und drang retaining italics with Abbyy. Using Abbyy is, as you rightly say, a PITA, overall, but in my opinion, compared to all the other methods to get a PDF to an editable form, it's the best way to go. Maybe I'm just lucky that way, that we get a lot of books that are bold/italics laden, but we do.

Or as long as you have Finereader 10 or higher, it has EPUB output. The EPUB output has relatively clean code with only a handful of inline styles.

I have 12 regex to help clean up after (5 to handle Italics, Bold, BoldItalics, Smallcaps, BoldSmallcaps + 7 to just clean up some <td> and some other anomalies).

Example:

Search: <span style="font-style:italic;">
Replace: <span class="italics">

Originally Posted by RobertDDL

Yes, you are right about all formatting being lost. For novels, in most cases that would be headings and italics. [...] The loss of italics can be quite unfortunate, if the writer relies on them - not all of them do, though.

Getting the OCR correct is just a portion of the work. Formatting is just as important (and is where a lot of other tools fail miserably).

I've written about this in-depth before:

https://www.mobileread.com/forums/sh...72#post2883972

Most of your time is going to be spent editing and correcting the text/formatting, so the better you can get the input, the easier/faster your life will be on those later steps.

[willus]

My two cents on PDF conversions. This topic has been visited in a number of past MR threads.

[darryl]

Originally Posted by JSWolf

Forget PDF exists. Problem with converting PDF solved.

I totally agree. I dabbled with converting pdf's years ago and decided it is simply not worthwhile. Of course Hitch would seem to have little choice given her business.

[BetterRed]

Originally Posted by willus

My two cents on PDF conversions. This topic has been visited in a number of past MR threads.

- I've only recently 'stumbled on' Word's PDF convert facility.

I am surprised (shocked even) at how well it does with PDF's from institutions like Rand, McKinsey, Stratfor etc - previously I often didn't even bother trying to convert their documents. Tables would end up as a meaningless shambles, sidebars as a nonsensical farrago etc. And the number of styles it creates for a document is significantly fewer than other pdf converters.

BR