Converting PDF to EPUB and automatically creating popup footnotes

[bleopaskom]

I use Abbyy to convert PDFs to EPUB, and the conversions are almost perfect.

When I open some of these converted epubs into Calibre, all the footnotes are linked and pop up correctly in the editor. The rest of the converted epubs do not have any linked footnotes, and to fix I would have to manually link each and every one of them.

Do I have to do something in Abbyy first?

Is there a Calibre addon that can help link them automatically?

There was maybe something about exporting to Word and fixing it there, or using Sigil, or just making use of Toxaris ePUB-Tools, but I don't really understand how to do it.


A step-by-step guide would be appreciated.


Thanks!

[Turtle91]

I don’t know the exact problems with the pdf conversions without being able to see examples. However, there are a couple things I can mention that will set you looking in the right direction.

There is no automatic process that can determine links if there is nothing to indicate a link in the original document. I suspect the successful pdfs have had those links/bookmarks already created in the pdf before the conversion, the failures didn’t.

When you say the pdf conversion was almost perfect, I start to think I’m in an episode of twilight zone (did I just age myself?!?) because it’s very difficult to get a ‘decent’ conversion from a pdf, let alone a near perfect one. It might look decent but I would suspect that when you open it up in Sigil/Calibre you will find either a bunch of nauseating code, or just a bunch of images of each page, that may cause problems in different readers/apps/systems.

Tread carefully!!

Cheers!

[bleopaskom]

Originally Posted by Turtle91

There is no automatic process that can determine links if there is nothing to indicate a link in the original document. I suspect the successful pdfs have had those links/bookmarks already created in the pdf before the conversion, the failures didn’t.

When you say the pdf conversion was almost perfect, I start to think I’m in an episode of twilight zone (did I just age myself?!?) because it’s very difficult to get a ‘decent’ conversion from a pdf, let alone a near perfect one. It might look decent but I would suspect that when you open it up in Sigil/Calibre you will find either a bunch of nauseating code, or just a bunch of images of each page, that may cause problems in different readers/apps/systems.

Tread carefully!!

Cheers!

Thanks, then I guess I won't be bothered with trying to manually link hundreds of footnotes, that would be a waste of time.


Even those really bad manually scanned books with slightly yellow/darker pages were OCRd correctly, with obviously a few characters misread, because the pages were not aligned and it wasn't in English.


Other than that, when I go to edit in Calibre, the only errors that I get are the missing rules in the main.css. Even then, if I just send them to my Kindle Paperwhite 4, there are no issues whatsoever.

[JSWolf]

Converting PDF to ePub and expecting it to be correct is not going to happen. There are going to be many many errors. There is nothing you can do about that.

The only way to fix things is to A/B compare and also hand fix the code. It's not worth the hassle.

[Hitch]

Originally Posted by bleopaskom

Thanks, then I guess I won't be bothered with trying to manually link hundreds of footnotes, that would be a waste of time.


Even those really bad manually scanned books with slightly yellow/darker pages were OCRd correctly, with obviously a few characters misread, because the pages were not aligned and it wasn't in English.


Other than that, when I go to edit in Calibre, the only errors that I get are the missing rules in the main.css. Even then, if I just send them to my Kindle Paperwhite 4, there are no issues whatsoever.

If you're doing this for personal use alone, and you don't care overmuch if the footnotes popup or not, then I guess this is the best approach--just leave them as they are.

I, like the others, find myself pretty speechless over the idea that Abbyy did an amazing job converting a PDF to an ePUB. I mean...I work hand-in-hand with a very great fellow, who does world-class scanning/OCR. We send all our querents there, who have printed books in-hand and all that (and image-layer-only PDFs and so on). I've seen what his ePUB exports look like, from Abbyy--and they make me wince. And yet, he's the best and most experienced scanner and Abbyy-user that I know. So...you are either incredibly lucky or you simply want different things from your conversions.

(n.b.: I cannot tell you how many people I've had show up at my shop, over the years, with "eBooks" made by Abbyy, from scanners like BlueLeaf, etc., needing help to fix them and honestly, it's cheaper and less time-consuming to take their OCR-output Word file and redo the book from the jump than it is to fix those messes. Boggling, really, that people can't/don't see how bad those auto-ePUBs really are....)

It sounds to me that you're using Abbyy on modern PDFs that already have a text layer. If you are, I'm honestly not sure why you'd do that. You can export a modern PDF from Acrobat Pro, DC, etc. into Word and then use that Word file to make an ePUB in Calibre, too. Why go to the AbbyyFineReader route, unless you have a PDF that's imaged-only?

Are the footnotes in the PDFs you are scanning already linked, do you know?

Hitch

[bleopaskom]

Originally Posted by Hitch

I, like the others, find myself pretty speechless over the idea that Abbyy did an amazing job converting a PDF to an ePUB. I mean...I work hand-in-hand with a very great fellow, who does world-class scanning/OCR. We send all our querents there, who have printed books in-hand and all that (and image-layer-only PDFs and so on). I've seen what his ePUB exports look like, from Abbyy--and they make me wince. And yet, he's the best and most experienced scanner and Abbyy-user that I know. So...you are either incredibly lucky or you simply want different things from your conversions.
)

It sounds to me that you're using Abbyy on modern PDFs that already have a text layer. If you are, I'm honestly not sure why you'd do that. You can export a modern PDF from Acrobat Pro, DC, etc. into Word and then use that Word file to make an ePUB in Calibre, too. Why go to the AbbyyFineReader route, unless you have a PDF that's imaged-only?

Are the footnotes in the PDFs you are scanning already linked, do you know?

Hitch

I tried to use Adobe a few times, but it was just terrible. With Abbyy 15 I don't have too many issues. Some of the PDFs are older and hand scanned, but the text is being recognized even if it's not in English, and sure the TOC isn't perfect, and some of the headers are not right, but if I am to just immediately send to my Kindle it works and reads great, I don't know what else it's supposed to do.


Honestly, I don't know how to check if it's a modern PDF that has a text layer and if the footnotes are already linked. But I will try to can export with Adobe to Word, and then use Calibre to make an epub, and see how the final result looks.

Thank you!

[Hitch]

Originally Posted by bleopaskom

I tried to use Adobe a few times, but it was just terrible. With Abbyy 15 I don't have too many issues. Some of the PDFs are older and hand scanned, but the text is being recognized even if it's not in English, and sure the TOC isn't perfect, and some of the headers are not right, but if I am to just immediately send to my Kindle it works and reads great, I don't know what else it's supposed to do.


Honestly, I don't know how to check if it's a modern PDF that has a text layer and if the footnotes are already linked. But I will try to can export with Adobe to Word, and then use Calibre to make an epub, and see how the final result looks.

Thank you!

You know, there's a world of difference between what an individual wants, for their own use and purposes (like putting it on your PPW) and what a professional needs. As long as you're reasonably happy with what you're getting and you're not using these commercially, why not stick with what works for you?

Hitch

[Tex2002ans]

Originally Posted by bleopaskom

I use Abbyy to convert PDFs to EPUB, and the conversions are almost perfect.

For Non-Fiction? With footnotes? Doubt it.

Perhaps for simple stuff like Fiction, you may be able to get near-perfect PDF->EPUB output from Finereader...

But with complicated Non-Fiction (footnotes/tables/charts), you'll need to manually correct lots of little issues.

See my response in 2017: "Converting pdf file of a scanned book to epub format":

Originally Posted by Tex2002ans

Depending on the export format, Finereader does try to do its best, but it botches the "linking back/forth footnotes" pretty badly. The only way to handle it is properly is to manually correct them.

There are some tools to kind of help speed up the process though:

[...]

You may also want to check out some of the other threads:

• 2016: "Epub3 Foot- End-notes"

or do a search for:

footnotes PDF tex2002ans site:mobileread.com
footnotes EPUB tex2002ans site:mobileread.com

Footnotes have been discussed to death.

Originally Posted by bleopaskom

Do I have to do something in Abbyy first?

[...]

There was maybe something about exporting to Word and fixing it there, or using Sigil, or just making use of Toxaris ePUB-Tools, but I don't really understand how to do it.

I'm still on Finereader 12, but I doubt much changed in 15.

Depending on the book, it might be easier to go:

• Finereader PDF
• -> Save As "Formatted Text" DOCX
• -> Word + Toxaris's EPUB Tools
• -> EPUB.

This will avoid some of the worst Finereader EPUB problems, because:

• Finereader's DOCX export is less buggy
  
  • The EPUB export sometimes randomly eats text, especially in footnotes.
    
• Toxaris's EPUB Tools was designed for stripping/cleaning/fixing a lot of Finereader's cruft.

Toxaris's tools will also generate a MUCH cleaner EPUB file for you to work from, saving you tons of time.

See my posts in 2020: "OCRing + EPUBing my first book: Tips?" (especially Post #15 where I explain some of that).

Originally Posted by bleopaskom

Converting PDF to EPUB and automatically creating popup footnotes

See my posts from 2020 in "I have a DOCX with footnotes that I wanna turn to pop-up notes in ePub on Kobo Forma"...

But there are lots of Finereader footnote issues that you're going to hit, like:

Originally Posted by Tex2002ans

• While many footnotes were detected properly, many weren't.
  
  • On top of that, the problem with PDF->DOCX "automated footnotes" is... the numbers may now be thrown way off. If 1-4 + 6-10 were detected fine... Word will only think there are "9 actual footnotes". 5 will be floating in the text, and 6-10 will now be off by 1.

Originally Posted by bleopaskom

Is there a Calibre addon that can help link them automatically?

No.

Learn regular expressions. They'll be your best friend.

Depending on the book/errors that crept in, it may be just as simple as changing:

<sup>5</sup>

into

<a href="#fn5" id="ft5">[5]</a>

but, much more likely, you'll have to renumber (+manually move) all your footnotes again.

Finereader may have automatically detected 90% footnotes, but accidentally missed 10%, leaving them in the text itself.

When Finereader exports to other formats, it links + "helpfully" renumbers all your footnotes for you.

Original PDF:

As studies show,<sup>1</sup> brown cows do not
produce brown milk.<sup>2</sup>

This is another sentence with a footnote.<sup>3</sup>

- - -

<sup>1</sup> One. The studies.

<sup>2</sup> Two. They produce white milk.

<sup>3</sup> Three. More footnote.

Let's say Finereader correctly detected Footnotes 1+3, but missed 2:

Finereader EPUB:

<p>As studies show,<a id="footnote1"></a><sup><a href="#bookmark1">1</a></sup> brown cows do not
produce brown milk.<sup>2</sup></p>

<p>This is another sentence with a footnote.<a id="footnote2"></a><sup><a href="#bookmark2">2</a></sup></p>

<p><sup>2</sup> Two. They produce white milk.</p>

[...]

- - -

<p><a id="bookmark1"></a><a href="#footnote1">1</a> One. The studies.</p>

<p><a id="bookmark2"><a href="#footnote2">2</a> Three. More footnote.</p>

Whoops, now you have to correct all the numbering. AND you have to manually move footnote 2 into its proper location.

... No automated tool can magically find/correct this stuff. You have to manually spot and fix this.

(There are a few tricks/speedier ways, but it's still a lot of manual legwork + various methods you have to stitch together. Every single book is going to have completely different issues crop up.)

Originally Posted by bleopaskom

A step-by-step guide would be appreciated.

Doesn't exist.

There's too many variables.

All the individual pieces have been explained on MobileRead though, in extreme depth.

Whatever problem you have, you can usually type this in your favorite search engine:

type out your exact problem Tex2002ans site:mobileread.com
type out your exact problem Hitch site:mobileread.com

and me and/or Hitch probably answered it already.

But what you want: a simple PDF->EPUB, one-button-press, turnkey solution... it just doesn't exist.

[Asteros65]

Where do you get Toxari's Epub tools? His webpage is not available

[Hitch]

Oh, dear. I wonder if he knows that?

Hitch

[Doitsu]

Originally Posted by Asteros65

Where do you get Toxari's Epub tools? His webpage is not available

Archive.org

[Tex2002ans]

Originally Posted by Asteros65

Where do you get Toxari's Epub tools? His webpage is not available

I have a copy of his latest version (1.27.0) saved.

I attached it to this post.

Note: I put it in a ZIP file so MobileRead will accept it. You'll have to unzip it to get the original MSI file.

Originally Posted by Hitch

Oh, dear. I wonder if he knows that?

Hmmm. I visited one of my old links about a month ago, and looks like his site didn't redirect correctly from his older English link:

- toxaris.nl/en

to his newer redesigned homepage:

- toxaris.nl

correctly. Wonder if he was poking around or updating the old website in the redesign.

And now it does look like it's currently down (404 error). Must've happened very recently.

PS. I miss my great friend Toxaris.

- - -

Side Note: For future info, here's the original link to a previous version of EPUB Tools (1.26.1):

http://www.mediafire.com/file/ltyh69...26.1-setup.msi

The URL I have saved for 1.27.0 isn't working anymore, but the previous ones are. According to his changelog, there was only one small thing added between 1.26->1.27:

v1.27 (10/12/2017)
New: Option to remove hidden text during conversion to HTML/ePUB.

[BetterRed]

1.27 working fine here - maybe even better than ever, the Add-on hasn't gone AWOL for months, nor have I had any Dialogue Check crashes of late.

But Toxaris logged on a few hours ago and --- look at the very last line ==>> https://www.mobileread.com/forums/sh...90&postcount=1

I can't see what he did, I'll ask an admin to have a look and let me know.

Strange he didn't post a calling card

BR

[Hitch]

Originally Posted by BetterRed

1.27 working fine here - maybe even better than ever, the Add-on hasn't gone AWOL for months, nor have I had any Dialogue Check crashes of late.

But Toxaris logged on a few hours ago and --- look at the very last line ==>> https://www.mobileread.com/forums/sh...90&postcount=1

I can't see what he did, I'll ask an admin to have a look and let me know.

Strange he didn't post a calling card

BR

I emailed him and he said he was going to post a link to it. He's decided that he doesn't need his website.

Hitch

[BetterRed]

Originally Posted by Hitch

I emailed him and he said he was going to post a link to it. He's decided that he doesn't need his website.

Hitch

I suspected that's what he'd done - added the zip.

His website had the doco, which was pretty darn good

It's available at the wayback link Doitsu posted

BR