EPUB3 Footnote Fixer-Formatter

WiseGuru · 02-08-2024, 02:08 PM

I recently discovered that you could read ebooks with the footnotes at the bottom of the page like in a real book (instead of a link to the end matter), and this lit my brain on fire. I've never seen this before, and it feels so obvious that it should just... be how it is for all books. This led me on a personal missing to convert all the footnotes for all of my ebooks into that properly formatted structure. I'm using KOReader as my ereader app of choice, and that's the one I'm testing with and want to make sure it works on.

Here's what I know so far.
1. The books have to be in EPUB3
2. The minimum effort required is to wrap the body of the footnote with an aside tag, like `<aside epub:type="footnote" id="[whatever the ID is]">`
3. This is complicated AF

I have two scripts (ChatGPT wrote them and I provided feedback and logs) that attempt to wrap the body of the footnote text with `<aside epub:type="footnote" id="[whatever the ID is]">`. Unfortunately, the footnotes are formatted differently from book to book (even with the same author and publisher), and the scripts have not worked consistently for most books, or even consistently within a single book.

For example, I bought 38 Terry Pratchett books from Humble Bundle. After converting the EPUB2 books to EPUB3, one of the scripts successfully converted around 20 books (although it left some nested footnotes that were not caught by the script), and the second script caught another 10 or so. In the end, there are 5 books that I have not been able to automatically convert.

At this point, I think there are two options; fix the scripts so that they wrap the footnotes in the aside tag, OR create a new script that scrapes the footnote text and completely rebuilds the footnotes using a cleaner format.

The new, cleaner format would look something like this:

Code:

<!--Converting to single lines-->
<!--Inline-->
<p>This is some text that needs a footnote.<a href="FN_Section0001.xhtml#fn1" epub:type="noteref" id="bookfn1">[1]</a></p>

<!-- Footnote content -->
<body>
<aside epub:type="footnote" id="fn1">
  <p>This is the content of the footnote. <a href="9780061807145_Chapter_1_split_000.xhtml#bookfn1">↩</a></p>
</aside>
</body>

If you want to take a look, I have the two python scripts available for review, as well as 70 ebooks (About 109 MB) with the copywritten information scrambled for script development and testing, linked here: Dropbox.

If anyone has any ideas or suggestions, I'm all ears; as it is, I will keep posting here as I experiment and make progress.

Quoth · 02-08-2024, 05:00 PM

What ereaders and apps have you tested it on?

WiseGuru · 02-08-2024, 05:26 PM

Quote:

Originally Posted by Quoth

What ereaders and apps have you tested it on?

I'm using a PocketBook Era with KOReader, and then ReadEra Premium on my android devices. The PocketBook reader app does not display the footnotes like KOReader or ReadEra do. I have not tried importing the modified EPUBs to Kobo or Kindle.

And just to clarify, the scrambled books are not currently functional; that said, I should probably scramble some of the ones I've updated for reference.

paperwhite13 · 02-09-2024, 04:19 AM

I’ve recently updated KOReader on my PW3 to v2024 after using v2020 for the past four years and the .epub I’m working on, exported as EPUB 3 from InDesign (CC2019), has footnotes appearing at the bottom at the page, even though I'm not using <aside> (they’re exported as end-of-chapter endnotes). The other epubs I've checked don't do this; I've yet to check the code or the epub version of each.

WiseGuru · 02-09-2024, 01:03 PM

Quote:

Originally Posted by paperwhite13

... KOReader on my PW3 to v2024... has footnotes appearing at the bottom at the page, even though I'm not using <aside> (they’re exported as end-of-chapter endnotes).

Very interesting! Any chance you could share the HTML formatting?

Like I said, it's been really weird and inconsistent, and I'm sure part of my struggles have to do with converting EPUB2 to EPUB3 books (EPUB2 books not having the formatting options that EPUB3 books do).

When I started this project, I had two functional points of reference; "A City on Mars" by Kelly and Zach Weinersmith (highly recommend), and the sample I shared in the original post (which is based on an old post I can't find, but also this one from Apple for its Pop-up Footnotes).

I tried removing and adding elements to the books "Sourcery" and "Pyramids" by Terry Pratchett, and eventually landed on wrapping the footnote with the <aside> tag.

WiseGuru · 02-09-2024, 01:35 PM

I somehow never came across this post here on mobileread by Tex2002ans which discusses the standardization of footnote formatting in EPUB3, and how certain devices display that information. It corroborates what I said earlier about the <aside> tag etc., though in my personal testing I also needed the footnote ID (e.g. <aside epub:type="footnote" id="note1") included for it to render properly on KOReader.

Also I forgot to mention, I'm using KOReader 2024.01 on my phone, and I believe the same version on my PocketBook Era, but whatever version is on my Era is rendering the ebooks the same way as on my phone.

paperwhite13 · 02-10-2024, 02:57 AM

Quote:

Originally Posted by WiseGuru

Very interesting! Any chance you could share the HTML formatting?.

Sure thing, but as far as I can tell, there's nothing special about it (or optimal, either). Of all the readers I've tried, only KOReader does this.

This is from the end-of-chapter endnotes section:

Code:

<div class="_idFootnotes">
. . .
<div id="footnote-154" epub:type="footnote">
<p class="Footnote"><a href="#footnote-154-backlink">1</a>. L. Chiarini, <em>Arte e tecnica del film</em>, Ban, Laterza, 1962, p. 7.</p>
. . .
</div>

paperwhite13 · 02-10-2024, 03:37 AM

I tried ReadEra (regular version) on my Android device and with my epub it doesn't do the page-bottom footnotes thing. They say ReadEra is based on CoolReader, but that didn't do it either. So, for me, it's still just KOReader displaying this behaviour.

Quoth · 02-10-2024, 06:21 AM

And KOReader, while interesting, is not pre-installed on anything, nor on Playstore. I installed it from Git on my Android devices.

So basically this idea is tested on hardly anything that people commonly use and doesn't work on Pocketbook Android App, which is free and has 5M+ downloads or regular Readera which has 10M+ downloads.

Kindle has about 92% of English language market, with majority reading on the Kindle app (Android, iOS, Mac, Windows, Amazon Fire), mostly on phones or tablets.
Most Amazon eink in use is now able for KFX. All run AWZ3/KF8 except the K1, K2, DX and DXG, which are really old. Likely few K1 or K2 are running due to exhusted battery/cell).

Edit:
Playstore has 100M+ for Kindle app downloads. Amazon Fire (one of the best selling tablets) has Kindle App pre-installed.

OTOH, Google claims 1 Billion+ downloads for Google PlayBooks & Audio Books app, yet Google has the lowest sales of any International ebook retailer.

paperwhite13 · 02-10-2024, 06:48 AM

Quote:

Originally Posted by Quoth

And KOReader, while interesting, is not pre-installed on anything, nor on Playstore. I installed it from Git on my Android devices.

So basically this idea is tested on hardly anything that people commonly use and doesn't work on Pocketbook Android App, which is free and has 5M+ downloads or regular Readera which has 10M+ downloads.

Kindle has about 92% of English language market, with majority reading on the Kindle app (Android, iOS, Mac, Windows, Amazon Fire), mostly on phones or tablets.
Most Amazon eink in use is now able for KFX. All run AWZ3/KF8 except the K1, K2, DX and DXG, which are really old. Likely few K1 or K2 are running due to exhusted battery/cell).

Edit:
Playstore has 100M+ for Kindle app downloads. Amazon Fire (one of the best selling tablets) has Kindle App pre-installed.

OTOH, Google claims 1 Billion+ downloads for Google PlayBooks & Audio Books app, yet Google has the lowest sales of any International ebook retailer.

I wouldn’t say it's exactly untested, I think most jailbroken Kindles and Kobos use KOReader, whether the owners are native English speakers or not. Also, in my opinion, sales figures aren't necessarily the same with the number of books read

There could be many readers reading many pirated ebooks on such jailbroken devices, therefore testing it extensively.

Quoth · 02-10-2024, 07:03 AM

Jailbroken Kindles is a tiny percentage of Kindle users. Pretty much only is done for KOReader. Also up-to-date FW usually can't be jailbroken.

I do have a jailbreak on a Kindle DX, partly out of curiosity and party to set the time, but not any other Kindle and none of the many Kindle users I know have any interest in Jailbreak or KOReader.

Also you can read a pirated book on any ereader or app. It's the DRM'ed commercially sold or library loans that take expertise to read on a different device to that intended, and that doesn't need a jailbreak either.

Jailbreaking is irrelevant to reading ebooks and the only thing that absolutely needs it is a wish to to use KOReader on a Kindle.

Quoth · 02-10-2024, 07:18 AM

Quote:

Originally Posted by paperwhite13

Also, in my opinion, sales figures aren't necessarily the same with the number of books read

Oh, that's not an opinion. It's fact, which why Amazon has the stupid system for KU. Real libraries in UK & Ireland both have to buy the book (or licence the ebook) and pay a per loan royalty. It's irrelevant if the book is read. Amazon's system of getting the content free, deciding on a pot and then sharing that according to pages read cheats authors/publishers and invades the privacy or readers/users/public. Amazon's KU is wrong, as is getting all but large publisher content exclusively. An author or small publisher can't even give away copies free if they are on KU.

But it's sales that feeds and houses the authors.

paperwhite13 · 02-10-2024, 10:47 AM

Quote:

Originally Posted by Quoth

Oh, that's not an opinion. It's fact, which why Amazon has the stupid system for KU. Real libraries in UK & Ireland both have to buy the book (or licence the ebook) and pay a per loan royalty. It's irrelevant if the book is read. Amazon's system of getting the content free, deciding on a pot and then sharing that according to pages read cheats authors/publishers and invades the privacy or readers/users/public. Amazon's KU is wrong, as is getting all but large publisher content exclusively. An author or small publisher can't even give away copies free if they are on KU.

But it's sales that feeds and houses the authors.

I agree with everything

Quote:

Originally Posted by Quoth

Jailbroken Kindles is a tiny percentage of Kindle users. Pretty much only is done for KOReader. Also up-to-date FW usually can't be jailbroken.

I do have a jailbreak on a Kindle DX, partly out of curiosity and party to set the time, but not any other Kindle and none of the many Kindle users I know have any interest in Jailbreak or KOReader.

Also you can read a pirated book on any ereader or app. It's the DRM'ed commercially sold or library loans that take expertise to read on a different device to that intended, and that doesn't need a jailbreak either.

Jailbreaking is irrelevant to reading ebooks and the only thing that absolutely needs it is a wish to to use KOReader on a Kindle.

Up until 2022, it wasn't as easy to get epubs on Kindle. On top of that, the Kindle reading app is very limited in comparison. Without it, my PW3 would've been an ok reader, but with KOReader and its many features, it's been nothing short of amazing. I wouldn't even have had Night Mode -- or dictionary-based hyphenation, Calibre connect etc. In my opinion, it's the best reader out there.

And the in page footnotes feature is great too, for me it doesn't matter it's not available in other apps. Perhaps it'll make even more people try it.

paperwhite13 · 02-10-2024, 10:56 AM

So, I'm certain you don't need <aside> for this to work (just enable the In Page Footnotes style tweak and you're good to go). As to why doesn't work with the other epubs I've tried, I've yet to determine, I'm certain most of those are EPUB3 as well. Maybe it's something really small, but necessary for this to work.

WiseGuru · 02-10-2024, 01:03 PM

Quote:

Originally Posted by Quoth

And KOReader, while interesting, is not pre-installed on anything, nor on Playstore. I installed it from Git on my Android devices.

So basically this idea is tested on hardly anything that people commonly use and doesn't work on Pocketbook Android App, which is free and has 5M+ downloads or regular Readera which has 10M+ downloads.

In fairness, I did say that I want to make this work with KOReader specifically. Adding ereaders that require you to convert from EPUB3 to other proprietary formats (e.g., Kindle) adds significant complexity and uncertainty to an already complicated project. Additionally, this only works on ebooks without DRM (since you have to modify the contents of the ebook itself), so it requires users to have a certain base-level of technical know-how and willingness to get in the weeds to even get started.

Also to clarify on ReadEra, I hadn't tested on the regular version since I upgraded to Premium and didn't have it installed. However, installing it, the two apps perform the same as Premium and displays footnotes at the bottom of the page (tested with "A City on Mars" by Kelly and Zach Weinersmith out-of-the-box, and other ebooks I've modified with <aside> tags). In my experience, it's more complicated to test books in ReadEra (since it creates a library index, and takes more taps to navigate to the new test file you just created), so I've been focusing on KOReader since it's more flexible for testing and it's what I use on my PocketBook Era.

Quote:

Originally Posted by paperwhite13

This is from the end-of-chapter endnotes section:

Code:

<div class="_idFootnotes">
. . .
<div id="footnote-154" epub:type="footnote">
<p class="Footnote"><a href="#footnote-154-backlink">1</a>. L. Chiarini, <em>Arte e tecnica del film</em>, Ban, Laterza, 1962, p. 7.</p>
. . .
</div>

Very interesting, I've got a book that also has a <div> tag with the ID and similar epub:type="footnote" attributes (IIRC), but it doesn't appear at the bottom of the page as is. From what I can see, the official syntax asks for an <aside> tag, so I would guess that if KOReader accepts the <div> tag as well, that's just KOReader being a cool app

If you want to share a scrambled version of the ebook you're reading, I'd like to take a look at it and do some tests.

I think the inline link also needs to be formatted a certain way. In "Lords and Ladies", I've been able to get the <aside> tag around the footnote text, but it only appears at the bottom of the page if I also modify the inline link.

I'm going to upload my latest version of the script (name suffix "2024.02.10", which I modified to be more inclusive and works better than the other two) so you can play with it if you'd like.

Additionally, sometime this weekend I'll update the Dropbox folder to include both the unmodified Scrambled epubs I have along with the modified versions as changed by the latest iteration of the script and the script's log output.

02-08-2024, 02:08 PM	#1
WiseGuru More Wiserer Posts: 9 Karma: 10 Join Date: Jan 2024 Device: Pocketbook Era (KOReader)	EPUB3 Footnote Fixer-Formatter I recently discovered that you could read ebooks with the footnotes at the bottom of the page like in a real book (instead of a link to the end matter), and this lit my brain on fire. I've never seen this before, and it feels so obvious that it should just... be how it is for all books. This led me on a personal missing to convert all the footnotes for all of my ebooks into that properly formatted structure. I'm using KOReader as my ereader app of choice, and that's the one I'm testing with and want to make sure it works on. Here's what I know so far. 1. The books have to be in EPUB3 2. The minimum effort required is to wrap the body of the footnote with an aside tag, like `<aside epub:type="footnote" id="[whatever the ID is]">` 3. This is complicated AF I have two scripts (ChatGPT wrote them and I provided feedback and logs) that attempt to wrap the body of the footnote text with `<aside epub:type="footnote" id="[whatever the ID is]">`. Unfortunately, the footnotes are formatted differently from book to book (even with the same author and publisher), and the scripts have not worked consistently for most books, or even consistently within a single book. For example, I bought 38 Terry Pratchett books from Humble Bundle. After converting the EPUB2 books to EPUB3, one of the scripts successfully converted around 20 books (although it left some nested footnotes that were not caught by the script), and the second script caught another 10 or so. In the end, there are 5 books that I have not been able to automatically convert. At this point, I think there are two options; fix the scripts so that they wrap the footnotes in the aside tag, OR create a new script that scrapes the footnote text and completely rebuilds the footnotes using a cleaner format. The new, cleaner format would look something like this: Code: <!--Converting to single lines--> <!--Inline--> <p>This is some text that needs a footnote.<a href="FN_Section0001.xhtml#fn1" epub:type="noteref" id="bookfn1">[1]</a></p> <!-- Footnote content --> <body> <aside epub:type="footnote" id="fn1"> <p>This is the content of the footnote. <a href="9780061807145_Chapter_1_split_000.xhtml#bookfn1">↩</a></p> </aside> </body> If you want to take a look, I have the two python scripts available for review, as well as 70 ebooks (About 109 MB) with the copywritten information scrambled for script development and testing, linked here: Dropbox. If anyone has any ideas or suggestions, I'm all ears; as it is, I will keep posting here as I experiment and make progress.

02-10-2024, 03:37 AM	#8
paperwhite13 Zealot Posts: 131 Karma: 9236 Join Date: Jun 2020 Device: Kindle PW3 [KOReader]	I tried ReadEra (regular version) on my Android device and with my epub it doesn't do the page-bottom footnotes thing. They say ReadEra is based on CoolReader, but that didn't do it either. So, for me, it's still just KOReader displaying this behaviour. Last edited by paperwhite13; 02-10-2024 at 03:42 AM.

02-10-2024, 06:21 AM	#9
Quoth Still reading Posts: 13,631 Karma: 103503445 Join Date: Jun 2017 Location: Ireland Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper	And KOReader, while interesting, is not pre-installed on anything, nor on Playstore. I installed it from Git on my Android devices. So basically this idea is tested on hardly anything that people commonly use and doesn't work on Pocketbook Android App, which is free and has 5M+ downloads or regular Readera which has 10M+ downloads. Kindle has about 92% of English language market, with majority reading on the Kindle app (Android, iOS, Mac, Windows, Amazon Fire), mostly on phones or tablets. Most Amazon eink in use is now able for KFX. All run AWZ3/KF8 except the K1, K2, DX and DXG, which are really old. Likely few K1 or K2 are running due to exhusted battery/cell). Edit: Playstore has 100M+ for Kindle app downloads. Amazon Fire (one of the best selling tablets) has Kindle App pre-installed. OTOH, Google claims 1 Billion+ downloads for Google PlayBooks & Audio Books app, yet Google has the lowest sales of any International ebook retailer. Last edited by Quoth; 02-10-2024 at 06:36 AM.

02-10-2024, 07:03 AM	#11
Quoth Still reading Posts: 13,631 Karma: 103503445 Join Date: Jun 2017 Location: Ireland Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper	Jailbroken Kindles is a tiny percentage of Kindle users. Pretty much only is done for KOReader. Also up-to-date FW usually can't be jailbroken. I do have a jailbreak on a Kindle DX, partly out of curiosity and party to set the time, but not any other Kindle and none of the many Kindle users I know have any interest in Jailbreak or KOReader. Also you can read a pirated book on any ereader or app. It's the DRM'ed commercially sold or library loans that take expertise to read on a different device to that intended, and that doesn't need a jailbreak either. Jailbreaking is irrelevant to reading ebooks and the only thing that absolutely needs it is a wish to to use KOReader on a Kindle. Last edited by Quoth; 02-10-2024 at 07:11 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Issue with footnote pop-up (epub3)	carmenchu	Viewer	5	01-06-2021 04:59 AM
EPUB3 footnote validation	roger64	Workshop	11	06-03-2015 07:56 AM
Mobi formatter wanted	AlexBell	Kindle Formats	4	04-19-2011 05:34 AM
Antenna House Formatter	canes816	Calibre	1	04-01-2010 12:46 AM
iLiad RFCToiLiad formatter application	scotty1024	iRex Developer's Corner	0	10-16-2006 12:56 PM

02-08-2024, 05:00 PM	#2
Quoth Still reading Posts: 13,631 Karma: 103503445 Join Date: Jun 2017 Location: Ireland Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper	What ereaders and apps have you tested it on?

02-09-2024, 04:19 AM	#4
paperwhite13 Zealot Posts: 131 Karma: 9236 Join Date: Jun 2020 Device: Kindle PW3 [KOReader]	I’ve recently updated KOReader on my PW3 to v2024 after using v2020 for the past four years and the .epub I’m working on, exported as EPUB 3 from InDesign (CC2019), has footnotes appearing at the bottom at the page, even though I'm not using <aside> (they’re exported as end-of-chapter endnotes). The other epubs I've checked don't do this; I've yet to check the code or the epub version of each.

02-09-2024, 01:35 PM	#6
WiseGuru More Wiserer Posts: 9 Karma: 10 Join Date: Jan 2024 Device: Pocketbook Era (KOReader)	I somehow never came across this post here on mobileread by Tex2002ans which discusses the standardization of footnote formatting in EPUB3, and how certain devices display that information. It corroborates what I said earlier about the <aside> tag etc., though in my personal testing I also needed the footnote ID (e.g. <aside epub:type="footnote" id="note1") included for it to render properly on KOReader. Also I forgot to mention, I'm using KOReader 2024.01 on my phone, and I believe the same version on my PocketBook Era, but whatever version is on my Era is rendering the ebooks the same way as on my phone.

02-10-2024, 10:56 AM	#14
paperwhite13 Zealot Posts: 131 Karma: 9236 Join Date: Jun 2020 Device: Kindle PW3 [KOReader]	So, I'm certain you don't need <aside> for this to work (just enable the In Page Footnotes style tweak and you're good to go). As to why doesn't work with the other epubs I've tried, I've yet to determine, I'm certain most of those are EPUB3 as well. Maybe it's something really small, but necessary for this to work.