Footnote problem in kepub conversion

[Liudprand]

I'm not 100% sure this is the right place to post this, but I wasn't sure where else to go. I think it's a Calibre issue, but I may be wrong.

Basically, I have a quite large book (240k words, 40 chapters) that I've converted from a pdf into Word so that I can copy-edit it properly, with a view to then converting it to an epub on Calibre, and finally using KoboTouchExtended for kepub conversion onto my Libra 2.

Everything seems to work fine, formatting-wise, except for an issue I have with the notes. Even though, in Word (2010), the note numbering behaves itself, re-starting at the beginning of each section (and a section break at the end of each chapter), as I've specified, the epub file that Calibre outputs when I convert from docs has the words running in one continuous sequence throughout the text (and therefore the kepub has the same).

Calibre seems able to preserve all other aspects of formatting (or nearly all) when converting from Word to epub - which is why I tend to use it for this purpose.

Is there any straigthtfoward fix for this?

[Liudprand]

I know this is possible in principle, because this tool:

https://daisy.org/activities/software/wordtoepub/

does the job of retaining the section breaks, and thus the footnote numbering. Unfortunately, it messes up a few other things (paragraph indention specified in Word being the most obvious).

So, there must be some way of coding this behaviour into Calibre's conversion process, surely... ?

[Liudprand]

Having messed around with this in various ways trying to find a solution, it doesn't seem to have anything to do with KTE - it seems that Calibre itself is unable to preserve footnote numbering hygiene between chapters. Can this really be true... ?!

[Quoth]

Don't use Word (or LO ) footnote numbering, or indeed "Insert footnote/endnote" feature at all, it has to be manually done. Explicitly number and link the word and number as one link to an explicitly numbered separate paragraph with anchor/bookmark. Put an anchor/book mark also at source and the word [back] with a link.

Why?
Delete a source number and the associated note can vanish in Word.

Never auto number lists or footnotes. Explicitly add them for 100% reliable.

have a psuedo chapter of notes, one per paragraph after the chapter so that this is separate file in the epub.

Only one-to-one footnotes. Repeat it if used more than once, because every note needs a text link [back], because many apps and ereaders have no "back" button, or the footnote may not be a popup.

Ebooks use html and css but are not websites.
Automatic footnote / endnote generation is buggy in Word and LO Writer, especially feature where if you delete reference in source paragraph the note can vanish.

Only use a numbered superscript on files for paper. For ebook use regular size and make preceeding word also part of the link. People have fat fingers compared to [2] to tap on.

Don't use Insert footnote/endnote (automatic process) at all, in any wordprocessor. Type a paragraph for each note and link manually, to and back.

For lists, create a paragraph style that mimics a list and type the numbers explicitly. Also beware fonts needed for bulleted lists. Automatic list support and especially numbering is erraticly or not at all supported in ebooks.

[Liudprand]

Thanks for the reply - most of which I understand!

What I'm finding frustrating is that it's difficult to find any step-by-step guide, either online or off (I'm talking about several books on ebook formatting that I've paid cold, hard cash for) for this kind of thing. I've looked at several books, and NONE of them has anything to say about this question. I'm very unfamiliar with HTML and CSS, so Word (or InDesign) are the only tools I can use to edit a epub file at all.

An example of my enduring confusion on this: Do you mean that the link to the paragraph with the note text has to be two-way, as it will eventually be in the ebook? That's not possible in Word, afaik. But does Calibre fix that in the conversion process?

And you say Word footnotes are "buggy". That may well be true in various senses - but I've been using them as a professional book editor for about 18 years now, and never had a problem. If you format everything correctly in Word, most things transfer cleanly to, for example, InDesign. Also, the Word add-on I mentioned above - WordToEpub - makes a perfectly good job of numbering notes within chapters when you make the conversion. So I'm confused about why other pieces of software, including Calibre, can't just implement the same behaviour. I don't mean it's not a significant amount of work, or that the developer of Calibre should get on it right away - I just mean in principle I don't see why it's not perfectly feasible - since both InDesign and WordToEpub have implemented it.

Anyway, thanks again for the response. And, by the way, I don't expect you to respond to my follow-ups unless you want to. I would, though, appreciate it if you or anyone else could point me to the most comprehensive, professional source available from which I could teach myself all of this stuff.

[Quoth]

Originally Posted by Liudprand

I'm very unfamiliar with HTML and CSS, so Word (or InDesign) are the only tools I can use to edit a epub file at all.

An example of my enduring confusion on this: Do you mean that the link to the paragraph with the note text has to be two-way, as it will eventually be in the ebook? That's not possible in Word, afaik. But does Calibre fix that in the conversion process?

It's simple in word!
In Word (same in LO Writer) you can add (insert) a bookmark at the start of a paragraph (actually anywhere, but a paragraph start is best). Do not highlight/select text. Just click where you want it.

Then anywhere else you can select a block of text (last word and[7] for instance) and Add Hyperlink/URL. Select target is document and you'll get a list of bookmarks. I'd use a naming scheme for the footnote bookmarks like fc6n3 for 3rd footnote in chapter 3. You can even type the bookmark directly, I forget how in Word but in LO Writer the url would be #fc6n3 instead of starting with http:// or https://

So you can create Chapter 6: Notes as a heading (then that gets a separate file in the epub) and a paragraph for each note, each ending with the word [back]

So each note starts with an invisible inserted bookmark (anchor) and ends with a visible link called [back], which uses an invisible bookmark (anchor such as url #c6n3) at the start of the paragraph that ends with a link to #fc6n3. That would be the word and [3] (no superscript).

It's simpler than it sounds.

The ONLY editing of epub I do after creation by adding a docx to calibre is some images need to fit width or height and the image properties in Wordprocessor can't seem to do that. That's two lines changed for the maybe 4 images needing it.

MS Word has been able to do this for decades.
I used MS Word almost exclusively 1997 to 2014 when I 100% switched to LO Writer on Windows.

So I make the wordprocessor file for epub first (no headings, footers all same page type and smaller than A5, no registration, no line spacings). That's proofed on an ereader.

Then for paper version you do page styles on a copy for title, front matter, contents, start of chapter, left page, right page, index, notes, appendix, end matter etc, with appropriate headers & footers, inner & outer margins. You might change fonts and sizes. You might use specialist spaces between numbers and units. Other print stuff that's risky in ebooks.
Then you can delete all the hyperlinks (if the print company objects). The bookmarks are OK. The footnote references can be found and changed to superscript.

InDesign is fudged for ebooks, it should only be used for paper and really books don't need it now, only magazines & papers. Maybe some textbooks might. MS Word (2007 or later) or LO Writer (nearly any version) can perfectly produce docx for Calibre to make an ebook, and a proofed and differently formatted copy can export directly (Earlier MS Word needs a plugin) to PDF. LO Writer has exported PDF perfectly at least a decade.

Note only odt should be edited with LO Writer, an extra Save As is used to make the docx.

Anyway for perfect epubs you only need to know how to use MS Word with styles, outlining and internal hyperlinks to bookmarks (=HTML anchors), all of which Word has done for over 20 years, though before Word 2007 you had to retrofit a plugin for docx and on 2002/XP and 2003 it only let you import docx.

This isn't a Calibre problem!

[Quoth]

Also InDesign produces terrible ebooks and you have to rent it. It's a Desktop Publishing system for paper!
WordToEpub makes poor ebooks, even if it understands footnotes.

The MS Word footnote system only works properly for PDF/Print or another user on a compatible version of MS Word. It can also break just editing (notes get deleted if reference is deleted).

So now we (not just me) do footnotes or endnotes manually as described.

I've a copy of Word 2007 somewhere on a VM

[Liudprand]

Thanks again.

I still don't know where I can go to get a proper, soup-to-nuts education on all this. I can't understand why no one seems to have written a book that assembles it all in one place, so that you have to stalk these forums and bother people like you.

One point, though. Among other things, I'm a professional typesetter of nonfiction books. Every publisher I've worked with is still very much relying on InDesign. If I tried to submit a "typeset" book to them in Word, I don't think I'd ever hear from them again. Among many other things, Word can't handle proper kerning, wordspacing, ligatures - and much else besides. I know they're not relevant for ebooks, but they're absolutely central to professional typesetting of books.

[Liudprand]

By the way, I'm using Word 2010 (2013 messed up several features I found, and still find, useful, apparently to make the interface "look cleaner" or some such garbage); and InDesign CS6 - the last pre-subscription version, which is perfectly adequate for my purposes.

[kovidgoyal]

No there is no fix for this. Remember that in ebooks the actual number used is not important since your readers will read footnotes by tapping the footnote link not scrolling to a list of foootnotes and looking up a number.

[Liudprand]

And there's one other thing I maybe should have made clearer at the start.

I'm not writing, or formatting-from-scratch, an ebook of my own. The project I'm working on involves converting a 260,000-word history book (with around 500 endnotes) from a pdf scan. Obviously, this involves lots of copy-editing, removal of page-headers, etc. But my instructions include reproducing the notes in the form they appear in in the original book - so, no highlighting of words in the text: just the note cue itself. The whole process involves a detour through Word, which is where I've done all the editing/proofing.

I completely get what you mean when you say it's "not a Calibre problem", and obviously the developer has no obligation to incorporate the functionality I sorely need. But, to this normie (at least in the specific area of ebook arcana), it seems very peculiar indeed to suggest that, rather than implementing a solution resembling that included in WordToEpub, I should be going through the whole book to format each note manually, which means many hours of extra work and the almost-certainty of introducing new errors. It's much more convenient for me, and I imagine many others in a similar situation, to rely on the numbering (sometimes re-numbering) of footnotes that is one of Word's simple virtues in this area. So, it still seems very strange to me that it's not possible to "bank" all the work I've done in Word and simply export into epub.

Needless to say, this isn't a shortcoming that's in any way exclusive to Calibre!

[Liudprand]

Originally Posted by kovidgoyal

No there is no fix for this. Remember that in ebooks the actual number used is not important since your readers will read footnotes by tapping the footnote link not scrolling to a list of foootnotes and looking up a number.

Thanks for this, Kovid.

I should have explained that I'm working on a book that was published way back in the analogue era, and it contains quite a few cross-references to notes in other chapters ("See Chapter 8, note 37", or whatever). I realise that this could be worked around by rephrasing/editing each of those cross-refs, but that's not what the publisher, or the author, wants. And I should mention, as someone who works regularly on profesionally published nonfiction, that this is a formulation ("See Chapter X, note Y") that's still widely used in freshly published books.

I guess I need some kind of conversion tool that meets paper/analogue publishing standards and traditions at least half-way, and Calibre isn't it (though it's enormously useful to me in all kinds of other ways!).

[kovidgoyal]

are these cross references not links themselves? That would be a very poor user experience, if they were not. If you use the footnote tool in word so that these are dynamic text linking to the actual footnote, the numbers in the converted document will be the correct number in the final footnote list. It wont be of chapter/note form, but again in an ebook that is pointless.

[Quoth]

You can't or shouldn't exactly duplicate how a paper book works as an ebook. It does links back and forth well. But you absolutely can't mimic the paper style of a source for a footnote. People need a larger area to tap on.

The ONLY sensible way to do it is how I suggest. I know it's more work, but it's far better quality, more reliable and safer to manually number.

Word (and InDesign) are designed for paper. Both predate ebooks (InDesign is ancient and kludged and may have bits of Pagemaker in it). Though ebooks are based on HTML and CSS, they certainly don't work the same way as web pages or web sites.

I've been creating content on computers since 1979 and used to teach this stuff. Do things to suit epub and get a Sage 8″ Kobo to proof on and copy back annotations to Calibre, then copy/paste to a text editor window beside Word or LO Writer.
It's then trivial to edit styles and page formats of a copy of Word/LO file to suit paper/POD/PDF. Working the other way (editing & styling for paper and trying to get an epub to work) is very very much harder.

It makes no difference that you are editing OCR from an old paper edition rather than writing from scratch. It just means the creative bit is replaced by proof comparison with the paper.

[Liudprand]

Well, they're not links in the original book, because it was published in 1998. And only on paper. I get what you mean - but my preferred solution (maybe I should say my "preferred reality"!) would be to add a link to the cross reference - "See Chapter 8, note 37" - and for that to point to the relevant note, which would be numbered "37", not 465, or whatever. I haven't consulted my client in detail on this - but I do know that preserving the original language and style of the original book, as much as possible, is a priority.