Standardizing ePub annotations

[DarkTrick]

Hello,

(1) I was wondering: why is there no standard to store annotations in an ePub book? Perhaps some insights?

(2) Couldn't you write highlights and annotations directly into the document? After all it's basically XML. So you would simply need to add a tag around a passage marking the area and adding text information for the notes.
How would that work in conjunction with DRMs?


Regards
DarkTrick

[Turtle91]

Mostly "adding to" or "changing" a copywritten publication (ePub) is a big no-no.

Besides the legal aspects, most reading apps don't want to spend the extra time, and/or overhead, to develop the functionality to store annotations in a linked document.

Having said that, there ARE some apps that allow annotations...you just need to track down which ones do it the way you want. (For example: Marvin on iOS, spent a lot of time making annotations possible)

[Quoth]

Originally Posted by DarkTrick

Hello,

(1) I was wondering: why is there no standard to store annotations in an ePub book?

(2) Couldn't you write highlights and annotations directly into the document?

3) How would that work in conjunction with DRMs?

1. Because it's pointless for almost anyone interested. A separate database or text file is better.

2. Madness. You could create an extra file in the epub (it's just a zip folder) for each existing file to append highlight, position, date, time and comment. But a separate file outside the epub is better. Or that the reader/app database has it and can export it as text (Kobo). Really crazy to edit the existing files inside the epub.

3. Depends on the DRM. But no copyright holder is going to agree to annotations being INSIDE the ebook. There is also no need.

4. Then there is Amazon, who have over 90% of the English Language ebook market partly due to people being conned into giving Amazon exclusivity or allowing titles in Amazon's subscription systems (usually also exclusive unless you are big publisher). Amazon won't care for your scheme and even they who control their ecosystem don't put users annotations INSIDE the ebook.

[DarkTrick]

Disclaimer:
As letters don't convey enough information I'm not trying to discuss against you, I'm trying to figure out the rationale behind the problem I appreciate the participation!

>> (1) I was wondering: why is there no standard to store annotations in an ePub book?

> 1. Because it's pointless for almost anyone interested.

Why is that?

>> (2) Couldn't you write highlights and annotations directly into the document?

> "adding to" or "changing" a copywritten publication (ePub) is a big no-no.

> Madness.

I don't see the "madness" or problem here. If you'd had a book, you would also write directly into it - even though the content is copyright protected.


> Really crazy to edit the existing files inside the epub

Why is that?

[Quoth]

It messes up any indexing to write into the files. Insert also is more risky (might damage file) and slower than append. Thus create a separate file. The eBook is actually a Zip archive, so adding to it all is a more major and dangerous operation than creating separate files, which is safer, faster and no less functionality.

While it's theoretically possible to edit an ebook on an ereader, copyright and DRMed ones shouldn't be edited at all, and it's a much slower riskier thing than editing on desktop, where backups are easy. The Flash on an ereader isn't at all like NVMe SSDs.

[Quoth]

Originally Posted by DarkTrick

> Madness.

I don't see the "madness" or problem here. If you'd had a book, you would also write directly into it - even though the content is copyright protected.

The madness doesn't refer to annotating. Personally I'd not annotate a physical book, but you don't give or lend copies – only the physical orginal, you do own the physical medium and can mutilate it anyway you fancy, or highlight passages or add ballpoint pen annotations. But I'd hate to borrow a physical book that someone has highlighted and annotated. If I accidentally buy such a book in a charity shop I bin it. What is wonderful about ebook annotations is that they are NOT in the ebook, yet you can use them as if they are.
PDFs can have annotations (graphic image or text) added if they don't have DRM because they can be added at the end of the file and defined as an extra layer. Each annotation then has a page and position on the page.
An ebook doesn't work like a PDF. Technically it's possible to put annotations inside an ebook, but that is madness compared to a database or a per ebook separate file.

[PeterT]

If you're interested, the Epub spec is maintained / managed by the Epub 3 working group https://www.w3.org/publishing/groups/epub-wg/

Remember that even if a standard were to be established there would be no guarantee that it would be implemented in devices / software; look at the number of solutions out there that don't fully implement the current standards

Sent from my Pixel 4a using Tapatalk

[Tex2002ans]

Originally Posted by DarkTrick

(1) I was wondering: why is there no standard to store annotations in an ePub book? Perhaps some insights?

A lot of this was discussed back in 2017: "Merge notes into epub file?".

To my knowledge, not too much has changed on that front.

In 2021, Calibre added support for a way to export notes/highlights to markdown from Calibre's reader...

But there's definitely no cross-format, cross-compatible way of doing this. Just individuals coming up with all their non-standard, (proprietary) ways for every individual format.

Side Note: If I remember correctly, the current "best"/most compatible way, might be baking all the notes/highlights into PDFs + only read using specific apps on your computer/tablet. (That was probably discussed in the 2017 thread above.)

Originally Posted by DarkTrick

How would that work in conjunction with DRMs?

That's a huge part of it.

A lot of it also gets very complicated when the underlying file itself may change slightly. (Fixing typos, updating code, etc.)

How will your highlights and everything stay synced + transfer over?

Amazon probably has the current, best ecosystem for this... but the second something gets updated in the ebook, poof, all your notes+highlights are gone.

Originally Posted by PeterT

Remember that even if a standard were to be established there would be no guarantee that it would be implemented in devices / software; look at the number of solutions out there that don't fully implement the current standards

[DarkTrick]

Thanks again very much for the participation!

The link to the other thread and the open annotation standard were nice resources.

@Changing Epubs direclty
The problem of "how to handle annotations in a changing document" would be solved with this approach. But I understand that it's possibly dangerous. And on the technical side difficult because of the /inserting/ nature (and noted above already).

> A lot of it also gets very complicated when the underlying file itself may change slightly. (Fixing typos, updating code, etc.)
I've never thought of that (probably as I don't let my ebooks being updated. But this might indeed be a hurdle. I could imaging though, that a git-diff approach might solve this rather easily.

Anyway, I guess the conclusion for now is mostly (1) use books or (2) use a reader with a very easy notes-export functionality.

[rantanplan]

Originally Posted by DarkTrick

(2) Couldn't you write highlights and annotations directly into the document? After all it's basically XML. So you would simply need to add a tag around a passage marking the area and adding text information for the notes.
How would that work in conjunction with DRMs?

Not every html file is actaully encrypted, at least with CARE-Drm. In the end you could very well define a subset of content that isn't protected. But I also think that annotations should not be part of the epub itself but maybe a standardized json file that is in the same folder or similar.

[Tex2002ans]

Originally Posted by DarkTrick

Thanks again very much for the participation!

I haven't followed it too much since that 2017 thread, but it seems like "Open Annotation" Standard has been accepted by the W3C.

You can see all the latest specs (February 2017) here:

https://w3c.github.io/web-annotation/

Also, I assume Hypothes.is has still been chipping away at this problem as well. They may be a decent resource to read up more on the latest annotation news.

Originally Posted by DarkTrick

@Changing Epubs direclty

The problem of "how to handle annotations in a changing document" would be solved with this approach. But I understand that it's possibly dangerous. And on the technical side difficult because of the /inserting/ nature (and noted above already).

The specs go into detail on:

• Embedding into HTML or external file?
  
  • JSON, XML, etc.
• How to handle morphing documents.
  
  • New paragraph/image inserted in the middle of text? How do your make your note "stick"?
    
  • Note: And remember, websites are much more dynamic than "static" ebooks.
    
• Different levels of annotations
  
  • From Private (only for you) -> Fully Public (everyone can see).
    

and a lot of other stuff.

Originally Posted by DarkTrick

I've never thought of that (probably as I don't let my ebooks being updated. But this might indeed be a hurdle.

When ebooks are updated on Amazon:

• Amazon sends you a popup saying the publisher has updated the ebook + if you want to update your copy.
• They "warn you" of losing all your highlights, etc.
• You then have to explicitly accept it.

It's up to you to then balance the ebook fixes vs. losing your notes.

- - -

Side Note: I assume other stores do similar.

Kobo, as of a few years ago, locks you to the ebook-version-when-you-purchased, unless you contacted them directly.

I know back in 2019, when I informed Kobo one of the ebooks I purchased was accidentally set to "French", they updated my copy (and let me know the publisher already had it fixed within the months of my purchase + when I began reading).

Originally Posted by DarkTrick

I could imaging though, that a git-diff approach might solve this rather easily.

lol. Don't make me laugh. You probably haven't seen the underlying changes that occur when updating books OR doing a Calibre-convert.

There ain't nothing "easy" about git diff!

- - -

Side Note: For some of the super in-depth discussion, see this fantastic 5-part blog post in 2021: "Optimizing Git's Merge Machinery".

You wouldn't believe all the crazy edge-cases that come up.

- - -

But, if you read the "Open Annotation" specs, they do describe how you might try to keep comments attached, even if the underlying text/code is changing often.

(Again, think something like Microsoft Word / LibreOffice / Google Docs comments. They've already been trying to tackle similar problems for years.)

- - -

Complete Side Note: The latest insanity I ran across while digging through LibreOffice "Comment bugs"... did you know that Word allows you to COLOR text/backgrounds/highlights and carry over all this insane extra formatting within comments?

Not to mention:

• nested replies
• commenting in headers/footers
• images-within-comments

... and all sorts of other insanity.

Originally Posted by DarkTrick

Anyway, I guess the conclusion for now is mostly (1) use books or (2) use a reader with a very easy notes-export functionality.

Yes, pretty much this. Find one tool/workflow that you like, test the export functionality, and stick with it.

One of my frustrations is, after export, most only carry the raw text of the highlight.

(Many times, I only highlight a handful of words—not entire sentences/paragraphs. I jump to the comment, then need to read the surrounding context.)

If you read the Open Annotate specs above, they also describe ways to export strings to the left/right of the highlight... to try to make it a little more robust.