Plugin to import Highlight information from external source

[fmagin]

I am using a reMarkable Tablet which added support for highlights that are aligned with the text. The tablet specific details aren't really relevant for my question though, the abstract situation is:

• I have some datafiles that contain the information about what I highlighted in an epub on my device
• I would like to import these highlights into Calibre, so they show up like any other highlight added via the epub viewer
• in the future I would maybe integrate further external dependencies to automatically retrieve the highlight data from the Cloud API of the tablet, but for now I just have some local JSON files I need to work with.

My core question is: What are the appropriate APIs to add an highlights/annotations to an existing book? Are there existing examples for this API, ideally as a headless script that doesn't require adding GUI actions?

In case there are any existing plugins/tools/ideas for something like this, or completely different ways to approach this I am also curious to hear about them.

[kovidgoyal]

look in db/cache.py at functions with *annotations* in their name. Annotations in calibre are simple JSON objects that store highlighted text and start and end positions as EPUB CFI paths, alongwith colors/styles. Just creating a few different annotations with the calibre viewer and examining the resulting obets will do the trick.

You can get the db/cache.py object like this:

from calibre.library import db
cache = db('/path/to/library/folder').new_api

this is from a standalone script run via calibre-debug if you are in a gui plugin then you get it as:

from calibre.gui2.ui import get_gui()
cache = get_gui().current_db.new_api

[fmagin]

Thanks, this covers the initial problem!

The hurdle I am encountering now is properly getting the exact location as Calibre expects it. E.g. in Calibre the JSON annotation data looks like this:

{
"end_cfi": "/2/4/56/1:250",
"highlighted_text": "frontal lobes represent about 7 per cent of the total brain volume of a relatively intelligent animal such as the dog, and take up about 17 per cent of the brain in the lesser apes, they represent as much as 35 per cent of the human brain",
"spine_index": 10,
"spine_name": "OPS/chapter_01.htm",
"start_cfi": "/2/4/56/1:12",
"style": {
"kind": "color",
"type": "builtin",
"which": "yellow"
},
"timestamp": "2021-08-06T15:39:28.644Z",
"toc_family_titles": [
"Part One The Divided Brain",
"Chapter 1 Asymmetry and the Brain"
],
"type": "highlight",
"uuid": "ffT8yQAnoxs0oSoXBXdK6w"
},

I need to somehow get the location info from just the text "frontal lobes [...] of the human brain" or ideally a regex, like the epub viewer allows via the search function + highlight creation. Is there some API to achieve this without a GUI, or is this too deeply intertwined with the GUI code?

[kovidgoyal]

This happens via Javascript that runs in either a browser or Qt WebEngine. Qt WebEngine can run headlessly but it wont be an easy task. In the long term you are better off writing python code to parse html, do the search and convert its matches into CFI paths. Sadly this wont be an easy task either.

[Bunker D]

Are there any advancement on this topic? From what I get, the “Annotations” plug-in becomes quite outdated, isn't integrated with Calibre native annotation system, and is on it's way to depreciation.
I might help developing something, but at this point I don't even know where to start, and wouldn't like to do work that has already been done.

[kovidgoyal]

No there is no further work on this that I am aware of.