Marvin's Annotation Format (importing iBooks annotations)

[gulielmus]

Hi, so I'd really like to love Marvin. It doesn't seem like it'd be a hard thing to do. (I'm slightly disappointed by the ePub3 warnings, but that's about it so far.)

What I'm particularly interested in is exporting my annotations (notes, highlights, bookmarks) from iBooks. Yes, I know iBooks provides no means of extracting these, but iBooks on Mavericks provides an opportunity that only existed with things like iOS Backup Extractor previously: the iBooks SQLite database is available for perusal.


% cd Library/Containers/com.apple.iBooksX/Data/Documents

% sqlite3 BKLibrary/BKLibrary-1-091020131601.sqlite
sqlite> SELECT
ZAUTHOR,
ZTITLE,
ZASSETID
FROM
ZBKLIBRARYASSET
WHERE
ZTITLE = 'Ulysses';
James Joyce|Ulysses|AFD7FD1805E4A50A67FDB24A0A7C5F39


% sqlite3 AEAnnotation/AEAnnotation_v10312011_1727_local.sqlite
sqlite> SELECT
ZANNOTATIONLOCATION,
ZANNOTATIONMODIFICATIONDATE,
ZANNOTATIONNOTE,
ZANNOTATIONSELECTEDTEXT,
ZANNOTATIONSTYLE,
ZANNOTATIONTYPE
FROM
ZAEANNOTATION
WHERE
ZANNOTATIONDELETED = 0
AND
ZANNOTATIONASSETID=‘AFD7FD1805E4A50A67FDB24A0A7C5F 39'
epubcfi(/6/36[episode-12]!/4/2[episode-12]/774/1,:0,:423)|379400715||They believe in rod, the scourger almighty, creator of hell upon earth, and in Jacky Tar, the son of a gun, who was conceived of unholy boast, born of the fighting navy, suffered under rump and dozen, was scarified, flayed and curried, yelled like bloody hell, the third day he arose again from the bed, steered into haven, sitteth on his beamend till further orders whence he shall come to drudge for a living and be paid.|3|2


Apple is using EPUB-CFI (http://www.idpf.org/epub/linking/cfi/epub-cfi.html) for their annotations! Not some quirky custom method, but the standards-compliant portable specification for annotations! At this point I'm doing a little happy dance.

So, satisfied with this I went off to see how easy it would be to get this CFI into Marvin (probably by way of a Calibre column or something…). I extracted the Marvin container from an iTunes backup, and opened the mainDb.sqlite to see what fields I might need to extract.

% sqlite3 com.appstafarian.MarvinIP/Library/mainDb.sqlite
sqlite> .schema
...

CREATE TABLE "Highlights" (
"ID" INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
"BookID" INTEGER NOT NULL REFERENCES "Books"("ID") ON DELETE CASCADE,
"Section" INTEGER NOT NULL,
"Colour" INTEGER NOT NULL,
"Note" TEXT NOT NULL,
"UUID" TEXT NOT NULL,
"StartXPath" TEXT NOT NULL,
"EndXPath" TEXT NOT NULL,
"StartOffset" INTEGER NOT NULL,
"EndOffset" INTEGER NOT NULL,
"Text" TEXT NOT NULL,
"Deleted" INTEGER NOT NULL,
"AncestorXPath" TEXT NOT NULL,
"NoteDateTime" REAL NOT NULL
);

...



Uh oh. What are StartXPath, EndXPath, and AncestorXPath? Where’s the CFI field? I've got a bad feeling about this...

sqlite> SELECT
Section,
StartXPath,
EndXPath,
StartOffset,
EndOffset,
Text,
AncestorXPath
FROM
Highlights;
18| /x:html[1]/x:body[1]/x:div[1]/x:div[1]/x:section[1]/x[387]/text()[1]|/x:html[1]/x:body[1]/x:div[1]/x:div[1]/x:section[1]/x[387]/text()[1]|0|423|They believe in rod, the scourger almighty, creator of hell upon earth, and in Jacky Tar, the son of a gun, who was conceived of unholy boast, born of the fighting navy, suffered under rump and dozen, was scarified, flayed and curried, yelled like bloody hell, the third day he arose again from the bed, steered into haven, sitteth on his beamend till further orders whence he shall come to drudge for a living and be paid.|/x:html[1]/x:body[1]/x:div[1]/x:div[1]/x:section[1]/x[387]

Oi. It looks like my project has died before it even began. In order to convert an iBooks EPUB CFI into the relevant xpaths, I would have to open each book, open the package document to discover the spine location, then open that spine document to resolve the rest of the CFI, then covert all of that into XPath.

So I have a question for Marvin’s developers: do you plan on supporting EPUB-CFI in the near future? I’m downright dismayed that Marvin appears to have rolled its own solution to content location within EPUB files when there’s a standard for portable content reference. I was expecting to have to wrangle whatever form Apple was using in iBooks to CFI; I never dreamed the problem would be the reverse.

If Marvin were open source, I’d see how I could add support for CFIs (migrating existing XPath references would be painful though, and impossible without opening each individual book). As it is, I’m left with a massive amount of notes in iBooks that I now know I can extract easily (yay!) but that I cannot easily import into Marvin. It's particularly disappointing since one of the key advantages that eBooks have over print is that marginalia [I]can be[I] portable. I just wish eBook reading software understood that.

[kguil]

You're right. Marvin doesn't use CFI. It was a decision very early on in development that, in hindsight, was a wrong one - and very difficult to revert without causing lots of problems to existing users.

There are no plans to switch to CFI in the current version of Marvin but would almost certainly happen in the next major release.

I'm not sure it'll help, but I'm more than happy to discuss the xpath stuff you uncovered.

[gulielmus]

Thanks for the reply. I can certainly appreciate the difficulties that changing the way annotations are stored could introduce.

Do you have any kind of timeline for CFI support (the next major release, as you say)? Apple thoroughly ruined iBooks in iOS8 by making the reader just a small part of the store (instead of having the store be a small part of the reader). Marvin is great except for this little niggle for me, but it's a show-stopper. I'm a bit obsessive about annotations and retaining them regardless of readers, and it's frustrating to me how much of an afterthought they seem to be for most readers — of course, for most readers the actual reading is an afterthought since they exist as a means to sell products first.

There's no chance of Marvin going open source anytime soon, is there? It's something I'd happily work on, but rolling my own reader is out of the question, and there aren't any satisfactory open source readers.

[Torrilin]

Originally Posted by gulielmus

What I'm particularly interested in is exporting my annotations (notes, highlights, bookmarks) from iBooks. Yes, I know iBooks provides no means of extracting these

Actually, it does for annotations. I haven't tested on other data yet. And it is kludgy and nonobvious how to do it.

Go to the contents/bookmarks/notes view of your book.

Go to notes.

Tap the share button.

Select edit.

Select all, or if you want to export specific notes, select them.

Tap the share button.

Choose email, and email the notes where you want them. You can pick yourself, but I can tell you that emailing them to Evernote works. You may be able to use the email trick with other software too.

I tried the same thing via the bookmarks view, but I haven't figured out a way to make it work to export bookmarks. There seems to be something wrong with bookmarks tho since you can't always get the ribbon marker to show on the exact page you want.

I haven't compared iBooks on Mac with the iOS version to check whether there is a similar method of getting annotations out. But I'm pretty used to some stuff working better on iOS than Mac and vice versa.

[gulielmus]

Originally Posted by Torrilin

Actually, it does for annotations. I haven't tested on other data yet. And it is kludgy and nonobvious how to do it.

Well, yes, but I'm interested in more than the highlighted or annotated text. I'd like to be given the epub-CFI which references the specific location within the epub file, and should (in theory) be portable between readers. I should be able to export annotations from one reader and import them in another — physical marginalia is tied to a specific copy of a book, but there's no technical reason why digital marginalia should be. A specific revision, yes, but not just the copy that resides in my reader software of choice.

In this particular case I figured out how I can extract the CFI from iBooks. Id love to be able to import them into Marvin so I can retain them.

[Torrilin]

Yeah, I figured. But given the buggyness of the bookmarks feature, I'm not sure it's possible to really export them. And I haven't worked out what triggers the various bits of weird.

The failure modes I've seen are the unbookmarkable page (fairly common), a double bookmark (rather less common), and you bookmark page X, and you actually get X+1 marked. I haven't checked that they all still happen in iOS 8's iBooks, but they existed from at least iOS 5 on. If I want to mark a specific spot and have it be the right one, a highlight is more accurate.

[gulielmus]

Well, yes, but it's not so much bookmarks I'm concerned with as it is highlights and notes. And bookmarks are still stored by iBooks as CFIs, the same as highlights and notes (indeed Apple, rightly IMHO, has modeled all three as a single "annotation" with differing types and styles, which are likely expressed as enums or something).

[crashnburn]

Originally Posted by kguil

You're right. Marvin doesn't use CFI. It was a decision very early on in development that, in hindsight, was a wrong one - and very difficult to revert without causing lots of problems to existing users.

There are no plans to switch to CFI in the current version of Marvin but would almost certainly happen in the next major release.

I'm not sure it'll help, but I'm more than happy to discuss the xpath stuff you uncovered.

I'd like to be able to export the iBooks high & anns associated with an Epub and TRANSLATE move/ import it into Marvin for that same Epub.

If I can understand how the 2 formats are structured, I wouldnt mind writing an XSLT / XPATH to do translations.

[Faterson]

Bold dreams, everyone. I wish if only a fraction of what you're discussing were possible one fine day – such as, at the very least, syncing annotations between various Marvins...

[crashnburn]

Originally Posted by gulielmus

Well, yes, but it's not so much bookmarks I'm concerned with as it is highlights and notes. And bookmarks are still stored by iBooks as CFIs, the same as highlights and notes (indeed Apple, rightly IMHO, has modeled all three as a single "annotation" with differing types and styles, which are likely expressed as enums or something).

As I said, if we can get an understanding of the structures, I could try to create something that does translation.

[It2Fr]

Just noticing this thread a year or so later... I'm wondering if any of these projects got successfully resolved? Thanks.