[GUI Plugin] Annotations

[DanH44]

David: I apologize for being unable to figure this out myself.

I bought a Kindle, but I hate it and never use it. I have been using Kindle for PC and have some books there that I annotated. I just discovered Calibre yesterday and found/install your plugin today.

Is it possible to bring the annotations from my Kindle for PC into Calibre with your plugin?

Thank you,
Dan

[davidfor]

Dan: No, or at least, not directly. The plugin is reading the annotations from the connected device. In the case of a Kindle, this is the MyClippings.txt file. I'm not sure, but I think the annotations should be synced to the Amazon cloud and then to the Kindle. If that is correct, if sync you Kindle over WiFi, then connect to calibre, the plugin should be able to get the annotations. Unfortunately, I don't have a Kindle to test if this will work.

[davidfor]

Originally Posted by richo

The function ParseKindleMyClippingsTxt.py does the parsing of the files, It may be not that difficult to use it for action triggered by dropping file.

Yes, that might be possible.

(It is funny that cursor icon indicates that drag and drop is not supported)

Same here. I can't see why it only sometimes works.

I tried your idea to put old file back and run Fetch annotation from Kindle. Interesting is that it doesn't fetch all annotations only some and I suspect only those whose books are still loaded in Kindle. Why is that? Is it done in step when progress bar is displayed with text "Getting installed books fro Kindle"?
All the info needed for import is in the MyClippings.txt file, so not really need to scan books on kindle.
That means that I am not able to fetch annotations from the books which are still in calibre but not on Kindle anymore.

Unfortunately, I didn't write that code, and am not completely sure how it works. The plugin only get annotations for books that both on the device and in the calibre library. By the sounds of it, for the Kindle, that might not be the best idea. One thing is, the plugin is reading metadata from the book. This is used to help match the book on the device with the book in the library. From what I can see, the link between the two is not as strong as I would expect. That is why the plugin has dialogs to confirm the annotations are put on the correct book.

Because of the error Aldous04 was having, I have considered either not checking the book, or catching the error and handling it better. This could be to continue with the annotations but using only the information from MyClippings.txt. It should work, but I don't have a way to test it.

[eschwartz]

Originally Posted by davidfor

Dan: No, or at least, not directly. The plugin is reading the annotations from the connected device. In the case of a Kindle, this is the MyClippings.txt file. I'm not sure, but I think the annotations should be synced to the Amazon cloud and then to the Kindle. If that is correct, if sync you Kindle over WiFi, then connect to calibre, the plugin should be able to get the annotations. Unfortunately, I don't have a Kindle to test if this will work.

It will indeed sync to the Kindle, but into the .mbp1 and .azw3(f|r) files.

"My Clippings.txt" is simply a logfile, and it only records annotations on the device they are made, as they are made.


Using "My Clippings.txt" to get the annotations is a hack, but of course, the best we can do until/unless someone reverse-engineers the annotations files and figures out how to extract them from there.
Might be worth it, IF there was some way to match those notes to notes taken from other devices. (Then we could translate notes from device to device.)

[davidfor]

Originally Posted by eschwartz

It will indeed sync to the Kindle, but into the .mbp1 and .azw3(f|r) files.

"My Clippings.txt" is simply a logfile, and it only records annotations on the device they are made, as they are made.


Using "My Clippings.txt" to get the annotations is a hack, but of course, the best we can do until/unless someone reverse-engineers the annotations files and figures out how to extract them from there.
Might be worth it, IF there was some way to match those notes to notes taken from other devices. (Then we could translate notes from device to device.)

Have you ever looked at the .mbp1 and .azw3(f|r) files? How easy it is to use them depends on the format. If they are some binary or encrypted format, that makes it hard. But, if they are text file of some sort, it should be possible. I'd be interested in looking at them, so if you or someone else wants to send me some examples, that would be good.

[eschwartz]

Never looked at them, they are some sort of binary data though.

Here is Hogfather (by Terry Pratchett).

I have highlighted "After all, what was the point of teaching children to be children? They were naturally good at it." and added the note "test note".

I believe there have been halfhearted attempts in the past, but nothing current.

[davidfor]

My first thought is that it is a serialization of an object. And that would mean know what the object was to reconstruct it. This has an application to do it, but, it probably doesn't work now. That page suggests the files are now encrypted. But, it also says different extensions were used for the encrypted files, so maybe they aren't. I might have a better look over the weekend.

[davidfor]

I thought of a test. I made an annotation in the Kindle app on my iPad. Then I opened Kindle for PC. The annotation was there.

When I looked in the directory the books are in, there was an mbp. The (reformatted) contents are:

{
"md5":"1456041f37626390ee733f102fcab643",
"payload": {
	"guid":"CR!VKB2NABKD94EBCKQ2R2DAAFQ4D8N:DDC1AF5F",
	"key":"B008ZG9KMA",
	"acr":"CR!VKB2NABKD94EBCKQ2R2DAAFQ4D8N",
	"type":"EBOK",
	"records": [
		{
			"lastModificationTime":"2015-01-30 03:41:09.0",
			"startPosition":"5853",
			"type":"kindle.note",
			"text":"This is a test!",
			"endPosition":"5853",
			"creationTime":"2015-01-30 03:41:09.0"
		},
		{
			"location":"5160",
			"type":"kindle.lpr",
			"creationTime":"2015-01-30 03:41:52.0"
		},
		{
			"lastModificationTime":"2015-01-30 03:40:39.0",
			"startPosition":"5674",
			"type":"kindle.highlight",
			"endPosition":"5853",
			"creationTime":"2015-01-30 03:40:39.0"
		}
	]
	}
}

That is clear and easy to handle. The issue is that it doesn't include the highlighted text. The positions are there. That probably makes it possible to extract the text, but I have no idea how to do that.

I don't know if this is what is encoded in the file eschwartz supplied. It probably is, unless there is a different format file for the different book formats. And suspect that the use of clear text might be dependent on something else. I added an annotation to another book, and it was not in this format. And a third one was. I think the DRM state of the book changes the annotations file. Whether the Kindle device follows the same rules, I don't know.

Right now, I won't be doing anything about this. If someone wants to write some code, I'll be happy to discuss it. And very happy to include it in the plugin. Especially if it works better than what we currently have.

[eschwartz]

FWIW my book was a DeDRMed Amazon book.

[davidfor]

Well, that blows that theory.

[DanH44]

David: In case this is of interest to you, I hacked the MBP format. This was prep for me to write a little auto-hotkey script to extract my annotations.

Anyway, here's the format for the amazon .MBP files when annotating a .PDF made by Calibre from a .EPUB. (My next step is to see if this is any different for .MOBI annotations.)

[Nels17]

Hello. Thank you so much for creating an annotation plugin for Calibre. This is a huge improvement. I am looking for ways of importing my notes, highlights and bookmarks from my Nook SimpleTouch (v. 1.2.1). I found the following script online, which does not seem to work, but may help finding a solution:

#!/usr/bin/env python2
import os
import tempfile
import sqlite3

class DBExportor(object):
    def __init__(self, db=None):
        if not db:
            db = self.pullDB()
        self.conn = sqlite3.connect(db)
        self.conn.row_factory = sqlite3.Row
        self.files = {}

    def pullDB(self):
        ''' pull Nook Reader's db from device and return the path of that db '''
        db_name = "annotations.db"
        pull_db = "/data/data/com.bn.nook.reader.activities/databases/" + db_name
        db_path = os.path.join(tempfile.gettempdir(), db_name)
        os.system("adb pull %s %s" % (pull_db, db_path))
        return db_path

    def saverow(self, row):
        ''' save one row to file '''
        # get fp
        fname = os.path.basename(row['ean'])
        if fname not in self.files: # create outfile if necessary
            base = os.path.splitext(fname)[0]
            export_file = base + '.txt'
            self.files[fname] = open(export_file, 'w')
        fp = self.files[fname]

        # composite a record
        record = "-%s-" % row["pagenumber"]
        if row['highlighttext']:
            record += ' ' + row['highlighttext']
        if row['note']:
            record += ' [Note: %s]' % row['note']
        record += '\n'

        # write to file
        fp.write(record.encode('utf8'))

    def export_db(self):
        c = self.conn.cursor()

        table = "annotations"
        c.execute("SELECT * FROM %s" % table)
        # since pagenumber is a string, need convert to int
        rows = c.fetchall()
        rows.sort(key=lambda c: int(c['pagenumber']))
        for row in rows:
            self.saverow(row)
        c.close()

    def cleanup(self):
        self.conn.close()
        for f in self.files.values():
            f.close()

    def run(self):
        self.export_db()
        self.cleanup()

if __name__ == '__main__':
    exp = DBExportor()
    exp.run()

I can’t get this script to work in Applescript Editor, nor do I understand how these things work. Is this something you could have a look at and hopefully integrate into your annotation plugin for Calibre? This seems to be the only viable solution for Nook users. A lot of people seem to be looking for this option on the internet and it seems that Calibre would be best and most user friendly way of achieving it. Thanks your help!

[davidfor]

That's a Python script to read the annotations database on the Nook and dump out a file with lines with the page number, highlighted text and the note. It creates one file per book.

That gives a lot of clues as to what is needed. Plus, someone in the past has sent me some parts of the database from a Nook. But, it was incomplete. It had the annotations database, which includes a path to the book, but no details of the book. Without this, I don't have a way to connect the annotation pulled from the database to the book in the calibre library.

What I need to to try this is the full database, or some other way to connect the books and annotations together. If I can get that, and someone willing to test the code, I can try and create the code to read the annotations from the Nook.

The other thing I am not sure of is access to the database. That script appears to be using adb to copy to database to a temporary file and then run the rest from there. I thought that the Nooks were mounted as a mass storage device and hence the database and books were accessible that way. If it isn't, that might complicate things.

[Nels17]

Thanks for your reply and for considering it. I’m afraid I don’t know anything about code, script and python, so I won’t be much help. I can confirm, however, that Nooks work as a storage drive. I drag and drop pdfs onto my Nook Simple Touch and it reads it.

I’m an academic. I like reading books and journal articles on my Nook because I can’t stare at the computer all day, but the problem is that I can’t retrieve my notes afterwards. I know I’m not the only one in this case. We’d be extremely grateful if you or someone else could develop a plugin for the Nook. I’d be happy to donate some money for it if it can make my work life much easier.

[davidfor]

Originally Posted by Nels17

Thanks for your reply and for considering it. I’m afraid I don’t know anything about code, script and python, so I won’t be much help. I can confirm, however, that Nooks work as a storage drive. I drag and drop pdfs onto my Nook Simple Touch and it reads it.

I’m an academic. I like reading books and journal articles on my Nook because I can’t stare at the computer all day, but the problem is that I can’t retrieve my notes afterwards. I know I’m not the only one in this case. We’d be extremely grateful if you or someone else could develop a plugin for the Nook. I’d be happy to donate some money for it if it can make my work life much easier.

I'm happy to look at it. At this point, I need a copy of the full database from a Nook. From the script, there should be a directory somewhere called "databases". The full path given in the script is "/data/data/com.bn.nook.reader.activities/databases/". But that is used with adb and I'm not sure if it is accessible when connected to the PC in any other way. A very quick search seems to suggest it isn't.