Calibre + Kindle: Fetching User Annotations

[GRiker]

Calibre 0.6.44 is now able to fetch user annotations from a connected Kindle. This means that any Bookmarks, Highlights or Notes you have created on the Kindle (as displayed in 'My Notes & Marks') may be merged into the book's comments field in calibre's database. 0.6.46 adds archiving of 'My Clippings.txt' to calibre's database.

Important note about migrating books from Kindle to calibre:

If you have not yet migrated your books from the Kindle to calibre, please read this section carefully.

Kindle supports three types of user annotation, plus Last Page Read:

• Bookmark: Type Alt-B on the Kindle keyboard to bookmark the current page. A small dog-ear triangle is displayed in the upper-right corner of the bookmarked page.
• Highlight: Toggle the joystick to move the cursor to the beginning of the section to highlight, click, move the cursor to the end of the section, click to highlight.
• Note: Toggle the joystick to move the cursor to the location where the note will be appended. Type your note; toggle the joystick to highlight 'save note', click to save.
  
• Last Page Read: The Kindle automatically saves your last location when closing the book, and re-opens to that location when you return.

To utilize the Fetch annotations feature, you will need to migrate your books into calibre following the steps listed below.

If you have existing annotations for Kindle books that you wish to keep, including Last Page Read locations, you need to migrate your annotations into calibre then resave them to the Kindle following the steps below.

If you don't care about any existing annotations, or keeping your Last Page Read locations, you may skip steps 3 & 7 for each book.

This is a one-time process – once you've migrated a book from the Kindle to calibre, you won't need to do it again.

Migrating books and annotations into calibre
(0.6.45 required for step 7):

1. Make a backup copy of the documents folder on the connected Kindle. A USB memory stick works well for this important step. If there's a problem, you'll be able to restore the contents of your Kindle/documents directory.
2. Add the book to the calibre database
   • Click the down arrow to the right of the Add books icon, then select Add books from a single directory
   • Navigate to the 'documents' folder on your connected Kindle
   • Select the book to add. Supported Kindle book extensions include azw, mobi, prc, tpz, and azw1.
   • Click 'Open'. The Adding ... dialog is displayed. When the dialog closes, the book has been added to the calibre database.
3. Add the annotations to the database
   • Select the book just added to calibre
   • Click the Edit meta information icon
   • Click the Add a new format for this book to the database icon.
   • Navigate to the 'documents' folder on your connected Kindle
   • Select the annotations file matching the book you added in step 2. Supported Kindle annotation extensions include .mbp and .tan. For example, if you added 'The-War-of-the-Worlds.azw' in step 2, you will now add 'The-War-of-the-Worlds.mbp' in this step.
   • Click 'Open'. The Adding ... dialog is displayed. When the dialog closes, the book's annotation file has been added to the calibre database.
4. Edit the metadata
   • Edit title, author, comments, etc. to your liking.
   • Click OK to accept the changes and close the Edit Meta Information dialog.
5. Delete the original file on the Kindle
   • Click on the Reader icon
   • Select the book
   • Remember that you backed everything up in step 1.
   • Click the Remove books icon to remove the original book from the Kindle
6. Send the book with edited metadata to the Kindle
   • Click on the Library icon to display the books in your database.
   • Select the book
   • Click the Send to device icon
7. Save the annotations to the Kindle (This step available with 0.6.45 and later)
   • Click the down arrow to the right of the Save to disk, then Save single format to disk ....
   • Select either MBP or TAN, matching the extension of the annotation file you added in step 3.
   • Select the 'documents' folder on your connected Kindle as the destination directory
   • The Saving dialog is displayed. When the dialog closes, the annotation file has been saved to the Kindle in the new location.
8. Repeat for each book to be migrated
9. Confirm the changes
   • Eject the Kindle
   • Confirm that the titles/authors conform to the metadata values in calibre
   • Select the book, toggle right, click 'My Notes & Marks' to view the annotations on the Kindle.
   • Confirm that the annotations have arrived safely.

Using Fetch annotations

Here's a 'My Notes & Marks' menu with some sample user annotations:

Fetch annotations merges user annotations created on the Kindle with a book's comments in the database.

Invoking Fetch annotations:

• Connect a Kindle and wait until the Reader icon is displayed next to the Library icon.
• Click the small down arrow to the right of the Send to device icon, then select Fetch annotations.
• The Jobs icon spins while calibre is retrieving annotations from the Kindle.
• A progress dialog is displayed while the annotations are merged into the database comments.
• The annotation file is added to the database.

After the dialog is dismissed, you can review your annotations in calibre. Select the book in the main window, then type 'I' or click in the lower metadata preview window ...

... which will bring up the metadata viewer:

Caveats:

1. Fetch annotations is an experimental feature introduced in version 0.6.44. I don't believe that it will harm your Kindle or your annotations.
2. If you're a new calibre user and you haven't yet migrated your books to calibre, see the section 'Migrating books and annotations into calibre' above to get started.
3. The Kindle must be connected for Send to device↓Fetch annotations to be enabled. Wait for the 'Reader' icon to display in the upper window.
4. Calibre looks for the annotation file in the same location that it stores the eBook on the Kindle.
5. Fetch annotations will not find annotation files (*.mbp, *.tan) in locations other than where calibre would store the eBook. See #4.
6. If you use more than one Kindle with calibre, Fetch annotations will import annotations from the currently connected Kindle, replacing any previous annotations.
7. Currently supported formats include MOBI, PRC, AZW, (*.mbp) TPZ and AZW1 (*.tan). PDF is not currently supported.
8. Fetch annotations is implemented only for the Kindle at this time.

Question, comments, suggestions? Please post them in this thread.

G

[daffy4u]

This is most excellent! Thanks for all your hard work!

[TuxGirl]

Yay!!!!!!!!!!!!!!!! thank you!!!!!!!!!!!!!!!!!!!!!!!!!!

[FoolforBooks]

Great, but.. when you edit the metadata, changing the title, and send that book back to the Kindle with Send to Device, are you re-naming the .mbp files to match the re-named book file? That is the only way those notes and highlights are still accessible through MyNotes&Marks.

The alternative would be not to re-name the file, leaving in Amazon's format, but change only the metadata within the file. Then the mbp files would still match up. I was very curious why you re-named the file at all.

And does #4 mean we must be using that sub-directory business with all the separate author and book files. I have disabled that.

[GRiker]

Originally Posted by FoolforBooks

Great, but.. when you edit the metadata, changing the title, and send that book back to the Kindle with Send to Device, are you re-naming the .mbp files to match the re-named book file? That is the only way those notes and highlights are still accessible through MyNotes&Marks.

I'll take a look at migrating the last-cached .mbp file with the book when using Send to device. That would obviously be helpful.

Originally Posted by FoolforBooks

The alternative would be not to re-name the file, leaving in Amazon's format, but change only the metadata within the file. Then the mbp files would still match up. I was very curious why you re-named the file at all.

That design decision precedes my involvement with calibre, but my sense is renaming the files is the right choice.

Originally Posted by FoolforBooks

And does #4 mean we must be using that sub-directory business with all the separate author and book files. I have disabled that.

No, you're not forced to use any particular scheme, but the scheme you use must be declared in Preferences|Add/Save|Sending to device. The storage pattern declared there is used to generate the path to find the associated .mbp file. So as long as you're using calibre to Send to device, Fetch annotations should be able to find the associated .mbp file. If the scheme you're using isn't finding the .mbp file, please post a bug report.

G

[FoolforBooks]

G--I misunderstood. I thought you were going to be managing the retention of the actual mbp and tan files. That is a concern when someone makes a change to the title of a book. Calibre changes the file name, and the mbp file no longer matches, so the notes and highlights are *lost*. They need re-named to the exact file name you used when Calibre sends the book to the device.

After doing metadata changes, I went in via usb and changed the mbp file names myself. I had left them on the Kindle. My understanding is that you are recommending people to clear their Kindle completely, and edit metadata, and re-send all their books via Send to Device. I think this is a little radical. If you are going to recommend this, then you need to tell people to backup those mbp files first (or leave them on the Kindle), and to rename them to match the new filenames on the books after they are sent back via Calibre. And to do it before they open the book, because the Kindle software will immediately create an mbp file with the new name. The re-naming process is complicated by having to delete that one and re-name the one containing the notes. (Or do all this within Calibre.)

I want to make sure I understand what you are going to do. If I do this, and I have 50 screens of highlights in a book, you will put all 50 pages in text format into the Comments box in the Calibre library? I'm not sure I understand the objective, but if people asked for this enhancement they must know what they will do with the data.

When you build a Calibre Catalog, and send it back to the Kindle, the Catalog will contain all 50 pages of highlights as Comments?

Does any of this require allowing Calibre to read the metadata when the Kindle connects to Calibre? I have disabled that as it was taking too long, and I didn't see why you needed to do it. I asked Kovid, but he didn't answer that part of the question.

[GRiker]

Originally Posted by FoolforBooks

G--I misunderstood. I thought you were going to be managing the retention of the actual mbp and tan files. That is a concern when someone makes a change to the title of a book. Calibre changes the file name, and the mbp file no longer matches, so the notes and highlights are *lost*. They need re-named to the exact file name you used when Calibre sends the book to the device.

There are some internal architecture changes for device drivers that need to be in place before calibre can store/replace annotations files. That work is currently under way.

Originally Posted by FoolforBooks

After doing metadata changes, I went in via usb and changed the mbp file names myself. I had left them on the Kindle. My understanding is that you are recommending people to clear their Kindle completely, and edit metadata, and re-send all their books via Send to Device. I think this is a little radical. If you are going to recommend this, then you need to tell people to backup those mbp files first (or leave them on the Kindle), and to rename them to match the new filenames on the books after they are sent back via Calibre. And to do it before they open the book, because the Kindle software will immediately create an mbp file with the new name. The re-naming process is complicated by having to delete that one and re-name the one containing the notes. (Or do all this within Calibre.)

I am working on some additional documentation to explain how to migrate from simple Kindle storage to calibre-assisted storage. You are correct that any pre-existing annotations that the user wishes to retain need to be renamed to continue to be available after calibre begins managing file storage on the Kindle.

Originally Posted by FoolforBooks

I want to make sure I understand what you are going to do. If I do this, and I have 50 screens of highlights in a book, you will put all 50 pages in text format into the Comments box in the Calibre library? I'm not sure I understand the objective, but if people asked for this enhancement they must know what they will do with the data.

Yes, that's what happens. If you're not interested in being able to see annotations in calibre, then you wouldn't use Fetch annotations.

Originally Posted by FoolforBooks

When you build a Calibre Catalog, and send it back to the Kindle, the Catalog will contain all 50 pages of highlights as Comments?

Currently (0.6.44), the user annotations are stored with the comments in the generated catalog, but that will change in the next release - user annotations will not be included in the generated catalog.

Originally Posted by FoolforBooks

Does any of this require allowing Calibre to read the metadata when the Kindle connects to Calibre? I have disabled that as it was taking too long, and I didn't see why you needed to do it. I asked Kovid, but he didn't answer that part of the question.

No, you don't need to have metadata reading enabled for the feature to work.

G

[andrys]

Originally Posted by FoolforBooks

G--I misunderstood. I thought you were going to be managing the retention of the actual mbp and tan files. That is a concern when someone makes a change to the title of a book. Calibre changes the file name, and the mbp file no longer matches, so the notes and highlights are *lost*. They need re-named to the exact file name you used when Calibre sends the book to the device.

After doing metadata changes, I went in via usb and changed the mbp file names myself. I had left them on the Kindle. My understanding is that you are recommending people to clear their Kindle completely, and edit metadata, and re-send all their books via Send to Device. I think this is a little radical. If you are going to recommend this, then you need to tell people to backup those mbp files first (or leave them on the Kindle), and to rename them to match the new filenames on the books after they are sent back via Calibre. And to do it before they open the book, because the Kindle software will immediately create an mbp file with the new name. The re-naming process is complicated by having to delete that one and re-name the one containing the notes. (Or do all this within Calibre.)

I want to make sure I understand what you are going to do. If I do this, and I have 50 screens of highlights in a book, you will put all 50 pages in text format into the Comments box in the Calibre library? I'm not sure I understand the objective, but if people asked for this enhancement they must know what they will do with the data.

When you build a Calibre Catalog, and send it back to the Kindle, the Catalog will contain all 50 pages of highlights as Comments?

Does any of this require allowing Calibre to read the metadata when the Kindle connects to Calibre? I have disabled that as it was taking too long, and I didn't see why you needed to do it. I asked Kovid, but he didn't answer that part of the question.

I'm one who makes highlights and notes on almost every book I read, but especially the non-fiction ones (which is most of what I read), so it's important to me what you're saying here...

I don't want to make or recommend making changes that would cause the notes and highlights to be gone unless we went to some work aligning the annotation/status auxiliary file with the newly edited book file when the title is actually changed.

I use the http://kindle.amazon.com page a lot, and it's sort of crucial I don't lose those annotations.


For non-Amazon books, no worry (I think).

[GRiker]

@FoolForBooks, @andrys: I've revised the first posting to help users who have not yet migrated book management to calibre with the process. I made some changes to the Save books functionality which won't be available until 0.6.45. With those changes, it is possible (though somewhat tedious) to migrate books and annotations from the native Kindle-style storage to calibre-style storage.

After the architectural changes are in place for the device drivers, it should be simpler for new users to migrate annotations with their books.

Thanks for the feedback.

G

[GRiker]

Originally Posted by andrys

I'm one who makes highlights and notes on almost every book I read, but especially the non-fiction ones (which is most of what I read), so it's important to me what you're saying here...

I don't want to make or recommend making changes that would cause the notes and highlights to be gone unless we went to some work aligning the annotation/status auxiliary file with the newly edited book file when the title is actually changed.

I use the http://kindle.amazon.com page a lot, and it's sort of crucial I don't lose those annotations.

@andrys, if you'd be willing to help me test the interactions with the Amazon page, that would be appreciated.

If you're already using calibre to manage the books on your Kindle, everything should just work. If it doesn't, I'd like to resolve any problems.

G

[FoolforBooks]

G-- Good news on all fronts. I tested, everything worked well with the fetch on the books I put in the expected folder structure. I certainly liked reviewing my first note (with all my tagging) in Calibre's preview pane. Very easy to see which books I've forgotten to tag.

I'lll look forward to the architectural changes. I have never let Calibre manage my books because of the mbp/tan file issue. It never caused me too much concern until I saw people so excited about re-titling all their books. I'm a worrier.

I think all of you have done an excellent job, except for that problem, so I'm glad you are working on it. I don't have a paypal account, or you would have had a donation from me. Good work.

(If I come here more often I am going to have to figure out how to do the quote thing.)

[FoolforBooks]

@Andrys

[/QUOTE] For non-Amazon books, no worry (I think).[/QUOTE]

Actually, in my own case, it is non-Amazon books that are a concern. I know Amazon backs up theirs, and all I need to do is archive and redownload and the mbp/tans are restored to me. I lose my metadata changes this way, but I haven't lost my annotations.

But there is no safety net for the non-Amazon books. If people don't follow G's procedure , --if instead they delete all those the mbp/tans off the Kindle, and then later change title and Send to Device from Calibre, those are gone. They probably have them somewhere in some backup they've made, but will they know how to find them and what to do? You know from your time on the Amazon forum how inexperienced some Kindle owners can be, and those are always the ones who panic about even losing their place in their book.


Edited: Obviously, I have more work to do on this quoting feature.

[FoolforBooks]

GRiker, tried out the procedure and it was successful. The problem I had though when I tried Fetch Annotations --the window opens saying this is in process and never closes. (I'm on XP). I couldn't get out without using Task Manager to End Process. It had completed all the fetches.

Do you realize the License Agreement has been blank on both releases 0.6.44 and 45 ? It's easy to agree to a big blank box.

[GRiker]

@FoolforBooks: Please try this:

• Open a command window in XP
• Type 'calibre-debug -g'
• Do the same sequence that you did causing the hang
• Post whatever traceback or error info comes up in the command window.

The license window is an issue for Kovid, please file a bug report on that one.

G

[Gwen Morse]

Is the information on Last Page Read eventually going to be accessible elsewhere (such as export to OPDS listing)? Reading this over, the first thing that came to mind was being able to use this information to read the same book on two different readers and keeping the location details in something resembling sync. I'm not sure if I understand how its going to be handled...

Gwen