[GUI Plugin] Annotations

[GRiker]

The Annotations plugin imports annotations from supported ebook readers and applications:

Main Features of v1.1.2:

• Imports annotations from supported eBook reader applications.
• Annotations may be added to Comments or a custom column.
• Annotation appearance is fully configurable via CSS.
• Collect News clippings from periodicals downloaded by Calibre (Kindle, Marvin).
• Extensible architecture allows adding support for other eBook reader applications.
• Optional use of 3rd party application iExplorer enables fetching annotations directly from a connected iDevice. Use of iExplorer also requires installation of Apple iDevice interface plugin. See additional comments about iExplorer in Special Notes section below.

Installation Steps:

• Open Preferences|Plugins|Get new plugins to install the 'Annotations' plugin. Or, download the attached zip file and install the plugin manually, then restart calibre as described in the Introduction to plugins thread.

Plugin configuration:

• By default, imported annotations will be added to calibre's Comments column.
• If you'd prefer to add imported annotations to a custom column, create a custom column in Preferences|Interface|Add your own columns with a lookup name of annotations, a column heading name of Annotations, and a column type of Long text, like comments, not shown in the tag browser. (The lookup name and column heading name are examples, but the column type should be 'Long text…'). Accept all changes, then restart calibre.
• If you choose to use a custom column instead of the Comments column to receive annotations, click on the Annotations plugin icon to open the Configuration dialog, then select the custom column in the Annotation options dropdown list.
• See Help in the plugin menu for additional information.

Typical Marvin workflow:

• Calibre: Transfer books from calibre to Marvin, either via iTunes, email, OPDS or Marvin's web browser.
• Marvin: Create some highlights/notes.
  
  Spoiler:

  
• Marvin: Export the annotations (see Marvin export options below).
  
  Spoiler:

  
• Calibre: Open the Annotations plugin context menu, select Import annotations from…Marvin
• Drag the exported file to the plugin window, then click OK to import the annotations.
• Select a book, click Preview to preview the annotations to be added to calibre.
  
  Spoiler:

  
• Click Import Annotations to add the imported annotations to the selected book in calibre's database.
• Open calibre's Metadata editor to view the imported annotations.
  
  Spoiler:

  

Typical Kindle workflow:

The Kindle UI must be set to English for the plugin to be able to retrieve locations and timestamps from your 'My Clippings.txt' file.

• Calibre: Transfer books from calibre to Kindle, either via email, Kindle's experimental web browser, or while directly connected to calibre.
• Kindle: Create some highlights/notes.
• Calibre: Open the Annotations plugin context menu, select Fetch annotations from Amazon Kindle.

Typical GoodReader workflow:

• Computer: Transfer books from calibre to GoodReader via iTunes. See this GoodReader explanation of how to transfer books from your computer to GoodReader via USB.
• GoodReader: Create some highlights/notes.
• GoodReader: While reading a book, tap the center of the screen so that the overlay controls are shown. Tap the Share icon (bottom right). Tap E-mail Summary, then email the annotations to yourself.
• Computer: After receiving the emailed annotations summary, select and copy the contents of the annotations summary email.
• Calibre: Select the book in your calibre library to receive the imported GoodReader annotations.
• Calibre: Open the Annotations plugin context menu, select Import annotations from…GoodReader, then paste the annotations summary into the dialog window.

Modifying the appearance of imported annotations:

• Click the Annotations plugin icon, or select Customize plugin from the context menu.
• Click Modify appearance…
  
  Spoiler:

  
• The Preview window shows a live preview of the formatting options.
• Edit the CSS in the Annotation elements window
• To move an element up or down, click the name of the element (Text, Note or Timestamp), then click the up or down arrow to move that element.
• To restore an element's CSS to its last saved values, select the element, then click the restore button.
• To reset an element's CSS to default values, select the element, then click the recycle button.
• To change the format of the timestamp, edit the Timestamp format. Click the Help icon to see formatting values for the timestamp. Click the recycle button to reset the timestamp to its default value.

How the plugin decides where to add imported annotations

Spoiler:

When the plugin imports annotations, it calculates a confidence index for the book receiving the annotations based upon three metadata items: Title, Author, and the calibre uuid. If all three are matched with a book in your calibre library, the annotations are added without any user intervention. With a partial metadata match, the plugin guesses which book should receive the annotations and asks for user confirmation. With no metadata matching, the user is offered the option to add the annotations to the currently selected book in the library.

Marvin export options:

Spoiler:

To export annotations from a single book:

From within an open book:

• Tap Table of Contents (book icon at top left)
• Tap Share (arrow icon at top right)
• Tap Export Annotations, then email the annotations file to yourself

From the Library screen:

• Slide the book to the left, then tap Share
• Tap Export Annotations, then email the annotations file to yourself

After receiving the emailed annotations file:

• Open the context menu for the Annotations plugin
• Select Import annotations from…Marvin
• Drag the exported annotations file to the Import Marvin annotations window
• Click OK

To export all annotations:

• In Marvin's Home screen tap Settings (gear icon at top left)
• Choose the Services tab, then swipe left to the Calibre icon

Via Email:

• Tap the Export File button, then choose Email
• Email the annotations to yourself
• Open the context menu for the Annotations plugin
• Select Import annotations from…Marvin
• Drag the exported annotations file to the Import Marvin annotations window
• Click OK

Via iTunes:

• Tap the Export File button, then choose Documents directory
• In iTunes, select your connected iDevice in the DEVICES menu
• Click the Apps tab
• Scroll down to File Sharing, then select Marvin in the Apps list
• Drag library.mrvi from Marvin Documents to your desktop
• Open the context menu for the Annotations plugin
• Select Import annotations from…Marvin
• Drag library.mrvi from your desktop to the Import Marvin annotations window
• Click OK

Special Notes:

• Requires calibre 0.9.22 or later.
• With the optional installation of iExplorer (OS X and Windows), the Annotations plugin can fetch annotations directly from a connected iDevice (no jailbreaking required). iExplorer will run in evaluation mode which will allow you to decide whether it's worth the purchase. I have no affiliation with iExplorer or Macroplant. I do not receive any compensation from the use of iExplorer with the Annotations plugin. Some iOS reader applications do not export their annotations, so the use of iExplorer is the only way to fetch their annotations.

Developer Notes:

The Annotations plugin can be extended to support devices whose annotations can be fetched while connected via USB, or exported via email. If you are a developer interested in adding support for a particular device or reader app meeting these requirements, contact me for more information.

Paypal Donations:

If you find my contributions to calibre useful, say thanks with PayPal!

Version History:

Spoiler:

v1.1.2 - 18 April 2013
• Handle non-English locations in Kindle 'My Clippings.txt'
v1.1.1 - 18 April 2013
• Handle non-English timestamps in Kindle 'My Clippings.txt'
v1.1.0 - 15 March 2013
• Add News clipping feature
• Add support for GoodReader
• Add support for hardware Kindle
v1.0.1 - 11 March2013
• Fixed rare crash condition after opening Preview dialog.
v1.0.0 - 11 March 2013
• Initial release for Linux, OS X and Windows.
• Support for Marvin, an iOS reader application.

[-axel-]

Hi,

was trying to use the plugin (as included in calibre portable 0.9.23) and it did not find any annotations on my Kindle Paperwhite.

Created a debug file via plugin options, which says:


c:\dokume~1\axel\lokale~1\temp\calibre_annotations _plugin\Kindle.py:_parse_my_clippings(failed with line: u'- Ihre Notiz auf Seite 14 | Position 203 | Hinzugef\xfcgt am Sonntag, 2. September 2012 um 00:40:38 Uhr\n')
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Traceback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "c:\dokume~1\axel\lokale~1\temp\calibre_annotation s_plugin\Kindle.py", line 361, in _parse_my_clippings
UnboundLocalError: local variable 'ts' referenced before assignment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
action:fetch_device_annotations(Amazon Kindle)
action:fetch_device_annotations(INFO: <p>Unable to find any annotations on Amazon Kindle matching books in your library.</p>)


It quotes the second line in My Clippings.txt, which is in deed localized to German by the Kindle. So it looks like the plugin can not handle Kindle set to language different from English, does it?

I could try to change Kindle language to English, but I wonder if this would change all entries in My Clippings.txt or only new ones...? Anyway I'd rather not switch language.

Best
Axel

[GRiker]

Originally Posted by -axel-

Hi,
was trying to use the plugin (as included in calibre portable 0.9.23) and it did not find any annotations on my Kindle Paperwhite.

@-axel-,
I need to see your My Clippings.txt file to better understand the problem.

Please copy 'My Clippings.txt' from your Kindle to your computer, then send me a PM with that file as an attachment. If it's too big, you can zip it to make it smaller.

G

[-axel-]

@GRiker
attached you find My Clippings.txt. I've removed some clipping entries from the beginning using notepad.exe.

Got into software tester mood, so I also created a note containing some interesting non-ASCII characters, copied the file back to Kindle renamed as test.txt and marked the "==========" separator in there.

By the way, sometimes the first line of an entry starts with a 3 byte UTF-8 character, which is displayed as a centered dot by notepad++ and as a caret by notepad. Do you know what that means? I don't have a binary editor here... but maybe Kindle repeatedly prefixes entries with UTF-8 BOM?

Cheers
Axel

[GRiker]

@Axel, the problem is definitely related to the fact that your My Clippings.txt stores the timestamp of the annotation in German. I will do some research to see if I can find a way of handling multiple languages. It won't be right away, I'm in the midst of another project.

I will add a note to the Kindle section explaining that the UI must be set to English.

G

[GRiker]

Originally Posted by -axel-

@GRiker
By the way, sometimes the first line of an entry starts with a 3 byte UTF-8 character, which is displayed as a centered dot by notepad++ and as a caret by notepad. Do you know what that means? I don't have a binary editor here... but maybe Kindle repeatedly prefixes entries with UTF-8 BOM?

Yes, that's the conclusion I came to as well.

G

[-axel-]

Thanks a lot for your fast feedback.

@Axel, the problem is definitely related to the fact that your My Clippings.txt stores the timestamp of the annotation in German.

Duh, yes, parsing back the date/time is even risky in a single language; too many ways to write human readable day/time. Doing so for many languages... even with a mature library it is still asking for trouble. Amazon may decide to "improve" the format at any time.

I guess if I had to program a parser I would opt to keep everything after the last "|" a string. Comparing for equality should still work. Sorting by date won't work, but preserving order of entries may be good enough for most uses.

In the end the question is what is to be / can be done with the notes and highlights. What I am currently looking for is a way to quickly and easily edit a book based on my notes&highlights. Considering that calibre isn't much of an ebook editor, I guess that anyway isn't something your plugin offers, or does it?

[GRiker]

@-axel-
I've uploaded a new version of the plugin, 1.1.1, which substitutes the current time if it can't parse the timestamp from My Clippings.txt. This will at least allow you to fetch your annotations.

If you want to try your hand at modifying the Kindle code, explode the plugin, find the document 'developer_notes.txt', follow the directions in step 4.

@chaley had a great suggestion - use the online translation service at http://www.bing.com/translator to try to translate the line to English. This would assume an internet connection, and could be slow, as it would need to execute a translation for every annotation, but it could potentially solve the problem. There's a limit of 2MB of text per month through that site for free, so it would mean that individual users would need to have their own API key. This isn't something that I'm going to do in the near term, but you're welcome to try.

G

[GRiker]

Originally Posted by -axel-

In the end the question is what is to be / can be done with the notes and highlights. What I am currently looking for is a way to quickly and easily edit a book based on my notes&highlights. Considering that calibre isn't much of an ebook editor, I guess that anyway isn't something your plugin offers, or does it?

I've come at this from a number of different angles, and the approach I'm currently using is to do proofreading on an iPad using Marvin. It supports multiple highlight colors, so you can color-code your markup.

I use Sigil to do the (ePub) editing. It has full-text search, so it's pretty quick to pull up the annotations (either in Calibre or on the iPad), then find the corresponding point in Sigil and correct it.

I tried doing this using the Kindle PW, but it was ultimately an exercise in frustration. The interface is just too slow for making annotations.

G

[-axel-]

@-axel-
I've uploaded a new version of the plugin, 1.1.1, which substitutes the current time if it can't parse the timestamp from My Clippings.txt. This will at least allow you to fetch your annotations.

Sadly the date is not the end of the internationalization problems. I still can not fetch my annotations, but I get a new error:

c:\dokume~1\axel\lokale~1\temp\calibre_annotations_plugin\Kindle.py:_parse_my_clippings(failed with line: u'- Ihre Notiz auf Seite 14 | Position 203 | Hinzugef\xfcgt am Sonntag, 2. September 2012 um 00:40:38 Uhr\n')
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  Traceback   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "c:\dokume~1\axel\lokale~1\temp\calibre_annotations_plugin\Kindle.py", line 406, in _parse_my_clippings
AttributeError: 'NoneType' object has no attribute 'group'

Note that instead of "Location" German Kindle uses "Position".

use the online translation service at http://www.bing.com/translator to try to translate the line to English

In general that will create more problems than it solves. For example "Position" is also an English word, so it is no surprise the Microsoft translation keeps it. Expecting a translation to always use "Location" when there are a lot of synonymous words would be naive. Auto-translation may work for the date/time, but I see some irregularities in the translations. Sometimes it translates one way, sometimes another. Who knows, tomorrow it may use something else again.

For .net I would try to use the built in "Culture" support. Python may have something similar? IIRC the Unix world often calls it "locale".

Also note that if you tackle internationalization, you also will have to adjust the constants 'Highlight', 'Bookmark', and 'Note' in lines 345 ff in Kindle.py.

I use Sigil to do the (ePub) editing. It has full-text search, so it's pretty quick to pull up the annotations (either in Calibre or on the iPad), then find the corresponding point in Sigil and correct it.

I tried doing this using the Kindle PW, but it was ultimately an exercise in frustration. The interface is just too slow for making annotations.

I find PW acceptable to highlight some place where an edit is needed.

I'm more annoyed by having to use text search to locate a highlight in Sigil. There is the click-work for copy, paste and search. That's ok if you have just a few edits to do. But do it a 100 times and it becomes annoying.

And there is the problem of ambiguity. An extreme example would be if I highlight an extra word "a" in the text. When I see it at the location, it becomes obvious why I marked it. But searching all "a" in the book takes some time. Some (approximate) support for "Go to location" would help.

But I guess that is another topic.

Thanks for your feedback and for trying to quick-fix the problem.
Axel

[GRiker]

@-axel-,
I fixed the Position/Location error, version 1.1.2 should work properly for you.

As far as your ambiguity problem, I always highlight context around the typo to make it easier to find the fragment. If I think I won't remember the typo or the reason for the markup, I add an annotation to the highlight explaining the markup.

G

[tilia]

With this plugin installed, I can't drag-and-drop new formats into existing book record in calibre. Calibre freezes, and I have to use the task manager to kill it.

Win 7, calibre 0.9.29 (and a previous version, don't remember which, both 64 and 32 bit)

See also this thread

Debug log:

Spoiler:

calibre Debug log
calibre 0.9.29 [64bit] isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.3
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Starting up...
action:init_options(disable_caching: False)
annotations_db:connect(db_version: 1)
action:load_dynamic_reader_classes()
loading built-in class 'GoodReader'
loading built-in class 'Kindle'
loading built-in class 'Marvin'
Started up in 24.14 seconds with 1721 books
Debugging DND event
application/x-qt-windows-mime;value="Shell IDList Array" 1416 '\x01\x00\x00\x00\x0c\x00\x00\x00>\x01\x00\x00\x14 \x00\x1fB%H\x1e\x03\x94{\xc3M\xb11\xe9F\xb4L\x8d\x d5\x1c\x01w\x00"\x00CFSF\x1c\x002\x00\xd0\x08\x00\ x00BA:S DOWNLO~1.LIB\x00\x00\x00\x00t\x1aY^\x96\xdf\xd3H\x 8dg\x173\xbc\xee(\xba\xf7df\x89\xe1\x12\x0fI\x87\x 82\xc0\x83Z\xfd\x98\xfcV\x00\x08\x00\x04\x00\xef\x beZ=O\x9eBA:S*\x00\x00\x00\xa8\x04\x02\x00\x00\x00 U\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00D\x00o\x00w\x00n\x00l\x00o\x00a\x00d\x0 0s\x00.\x00l\x00i\x00b\x00r\x00a\x00r\x00y\x00-\x00m\x00s\x00\x00\x00H\x00*\x00\x00\x00\x00\x00\x ef\xbe\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01 \x00\x00\x00H\x00*\x00\x00\x00\x19\x00\xef\xbe\x9a \xabN_3haO\x89\x9d1\xcfF\x97\x9dI\x82\x07\xba\x82z[iE\xb5\xd7\xec\x83\x08_\x08\xccH\x00*\x00\x00\x00\ x13\x00\xef\xbe\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00'

application/x-qt-windows-mime;value="UsingDefaultDragImage" 4 '\x01\x00\x00\x00'

application/x-qt-windows-mime;value="DragImageBits" 36896 '`\x00\x00\x00`\x00\x00\x000\x00\x00\x00Y\x00\x00\ x00\x87\x16\x05r\x00\x00\x00\x00\xff\xff\xff\xff\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00'

application/x-qt-windows-mime;value="DragContext" 16 '\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00\x00'

application/x-qt-windows-mime;value="DragSourceHelperFlags" 4 '\x01\x00\x00\x00'

application/x-qt-windows-mime;value="InShellDragLoop" 4 '\x01\x00\x00\x00'

text/uri-list 73 'file:///C:/Users/tilia/Downloads/Daughter%20of%20Smoke%20and%20Bone.azw3\r\n'

application/x-qt-windows-mime;value="FileName" 36 'C:\\Users\\tilia\\DOWNLO~1\\DAUGHT~1.AZW\x00'

application/x-qt-windows-mime;value="FileNameW" 112 'C\x00:\x00\\\x00U\x00s\x00e\x00r\x00s\x00\\\x00J\ x00a\x00n\x00e\x00\\\x00D\x00o\x00w\x00n\x00l\x00o \x00a\x00d\x00s\x00\\\x00D\x00a\x00u\x00g\x00h\x00 t\x00e\x00r\x00 \x00o\x00f\x00 \x00S\x00m\x00o\x00k\x00e\x00 \x00a\x00n\x00d\x00 \x00B\x00o\x00n\x00e\x00.\x00a\x00z\x00w\x003\x00\ x00\x00'


Firefox rurl: None None
URLS: [u'file:///C:/Users/tilia/Downloads/Daughter of Smoke and Bone.azw3']
Paths: [u'/C:/Users/tilia/Downloads/Daughter of Smoke and Bone.azw3']
Extensions: frozenset([u'azw3'])
Firefox rurl: None None

[GRiker]

tilia, in the other thread you cited you posted a message saying that you resolved the problem. Is that correct?

G

[tilia]

No, the problem with adding new formats to books is still there. (I wrote drag-and-drop, but it's not possible from the meta-data window either). I also had a problem using the plugin. After deleting three entries in My Clippings.txt, I can now use the plugin to import annotations from my K3. That's what my last post in the other thread was about.

There actually seems to be some conflict between your plugin and Alf's DRM-plugin. If I disable the DRM-plugin, I can add new formats to book records with Annotations installed. So I'm not sure if this is something that can be fixed by you, or by them.