[GUI Plugin] Annotations

[davidfor]

The Annotations plugin is under new management. With GRiker taking a long vacation, he has passed the development and support for the plugin to me. From now on, this thread will be used for any support questions and to host the plugin. All future questions should be posted here, but the older thread is here for reference.

The Annotations plugin imports annotations from supported ebook readers and applications described in the following table:

Main Features:

• Imports annotations from supported eBook reader applications.
• Annotations may be added to Comments or a custom column.
• Annotation appearance is fully configurable via CSS.
• Collect News clippings from periodicals downloaded by Calibre (Kindle).
• Extensible architecture allows adding support for other eBook reader applications.
• Enables fetching annotations directly from a connected iDevice without additional 3rd party software.
• Adds improved support for international Kindle timestamps contributed by @-axel-.
• Support for:
  • Kindle - annotations fetched from "My Clippings.txt" file.
  • Kobo ereaders - annotations fetched from database.
  • tolino ereaders - annotations fetched from notes.txt file.
  • GoodReader

Installation Steps:

• Open Preferences|Plugins|Get new plugins to install the 'Annotations' plugin. Or, download the attached zip file and install the plugin manually, then restart calibre as described in the Introduction to plugins thread.

Plugin configuration:

• By default, imported annotations will be added to calibre's Comments column.
• If you'd prefer to add imported annotations to a custom column (recommended), open the Customize Annotations dialog by selecting Customize plugin… from the plugin's context menu. Use the custom column wizard in the Annotation options section to create a custom column.
• If you choose to use a pre-existing custom column to receive annotations, click on the Annotations plugin icon to open the Configuration dialog, then select the custom column in the Annotation options dropdown list.
• See Help in the plugin menu for additional information.

Typical Kindle workflow:

The Kindle UI must be set to English, French, German, Spanish, Italian, Portuguese, Japanese or Chinese for the plugin to be able to retrieve locations and timestamps from your 'My Clippings.txt' file.

• Calibre: Transfer books from calibre to Kindle, either via email, Kindle's experimental web browser, or while directly connected to calibre.
• Kindle: Create some highlights/notes.
• Calibre: Open the Annotations plugin context menu, select Fetch annotations from Amazon Kindle.

Typical GoodReader workflow:

• Computer: Transfer books from calibre to GoodReader via iTunes. See this GoodReader explanation of how to transfer books from your computer to GoodReader via USB.
• GoodReader: Create some highlights/notes.
• GoodReader: While reading a book, tap the center of the screen so that the overlay controls are shown. Tap the Share icon (bottom right). Tap E-mail Summary, then email the annotations to yourself.
• Computer: After receiving the emailed annotations summary, select and copy the contents of the annotations summary email.
• Calibre: Select the book in your calibre library to receive the imported GoodReader annotations.
• Calibre: Open the Annotations plugin context menu, select Import annotations from…GoodReader, then paste the annotations summary into the dialog window.

Modifying the appearance of imported annotations:

• Click the Annotations plugin icon, or select Customize plugin from the context menu.
• Click Modify appearance…
  
  Spoiler:

  
• The Preview window shows a live preview of the formatting options.
• Edit the CSS in the Annotation elements window
• To move an element up or down, click the name of the element (Text, Note or Timestamp), then click the up or down arrow to move that element.
• To restore an element's CSS to its last saved values, select the element, then click the restore button.
• To reset an element's CSS to default values, select the element, then click the recycle button.
• To change the format of the timestamp, edit the Timestamp format. Click the Help icon to see formatting values for the timestamp. Click the recycle button to reset the timestamp to its default value.

How the plugin decides where to add imported annotations

Spoiler:

When the plugin imports annotations, it calculates a confidence index for the book receiving the annotations based upon three metadata items: Title, Author, and the calibre uuid. If all three are matched with a book in your calibre library, the annotations are added without any user intervention. With a partial metadata match, the plugin guesses which book should receive the annotations and asks for user confirmation. With no metadata matching, the user is offered the option to add the annotations to the currently selected book in the library.

Special Notes:

• Requires calibre 1.0.0 or later.

Developer Notes:

The Annotations plugin can be extended to support devices whose annotations can be fetched while connected via USB, or exported via email. If you are a developer interested in adding support for a particular device or reader app meeting these requirements, contact me for more information.

Version History:

Spoiler:

1.17.13 - 19 July 2022

• Fix: Qt6 compatibility - Yet another enum issue.

1.17.12 - 17 July 2022

• Fix: Qt6 compatibility - Yet another enum issue.

1.17.11 - 14 July 2022

• Fix: Qt6 compatibility - Another enum issue.

1.17.10 - 13 July 2022

• Fix: Qt6 compatibility - Another enum issue.
• New: Enable support for import file chooser - thanks to @stbenjam
• Fix: Qt6 compatibility - Check boxes in the annotations import screen were not working.

1.17.8 - 09 July 2022

• Fix: Qt6 compatibility - time conversions and column sort order

1.17.6 - 30 June 2022

• Change: Remove Qt compileUi as no longer used and blocking calibre6/Qt6.
• Fix: Missed a spot where need to use "current_friendly_name" for MTP devices
• Fix: PocketBook - Workaround issues with nulls in database and links.
• Fix: Change regex used for Kindle "Clippings.txt" to case insensitive. Kindle set to pt_BR uses lower case version of strings.
• Fix: Threadsafe method of getting device name for MTP devices on Linux.

1.17.0 - 10 April 2022

• New: Added support for AlReaderX reader app on the Boox Android-based devices. Thanks to @aik099.
• Change: Some code cleanup.
• Fix: Handle change collections library.

1.16.0 - 19 July 2021

• New: Add PocketBooks support - Thanks to @wold.
• Fix: Error "TypeError: unhashable type: 'SortableTableWidgetItem'" when importing from Kindle.
• Fix: Error rendering annotations as HTML.
• Fix: Change some handling of confidence generation as some devices have already matched the books.

1.15.0 - 04 April 2021

• Fix: Spanish My Clippings.txt has "La" for the annotations.
• Fix: Change method of getting books on device to use a search similar to what is used for Kobo and Tolino devices.
• Fix: Sorting of annotations for kepubs on Kobo devices.
• Fix: Opening My Clippings.txt should have been binary. Missed this when reverted earlier change.
• Fix: Display more details of an exception when parsing My Clippings.txt.
• Fix: Goodreader import not working.
• Fix: Broke Python 2 support with the open when reading My Clippings.txt.
• Fix: If already have book id in library, use confidence of 5.
• Fix: Encode and decode the title from My Clippings.txt to remove the Unicode BOM character.
• Fix: A couple of changes to solve some obvious problems with the Kindle fetch. Plus more logging.
• Fix: Python 3 compatibility: Help dialogs.
• Fix: Sorting columns in fetched annotations dialog.

1.14.0 - 26 December 2020

• Fix: Python 3 compatiblity: Replace use of "file" with "open".
• Fix: Python 3 compatiblity: Sorting on the Annotations Import dialog.
• Fix: Python 3 compatiblity: Completed work for tolinos and Kindles.
• New: Add support for Sony devices. From @Cibes.
• Fix: Remove references to older calibre IPC code as that was removed in calibre 5.7.
• Update: Various backend changed that should improve performance.

1.13.0 - 18 October 2020

• Update: Changes for Python 3 support in calibre.
• Update: Internal changes to how dialogs are handled.

1.12.0 - 08 January 2020

• Fix: Errors fetching from Kobo devices if the dates in the database were null.

1.11.0 - 3 December 2019

• Fix: Problem when couldn't get old annotations.
• Fix: Support calibre version 4. Use QTextEdit instead of QWebView. Should be compatible with 3.x.

1.10.0 - 4 October 2019

• Change: Add PDF as supported format for Kindle devices.
• Fix Kobo: Not getting annotations for kepubs if filename contains spaces.
• Internal Change: Use rowfactory and names in database.
• Fix: Some issues with merging with existing annotations.

1.9.2 - 25 May 2019

• Update: Remove dynamic_property in preparation for change in calibre.

1.9.1 - 5 May 2019

• Update: Small change to make plugin backwardly compatible with older versions of calibre.

1.9.0 - 22 April 2019

• New: Add support for tolino devices.
• Fix: Parsing error when annotations in library missing color attribute.
• Fix: Apparently some Kindles use lower case for location and page.
• Fix: Was writing empty p-tag after the annotation and note.
• New: Change device fetch menu option to the same name all the time.
• New: Add some shortcuts.
• Update: Internal changes for prints and testing in ParseKindleMyClippingsTxt.py.
• Fix: Update to match BeautifulSoup changes in calibre.

Spoiler:

1.8.0 - 2 Jun 2018
• Fix: Version of BeautifulSoup was updated in calibre 3.24.0 and the plugins usage needed to change.
• Fix: Error when moving/updating annotations.
1.7.0 - 27 Jan 2018
• New: Add support for translations
• New: Add German translation. Thanks to schnesim.
• New: Add Spanish translation. Thanks to Terisa de Morgan.
• New: Add Italian translation. Thanks to dsestero.
• New: Add French translation. Thanks to Nicolas F.
• Fix: Extra Italian, German and Polish strings for the locations in Kindle 'My Clippings.txt'
• New: Add German name of 'My Clippings.txt' file.
• New: Add support for KindleXRay driver.
• Fix: Normalize title collected by scanner for matching title and when comparing for confidence calculation.
• New: Added support for newer formats for Kindles - kfx and awz3
• Fix: Update regex for getting timestamp from My Clippings.txt
• Fix: Some of the icons weren't valid and displayed an error in the debug log. Fixed by Nicolas F.
1.6.0 - 25 Aug2017
• Fix: Extra Spanish string for locations in Kindle 'My Clippings.txt'
1.5.0 - 29 Jun 2017
• Fix: Handle that idevice.libimobiledevice doesn't work in calibre v3+ and is blacklisted.
• Fix: Problem opening config dialog of annotations column does not exist.
• Fix: Display error if fetch annotations if annotations column has not been configured or does not exist.
1.4.11 - 20 Aug 2016
• Kindle: Add second Spanish identifiers.
• KoboTouch: Fix "Last Annotation" timestamp in import dialog
• KoboTouch: Annotation timestamp now shown in local time
• KoboTouch: Change database queries to improve performance.
• KoboTouch: Change to use apsw instead of sqlite3
1.4.7 - 28 Nov 2015
• Handle if the libiMobileDevice library cannot be opened.
1.4.6 - 22 Nov 2015
• The reported error when a Pocketbook device is connected
• Remembering the size of the Import Annotations dialog.
• Default to using the Comments column if one hasn't been configured.
v1.4.3 - 28 Sep 2014
• Fixed matched book confirmation dialog
v1.4.2 - 01 Aug 2014
• Fix error in logging for Kindle
v1.4.1 - 28 Jul 2014
• Support for upcoming calibre 2.0
v1.4.0 - 16 June 2014
• Added support for multiple libraries
• Various bug fixes
• Remove support for Marvin as that is in the Marvin XD plugin.
v1.3.0 - 9 December 2013
• Added support for multiple libraries
• Various bug fixes
• Remove support for Marvin as that is in the Marvin XD plugin.
v1.2.0 - 13 September 2013
• Removed dependency on iExplorer for iOS reader applications.
• Added custom column wizard
• Various bug fixes
• Added improved international support for Kindle 'My Clippings.txt' contributed by MobileRead member -axel-.
v1.1.2 - 18 April 2013
• Handle non-English locations in Kindle 'My Clippings.txt'
v1.1.1 - 18 April 2013
• Handle non-English timestamps in Kindle 'My Clippings.txt'
v1.1.0 - 15 March 2013
• Add News clipping feature
• Add support for GoodReader
• Add support for hardware Kindle
v1.0.1 - 11 March2013
• Fixed rare crash condition after opening Preview dialog.
v1.0.0 - 11 March 2013
• Initial release for Linux, OS X and Windows.
• Support for Marvin, an iOS reader application.

[pedrogaza]

Hi,

Would it possible to export annotations to Kobo ? For the ADE annotations it is not a problem, but for annotations stored in Kobo database, it is more difficult to manage it by hand.

I wait for this function since long time... I do not know how to program a plugin, so if I can't find a way to do this before this summer, I will make it in bash script.

Pierre

[theducks]

Originally Posted by pedrogaza

Hi,

Would it possible to export annotations to Kobo ? For the ADE annotations it is not a problem, but for annotations stored in Kobo database, it is more difficult to manage it by hand.

I wait for this function since long time... I do not know how to program a plugin, so if I can't find a way to do this before this summer, I will make it in bash script.

Pierre

Have you looked at what the Kobo utilities PI can do for you?

[davidfor]

Originally Posted by pedrogaza

Hi,

Would it possible to export annotations to Kobo ? For the ADE annotations it is not a problem, but for annotations stored in Kobo database, it is more difficult to manage it by hand.

I wait for this function since long time... I do not know how to program a plugin, so if I can't find a way to do this before this summer, I will make it in bash script.

Version 1.4.0 of the plugin added support for the Kobo devices. I still need to update the first post completely. And add it to the help.

There are some alternatives:

• As theducks said, the Kobo Utilities plugin includes a function.
• Calibre has a a "Fetch annotations" function that gets them and adds them to the comments field.
• If you add "ExportHighlights=true" to the "FeatureSettings" section of the devices config file, a menu item will be added to the books menu in in the library list on the device to export them as a text file on the device.
• There have been a few scripts or apps posted in the Kobo forum over the years. A search there should find them.

[Rains]

Hi! Is this plugin usable in Linux? Because I can't get it to work, it installs fine and then I reboot the program but I can't import annotations from the Kindle and when I try to click on "Customize plugin" it shows me this:



These are the details:

calibre 1.40  isfrozen: False is64bit: True
Linux-3.15.0-1-mainline-x86_64-with-glibc2.2.5 Linux ('64bit', 'ELF')
('Linux', '3.15.0-1-mainline', '#1 SMP PREEMPT Mon May 26 09:43:21 CEST 2014')
Python 2.7.7
Linux: ('', '', '')
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/preferences/plugins.py", line 286, in customize_plugin
    self.modify_plugin(op='customize')
  File "/usr/lib/calibre/calibre/gui2/preferences/plugins.py", line 361, in modify_plugin
    if plugin.do_user_config(self.gui):
  File "/usr/lib/calibre/calibre/customize/__init__.py", line 150, in do_user_config
    config_widget = self.config_widget()
  File "calibre_plugins.annotations.__init__", line 28, in config_widget
  File "calibre_plugins.annotations.config", line 60, in __init__
AttributeError: 'AnnotationsAction' object has no attribute 'opts'

As you can see, my Calibre is in Spanish, but I don't think that's the issue, or is it?

[eschwartz]

I do know that according to your error message [isfrozen: False] you do not seem to be running the version that came from Kovid, but are instead using a distro calibre.

It is generally recommended to use the official calibre build. Most distros do not update as frequently as calibre does (weekly) and Kovid prefers to have one self-contained package to support.

The plugin does work on linux -- I use it.

[Rains]

You're right. I just removed the version I had installed (which wasn't too old, 1.40) and installed the binary package from Calibre's website and now the plugin works.

Thanks.

[eschwartz]

Originally Posted by Rains

You're right. I just removed the version I had installed (which wasn't too old, 1.40) and installed the binary package from Calibre's website and now the plugin works.

Thanks.

There are also random ways in which the binaries are not necessarily the same as the distro. For instance, calibre (binary) uses a different version of python (2.7.5) than you seem to have installed (2.7.7).

Perhaps the plugin is written in a way that depends on 2.7.5?

[davidfor]

Originally Posted by eschwartz

There are also random ways in which the binaries are not necessarily the same as the distro. For instance, calibre (binary) uses a different version of python (2.7.5) than you seem to have installed (2.7.7).

Perhaps the plugin is written in a way that depends on 2.7.5?

To the best of my knowledge it doesn't. Or at least not directly. The plugin is dependent on calibre. And it is packaged with a version of Python, so anything else could cause a problem.

Rain: Where did you try to configure the plugin from? The toolbar button or the plugin list?

[eschwartz]

Originally Posted by davidfor

To the best of my knowledge it doesn't. Or at least not directly. The plugin is dependent on calibre. And it is packaged with a version of Python, so anything else could cause a problem.

You mean python is installed in the plugin itself??

Doesn't it use calibre's python? Which in this case is updated distro python.

[davidfor]

Yes, that was badly phrased. The plugin doesn't have its own version of Python. It is dependent on the version of Python included with calibre.

[Rains]

Originally Posted by davidfor

Rain: Where did you try to configure the plugin from? The toolbar button or the plugin list?

I'd tried both from the plugin list and the toolbar button and the result was the same.

[Digital_Restrict]

I am trying to use this plugin but I'm encountering an error message. I wonder if someone is still providing support for this apparently glorious plugin? If I get a reply, I will post the exact error message and jobs log, and all the relevant details. I would love to be able to use this plugin ! Thanks !

[PeterT]

Davidfor is supporting it....

[davidfor]

Originally Posted by Digital_Restrict

I am trying to use this plugin but I'm encountering an error message. I wonder if someone is still providing support for this apparently glorious plugin? If I get a reply, I will post the exact error message and jobs log, and all the relevant details. I would love to be able to use this plugin ! Thanks !

Yes, post the error and any information that you think could help.