[GUI Plugin] Annotations

[sweth]

Recently got a Kindle Oasis and just tried using the Annotations plugin (which worked fine w/ my old Kindle Paperwhite), but it says it finds no annotations matching any books in the library. I can see the annotations on the device in the My Clippings file when the device is mounted, and the Title and Author fields appear to match between what the Kindle has and what's in the Calibre metadata. Any suggestions for what to check next?

(Plugin v1.14.0, Calibre v4.23.0 as I am waiting to go to v5 until all of the plugins I use regularly are py3-happy.)

EDIT: I turned on debug and it looks like get_installed_books() and get_imported_books() are not finding what I'd expect. ie the former gets a path_map w/ 162 entries none of which are on this device (so resolved_path_map is empty) and then get_imported_books() only finds the Kindle Users Guide (all of the books I've installed so far on this device have been via Whispersync, so I'd expect them to show up in this step?). So then get_active_annotations() finds all of the annotations but thinks none of them are books on the device.

[davidfor]

Originally Posted by sweth

Recently got a Kindle Oasis and just tried using the Annotations plugin (which worked fine w/ my old Kindle Paperwhite), but it says it finds no annotations matching any books in the library. I can see the annotations on the device in the My Clippings file when the device is mounted, and the Title and Author fields appear to match between what the Kindle has and what's in the Calibre metadata. Any suggestions for what to check next?

(Plugin v1.14.0, Calibre v4.23.0 as I am waiting to go to v5 until all of the plugins I use regularly are py3-happy.)

EDIT: I turned on debug and it looks like get_installed_books() and get_imported_books() are not finding what I'd expect. ie the former gets a path_map w/ 162 entries none of which are on this device (so resolved_path_map is empty) and then get_imported_books() only finds the Kindle Users Guide (all of the books I've installed so far on this device have been via Whispersync, so I'd expect them to show up in this step?). So then get_active_annotations() finds all of the annotations but thinks none of them are books on the device.

Try the beta published at the end of last year. It should fix things. I expect that the books on the new device are KFX and there was a problem with handling these. It was actually a more general problem in how the plugin matched books on Kindle devices. It always failed for KFX and would occasionally fail for other formats.

[genoasalami]

Hoping you can help with this error. I am trying to create a custom column for the annotations because I don't want them showing up in all my book descriptions, which is happening right now because of use of the default "comments" column. Following the directions in the help file produces the below error.

calibre, version 5.9.0
ERROR: Unhandled exception: <b>ConstraintError</b>:ConstraintError: UNIQUE constraint failed: custom_columns.label

calibre 5.9 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 36, 0) && DeDRM (7, 0, 3) && Package KFX (from KFX Input) (1, 36, 0) && Annotations (1, 14, 0) && Find Duplicates (1, 8, 3) && KFX metadata reader (from KFX Input) (1, 36, 0) && KFX Input (1, 36, 0) && KePub Input (3, 4, 3) && KePub Metadata Reader (3, 4, 3) && KePub Output (3, 4, 3) && Kobo Books (1, 8, 0)
Traceback (most recent call last):
  File "C:\Users\genoasalami\AppData\Roaming\calibre\plugins\annotations_resources\dialogs\cc_wizard.py", line 160, in dispatch_button_click
  File "C:\Users\genoasalami\AppData\Roaming\calibre\plugins\annotations_resources\dialogs\cc_wizard.py", line 100, in custom_column_add
  File "calibre\db\legacy.py", line 808, in create_custom_column
  File "calibre\db\cache.py", line 72, in call_func_with_lock
  File "calibre\db\cache.py", line 2014, in create_custom_column
  File "calibre\db\backend.py", line 993, in create_custom_column
  File "calibre\db\backend.py", line 920, in execute
apsw.ConstraintError: ConstraintError: UNIQUE constraint failed: custom_columns.label

Thanks!

[DNSB]

Originally Posted by genoasalami

Hoping you can help with this error. I am trying to create a custom column for the annotations because I don't want them showing up in all my book descriptions, which is happening right now because of use of the default "comments" column. Following the directions in the help file produces the below error.

After selecting MyAnnotations, did you click on the +CreateCustomColumn button and then restart calibre? If you go into calibre preferences => Add your own columns, do you now see a column called Annotations with the lookup name #mm_annotations with the type Long text, like comments, not shown in the Tag browser?

[genoasalami]

Originally Posted by DNSB

After selecting MyAnnotations, did you click on the +CreateCustomColumn button

Yep

and then restart calibre? If you go into calibre preferences => Add your own columns, do you now see a column called Annotations with the lookup name #mm_annotations with the type Long text, like comments, not shown in the Tag browser?

Nope. Nothing happens once that error is thrown.

[davidfor]

Originally Posted by genoasalami

Hoping you can help with this error. I am trying to create a custom column for the annotations because I don't want them showing up in all my book descriptions, which is happening right now because of use of the default "comments" column. Following the directions in the help file produces the below error.

calibre, version 5.9.0
ERROR: Unhandled exception: <b>ConstraintError</b>:ConstraintError: UNIQUE constraint failed: custom_columns.label

calibre 5.9 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 36, 0) && DeDRM (7, 0, 3) && Package KFX (from KFX Input) (1, 36, 0) && Annotations (1, 14, 0) && Find Duplicates (1, 8, 3) && KFX metadata reader (from KFX Input) (1, 36, 0) && KFX Input (1, 36, 0) && KePub Input (3, 4, 3) && KePub Metadata Reader (3, 4, 3) && KePub Output (3, 4, 3) && Kobo Books (1, 8, 0)
Traceback (most recent call last):
  File "C:\Users\genoasalami\AppData\Roaming\calibre\plugins\annotations_resources\dialogs\cc_wizard.py", line 160, in dispatch_button_click
  File "C:\Users\genoasalami\AppData\Roaming\calibre\plugins\annotations_resources\dialogs\cc_wizard.py", line 100, in custom_column_add
  File "calibre\db\legacy.py", line 808, in create_custom_column
  File "calibre\db\cache.py", line 72, in call_func_with_lock
  File "calibre\db\cache.py", line 2014, in create_custom_column
  File "calibre\db\backend.py", line 993, in create_custom_column
  File "calibre\db\backend.py", line 920, in execute
apsw.ConstraintError: ConstraintError: UNIQUE constraint failed: custom_columns.label

That error says you already have a column with the lookup name "#myannotations". Or whatever name you used in the "Custom column wizard". The name you enter is used as the column header and is converted to lower case to be used as the lookup name. The error would only happen if you had used that name before.

A possibility is if you created the column, and didn't restart calibre. Then tried to do the configuration again (which is something I have done more times than I want to admit). At that point, I do not think the new column will not be in the list of suitable columns. But, if you tried to create it again, you would get this error.

I will look at putting some error handling in, but, in the meantime, check what calibre has in the the custom column list. The above should be there.

[davidfor]

Originally Posted by Noirtier

Hi David,

Thanks for the prompt response and apologies for the delayed reply. Despite being subscribed to this thread it does not appear in my User CP and I do not seem to receive notifications of any kind for it.

I will PM you a link now for the requested files which will include:
- Debug log file after a fetch with v1.14.2
- Kobo database file
- A few example epub files where the annotations are not importing properly
- AnnotationsPreview.txt showing the annotations as appear correctly in the preview window
- AnnotationsCalibre.txt showing the annotations as imported into Calibre with some sections missing as detailed in my previous post (apart from in the final example where nothing is missing for comparison).

Let me know if anything is unclear or if I can do anything more to help.

Can I check a couple of things:

• It looks like you have fetched the annotations into the calibre before. Is that correct?
• If so, did you edit the annotations? Since calibre 4.0, editing changes the structure of the HTML code. Even viewing it and then saving from the metadata editor does this.
• Do the books you included come from the calibre library or the device? I'm assuming the library as they don't include the custom columns on the OPF file. I'd like to see the actual annotations HTML. Could you send the OPF files from the library directory? That will include the latest metadata that is in the library database.
• Can you test adding an annotation to a book that did not have an annotations before? Fetch that and check it is in the library. Then add another annotation and do another fetch. Both annotations should be there. It does work here, but, I want check for someone else.

[Noirtier]

Originally Posted by davidfor

Can I check a couple of things:

• It looks like you have fetched the annotations into the calibre before. Is that correct?
• If so, did you edit the annotations? Since calibre 4.0, editing changes the structure of the HTML code. Even viewing it and then saving from the metadata editor does this.
• Do the books you included come from the calibre library or the device? I'm assuming the library as they don't include the custom columns on the OPF file. I'd like to see the actual annotations HTML. Could you send the OPF files from the library directory? That will include the latest metadata that is in the library database.
• Can you test adding an annotation to a book that did not have an annotations before? Fetch that and check it is in the library. Then add another annotation and do another fetch. Both annotations should be there. It does work here, but, I want check for someone else.

Thanks David,

• Yes, that is correct. It had worked for me previously. I thought without fault though maybe I just never noticed any issues before.
• Do you mean editing the annotations in Calibre? If so, no – I only copy them out of Calibre into my Zim notes. Very occasionally Calibre seems to think I have edited the meta-data and asks if I want to save changes. Though I always say no in that case.
• Correct – all books loaded on with Calibre. I'm afraid I couldn't find any .OPF files on my Kobo Forma at all. I tried searching both with Windows and a third part search utility and none came up. Are you sure that is the correct extension?
• I have just done that now, and it does indeed appear that there is an issue with updating pre-existing annotations. Testing with a new book, the first fetched annotation worked correctly. But then when I tried to add a second one, they both showed up in preview OK, but on importing into Calibre, the first one has now disappeared with just the chapter title header remaining, and only the second one appears correctly.

[davidfor]

Originally Posted by Noirtier

Thanks David,

• Yes, that is correct. It had worked for me previously. I thought without fault though maybe I just never noticed any issues before.
• Do you mean editing the annotations in Calibre? If so, no – I only copy them out of Calibre into my Zim notes. Very occasionally Calibre seems to think I have edited the meta-data and asks if I want to save changes. Though I always say no in that case.
• Correct – all books loaded on with Calibre. I'm afraid I couldn't find any .OPF files on my Kobo Forma at all. I tried searching both with Windows and a third part search utility and none came up. Are you sure that is the correct extension?
• I have just done that now, and it does indeed appear that there is an issue with updating pre-existing annotations. Testing with a new book, the first fetched annotation worked correctly. But then when I tried to add a second one, they both showed up in preview OK, but on importing into Calibre, the first one has now disappeared with just the chapter title header remaining, and only the second one appears correctly.

OK. For the OPF file, I meant from the calibre library. Click the "Open path" in the details panel for the book and it will show the directory with the book in it. There should be an OPF file. Calibre uses this as part of its backup.

I'll have to retest this. It seemed to work adding the annotation, but, maybe I missed something.

[dunhill]

Hello people, I have been reading carefully all the instructions and the plugin does not work for me. I'll be doing something wrong, you could guide me.
Create a custom column for annotations in Caliber, (Annotations).
In the kindle I create a note in a book.
I install the plugin.
In the plugin settings "Add retrieved annotations to" I select the custom column (Annotations).
I connect the kindle and Caliber recognizes it as a device and I give the plugin in Fetch annotations form device connected device, it runs the process bar and I get a sign "Unable to find any annotations on Amazon Kindle matching books in your library"

[davidfor]

Originally Posted by dunhill

Hello people, I have been reading carefully all the instructions and the plugin does not work for me. I'll be doing something wrong, you could guide me.
Create a custom column for annotations in Caliber, (Annotations).
In the kindle I create a note in a book.
I install the plugin.
In the plugin settings "Add retrieved annotations to" I select the custom column (Annotations).
I connect the kindle and Caliber recognizes it as a device and I give the plugin in Fetch annotations form device connected device, it runs the process bar and I get a sign "Unable to find any annotations on Amazon Kindle matching books in your library"

The simplest reason for that message is also the obvious: There are no annotations to be found

But, I'm sure that is not the case.

There are reasons:

• The plugin reads the My Clippings.txt file to get the annotations. This is more a log file than a store. If you remove the file, the plugin will not find any annotations. If you make the annotations on another device/app and they sync to this device, the annotations will not be in the file.
• Calibre matches the books on the device to books in the library. The plugin partly uses this and partly does it's own thing. If that fails, you will get this error. The current released version has a problem with KFX books and will never match them to the library. The last beta fixes that and should improve the the book matching for Kindles.
• Amazon has changed something. They have made some not obvious changes to the "My Clippings.txt" file that have broken the plugin.
• There is a bug. And I am sure that there are bugs. Finding them is the problem.

If you haven't installed the beta, try it. And test by making an new annotation on the device for a book that is matching to you calibre library. That should check that it is working.

If it is still not working, I need a debug log and a copy of the "My Clippings.txt" file. Plus some details of which books are having problems.

[misaifan]

Originally Posted by davidfor

As mentioned a few times, and copied in a post above, I intend to do this. I don't know when I will do it. And, it will only be useful for devices where we can get the actual location of the annotations. That doesn't include Kindle devices.

Thank you for your response. I did try hard to find the answer for it before I decided to actually write a post. Sorry for having missed it.
Then I will use the plugin to the extent my device is able to.

[dunhill]

I have downloaded the beta of post 824 and I still cannot connect the annotations from a kindle paperwhite and Calibre
Image 1 book in the kindle,
image 2 Note of said book,
image 3 in process in Calibre.
Image 4 The file My Clippings.txt exists

[genoasalami]

This was the answer; thanks. I will have to dig around and see if I can figure out how to destroy the duplicate column name from the database.

Originally Posted by davidfor

That error says you already have a column with the lookup name "#myannotations". Or whatever name you used in the "Custom column wizard". The name you enter is used as the column header and is converted to lower case to be used as the lookup name. The error would only happen if you had used that name before.

A possibility is if you created the column, and didn't restart calibre. Then tried to do the configuration again (which is something I have done more times than I want to admit). At that point, I do not think the new column will not be in the list of suitable columns. But, if you tried to create it again, you would get this error.

I will look at putting some error handling in, but, in the meantime, check what calibre has in the the custom column list. The above should be there.

[sweth]

Originally Posted by davidfor

Try the beta published at the end of last year. It should fix things. I expect that the books on the new device are KFX and there was a problem with handling these. It was actually a more general problem in how the plugin matched books on Kindle devices. It always failed for KFX and would occasionally fail for other formats.

The last beta I can find (1.14.3) doesn't find anything either. There are a few KFX files on the device--ones that I bought on Amazon--but most of them are mobi files that I emailed and that were delivered (looked at the filesystem) as AZW3. There was only one file that was showing up in the final resolved path map, the user guide that comes pre-installed, so I added that to my Calibre library, added an annotation to the version on the Kindle, and tried to import annotations, but even though I see that annotation in the clippings file when the Kindle is mounted, the plugin still thinks there are no annotations and returns an error when parsing the clippings file (not sure if it was doing that before I installed the beta).

DEBUG: 534.1 After getting installed books: resolved_path_map={}
DEBUG: 534.1 List of storage devices - storage=[u'/Volumes/Kindle/documents']
DEBUG: 534.1 Searching for books on vol=/Volumes/Kindle/documents using template=*.azw
DEBUG: 534.1 List of books from glob - imported_books=<generator object iglob at 0x12db4c690>
DEBUG: 534.1 Searching for books on vol=/Volumes/Kindle/documents using template=*.azw3
DEBUG: 534.1 List of books from glob - imported_books=<generator object iglob at 0x12c54a320>
DEBUG: 534.1 Have possible book with path=/Volumes/Kindle/documents/Kindle_Users_Guide.azw3
DEBUG: 534.2 Did not find book UUID - book_mi.uuid='None'
DEBUG: 534.2 Book not already in resolved_path_map path=='/Volumes/Kindle/documents/Kindle_Users_Guide.azw3'
DEBUG: 534.2 Searching for books on vol=/Volumes/Kindle/documents using template=*.kfx
DEBUG: 534.2 List of books from glob - imported_books=<generator object iglob at 0x12c54a370>
DEBUG: 534.2 Searching for books on vol=/Volumes/Kindle/documents using template=*.mobi
DEBUG: 534.2 List of books from glob - imported_books=<generator object iglob at 0x12c54a550>
DEBUG: 534.2 Searching for books on vol=/Volumes/Kindle/documents using template=*.pobi
DEBUG: 534.2 List of books from glob - imported_books=<generator object iglob at 0x12c54a320>
DEBUG: 534.2 Searching for books on vol=/Volumes/Kindle/documents using template=*.pdf
DEBUG: 534.2 List of books from glob - imported_books=<generator object iglob at 0x12c54a370>
DEBUG: 534.2 After getting imported books: resolved_path_map={-1: u'/Volumes/Kindle/documents/Kindle_Users_Guide.azw3'}
DEBUG: 534.2 Getting metadata from book. path='/Volumes/Kindle/documents/Kindle_Users_Guide.azw3'
DEBUG: 534.2 Book on device title: 'Kindle Oasis User's Guide, 3rd Edition'
DEBUG: 534.2 Adding title to self.installed_books_by_title: 'Kindle Oasis User's Guide, 3rd Edition'
DEBUG: 534.2 Kindle:get_active_annotations()
Job: 4 Send metadata to device finished
No details available.
DEBUG: 534.3 ParseKindleMyClippingsTxt ERROR: Error trying to read clippings file: 'encoding' is an invalid keyword argument for this function
DEBUG: 534.3 Number of entries retrieved from 'My Clippings.txt'=0
DEBUG: 534.3 Dictionary of installed_books_by_title ={u"Kindle Oasis User's Guide, 3rd Edition": {u'book_id': -1, u'author_sort': u'Unknown'}}
DEBUG: 534.3 Keys/Titles of installed_books_by_title =[u"Kindle Oasis User's Guide, 3rd Edition"]
DEBUG: 534.3 Keys/Titles of installed_books_by_title =[u"Kindle Oasis User's Guide, 3rd Edition"]
DEBUG: 534.5 AnnotationsAction:fetch_usb_device_annotations(DEB UG: [])
DEBUG: 534.5 AnnotationsAction:fetch_device_annotations(Amazon Kindle)
DEBUG: 536.2 AnnotationsAction:fetch_device_annotations(INFO: <p>Unable to find any annotations on Amazon Kindle matching books in your library.</p>)