[GUI Plugin] Annotations

[zenwolfe]

Originally Posted by davidfor

The log has:

DEBUG:   21.8    Annotation for Title=='The Circadian Code: Lose Weight, Supercharge Your Energy, and Transform Your Health from Morning to Midnight'
DEBUG:   21.8      Title not found in books on device

Which means that the plugin cannot match the title that is in the clippings file, with a book on the device. Can you tell me:

• How did the book get on the device? Sideloaded, purchased or via email?
• What is the full file name and path for the book on the device?
• What is the exact title for the book in your calibre library?
• When the device is connected, does calibre show that the book on the device is matched to the library?
• What is the save template in the driver?

I have never liked how the plugin does the book matching for Kindles. This is from before I took over the plugin. The way that it is done obviously has issue. There is a "confidence level" that is used to prompt to check, but, it needs a reasonable match before that happens. Answers to the above questions might help work out how to fix it.

• Sideloaded via Calibre
• Do you mean like, on the Kindle drive when it's attached? If so, "/Volumes/Kindle/documents/Panda, Satchin/Circadian Code_ Lose Weight, Supercharge Your Energy, and Trans Your Health from Morning to Midnight, The - Satchin Panda.azw3"
• The Circadian Code: Lose Weight, Supercharge Your Energy, and Transform Your Health from Morning to Midnight
• It does show that it's matched when the device is connected
• Not sure how to answer this question or what you mean exactly? If you can clarify I can try and figure it out, hopefully?

[droopy]

After I successfully import annotations from my Kobo reader, I'm able to click on the "Annotations" column to bring all the annotated books to rise to the top.
But how can sort the books in a finer way? Let's say these are dates of last annotation:

Alphabet book for Guide Dogs -- Sept 4 2019
Bob the Builder construction textbook for engineers -- March 3 2020
Cat in the Hat -- Jan 1 2020
Dictionary -- Feb 1 2020
Everything You need to know about GUI Plugin Annotations -- Dec 20 2015

I'd like the ability to sort in this way (most recent annotation at the top):

Bob the Builder construction textbook for engineers -- March 3 2020
Dictionary -- Feb 1 2020
Cat in the Hat -- Jan 1 2020
Alphabet book for Guide Dogs -- Sept 4 2019
Everything You need to know about GUI Plugin Annotations -- Dec 20 2015

[davidfor]

Originally Posted by droopy

After I successfully import annotations from my Kobo reader, I'm able to click on the "Annotations" column to bring all the annotated books to rise to the top.
But how can sort the books in a finer way? Let's say these are dates of last annotation:

Alphabet book for Guide Dogs -- Sept 4 2019
Bob the Builder construction textbook for engineers -- March 3 2020
Cat in the Hat -- Jan 1 2020
Dictionary -- Feb 1 2020
Everything You need to know about GUI Plugin Annotations -- Dec 20 2015

I'd like the ability to sort in this way (most recent annotation at the top):

Bob the Builder construction textbook for engineers -- March 3 2020
Dictionary -- Feb 1 2020
Cat in the Hat -- Jan 1 2020
Alphabet book for Guide Dogs -- Sept 4 2019
Everything You need to know about GUI Plugin Annotations -- Dec 20 2015

You can't. The annotations are stored as HTML. There isn't any way to sort on the timestamps in it.

[zenwolfe]

Originally Posted by davidfor

The log has:

DEBUG:   21.8    Annotation for Title=='The Circadian Code: Lose Weight, Supercharge Your Energy, and Transform Your Health from Morning to Midnight'
DEBUG:   21.8      Title not found in books on device

Which means that the plugin cannot match the title that is in the clippings file, with a book on the device. Can you tell me:

• How did the book get on the device? Sideloaded, purchased or via email?
• What is the full file name and path for the book on the device?
• What is the exact title for the book in your calibre library?
• When the device is connected, does calibre show that the book on the device is matched to the library?
• What is the save template in the driver?

I have never liked how the plugin does the book matching for Kindles. This is from before I took over the plugin. The way that it is done obviously has issue. There is a "confidence level" that is used to prompt to check, but, it needs a reasonable match before that happens. Answers to the above questions might help work out how to fix it.

Hi David, any update on if you've been able to take a look at this issue? Replied to this particular set of questions a bit ago, so was curious. Thanks!

[droopy]

Originally Posted by droopy

After I successfully import annotations from my Kobo reader, I'm able to click on the "Annotations" column to bring all the annotated books to rise to the top.
But how can sort the books in a finer way? Let's say these are dates of last annotation:

Alphabet book for Guide Dogs -- Sept 4 2019
Bob the Builder construction textbook for engineers -- March 3 2020
Cat in the Hat -- Jan 1 2020
Dictionary -- Feb 1 2020
Everything You need to know about GUI Plugin Annotations -- Dec 20 2015

I'd like the ability to sort in this way (most recent annotation at the top):

Bob the Builder construction textbook for engineers -- March 3 2020
Dictionary -- Feb 1 2020
Cat in the Hat -- Jan 1 2020
Alphabet book for Guide Dogs -- Sept 4 2019
Everything You need to know about GUI Plugin Annotations -- Dec 20 2015

Originally Posted by davidfor

You can't. The annotations are stored as HTML. There isn't any way to sort on the timestamps in it.

I see. Does anybody have any suggestions then? Or any workarounds?

Update 1: Are annotations and highlights accounted as modifications, to the eyes of Calibre or to the eyes of Kobo?
Update 2: Maybe there's a way, in Calibre, to show just the ebooks with annotations/highlights, and then sort by date, assuming the timestamp changes when there's a new highlight or note on the ebook?

[davidfor]

Originally Posted by droopy

I see. Does anybody have any suggestions then? Or any workarounds?

Update 1: Are annotations and highlights accounted as modifications, to the eyes of Calibre or to the eyes of Kobo?

The modified timestamp is updated. So, you can sort by that. But, you asked if you could sort by the timestamps within the annotations. That you cannot do.

Update 2: Maybe there's a way, in Calibre, to show just the ebooks with annotations/highlights, and then sort by date, assuming the timestamp changes when there's a new highlight or note on the ebook?

You can do a search for all books with an annotation with something like:

#annotations:true

With your column replacing "#annotations".

The plugin also has its own search. That has several options to search what is in the annotations.

[kamichai]

Hi David,
What do I do in the case of a mismatch on uuid? Is there a way to force the annotation to match with the correct book?

[thiago.eec]

Originally Posted by davidfor

Typical Kindle workflow:

The Kindle UI must be set to English for the plugin to be able to retrieve locations and timestamps from your 'My Clippings.txt' file.

My Kindle is in Portuguese (pt-BR). It fetches annotations just the same.

[davidfor]

Originally Posted by thiago.eec

My Kindle is in Portuguese (pt-BR). It fetches annotations just the same.

Support for other languages was added to the plugin a long time ago. I think it might have been before I took over support, but, that statement wasn't updated. I have updated the statement to list the supported languages. I don't know if that is all the languages the Kindle UI supports or not.

[thiago.eec]

Originally Posted by davidfor

Support for other languages was added to the plugin a long time ago. I think it might have been before I took over support, but, that statement wasn't updated. I have updated the statement to list the supported languages. I don't know if that is all the languages the Kindle UI supports or not.

That's it, then.

[mfor]

I'm fairly new to Calibre and the plugin, but I have a question. I'm using a Kobo Forma with kepubs and am trying to back up annotations in Calibre. I've been able to "Fetch annotations from connected device" successfully and pull the annotations into a custom column.

My primary question is the sorting of the annotations. I have not been able to figure out how exactly the annotations are sorted. It's not by date or page. Is this the intended behavior? Should there be a way to control the sort?

A secondary question is that I have the same book on two devices and have annotations on both. When I fetch from one or the other, it always overwrites the annotations on the book in the library. I think it's supposed to merge them? But I'm not totally sure.

Any help with these questions would be appreciated. I looked around but sorry if I missed answers to these elsewhere in the thread.

[kawls]

I load and store my books in Calibre and then sideload them to a Kobo Clara HD. For months, I’ve used the fetching command on the Calibre annotations plugin to sync book annotations made on the Kobo device to the book’s parent copy in my Calibre library. Last night, the Kobo device stopped recognizing these annotations, although Calibre still sees them and they’re still in the book’s metadata. The Kobo device will make and sync new annotations, but it does not find the old ones, even though Calibre matches those books with older annotations to the copies on the device, showing a green checkmark in the “In Library” column.*

Can you help me to understand why the Kobo device is not recognizing these annotations, and how to resolve the bug? *

Steps taken so far:
Updated metadata on device through Calibre…bug persists.
Repaired the device’s Kobo account and then sideloaded…bug persists.
Factory-reset the device through settings page and then sideloaded…bug persists.
Reset the device with new Kobo account and then sideloaded…bug persists.
Downgraded Calibre to 4.21 and then sideloaded…bug persists.
Factory-reset the device manually and then sideloaded…bug persists.
Imported library to new installation of Calibre on different computer and then sideloaded…bug persists.*

Here are my specs:
Kobo Clara HD 4.22.15190
Calibre 4.22
MacOS 10.15.4 *

I’ve attached three (public domain) epubs to this post. Silas Marner contains a new annotation that both the Kobo device and Calibre annotation plugin recognize. The other two books—Whirligigs and The Adventures of Tom Sawyer—contain older annotations that Calibre sees but the Kobo device does not.*I’ve also attached a debugging report for fetching annotations from these three books. The fetching process found only the new annotation in Silas Marner, returning no results for The Adventures of Tom Sawyer or Whirligigs, even though they’re matched in the library. If I delete those two epubs from the Calibre library, move their copies from the Kobo device to my desktop, and then load them from the desktop into Calibre, the annotations plugin does find the annotations.*

[davidfor]

I'm not sure what the problem is because some of the things you state don't make sense. Or at least, don't make sense for the plugin. Or I'm reading it wrong.

The plugin fetches the annotations from the device. And stores them in the calibre library. Each time you do this, any new annotations found on the device for a book should be merged with the annotations in the library. These are there all the time, whether the device is connected or not.

The device has no knowledge of what annotations the plugin has fetched. And there is no way to send them back to the device. The device only knows what annotations are in the database on the device. If the books are purchased from Kobo, the annotations can sync to the Kobo server and can later be retrieved by the device from there. For sideloaded books, the annotations are only on the device. For epubs, they are actually in two places, but the plugin only gets the annotations from the database.

From the log, the annotations on the device match those in the library. So, there are no changes seen. For the other books, the plugin didn't find any annotations.

From what you have said, I can't tell when you took this log. But, some of the actions you took on the device will have wiped the annotations for sideloaded books from the device:

• The Repair would not have done anything for sideloaded books.
• The signout creates a new database, so you would lose the annotations. The plugin would not find any annotations. But, if you opened a sideloaded book, the annotations would have been loaded from the other store. The plugin should find these.
• The factory reset formats the partition the books are on, so you completely lose annotations. The plugin would find no annotations for sideloaded books.
• The same goes for using a new account.
• The calibre download would not have done anything.
• The manual factory reset formats everything. It has the same affect as the other factory reset.
• Exporting and importing the library elsewhere would not have done anything.

The log doesn't tell me much. Enabling the debug option in the plugins configuration will show a lot more information. Including when it is reading the actual database.

I have done a quick check here and it working correctly both with new annotations and when there are no changes. And I need to know what it is you actually have on the device and in the library for the book.

[kawls]

Thank you for puzzling this out with me. The problem seems simpler to communicate now that I have the terminology. This is the problem: annotations embedded in an epub's contents.opf file (in <meta name="calibre:user_metadata:#mm_annotations">) are not being passed into the Kobo device's sqlite database when the epub is sideloaded to it from Calibre. Other metadata like the book's title, author, etc. are passed into the database, but annotations metadata are not.

So this is the flow:
SomeBook.epub is downloaded from the Internet, loaded into Calibre, and then sideloaded to a Kobo Clara HD. The Kobo device is then properly ejected from Calibre. Annotations are made in SomeBook.epub using the Kobo device. The device is then re-connected to Calibre, and those annotations are fetched using the annotations plugin. They appear in the "Annotations" column in Calibre. So far everything is working fine. Later, SomeBook.epub is erased from the Kobo device, including from its sqlite database. The device is then re-connected to Calibre and SomeBook.epub is sideloaded again. When it is sideloaded, the original annotations are still in SomeBook.epub's content.opf file. But when the Kobo device is properly ejected and SomeBook.epub is opened on the device, no annotations appear in the book. Upon reconnecting the device to Calibre, the annotations plugin finds zero annotations to import. Meanwhile, the original annotations still remain in the content.opf file of both Calibre's copy and the Kobo device's copy of SomeBook.epub.

I have reset the Kobo device multiple times in hopes of fixing any database corruptions. But when the epub is again re-sideloaded, the annotations in the content.opf file are still not passed into the device's database.

I've attached a debugging report from the plugin. In debugging mode, I loaded an epub with annotations into the Calibre library, sideloaded it to a Kobo device, ejected the device, reconnected the device, and fetched annotations (which the plugin did not find), updated metadata on the device, and fetched annotations again (with same result).

Now that I know of your Kobo Utilities plugin that backs-up the Kobo database, I seemingly won't get in this situation again. But I am in it at present, alas. Any ideas why the annotations aren't being passed into the Kobo database?

[DNSB]

Originally Posted by kawls

Thank you for puzzling this out with me. The problem seems simpler to communicate now that I have the terminology. This is the problem: annotations embedded in an epub's contents.opf file (in <meta name="calibre:user_metadata:#mm_annotations">) are not being passed into the Kobo device's sqlite database when the epub is sideloaded to it from Calibre. Other metadata like the book's title, author, etc. are passed into the database, but annotations metadata are not.

That annotations are not processed into the SQLite database makes sense since they are not part of the standard epub metadata fields. If it's not Dublin Core metadata, it will not be processed. Much as calibre's series metadata is not processed by a Kobo. Check the Dublin Core entry in the MobileRead wiki for more information.

Originally Posted by kawls

Any ideas why the annotations aren't being passed into the Kobo database?

See the above. Visualize Steve Balmer screaming "Dublin Core! Dublin Core!"