[GUI Plugin] Annotations

[davidfor]

Originally Posted by kovidgoyal

Qt's rich text editor does not preserve html classes sadly. It might preserve id/name/title attributes, you could try with those.

That's a pretty severe omission on Qt's part. I think it is losing the it as well, but, I don't know about the others.

For the record, this will probably affect the built-in but experimental annotation fetching implemented for Kindles and Kobo. From memory, it also uses a class attribute to identify where the annotations start.

[kovidgoyal]

probably the safest solution is to use a text pattern of invisible unicode chars at the start to identify it.

[Kicweed]

Originally Posted by davidfor

I can see the problem. It looks like the annotation has been edited in the library. In fact, it looks like the text in the annotation column was never put there by the plugin. Is this the case?

I don't know to be honest, unfortunately I don't know when it stopped working so I have no idea why it might have stopped working.

I tried the following which worked (apart from a strange issue, more on that latere): I created a new column to hold the plugin's fetched annotations and pointed the plugin to that new column. It handily offered to move existing annotations to the new column, which I accepted, a very thoughtful touch, thank you for that.

The strange this is that some of my annotations seem to be appearing out of order. For example, they appear as follows:

• Chapter 4 - Annot 2
• Chapter 1 - Annot 1
• Chapter 4 - Annot 1
• Chapter 2 - Annot 1
• Chapter 1 - Annot 2
• etc

The annotations all seem to be there but in some cases in the wrong order. I haven't had the time to look at this more thoroughly, I'm thinking it might be an issue with the format of the new column but I can't be sure. I'll try to investigate again the following days.

[edderf]

Originally Posted by jopping

Thanks for making such a great plugin @davidfor!

I'm trying to customize the Annotations plugin to do this:


However, every time I click on the Customize plugin... button, Calibre returns the following error:

This occurs despite me trying to restart Calibre multiple times.

I was wondering if you could give me any help into how to solve this problem? It'd be amazing if I could separate my annotations into a custom field! Thanks!

Here's the debug log when I try to customize the plugin in debug mode:

I have the same issue using calibre 4.x , plugin do not load at all, asking to restart on customize.
It work on calibre 3.48.0 (portable)

[DNSB]

Originally Posted by edderf

I have the same issue using calibre 4.x , plugin do not load at all, asking to restart on customize.
It work on calibre 3.48.0 (portable)

To quote davidfor's post #645 in this thread (bolding mine):

Originally Posted by davidfor

I have just released version 1.10.0 of the plugin. I should have done this before, but kept forgetting. The release of calibre 4.0 has triggered this release as it has a change that breaks this plugin. I'll be fixing this soon, but am making this release as the last version to support calibre 3.x and earlier.

[edderf]

Originally Posted by DNSB

To quote davidfor's post #645 in this thread (bolding mine):

Oh, sorry i miss the post

[davidfor]

Thanks for reminding me. I have had a build for 4.x but had forgotten to upload it.

The attached beta has the following changes:

• Fix: Problem when couldn't get old annotations.
• Fix: Support calibre version 4. Use QTextEdit instead of QWebView. Should be compatible with 3.x.

This seems to be compatible with calibre 3.x. I have only tested a couple of the more recent builds, but it should be OK. It might not be compatible with calibre version 2. If anyone tests the older versions, please tell me the results.

It will partly fix the problem reported by @Kicweed. It will not error, but, the existing annotations will probably be lost. I am still thinking about how to best handle this.

[Kicweed]

Originally Posted by davidfor

Thanks for reminding me. I have had a build for 4.x but had forgotten to upload it.

The attached beta has the following changes:

• Fix: Problem when couldn't get old annotations.
• Fix: Support calibre version 4. Use QTextEdit instead of QWebView. Should be compatible with 3.x.

This seems to be compatible with calibre 3.x. I have only tested a couple of the more recent builds, but it should be OK. It might not be compatible with calibre version 2. If anyone tests the older versions, please tell me the results.

It will partly fix the problem reported by @Kicweed. It will not error, but, the existing annotations will probably be lost. I am still thinking about how to best handle this.

How do we manually install this? I assume it has to be installed manually because you said it's a beta and also because I had a look in Calibre's own plugin page for available plugin updates and nothing came up.

Also, to continue with something I mentioned earlier. The annotations are definitely being fetched out of order. I don't know if they are being fetched out of order or if they're being saved out of order in Calibre, the end result is the same to how they appear :-)

I've created a blank, new column via the plugin's setup page to have a clean(er) test case. The annotations appear in the wrong order in the original, 'entry' column, not in the joined one.

I've attached an image to make it clear, this is how they appear right after the plugin fetched the annotations in a freshly created column. You can see both the Chapter and the date order are wrong:
[oversized, inline image made as attachment. Please follow MR image guidelines. [/mod]

[theducks]

Plugins can be installed from Preferences: Plugins: Load from File (button near bottom)

[Philantrop]

Originally Posted by davidfor

It will partly fix the problem reported by @Kicweed. It will not error, but, the existing annotations will probably be lost. I am still thinking about how to best handle this.

Thanks, David, this beta works fine here with Calibre 4.3.0 using a Kindle. If I edit fetched annotations in Calibre and re-fetch from the ebook, the edited annotations are indeed lost.

I'm wondering, though, if such a usecase is really important enough to warrant much effort: I for one take my notes, highlight stuff, etc. while reading a book. Once I'm finished with the entire book, I fetch the annotations and edit them. I never re-fetch from the Kindle and, thus, wouldn't be bothered if the annotations in Calibre were overwritten.If I really cared, I guess I'd just have two custom columns - one for the "raw" un-edited annotations and one for the edited version.
I certainly could be "special" in that, though. ;-)

Or you could issue a warning (with an option to cancel the operation) if there are existing annotations before overwriting them. Like that, people could "evacuate" their annotations and merge them manually and you'd have all the time in the world to find a good solution.

[davidfor]

Originally Posted by Kicweed

Also, to continue with something I mentioned earlier. The annotations are definitely being fetched out of order. I don't know if they are being fetched out of order or if they're being saved out of order in Calibre, the end result is the same to how they appear :-)

I've created a blank, new column via the plugin's setup page to have a clean(er) test case. The annotations appear in the wrong order in the original, 'entry' column, not in the joined one.

I've attached an image to make it clear, this is how they appear right after the plugin fetched the annotations in a freshly created column. You can see both the Chapter and the date order are wrong:

What book format are you using? If these are epubs, are there is a bug that messes with the chapter and position shown in the annotations list if you close and reopen the book. There has also been a bug in recent firmware where the location isn't stored properly. But, for that bug, the annotations are not likely to be retrieved.

Can you run calibre in debug mode when you do a fetch? There should be enough information in the log to show what is happening. But, I might also need to look at the database to see the order that is there.

[davidfor]

Originally Posted by Philantrop

Thanks, David, this beta works fine here with Calibre 4.3.0 using a Kindle. If I edit fetched annotations in Calibre and re-fetch from the ebook, the edited annotations are indeed lost.

I'm wondering, though, if such a usecase is really important enough to warrant much effort: I for one take my notes, highlight stuff, etc. while reading a book. Once I'm finished with the entire book, I fetch the annotations and edit them. I never re-fetch from the Kindle and, thus, wouldn't be bothered if the annotations in Calibre were overwritten.If I really cared, I guess I'd just have two custom columns - one for the "raw" un-edited annotations and one for the edited version.
I certainly could be "special" in that, though. ;-)

Or you could issue a warning (with an option to cancel the operation) if there are existing annotations before overwriting them. Like that, people could "evacuate" their annotations and merge them manually and you'd have all the time in the world to find a good solution.

There is a fair amount of code in the plugin attempting to merge the annotations fetched at different times and from different devices. The intention of the plugins original author was that it would allow collecting and storing annotations in one place. With that, being able to edit them, either to remove an annotation, or change the comment is desirable. The current editor is terrible for this.

Having said that, I have considered an option to replace the annotations. My main use of annotations is to mark errors, or something to look up later. I don't tend to keep them at all. An option to replace rather than merge would work for me.

Another way would be to add the annotations to the end of the existing text if what is there cannot be identified. This is what is done if the comments column is used. If the annotations are edited, it could mean they get repeated.

The last option is to not update the annotations if there is any question about merging them. Or prompt for what to do, either individually or as list after the first pass of updating. This is probably the most work.

[Kicweed]

Originally Posted by davidfor

What book format are you using? If these are epubs, are there is a bug that messes with the chapter and position shown in the annotations list if you close and reopen the book. There has also been a bug in recent firmware where the location isn't stored properly. But, for that bug, the annotations are not likely to be retrieved.

Can you run calibre in debug mode when you do a fetch? There should be enough information in the log to show what is happening. But, I might also need to look at the database to see the order that is there.

I'm always converting my .epubs to .kepubs when sending to my device. Do you know if they are also affected by the bugs you've mentioned? I'll try to get a log at some point and send it to you, thank you in advance.

To be honest, and I say this with deep appreciation for the work you've done, I'd settle for a robust way to "simply" extract annotations from books. It wouldn't even have to sync to Calibre, just extract them to a .txt file. A fancy alternative would be a single, plain .html with a clickable TOC at the top so I can click around to each book's annotations.

I know you can enable a hidden setting in Kobo devices to extract annotations to a plain .txt file, something I've been relying more and more on.

I understand that trying to accommodate two-way updates to annotations increases the complexity dramatically. I won't pretend having an easy solution to that. Still, for my personal use case, all I'm looking for is a robust way to extract annotations from a book to a static, text/html file I can easily go back to when I want to.

[davidfor]

Originally Posted by Kicweed

I'm always converting my .epubs to .kepubs when sending to my device. Do you know if they are also affected by the bugs you've mentioned? I'll try to get a log at some point and send it to you, thank you in advance.

The bugs I mentioned are epub only problems. I don't think that there is anything currently affecting kepubs. I have done some recent changes for kepubs. I'll have to recheck them.

To be honest, and I say this with deep appreciation for the work you've done, I'd settle for a robust way to "simply" extract annotations from books. It wouldn't even have to sync to Calibre, just extract them to a .txt file. A fancy alternative would be a single, plain .html with a clickable TOC at the top so I can click around to each book's annotations.

I know you can enable a hidden setting in Kobo devices to extract annotations to a plain .txt file, something I've been relying more and more on.

I understand that trying to accommodate two-way updates to annotations increases the complexity dramatically. I won't pretend having an easy solution to that. Still, for my personal use case, all I'm looking for is a robust way to extract annotations from a book to a static, text/html file I can easily go back to when I want to.

There are five or six ways to get annotations from Kobo devices. I've detailed them a few times over the years. A search of the Kobo forum should find the post.

[Kicweed]

Originally Posted by davidfor

The bugs I mentioned are epub only problems. I don't think that there is anything currently affecting kepubs. I have done some recent changes for kepubs. I'll have to recheck them.

Let me know if there's anything I could do to help with debugging this :-)

Originally Posted by davidfor

There are five or six ways to get annotations from Kobo devices. I've detailed them a few times over the years. A search of the Kobo forum should find the post.

I hope I didn't get that out wrong, I really appreciate the work you've done. I'm not suggesting your plugin should do something different, you do what you need to scratch your own itch.

I'll have another look around the forums, I might stumble on a useful trick I haven't seen before.