[GUI Plugin] Annotations

[davidfor]

Originally Posted by bernard.ryefield

With 7300+ books now it should take under 3mn. Hence my remark about the PDFs above.

The problem with eliminating PDFs from the fetch is that you can add bookmarks for them. And then add an annotation to the bookmark.

Edit:
Actually the first pass (Fetching annotations from device) spends 4mn, it's the second pass (Getting active annotations for KoboTouchExtended) that spends an undefined amount of time.

Unfortunately, I think the only way to find out is to let it run and see. The phases are not well defined. And sort of changed. The released version gets a list of all books on the device and then looks for annotations for these books. The beta version above gets a list of all annotations and then checks if the books are in the current calibre library. Which is better probably depends on the number of books with annotations and what percentage are in the calibre library. If most books have annotations, it is probably the former. But, if there are very few annotations, the latter is probably better.

I've just thought of something to do to help with the latter version. Plus, I might add some statistics gathering to the log file to get a better idea of the counts. I'll try and do these in the next couple of days, but if you can run the last version in debug mode and post the output, that would be good.

Also I don't get the message for the log file any more. Is it removed ?

Do you mean the debug log? If so, I don't think it has changed. But, there probably was an option to not show the message again on the message box. Did you select that?

[bernard.ryefield]

Originally Posted by davidfor

The problem with eliminating PDFs from the fetch is that you can add bookmarks for them. And then add an annotation to the bookmark.

If I understand correctly, bookmarks saved in PDFs are saved as annotations by the plugin? That's great! As I usually only save comments and not bookmarks in my PDF files, I would change my workflow to use this feature.

Originally Posted by davidfor

Unfortunately, I think the only way to find out is to let it run and see. The phases are not well defined. And sort of changed. The released version gets a list of all books on the device and then looks for annotations for these books. The beta version above gets a list of all annotations and then checks if the books are in the current calibre library. Which is better probably depends on the number of books with annotations and what percentage are in the calibre library. If most books have annotations, it is probably the former. But, if there are very few annotations, the latter is probably better.

I've just thought of something to do to help with the latter version. Plus, I might add some statistics gathering to the log file to get a better idea of the counts. I'll try and do these in the next couple of days, but if you can run the last version in debug mode and post the output, that would be good.

I'll wait then. I have the 1.4.8 version but Calibre tells me the last version is 1.4.7. Which one do I use for the test?

Originally Posted by davidfor

Do you mean the debug log? If so, I don't think it has changed. But, there probably was an option to not show the message again on the message box. Did you select that?

Nevermind, I forgot to restart Calibre in debug mode...

[davidfor]

Originally Posted by bernard.ryefield

If I understand correctly, bookmarks saved in PDFs are saved as annotations by the plugin? That's great! As I usually only save comments and not bookmarks in my PDF files, I would change my workflow to use this feature.

The plugin reads the database on the device. If there are bookmarks/annotations/highlights in there for books in your calibre library, they will be fetched. Someone has told me that the Kobo firmware reads the bookmarks already in PDFs and displays them in annotations list on the device. But, I haven't been able to reproduce this. Including with a file they gave me.

What I meant was that you can tap the top-right corner of the screen when reading a PDF and add a bookmark. Then, in the annotations list for the PDF, you can add an annotation to the bookmark. Because of that, I can't ignore the PDFs when fetching the annotations.

I'll wait then. I have the 1.4.8 version but Calibre tells me the last version is 1.4.7. Which one do I use for the test?

I'd like it with 1.4.8. Then I can compare this to the changes I'm planning to see if it helps.

Nevermind, I forgot to restart Calibre in debug mode...

Which is something that I would never, ever do. And, wondering why the code changes I just made didn't do anything and then noticing I hadn't actually saved them, that's another thing I never, ever do. More than once an hour.

[davidfor]

Here's another test with the small change I referred to the other day. This should help a little when there are multiple bookmarks for each book. But, it probably won't help as much as I was thinking the other day.

I also added a log statement to print the number of bookmarks that were found. This will add some time at the start as I have to run the query as a count before fetching the actual bookmarks. But, the count should help to predict the time it will take.

This has version number 1.4.9.

[bernard.ryefield]

Originally Posted by davidfor

Here's another test with the small change I referred to the other day. This should help a little when there are multiple bookmarks for each book. But, it probably won't help as much as I was thinking the other day.

Well, I lost the counts of the changes so I trust you on this one.

Originally Posted by davidfor

I also added a log statement to print the number of bookmarks that were found. This will add some time at the start as I have to run the query as a count before fetching the actual bookmarks. But, the count should help to predict the time it will take.

This has version number 1.4.9.

I'll try it tomorrow, with the PDF bookmarking+note on a document.

[davidfor]

Originally Posted by bernard.ryefield

Well, I lost the counts of the changes so I trust you on this one.

From the logs, version 1.4.8 took 4435.8 seconds and 1.4.9 took 3882.9 seconds. Or about 74 and 64 minutes respectively. That's a long time.

Looking at the logs, it is the kepubs. To get the chapter details, a second query is needed. That second query is taking a long time. I have had a look at the queries in use, and have split into two queries. One fetches for the kepubs and the other for the other book types. They get run one after the other and the annotations combined into one list.

This has made a big change here, but it does depend on the calibre library. Before, it was taking about over 40 seconds with either library. Now, I get 5 seconds with one library, and 15 seconds with the other! The slower library has more books on the device, but less books with annotations. The faster has less books, but more books with annotations.

The attached version of the plugin has the change. It is version 1.4.10. It should be a lot faster. But, I'm not expecting it to get under 30 minutes. Hopefully I am wrong.

[bernard.ryefield]

Originally Posted by davidfor

The attached version of the plugin has the change. It is version 1.4.10. It should be a lot faster. But, I'm not expecting it to get under 30 minutes. Hopefully I am wrong.

I'll try it and keep you posted as usual

[bernard.ryefield]

This version is much faster on Kobo now. I still noticed some glitches: one of my books has not its annotations retrieved (and not mentioned in the log file). The final dialog box is way too wide and I have to arrange manually the columns to read the infos.

[davidfor]

Originally Posted by bernard.ryefield

This version is much faster on Kobo now.

Good to hear.

I still noticed some glitches: one of my books has not its annotations retrieved (and not mentioned in the log file).

To look at that, I would need to see the database from the device. Can you send it to me? Along with the title of the missing book.

The final dialog box is way too wide and I have to arrange manually the columns to read the infos.

Do you mean the import list? If so, you can resize it. The size should be remembered for next time. But, I can't remember if the column sizes are remembered.

[bernard.ryefield]

Originally Posted by davidfor

Good to hear.

To look at that, I would need to see the database from the device. Can you send it to me? Along with the title of the missing book.


Do you mean the import list? If so, you can resize it. The size should be remembered for next time. But, I can't remember if the column sizes are remembered.

Unfortunately my internet connection is too unstable at the moment to send 600MB, I have to try 2 or 3 times before I can download a 40MB PDF and the upload is worse. I'll try again to resize the window and keep you posted.

[bradbowman]

Marvin 3 offers a number of options for exporting annotations (epub, html, csv, pdf, and its own own .marvinz format). Given that Marvin XD is now defunct, it would be very useful if importing from any of these Marvin 3 export files could be added to the Annotations plugin.

Is that something you might consider adding in the future?

[davidfor]

Yes, I will consider adding that. I would need some sample files to look at. The simplest is probably the CSV, but the marvinz would be good to look at.

Having said that, the Marvin XD plugin will probably be revived. From it's thread, the changes in Marvin should mean external applications can reach the data files.

[bradbowman]

If you would like exemplar export files, I will certainly provide some for you. If it makes more sense to wait for Marvin XD then that's fine too. Let me know how I can help

[davidfor]

Having the files won't harm. But, I'm not promising anything. It's most likely to happen when I get bored or curious about the files. There's a pretty good chance that Marvin XD will be updated first

[petzi]

This plugin is very important for me as I would like to read (and annotate) on my new kindle oasis some ebooks not bought by amazon. I tried the plugin two weeks ago and it worked. But today I got the following error messages:

Unable to fetch Annotation from Kindle

with the following details:

calibre, version 2.61.0
Error fetching annotations: <p>Unable to fetch annotations from Amazon Kindle.</p>

Traceback (most recent call last):
File "calibre_plugins.annotations.action", line 289, in fetch_device_annotations
File "calibre_plugins.annotations.action", line 988, in process_selected_books
File "calibre_plugins.annotations.action", line 141, in add_annotations_to_calibre
TypeError: 'NoneType' object has no attribute '__getitem__'

This error messages appear whenever I choose "Fetch Annotation from Amazon Kindle": I will get the window with all the books (two) where the plugin has found annotations. But when I try to fetch them I get the above message.

Whenever I try the other command "Find Annotation" I get another error message:

TypeError: 'NoneType’ Object has no attribute ‚__getitem__'

With this error message calibre crashes after I answer the dialog box with whatever choice I choose. (Show Details, Copy to Clipboard, OK).

Two weeks ago this plugin worked. with the same device and *almost* same circumstances

• besides the installation of Zotero Metadata Importer (ZMI) and Import List (which I de-installed for test purposes but with no effect)
• Changes in the Plugins "Goodreads" and "View Manager" which I cannot de-install because of many adaptations I have done with them
• and the transfer of a new (second) book to the kindle to annotate and later to fetch the annotation

For test purposes I have deleted these two books from kindle oasis and ran the programme in debug modus again. Fetching produced a correct message that there are no books with annotation. But "Find Annotation" produced the same error as before. I included the log file as attachment.

Any help will be highly appreciated.