[GUI Plugin] Annotations

[jhowell]

Originally Posted by dsestero

DDEBUG: 16.7 Exception thrown was=CONT magic number is incorrect (ea 44 52 4d)

You can expect to see that type of error when reading a the main KFX file of a book from a Kindle. The ASCII code 44524d is "DRM", indicating that the file is encrypted.

[davidfor]

Originally Posted by jhowell

You can expect to see that type of error when reading a the main KFX file of a book from a Kindle. The ASCII code 44524d is "DRM", indicating that the file is encrypted.

I thought that was going to be the case when I saw the error. Is there a way to get the metadata? I'll look at the code in calibre tonight, but any hints would be good.

[jhowell]

Originally Posted by davidfor

I thought that was going to be the case when I saw the error. Is there a way to get the metadata? I'll look at the code in calibre tonight, but any hints would be good.

On Kindle devices books in KFX format delivered by Amazon have the metadata in a separate unencrypted KFX file. If the main file is "bookname.kfx" then the associated metadata file is "bookname.sdr/assets/metadata.kfx". If no file with that name exists then the metadata will instead be in the main file.

[davidfor]

With that hint, I looked at the Kindle driver, and there appears to be a method I can use there. So this one just calls the driver to get the metadata for the books on the device.

What's the bet I broke it?

[dsestero]

Good job David, we are getting closer.

As you can see from attached screenshot, now the plugin finds more annotated books. For example, it now finds KFX books downloaded from Amazon.

Yet, some annotated books are not found. I can't understand, for example, why it don't find the annotated book 'Ecologia affettiva' that I sideloaded from Calibre to the device in plain AZW3. It has a lot of annotations, as you can see from the MyClippings.txt that I have uploaded in a previous post.

Furthermore, it seems that selecting more than a book yelds an error and no annotation get fetched. I tried to select 7 out of the 9 green titles and push the button 'Fetch annotations' than I got the following error:

calibre, version 3.8.0
Errore recuperando le annotazioni: <p>Non sono in grado di recuperare annotazioni da Amazon Kindle.</p>

Traceback (most recent call last):
  File "calibre_plugins.annotations.action", line 300, in fetch_device_annotations
  File "calibre_plugins.annotations.action", line 1021, in process_selected_books
  File "calibre_plugins.annotations.action", line 157, in add_annotations_to_calibre
  File "calibre_plugins.annotations.annotations", line 251, in merge_annotations
  File "calibre_plugins.annotations.annotations_db", line 241, in capture_content
TypeError: 'NoneType' object has no attribute '__getitem__'

and the log is attached.

Thanks for your commitment, ciao
Dario

[davidfor]

Originally Posted by dsestero

Good job David, we are getting closer.

As you can see from attached screenshot, now the plugin finds more annotated books. For example, it now finds KFX books downloaded from Amazon.

Great!

Yet, some annotated books are not found. I can't understand, for example, why it don't find the annotated book 'Ecologia affettiva' that I sideloaded from Calibre to the device in plain AZW3. It has a lot of annotations, as you can see from the MyClippings.txt that I have uploaded in a previous post.

The clipping file seems to have that book with the title "Ecologia affettiva (Italian Edition)". But, the metadata coming from the book seems to be "Ecologia affettiva". What does the Kindle display as the title? I'm wondering where the "(Italian Edition)" comes from.

Furthermore, it seems that selecting more than a book yelds an error and no annotation get fetched. I tried to select 7 out of the 9 green titles and push the button 'Fetch annotations' than I got the following error:

The error seems to be happening in the code to merge old annotations with the new. Someone reported that recently and it was because the existing annotation had been edited and wasn't complete. Are you sure it is happening when multiple annotations are selected, or when a particular annotation is selected? Or there might be more than one that is bad.

Can you have a look and tell me? And if there are, can you copy the HTML code for the annotation and send it to me. I'll have a look at it and see if I can come up with a solution that doesn't lose anything.

[dsestero]

Originally Posted by davidfor

Great!

Originally Posted by davidfor

The clipping file seems to have that book with the title "Ecologia affettiva (Italian Edition)". But, the metadata coming from the book seems to be "Ecologia affettiva". What does the Kindle display as the title? I'm wondering where the "(Italian Edition)" comes from.

OK, you got it! The '(Italian Edition)' in the clippings files comes from the fact that I annotated the book after downloading it and so it doesn't matter if afterwards I replaced it with another version with updated metadata.
What now I clearly understand is this: the plugin looks for books which should have the same title in all three places: 1: the metadata in the Calibre library; 2: the metadata on the device; 3: the clipping file. Perfect, now I have a clear rule that I didn't understood before.

Originally Posted by davidfor

The error seems to be happening in the code to merge old annotations with the new. Someone reported that recently and it was because the existing annotation had been edited and wasn't complete. Are you sure it is happening when multiple annotations are selected, or when a particular annotation is selected? Or there might be more than one that is bad.

Can you have a look and tell me? And if there are, can you copy the HTML code for the annotation and send it to me. I'll have a look at it and see if I can come up with a solution that doesn't lose anything.

[/QUOTE]
You are right: it was caused by only one book and it has some annotations previously fetched.
Here is their HTML:

<div class="user_annotations" style="margin:0"><div class="annotation" genre="Insegnamenti, Buddhismo" hash="49d65b1bdec994c1ed118fddcf9514f5" location_sort="000029" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" color="Gray" style="background-color: rgb(211, 211, 211); color: black; font-size: 80%; font-weight: bold; margin: 0px;"><tbody><tr><td class="location">34</td><td class="timestamp" uts="1504093561.0" style="text-align:right">Wed Aug 30 13:46:01 2017</td></tr></tbody></table></div><div class="annotation" genre="Insegnamenti, Buddhismo" hash="26077f5e3c2a9e81ccb7d397699d9ae8" location_sort="000034" reader="Kindle" style="margin:0 0 0.5em 0"><p class="highlight" style="margin:0;text-indent:0.5em;">Talvolta, è necessario fare lo sforzo di generare qualità positive; altre volte ci è richiesto lo sforzo di eliminare quel che non è granché positivo, interiormente o esteriormente. Ci sono momenti in cui è necessario essere guerrieri coraggiosi che combattono con le forze della confusione illusoria per evitare che prendano il comando del nostro cuore e della nostra mente; e in altri, abbiamo bisogno di essere come un genitore che alleva e si prende cura della bontà che è già presente. L’agilità è una grande virtù spirituale.</p><p class="highlight" style="margin:0;text-indent:0.5em;"></p></div><div class="annotation" genre="Insegnamenti, Buddhismo" hash="e81e407a6b5d72e3801d631f29e2a406" location_sort="000068" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" style="background-color:LightGray;color:black;font-size:80%;font-weight:bold;margin:0;" color="Gray"><tbody><tr><td class="location" style="text-align:left">68</td><td class="timestamp" uts="1502699605.0" style="text-align:right">Mon Aug 14 10:33:25 2017</td></tr></tbody></table><p class="highlight" style="margin:0;text-indent:0.5em;">Forse ci accorgiamo che qualunque critica, da chiunque provenga, ci ferisce; se è così, allora abbiamo bisogno di osservare più in profondità. Se perfino il Buddha si trovava a dover fare da ricettacolo della sofferenza non riconosciuta degli altri, perché questo è il biasimo, allora tanto più è probabile che accada a noi. Può essere utile riflettere che le persone buttano all’esterno solo il dolore che non riescono a contenere; lo esprimono come ipercritica. Quando abbiamo la consapevolezza e la capacità di ricevere pienamente noi stessi, semplicemente non siamo portati a biasimare nessuno, né noi stessi, né gli altri.</p></div><div class="annotation" genre="Insegnamenti, Buddhismo" hash="e81e407a6b5d72e3801d631f29e2a406" location_sort="000082" reader="Kindle" style="margin:0 0 0.5em 0"><p class="highlight" style="margin:0;text-indent:0.5em;"></p></div><div class="annotation" genre="Insegnamenti, Buddhismo" hash="2e98a272367b0d9a1aa42446f8a260b7" location_sort="000116" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" style="background-color:LightGray;color:black;font-size:80%;font-weight:bold;margin:0;" color="Gray"><tbody><tr><td class="location" style="text-align:left">116</td><td class="timestamp" uts="1503292850.0" style="text-align:right">Mon Aug 21 07:20:50 2017</td></tr></tbody></table><p class="highlight" style="margin:0;text-indent:0.5em;">tutto è pratica; inclusa la sensazione di non poter praticare con ‘questo’. La saggezza riconosce questa verità.</p><p class="highlight" style="margin:0;text-indent:0.5em;"></p></div><div class="annotation" genre="Insegnamenti, Buddhismo" hash="2e98a272367b0d9a1aa42446f8a260b7" location_sort="000141" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" color="Gray" style="background-color: rgb(211, 211, 211); color: black; font-size: 80%; font-weight: bold; margin: 0px;"><tbody><tr><td class="location">148</td><td class="timestamp" uts="1503896359.0" style="text-align:right">Mon Aug 28 06:59:19 2017</td></tr></tbody></table></div><div class="annotation" genre="Insegnamenti, Buddhismo" hash="c749aa01fe3a1f320c8746074b75f327" location_sort="000148" reader="Kindle" style="margin:0 0 0.5em 0"><p class="highlight" style="margin:0;text-indent:0.5em;">Domando il cuore, ci ritroviamo a non giudicare con reattività tutto quello che sorge; non prendiamo più così facilmente posizione pro o contro. E scopriamo un’accresciuta disponibilità e audacia a offrirci a quel che stiamo facendo proprio in questo momento, con tutto il corpo, con tutto il cuore.</p><p class="highlight" style="margin:0;text-indent:0.5em;"></p></div><div class="annotation" genre="Insegnamenti, Buddhismo" hash="c749aa01fe3a1f320c8746074b75f327" location_sort="000181" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" color="Gray" style="background-color: rgb(211, 211, 211); color: black; font-size: 80%; font-weight: bold; margin: 0px;"><tbody><tr><td class="location">307</td><td class="timestamp" uts="1505422538.0" style="text-align:right">Thu Sep 14 22:55:38 2017</td></tr></tbody></table></div><div class="annotation" genre="Insegnamenti, Buddhismo" hash="d9828b38ae4bec72246a2092ce2e4dc6" location_sort="000307" reader="Kindle" style="margin:0 0 0.5em 0"><p class="highlight" style="margin:0;text-indent:0.5em;">Praticare significa trovare lo spazio dentro di noi per sentire quel che sentiamo, senza ‘diventare’quelle sensazioni.</p></div></div>

I am very satisfied of the work we did together and would like to thank David, jhowell, schnesim and the other contributors to this thread very much.

One last thing: David if you answer to me about the question I asked in post #394 I can provide a more correct translation.

Ciao,
Dario

[Nicolas F]

Originally Posted by dsestero

One last thing: David if you answer to me about the question I asked in post #394 I can provide a more correct translation.

Ciao,
Dario

The & character marks the character used as a shortcut.
In your example, for 'Show &details' the shortcut would be 'd' and if you translate it to '&Visualizza dettagli', the shortcut for italian would then be 'v'.

[dsestero]

Originally Posted by Nicolas F

The & character marks the character used as a shortcut.

Thank you Nicolas.

David, attached you find the updated italian translation.

Have a good day, ciao
Dario

[schnesim]

Originally Posted by davidfor

The attached beta adds a lot of extra debug statements. It should show more what is going on.

Here is my log with the latest beta:

[Terisa de morgan]

Here you have the Spanish translation. Sorry for the delay.

[davidfor]

Originally Posted by schnesim

Here is my log with the latest beta:

Sorry, I completely forgot to look at this.

The good thing is that there are plenty of matches. And plenty that don't match, but I there doesn't seem to be anything they should match. But, there are plenty that don't match that look like they should. Such as:

DEBUG:   72.0  Book on device title:   Einführung in die Analytische Psychologie C.G. Jungs 
DEBUG:   72.5    Annotation for Title==Einführung in die Analytische Psychologie C.G. Jungs
DEBUG:   72.5      Title not found in books on device

DEBUG:   72.0  Book on device title:   Der tägliche Terror der agilen Meetings 
DEBUG:   72.5    Annotation for Title==Der tägliche Terror der agilen Meetings
DEBUG:   72.5      Title not found in books on device

DEBUG:   72.2  Book on device title:   Iwan Krastew und Oliver Jens Schmitt zu Osteuropa 
DEBUG:   72.5    Annotation for Title==Iwan Krastew und Oliver Jens Schmitt zu Osteuropa
DEBUG:   72.5      Title not found in books on device

Looking at these in the log, I did find something. The attached beta has a change to the logging to help see this. And a change to see if I'm right. And I'm torn between hoping it does fix it, and hoping it doesn't because it's something stupid I should have checked before: It looks like the books that aren't matching have a space at the end of the titles in the metadata. I've added quotes around the title in the logging and then stripped the spaces before using it.

The attached beta also has the Spanish translation from Terisa de morgan and an updated French translation from dsestero. Thanks to both of you.

@schnesim: Could you post the German PO file for your translation. I'd like to keep these together for any future needs.

[schnesim]

Awesome, it looks like with your latest changes the plugin now finds all the annotations

Since everything seems to work, I didn't add a log file. Let me know, in case you need it nevertheless.

[davidfor]

Originally Posted by schnesim

Awesome, it looks like with your latest changes the plugin now finds all the annotations

Since everything seems to work, I didn't add a log file. Let me know, in case you need it nevertheless.

Good to hear. But, I would like to see a log file to check that I thought what was happening is what is happening.

And thanks for the translation. I'll probably publish one more beta with this in it and some logging removed and then do a release on the weekend.

[schnesim]

Originally Posted by davidfor

Good to hear. But, I would like to see a log file to check that I thought what was happening is what is happening.

And thanks for the translation. I'll probably publish one more beta with this in it and some logging removed and then do a release on the weekend.

Sure, here is the log.