[GUI Plugin] Annotations

[Nicolas F]

Originally Posted by davidfor

Good to hear. But, I would like to see a log file to check that I thought what was happening is what is happening.

And thanks for the translation. I'll probably publish one more beta with this in it and some logging removed and then do a release on the weekend.

One small thing: some images in this plugin gives me an error (I had not noticed since I use a theme usually)

libpng warning: iCCP: known incorrect sRGB profile

Its not a big problem, but I've compressed the images. You can switch them, this way no more libpng error and less space for this plugin

[davidfor]

Originally Posted by Nicolas F

One small thing: some images in this plugin gives me an error (I had not noticed since I use a theme usually)

libpng warning: iCCP: known incorrect sRGB profile

Its not a big problem, but I've compressed the images. You can switch them, this way no more libpng error and less space for this plugin

Is that were those errors are coming from. I've seen them in the logs, but have never bothered looking for it. I definitely hadn't connected them with the plugin.

Thanks.

[davidfor]

Originally Posted by schnesim

Sure, here is the log.

Thanks, and look at all those titles with a space on the end of them. I would have expected the metadata reader to strip them. I might have a look, but doing the strip in plugin is no real harm.

[davidfor]

Originally Posted by Nicolas F

One small thing: some images in this plugin gives me an error (I had not noticed since I use a theme usually)

libpng warning: iCCP: known incorrect sRGB profile

Its not a big problem, but I've compressed the images. You can switch them, this way no more libpng error and less space for this plugin

Hmm, they don't work here. I still get these errors. This is on a Win10 machine. But, I'll include them in a beta few minutes.

[davidfor]

Attached is a beta that I hope will be the last before a release. It has all the fixes for the Kindles, translations for German, Italian, French and Italian. If there are no problems reported with this, I will release it officially in the next few days.

[Nicolas F]

Originally Posted by davidfor

Hmm, they don't work here. I still get these errors. This is on a Win10 machine. But, I'll include them in a beta few minutes.

For me in windows 10 64bits, I get 4 libpng warning with the old version, and none with the latest beta.
Your remaining errors might come from another plugin?

[davidfor]

Originally Posted by Nicolas F

For me in windows 10 64bits, I get 4 libpng warning with the old version, and none with the latest beta.
Your remaining errors might come from another plugin?

The number of errors haven't changed, and, now that I look at them properly, the positions in the logs suggest they are coming from somewhere in the annotations plugins initialization. I am running from source, so it is possible I've missed updating a library somewhere. Or maybe I have something leftover somewhere that needs to be cleaned up. I'll have a better look over the weekend.

[dsestero]

Hi David,

I would like to ask you to have a look at the attached log before making the release. Since the latest betas I thought the plugin works fine but yestarday I tried to fetch the annotations from a recently added book downloaded from Amazon and then added to the Calibre library. It is in kfx format. May be because of the inversion of first/last author name it did not get matched, so I synchronized it in the way you explained to me. You can see in the attached screenshot it is matched now. I then restarted Calibre in debug mode.
Looking at the log I can't understand why the book seems to be recognized, yet when the annotations are found no associated book is found. I attached also the fragment of the MyClippings file.
Do you have an idea why is it happening?
Thanks,
Dario

[davidfor]

Originally Posted by dsestero

Hi David,

I would like to ask you to have a look at the attached log before making the release. Since the latest betas I thought the plugin works fine but yestarday I tried to fetch the annotations from a recently added book downloaded from Amazon and then added to the Calibre library. It is in kfx format. May be because of the inversion of first/last author name it did not get matched, so I synchronized it in the way you explained to me. You can see in the attached screenshot it is matched now. I then restarted Calibre in debug mode.
Looking at the log I can't understand why the book seems to be recognized, yet when the annotations are found no associated book is found. I attached also the fragment of the MyClippings file.
Do you have an idea why is it happening?

The following entries in the log are the relevant ones:

DEBUG:   54.2  resolve_paths. id=505, path=/<storage>/Nhat Hanh, Thich/Lettera d'amore alla Madre Terra (Italian Edition) - Nhat Hanh, Thich.bookmark
DEBUG:   54.2  resolve_paths. fmt=[u'azw3', u'kfx']
DEBUG:   54.2  resolve_paths. fmt=[u'azw3', u'kfx']
DEBUG:   54.2  resolve_paths. looking for book on device: book_path=/Volumes/Kindle/documents/Nhat Hanh, Thich/Lettera d'amore alla Madre Terra (Italian Edition) - Nhat Hanh, Thich.bookmark
DEBUG:   54.2  resolve_paths. looking for book on device: this_fmt=/Volumes/Kindle/documents/Nhat Hanh, Thich/Lettera d'amore alla Madre Terra (Italian Edition) - Nhat Hanh, Thich.kfx
DEBUG:   54.2  resolve_paths. looking for book on device: this_fmt=/Volumes/Kindle/documents/Nhat Hanh, Thich/Lettera d'amore alla Madre Terra (Italian Edition) - Nhat Hanh, Thich.azw3

DEBUG:   64.3      Have possible book with path=/Volumes/Kindle/documents/Lettera d'amore alla Madre Terra_B018EMPPCI.kfx

DEBUG:   65.3  Book on device title: 'Lettera d’amore alla Madre Terra'

DEBUG:   74.7    Annotation for Title=='Lettera d'amore alla Madre Terra (Italian Edition)'
DEBUG:   74.7      Title not found in books on device
DEBUG:   74.7    Annotation for Title=='Lettera d'amore alla Madre Terra (Italian Edition)'
DEBUG:   74.7      Title not found in books on device
DEBUG:   74.7    Annotation for Title=='Lettera d'amore alla Madre Terra (Italian Edition)'
DEBUG:   74.7      Title not found in books on device
DEBUG:   74.7    Annotation for Title=='Lettera d'amore alla Madre Terra (Italian Edition)'
DEBUG:   74.7      Title not found in books on device
DEBUG:   74.7    Annotation for Title=='Lettera d'amore alla Madre Terra (Italian Edition)'
DEBUG:   74.7      Title not found in books on device

The title in the clippings file is "Lettera d'amore alla Madre Terra (Italian Edition)". The title in the books metadata is "Lettera d’amore alla Madre Terra". The plugin is does a first match using the title. That isn't matching, so it doesn't go to the next step with these annotations. So, the version of the book you made the annotations against has "(Italian Edition)", but the version you sent from calibre doesn't. A test of this would be to edit the clippings file and remove the "(Italian Edition)" from the title.

[dsestero]

Originally Posted by davidfor

The title in the clippings file is "Lettera d'amore alla Madre Terra (Italian Edition)". The title in the books metadata is "Lettera d’amore alla Madre Terra". The plugin is does a first match using the title. That isn't matching, so it doesn't go to the next step with these annotations. So, the version of the book you made the annotations against has "(Italian Edition)", but the version you sent from calibre doesn't. A test of this would be to edit the clippings file and remove the "(Italian Edition)" from the title.

Well, the fact is that book was downloaded from Amazon and annotated. When I imported it in the Calibre library I resisted the temptation to strip out the '(Italian edition)' just to have the possibility, afterwards, to match the annotations. You can see that part of the title was retained in the metadata in the Calibre library from the screenshot I attached to the previous post. Now I tried three things:

1. to replace the title with the one with '(Italian Edition)' stripped off in the MyClippings file, as you suggested, but nothing changes;
2. to change the metadata in the Calibre library to match those on the device (hence removing '(Italian Edition)' from the Calibre metadata) but nothing changes either;
3. to synchronize the book on the device to that in the library and even this solution doesn't work

What do you think about it?

Ciao
Dario

[dsestero]

... and here is the log after both renaming in the MyClippings file and in the Calibre library.

Ciao
Dario

[davidfor]

Originally Posted by dsestero

Well, the fact is that book was downloaded from Amazon and annotated. When I imported it in the Calibre library I resisted the temptation to strip out the '(Italian edition)' just to have the possibility, afterwards, to match the annotations. You can see that part of the title was retained in the metadata in the Calibre library from the screenshot I attached to the previous post. Now I tried three things:

1. to replace the title with the one with '(Italian Edition)' stripped off in the MyClippings file, as you suggested, but nothing changes;
2. to change the metadata in the Calibre library to match those on the device (hence removing '(Italian Edition)' from the Calibre metadata) but nothing changes either;
3. to synchronize the book on the device to that in the library and even this solution doesn't work

What do you think about it?

The non-matching is caused by the apostrophe. I didn't notice this before, but the apostrophe in the clippings file is a single straight quote. The one in the title of the book you sent is a "curly" apostrophe. You should be able to see the difference in my post above. This is in addition to the "(Italian Edition)" not matching.

That is a real pain that I hadn't thought about. I might try "dumbing down" the titles before matching them. If I convert them to lower case and to plain ASCII, it should improve the matching. That would be doing the what calibre does when generating file names from title, plus the lower case.

But, for curiosity, could you add an annotation to the book now on you Kindle? I'd like to know what is used in the clipping file. I can see some accented letters in the log, but I would like to know exactly what the device uses before going ahead.

Also, if this looks like it will take a bit of time, I will probably release the plugin as is and then fix this later. I think were the plugin is now, it is working better than the current release version. And gets the translations out to anyone who wants to use them.

[dsestero]

Originally Posted by davidfor

The non-matching is caused by the apostrophe. I didn't notice this before, but the apostrophe in the clippings file is a single straight quote. The one in the title of the book you sent is a "curly" apostrophe. You should be able to see the difference in my post above. This is in addition to the "(Italian Edition)" not matching.

Correct!

Originally Posted by davidfor

That is a real pain that I hadn't thought about. I might try "dumbing down" the titles before matching them. If I convert them to lower case and to plain ASCII, it should improve the matching. That would be doing the what calibre does when generating file names from title, plus the lower case.

It sounds to me like a very good idea!

Originally Posted by davidfor

But, for curiosity, could you add an annotation to the book now on you Kindle? I'd like to know what is used in the clipping file. I can see some accented letters in the log, but I would like to know exactly what the device uses before going ahead.

OK. I did another annotation that you can see below, but please consider that it is the same as the previous ones because that book was never sent to the device from Calibre but only downloaded from Amazon:

Lettera d'amore alla Madre Terra (Italian Edition) (Nhat Hanh, Thich)
- La tua evidenziazione alla posizione 265-268 | Aggiunto in data domenica 22 ottobre 2017 16:27:40

assaporare ogni respiro. Contiene azoto, ossigeno e idrogeno, così come altri oligoelementi, quindi ogni respiro che incameriamo e buttiamo fuori racchiude la Terra. Con ognuno di essi ci viene ricordato che facciamo parte di questo bellissimo pianeta che dà la vita.
==========

Furthermore, I tried to change one annotation in the MyClippings file replacing the straight quote with the curly apostrophe and, even without changing the metadata in the Calibre library (where in the title there is the straight quote), the plugin matched the annotation, as you can see in the attached screenshot!

Originally Posted by davidfor

Also, if this looks like it will take a bit of time, I will probably release the plugin as is and then fix this later. I think were the plugin is now, it is working better than the current release version. And gets the translations out to anyone who wants to use them.

Of course, it is much much better and functional now; thanks a lot for all the work!

Ciao,
Dario

[BeckyEbook]

For samples from Amazon (*_sample.kfx) skip fetch metadata from kindle. Samples do not have a metadata file in the default format (file \assets\metadata.kfx) for the KFX.

…
DEBUG:   62.1      Searching for books on vol=J:\documents using template=*.kfx
DEBUG:   62.1      Have possible book with path=J:\documents\The Son_B009NF6YLM_sample.kfx
Traceback (most recent call last):
  File "site-packages\calibre\devices\kindle\driver.py", line 101, in metadata_from_path
  File "site-packages\calibre\startup.py", line 116, in local_open
IOError: [Errno 2] No such file or directory: u'J:\\documents\\The Son_B009NF6YLM_sample.sdr\\assets\\metadata.kfx'
failed kfx path: J:\documents\The Son_B009NF6YLM_sample.sdr\assets\metadata.kfx
…

---

In file common_utils.py (line 1119):

msg = _("Annotations updated to new appearance settings for {0} {1}.</p>").formt(len(annotation_map), book_word)

should be format.

---

Why mismatch for title and author?

[jhowell]

Originally Posted by BeckyEbook

For samples from Amazon (*_sample.kfx) skip fetch metadata from kindle. Samples do not have a metadata file in the default format (file \assets\metadata.kfx) for the KFX.

Sample books in KFX are an odd exception. For those the metadata is in the main KFX file in the documents folder. That file always has DRM encryption.