[GUI Plugin] Annotations - Page 27

schnesim · 09-24-2017, 02:23 AM

For me the latest beta doesn't find any annotations whatsoever.

Here is the log:

davidfor · 09-24-2017, 06:44 AM

Quote:

Originally Posted by schnesim

For me the latest beta doesn't find any annotations whatsoever.

Here is the log:

Looks like I screwed up and took out one of the fixes for getting the type of annotations for German. Sorry. The attached beta should fix it.

Nicolas F · 09-24-2017, 09:41 AM

Quote:

Originally Posted by davidfor

That's one of the things I forgot about. Those two strings are used in comparisons in the code. I have to check if translating them breaks something. And if it does, change some of the other code.

Just letting you know that the translation doesn't seem to break anything. ("All style" and "All reader" at line 89 and 90 of find_annotation.py)

The search works as intended, and the two sentences aren't used anywhere else in the code.

dsestero · 09-25-2017, 04:45 PM

OK David. I have my ebooks subdivided in different libraries and I work with both a GNU/Linux workstation and a Mac.

Attached you find two images from my Linux workstation showing fetched books with the previous 1.4.6 beta and the last 1.4.8 one when the library 'Informatica' is selected as current Calibre library. As a side note, I would like to mention that this window now is correctly resizable on Linux too, while previously on Linux appeared blocked with a size bigger than the screen estate.
Attached you find also the Calibre log obtained enabling the plugin debug log and restarting Calibre in debug mode with the kindle connected.

Furthermore, you find other two images from my Mac showing fetched books with the previous 1.4.6 beta and the last 1.4.8 one when the library ‘Calibre Library Dario’ (my main library) is selected as current Calibre library.

Finally, the full MyClippings.txt is also attached.

Hope you have enough information for debug :-)

I would like to ask you something about the translation
In the translation: what is the meaning of '&' in strings like: '&Copy to clipboard' or 'Show &details'? I noticed today that I haven't translated the words with the '&' because thought it was a kind of variable but, in fact, it appears untranslated in italian. Am I safe to translate 'Show &details' in 'Visualizza dettagli'? If the answer is yes you find attached the updated translation (without the &).

Ciao,
Dario

glire · 09-29-2017, 09:47 AM

Quote:

Originally Posted by davidfor

The error is in code that is interpreting the annotations previously fetched ready to add the new annotations. It looks like one of the existing annotations has been changed and no longer contains the expected HTML code. If you can work out which book it is, then you can fix or remove the annotation.

I'll have a think of how to handle this better, but, I'm not sure if there is anything that can be done except removing the existing annotation or reporting an error and not updating it.

You were right, I had manually changed the HTML from a book

I removed it and works just fine.

Thank you

schnesim · 10-02-2017, 04:22 AM

Quote:

Originally Posted by davidfor

Looks like I screwed up and took out one of the fixes for getting the type of annotations for German. Sorry. The attached beta should fix it.

Unfortunately the latest beta didn't resolve the issue. It still doesn't find any annotations.

davidfor · 10-02-2017, 08:00 AM

Quote:

Originally Posted by schnesim

Unfortunately the latest beta didn't resolve the issue. It still doesn't find any annotations.

According to the log, it is finding plenty of annotations. But, they aren't matching back to a book.

There are a lot of errors getting the metadata. The stack dump doesn't suggest they are from the plug in, but the numbers suggest maybe they are. I've made a change in the attached beta to pass the format type when getting the metadata from the books. Hopefully that will fix it.

schnesim · 10-02-2017, 03:39 PM

Thanks, now we're back to normal. Meaning it finds some of the annotations but not all.

Upon analysing the log file I noticed a ton of statements saying that it doesn't find the book on the device

Code:

DEBUG:   27.0  Kindle:get_active_annotations()
DEBUG:   27.2   Number of entries retreived from 'My Clippings.txt'=452
DEBUG:   27.2    Annotation for Title==Italienische Reise — Band 1 (German Edition)
DEBUG:   27.2      Title not found in books on device

I dug deeper, added a few log statements and what I found out is:
The log says there are 147 books on the device, but the routine titled

Code:

#  Add installed books to the database

only logs out 122.

Could you add more log statements to figure out, why it's missing a few books?

davidfor · 10-02-2017, 06:56 PM

Quote:

Originally Posted by schnesim

Thanks, now we're back to normal. Meaning it finds some of the annotations but not all.

Good to hear. I'm not sure if this got broken because of something I did or a change in calibre that I didn't notice.

Quote:

Upon analysing the log file I noticed a ton of statements saying that it doesn't find the book on the device

Code:

DEBUG:   27.0  Kindle:get_active_annotations()
DEBUG:   27.2   Number of entries retreived from 'My Clippings.txt'=452
DEBUG:   27.2    Annotation for Title==Italienische Reise — Band 1 (German Edition)
DEBUG:   27.2      Title not found in books on device

I dug deeper, added a few log statements and what I found out is:
The log says there are 147 books on the device, but the routine titled

Code:

#  Add installed books to the database

only logs out 122.

Could you add more log statements to figure out, why it's missing a few books?

I'll have a look at where there are some useful spots for logging. But, are there any patterns you can see? Format is the obvious one. And source - sideloaded, emailed to Kindle or bought and downloaded - could also be involved. Are the books shown as on the device in calibre?

dsestero · 10-03-2017, 02:47 AM

Quote:

Originally Posted by davidfor

are there any patterns you can see? Format is the obvious one. And source - sideloaded, emailed to Kindle or bought and downloaded - could also be involved. Are the books shown as on the device in calibre?

Hi, just to summarize my findings until now:
email to kindle: ok
sideloading mobi, azw3: ok
downloading mobi, azw3: I need to investigate
downloaded kfx (the format that nowadays Amazon use for most books): no

Attached you find the image of the fetched books from the latest beta: like schnesim I got more annotated books fetched than with 1.4.8.

Ciao,
Dario

schnesim · 10-03-2017, 03:32 AM

Quote:

Originally Posted by davidfor

I'll have a look at where there are some useful spots for logging. But, are there any patterns you can see? Format is the obvious one. And source - sideloaded, emailed to Kindle or bought and downloaded - could also be involved. Are the books shown as on the device in calibre?

I can't spot any patterns since it finds annotations in books of all kinds of format and source.

And yes, all the 147 books are being shown in calibre.

davidfor · 10-03-2017, 07:06 AM

That's encouraging. It looks like we are getting closer.

The attached beta adds a lot of extra debug statements. It should show more what is going on.

But, looking at the code while adding these, I can see a possibility. It looks like the code is getting all the books and formats from calibre and trying to match them to what is on the device. It looks like if there is a mismatch of formats, those books won't be found. So, if you have a MOBI in the library and a KFX on the device, it probably isn't going to be selected by the plugin. Or at least that's how I read the code. The debug statements should show if this is the case.

dsestero · 10-03-2017, 07:44 AM

Hi David, unfortunately last beta presents the following error when 'Fetch annotations from device' is selected:

Code:

calibre, version 3.8.0
ERRORE: Eccezione non gestita: <b>TypeError</b>:_log() takes at most 2 arguments (3 given)

calibre 3.8  embedded-python: True is64bit: True
Darwin-15.6.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '15.6.0', 'Darwin Kernel Version 15.6.0: Sun Jun  4 21:43:07 PDT 2017; root:xnu-3248.70.3~1/RELEASE_X86_64')
Python 2.7.12
OSX: ('10.11.6', ('', '', ''), 'x86_64')
Interface language: it
Successfully initialized third party plugins: Generate Cover (1, 5, 21) && Ex Libris (1, 0, 19) && Reading List (1, 6, 6) && EpubMerge (2, 1, 1) && Count Pages (1, 8, 0) && KFX Output (1, 10, 0) && Annotations (1, 6, 10)
Traceback (most recent call last):
  File "calibre_plugins.annotations.action", line 360, in fetch_usb_device_annotations
  File "calibre_plugins.annotations.action", line 555, in get_annotated_books_on_usb_device
  File "/var/folders/1t/t88g5dl97jd29gq0kxqldc6w0000gn/T/calibre_annotations_plugin/Kindle.py", line 148, in get_installed_books
  File "/var/folders/1t/t88g5dl97jd29gq0kxqldc6w0000gn/T/calibre_annotations_plugin/Kindle.py", line 272, in _get_installed_books
  File "/var/folders/1t/t88g5dl97jd29gq0kxqldc6w0000gn/T/calibre_annotations_plugin/Kindle.py", line 248, in resolve_paths
TypeError: _log() takes at most 2 arguments (3 given)

and then it hangs.

I am curious about your hypothesis because I have books downloaded in KFX format on the device and that same book available in different formats, including KFX, in the Calibre library. I will try with next beta.

The strange thing is that it fetches books that are not marked as both in Calibre and device (though they are there: different formats and/or different title/author data order) while it does'nt fetch books marked as in both or synced.

Ciao
Dario

davidfor · 10-03-2017, 09:07 AM

Quote:

Originally Posted by dsestero

Hi David, unfortunately last beta presents the following error when 'Fetch annotations from device' is selected:

Code:

calibre, version 3.8.0
ERRORE: Eccezione non gestita: <b>TypeError</b>:_log() takes at most 2 arguments (3 given)

calibre 3.8  embedded-python: True is64bit: True
Darwin-15.6.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '15.6.0', 'Darwin Kernel Version 15.6.0: Sun Jun  4 21:43:07 PDT 2017; root:xnu-3248.70.3~1/RELEASE_X86_64')
Python 2.7.12
OSX: ('10.11.6', ('', '', ''), 'x86_64')
Interface language: it
Successfully initialized third party plugins: Generate Cover (1, 5, 21) && Ex Libris (1, 0, 19) && Reading List (1, 6, 6) && EpubMerge (2, 1, 1) && Count Pages (1, 8, 0) && KFX Output (1, 10, 0) && Annotations (1, 6, 10)
Traceback (most recent call last):
  File "calibre_plugins.annotations.action", line 360, in fetch_usb_device_annotations
  File "calibre_plugins.annotations.action", line 555, in get_annotated_books_on_usb_device
  File "/var/folders/1t/t88g5dl97jd29gq0kxqldc6w0000gn/T/calibre_annotations_plugin/Kindle.py", line 148, in get_installed_books
  File "/var/folders/1t/t88g5dl97jd29gq0kxqldc6w0000gn/T/calibre_annotations_plugin/Kindle.py", line 272, in _get_installed_books
  File "/var/folders/1t/t88g5dl97jd29gq0kxqldc6w0000gn/T/calibre_annotations_plugin/Kindle.py", line 248, in resolve_paths
TypeError: _log() takes at most 2 arguments (3 given)

and then it hangs.

Not paying attention to the log statements. The attached should fix it.

Quote:

I am curious about your hypothesis because I have books downloaded in KFX format on the device and that same book available in different formats, including KFX, in the Calibre library. I will try with next beta.

Reading the code seems to suggest this, but I'll need to see a log to be sure.

Quote:

The strange thing is that it fetches books that are not marked as both in Calibre and device (though they are there: different formats and/or different title/author data order) while it does'nt fetch books marked as in both or synced.

I'm not that surprised to hear either of those. The book matching is different. There is some attempt in the plugin to match on ids, but it will fall back to title and author. If there are examples of each of these in the log, can you mark a couple. That should help me see what is happening.

dsestero · 10-03-2017, 11:38 AM

It seems to me that the plugin is looking correctly to find books on the device on the basis of known patterns, like *.azw3 or *.kfx but then there are books (kfx mostly) for which there are errors like those shown below:

Code:

DEBUG:   16.7      Searching for books on vol=/media/dario/Kindle/documents using template=*.kfx
DEBUG:   16.7      Have possible book with path=/media/dario/Kindle/documents/Ultimo viene il corvo_B00C2H3ZO2.kfx
DEBUG:   16.7  _get_metadata. path=/media/dario/Kindle/documents/Ultimo viene il corvo_B00C2H3ZO2.kfx, stream_type=kfx
DEBUG:   16.7      Unable to get metadata from book. path=/media/dario/Kindle/documents/Ultimo viene il corvo_B00C2H3ZO2.kfx
DEBUG:   16.7      Exception thrown was=CONT magic number is incorrect (ea 44 52 4d)
DEBUG:   16.7      Have possible book with path=/media/dario/Kindle/documents/Il signore degli anelli (Vintage)_B00671AB3M.kfx
DEBUG:   16.7  _get_metadata. path=/media/dario/Kindle/documents/Il signore degli anelli (Vintage)_B00671AB3M.kfx, stream_type=kfx
DEBUG:   16.7      Unable to get metadata from book. path=/media/dario/Kindle/documents/Il signore degli anelli (Vintage)_B00671AB3M.kfx
DEBUG:   16.7      Exception thrown was=CONT magic number is incorrect (ea 44 52 4d)

or:

Code:

DEBUG:   22.4      Have possible book with path=/media/dario/Kindle/documents/Il cuore saggio_B01KTTPZG0.kfx
DEBUG:   22.4  _get_metadata. path=/media/dario/Kindle/documents/Il cuore saggio_B01KTTPZG0.kfx, stream_type=kfx
DEBUG:   22.4      Unable to get metadata from book. path=/media/dario/Kindle/documents/Il cuore saggio_B01KTTPZG0.kfx
DEBUG:   22.4      Exception thrown was=CONT magic number is incorrect (ea 44 52 4d)

Of couse, when looking at annotations the plugin writes log like:

Code:

DEBUG:   29.8    Annotation for Title==Il cuore saggio (Italian Edition)
DEBUG:   29.8      Title not found in books on device

Conversely, when books are found, they present a log like this:

Code:

DEBUG:   16.1      Searching for books on vol=/media/dario/Kindle/documents using template=*.azw3
DEBUG:   16.1      Have possible book with path=/media/dario/Kindle/documents/Ascolta il tuo cane (Salani Saggi)_B006664BDO.azw3
DEBUG:   16.1  _get_metadata. path=/media/dario/Kindle/documents/Ascolta il tuo cane (Salani Saggi)_B006664BDO.azw3, stream_type=azw3
DEBUG:   16.1      Have possible book with path=/media/dario/Kindle/documents/Palomar (Oscar opere di Italo Calvino Vol. 12)_B00C2H3Y3Y.azw3
DEBUG:   16.1  _get_metadata. path=/media/dario/Kindle/documents/Palomar (Oscar opere di Italo Calvino Vol. 12)_B00C2H3Y3Y.azw3, stream_type=azw3

followed by:

Code:

DEBUG:   29.0  _get_metadata. path=/media/dario/Kindle/documents/Palomar (Oscar opere di Italo Calvino Vol. 12)_B00C2H3Y3Y.azw3, stream_type=azw3
DEBUG:   29.1  Book on device title: Palomar (Oscar opere di Italo Calvino Vol. 12) (Italian Edition)
DEBUG:   29.1  _get_metadata. path=/media/dario/Kindle/documents/Ascolta il tuo cane (Salani Saggi)_B006664BDO.azw3, stream_type=azw3
DEBUG:   29.1  Book on device title: Ascolta il tuo cane (Salani Saggi) (Italian Edition)

and then:

Code:

DEBUG:   29.7    Annotation for Title==Palomar (Oscar opere di Italo Calvino Vol. 12) (Italian Edition)
DEBUG:   29.7      Found book_id=-52

Hope this would help.

Ciao,
Dario

10-03-2017, 07:06 AM	#402
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	That's encouraging. It looks like we are getting closer. The attached beta adds a lot of extra debug statements. It should show more what is going on. But, looking at the code while adding these, I can see a possibility. It looks like the code is getting all the books and formats from calibre and trying to match them to what is on the device. It looks like if there is a mismatch of formats, those books won't be found. So, if you have a MOBI in the library and a KFX on the device, it probably isn't going to be selected by the plugin. Or at least that's how I read the code. The debug statements should show if this is the case. Last edited by davidfor; 10-03-2017 at 09:07 AM. Reason: Removed beta as it was no good.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] KindleUnpack - The Plugin	DiapDealer	Plugins	523	07-15-2025 06:45 PM
[GUI Plugin] Open With	kiwidude	Plugins	404	02-21-2025 05:42 AM
[GUI Plugin] Annotations (closed)	GRiker	Plugins	68	06-17-2014 06:11 AM
[GUI Plugin] KiNotes	-axel-	Plugins	0	07-14-2013 06:39 PM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

Advert

Advert