[GUI Plugin] Annotations

[davidfor]

Originally Posted by schnesim

According to my investigation so far the cause for the issue appears to be the parsing of the timestamp.

I'll look at that, but I can't remember

If never done any Python coding but nevertheless my plan right now is, extract the timestamp parsing method so I can run it in an IDE and debug what's actually happening. (Also there is a nasty rege which I need to dissect.)

I don't think "nasty" describes that regex properly. I used https://regex101.com/ to test it with a couple of the samples and there wasn't any problems. Except with the initial words on the line it is matching. I can see differences in the timestamp format between the German (Hinzugefügt am Mittwoch, 11. Mai 2016 7.10 Uhr GMT+01:01) and Italian samples (Aggiunto in data venerdì 1 settembre 2017 13:56:10). I'll have to look a the code again to see if that difference could be tripping it up.

[davidfor]

Looking at the book and annotations matching, there are two probable reasons:

- Cannot match the title information in the "My Clippings.txt" with a book in the calibre library.
- Parsing the annotation record doesn't work.

For the latter problem, the debug log should show an entry like:

DEBUG:   31.0  ParseKindleMyClippingsTxt ERROR: could not detect type of record 'Markierung Pos. 103  | Hinzugefügt am Freitag, 10. Juni 2016 20.48 Uhr GMT+01:01'

That error comes from schnesim debug log and was caused by a stray space in the matching string. But, any other problems working out what type of annotation it is, should show this sort of error. If the timestamp doesn't parse, that will cause a problem, but the error is different.

For the title match, the plugin is comparing the title from the annotation with the titles of books it thinks are on the device. This is a simple equality, so there might be some issues there. Also, I just noticed it has two lists of formats. For sideloaded books, they need to have formats in "[u'azw', u'azw1', u'azw3', u'mobi']" and for synced books, extensions in "['*.azw', '*.mobi', '*.pobi']". When it is looking for books, it uses these. I assume neither of those lists is up to date.

[schnesim]

Originally Posted by dsestero

Schnesim, I don't know Python but in the past I was a Java programmer. May be I can help with the regular expression if you point it to me.

Good night,
Dario

date_time_re = r'([0-2]?[0-9] ) : ([0-5][0-9])(?:: ([0-5][0-9])(?:\.([0-9]+))?)?\s*([AP]\.?M)?\s*(?:[A-Z]{3}?([+-][0-2]?[0-9](?::[0-5][0-9])?))?'

Have fun

(I had to add some spaces to prevent some parts being converted to smileys.)

[davidfor]

Originally Posted by schnesim

Have fun

(I had to add some spaces to prevent some parts being converted to smileys.)

If you wrap it in CODE tags, then it doesn't do the smiley substitutions.

date_time_re = r'([0-2]?[0-9]):([0-5][0-9])(?::([0-5][0-9])(?:\.([0-9]+))?)?\s*([AP]\.?M)?\s*(?:[A-Z]{3}?([+-][0-2]?[0-9](?::[0-5][0-9])?))?'

So, from the My Clippings files - German then Italian:

- Markierung Pos. 4921-27  | Hinzugefügt am Montag, 5. Mai 2014 6.49 Uhr GMT+01:01
-  La tua evidenziazione alla posizione 750-750 | Aggiunto in data venerdì 4 settembre 2015 07:44:20

The regex matches the time OK for the Italian version, but it matches the timezone for the German version.

The month selection then fails for the Italian version as it the selection is looking for either a comma or semi-colon to separate the date from preamble on the line. The German version should work if the time was extracted properly.

[davidfor]

Ok, changing the regex to:

date_time_re = r'([0-2]?[0-9])[.:]([0-5][0-9])(?::([0-5][0-9])(?:\.([0-9]+))?)?\s*([AP]\.?M|Uhr)?\s*(?:[A-Z]{3}?([+-][0-2]?[0-9](?::[0-5][0-9])?))?'

That should fix the timestamp matching for German. From schnesim last debug log, I thing that will fix the errors " Unable to parse entries from 'My Clippings.txt'". I shouldn't break the other languages. At least no more than they are now.

And looking at the log file, the grouping of errors isn't obvious. There is some indentation, but the above isn't indented properly. The following are how they should be grouped:

DEBUG:  156.7    title==Kulturgeschichte der Neuzeit (German Edition)
DEBUG:  156.7     Found book_id=-16
DEBUG:  156.7   Unable to parse entries from 'My Clippings.txt'

I'll try and post a beta with the regex tonight. And a more obvious grouping of the messages.

[dsestero]

Originally Posted by davidfor

The month selection then fails for the Italian version as it the selection is looking for either a comma or semi-colon to separate the date from preamble on the line. The German version should work if the time was extracted properly.

Hi,

I found something about it: old annotations, dating up to May 2014, look like this:

La saga di Terramare (I Grandi) (Italian Edition) (Le Guin, Ursula K.)
- La mia evidenziazione Posizione 7030-7031 | Aggiunto il venerdì 7 marzo 14, 05:57:08

Lebannen, questo è. E tu sei. Non c’è sicurezza, e non c’è fine. La parola dev’essere sentita nel silenzio. Deve esserci il buio per vedere le stelle. La danza viene sempre danzata sopra quell’enorme cavità, sopra il terribile abisso.

but, starting from June 2014 they are like:

Il principio passione (Garzanti Saggi) (Italian Edition) (Mancuso, Vito)
- La tua evidenziazione alla posizione 2339-2342 | Aggiunto in data domenica 8 giugno 2014 17:19:05

in natura non esistono mattoncini originari precostituiti e indivisibili dotati di massa propria, ma tutti gli elementi vengono a costituirsi in base alla forza sviluppata dalle relazioni reciproche; l’unica particella dotata di massa propria e generatrice della massa altrui, il bosone di Higgs, esiste ma non in natura, in questa rerum natura che è il nostro mondo, dove decade prontamente e se ne può identificare solo la scia.

and the good news is that it is remained like this until now.

I will try to adapt the regular expression to match both otherwise either is it possible to easily add an option or please consider matching the latter case.

Thanks, together we are nearing to the solution

Dario

[dsestero]

Originally Posted by dsestero

I will try to adapt the regular expression to match both otherwise either is it possible to easily add an option or please consider matching the latter case.

but: do we really need this detailed matching? Would'nt be possible just to match everything from the pipe sign to the end of line? In that way everybody would get the date/time string in his/her own language and the code would be simpler and more general. What do you think?

Dario

[dsestero]

Originally Posted by davidfor

For the title match, the plugin is comparing the title from the annotation with the titles of books it thinks are on the device. This is a simple equality, so there might be some issues there. Also, I just noticed it has two lists of formats. For sideloaded books, they need to have formats in "[u'azw', u'azw1', u'azw3', u'mobi']" and for synced books, extensions in "['*.azw', '*.mobi', '*.pobi']". When it is looking for books, it uses these. I assume neither of those lists is up to date.

Please add to both lists azw3 and kfx (see also post #366).

And, would'nt it be possible to match only the first part of the title? When I import a book in the Calibre library, I usually get rid of some words i.e. '(Italian Edition)' and the like.

Thanks,
Dario

[davidfor]

Originally Posted by dsestero

but: do we really need this detailed matching? Would'nt be possible just to match everything from the pipe sign to the end of line? In that way everybody would get the date/time string in his/her own language and the code would be simpler and more general. What do you think?

Internally, the date/time string is converted to a timestamp. That is used in a few ways including sorting and as part of the key. And as part of the mechanism for merging annotations that have been fetched over time.

Also, I realised I was wrong about the matching for Italian. It should be working. But, the example file you sent, the month for a lot of the entries is "Luglio". The code I have has this in lower case only. That will break the parsing of the timestamp.

[davidfor]

Originally Posted by dsestero

Please add to both lists azw3 and kfx (see also post #366).

I missed that before.

And, would'nt it be possible to match only the first part of the title? When I import a book in the Calibre library, I usually get rid of some words i.e. '(Italian Edition)' and the like.

Part of the problem with that is defining what "the first part of the title" is. How many characters, or do you want it up to character? But, the title in the My Clippings.txt should match whatever you sent to the device from calibre. If you fix the metadata and then send the book to the device before you start reading it, then any annotations should be under the correct title.

[davidfor]

Here is another beta that attempts to fix the problems discussed in today. The changes include:

• Fix: Added newer formats for Kindles - kfx and awz3
• Fix: Update regex for getting timestamp from My Clippings.txt
• New: Add German name of 'My Clippings.txt' file.
• Fix: Try to compensate for case in month names.

Unfortunately, I can't test these changes very well. I don't have a Kindle, so cannot run all of the code. I'm pretty sure there are no typos or anything to stop them running. Whether it does what it is supposed to do it the question.

Please test and tell me how it goes. If there are problems, I'd like to see a debug log and the "My Clippings.txt" file.

[dsestero]

David,

now the plugin is broken: when I select 'Fetch annotations' I get the following error (exctly as you see it, with those special characters):

calibre, version 3.7.0
ERRORE: Eccezione non gestita: <b>MobiError</b>:Unknown book type: ������

calibre 3.7 embedded-python: True is64bit: True
Linux-4.10.0-33-generic-x86_64-with-debian-stretch-sid Linux ('64bit', 'ELF')
('Linux', '4.10.0-33-generic', '#37-Ubuntu SMP Fri Aug 11 10:55:28 UTC 2017')
Python 2.7.12
Linux: ('debian', 'stretch/sid', '')
Interface language: it
Successfully initialized third party plugins: Count Pages (1, 8, 0) && Wiki Reader (2, 1, 0) && Annotations (1, 6, 5) && EpubMerge (2, 1, 1) && Reading List (1, 6, 6) && Generate Cover (1, 5, 21)
Traceback (most recent call last):
File "calibre_plugins.annotations.action", line 360, in fetch_usb_device_annotations
File "calibre_plugins.annotations.action", line 555, in get_annotated_books_on_usb_device
File "/tmp/calibre_annotations_plugin/Kindle.py", line 151, in get_installed_books
File "/tmp/calibre_annotations_plugin/Kindle.py", line 292, in _get_imported_books
File "/tmp/calibre_annotations_plugin/Kindle.py", line 266, in _get_metadata
File "site-packages/calibre/ebooks/metadata/mobi.py", line 494, in get_metadata
File "site-packages/calibre/ebooks/mobi/reader/headers.py", line 284, in __init__
File "site-packages/calibre/ebooks/mobi/reader/headers.py", line 312, in identity
MobiError: Unknown book type: ������

and then the popup window 'Fetchin annotations from device' remains blocked.

I can assure you that the previous beta works and I have not added any books neither to library nor to the device in the meanwhile.

Attached are the CalibreLog and also an updated italian translation.

For the other points you answered previously you are right: in the previous beta the annotations were in fact merged correctly, and I understand your point of view about matching titles, although I often download from Amazon then start to read and highlight, especially when I don't have my computer at hand.

Thanks, ciao
Dario

[davidfor]

Originally Posted by dsestero

David,

now the plugin is broken: when I select 'Fetch annotations' I get the following error (exctly as you see it, with those special characters):

Of course it doesn't work. Calibre doesn't have any support for getting the metadata from KFX books. Or at least not via the mechanism used. I've attached another beta with a workaround. It's simply catching the exception thrown, logging it and ignoring that book. This a little better than before as it does find the books and reports them, and if they are in a format that calibre supports, they will be handled.

I'm assuming the books that cause this are KFX. Does calibre show these in the device list, do they match with books in the library and is the metadata displayed correctly?

and then the popup window 'Fetchin annotations from device' remains blocked.

There are some missing error handling that leaves dialogs open when there are errors. When I find them, I fix them.

I can assure you that the previous beta works and I have not added any books neither to library nor to the device in the meanwhile.

Attached are the CalibreLog and also an updated italian translation.

Thanks, I've updated the translation in the beta.

For the other points you answered previously you are right: in the previous beta the annotations were in fact merged correctly, and I understand your point of view about matching titles, although I often download from Amazon then start to read and highlight, especially when I don't have my computer at hand.

I tend to do the same thing. Start reading the book that I buy from Kobo and downloaded to the device, and then change to a sideloaded version if I have any problems with the downloaded version.

If what you want is the merged annotations, then I think the following will work:

1. Read an annotate on the downloaded version.
2. Add the book to calibre and fix the metadata.
3. Connect the device.
4. If the book on the device doesn't match with the book in the calibre library, go to the device list and do the matching.
5. Fetch the annotations. The annotations should get put on the book in the calibre library, but you might get prompted to confirm they are going to the correct book.
6. Send the calibre version to the device.
7. Any annotations made on this version should be fetched to the book in the calibre library and merged. I wouldn't expect you to get prompted about this during the fetch.

It's not ideal, but I think that will work. If you look at the My Clippings.txt after doing this, I expect you will find the annotations under the two different titles for the book. I know this works for the Kobo devices, but they do the book matching very differently. If it doesn't work, I'd be interested in seeing the clipping file and log to see what has happened.

[dsestero]

Good job David, now I find more annotated books, although not all yet.

I did several experiments but haven't found a satisfying sense of the things at this time.

Here I try to tell some:

1) I had a book sideloaded from calibre in mobi format and it didn't get fetched. If I convert the book from mobi to azw3 and substitute this new version on the device its annotations get fetched.

Path fetched book after conversion from mobi to azw3:
/media/dario/Kindle/documents/Vari, Autori/viaggio chiamato LibreUmbria, Un - Vari, Autori.azw3
previously was:
/media/dario/Kindle/documents/Vari, Autori/viaggio chiamato LibreUmbria, Un - Vari, Autori.mobi
Calibre title:
Un viaggio chiamato LibreUmbria
Calibre sorted title:
viaggio chiamato LibreUmbria, Un
Calibre Authors:
Vari, Autori
Calibre sorted authors:
Vari, Autori

but now even if I reload the mobi version its annotations get fetched.

2) I repeated the same procedure for another book, it seems to me the same conditions, but cannot get any annotations.

Path not fetched book:
/media/dario/Kindle/documents/Chacon, Scott & Ben, Straub/Pro Git - Chacon, Scott & Straub Ben.mobi
or
/media/dario/Kindle/documents/Chacon, Scott & Ben, Straub/Pro Git - Chacon, Scott & Straub Ben.azw3
Calibre title:
Pro Git (second edition)
Calibre sorted title:
Pro Git
Calibre Authors:
Chacon, Scott & Straub Ben
Calibre sorted authors:
Chacon, Scott & Ben, Straub

MyClippings.txt (fragment):

==========
Un viaggio chiamato LibreUmbria (AA.VV.)
- La tua evidenziazione a pagina 4 | posizione 58-60 | Aggiunto in data martedì 22 settembre 2015 12:16:37

sulla base delle best practice dei progetti di maggiore successo, come quelli del Comune di Bologna e della Città di Monaco di Baviera, dove la comunicazione è stata il filo conduttore della migrazione, e ha permesso di ridurre al minimo la resistenza al cambiamento),
==========
Un viaggio chiamato LibreUmbria (AA.VV.)
- La tua evidenziazione a pagina 6 | posizione 80-84 | Aggiunto in data martedì 22 settembre 2015 12:18:42

comunicazione (rivolta prima ai Dirigenti, che sono stati coinvolti nel processo e ne sono diventati parte attiva, poi agli utenti finali) ed alla specifica e massiccia attività di formazione all'uso di LibreOffice. Il risultato maggiormente significativo è stato quello dell'altissimo gradimento da parte degli utenti finali, che ha peraltro permesso il completamento del processo in tempi contenuti (circa un anno per la migrazione e la formazione di tutto il personale della Provincia di Perugia, 1200 persone).
==========
Un viaggio chiamato LibreUmbria (AA.VV.)
- La tua evidenziazione a pagina 36 | posizione 544-547 | Aggiunto in data mercoledì 23 settembre 2015 09:21:24

Non c'è cosa migliore per il progetto che mettere nel gruppo di coordinamento persone motivate, curiose e magari con diverse professionalità ed attitudini alle spalle (informatica, comunicazione, organizzazione e risorse, etc.) che inizino a pianificare insieme l'itinerario di viaggio, sognando la meta e tutto il tragitto. Un gruppo di persone motivate che presidino con passione l'evoluzione del progetto nell'Ente, apportando in tempo azioni correttive ad eventuali rischi, sono la migliore garanzia per il raggiungimento del risultato.
==========
Un viaggio chiamato LibreUmbria (AA.VV.)
- La tua evidenziazione a pagina 52 | posizione 796-799 | Aggiunto in data mercoledì 23 settembre 2015 16:47:58

“Libertà di studiare come funziona il programma e di modificarlo in modo da adattarlo alle proprie necessità”. La seconda libertà offerta dal Software Libero, libertà fondamentale al fine di migliorare il programma e, soprattutto, la vita quotidiana degli utilizzatori.
==========
Un viaggio chiamato LibreUmbria (AA.VV.)
- La tua evidenziazione a pagina 37 | posizione 553-554 | Aggiunto in data giovedì 31 agosto 2017 17:06:24

L’attuazione ottimale di un piano di progetto prevede di esercitare una costante attenzione al fine di valutare lo stato e le tendenze di sviluppo del progetto.
==========
Pro Git
- La tua evidenziazione a pagina 344 | posizione 4566-4569 | Aggiunto in data venerdì 17 giugno 2016 07:19:32

Since the URL in the .gitmodules file is what other people will first try to clone/fetch from, make sure to use a URL that they can access if possible. For example, if you use a different URL to push to than others would to pull from, use the one that others have access to. You can overwrite this value locally with git config submodule.DbConnector.url PRIVATE_URL for your own use. When applicable, a relative URL can be helpful.
==========
Pro Git
- La tua evidenziazione a pagina 343 | posizione 4554-4556 | Aggiunto in data venerdì 22 settembre 2017 10:31:43

By default, submodules will add the subproject into a directory named the same as the repository, in this case “DbConnector”. You can add a different path at the end of the command if you want it to go elsewhere.
==========
Pro Git (second edition) (Chacon, Scott - Straub, Ben)
- La tua evidenziazione a pagina 2 | posizione 22-23 | Aggiunto in data venerdì 22 settembre 2017 11:41:51

The Open Source community using Git has also exploded.
==========

3) Another experiment was with a kfx book downloaded from Amazon and synchronized with Calibre: no fetched annotations.
Here sideloading the azw3 version allows to fetch the annotations. Removing the azw3 version from the device, converting it to kfx and sideloading the kfx yeld no annotations, even if I made some new ones.

One more thing: I thought Calibre detects book in KFX format since version 2.40 (see: https://calibre-ebook.com/whats-new) but, as you said, maybe it is not the same mechanism used by the plugin. If that would not possible to change it would be sad because on the kindle kfx books looks really improved with respect to appearance, navigation and speed over azw3.

Well, I'll collect more cases using the plugin and will report when I have more to say.

Thank you and nice week-end,
Dario

[davidfor]

Firstly, I had a vague feeling my statement about KFX was wrong, but couldn't see how they were being handled. I have since found out how, and the attached beta should pick them up. Plus, for the other formats, I have changed to another method that should be better over all. And one extra bit of debug output that should tell me what goes wrong.

For the rest, I'm not to surprised by any of that. As KFX books have to be handled differently, they won't have been handled by any previous version of the plugin. For the MOBI/AZW3 swaps, there are things going on with UUIDs in the files that probably explain it.

But, I really need to see the debug log. It has information about what the plugin thinks is on the device. What files it sees and what titles it thinks these files have. And I can map that back to the code to work out what is happening. And the complete clippings file will help. As I don't have a Kindle of any sort, I am hamstrung about testing the code. If there is a problem posting this in the thread, then send it in a PM, or upload it somewhere and PM the link. Or send an email to the address in the code.

The attached beta should get closer to either getting all the annotations or telling me why they aren't retrieved. This has all been work on code in the Kindle part of the plugin. But, until I know all the annotations are read from the file and the plugin has done it's first pass of matching them to books.