[GUI Plugin] Annotations

[zeekles]

Soooo... has anyone come up with a solution for exporting notes and annotations from Marvin and importing them into Calibre yet?? (pleeeeeeease??)

I have found that I can export a file from my Marvin app on my iPad and send the export via email - it produces an *.mrvi file with most of the details in it. Is there any way to use this to import into Calibre?

I put some dummy / test data in to Marvin then exported - to summarise my test data:

For the book '01 Unicorn Lane - Curran POV Collection' by author 'Gordon Andrews': I added a Note, but this doesn't seem to be included in the export - I am happy to chase this up with the developer as I put a lot of my information in to the 'Notes' field, but I can work around this if nothing can be done (hopefully they fix this though!). I also added a highlight, with annotation. Highlighted text: 'I was in Unicorn Lane at night.' , Annotation: 'Testing highlights and annotations
Test test' . I also added 'supernatural' to my vocabulary list.

See below for the entry from the *.mrvi file:


<?xml version='1.0' encoding='utf-8'?><marvin scope='library' version='1.0' timestamp='1454503949.909970'><book hash='7116d1b3028d7f1eeff97dec2837f000' bookid='636' title='01 Unicorn Lane - Curran POV Collection' author='Gordon Andrews' uuid='8cb8b983-4e32-4d9c-855c-d77a098d05f9' sections='2' authorSort='Andrews, Gordon' calibreTitleSort='01 Unicorn Lane - Curran POV Collection' rating='0' filesize='68579' filename='8cb8b983-4e32-4d9c-855c-d77a098d05f9.epub' dtadded='1416044699.152995' ><subjects></subjects><bookmarks></bookmarks><highlights><highlight section='2' startX='/x:html[1]/x:body[1]/x:div[1]/x:div[1]/x[3]/text()[1]' endX='/x:html[1]/x:body[1]/x:div[1]/x:div[1]/x[3]/text()[1]' parentX='/x:html[1]/x:body[1]/x:div[1]/x:div[1]/x[3]' startOffset='1' endOffset='32' color='4' dateTime='1454503766.925537' key='EEC1F5CF-6AB6-4D4F-83AF-9F24F8266E94' note='Testing highlights and annotations
Test test' text='I was in Unicorn Lane at night.' deleted='0'/></highlights><vocabulary><entry dateTime='1454503934.795547'>supernatural</entry></vocabulary></book></marvin>

[davidfor]

The plugin does support importing annotations from files that have come from somewhere else. This done for Goodreader. The first post has an explanation of how to do this and it matches fairly closely to your suggestion. It just needs someone to write the import function. For Marvin, it might be easy. The plugin has Marvin support, but it was disabled when the the Marvin plugin was released with annotations support. I'd have to look, but the format of the file is probably related to the Marvin database. If so, that should make things a lot easier.

I'll put it on my todo list to look at, but it won't be to soon. I need to clean up some other work I have around. But, if someone wants to investigate, I'll be happy to answer any questions.

[zeekles]

Originally Posted by davidfor

The plugin does support importing annotations from files that have come from somewhere else. This done for Goodreader. The first post has an explanation of how to do this and it matches fairly closely to your suggestion. It just needs someone to write the import function. For Marvin, it might be easy. The plugin has Marvin support, but it was disabled when the the Marvin plugin was released with annotations support. I'd have to look, but the format of the file is probably related to the Marvin database. If so, that should make things a lot easier.

I'll put it on my todo list to look at, but it won't be to soon. I need to clean up some other work I have around. But, if someone wants to investigate, I'll be happy to answer any questions.

Just the fact that you're willing to investigate is good enough for me, thank you.

Is there a way I can find out what the format of the goodReader file is? maybe I can bastardise my Marvin file in the meantime to make it compatible...

[davidfor]

Originally Posted by zeekles

Is there a way I can find out what the format of the goodReader file is? maybe I can bastardise my Marvin file in the meantime to make it compatible...

The Goodreader annotations appears to be a report generated and emailed from the app. You could reverse-engineer it from the code to read the file, but it would probably be easier to install Goodreader on you iOS device, add a book, make some annotations and follow the instruction in the first post to see it.

The Marvin file you posted is XML. That should make it easier to write the code to import it.

[NoOneHere]

Kobo annotation timestamps seem to be stored as UTC, it would be nice if on import they could be converted to local time.

Note: The formatting option is already %c

[Spyral]

Hi! First of all thanks for all your time and work invested in developing and supporting this plugin! I was having trouble making it work and after a while I found out why it wasn't working. I looked at the plugin's code and found that inside "ParseKindleMyClippingsTxt.py" under "_LANG_AND_KIND_DETECT_BY_START_WORDS = {" there is:
"Mi subrayado": ('es', 'highlight'),
"Mi nota": ('es', 'note'),
"Mi marcador": ('es', 'bookmark'),
and should be, at least for this device (I noticed this looking at "My Clippings.txt"):
"Tu subrayado": ('es', 'highlight'),
"Tu nota": ('es', 'note'),
"Tu marcador": ('es', 'bookmark'),
and got that working: the plugin detected notes, highlights and bookmarks.

Then I encountered another problem, it could not detect the position correctly.
So I kept looking at the "ParseKindleMyClippingsTxt.py" file, and found that
under "_LOCATION_REGEX = {" was:
'es': (r"\sPosición\s*%s",),
and should be (notice the lowercase):
'es': (r"\sposición\s*%s",),
And now its working perfectly! I hope this helps others with the same problem

PS: I was thinking that it could be nice if we could select to import only the highlights, or notes or bookmars. The kindle has a problem that highlights a word when you make a note, and this plugin imports the note and highlight as separated things (which is not a plugin's problem I know), and that highlight has no use for me.

[biojack]

First of all I want to say thanks for this great plugin. I take notes when I read non-fiction on my kobo and this allows me to export them to an annotation column in Calibre. As part of my workflow I want to me able to export my highlights and notes to Evernote/Google Docs. I can't seem to figure out how to keep the formatting. I've slightly modified the CSS and it's great. I can clearly see my personal notes in italic but when I copy/paste to other programs I can't figure out what is a highlight and what is a personal notes. I'm thinking I could add a keyword or symbol when I write a note but perhaps there is a way to keep the formating. I've tried control-v or control-shift-v, paste with formating, no luck.

OK.... I just had the idea of trying Microsoft Explorer instead of Chrome and it works better... Google Docs looks good, evernote weird... but in case someone else is wondering I'll keep this here. If anyone else has a great workflow let me know. Cheers

[bernard.ryefield]

With only two books selected the plugin/Calibre keeps churning for ages and becomes unresponsive. The approximate operation time was twenty minutes before I killed the task (before completion).

[davidfor]

Originally Posted by bernard.ryefield

With only two books selected the plugin/Calibre keeps churning for ages and becomes unresponsive. The approximate operation time was twenty minutes before I killed the task (before completion).

The plugin isn't fetching annotations just for the selected book. It is looking for all annotations on the device. It looks for all annotations and then matches the books they are on against those in the current calibre library. The more books you have on the device with annotations, and the more books ing the library, the longer it takes. How many do you have?

[davidfor]

Originally Posted by biojack

First of all I want to say thanks for this great plugin. I take notes when I read non-fiction on my kobo and this allows me to export them to an annotation column in Calibre. As part of my workflow I want to me able to export my highlights and notes to Evernote/Google Docs. I can't seem to figure out how to keep the formatting. I've slightly modified the CSS and it's great. I can clearly see my personal notes in italic but when I copy/paste to other programs I can't figure out what is a highlight and what is a personal notes. I'm thinking I could add a keyword or symbol when I write a note but perhaps there is a way to keep the formating. I've tried control-v or control-shift-v, paste with formating, no luck.

OK.... I just had the idea of trying Microsoft Explorer instead of Chrome and it works better... Google Docs looks good, evernote weird... but in case someone else is wondering I'll keep this here. If anyone else has a great workflow let me know. Cheers

The problem here is that copy-and-paste depends on both the source of the copy, and the paste destination. When you copy to the clipboard, the source application can put multiple formats into the clipboard. When you paste from the clipboard, the destination application queries the clipboard for the available formats and picks the most suitable. So, if you copy from a browser and paste into a text editor, the text of the page is usually what it picks. If you post to a word processor, the formatted version is usually used. Some applications will let you pick which version to paste.

Depending on the application you are pasting to, you might be better to copy the source from the metatada editor. That will put the actual source into the clipboard. Then, you can paste that. For this to be useful, it needs the destination application to be able to handle this.

[davidfor]

Originally Posted by NoOneHere

Kobo annotation timestamps seem to be stored as UTC, it would be nice if on import they could be converted to local time.

Note: The formatting option is already %c

The Kobo devices store the timestamp as UTC. The plugin doesn't change this. I'll look at fixing it.

Question for the Kindle users: How do they store the timestamp?

[davidfor]

Originally Posted by Spyral

Hi! First of all thanks for all your time and work invested in developing and supporting this plugin! I was having trouble making it work and after a while I found out why it wasn't working. I looked at the plugin's code and found that inside "ParseKindleMyClippingsTxt.py" under "_LANG_AND_KIND_DETECT_BY_START_WORDS = {" there is:
"Mi subrayado": ('es', 'highlight'),
"Mi nota": ('es', 'note'),
"Mi marcador": ('es', 'bookmark'),
and should be, at least for this device (I noticed this looking at "My Clippings.txt"):
"Tu subrayado": ('es', 'highlight'),
"Tu nota": ('es', 'note'),
"Tu marcador": ('es', 'bookmark'),
and got that working: the plugin detected notes, highlights and bookmarks.

Then I encountered another problem, it could not detect the position correctly.
So I kept looking at the "ParseKindleMyClippingsTxt.py" file, and found that
under "_LOCATION_REGEX = {" was:
'es': (r"\sPosición\s*%s",),
and should be (notice the lowercase):
'es': (r"\sposición\s*%s",),
And now its working perfectly! I hope this helps others with the same problem

Could this be a national difference? Or maybe a firmware version difference? In any case, the code can support both versions. Adding extra entries in both places should solve the problem and not break it for others.

PS: I was thinking that it could be nice if we could select to import only the highlights, or notes or bookmars. The kindle has a problem that highlights a word when you make a note, and this plugin imports the note and highlight as separated things (which is not a plugin's problem I know), and that highlight has no use for me.

I'll have a look at this. The structure of the code doesn't really support it, but it probably should be possible to do.

[bernard.ryefield]

Originally Posted by davidfor

The plugin isn't fetching annotations just for the selected book. It is looking for all annotations on the device. It looks for all annotations and then matches the books they are on against those in the current calibre library. The more books you have on the device with annotations, and the more books ing the library, the longer it takes. How many do you have?

I have 7200 books, kepubs and pdfs.
If I don't select any book there's this pop-up window "Select a book to receive annotations", do you confirm it's ok?

[davidfor]

Originally Posted by bernard.ryefield

I have 7200 books, kepubs and pdfs.

Is that 7200 books on the device and in the calibre library?

If I don't select any book there's this pop-up window "Select a book to receive annotations", do you confirm it's ok?

That is expected. There are some rules about matching the fetched annotations to the books. Any that can't be matched are put into the selected book. This is a general rule and hence a single book has to be selected. But for the Kobo devices, the matching always works, so there are no "extra" annotations.

[bernard.ryefield]

Originally Posted by davidfor

Is that 7200 books on the device and in the calibre library?


That is expected. There are some rules about matching the fetched annotations to the books. Any that can't be matched are put into the selected book. This is a general rule and hence a single book has to be selected. But for the Kobo devices, the matching always works, so there are no "extra" annotations.

7200 on the device, more in the Calibre library