[GUI Plugin] Annotations

[Katja_hbg]

hi @davidfor,
in your help with my issue fetching annotation using kobo-utility, you mentioned this plugin. So I gave it a try. I did not finally decide as there are pros and cons on both.
The KU has still too many empty rows while this one has layout instead plain text.
With KU I have to do it one by one while this fetches in one go.
Lets see where I end with.

This text is to ask you to add something in the very first text. I searched whether and how to suppress the timestamp and found the answer in blog #48 page 4.
There is a section in blog #1 about modifying the timestamp. Perhaps you like to mention this hidden function ... To suppress write display=none in the timeout css field

[loFreaQ]

Hello!

I'm trying to use your plugin with my Paperwhite 1, but I have two problems that neither the latest beta resolves.

When I select the option to retrieve annotations from my kindle, the list of books installed on my Kindle appears correctly, but when I click the button to fetch the annotations, i get this error:

calibre, version 3.29.0
Errore recuperando le annotazioni: <p>Non sono in grado di recuperare annotazioni da Amazon Kindle.</p>

Traceback (most recent call last):
  File "calibre_plugins.annotations.action", line 301, in fetch_device_annotations
  File "calibre_plugins.annotations.action", line 1027, in process_selected_books
  File "calibre_plugins.annotations.action", line 135, in add_annotations_to_calibre
  File "calibre_plugins.annotations.annotations_db", line 180, in annotations_to_html
  File "calibre_plugins.annotations.annotations", line 170, in to_HTML
UnboundLocalError: local variable 'text_style' referenced before assignment

Also, if I try to modify the aspect by clicking the proper button in the plugin settings window, I get this error:

calibre, version 3.29.0
ERRORE: Eccezione non gestita: <b>KeyError</b>:u'Testo'

calibre 3.29 Portable embedded-python: True is64bit: False
Windows-10-10.0.17134-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.17134')
Python 2.7.12+
Windows: ('10', '10.0.17134', 'SP0', u'Multiprocessor Free')
Interface language: it
Successfully initialized third party plugins: DeDRM (6, 1, 0) && Kindle Collections (1, 7, 20) && DOC Input (1, 0, 1) && Quality Check (1, 9, 11) && Annotations (1, 8, 2) && Generate Cover (1, 5, 21) && Save To Format (0, 0, 6)
Traceback (most recent call last):
  File "calibre_plugins.annotations.config", line 325, in configure_appearance
  File "calibre_plugins.annotations.appearance", line 496, in __init__
  File "calibre_plugins.annotations.appearance", line 293, in initialize
  File "calibre_plugins.annotations.appearance", line 346, in populate_table
  File "calibre_plugins.annotations.appearance", line 353, in populate_table_row
KeyError: u'Testo'

Anything I can do to help you debug this?
Windows 10, Calibre 3.29 portable 64 bit.

Thanks!

[bopuc]

Originally Posted by davidfor

@GoNico: I don't consider the issue with annotation sharing or retrieving a issue with copyrights. It is more an issue of standards. To the best of my knowledge, there is no real standard for how annotations are marked in ebooks or stored for sharing. Each ereading device and application basically does its own thing. Some have ways to export to things like EverNote, for others, it is completely contained.

Unfortunately, it doesn't look like something the big companies are interested in fixing, so I can't see anything is likely to change in the near future.

Not that this is the place to debate such things but just to point out (as someone who has experience with standards bodies, most recently the W3C's new Digital Publishing Working Group)… you are half way to understanding how this is fundamentally a Copyright issue.

Copyright law disallows copying of "too much text" (and a bunch of other stuff). Any move to make it easy for the (human) reader to manipulate too much text—nevermind to share or somehow republish!—takes the developer into dangerous waters. Bad enough for individual companies, from Amazon down, totally a no-go for any type of online, hosted services / platform, let alone a standardized, interoperable one. The legal, political will is not there, and not enough mass-market readers want or need annotation features, so… no one bothers. They do the bare minimum they can get away with, and sometimes we're lucky and they leave a raw text file lying around for plugins like this one to do something with them.

In other words Copyright law has a chilling effect on technology development and marketing for consumer manipulation of content. This has severe repercussions on human cultures, but that's another topic.

In any case, I am SO SO grateful to you do, @davidfor, for maintaining and advancing Annotations. Thank you.

[danhof]

Originally Posted by davidfor

Attached is a beta that adds support for the tolino devices. The annotations are read from the "notes.txt" file in the root partition of the device. This is similar to the clippings file in Kindle devices.

[...]

If anyone with a tolino can test this and tell me how it goes, it would be much appreciated.

Hello David,

Thank you so much for providing this plugin.

I have checked it with my Tolino Vision 4 HD and it works fine. However, I assume that there are different locations where Notes are stored on the device. There is the file notes.txt where notes are included as soon as they are created. But if you change or delete these notes later, these changes are not reflected in this file. Thererefore, the 'actual' notes must be saved at a different location.

Do you happen to know where these data are stored and do you see any chance to fetch these "original" notes as well?

Warm regards,
Daniel

[davidfor]

Originally Posted by danhof

Hello David,

Thank you so much for providing this plugin.

I have checked it with my Tolino Vision 4 HD and it works fine. However, I assume that there are different locations where Notes are stored on the device. There is the file notes.txt where notes are included as soon as they are created. But if you change or delete these notes later, these changes are not reflected in this file. Thererefore, the 'actual' notes must be saved at a different location.

Do you happen to know where these data are stored and do you see any chance to fetch these "original" notes as well?

All that is correct. The "notes.txt" file is effectively a log of annotations. It isn't used when you look at the annotations while reading on the device. Those are somewhere else. I can't see any likely files when the device is connected, so I assume they are on a partition that isn't made available to the PC. That makes them inaccessible and impossible to use.

There are instructions for rooting the tolino devices. I assume that will make more of the filesystem available. Apart from a little curiosity about what is hidden, and how various pieces of information are stored, I'm not very interested in doing this.

At this point, this is the only way to get any sort of annotations from the device. It's better than nothing, but I'm not sure how much.

[TGSET]

On my Kobo I store fetched annotations in comments and want to change that behavior. Under customize the "custom column wizard" asked me if I want to move the existing annotations to the new column. Fabulous! but the task threw an error (which of course I didn't copy). The column was created but no notes appeared. I've tried several times since with various column names but it never asks if I want existing notes moved, nor has it had further errors.

It appears I can create and likely populate a new column with new notes but I'm unsure it will move existing notes as well since it no longer asks permission. I fear completing the new column add and having notes in two places. It appears to work but I'd like to be absolutely sure ALL notes will be in only one place. Any guidance or suggestions?

BTW, much thanks to all Calibre developers - it's the best FOSS out there!

[davidfor]

Originally Posted by TGSET

On my Kobo I store fetched annotations in comments and want to change that behavior. Under customize the "custom column wizard" asked me if I want to move the existing annotations to the new column. Fabulous! but the task threw an error (which of course I didn't copy).

Had you restarted calibre after creating the new column? That's the simplest reason for an error. Another is if the annotations in the comments field had been broken in some form. If it happens again, seeing the details will help.

The column was created but no notes appeared. I've tried several times since with various column names but it never asks if I want existing notes moved, nor has it had further errors.

It appears I can create and likely populate a new column with new notes but I'm unsure it will move existing notes as well since it no longer asks permission. I fear completing the new column add and having notes in two places. It appears to work but I'd like to be absolutely sure ALL notes will be in only one place. Any guidance or suggestions?

From memory, if there are no annotations in the column, then it doesn't offer to move them. What you can do is change the annotations back to the comment column. That should work without any actual data changes. Then you can change it to the new column. And hopefully it will work without error. If there is an error, grab it and post it.

But, before doing that, do a backup of your calibre library. Simply copying the library directory will be enough. You can also, take a copy of the database. This is the file "metadata.db" inside your library. Take a copy of this, do the experiment and if it fails, restore it immediately.

[TGSET]

Well, that worked (Thank you) but I think I lost all my notes the first time through. I have a new column but all that it contains are bookmarks. I suppose I lost them somewhere in prior attempts. The comment field of my books that contained notes end with a separator, just FYI.

Backuos: Lesson learned (once again).

[ilovejedd]

Originally Posted by davidfor

Yes, I will consider adding that. I would need some sample files to look at. The simplest is probably the CSV, but the marvinz would be good to look at.

Having said that, the Marvin XD plugin will probably be revived. From it's thread, the changes in Marvin should mean external applications can reach the data files.

I saw mention of this plugin in the Understanding Metadata thread.

Given Apple closed access to iOS devices and I believe Marvin XD is effectively defunct, I wonder if you might revisit considering adding Marvin 3 to the plugin's list of supported apps?

The csv export is as follows:

ID,Type,Colour,Title,Author,Date,Created,Updated,Starred,Latitude,Longitude,Address,City,State,Country,HasPhoto,PhotoDate,PhotoLatitude,PhotoLongitude,EntryText,HighlightText
"0051DBCF-6143-4349-9FC7-ED18719666D6","2","0","A Song of Ice and Fire: A Game of Thrones, A Clash of Kings, A Storm of Swords, A Feast for Crows, and A Dance with Dragons","George R.R. Martin","2017-12-28T17:34:44Z","2017-12-28T17:34:44Z","2017-12-28T17:34:44Z","0","","","","","","","0","","","","","An oak may live three hundred years, a redwood tree three thousand. A weirwood will live forever if left undisturbed. To them seasons pass in the flutter of a moth’s wing, and past, present, and future are one. Nor will your sight be limited to your godswood. The singers carved eyes into their heart trees to awaken them, and those are the first eyes a new greenseer learns to use*…*but in time you will see well beyond the trees themselves."

The marvinz export is pretty much just a zip file containing a bunch of json files.

Each annotation is stored in a separate json file named <uuid>.entry.marvinj and a list of all these files is in manifest.json.

Sample manifest.json:

{
  "DeviceVersion" : "10.3.3",
  "Payload" : [
    {
      "filename" : "0051DBCF-6143-4349-9FC7-ED18719666D6.entry.marvinj"
    }
  ],
  "FileType" : "Marvin interchange file format",
  "ContentType" : "marvin\/annotations",
  "DateCreated" : 539657301.130649,
  "HardwareName" : "iPad6,11",
  "DeviceName" : "iPad",
  "MarvinBuild" : "34",
  "PayloadVersion" : 1,
  "MarvinVersion" : "3.1.2"
}

Sample entry.marvinj:
0051DBCF-6143-4349-9FC7-ED18719666D6.entry.marvinj

{
  "DeviceVersion" : "10.3.3",
  "Payload" : {
    "address" : null,
    "selectionSectionName" : "A Dance with Dragons, 34. Bran III",
    "country" : null,
    "photoBase64" : null,
    "tags" : [

    ],
    "dateUpdated" : 536175284.265789,
    "photoDate" : null,
    "uuid" : "0051DBCF-6143-4349-9FC7-ED18719666D6",
    "city" : null,
    "latitude" : null,
    "entryText" : "",
    "dateCreated" : 536175284.265787,
    "state" : null,
    "type" : 2,
    "photoLatitude" : null,
    "selectionSectionIndex" : 362,
    "starred" : false,
    "entryDate" : 536175284.265789,
    "longitude" : null,
    "selectionStart" : 27294,
    "photoLongitude" : null,
    "ID" : 459,
    "selectionEnd" : 27736,
    "selectionText" : "An oak may live three hundred years, a redwood tree three thousand. A weirwood will live forever if left undisturbed. To them seasons pass in the flutter of a moth’s wing, and past, present, and future are one. Nor will your sight be limited to your godswood. The singers carved eyes into their heart trees to awaken them, and those are the first eyes a new greenseer learns to use*…*but in time you will see well beyond the trees themselves.",
    "bookTitle" : "A Song of Ice and Fire: A Game of Thrones, A Clash of Kings, A Storm of Swords, A Feast for Crows, and A Dance with Dragons",
    "hasPhoto" : false,
    "colour" : 0,
    "bookAuthor" : "George R.R. Martin",
    "bookFileHash" : "a06fcbc831509ca9f5d1762585b28edb"
  },
  "FileType" : "Marvin interchange file format",
  "ContentType" : "marvin\/journalentry",
  "DateCreated" : 539657299.963053,
  "HardwareName" : "iPad6,11",
  "DeviceName" : "iPad",
  "MarvinBuild" : "34",
  "PayloadVersion" : 1,
  "MarvinVersion" : "3.1.2"
}

From my observations:

• selectionSectionName - corresponds to the chapter title
• selectionSectionIndex - index number of file based on epub's content.opf manifest?
• selectionStart & selectionEnd - word count within the chapter/section's html file only?
• ID - I believe Marvin stores all annotations in a single database and this ID corresponds to its index number in Marvin's database
• bookFileHash - MD5 hash of the epub file

I've got a crap ton of highlights for this particular book (probably too much for fair use sharing) so I didn't attach the actual files but if you want, I can make annotation export files for a book in the public domain for your review.

Thanks!

[davidfor]

Originally Posted by ilovejedd

I saw mention of this plugin in the Understanding Metadata thread.

I knew I shouldn't have said that Both today or back then.

Given Apple closed access to iOS devices and I believe Marvin XD is effectively defunct, I wonder if you might revisit considering adding Marvin 3 to the plugin's list of supported apps?

The csv export is as follows:

ID,Type,Colour,Title,Author,Date,Created,Updated,Starred,Latitude,Longitude,Address,City,State,Country,HasPhoto,PhotoDate,PhotoLatitude,PhotoLongitude,EntryText,HighlightText
"0051DBCF-6143-4349-9FC7-ED18719666D6","2","0","A Song of Ice and Fire: A Game of Thrones, A Clash of Kings, A Storm of Swords, A Feast for Crows, and A Dance with Dragons","George R.R. Martin","2017-12-28T17:34:44Z","2017-12-28T17:34:44Z","2017-12-28T17:34:44Z","0","","","","","","","0","","","","","An oak may live three hundred years, a redwood tree three thousand. A weirwood will live forever if left undisturbed. To them seasons pass in the flutter of a moth’s wing, and past, present, and future are one. Nor will your sight be limited to your godswood. The singers carved eyes into their heart trees to awaken them, and those are the first eyes a new greenseer learns to use*…*but in time you will see well beyond the trees themselves."

The marvinz export is pretty much just a zip file containing a bunch of json files.

Each annotation is stored in a separate json file named <uuid>.entry.marvinj and a list of all these files is in manifest.json.

Sample manifest.json:

{
  "DeviceVersion" : "10.3.3",
  "Payload" : [
    {
      "filename" : "0051DBCF-6143-4349-9FC7-ED18719666D6.entry.marvinj"
    }
  ],
  "FileType" : "Marvin interchange file format",
  "ContentType" : "marvin\/annotations",
  "DateCreated" : 539657301.130649,
  "HardwareName" : "iPad6,11",
  "DeviceName" : "iPad",
  "MarvinBuild" : "34",
  "PayloadVersion" : 1,
  "MarvinVersion" : "3.1.2"
}

Sample entry.marvinj:
0051DBCF-6143-4349-9FC7-ED18719666D6.entry.marvinj

{
  "DeviceVersion" : "10.3.3",
  "Payload" : {
    "address" : null,
    "selectionSectionName" : "A Dance with Dragons, 34. Bran III",
    "country" : null,
    "photoBase64" : null,
    "tags" : [

    ],
    "dateUpdated" : 536175284.265789,
    "photoDate" : null,
    "uuid" : "0051DBCF-6143-4349-9FC7-ED18719666D6",
    "city" : null,
    "latitude" : null,
    "entryText" : "",
    "dateCreated" : 536175284.265787,
    "state" : null,
    "type" : 2,
    "photoLatitude" : null,
    "selectionSectionIndex" : 362,
    "starred" : false,
    "entryDate" : 536175284.265789,
    "longitude" : null,
    "selectionStart" : 27294,
    "photoLongitude" : null,
    "ID" : 459,
    "selectionEnd" : 27736,
    "selectionText" : "An oak may live three hundred years, a redwood tree three thousand. A weirwood will live forever if left undisturbed. To them seasons pass in the flutter of a moth’s wing, and past, present, and future are one. Nor will your sight be limited to your godswood. The singers carved eyes into their heart trees to awaken them, and those are the first eyes a new greenseer learns to use*…*but in time you will see well beyond the trees themselves.",
    "bookTitle" : "A Song of Ice and Fire: A Game of Thrones, A Clash of Kings, A Storm of Swords, A Feast for Crows, and A Dance with Dragons",
    "hasPhoto" : false,
    "colour" : 0,
    "bookAuthor" : "George R.R. Martin",
    "bookFileHash" : "a06fcbc831509ca9f5d1762585b28edb"
  },
  "FileType" : "Marvin interchange file format",
  "ContentType" : "marvin\/journalentry",
  "DateCreated" : 539657299.963053,
  "HardwareName" : "iPad6,11",
  "DeviceName" : "iPad",
  "MarvinBuild" : "34",
  "PayloadVersion" : 1,
  "MarvinVersion" : "3.1.2"
}

From my observations:

• selectionSectionName - corresponds to the chapter title
• selectionSectionIndex - index number of file based on epub's content.opf manifest?
• selectionStart & selectionEnd - word count within the chapter/section's html file only?
• ID - I believe Marvin stores all annotations in a single database and this ID corresponds to its index number in Marvin's database
• bookFileHash - MD5 hash of the epub file

I've got a crap ton of highlights for this particular book (probably too much for fair use sharing) so I didn't attach the actual files but if you want, I can make annotation export files for a book in the public domain for your review.

That doesn't look to bad. I'll have a look at it, but, I won't promise when or if I will do anything. But, if someone is interested in doing it, I will be happy to answer any questions and add it the plugin.

[dunhill]

Will this plugin work with the moon + readers application?
Thank you.

[davidfor]

Originally Posted by dunhill

Will this plugin work with the moon + readers application?

The list of supported devices and apps are in the first post of the thread, so no. But, it probably can. It depends on exactly how the annotations are stored and whether this can be seen by the plugin. For example, the support for the Goodreader app is done by emailing the annotations from the app and pasting the content of the email into the plugin. But, if there is a visible database or something, that could be used instead. The other thing is that someone needs to write the code to add the support. This is something I do for apps or devices I have, but, for others, I hope someone who is interested will do it. I can help, but I have little interest in doing the work for all the apps out there.

[GregLeBlonde]

Hi,

I recently upgraded devices. I was wondering if the plugin can send the backed up annotations from Calibre to my new device so they can be displayed there as well as on the PC GUI.

I apologize if this has been asked before; a cursory look at the thread did not reveal any answers.

Happy New Year!

[davidfor]

Originally Posted by GregLeBlonde

I recently upgraded devices. I was wondering if the plugin can send the backed up annotations from Calibre to my new device so they can be displayed there as well as on the PC GUI.

The short answer is no. And I'm sure that it has been mentioned many times in this thread. Though maybe I'm used to saying it elsewhere.

And for the record, the plugin does not back up the annotations. It is taking an independent copy that could be merged from multiple devices of different types.

[GregLeBlonde]

Originally Posted by davidfor

The short answer is no. And I'm sure that it has been mentioned many times in this thread. Though maybe I'm used to saying it elsewhere.

And for the record, the plugin does not back up the annotations. It is taking an independent copy that could be merged from multiple devices of different types.

Thanks for your response. On a 30+ page thread that covers many iterations of the plugin it is tough to find of a question has been asked previously and know whether that answer is still true. If it is a common question, maybe it should be added to the FAQ on the plugin page; that was the first place I looked.