[GUI Plugin] Annotations

[norbusan]

Hi everyone,

I am running calibre 2.45 with the latest version of this plugin, and using a Kobo eink reader. Unfortunately, nothing works:
- pressing the annotation button in the menu
- no entry in the conext menu
- no way to get annotations

On the command line I see errors:

Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/ui.py", line 227, in initialize
    ac.do_genesis()
  File "/usr/lib/calibre/calibre/gui2/actions/__init__.py", line 143, in do_genesis
    self.genesis()
  File "calibre_plugins.annotations.action", line 365, in genesis
  File "/usr/lib/calibre/calibre/devices/idevice/libimobiledevice.py", line 205, in __init__
    self.load_library()
  File "/usr/lib/calibre/calibre/devices/idevice/libimobiledevice.py", line 443, in load_library
    self.lib = cdll.LoadLibrary('libimobiledevice.so.5')
  File "/usr/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libimobiledevice.so.5: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/ui.py", line 404, in initialize
    ac.initialization_complete()
  File "calibre_plugins.annotations.action", line 729, in initialization_complete
  File "calibre_plugins.annotations.action", line 1073, in rebuild_menus
AttributeError: 'AnnotationsAction' object has no attribute 'menu'

Any hints?

Thanks

Norbert

[NullNix]

Originally Posted by norbusan

Hi everyone,

I am running calibre 2.45 with the latest version of this plugin, and using a Kobo eink reader. Unfortunately, nothing works:
- pressing the annotation button in the menu
- no entry in the conext menu
- no way to get annotations

On the command line I see errors:

Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/ui.py", line 227, in initialize
    ac.do_genesis()
  File "/usr/lib/calibre/calibre/gui2/actions/__init__.py", line 143, in do_genesis
    self.genesis()
  File "calibre_plugins.annotations.action", line 365, in genesis
  File "/usr/lib/calibre/calibre/devices/idevice/libimobiledevice.py", line 205, in __init__
    self.load_library()
  File "/usr/lib/calibre/calibre/devices/idevice/libimobiledevice.py", line 443, in load_library
    self.lib = cdll.LoadLibrary('libimobiledevice.so.5')
  File "/usr/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libimobiledevice.so.5: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/ui.py", line 404, in initialize
    ac.initialization_complete()
  File "calibre_plugins.annotations.action", line 729, in initialization_complete
  File "calibre_plugins.annotations.action", line 1073, in rebuild_menus
AttributeError: 'AnnotationsAction' object has no attribute 'menu'

Any hints?

Thanks

Norbert

I can confirm this. Your problem is the first error: AnnotationsAction.genesis() tries to instantiate a libiMobileDevice right before initializing self.menu, so when the exception is thrown you wind up with, well, no menu.

It is definitely incorrect to assume that this class can always be instantiated: it depends on shared libraries that may well not be present.

Fix that works for me attached. (As a zipped diff because mobileread doesn't accept diff attachments: why, mobileread, why?!)

[norbusan]

Thanks, works for me, too. I checked it just now.

All the best and thanks for the quick fix

Norbert

[davidfor]

Originally Posted by NullNix

I can confirm this. Your problem is the first error: AnnotationsAction.genesis() tries to instantiate a libiMobileDevice right before initializing self.menu, so when the exception is thrown you wind up with, well, no menu.

It is definitely incorrect to assume that this class can always be instantiated: it depends on shared libraries that may well not be present.

Fix that works for me attached. (As a zipped diff because mobileread doesn't accept diff attachments: why, mobileread, why?!)

You're probably right, but the library in question should be part of the calibre install. From the line generating the error, this looks to be a Linux install. I can see libimobiledevice.so.5 in both the 32 and 64 bit tarballs.

Has this only started with 2.45, or was it already happening?

In any case, I'll include the change and post an update in a little while.

[kovidgoyal]

@davidfor: That is because the OP is using a distro package not the official calibre binary package.

[davidfor]

Originally Posted by bopuc

(For context, I ask because I have built a system for myself* which pulls directly from calibre's SQL database.)

You're extracting the highlighted text and posting them as part of your reviews?

"there until next time I fetch annotations" is not a reliable, permanent database. By the looks of it (I have opened the "annotations.db" database in a MySQL viewer), there is a table named "Kindle_Amazon_Kindle_annotations" and it appears to actually hold (mysteriously enough) most if not all of my Kindle annotations ever (I have been meticulous about keeping my My_Clippings.txt around.)

I want to know if the plugin is acting strangely or if the code, by design, keeps and maintains that full record (which would be great!).

The database holds the annotations from the last fetch. The tables are actually dropped and recreated when the fetch is done. This mean that if you don't delete annotations from the device, they will all be there. For me, the annotations are temporary. I mark things I want to fix or check later. Once I do that, I delete them.

The plugin merges whatever is fetched with whatever is in the annotations column. This will work for annotations from different devices. If you are moving the My_Clippings.txt between devices, you should be OK. But, if you started using a Kobo device, it should also be OK, but you wouldn't have the annotation on the device.

Now, my limitation is that Python is not a programming language I know and so I do not know what your code is doing.

Python is not a bad language. Though I'm still not used to the idea of not having any delimiters around code blocks.

In my case I do not need an export function (because I already have one, one that needs to parse the HTML-encoded annotations through an XML parser), I would simply love it if the plugin took a copy of the data structure of what gets HTML-ified and stuck into the calibre database, JSON-ify it as well and stick that into an adjacent field (in the calibre database). No one needs to see it in the UI necessarily, but having access to it in the DB would be sweet. Well, for me anyways… cough… heh

If you pushed the current revision up to your GitHub repo for the project, I might clone it and try my hand at learning some Python (finally!).

I have just updated the repository. It wasn't missing much, just the changes in the last release.

Ideally, if your Annotations plugin can be *the* trusted Annotations importer, formatter and manager for calibre, I think that provides a huge amount of value if some better way of then re-consuming annotations, as I have built for myself*, also becomes available.

I'd like to think it can be trusted. But, there isn't an easy way to get the annotations elsewhere. Because of my usage, I haven't been that interested in doing it. There is code in the plugin to parse the HTML version of the annotation. Using that to produce another format, shouldn't be that hard. Any contributions to the code will be welcome.

[davidfor]

Originally Posted by kovidgoyal

@davidfor: That is because the OP is using a distro package not the official calibre binary package.

Of course. Though, I wasn't expecting any of the distros to be that up to date with their calibre releases.

In any case, checking the load won't harm. I just need to check it doesn't break anything later. Well, no iOS device will work, but they won't anyway.

[eschwartz]

Well, my distro is up to date (ArchLinux).

You can always tell if it is a distro calibre, because the traceback will reference source files from "/usr/lib/calibre/calibre/"

Or in my case, "/home/eschwartz/git/calibre/src/calibre/"

I've been ignoring that error for a long time now, since I don't use iDevices and most of them are probably running iOS>=8.3 and have issues with Marvin (#1 user of libimobiledevice in calibre I believe) it seemed pointless to care just for linux users who compiled calibre from source (<=distros).

Never realized that that was causing this plugin to do weird things. Anyway, Kovid committed a fix so calibre will try to fallback on the unversioned soname: https://github.com/kovidgoyal/calibr...bd338e16c3fe59

So the upcoming calibre 2.46 will no longer run into problems when a linux distro updates libimobiledevice and the calibre binary package doesn't.
I can't promise distros will package the new version.

[davidfor]

This is a quick update to remove the error seen with a missing libimobiledevice under linux. This should only happen if using the distro version of calibre. In these circumstances, the plugin will not work with iOS devices, but all other function will work.

As usual, the update will be available through calibre in the next hour or so. Or you can download it from the first post. And if you see any problems, please report them here

[norbusan]

Originally Posted by davidfor

You're probably right, but the library in question should be part of the calibre install. From the line generating the error, this looks to be a Linux install. I can see libimobiledevice.so.5 in both the 32 and 64 bit tarballs.

Has this only started with 2.45, or was it already happening?

In any case, I'll include the change and post an update in a little while.

As mentioned, this is the Debian package where libimobiledevice is available, but not the .5 version but .4.

Thanks

Norbert

[watersailor]

...i still have some questions about your great plugin. Thanks a lot for your work!

1.Annotations Plugin
...do i still have to change my Kindle UI from German to English so that i can use the plugin? Why? I mean it just uses the "My Clippings.txt" File from my Kindle. So why do i have to change the Speech of my Kindle?

2.How can i import an additional "old" "My Clippings.txt"
...first i used my Kindle Paperwhite with German UI. And i read a book and i made some Comments. Before i changed the speech of my kindle to english and before i installed "Annotations" i safed this old "My Clippings.txt" to my PC. Then i changed the UI etc.!!! Now i testet your plugin and it worked fine. But how can i import also the old "My Clippings.txt" (its from another book i read earlier) additionaly into calibre?

3.Where are this informations stored?
...is it possible that the comments or "highlight's" are safed to each book sepparately? Or are they also "after the Annotations" Import functions still stored just in the "My Clippings.txt"???

4.Now i see my "highlight's" just under "Book Details"
...after i "Fetch annotations from Amazon Kindle" i can read this highlights only when i open the window "Book Details" (i am not sure if this is the correct point in english because i use a german calibre version).
But why cant i see my highlight's also if i "Open the Book" (i mean in the read mode)???

5.Highlights in "Book Details" without page nr.
...unfortunately i cant see the page number (the original book page nr.) of my hightlights. I think it is importand on which page of the book i made a highlight. And in My Clippings i can see this information of the original page nr.! So why cant you use this information in your Plugin and Calibre?

6.Another Question is about "Add Note's"
...for my notes i can just see the number e.g. 141 (its not the page etc.) and the date and time, but not the text passage i wrote this note to! Is it possible to also see the marked text passage where i made my note on?

7.Is it possible to synchronize Notes and / or Highlights?
...can i read today on my kindle, make highlights! Then tommorrow after i connected my kindle to calibre and after i imported my highlights with annotations, i read the next passage in the calibre reader (and how can i also use a textmarker function there). And on the third day i want to read in a android mobile ereader app and also there i want to make highlight's. Is this or a part of it possible. For me it would be great.

Please Help & Thanks a Lot

[davidfor]

Originally Posted by watersailor

...i still have some questions about your great plugin. Thanks a lot for your work!

1.Annotations Plugin
...do i still have to change my Kindle UI from German to English so that i can use the plugin? Why? I mean it just uses the "My Clippings.txt" File from my Kindle. So why do i have to change the Speech of my Kindle?

It is explained in the first post:

The Kindle UI must be set to English for the plugin to be able to retrieve locations and timestamps from your 'My Clippings.txt' file.

Basically, the plugin is looking for strings and is looking for the English version of them. But, that might not be correct. The Kindle code was written by someone else and I haven't looked at it much. But, there appears to be support for other languages. It has English, German, French, Italian, Portuguese, Japanese and Chinese.

2.How can i import an additional "old" "My Clippings.txt"
...first i used my Kindle Paperwhite with German UI. And i read a book and i made some Comments. Before i changed the speech of my kindle to english and before i installed "Annotations" i safed this old "My Clippings.txt" to my PC. Then i changed the UI etc.!!! Now i testet your plugin and it worked fine. But how can i import also the old "My Clippings.txt" (its from another book i read earlier) additionaly into calibre?

My first reaction is "Haven't a clue". As I said above, I didn't write that part of the plugin and I don't have a Kindle. But the "My Clippings.txt" is a text file. You can probably merge them into one. Each clipping should have some labels attached to it. These are what the plugin is looking for. You would need to change the labels to use only the one language.

3.Where are this informations stored?
...is it possible that the comments or "highlight's" are safed to each book sepparately? Or are they also "after the Annotations" Import functions still stored just in the "My Clippings.txt"???

4.Now i see my "highlight's" just under "Book Details"
...after i "Fetch annotations from Amazon Kindle" i can read this highlights only when i open the window "Book Details" (i am not sure if this is the correct point in english because i use a german calibre version).
But why cant i see my highlight's also if i "Open the Book" (i mean in the read mode)???

The annotations are stored per book, but the column used can be set. But default they are appended to whatever is in the "comments" column. Look at the first post and the help for details.

5.Highlights in "Book Details" without page nr.
...unfortunately i cant see the page number (the original book page nr.) of my hightlights. I think it is importand on which page of the book i made a highlight. And in My Clippings i can see this information of the original page nr.! So why cant you use this information in your Plugin and Calibre?

Sorry, I don't really know why the page number wasn't extracted. Part of it is probably that the other devices and apps don't have the page numbers.

6.Another Question is about "Add Note's"
...for my notes i can just see the number e.g. 141 (its not the page etc.) and the date and time, but not the text passage i wrote this note to! Is it possible to also see the marked text passage where i made my note on?

If you selected text when you made the annotation, that should be displayed. If the annotation was added to a page, then only some location information will be displayed.

7.Is it possible to synchronize Notes and / or Highlights?
...can i read today on my kindle, make highlights! Then tommorrow after i connected my kindle to calibre and after i imported my highlights with annotations, i read the next passage in the calibre reader (and how can i also use a textmarker function there). And on the third day i want to read in a android mobile ereader app and also there i want to make highlight's. Is this or a part of it possible. For me it would be great.

The list of applications and devices supported by the plugin is in the first post. If you fetch the annotations from one of these, they are merged with the existing annotations in the calibre library. The calibre viewer is not supported. It could be, but no-one has written the code to do this.

[PapaJohn]

Just wondering if your plugin can be modified to provide read status.
I had been using a template that would update (via Calibre's built-in Fetch Annotations) a Read_Status custom column (with either Read or Reading) and add the date and time and % Read to the comments section in Calibre.
This no longer works for me.
Would your plugin be capable of doing the same?
Hope this makes sense.

I am using Calibre 2.49 on Mac OS 10.10.5
Kindle DX and Kindle Paperwhite.

[davidfor]

I saw you other posts and wondered about it as I hadn't realised that the built-on fetch annotations for Kindles set a status column. But, from this I think you are saying your status column is uses a template to parse the comments and find the percent read and the date.

Adding the reading status to the annotations is possible, but I don't really think it fits with the plugin. The plugin is about getting the annotations, and the last reading position isn't really one. You could say it is a bookmark, which are fetched (at least for the Kobo devices), but they are not deleted if they are removed or changed. One of the functions of the annotations plugin is to merge the annotations from multiple devices over time. To do this, it doesn't remove any annotations it has already fetched. Hence, if the reading position was treated as a bookmark, you would either get a succession of them showing your progress over time, or they would have to be treated in a different way to be updated or moved each time.

Another problem is actually making changes. While I maintain the basic plugin, the device specific code is different. I do not have a Kindle, so cannot make big changes as I cannot test them. If there is someone who wants to make the changes, I'll be happy to help and include them.

Also, I don't believe there have been any changes in calibre that affect the experimental Fetch Annotations. Because of this, I think it is more likely a change in the devices that has broken the function. Has the firmware been updated on them recently?

[PapaJohn]

DX has not had any updates for a few years.
Paper White might have been updated....I'm not sure. It is my wife's