[GUI Plugin] Annotations

[Atcold]

Hum, having some issues here. I've just clicked on “Fetch annotations […]” and this pops up.

calibre, version 5.9.0
ERROR: Unhandled exception: <b>TypeError</b>:'&lt;' not supported between instances of 'NoneType' and 'NoneType'

calibre 5.9  embedded-python: True is64bit: True
macOS-10.16-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '20.2.0', 'Darwin Kernel Version 20.2.0: Wed Dec  2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64')
Python 3.8.5
OSX: ('10.16', ('', '', ''), 'x86_64')
Interface language: en_GB
Successfully initialized third party plugins: Annotations (1, 14, 0)
Traceback (most recent call last):
  File "calibre_plugins.annotations.annotated_books", line 168, in sort
    self.arraydata = sorted(self.arraydata, key=operator.itemgetter(Ncol), reverse=(order == Qt.DescendingOrder))
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'

How can I debug this?

[Atcold]

I've installed the 1.14.4 and now I get the window “No annotated books found on device” popup telling me that it was “Unable to find any annotations on Amazon Kindle matching books in your library.”.

Before I was getting a partial list, despite the error.
Now I'm getting nothing.

[Atcold]

So, I figured that the bug is in annotated_books.py at the else below:

def sort(self, Ncol, order=Qt.AscendingOrder):
    """
    Sort table by given column number.
    """
    self.layoutAboutToBeChanged.emit()
    if Ncol == self.ENABLED_COL: # Don't sort on the checkbox column.
        self.arraydata = sorted(self.arraydata, key=lambda row: row[Ncol].checkState(), reverse=(order == Qt.DescendingOrder))
    else:
        self.arraydata = sorted(self.arraydata, key=operator.itemgetter(Ncol), reverse=(order == Qt.DescendingOrder))
    self.layoutChanged.emit()

I printed out self.arraydata and I get

DEBUG:   18.9 Trying to sort this:  [['b99a2e76-53bf-4100-948b-3c41f42a2c03', '2', 'COMPUTERS / Programming / General', <PyQt5.QtWidgets.QCheckBox object at 0x12f5a2ee0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x13090ea60>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x13090eaf0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x13090eb80>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x13090e9d0>, 5, 5], ['86c19abd-5cb6-41c5-8da5-d641c4b6b8a0', '1', 'History, Ancient, Greece, Philosophy, Political, History & Surveys, Ancient & Classical', <PyQt5.QtWidgets.QCheckBox object at 0x13090ec10>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x13090ed30>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x13090ee50>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x13090eee0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x13090eca0>, 1, 5], [None, '-8', '', <PyQt5.QtWidgets.QCheckBox object at 0x13090ef70>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x1309120d0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912160>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x1309121f0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912040>, 7, 0], [None, '-9', '', <PyQt5.QtWidgets.QCheckBox object at 0x130912280>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x1309123a0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912430>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x1309124c0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912310>, 13, 0], [None, '-21', '', <PyQt5.QtWidgets.QCheckBox object at 0x130912550>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912670>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912700>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912790>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x1309125e0>, 21, 0], [None, '-22', '', <PyQt5.QtWidgets.QCheckBox object at 0x130912820>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x1309129d0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912a60>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912af0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912940>, 3, 0], [None, '-26', '', <PyQt5.QtWidgets.QCheckBox object at 0x130912c10>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912ca0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912d30>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912dc0>, <calibre_plugins.annotations.annotated_books.SortableTableWidgetItem object at 0x130912b80>, 49, 0]]

Not sure how to move forward.

[davidfor]

@Atcold: I'm not sure how that can happen with the recent betas. The first element in each of the lists should be None. And, there should be no way for the dialog to be sorted by that column. Could you send me a copy of you "annotations.json" file? This is in your calibre configuration directory in the plugins sub-directory and contains the options you have used for the plugin.

But, the attached beta should fix this. I have tested it with my Kobo devices. There is differences in how the other devices are handled, and this problem has come up because of KFX format books (and other changes I made).

Edit: Now with the actual attachment!

[Atcold]

With this 1.14.6 I get “No annotated books found on device” again.
Also, when I was bypassing the sorting, not all books / annotations were fetched (and specifically the last book I've been reading).

Thanks for your support and apologies for the double post (it went in a moderation queue for its length, but I thought the forum glitched instead).

[davidfor]

Originally Posted by Atcold

With this 1.14.6 I get “No annotated books found on device” again.
Also, when I was bypassing the sorting, not all books / annotations were fetched (and specifically the last book I've been reading).

Thanks for your support and apologies for the double post (it went in a moderation queue for its length, but I thought the forum glitched instead).

There isn't anything obviously wrong with those files. Could you run calibre in debug mode and send me the log. I'll have to look through it to see what is happening.

[Atcold]

Originally Posted by davidfor

There isn't anything obviously wrong with those files. Could you run calibre in debug mode and send me the log. I'll have to look through it to see what is happening.

Here you have the log using the latest version of your plugin.

[Noirtier]

Originally Posted by davidfor

OK. For the OPF file, I meant from the calibre library. Click the "Open path" in the details panel for the book and it will show the directory with the book in it. There should be an OPF file. Calibre uses this as part of its backup.

I'll have to retest this. It seemed to work adding the annotation, but, maybe I missed something.

Ah thank you, I see now. I will send you a new PM with the Calibre folders for each of the examples I sent you previously, so that you have a copy of the metadata.opf files.

P.S. I think I have figured out how to get email notifications in this forum now so hopefully will be a bit quicker to reply next time

[davidfor]

Originally Posted by Atcold

Here you have the log using the latest version of your plugin.

Sorry, I forget to ask people to turn the debug option in the plugin. It is at the top of the plugin's configuration dialog. Not having that turned means there is very little information in the debug log.

[Atcold]

Originally Posted by Noirtier

P.S. I think I have figured out how to get email notifications in this forum now so hopefully will be a bit quicker to reply next time

How did you do that? I've just sent a help request to the moderators of this forum (setting the Instant notification for subscribed threads does not work for me).

Originally Posted by davidfor

Sorry, I forget to ask people to turn the debug option in the plugin. It is at the top of the plugin's configuration dialog. Not having that turned means there is very little information in the debug log.

No worries. Yeah, in fact I was a little dubious about the log utility upon inspection. Hehe

Could you also explain to me what your plug-in does? I cannot manually locate any bookmark file in /Volumes/Kindle/documents/<author name>.
Also, I see Atomic Habits in the resolve_path section (line 118) but not after List of storage devices (line 136). So, it does not pop up in the resolved_path_map (line 232).

[kch]

I am using the Annotations plug-in to extract notes and highlights from a synchronised KePub on my Kobo Libra H2O.
The annotations fetched are in a random order which is very hard to follow (see screenshot).
It is neither in the chapter order or chronological order.

Some notes seem to be either missing in location or indexed to more than one chapter. For example, the first note in the screenshot is under both ‘Introduction’ and Chapter 2. I am not sure whether there is a highlight missing under ‘Introduction’ or if the annotation from Chapter 2 below just got randomly anchored to also ‘Introduction’.

Is there a way to extract annotations in a more logical order, such as following the chapter order or chronological, added order?

Thanks a lot in advance.

[davidfor]

Originally Posted by Atcold

No worries. Yeah, in fact I was a little dubious about the log utility upon inspection. Hehe

This time it gave me the hint. There was an error reading the My Clippings.txt file so that it didn't find any. This was because I reverted an earlier change when I realised it broke pre-5.0 support and I missed something. The attached beta should fix this. And, in case it doesn't, logs the error a bit better. Can you send me a debug log even if it does work?

Could you also explain to me what your plug-in does? I cannot manually locate any bookmark file in /Volumes/Kindle/documents/<author name>.

The "bookmark" files are a pseudo file. For Kindles, the plugin uses a function in calibre to get a list of books. It uses this as the extension but also returns the known formats. Then it looks for files that match the file name but with the book extension.

Also, I see Atomic Habits in the resolve_path section (line 118) but not after List of storage devices (line 136). So, it does not pop up in the resolved_path_map (line 232).

That's interesting. Calibre seems to think there are two books: "Atomic Habits: Tiny Changes, Remarkable Results _ An Easy & Proay to Build Good Habits & Break Bad Ones" and "Atomic Habits: Tiny Changes, Remarkable Results" (though the punctuation in titles might not be correct and there might be some shortening). But, when the plugin checks, it cannot find them. In the library, the first one has epub and azw3, the second has mobi. When the plugin searches for these books, it doesn't find them with these names and extensions and hence drops them. The plugin also does a scan of the device using the extensions and doesn't find them that way as well.

Is the book actually on the device? If so, what is the full path for it? Does calibre show it as on the device in the library?

[davidfor]

Originally Posted by kch

I am using the Annotations plug-in to extract notes and highlights from a synchronised KePub on my Kobo Libra H2O.
The annotations fetched are in a random order which is very hard to follow (see screenshot).
It is neither in the chapter order or chronological order.

Some notes seem to be either missing in location or indexed to more than one chapter. For example, the first note in the screenshot is under both ‘Introduction’ and Chapter 2. I am not sure whether there is a highlight missing under ‘Introduction’ or if the annotation from Chapter 2 below just got randomly anchored to also ‘Introduction’.

Is there a way to extract annotations in a more logical order, such as following the chapter order or chronological, added order?

It is a bug. The order is messed wrong under a couple of circumstance and the could be some missing annotations. But, I don't think there will be an actual highlights or notes missing for the ones that are there. The first "Introduction" looks like what you would see for a dog-ear bookmark.

The attached beta should fix this. There was also a case where there would be no chapter name. This happened if the annotation was in an internal file that did not have a ToC entry. These were also part of the cause for annotations being out of order.

[Atcold]

Originally Posted by davidfor

[…] I reverted an earlier change when I realised it broke pre-5.0 support and I missed something.

Are we talking about Calibre's version?

Can you send me a debug log even if it does work?

tmpy3teis29.txt.zip

That's interesting. Calibre seems to think there are two books: "Atomic Habits: Tiny Changes, Remarkable Results _ An Easy & Proay to Build Good Habits & Break Bad Ones" and "Atomic Habits: Tiny Changes, Remarkable Results" (though the punctuation in titles might not be correct and there might be some shortening).

Yes, indeed there are two books. There's an ugly MOBI that was found and then I got an EPUB via my library. Converting the EPUB to MOBI was giving ugly results, therefore I turned it into a pretty AZW3.
I've been loading these, and intermediate results, a few times to check the results.

When the plugin searches for these books, it doesn't find them with these names and extensions and hence drops them. The plugin also does a scan of the device using the extensions and doesn't find them that way as well.

Is the book actually on the device? If so, what is the full path for it? Does calibre show it as on the device in the library?

Yes, the book is on the device (I'm currently reading it).
Full path: /Volumes/Kindle/documents/Clear, James/Atomic Habits_ Tiny Changes, Remarkable Results _ An Easy & Prn Way to Build Good Habits & Break Bad Ones - James Clear.azw3
Yes, Calibre has a green check on it.

Here you have the tree for Clear, James: Clear_James-tree.txt.zip.

[davidfor]

Originally Posted by Atcold

Are we talking about Calibre's version?

Yes, calibre version 5 uses Python 3. There are differences in the language and the plugins needed to be updated to handle this. A lot is backwardly compatible with Python 2 but some things need to be handled for each version. At the moment, I am maintaining compatibility with older calibre versions of calibre. I broke the compatibility with a change, and then broke everything fixing that.

Attachment 184971



Yes, indeed there are two books. There's an ugly MOBI that was found and then I got an EPUB via my library. Converting the EPUB to MOBI was giving ugly results, therefore I turned it into a pretty AZW3.
I've been loading these, and intermediate results, a few times to check the results.

That looks better. I think that there are still books missingin the results, but, the ones that are retrieved should have the full annotations.

Yes, the book is on the device (I'm currently reading it).
Full path: /Volumes/Kindle/documents/Clear, James/Atomic Habits_ Tiny Changes, Remarkable Results _ An Easy & Prn Way to Build Good Habits & Break Bad Ones - James Clear.azw3
Yes, Calibre has a green check on it.

Here you have the tree for Clear, James: Attachment 184973.

The problem is that the plugin is asking for the file name from calibre. Calibre is generating what it thinks should be the name based on the parameters passed. This gives:

/<storage>/Clear, James/Atomic Habits_ Tiny Changes, Remarkable Results _ An Easy & Proay to Build Good Habits & Break Bad Ones - James Clear.bookmark

The plugin then converts this to an actual file name and hence looks for:

/Volumes/Kindle/documents/Clear, James/Atomic Habits_ Tiny Changes, Remarkable Results _ An Easy & Proay to Build Good Habits & Break Bad Ones - James Clear.azw3

From your listing, the actual file name is:

/Volumes/Kindle/documents/Clear, James/Atomic Habits_ Tiny Changes, Remarkable Results _ An Easy & Prn Way to Build Good Habits & Break Bad Ones - James Clear.azw3

All those together look like:

/<storage>/Clear, James/Atomic Habits_ Tiny Changes, Remarkable Results _ An Easy & Proay to Build Good Habits & Break Bad Ones - James Clear.bookmark
/Volumes/Kindle/documents/Clear, James/Atomic Habits_ Tiny Changes, Remarkable Results _ An Easy & Proay to Build Good Habits & Break Bad Ones - James Clear.azw3
/Volumes/Kindle/documents/Clear, James/Atomic Habits_ Tiny Changes, Remarkable Results _ An Easy & Prn Way to Build Good Habits & Break Bad Ones - James Clear.azw3

Those do not match ("Proay" vs "Prn"). The difference is in how calibre generates the file path. It has a maximum length and shortens the it somehow. That seen in how the word "Proven" is shortened to "Proay" in one and "Prn" in the other.

And the scan for books doesn't find it as it is not recursive. So, it is only searching "/Volumes/Kindle/documents". And that is something I hadn't noticed before.

This is all in that was written by the original developer and I never quite understood why it was done this way. I have written the code for the Kobo and tolino device and have handled getting the list of books in a very different way. For the Kobo, no reference is actually needed to the metadata in the books. For the tolino, it is much closer to the way the Kindle works and the metadata has to be retrieved from the books on the device.

So, in the attached beta I have change the way to get the list of books. This is based on the methods mentioned above. It should be accurate as it uses a calibre search to get the books on the device that are in the library and then uses other calibre functions to get the path to the book. This should work better than the existing method. Unfortunately, I have no way to test it. I don't have a Kindle or a good way to simulate it. The code should be good, but, there might be something. Please run it and send me the debug lot to look at.