[GUI Plugin] Annotations

[Bunker D]

Hi,

Like @readandread* above, I'm trying to import annotations from the NeoReader app of an Onyx Boox to Calibre. My doesn't even provide page numbers. (I'm guessing @readandread wanted to import rather than export annotation.)

I was and am still ready to code a bit if need be, but I'm not sure it is even doable considering that the annotations exported by Boox don't have any indication of location, which means that I guess it must be localized, and there might even be some uncertainty.

And here comes the other issue: I tried to fiddle around, to see what the Annotations could do, what it really does, if I could consider just converting my annotations file into a Kindle annotation files or something… I couldn't have the plugin to do anything.
I found information on what a coder should do to expand its capabilities, but I could not even find how I could import annotations from a file. It seems to me like there is a serious lack of accessible documentation for users.

[davidfor]

Originally Posted by Bunker D

Hi,

Like @readandread* above, I'm trying to import annotations from the NeoReader app of an Onyx Boox to Calibre. My doesn't even provide page numbers. (I'm guessing @readandread wanted to import rather than export annotation.)

I was and am still ready to code a bit if need be, but I'm not sure it is even doable considering that the annotations exported by Boox don't have any indication of location, which means that I guess it must be localized, and there might even be some uncertainty.

The plugin can only import details about the annotations that available to it. If the location is described as a page number in whatever the annotations are being imported from, then that is the best the plugin can do.

And here comes the other issue: I tried to fiddle around, to see what the Annotations could do, what it really does, if I could consider just converting my annotations file into a Kindle annotation files or something… I couldn't have the plugin to do anything.
I found information on what a coder should do to expand its capabilities, but I could not even find how I could import annotations from a file. It seems to me like there is a serious lack of accessible documentation for users.

The very first post of the thread describes how the plugin works. Which includes where it reads how to read the annotations from the supported apps and devices.

And the key word there "supported". The support for each app and device has to be created. If you want to read the annotations from a file then you need to create the code that does this. Once that is implemented and added to the plugin, then it should either automatically find the file (such as the My Clippings.txt for a Kindle) or it could prompt the user to select the file. The that code reads and parses the file into the structure the plugin uses for everything else.

[aik099]

Originally Posted by Bunker D

Hi,

Like @readandread* above, I'm trying to import annotations from the NeoReader app of an Onyx Boox to Calibre. My doesn't even provide page numbers. (I'm guessing @readandread wanted to import rather than export annotation.)

I was and am still ready to code a bit if need be, but I'm not sure it is even doable considering that the annotations exported by Boox don't have any indication of location, which means that I guess it must be localized, and there might even be some uncertainty.

And here comes the other issue: I tried to fiddle around, to see what the Annotations could do, what it really does, if I could consider just converting my annotations file into a Kindle annotation files or something… I couldn't have the plugin to do anything.
I found information on what a coder should do to expand its capabilities, but I could not even find how I could import annotations from a file. It seems to me like there is a serious lack of accessible documentation for users.

Please note, that the recent plugin update doesn't import annotations from the built-in NeoReader app. It does work with the AlReaderX reader app only, which does provide location information (relative location of a bookmarked fragment from the start of the book in percent, because page number is calculated on the fly based on chosen reader settings, like margins, line spacings, font size, etc.).

You can get it from https://4pda.to/forum/index.php?showtopic=945616 page (the page is in Russian (the app itself has an English interface as well), but your web browser can translate it. The line with the download link looks like this:

AlReaderX E-Ink AlReaderXEInk.apk для читалок Onyx, в которых AlReaderX не предустановлен

The download link might change with every release and you have to manually download that APK and put it on the Boox device and open it to install the updates. The app version on Goggle Play is built differently and won't work on e-ink devices. The one I've mentioned is a link for a build for the Boox devices specifically with library integration (shows reading positions in the library, shows recently read books) and background light blue/yellow/mixed changing support.

[icanhardlyread]

I have recently started using a Kobo Libra 2 and, as a consequence, making more use of Calibre and its plugins. So far, along with the extended Kobo driver, the annotations plugin is is one of the most useful I've encountered.

Unfortunately, I'm now running into issues with the location the plugin reports for some books. Where it doesn't work as expected, the plugin saves the name of the section's XML file rather than the chapter title. While I'm still new to Calibre and the inner workings of ebook formats, I poked around in the files a little and I'm getting the distinct impression that the problematic books use different XML tags/arguments to identify chapters.

I originally thought that this must be an issue related to the plugin, but as I'm typing this, I'm beginning to think the issue might already be present in the annotations stored on the device. How would I best troubleshoot this further?

In case it's helpful information, I let the extended driver convert books I send to the device to KEPUB, and it seems like annotations only show a proper location when the original file was an EPUB.

[stbenjam]

I've cobbled together a basic support for Moon+ Reader, would be interested in feedback and for someone else to try it.

https://github.com/davidfor/calibre-annotations/pull/23

[aik099]

Originally Posted by stbenjam

I've cobbled together a basic support for Moon+ Reader, would be interested in feedback and for someone else to try it.

https://github.com/davidfor/calibre-annotations/pull/23

That's interesting. Thank you.

I've done some review (not a Phyton developer myself). There are really some cool features present in your PR, that deserve a separate PR. Also some nice bug fixes in there.

What I'm not sure about is https://github.com/davidfor/calibre-...cde#r851656646 change. @davidfor, can you please explain the use case of the to-be-removed code so that we can address the issue that Moon Reader import has with it?

[stbenjam]

Originally Posted by aik099

That's interesting. Thank you.

I've done some review (not a Phyton developer myself). There are really some cool features present in your PR, that deserve a separate PR. Also some nice bug fixes in there.

What I'm not sure about is https://github.com/davidfor/calibre-...cde#r851656646 change. @davidfor, can you please explain the use case of the to-be-removed code so that we can address the issue that Moon Reader import has with it?

I removed the code you were concerned about, and went a different direction. It looks like most of the other exporting readers take a paste of some data in a text box. For moon reader, we want to import the backup zip file, so I added a file picker option. I broke that and the default colors as per your suggestion to different PR's.

Until those land I'd hold off on reviewing the Moon reader PR, so I can rebase and you'll see a clean diff.

[stbenjam]

@davidfor

I extracted some standalone features from my Moon+ Reader PR that also need review:

- Configurable default highlights, with support for HTML colors
https://github.com/davidfor/calibre-annotations/pull/24

- Optional file picker dialog for ExportingReaders
https://github.com/davidfor/calibre-annotations/pull/25

There may be a few more while I continue working on Moon+ Reader.

[stbenjam]

I also have some questions about the behavior of matching in the plugin.

- If there is no match, the plugin's behavior adds the annotations *by default* to whatever the selected book in the library is. I can get on board with that happening if the user asks, but could this be changed so the red entries (no confidence) aren't checked by default?

Moderator Notice
Oversize image moved to attachment. READ the guidelines on inline sizes.

- If there is a match, but not on UUID, the plugin prompts for each book to confirm the match. For Moon+, I obviously don't have the Calibre UUID's. Could I maybe change the behavior to add filename into the mix? I do have that, and I think if title + author + filename match, confidence of "3" should be OK and the annotations automatically imported. WDYT?

Moderator Notice
Oversize image moved to attachment. READ the guidelines on inline sizes.

[stbenjam]

Is this plugin actively maintained anymore? I saw AIReaderX support was recently merged but I don't see much other progress, and the only feedback on my PR's is from community members.

[davidfor]

Originally Posted by stbenjam

Is this plugin actively maintained anymore? I saw AIReaderX support was recently merged but I don't see much other progress, and the only feedback on my PR's is from community members.

The plugin is maintained. But, it isn't my first priority. I have other things happening, work, home and other things that have meant I haven't gotten to look at this. Hopefully it will be this weekend, but, I can't promise that.

And for the record, you comment in Github is pretty much how I have felt about this plugin for a long time. No-one really cares, and it has basically been on life support for years. Now suddenly someone wants something and I am expected to jump-to-it and ignore everything else.

[tlwright]

Having an error message, and Koyal pointed me here. I pasted below my original post. It is reproducible but only started in 5.4.0. The plugin is updated to 1.17.

This occurred on 5.40 & 5.41. It happens when I plug in my android tablet to transfer books but doesn't affect the actual transferring of books. Tablet is an Asus Zenbook 7" on Android 7 connecting to Windows 10. The laptop is an HP Envy 17, with an i7 processor from 2016. It's only occurred on 5.4 and 5.41, and didn't happen on earlier versions.

Here's the error info:

calibre, version 5.41.0
ERROR: Unhandled exception: <b>ThreadingViolation</b>:You cannot use the MTP driver from a thread other than the thread in which startup() was called

calibre 5.41 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 48, 0) && DeDRM (10, 0, 2) && Package KFX (from KFX Input) (1, 48, 0) && Action Chains (1, 16, 3) && Amazon.com Multiple Countries (1, 0, 0) && Annotations (1, 17, 0) && Audit Log (1, 0, 16) && Barnes & Noble (1, 2, 16) && Calibre Power Search Plugin (2, 1, 2) && CalibreCV (0, 0, 6) && CalibreSpy (1, 0, 84) && Category Tags (0, 2, 3) && Check Books (0, 1, 0) && Consolidate All Library Metadata (2, 0, 40) && Count Pages (1, 11, 2) && Drop Search Results (1, 0, 11) && Embed Comic Metadata (1, 6, 1) && EpubCheck (0, 2, 4) && EpubMerge (2, 13, 0) && EpubSplit (3, 3, 0) && Extract ISBN (1, 5, 0) && FanFicFare (4, 11, 0) && Fantastic Fiction (1, 5, 1) && Fantastic Fiction Adults (1, 2, 0) && Find Duplicates (1, 9, 4) && Generate Cover (2, 1, 0) && Goodreads (1, 5, 3) && Goodreads More Tags (1, 2, 1) && Goodreads Sync (1, 15, 1) && Goodreads_Rating (3, 1, 0) && ISFDB3 (1, 0, 2) && Import List (1, 8, 2) && Job Spy (1, 0, 190) && KFX metadata reader (from KFX Input) (1, 48, 0) && KFX Input (1, 48, 0) && Set KFX metadata (from KFX Output) (1, 61, 0) && KFX Output (1, 61, 0) && KOReader Sync (0, 2, 7) && KePub Input (3, 4, 3) && KePub Metadata Reader (3, 4, 3) && KePub Metadata Writer (3, 4, 3) && KePub Output (3, 5, 3) && Kindle Collections (1, 7, 29) && Kindle hi-res covers (0, 5, 0) && Kobo Books (1, 9, 0) && Kobo Utilities (2, 15, 1) && Last Modified (0, 8, 2) && Library Codes (1, 0, 50) && LibraryThing Match (0, 2, 0) && Manage Series (1, 3, 0) && Modify ePub (1, 7, 0) && MultiColumnSearch (1, 0, 87) && Obok DeDRM (10, 0, 0) && Overdrive Link (2, 51, 0) && Quality Check (1, 12, 0) && Reading List (1, 8, 0) && Save Composite Custom Columns (1, 1, 1) && Save Virtual Libraries To Column GUI (2, 3, 0) && Skoob Books (1, 5, 4) && SmartEject (2, 4, 0) && Smashwords Metadata (1, 0, 2) && Standard Ebooks (1, 0, 0) && User Category (1, 5, 5) && WordDumb (3, 19, 0) && Overdrive Link Metadata Source (2, 51, 0)
Traceback (most recent call last):
File "calibre_plugins.annotations.action", line 1043, in on_device_connection_changed
File "calibre_plugins.annotations.action", line 1261, in rebuild_menus
File "calibre_plugins.annotations.action", line 368, in get_connected_device_primary_name
File "calibre\devices\mtp\driver.py", line 196, in get_device_information
File "calibre\devices\mtp\windows\driver.py", line 34, in check_thread
calibre.devices.mtp.windows.driver.ThreadingViolat ion: You cannot use the MTP driver from a thread other than the thread in which startup() was called

[davidfor]

Originally Posted by tlwright

Having an error message, and Koyal pointed me here. I pasted below my original post. It is reproducible but only started in 5.4.0. The plugin is updated to 1.17.

This occurred on 5.40 & 5.41. It happens when I plug in my android tablet to transfer books but doesn't affect the actual transferring of books. Tablet is an Asus Zenbook 7" on Android 7 connecting to Windows 10. The laptop is an HP Envy 17, with an i7 processor from 2016. It's only occurred on 5.4 and 5.41, and didn't happen on earlier versions.

Here's the error info:

calibre, version 5.41.0
ERROR: Unhandled exception: <b>ThreadingViolation</b>:You cannot use the MTP driver from a thread other than the thread in which startup() was called

calibre 5.41 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 48, 0) && DeDRM (10, 0, 2) && Package KFX (from KFX Input) (1, 48, 0) && Action Chains (1, 16, 3) && Amazon.com Multiple Countries (1, 0, 0) && Annotations (1, 17, 0) && Audit Log (1, 0, 16) && Barnes & Noble (1, 2, 16) && Calibre Power Search Plugin (2, 1, 2) && CalibreCV (0, 0, 6) && CalibreSpy (1, 0, 84) && Category Tags (0, 2, 3) && Check Books (0, 1, 0) && Consolidate All Library Metadata (2, 0, 40) && Count Pages (1, 11, 2) && Drop Search Results (1, 0, 11) && Embed Comic Metadata (1, 6, 1) && EpubCheck (0, 2, 4) && EpubMerge (2, 13, 0) && EpubSplit (3, 3, 0) && Extract ISBN (1, 5, 0) && FanFicFare (4, 11, 0) && Fantastic Fiction (1, 5, 1) && Fantastic Fiction Adults (1, 2, 0) && Find Duplicates (1, 9, 4) && Generate Cover (2, 1, 0) && Goodreads (1, 5, 3) && Goodreads More Tags (1, 2, 1) && Goodreads Sync (1, 15, 1) && Goodreads_Rating (3, 1, 0) && ISFDB3 (1, 0, 2) && Import List (1, 8, 2) && Job Spy (1, 0, 190) && KFX metadata reader (from KFX Input) (1, 48, 0) && KFX Input (1, 48, 0) && Set KFX metadata (from KFX Output) (1, 61, 0) && KFX Output (1, 61, 0) && KOReader Sync (0, 2, 7) && KePub Input (3, 4, 3) && KePub Metadata Reader (3, 4, 3) && KePub Metadata Writer (3, 4, 3) && KePub Output (3, 5, 3) && Kindle Collections (1, 7, 29) && Kindle hi-res covers (0, 5, 0) && Kobo Books (1, 9, 0) && Kobo Utilities (2, 15, 1) && Last Modified (0, 8, 2) && Library Codes (1, 0, 50) && LibraryThing Match (0, 2, 0) && Manage Series (1, 3, 0) && Modify ePub (1, 7, 0) && MultiColumnSearch (1, 0, 87) && Obok DeDRM (10, 0, 0) && Overdrive Link (2, 51, 0) && Quality Check (1, 12, 0) && Reading List (1, 8, 0) && Save Composite Custom Columns (1, 1, 1) && Save Virtual Libraries To Column GUI (2, 3, 0) && Skoob Books (1, 5, 4) && SmartEject (2, 4, 0) && Smashwords Metadata (1, 0, 2) && Standard Ebooks (1, 0, 0) && User Category (1, 5, 5) && WordDumb (3, 19, 0) && Overdrive Link Metadata Source (2, 51, 0)
Traceback (most recent call last):
File "calibre_plugins.annotations.action", line 1043, in on_device_connection_changed
File "calibre_plugins.annotations.action", line 1261, in rebuild_menus
File "calibre_plugins.annotations.action", line 368, in get_connected_device_primary_name
File "calibre\devices\mtp\driver.py", line 196, in get_device_information
File "calibre\devices\mtp\windows\driver.py", line 34, in check_thread
calibre.devices.mtp.windows.driver.ThreadingViolat ion: You cannot use the MTP driver from a thread other than the thread in which startup() was called

OK, I can see the problem. I had my phone as an ignored device, so calibre wasn't going through the code that is failing for you. The attached beta fixes this. But, the error is in code that was added to support the Boox devices. I do not have one of these devices, so I do not have a way to test if it still works for these devices. And I do not have a way to test under Linux or Mac.


@aik099: Could you test this on your device? The problem is in the code in action.py to work out if the connected device is one of the device supported by the BooxReaderApp additions. In the method "get_connected_device_primary_name", there is the following line:

device_name = self.connected_device.get_device_information()[0]

I believe this should be changed to:

device_name = self.connected_device.current_friendly_name

If you look at the code for "get_device_information" in two versions of driver.py for the MTP devices, that is what is being used there.

I have tested this here with my Android phone, and it does not fail. But, it isn't a device that will is expected to match the test. And for "friendly name", it is getting "David's S20 FE". That was set when I setup the phone, but, if the Boox devices can be configured in the same way, it could be a problem.

[aik099]

Originally Posted by davidfor

OK, I can see the problem. I had my phone as an ignored device, so calibre wasn't going through the code that is failing for you. The attached beta fixes this. But, the error is in code that was added to support the Boox devices. I do not have one of these devices, so I do not have a way to test if it still works for these devices. And I do not have a way to test under Linux or Mac.


@aik099: Could you test this on your device? The problem is in the code in action.py to work out if the connected device is one of the device supported by the BooxReaderApp additions. In the method "get_connected_device_primary_name", there is the following line:

device_name = self.connected_device.get_device_information()[0]

I believe this should be changed to:

device_name = self.connected_device.current_friendly_name

If you look at the code for "get_device_information" in two versions of driver.py for the MTP devices, that is what is being used there.

I have tested this here with my Android phone, and it does not fail. But, it isn't a device that will is expected to match the test. And for "friendly name", it is getting "David's S20 FE". That was set when I setup the phone, but, if the Boox devices can be configured in the same way, it could be a problem.

@davidfor, it works perfectly. Actually, on the Boox device both versions of a code (old and new one) return the same "Nova3" string.

[tlwright]

I've tested the beta version and no error message on my tablet, and my phone (a Pixel). Thank you so much!