[GUI Plugin] Annotations

[squizzie]

Hi,

My Kindle screen has broken so I'm hoping to transfer all of my annotations before I look into alternative reader applications (this time on my phone, most likely). I'm having some trouble importing the annotations using the Annotations plugin, though. I initially got an error message that comments wasn't set up, but this self resolved after I opened the Annotation settings and closed it again (without making any changes). It then seemed like it was going to work and showed me the books it was going to import from, but failed to successfully bring the data across, returning instead a new error message "Error fetching annotations from Amazon Kindle", with the details:

"calibre, version 3.38.1
Error fetching annotations: <p>Unable to fetch annotations from Amazon Kindle.</p>

Traceback (most recent call last):
File "calibre_plugins.annotations.action", line 324, in fetch_device_annotations
File "calibre_plugins.annotations.action", line 1083, in process_selected_books
File "calibre_plugins.annotations.action", line 146, in add_annotations_to_calibre
File "calibre_plugins.annotations.annotations_db", line 180, in annotations_to_html
File "calibre_plugins.annotations.annotations", line 235, in to_HTML
AttributeError: 'NoneType' object has no attribute 'contents'"

I've been able to find this error on a handful of posts with the same message on Google although only one would redirect successfully and appeared to indicate that this error should have been resolved with the (1.14.0) update prior to the version of the plugin that I have (1.15.0), so I'm at a bit of a loss of what I can do to resolve.

It might be the case that my version of Calibre is not fully compatible with the version of the plugin that I have; if that's the case, is it possible to download a previous version that should work (and if so which one and where)? I'm not keen to update Calibre itself as there is another plugin that I use that no longer gets updates and will stop working if I do update Calibre.

Thanks!!

[davidfor]

@squizzie: That looks like a calibre version compatibility issue. I will have a look and see if I can solve this. But, it would be better if you upgrade calibre. If you are stuck on the 3.x versions of calibre, then I recommend installing 3.48. That is the last 3.x version and should be the best version to use. It is also the version I will do my compatibility testing with.

[davidfor]

I have just added the following to the first post of this thread.

Supported calibre Versions

The intention is for the plugin to support all versions of calibre. But, the effort put in to support older versions will be minimal. Any changes in the plugin will be developed and tested with the most recent released version of calibre. They will also be tested with the last dot release of older calibre versions. But, this testing may be just validating that nothing has been obviously broken. At the point, this means that the plugin is fully supported in calibre version 5. Releases will be checked against version 4.23.0 and 3.48.0.

If problems are found with older versions, please report them and I will see what I can do. But, upgrading to one of the above versions will be the first suggestion.

The post from @squizzie reminded me about it. I had already stated this for some of my other plugins. I haven't yet seen a good reason to drop support for older versions of calibre, and will keep supporting them as long as it is practical. But, mainly for situations were the user cannot use a newer version of calibre.

[Noirtier]

Originally Posted by davidfor

If there are still issues, then I need to see a debug log and details of what is missing. And the database from the device.

Thanks David. You previously asked me for the OPF files for the examples I had sent, so I sent those also in a second PM. I will send it on again now in case it has got buried. The initial link included above it has the debug log, database files, and text files showing what is missing by comparison between what appears in preview versus what is imported into metadata.

Would it help if I generated a new example set of this data using the new version of the plugin you have just released?

Originally Posted by davidfor

Yes, the ordering in the preview should be by timestamp and in the metadata by location. With the latter, new annotations are merged with any existing annotations. I think the location order works better for that.

I agree that location should in theory be the better approach. Though if you are saying that there is a difference between how things are ordered in the preview vs. imported metadata, might this be a clue as to where the problem lies (considering it seems to work correctly without missing content in the preview)? Just clutching at straws here

[wold]

@davidfor, as far as I'm concerned you can roll a beta with the added Pocketbook support.
When you have the time, off course. No hurry

Matching for PB is done now using the path, with fallback to author/title or just title.
When things are added outside of calibre, moved around and changed, some extra handling is required. As on my unit...
Beta testers should enable debug, and pay attention to lines with "_read_database_annotations" and "_fetch_annotations".

BTW, changed were also the "book_mi.author" definition (this seems to expect a list of authors?), and authors_fixed was added to "installed_books_by_title". A bit of a hack to be honest, but it doesn't seem to break anything.

[davidfor]

As strongly hinted above, @wold has been working on support for Pocketbook devices. The attached beta is the result.

If anyone has a Pocketbook and can test the plugin, please report the result here. If there are any issues run turn on debug logging in the plugins configuration and run calibre in debug mode. Then post the log for us to look at. Along with details of what the problem is. We may also need to see the database from the device to understand what is going on.

[bernard.ryefield]

On Windows 10, last versions of Calibre and Annotations plugin, Kobo Forma:
Annotations plugin is stuck at 0% for hours on "Compiling annotations for a book". No disk access nor UC used from the processes.

[wold]

Originally Posted by bernard.ryefield

On Windows 10, last versions of Calibre and Annotations plugin, Kobo Forma:
Annotations plugin is stuck at 0% for hours on "Compiling annotations for a book". No disk access nor UC used from the processes.

A number of things may be related. I will help out tomorrow.

@Davidfor: my fix retained annotations with an empty (content) paragraph. While I do not experience problems (Linux/Win10 with PB), it seems reasonable the Kobo reader breaks somewhere on re-reading these paragraphs. It might even involve the seperate hash and timestamp routines, though hash seems standard. I Should have checked more extensively.

My sync option did just failed to appear for Pocketbooks, though very rarely the Pocketbook740 driver overrules the Lux2 driver. Ruled out seem to git branch mistakes, partially updated plugins (not using zip -ru anymore). The bs4 code was tested separately, though only in Python3. caching issue.

I did get one error messages, but since it posed no problem, I judging it to be specific to that readers and ignored it:

A) Involving _Marvin.py:

  File "/usr/lib/calibre/calibre/gui2/ui.py", line 236, in initialize
    ac.do_genesis()
  File "/usr/lib/calibre/calibre/gui2/actions/__init__.py", line 147, in do_genesis
    self.genesis()
  File "calibre_plugins.annotations.action", line 457, in genesis
  File "calibre_plugins.annotations.action", line 872, in load_dynamic_reader_classes
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/calibre_annotations_plugin/_Marvin.py", line 15, in <module>
    from calibre.ebooks.BeautifulSoup import UnicodeDammit
ImportError: cannot import name 'UnicodeDammit' from 'calibre.ebooks.BeautifulSoup' (/usr/lib/calibre/calibre/ebooks/BeautifulSoup.py)

Removing _Marvin.py made syncing work. But placing it back, with the same error, did not break fetching...

B) Removing _Marvin.py did reveal another 'masked' error in _iBooks.py:

Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/ui.py", line 236, in initialize
    ac.do_genesis()
  File "/usr/lib/calibre/calibre/gui2/actions/__init__.py", line 147, in do_genesis
    self.genesis()
  File "calibre_plugins.annotations.action", line 457, in genesis
  File "calibre_plugins.annotations.action", line 872, in load_dynamic_reader_classes
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/calibre_annotations_plugin/_iBooks.py", line 11, in <module>
    import cStringIO, os, re, sqlite3
ModuleNotFoundError: No module named 'cStringIO'

This is a Python3 error:
https://stackoverflow.com/questions/...amed-cstringio

PS. notes with empty paragraphs (now retained by the fix) are not searchable using find annotations. Probably needs a loop.

[davidfor]

Separating the two issues and answering the easy one first...

Originally Posted by wold

I did get one error messages, but since it posed no problem, I judging it to be specific to that readers and ignored it:

A) Involving _Marvin.py:

  File "/usr/lib/calibre/calibre/gui2/ui.py", line 236, in initialize
    ac.do_genesis()
  File "/usr/lib/calibre/calibre/gui2/actions/__init__.py", line 147, in do_genesis
    self.genesis()
  File "calibre_plugins.annotations.action", line 457, in genesis
  File "calibre_plugins.annotations.action", line 872, in load_dynamic_reader_classes
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/calibre_annotations_plugin/_Marvin.py", line 15, in <module>
    from calibre.ebooks.BeautifulSoup import UnicodeDammit
ImportError: cannot import name 'UnicodeDammit' from 'calibre.ebooks.BeautifulSoup' (/usr/lib/calibre/calibre/ebooks/BeautifulSoup.py)

Removing _Marvin.py made syncing work. But placing it back, with the same error, did not break fetching...

B) Removing _Marvin.py did reveal another 'masked' error in _iBooks.py:

Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/ui.py", line 236, in initialize
    ac.do_genesis()
  File "/usr/lib/calibre/calibre/gui2/actions/__init__.py", line 147, in do_genesis
    self.genesis()
  File "calibre_plugins.annotations.action", line 457, in genesis
  File "calibre_plugins.annotations.action", line 872, in load_dynamic_reader_classes
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/calibre_annotations_plugin/_iBooks.py", line 11, in <module>
    import cStringIO, os, re, sqlite3
ModuleNotFoundError: No module named 'cStringIO'

This is a Python3 error:
https://stackoverflow.com/questions/...amed-cstringio

I assume the above is with your build of the plugin. The official build doesn't include the reader plugins whose name starts with an underscore. Those plugins do not work because of issues with iOS. They have not worked for a long time and are just around in case someone was looking for sample code.

But, for safety, I will add a check in the reader loading code to exclude them here as well.

[davidfor]

Originally Posted by bernard.ryefield

On Windows 10, last versions of Calibre and Annotations plugin, Kobo Forma:
Annotations plugin is stuck at 0% for hours on "Compiling annotations for a book". No disk access nor UC used from the processes.

Which version of the plugin are you running? The released version or the above beta? In any case, I don't think it is the problem that @word mentioned. But, I need some details. Could you turn on debugging in the plugin's configuration and then run calibre in debug mode? After you hit this point, close calibre and post the log it shows. If you don't want to post it, you can upload it somewhere and send me a PM with the link. Hopefully that will point me in the right direction.

[bernard.ryefield]

Originally Posted by davidfor

Which version of the plugin are you running? The released version or the above beta? In any case, I don't think it is the problem that @word mentioned. But, I need some details. Could you turn on debugging in the plugin's configuration and then run calibre in debug mode? After you hit this point, close calibre and post the log it shows. If you don't want to post it, you can upload it somewhere and send me a PM with the link. Hopefully that will point me in the right direction.

I'm running the last release. Will try this WE. I just hope killing the Calibre processes won't affect the debug log file, since I couldn't close Calibre normally.

[wold]

Originally Posted by wold

PS. notes with empty paragraphs (now retained by the fix) are not searchable using find annotations. Probably needs a loop.

@davidfor, you'll receive a fix for this soon. It's the same one as for annotations_db.

Originally Posted by davidfor

I assume the above is with your build of the plugin.

Correct. I forgot about unshared changes from your side, but diff shows only a version bump.

Originally Posted by davidfor

They have not worked for a long time and are just around in case someone was looking for sample code.

OK. Expected something like that.

Originally Posted by davidfor

In any case, I don't think it is the problem that @word mentioned.

It doesn't seem to be indeed. The PB driver closely follows Kobo's, and there's no re-use of these 'souped' items. Got a bit spooked by this and fetch not working here

[davidfor]

Originally Posted by bernard.ryefield

I'm running the last release. Will try this WE. I just hope killing the Calibre processes won't affect the debug log file, since I couldn't close Calibre normally.

If it does, you can use the command "calibre-debug -g" from the command line. You can copy the contents of the command window to get the debug log.

[davidfor]

Originally Posted by bernard.ryefield

I'm running the last release. Will try this WE. I just hope killing the Calibre processes won't affect the debug log file, since I couldn't close Calibre normally.

Looking at the log, I'm not sure what is happening. The place the code hangs doesn't quite make sense. But, did you try pressing the escape or enter key when this happened? There could be an error dialog opened and it might be opening off the screen or behind something.

I can think of one place that something like this could happen, but, not exactly why. It is a check that is not needed for Kobo devices, and possibly not for any now. Reorganising the code should avoid this. I will be posting another beta with this and some other changes soon.

[davidfor]

Attached is a beta that will hopefully fix some recently reported problems. And some small changes for the Pocketbooks.

The changes are:

• Update: Minor changes for PocketBooks support - Thanks to @wold.
• Fix: Error "TypeError: unhashable type: 'SortableTableWidgetItem'" when importing from Kindle. This was reported on GitHub.
• Fix: Change some handling of confidence generation as some devices have already matched the books. This hopefully solves @bernard.ryefield's problem.

As usual, if there are any problems, please report them here.