[GUI Plugin] Kobo Utilities

[davidfor]

Originally Posted by matthrie

Its good

I uninstall the drivers, delete the configuration files and install the beta program.

Don't use the beta that I posted as an update has already been released. And I have just removed the beta. And you should just need to remove the "device_drivers_KOBOTOUCHEXTENDED.py.json". So long as you have the latest version of the plugin installed, and are not running calibre v3.45, the older style configuration file will be converted to the new style properly.

[Quoth]

Book Annotated three days ago was ok (Started 31st, finished 1st), but error is back

Started 1st

calibre, version 3.46.0
ERROR: Unhandled exception: <b>TypeError</b>:unsupported operand type(s) for *: 'NoneType' and 'int'

calibre 3.46  embedded-python: True is64bit: True
Linux-4.15.0-55-generic-x86_64-with-debian-stretch-sid Linux ('64bit', 'ELF')
('Linux', '4.15.0-55-generic', '#60~16.04.2-Ubuntu SMP Thu Jul 4 09:03:09 UTC 2019')
Python 2.7.15
Linux: ('debian', 'stretch/sid', '')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 17, 1) && DeDRM (6, 6, 1) && K4PC, K4Mac, Kindle Mobi and Topaz DeDRM (0, 3, 1) && Package KFX (from KFX Input) (1, 17, 1) && Kobo Books (1, 7, 0) && KePub Output (3, 1, 3) && X-Ray Creator (3, 2, 1) && Amazon.com Multiple Countries (1, 0, 0) && Media File Importer (1, 0, 21) && KePub Metadata Reader (3, 0, 2) && Manage Series (1, 2, 8) && Barnes & Noble (1, 2, 15) && Generate Cover (1, 5, 21) && EpubMerge (2, 5, 0) && KoboTouchExtended (3, 1, 5) && Obok DeDRM (6, 5, 4) && Author Book Count (2, 1, 7) && FictionDB (1, 2, 0) && KePub Input (3, 0, 0) && KindleUnpack - The Plugin (0, 81, 5) && Annotations (1, 9, 2) && KePub Metadata Writer (3, 0, 3) && Hyphenate This! (0, 0, 8) && Wiki Reader (2, 1, 0) && EpubCheck (0, 1, 9) && Kindle covers (G) (0, 3, 0) && Extract ISBN (1, 4, 4) && Library Splitter (1, 0, 1) && Manage Sony x50 Reader Book List (1, 2, 11) && Modify ePub (1, 3, 13) && Quality Check (1, 9, 11) && Goodreads (1, 4, 0) && Resize Cover (1, 0, 2) && EpubSplit (2, 7, 0) && Open With (1, 5, 10) && Kindle Collections (1, 7, 24) && Reading List (1, 6, 6) && KFX metadata reader (from KFX Input) (1, 17, 1) && KFX Input (1, 17, 1) && Kobo Utilities (2, 8, 0) && Quick Preferences (1, 4, 1) && Smashwords Metadata (1, 0, 0) && Kindle hi-res covers (0, 4, 0) && Copy Cover To Device (2, 0, 0) && Find Duplicates (1, 6, 3)
Traceback (most recent call last):
  File "calibre_plugins.koboutilities.action", line 1906, in getAnnotationForSelected
  File "calibre_plugins.koboutilities.action", line 2644, in _getAnnotationForSelected
  File "site-packages/calibre/devices/kobo/driver.py", line 1191, in get_annotations
  File "site-packages/calibre/devices/kobo/bookmark.py", line 30, in __init__
  File "site-packages/calibre/devices/kobo/bookmark.py", line 94, in get_bookmark_data
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

Annotated last night

calibre, version 3.46.0
ERROR: Unhandled exception: <b>TypeError</b>:unsupported operand type(s) for *: 'NoneType' and 'int'

calibre 3.46  embedded-python: True is64bit: True
Linux-4.15.0-55-generic-x86_64-with-debian-stretch-sid Linux ('64bit', 'ELF')
('Linux', '4.15.0-55-generic', '#60~16.04.2-Ubuntu SMP Thu Jul 4 09:03:09 UTC 2019')
Python 2.7.15
Linux: ('debian', 'stretch/sid', '')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 17, 1) && DeDRM (6, 6, 1) && K4PC, K4Mac, Kindle Mobi and Topaz DeDRM (0, 3, 1) && Package KFX (from KFX Input) (1, 17, 1) && Kobo Books (1, 7, 0) && KePub Output (3, 1, 3) && X-Ray Creator (3, 2, 1) && Amazon.com Multiple Countries (1, 0, 0) && Media File Importer (1, 0, 21) && KePub Metadata Reader (3, 0, 2) && Manage Series (1, 2, 8) && Barnes & Noble (1, 2, 15) && Generate Cover (1, 5, 21) && EpubMerge (2, 5, 0) && KoboTouchExtended (3, 1, 5) && Obok DeDRM (6, 5, 4) && Author Book Count (2, 1, 7) && FictionDB (1, 2, 0) && KePub Input (3, 0, 0) && KindleUnpack - The Plugin (0, 81, 5) && Annotations (1, 9, 2) && KePub Metadata Writer (3, 0, 3) && Hyphenate This! (0, 0, 8) && Wiki Reader (2, 1, 0) && EpubCheck (0, 1, 9) && Kindle covers (G) (0, 3, 0) && Extract ISBN (1, 4, 4) && Library Splitter (1, 0, 1) && Manage Sony x50 Reader Book List (1, 2, 11) && Modify ePub (1, 3, 13) && Quality Check (1, 9, 11) && Goodreads (1, 4, 0) && Resize Cover (1, 0, 2) && EpubSplit (2, 7, 0) && Open With (1, 5, 10) && Kindle Collections (1, 7, 24) && Reading List (1, 6, 6) && KFX metadata reader (from KFX Input) (1, 17, 1) && KFX Input (1, 17, 1) && Kobo Utilities (2, 8, 0) && Quick Preferences (1, 4, 1) && Smashwords Metadata (1, 0, 0) && Kindle hi-res covers (0, 4, 0) && Copy Cover To Device (2, 0, 0) && Find Duplicates (1, 6, 3)
Traceback (most recent call last):
  File "calibre_plugins.koboutilities.action", line 1906, in getAnnotationForSelected
  File "calibre_plugins.koboutilities.action", line 2644, in _getAnnotationForSelected
  File "site-packages/calibre/devices/kobo/driver.py", line 1191, in get_annotations
  File "site-packages/calibre/devices/kobo/bookmark.py", line 30, in __init__
  File "site-packages/calibre/devices/kobo/bookmark.py", line 94, in get_bookmark_data
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

Again, at least I could read the annotations using the far slower and less suitable Annotations Plugin. So if it's the Kobo database or some unliked text, I don't know.

[davidfor]

Originally Posted by FrustratedReader

Book Annotated three days ago was ok (Started 31st, finished 1st), but error is back

Started 1st

Again, at least I could read the annotations using the far slower and less suitable Annotations Plugin. So if it's the Kobo database or some unliked text, I don't know.

No, it is not back. It never left. Why? Because NOTHING HAS CHANGED. I have NOT FIXED THIS PROBLEM. When I do, I will mention it here. If you are only seeing it sometimes, it is because of something in the book. Which I'm pretty sure I said the first time you reported the problem.

[Quoth]

I meant by saying it's back, that I have annotated maybe 3 or 4 books without a problem.

What ever it is that goes wrong the problem is intermittent. I had assumed that when something doesn't work that the procedure is to copy & paste the error report.

The books are created by Calibre from docx files exported by LibreOffice. All the same document formats and conversion settings. All the epubs pass epub checking.

I wasn't suggesting you'd fixed it. I was simply copy/pasting the error report and pointing out that what ever it is, it only happens sometimes. All the books are produced the same way, from the same LibreOffice template and same Calibre settings.

Obviously I've no idea why some are OK with the Kobo Utilities Fetch Annotation and some don't work at all, yet all are fine on the Kobo and the Annotations Plugin works, which suggests it's not some major issue in the Kobo H2O original.

It's a mystery to me.

Also surely it's a database interpretation issue and not "something in the book" as it's do with annotations? The Kobo displays them fine, even after a power off. The Annotations Plugin is reading them. All the books are produced the same way. About 42,000 words to 180,000 words. The two most recent fails are 66,000 and 71,700 words. Number of Annotations: 72 & 40
The TOC is looking fine on the Kobo.
I can see no difference between books that "fail" with Kobo Utilities fetching annotation and those that work.

I can't give you either the source epub books or a copy of my kobo database.

[davidfor]

Sorry, but everything about your post suggested that you thought it had been fixed but had returned. And repeatedly posting the same error is pointless. It was identical to the other times you posted the error, so how was it helping?

And as I stated the last time, and implied the first time you posted the error, there is no code shared between what this plugin is using and the Annotations plugin. As you have so politely commented, they don't behave the same, so why should they suffer from the same bugs?

And of course it is database interpretation. I mean, that's the whole point of this code. Interpret what is in the database and display it. And if it fails sometimes and not others, how could it possibly not be related to the books? But, it is almost definitely the ToC and internal file names. There are spaces or something else in them that gets URL encoded or otherwise mangled so that the current code doesn't match them. There is a possibility it is related to illegally, for a kepub, not having a one-to-one relationship between the ToC and the internal files. Or some other breakage of the kepubs rules.

[Quoth]

I have no idea why sometimes it seems fine.


1) The Annotations plugin fetches the annotations the Kobo Utilies don't. So though they are different code the Annotations "copes silently" with what ever is the problem. This might be a clue. What extra thing or assumption is one or the other doing?
2) All the books are epubs. Not kepubs. I checked driver settings and on the Kobo info for book.
3) All the books pass the Daisy group epub check
4) All the books are produced using the same LibreOffice template, identical styles, workflow etc. Though the SF & F titles differ slightly, there is no correlation between failing imports and that.
5) There are no soft spaces. All spaces are ordinary and only single. No paragraph starts or ends with a space. There are no empty paragraphs. Checked with regex.
6) The sections at the start with URLS and at the end with images are identical. Only the body text and TOC differ. The main body and human readable contents page and the copyright page have no images. The failing & working "books" have exactly the same images and fonts.
7) TOC checked on every book before transfer to Kobo.
8) There are ONLY notes. No highlights or bookmarks.
9) No obvious difference between working and failing annotations imports.
10) The actual system Contents menu items match the human readable contents page that is always just before Chapter 1
11) Some books start the chapter title with Chapter (Fantasy) and others with just a number (SF titles). Makes no difference.
12) Logically it's SOMETHING in some books OR the way Kobo H20 original stores annotations. Could it be caused by adding an annotation and deleting it, or re-editing one, or adding bookmarks and deleting them. I do all of those. I could be how I'm using annotating and not the book itself?

The differences I see between Kobo Utilities and Annotations plugin fetching annotations:
a) Annotations has date & time on each note. Utilities has book progress instead, though it's called Chapter.
b) Utilities prefixes every chapter heading from the book with "Chapter nn: ", so you might see on each note: Chapter 0: Chapter 1: The Village
This is mysteriously zero based. The Annotations plug in simply has the actual text heading.

Also I didn't read the error as I'm not planing to fix it, so I didn't know I was posting something identical. They are not a personal criticism but simply a report.

I have no idea what goes on inside the plugins so all I can do is report when there is a problem. I do not keep a log of Kobo updates, Calibre Updates, Plug in Updates, Linux OS updates so when the error doesn't occur for ages, I might think something has changed and it's thus "fixed". This is in theory possible even if NO changes are made to the annotations fetching part of the Kobo Utilities as it's some unlikely event the software somewhere doesn't like. As the Annotations plugin works differently, I'd not assume its a bug in the Kobo Utilities, though it might be. The problem could be the Kobo, Calibre, the Plug in, Python or whatever.

[davidfor]

Originally Posted by FrustratedReader

I have no idea why sometimes it seems fine.


1) The Annotations plugin fetches the annotations the Kobo Utilies don't. So though they are different code the Annotations "copes silently" with what ever is the problem. This might be a clue. What extra thing or assumption is one or the other doing?

Please, please pay attention:

I fixed a problem with fetching annotations in the Annotations plugin. It was a problem that didn't affect every annotation, or every book. It had a very specific trigger related to the internal file names. The code used by the Kobo Utilities plugin to fetch annotations suffers from the same problem. I have not fixed this. I have had a quick look to makes sure it will have the same problem and know it will a pain to fix.

2) All the books are epubs. Not kepubs. I checked driver settings and on the Kobo info for book.

And I could have sworn that you used the KoboTouchExtended driver to send the books to the device as kepubs. Didn't you choose to do this because the selection works better in kepubs? Has something changed?

3) All the books pass the Daisy group epub check
4) All the books are produced using the same LibreOffice template, identical styles, workflow etc. Though the SF & F titles differ slightly, there is no correlation between failing imports and that.
5) There are no soft spaces. All spaces are ordinary and only single. No paragraph starts or ends with a space. There are no empty paragraphs. Checked with regex.
6) The sections at the start with URLS and at the end with images are identical. Only the body text and TOC differ. The main body and human readable contents page and the copyright page have no images. The failing & working "books" have exactly the same images and fonts.
7) TOC checked on every book before transfer to Kobo.
8) There are ONLY notes. No highlights or bookmarks.
9) No obvious difference between working and failing annotations imports.
10) The actual system Contents menu items match the human readable contents page that is always just before Chapter 1
11) Some books start the chapter title with Chapter (Fantasy) and others with just a number (SF titles). Makes no difference.
12) Logically it's SOMETHING in some books OR the way Kobo H20 original stores annotations. Could it be caused by adding an annotation and deleting it, or re-editing one, or adding bookmarks and deleting them. I do all of those. I could be how I'm using annotating and not the book itself?

The differences I see between Kobo Utilities and Annotations plugin fetching annotations:
a) Annotations has date & time on each note. Utilities has book progress instead, though it's called Chapter.
b) Utilities prefixes every chapter heading from the book with "Chapter nn: ", so you might see on each note: Chapter 0: Chapter 1: The Village
This is mysteriously zero based. The Annotations plug in simply has the actual text heading.

And all of that demonstrates how little attention you are paying to what I have said. The issue is with the internal file names. And how the ToC is built and interpreted by the device and stored in the database. And how the references to the bookmarks/annotations/hightlights are store. Nothing you have said has any relationship to what is causing the error.

Also I didn't read the error as I'm not planing to fix it, so I didn't know I was posting something identical. They are not a personal criticism but simply a report.

Really, you didn't compare them to your previous reports to see if they had occurred before?

I have no idea what goes on inside the plugins so all I can do is report when there is a problem. I do not keep a log of Kobo updates, Calibre Updates, Plug in Updates, Linux OS updates so when the error doesn't occur for ages, I might think something has changed and it's thus "fixed". This is in theory possible even if NO changes are made to the annotations fetching part of the Kobo Utilities as it's some unlikely event the software somewhere doesn't like. As the Annotations plugin works differently, I'd not assume its a bug in the Kobo Utilities, though it might be. The problem could be the Kobo, Calibre, the Plug in, Python or whatever.

And yet, with all your theorising, you aren't paying any attention to anything said by the person who wrote all the code in question. I have told you what is wrong. Maybe, when I do find the time to look at the code with the intention of fixing it, I will find I am wrong about what the cause of the problem you are seeing is. But, until that happens, there isn't any point in further discussing this. So, I'm going to ignore your posts.

Oh, if you have really changed to annotating epubs on the device, there is another reason. Which has been heavily discussed in the Kobo forum. But, as you have stated the annotations are working correctly on the device, that isn't the cause here.

[Quoth]

I stopped using kepubs as soon as I discovered that the the driver was doing that. The epubs render the same as other ereaders and kindle and source in WP, the kepubs don't.

I do read what you say and pay attention.

Yes, if there is a TOC error, then the annotation doesn't work properly on the device. I saw that once on a draft that I'd not checked the headings/human contents page. So I always make sure that's correct.

Presumably Calibre has some internal recipe for generation of the file names. The only user control is the outer filename?

And no, I didn't look at the reports except at the start. I suspected it might be the same thing as before, but I'd not kept a copy, I'd have had to have looked for the previous post I made and compared line by line. I thought posting might be some help.

Obviously I was wrong.

I'll assume what ever this is will continue to be erratically appearing till there is a new Kobo Utilities that maybe has some reference to this.

Possible recreation if it's how I'm using Annotations:
Use 3 font faces: Chapter, body and divider/break. Ones that will work on Kobo, Kindle via Calibre embedding. All my embedded fonts working in epub (Kobo, Sony, Nook, phone apps), dual mobi on Kindle Keyboard and dual mobi and AZW3 on Kindle PW3.
Have a LibreOffice document with Chapter headings using a custom style, but level set to 1. Text Flow set to insert Page Before (but not with named style).
Put an anchor / bookmark in front of each heading.
Manually create text of contents and hyperlink to #ch1 etc. I do this by auto creation of a TOC without page numbers, cut and paste to a notepad, paste back. This means the text of all headings is in contents, but simply as plain text and then manually add the bookmarks/anchors as URLs (Select Line, Ctrl K, type #<anchor label> as URL, faster than using document url.
Only three styles body text needed: First Paragraph after chapters or breaks has no indent but a margin above. Break is centred Courier with bigger before margin and zero after, Default style has no margins above or below but a 1st line indent.
No other stuff needed.
Save as .odt
Save as MS Word .docx
Import to Calibre.
Convert to epub, but with setting to embed fonts used. Otherwise settings that don't change anything (I use "Tablet" to avoid image rescaling),
Convert epub rather than docx to dual mobi with all margins set to zero. View this and not the epub, as viewing the epub adds a bookmark that Daisy epub check complains about!
See that all TOC on dual mobi made from epub work. See that First Paragraph matches font size & internal line spacing of default paragraphs.
Transfer the epub from the docx to Kobo as epub
Do at least 40 annotations EVENTUALLY, but before done delete some, edit some, add book marks and delete them. Use some to jump to places in book. Take breaks and sleep Kobo.

Sometimes then the Annotations can be read by Kobo Utilities, sometimes not. They always can be read by the Annotations plugin.

Obviously I've no control over Calibre internal filename creation, though it's technically possible to edit those?
Only punctuation in outer file names is <single space> ’ , ( ) No accented letters, though I'd like á é í ó ú some things stupidly break with those as file names.
Every text has in the body and sometime chapters some combination of á é í ó ú also capitalised. That doesn't seem to be a problem for body. I've not checked if it's a problem for the TOC or headings/Chapters.
The Anchors/bookmarks are all each one string, no space or accents or punctuation, stuff like ch1 ch23 epilogue, not the actual chapter visible text. So the URLs in the contents page are simple, just those with # prefix, as you'd expect.

[Quoth]

I do occasionally have ì and ð in words (Scottish Gaelic words and Norse names) as well as loads of áéíóú ÁÓÉ on Irish and Irish names. I'd be surprised if that is the issue, because Calibre wouldn't be doing file names like that.
I suppose a font file name can be a problem, except the times it works and doesn't the embedded fonts are identical.

[PeterT]

Why not attach a document created using your instructions....

Sent from my moto g(7) power using Tapatalk

[Quoth]

Originally Posted by PeterT

Why not attach a document created using your instructions....

A very good point. I would if I could. Likely I'd need to attach the Kobo database if it's related NOT to the book, but how it's annotated!

It would have to be one that doesn't work.

The only "not working ones" I have can't be attached, or no longer exist. I'm proofing & editing for 3 people. The books get edited quickly after annotation so it still makes sense and then replaced by new revision.
Currently I have none with the annotation that's only readable by the "Annotations Plugin".
It's a problem in Kobo Utilities, possibly triggered by something the Kobo H2O original is doing with annotations of epubs. As Calibre decides the internal file names and different revisions of the SAME book with unchanged TOC, same fonts and minor proof edits have worked, it's likely something mysterious related to Kobo editing or deletion of individual bookmarks and notes.

It's not due to something I've changed in the books or the way I make them, it's either the Kobo firmware, Calibre, Kobo Utilities or some Linux resource used to read the annotations, that is all used differently in the Annotations Plug in. Which is WAY slower to use (maybe due to number of books) and doesn't report Progress. Though I have a regex to take out all the date/time stamps on the Annotations Plugin as they are just clutter. All annotations for a book are done over 1 to 3 days. All that matters to me is the finished date.

[Quoth]

I can't currently recreate the "bug".
Even a 140K epub with 200 annotations was fine.

[alexantao]

Hi,

Today I noticed that some books on my calibre library that I've already read do not have the day it was Finished stored. I pluged my Kobo Aura HD again and it does not get updated.

Is there a way to get all dates from my Kobo and store on a column I created on Calibre ?
On KoboUtilities I have configured the Last Read Column correctly set.

My Calibre is updated to the last version of al, including Plug-Ins. Calibre is 3.46
Thanks

[davidfor]

Originally Posted by alexantao

Hi,

Today I noticed that some books on my calibre library that I've already read do not have the day it was Finished stored. I pluged my Kobo Aura HD again and it does not get updated.

Is there a way to get all dates from my Kobo and store on a column I created on Calibre ?
On KoboUtilities I have configured the Last Read Column correctly set.

My Calibre is updated to the last version of al, including Plug-Ins. Calibre is 3.46

Are the books on the device matching to books in the library when it is connected? That means they have the green tick in front of them in the list. If not, then the plugin cannot fetch the reading status from the device.

If they do match, check the options you are using. The "Not if finished in library" option could be the culprit. This checks the percent read in the library and doesn't fetch the status if it is 100% or higher.

And are you running it automatically, or manually? The automatic option will do it when connected and do it for all books on the device. The manual option only tries to fetch the status for the selected books.

If the options are correct, and it doesn't fetch the status, then I need to see a debug log. Restart calibre in debug mode, connect the device, wait for the fetch to finished and close calibre. The debug log will be displayed. Post the complete log, or PM it to me. Hopefully I will see what is wrong from the log.

[alexantao]

Thanks for the reply. Here are the answers:

Originally Posted by davidfor

Are the books on the device matching to books in the library when it is connected? That means they have the green tick in front of them in the list. If not, then the plugin cannot fetch the reading status from the device.

yes, they are. All of then synced and green check.

If they do match, check the options you are using. The "Not if finished in library" option could be the culprit. This checks the percent read in the library and doesn't fetch the status if it is 100% or higher.

The "Not if finished in library" is NOT selected. The books I want to fetch date read are all 100% read in Library.

And are you running it automatically, or manually? The automatic option will do it when connected and do it for all books on the device. The manual option only tries to fetch the status for the selected books.

It is running automatically, and sets the % read correctly. Just the dates are not fetched. Some books does have the dates set and I am setting now mannually, but some are set or are wrong set (there are some I read 2 months ago, but were set red on 2012).

If the options are correct, and it doesn't fetch the status, then I need to see a debug log. Restart calibre in debug mode, connect the device, wait for the fetch to finished and close calibre. The debug log will be displayed. Post the complete log, or PM it to me. Hopefully I will see what is wrong from the log.

I'll see how to start it in debug mode. I thought there were a way to activate debug only on the plug-in. thanks.