[GUI Plugin] Kobo Utilities

[Jellby]

Originally Posted by davidfor

I've also realised while writing this that I have probably said, "I'll look into that" for some things over the last few months. I'll try to go back over the thread in the next few days to see what I've missed, but if you want to remind me, that's good to.

OK, then have a look at this post and ff.:
https://www.mobileread.com/forums/sh....php?p=3499657

- Clean-up database (delete e.g. finished status in events and settings for deleted books)

- Warning/abort if new filename for a book is different from the old existing filename (due to change in metadata, or templates, or internal routines)

[JSWolf]

Is there a setting to have the reading position removed when the book is finished or no longer on the device?

[davidfor]

Originally Posted by Jellby

OK, then have a look at this post and ff.:
https://www.mobileread.com/forums/sh....php?p=3499657

- Clean-up database (delete e.g. finished status in events and settings for deleted books)

The settings will be deleted. The device does a good job of cleaning up the database when you deleted a book. Rows in the Event table should be the only ones left.

Deleting the Events might be a bit harder. I did some fiddling a little while ago for an unrelated reason. When I deleted some Event records, the total hours read suddenly became zero. I think deleting some will be safe, and it's possible the total hours is recalculated when the next book is finished. If so, it all should be OK.

Another delete I want is to remove the full size cover. I hadn't realised that this was generated for each book when the smaller covers are generated. It's obvious now that I think of it, but it hadn't occurred to me. Removing the full size covers for all books will save a lot of space. Assuming they are regenerated when needed. If this work, I'll probably add an option to the driver to only send the thumbnail images.

- Warning/abort if new filename for a book is different from the old existing filename (due to change in metadata, or templates, or internal routines)

That has been in the back of my mind for a long time. I haven't been happy with anything I have come up with. Ideally it would be the driver, but I can't see where to step in and do that. I might be able to do something in the plugin that wraps it.

Unfortunately, none of those are going to happen soon.

[davidfor]

Originally Posted by JSWolf

Is there a setting to have the reading position removed when the book is finished or no longer on the device?

Where do you want the reading position removed from? When you delete it from the device, this will be removed. If it is on the device but finished, the reading position is normally set to the start of the book. Plus you can reset this using the "Update metadata in Device Library". Or the "Change Reading status in Device Library" function from the Device list.

[Blas]

I'd like know how many awards you guys have there to unlock in your kobo touches. Cuz mine just have 17 awards to be unlocked, and I saw that other people have more awards... I'm asking here cuz the other thread is dead I think

[davidfor]

I put a reply in your other thread. I think the issue there is that Kobo doesn't really push these any more, so there is no focus on them. And I keep being surprised that they don't disappear with a firmware updated

[Blas]

I will check there thx man!

[glasnost]

@davidfor: Thank you for all of the work you have done on this plugin. It seems like you have dramatically extended the interaction b/t Kobo devices and Calibre for many people.
Unfortunately, I am not one yet :-(

Try as I might, the plugin does not respond to my configuration - and searching through this thread and the help doc have not fixed things. Please let me know what I am missing!

Device
Kobo H2O, factory reset last night to: 4.5.9587 no firmware modification
Changed ...eReader.conf file to enable removal of header/footer, removed header/footer
epubs & pdfs, ~exclusively sideloaded NOT kepubs, but for one from the kobostore, all added last night
Using only ondevice storage

*Used the 'dogear' bookmark function on a few books, b/c earlier in the thread you wrote that the plugin tracked that and not the device reported 'percentage'

Kobo reading analytics turned off
KoboUtilities - analytics trigger added

Calibre
Windows 10, 64b
Version 3.6.0
KoboUtilities v. 2.5.2
KoboTouch driver 2.3.3 (have also tried KoboTouchExtended 2.8.2)

KoboUtilities Configuration
Default Profile:
Current Reading Location Columns #lastread Date
Percent Read#percentread Integers
Last Read#readloc Text, column shown in the tag browser
Checked: Store current bookmarks, prompt to store changes, only if more recent, not if finished in library

Devices:
Checked Backup database on each connection to ...\Documents\...\KoboDeviceBackup
Checked compress
Number 5

Kobo Driver Configuration
Checked:
Read metadata, use subdirectories, default save template
Collections (column #genre), create & destroy empty bookshelves
Upload b&2 covers
Update metadata, set series info
Show recommendations, show previews

Use Case
Automatically storing reading information (book status, percent read, ideally annotations)

Automatically backing up database upon connection

Integrating Kobo & Goodreads to autopopulate shelves:
Book added to device (i.e. "unread") -> "to read"
Book reading -> "reading" ideally including percent read

"read"(finished) "reading"(reading and/or percent read?) "to read"(unread)

Maybe more advanced functionality later, but right now this is good enough for me!

Testing

Manually storing the device database works great. Nothing else appears to.
(nothing appears to happen on connect, nothing is added when 'storing' bookmarks, but manually storing the database works.)

I've tried fresh installs of the KoboUtilities. I've tried both KoboTouch drivers. I have done a full factory reset of the device. I'm at a loss.

I've sent you my debug log, but I'd prefer to get any help on thread in case someone else runs into a similar problem.
Thanks!

[davidfor]

Originally Posted by glasnost

Testing

Manually storing the device database works great. Nothing else appears to.
(nothing appears to happen on connect, nothing is added when 'storing' bookmarks, but manually storing the database works.)

I've tried fresh installs of the KoboUtilities. I've tried both KoboTouch drivers. I have done a full factory reset of the device. I'm at a loss.

I've sent you my debug log, but I'd prefer to get any help on thread in case someone else runs into a similar problem.

Where did you send the debug log? I haven't seen anything yet.

Are the automatic backup and reading position storage setup to only happen for particular devices? If so, while the device is connected and seen by calibre, go to the second tab of the plugins configuration. The list of devices at the top should show your device with an icon to indicate it is connected. If this the isn't case, delete the entries for the Aura H2O, and then press the add current device button under list. Then change the rest of the configuration to use this device.

Or, and simpler if you only have the one Kobo, change the configuration to do the actions for any connected Kobo device. The backup is a single check box. For the reading status saving, it should be selecting "*Any device" in the "Device this profile is for" drop-down on the first tab.

[davidfor]

Of course, the PM arrived as soon as I hit the post button

Looking through the log, I can see what the problem is.

It looks like it is what I said in the earlier post. The configuration is for a specific device, but that isn't what is being seen. The problem here is that the match is against an id that calibre generates when the device is first seen and then stored on the device in the file "driveinfo.calibre". If that file is removed, then calibre will generate a new id the next time the device is seen. Under normal circumstances, the file shouldn't be removed, but if you did a factory reset it will disappear along with everything else.

The simple solution is to change the configuration to not be device dependent. That works fine if you only have one device. If you have multiple, you need to decide if you want them treated differently.

[glasnost]

Originally Posted by davidfor

The simple solution is to change the configuration to not be device dependent. That works fine if you only have one device. If you have multiple, you need to decide if you want them treated differently.

You were spot on. For some reason, Calibre was remembering an old device. Thank you!

So, obviously, rather than going with your simple suggestion, I decided to set the profile to *Any AND delete the old device from the device list. But, I couldn't remember the SN, so I figured I'd just delete both.

NO. That caused all sorts of exceptions, as the python scripts tried to call nul data (i.e. the device list was empty.) I couldn't even open the plugin configuration. But automagically, my reading position was added & shelves sorted on goodreads. (b/c I had set the profile to *any). I removed the plugin, restarted, added back in the plugin, restarted.

Still couldn't open plugin configuration!

This is when I learned that removing the plugin from Calibre does not remove the file: "Kobo Utilities.json" or "KoboUtilities_Help.html"

I deleted them, removed/added the plugin, opened config, a new .json config file was generated, and everything worked like a charm.

Suggestions:
1. Default plugin configuration profile to Any* ? Multiple, unique device config seems like it would be the exception.
2. remove the config file when removing the plugin. This probably requires changes to calibre plugin handling code, but if it can be done on your side, that'd be awesome!

Thank you again, I am so thrilled that everything is working smoothly!

[glasnost]

David

A couple of questions: it looks like you have integrated Goodreads functions into the Kobo utilities plugin. (though, I suppose it could be the goodreads plugin doing the integration, lemme know if I should direct this their way.)

When I update my reading reading position on device, plug in, and K.U. does its sync, a popup shows up asking me if I want to update reading position and add finished books to shelf 'read'

A. I love this idea
B. I think that the function calls re shelves are slightly off. Here's what I've tested:
1. Currently, if a book is not on a shelf, the popup to link it and add it to a shelf does not actually add the book to a shelf.
2. Once a book is on the 'to read' shelf (or, presumably any other shelf) - reading progress is correctly synced.
3. When a book is 100%, it does not move to the 'read' shelf, as it ought at 100%

Is this another PEBCAK issue of mine, or is this a work in progress? Lemme know how I can help you debug, because I am ~thrilled~ about this

Why I suspect this comes from K.U. : DEBUG: 403.9 KoboUtilitiesAction::_store_completed - goodreads_sync_plugin.users.keys()= [u'Ted']

and when I add a book to a shelf using the goodreads plugin, it gets added :-)

[Terisa de morgan]

Originally Posted by glasnost

3. When a book is 100%, it does not move to the 'read' shelf, as it ought at 100%

I'm not David but I want to comment on this suggestion: Not everybody send the books at the "read" shelf when it is finished (I do, but I know people who doesn't), so ought to...

[davidfor]

Originally Posted by glasnost

You were spot on. For some reason, Calibre was remembering an old device. Thank you!

So, obviously, rather than going with your simple suggestion, I decided to set the profile to *Any AND delete the old device from the device list. But, I couldn't remember the SN, so I figured I'd just delete both.

NO. That caused all sorts of exceptions, as the python scripts tried to call nul data (i.e. the device list was empty.) I couldn't even open the plugin configuration. But automagically, my reading position was added & shelves sorted on goodreads. (b/c I had set the profile to *any). I removed the plugin, restarted, added back in the plugin, restarted.

Still couldn't open plugin configuration!

You didn't happen to record the details of the error? That would help to look at it.

This is when I learned that removing the plugin from Calibre does not remove the file: "Kobo Utilities.json" or "KoboUtilities_Help.html"

I deleted them, removed/added the plugin, opened config, a new .json config file was generated, and everything worked like a charm.

Suggestions:
1. Default plugin configuration profile to Any* ? Multiple, unique device config seems like it would be the exception.

As there is no way to use the store function without configuring it, I don't think that will help much.

2. remove the config file when removing the plugin. This probably requires changes to calibre plugin handling code, but if it can be done on your side, that'd be awesome!

It can't be done. Nothing in the plugin is called when the plugin is uninstalled. It's been suggested before but Kovid hasn't wanted to do it. And, personally, I uninstall and reinstall the various plugins enough that I don't want the configuration removed. Once I fix the problem you are seeing, it won't matter as much.

[davidfor]

Originally Posted by glasnost

David

A couple of questions: it looks like you have integrated Goodreads functions into the Kobo utilities plugin. (though, I suppose it could be the goodreads plugin doing the integration, lemme know if I should direct this their way.)

The Kobo Utilities updates the status in the library, selects the books and calls the Goodreads Sync "Update reading progress" function. After that, it's the Goodreads Sync plugin doing the work. But, as it happens, I wrote that part of the Goodreads Sync plugin and maintain the plugin now.

When I update my reading reading position on device, plug in, and K.U. does its sync, a popup shows up asking me if I want to update reading position and add finished books to shelf 'read'

A. I love this idea
B. I think that the function calls re shelves are slightly off. Here's what I've tested:
1. Currently, if a book is not on a shelf, the popup to link it and add it to a shelf does not actually add the book to a shelf.
2. Once a book is on the 'to read' shelf (or, presumably any other shelf) - reading progress is correctly synced.
3. When a book is 100%, it does not move to the 'read' shelf, as it ought at 100%

Is this another PEBCAK issue of mine, or is this a work in progress? Lemme know how I can help you debug, because I am ~thrilled~ about this

Why I suspect this comes from K.U. : DEBUG: 403.9 KoboUtilitiesAction::_store_completed - goodreads_sync_plugin.users.keys()= [u'Ted']

and when I add a book to a shelf using the goodreads plugin, it gets added :-)

The update reading progress doesn't try to put the books onto the "Currently Reading" shelf. I have considered it, but, I always do my shelving on Goodread when I start reading a book. Plus it would need to be checked whenever the progress was updated. And I don't like that.

As to putting the finished book on the Read shelf, this works perfectly here if I have the checkbox in the bottom left of the dialog selected. But, do you have multiple users configured in the Goodreads Sync plugin?