Force-updating books on a Kobo H2O from code

[ceridwen]

I'm writing a command-line tool to update online serials on my Kobo using Calibre and the Kobo plugin. From testing I've done using the GUI, the Kobo doesn't seem to recognize that a book file has changed unless I first delete it from the Calibre database and the Kobo, then add the updated file as a new book.

I want to be able to do this same thing by running a plugin from the command line (written as an __init__.py that I then execute with calibredb). However, while there's an API function to remove books from the database https://manual.calibre-ebook.com/db_...e.remove_books, it's not clear to me how I make it remove books from the device also. Is there a way to do this? Alternately, I could have my code delete books directly from the file system on the Kobo, but how do I go from a book in the Calibre db to the corresponding file on the Kobo's file system? With simple heuristics like matching title I could end up deleting the wrong book.

Is there another way of forcing the Kobo to update a book other than this delete-and-recreate flow?

[davidfor]

What problem are you actually trying to achieve? What you say you are doing doesn't quite make sense for anything I can think of. Nothing I can think of would need you to remove the book from the calibre library.

As you mention updating online serials, I think your problem is that when you to send an updated copy of the book to the Kobo device, the ToC and metadata do not get updated. If that is the problem, there is something really simple you can do: Turn off the "Do not treat replacements as new books" option in the driver. This is on the second tab of both the KoboTouch and KoboTouchExtended drivers. For updating the metadata, there are options in the driver to do this each time the device is connected.

The reason for this is that when the Kobo device sees a new sideloaded book, it imports details from it to the internal database. These details include the metadata and the ToC. When a book is changed (based on file size), it deletes the book details from the internal database and imports it as a new book. But, this means you lose the reading status, annotations and collections.

The early firmware didn't do this. If you updated a book, it just accepted it. When this changed, I worked out how to prevent it and updated the KoboTouch driver (and hence the KoboTouchExtended driver) to handle resending a book this way. My logic was that if you were only making changes to things like spelling or styles, you didn't want to lose the reading status. And, if you made deeper changes you could delete the book first. That was the best of both worlds. But, due to several requests, I added the above option to the driver last year. That way people could chose which behaviour they want. The option is on by default as that is what the behaviour was before the option was added. I think turning off the option will do what you want.


For updating the book in calibre, I normally select the book in the list and drop the updated version in the details pane. That replaces the version in the library without changing the metadata. I'm pretty sure there is a way to do this using the command-line, but, I've never needed try.

[BetterRed]

Originally Posted by davidfor

I'm pretty sure there is a way to do this using the command-line, but, I've never needed try.

The calibredb add_format command does it, you need the book id. There's switch to allow/prevent overwriting of existing format.

BR

[ceridwen]

Originally Posted by davidfor

As you mention updating online serials, I think your problem is that when you to send an updated copy of the book to the Kobo device, the ToC and metadata do not get updated. If that is the problem, there is something really simple you can do: Turn off the "Do not treat replacements as new books" option in the driver. This is on the second tab of both the KoboTouch and KoboTouchExtended drivers. For updating the metadata, there are options in the driver to do this each time the device is connected.

The reason for this is that when the Kobo device sees a new sideloaded book, it imports details from it to the internal database. These details include the metadata and the ToC. When a book is changed (based on file size), it deletes the book details from the internal database and imports it as a new book. But, this means you lose the reading status, annotations and collections.

The early firmware didn't do this. If you updated a book, it just accepted it. When this changed, I worked out how to prevent it and updated the KoboTouch driver (and hence the KoboTouchExtended driver) to handle resending a book this way. My logic was that if you were only making changes to things like spelling or styles, you didn't want to lose the reading status. And, if you made deeper changes you could delete the book first. That was the best of both worlds. But, due to several requests, I added the above option to the driver last year. That way people could chose which behaviour they want. The option is on by default as that is what the behaviour was before the option was added. I think turning off the option will do what you want.

I may have been mistaking the lack of ToC and metadata update for lack of change. I just tested this again now, with "Do not treat replacements as new books" off, and it did update the title and metadata correctly. (I'm sure the option was checked because that was the default.)

However, in Calibre itself, because I had an existing copy of this book, I now end up with two copies of the same book (one with the old file, one with the new), though only one copy can be on the device at one time. How does this option interact with the main Calibre option to "Auto-merge existing books" under Adding Books in the Preferences menu?

Originally Posted by BetterRed

The calibredb add_format command does it, you need the book id. There's switch to allow/prevent overwriting of existing format.

This updates the underlying file in the Calibre DB, but doesn't send the file to the Kobo, even if I have it connected when I run the command. FYI, davidfor, when I executed the following steps I encountered what might be a bug:

1. Add calibredb add_format <book_id> <file_name>
2. Unplug Kobo.
3. Check on the Kobo that the book didn't update (the underlying file, the metadata, and the ToC were still using the same version).
4. Plug Kobo back in to the computer.
5. Start the Calibre GUI, find the correct entry, and hit "Send book to device."
6. Close Calibre GUI and unmount the Kobo file system.
7. Open the book.

The first time I tried to open it, the Kobo told me that I was trying to open an Adobe DRM-protected Epub and reset my reading position. This is an AO3 book, so it's definitely not DRMed. (I didn't have bookmarks so I don't know if it reset that too.) I could open it and read it on a subsequent attempt, but it seems something with the database was slightly broken.

I'm assuming this is the matching DB function to the CLI add_format: https://manual.calibre-ebook.com/db_...che.add_format. What function do I need to call to execute the second step of this flow, getting it to send the updated book to the device?

[davidfor]

Originally Posted by ceridwen

I may have been mistaking the lack of ToC and metadata update for lack of change. I just tested this again now, with "Do not treat replacements as new books" off, and it did update the title and metadata correctly. (I'm sure the option was checked because that was the default.)

I'm pretty sure it was being replaced, but with no ToC changes. If you were updating a downloaded book because of an extra chapter being added, not seeing it in the ToC would have seemed to be evidence that it wasn't changed.

However, in Calibre itself, because I had an existing copy of this book, I now end up with two copies of the same book (one with the old file, one with the new), though only one copy can be on the device at one time. How does this option interact with the main Calibre option to "Auto-merge existing books" under Adding Books in the Preferences menu?

The calibredb command with the appropriate options should solve this. I normally do this through the GUI so haven't looked at the command options properly.

This updates the underlying file in the Calibre DB, but doesn't send the file to the Kobo, even if I have it connected when I run the command. FYI, davidfor, when I executed the following steps I encountered what might be a bug:

1. Add calibredb add_format <book_id> <file_name>
2. Unplug Kobo.
3. Check on the Kobo that the book didn't update (the underlying file, the metadata, and the ToC were still using the same version).
4. Plug Kobo back in to the computer.
5. Start the Calibre GUI, find the correct entry, and hit "Send book to device."
6. Close Calibre GUI and unmount the Kobo file system.
7. Open the book.

That's what I would be doing, though normally through the UI.

The first time I tried to open it, the Kobo told me that I was trying to open an Adobe DRM-protected Epub and reset my reading position. This is an AO3 book, so it's definitely not DRMed. (I didn't have bookmarks so I don't know if it reset that too.) I could open it and read it on a subsequent attempt, but it seems something with the database was slightly broken.

I get the DRM error occasionally. It is can be a sign the database is corrupt. But, it can also be cleared by restarting the device. In that case, I assume a memory leak or a stuck process. But, I haven't had it for a while.

I'm assuming this is the matching DB function to the CLI add_format: https://manual.calibre-ebook.com/db_...che.add_format. What function do I need to call to execute the second step of this flow, getting it to send the updated book to the device?

The API for the device driver is at https://manual.calibre-ebook.com/plu...ices.interface. I don't see anything suitable to use there. "upload_books" would be the first thought, but that isn't where I would have started. It uses a list of file names rather than a list of ids from the library. It actually looks like you could use it to send a book to the device without adding it to the calibre library.

There doesn't seem to be a send-to-device command for the calibredb. But, there is an "export" command which will save the book to disk. You could use this with the path on the device. That would actually bypass all the driver options which would mean the device would import it as new all the time.