[GUI Plugin] Kobo Utilities

[davidfor]

Originally Posted by PeterT

I seem to remember from the ancient days of school that if you re-balanced a tree it was possible for the new one to be "larger" than the source.

That's the only thing I can think of too.

[DNSB]

Originally Posted by PeterT

I seem to remember from the ancient days of school that if you re-balanced a tree it was possible for the new one to be "larger" than the source.

Sounds like a decent possibility. I ran several tests on both copies of the database and no errors or oddities popped up. It tickled my sense of humour when the "compressed" database was larger than the original.

[Lofwyr23]

Something I played over in my mind for a while...

I have a custom column for cycles in a large series, which I want to use instead of the larger series when using the "Update Metadata on device" function. Is there an option or some trick I can use for defining which column is used other then in the code (which is rather inconvenient as I would only apply that change to this one series)?

[davidfor]

Sorry, there isn't any way to do it at the moment. Is suppose that I could add an option to the dialog to choose the series column to use. I'm a little opposed to that as then I have to work out what other attributes need to have their own columns. Is anyone else interested?

One issue with doing something like this is that you will also need to turn off the "Set Series information" option in the driver configuration. If you didn't, the next time you connect, it would be changed back to the main series column.

[Lofwyr23]

Hm, if it isn'T that easy to do, it's no big deal, I'll use shelves then (although those are meh at best).

[Jellby]

I keep getting this problem where, when I re-upload a book I'm reading with a new version (no change in structure or metadata, just some fixes in the text, maybe styles and the odd HTML tag), the book is reset by the Kobo: it disappears from the home screen, and is listed as unread, with reading location gone.

Could this be due to the use or misuse of this plugin? I usually click on "Update metadata in the device" right after sending the update, maybe I should only do it after the device has processed the books (as with new uploads)? Is there anything else I can do to try to avoid the problem? Could the plugin perform some king of check to warn me, something like "Hey, this book matches one in the device, but if you upload it the device will probably re-process it"?

[davidfor]

I do frequently replace books with small changes just like you describe. When doing this, the driver will update the size of the file in the devices database. This is done to prevent the device treating it as a new book.

But, I rarely update the metadata after sending. I never think of it and do it in a big batch sometime later. I know I have done this in the past with no problem. And as I had just fixed some spelling in the book I was reading, I tried it. The book went, the metadata was updated and it wasn't reprocessed.

[Jellby]

Well, I seem to run into this kind of problem almost every time I connect the reader (every 2-4 weeks), and updating the library has become something that takes me between 2 hours and a whole afternoon, including making a backup before starting, several failed attempts, restoring the backup, sending books one by one, etc.

[davidfor]

Is that happening every time? If so, can you run calibre in debug mode then next time you plan to do it? The logs could give a clue as to what is happening. Also, check the database in case there are any errors when starting.

How long between sending the book and updating the metadata? If you did the update while the send job is running, that might cause problems. You would be fairly unlucky to consistently hit an error, but the two updates could clash. If it looks like that is the problem, I can add a check for running device jobs to prevent the other updates happening at the same time.

[Jellby]

Not every time, as I have been able, so far, to always perform the updates at the end. I've now adopted the habit of making a backup and checking the database is OK ("pragma integrity_check" with sqlitebrowser) before connecting to calibre. I've also checked the card and internal memory with dosfsck (no errors).

I always wait for the transfer to finish before updating the metadata, and for any task to finish before starting the next one, for what is worth. But I've had the same issue just with transferring books, no metadata update, so that's not the culprit. It seems to be a random event, I can't find any consistent cause, but there are a few variables that could be behind the issue and I don't always keep in mind (connecting from pure nickel, or from nickel within KSM; launching nickel from KSM right after booting, or after exiting a previous nickel session; mounting manually or letting calibre do it; trying to force a sync before disconnecting the device, or just clicking on "disconnect device" right after the transfer finishes...).

I've even had this happening:

- Connect the device to calibre: calibre reads the list of books, the plugin makes database backup. All fine.
- Select a few books, click on the plugin's "update database" menu, and bang! error message about a corrupted database.
- After restoring a previous backup and repeating the same actions, no problem at all.

I'm thinking there could be a problem with the cable or USB port, but I haven't had any issue when accessing the device outside calibre.

[Francois_C]

I also do frequent replaces of a book with an improved version, not twice a month but often twice a day, but I didn't even notice that Calibre had been mended since the time it created a new entry every time you would upload a book without deleting the previous one! So I still remove the previous version before updating with the new one, and I did not think to regret to see it considered new: I only kept in mind the chapter I was reading.
But I never update the metadata either. If the cover has been changed and the old one still appears, I switch off the Kobo off and the new one is displayed at restart.
I never experienced a 2-hour wait (we are avid readers and own three different Kobos), except when I start the Kobo with 200 new books on the SD card at once.
The simplest way to update a slightly modified ebook and keep bookmarks and annotations is to overwrite it directly from the HD to the Kobo without using Calibre nor Kobo utilities! The only risk is that bookmarks may become slightly wrong.

[davidfor]

Originally Posted by Francois_C

I also do frequent replaces of a book with an improved version, not twice a month but often twice a day, but I didn't even notice that Calibre had been mended since the time it created a new entry every time you would upload a book without deleting the previous one!

When was it doing this? It should never have done this unless you changed the metadata that is used in the save template. Or you send a different format.

So I still remove the previous version before updating with the new one, and I did not think to regret to see it considered new: I only kept in mind the chapter I was reading.
But I never update the metadata either. If the cover has been changed and the old one still appears, I switch off the Kobo off and the new one is displayed at restart.
I never experienced a 2-hour wait (we are avid readers and own three different Kobos), except when I start the Kobo with 200 new books on the SD card at once.
The simplest way to update a slightly modified ebook and keep bookmarks and annotations is to overwrite it directly from the HD to the Kobo without using Calibre nor Kobo utilities! The only risk is that bookmarks may become slightly wrong.

That should only work if the file size of the book has not changed. Unless you are on a very old firmware. I hadn't checked for a while, but a test right now showed that this is still happening. Calibre avoids this by updating the file size in the database so the book doesn't get reprocessed and hence the bookmarks and reading status are not lost.

[Jellby]

Originally Posted by Francois_C

I never experienced a 2-hour wait

I didn't mean a 2-hour wait, but the process of repeatedly trying to update the books, restoring earlier backups and trying again until everything is fine often takes me a few hours.

[JimmXinu]

Originally Posted by davidfor

When you disconnect the Kobo ereader from the PC, it looks for changes in the books. Any new books will be processed into the internal database and hence the library. Any books that have been deleted, will have nearly all evidence of them removed from the database. Any changed books will be removed from the database and then processed as a new book.

The behaviour of the last one has changed over time. Early firmware didn't do anything. Current firmware does the above.

For this, what the firmware considers as changed is the file size. That means you could unpack an epub and repack it without any other changes and get a different size. Or open it, change one letter somewhere in the text and repack it and the device will see it as a new book.

Because of this, when you send a book to a Kobo device that is already on the device, the driver will update the files size stored in the database. I chose to do this in the driver because that kept older behaviour, and my impression from discussion here is that how people wanted it to work. And to be honest, it is how I wanted it to work. I frequently edit books I read to fix spelling and layout. As long as I don't fiddle with the internal file structure or the ToC, everything will work OK.

If you make changes to the ToC or the internal file structure, the book can probably still be read, but, there might be problems. One of the things that the processing of the book does is to store the ToC in the database. If you open the ToC when reading the book, the database is used. If it doesn't match the book, selecting an entry will probably take you to the start of the book.

If the file structure changes, again the book can probably be read. But, the ToC won't work and the reading position won't be stored properly. That means it will probably reopen to the start of the book.

None of this has anything to do with the utilities plugin. It's how the device works and how the driver works.

Sorry for quoting an old post, but I want to check if this is still how it works--it certainly matches the behavior I'm seeing.

I just bought a Kobo Glo HD that I want to use with fanfiction books from FanFicFare. My use pattern is to frequently add new chapters (and TOC entries) to existing books and then update them on the device. Often several times a day.

The post I quoted says that the device driver deliberately fudges the file size in the Kobo's DB to prevent it from reprocessing updated books--which pretty much prevents me from reading new chapters as they don't appear in the TOC and the Kobo crashes if I page into them and then try to view the TOC.

Alternatively, it sounds like even if I patched my copy of the driver to not fudge the file size, the Kobo would discard the reading position and any bookmarks when it reprocessed the book.

Am I missing anything that may help me here?

[davidfor]

Originally Posted by JimmXinu

Sorry for quoting an old post, but I want to check if this is still how it works--it certainly matches the behavior I'm seeing.

I just bought a Kobo Glo HD that I want to use with fanfiction books from FanFicFare. My use pattern is to frequently add new chapters (and TOC entries) to existing books and then update them on the device. Often several times a day.

The post I quoted says that the device driver deliberately fudges the file size in the Kobo's DB to prevent it from reprocessing updated books--which pretty much prevents me from reading new chapters as they don't appear in the TOC and the Kobo crashes if I page into them and then try to view the TOC.

Alternatively, it sounds like even if I patched my copy of the driver to not fudge the file size, the Kobo would discard the reading position and any bookmarks when it reprocessed the book.

Am I missing anything that may help me here?

No, you are not missing anything. The behaviour of either the Kobo devices or the driver has not changed. The driver fiddling with the file size in the database is a compromise and I think fits most scenarios for resending a book. Unfortunately, the one scenario it is a bad fit for is fanfiction.

But, the way the device would handle this is just as bad. You can get around this by having the utilities store the book progress, send the updated version, disconnect and let the device process the book and then reconnect and restore the progress. This will work, but is a pain.

I suppose what I should really do is clean up and release some code I have to fix the ToC in this type of book. It compares the ToCs the different places (the book in the calibre library, in the device and in the database) and reports the count of entries in each place. Then it can add the extra entries if they are missing from the database. It is fairly dumb, simply comparing counts and assumes the extra entries are at the end of the ToC. But, it works with the FanFicFare downloaded books I have tried. And before anyone asks, it only works for epubs.

Oh, the crash is something new. In the past, this would all work except that the ToC wouldn't match the book. But, it has been a while since I tried opening the ToC in this situation. I'll have a try later, but could you look and see if there are any logs on the device? These will be in the ".kobo" directory with the name "stack_xx.log". They contain a stack dump and can help Kobo debug problems like this. I should get one when I reproduce this and I'll report it via the beta list, but a report from another user is good.

And completely off topic, the stats on my Glo HD shows me as having read for 999.0 hours!