Aura HD Database update freezes

[Mellobob]

A few weeks ago I noticed that I'd sideloaded some books with not-so-correct formatting to my Kobo Aura HD. Okay, no problem ... redid the books with Calibre. Plugged in my Kobo and installed the books again.

Now, when I unplugged the "Updating Database" message froze at 72%. I turned my kobo off/on and all appeared to be fine. Well, not quite ... the book I was reading was one of the updated books and it didn't display the new formatting. So, I deleted the 5 books, connected to Calibre and uploaded again. This time all was just fine.

Since then I've installed other books and the "updating database" message was fine. No problems.

Yesterday, I sideloaded some more books. This time the message halted at 90%. Again, I needed to turn the kobo off/on. And, again, all seems to be okay.

Any idea what's going on? Possible solution?

[MacEachaidh]

I've noticed the "updating" cycle often doesn't complete, on either my Aura HD or Touch 905B, since upgrading to 3.18. It may be a database issue, or it may be a firmware one. I haven't personally bothered enough about it yet to go through the hassle of factory resetting, upgrading, reinstalling and recreating all my Collections memberships -- the true pain of Kobo databases apparently being so fragile. If the next firmware doesn't address this, though, I probably will have to. In the meantime, I've found (once I've left the device long enough to be sure it's not just chewing on a particularly meaty cover image, and in fact isn't going to refresh its own display), that pressing the Home button on the Touch or sliding the sleep slider on the Aura HD is enough to bring the device back to itself.

As for updating with new copies of a book sideloaded over an older copy, I've never found that to work. The only thing that works imo is to delete the unwanted copy, reboot the device, and then connect it and sideload the new one. (I reboot because the device often doesn't seem to delete the record of the book until you do, and I've had instances of simply deleting the book and connecting, then copying the new one over, only to find the old cover had been retained, even though it doesn't actually exist in the new copy. So I always shut the device down after a deletion. In fact, if the book I'm deleting has annotations or bookmarks, I find it necessary to delete them, then reboot, before deleting the actual book, or else they're retained and applied to the new copy.)

[GeoffR]

If it looks like it has frozen, you might just try tapping the screen and see if that helps before resorting to a pinhole reset.

I had never seen this happen until firmware 3.18.0, but occassionally the processing screen will appear to be stuck when in fact it has actually finished processing and is just waiting for a tap before it will display the homescreen.

[compurandom]

Originally Posted by MacEachaidh

I've noticed the "updating" cycle often doesn't complete,

I'd guess this is a corrupted book or a database issue. From the rest of your description, I'd lean towards a database issue from resetting the device while hung. Maybe the first time it was a bad book or a mis-synchronized book or a huge cover.

I've had good luck repairing the database with sqlite3, as described elsewhere in this forum. That's a lot easier than a factory reset, and at the very least, sqlite3 can tell you if the database is damaged or not.

But even if sqlite3 does successfully repair the database, there can be metadata corruption left over, which won't hurt much, but might leave you with undeletable non-existent books and other strangeness. I've wondered how hard it would be to write something to scour the database of problems like that.

Originally Posted by MacEachaidh

[...] that pressing the Home button on the Touch or sliding the sleep slider on the Aura HD is enough to bring the device back to itself.

I've also found that occasionally it isn't hung, just the screen didn't update after it finished indexing the new books.

Originally Posted by MacEachaidh

As for updating with new copies of a book sideloaded over an older copy, I've never found that to work. The only thing that works imo is to delete the unwanted copy, reboot the device,

A good shortcut is to delete the book from within the device itself (long press on the book from the library or a collection and select delete). If you delete it that way, it deletes the metadata for the book from your library, and then side loading can be done without that extra reboot.

When you side load over top of a book that has already been indexed, the metadata from the old book (which includes TOC, book length, offset to pages, cover, and some other stuff) stays in the database, and all kinds of strange things happen when you try to read the new book, including hangs and crashes and inability to correctly jump to pages or the end of the book.

[davidfor]

Originally Posted by MacEachaidh

As for updating with new copies of a book sideloaded over an older copy, I've never found that to work. The only thing that works imo is to delete the unwanted copy, reboot the device, and then connect it and sideload the new one. (I reboot because the device often doesn't seem to delete the record of the book until you do, and I've had instances of simply deleting the book and connecting, then copying the new one over, only to find the old cover had been retained, even though it doesn't actually exist in the new copy. So I always shut the device down after a deletion. In fact, if the book I'm deleting has annotations or bookmarks, I find it necessary to delete them, then reboot, before deleting the actual book, or else they're retained and applied to the new copy.)

All of which are known problems that have been discussed here plenty of times.

As well as on disk, the covers are cached in memory. Replacing the book doesn't force the on disk image or the memory cached version to be replaced. And unfortunately, deleting the book only deletes the on disk image. The memory cached version isn't cleared. Restarting the device clears the cache and reloads/regenerates the disk version.

The annotations problem is caused by the RMSDK saving copies of the annotations to disk as well as the Kobo also putting it in the database. And again, deleting the book doesn't remove the file, but it does remove the entries from the database. Putting a book with the same file path on the device will reuse this file. The first time you open the book, the annotations are loaded from the file and are put into the database. This file is also the reason that the chapter reference for annotations in epubs and PDFs is messed up when you reopen a them. These files are in the "Digital Editions\Annotations" directory. I sometimes clean them up, but at least one person here has replaced this directory with a file so that they cannot be created.

From memory, you don't use calibre. In this case, if you replace an existing file with a new one, and the file size changes, the processing will remove the book details from the database an delete the cover images before processing the book as a new book. This doesn't remove the annotations file or remove the covers from the memory cache. The calibre driver updates the file size stored in the database so that the book doesn't get reprocessed. That means you don't get any updated metadata or ToC, but you also don't lose the collections, reading status and bookmarks.