Help please - updating to 4.4.9344 has really scrambled my Aura H20!

[davidfor]

Originally Posted by lealla

I will try to manually install the latest update - just to confirm, it doesn't matter that my device hasn't registered it automatically yet? I don't want to mess up the device further by installing an incorrect update.

It is reasonably safe to manually install the firmware. Kobo rolls it out slowly to start in case there are problems and to balance the load in their servers. This has been out long enough that there does not appear to be any show-stoppers.

If that doesn't help, I'll try a factory reset - I'm sad to hear I won't be able to save my information / account settings, but it's more important to me to have the device back up and running. I hope kobo somehow manages to implement a user settings back up/restore option someday.

Kobo sync details of all purchased books to their server. They don't sync anything for sideloaded books. This is a decision they have made. You can use calibre and my Kobo Utilities to sync reading status for any book, but there is nothing particularly good for annotations.

I understand that it wasn't the firmware upgrade that scrambled the device, however it seems to have been a trigger for the corruption to either occur or for a previous corruption to reveal itself. I didn't mean to suggest that Kobo is releasing corrupting firmware, rather it was during the timing of updating that my device seems to have become corrupted.

I hadn't updated my kobo for a long time, due to having changed wifi settings a long time ago and neglecting to update the e-reader settings. Perhaps the update was just too big a change and exposed the corruption.

@davidfor - I get the normal "error communicating with device" in Calibre, however once choosing OK, the device shows up as connected, though with apparently no books on the device.

There is no such thing as a 'normal "error communicating with device" in Calibre'. The only one that comes close to that is the message displayed about unsupported firmware. If that is the message you are seeing, it includes instructions to allow calibre to handle the newer firmware. But, the update to calibre to include 4.4.9344 as a supported firmware version came out at most two weeks after that firmware was released. You should upgrade to the latest version of calibre. Though if you wait for this weeks release, it will include the 4.5.x firmware.

If you are getting any other message, it is an error. Click the copy button on the message to copy the error details and post that. I'll have a look and see what can be done.

[davidfor]

Originally Posted by lealla

Thanks DNSB - I suspected as much, but was hoping that an update might just refresh everything and sort itself out. I can see that's unlikely given the likelyhood of a corrupted database.

I have downloaded the link you supplied but am wondering if I should factory reset first, and then install the update - or install update and then factory reset. Which order do you think makes most sense?

A factory reset is usually overkill. Signing out of the account is usually enough. With that, it takes you through the setup process, and then reloads any sideloaded books. If you do a factory reset, you will need to copy all those books over again. But, if you want to start with a clean slate, then do a factory reset.

For installing the update, don't bother if you are doing a factory reset. That will set the firmware back to what came out of the factory. Then during the setup, it will download the latest available to you. That will probably be 4.4.9344, but, you could be getting 4.5.9587.

The signout/in will also check for the latest firmware during the setup. If you do that, installing 4.5.9587 first won't harm.

[El Duderino]

Originally Posted by DNSB

Are you thinking of when Kobo's 4.x firmware switched to using journaling with their SQLite database? OTOH, the restart after a firmware install can and will expose a corrupted database.

Originally Posted by davidfor

As DNSB said, Kobo changed some options in the database starting with the 3.20.x firmware. I don't know if they changed the version of the SQLite library they used at the same time. It's more likely a corruption would happen when migrating to either the 4.1 or 4.2 firmware. In one of those, they changed how some data was stored and, from memory, the migration updated all of that. That would affect anyone coming from earlier firmware to a one of those or later. The main issue with this would be the amount of data updated. For anyone with a lot of books, and had done a lot of reading, then there would have been a lot of updates. The chance of something going wrong, increases with the number of changes needed.

Absolutely what I was referring to. I had just side-loaded my whole collection and spent hours manually sorting them on the device into bookshelves before the corruption occurred while updating to 4.x. I couldn't restore the database backup because it was a different SQLite version, so ended up performing a factory reset.

@ lealla, If I can offer any advice, resolve to manage every aspect of your Kobo library in Calibre including bookshelves with regular external backups, so restoration isn't a painful process. A healthy respect for the Kobo Utilities plugin by not pushing every button to see what it does is also advised.

Best of luck on this.

[lealla]

Originally Posted by davidfor

If you are getting any other message, it is an error. Click the copy button on the message to copy the error details and post that. I'll have a look and see what can be done.

Apologies for not being more specific - I was assuming that my error message was just an ordinary 'can't find the device' kind of thing, but looking at the details shows:

CorruptError: database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 88, in run
File "site-packages\calibre\gui2\device.py", line 511, in _books
File "site-packages\calibre\devices\kobo\driver.py", line 1836, in books
File "c:\t\t\apsw-j6gj96\apsw-3.13.0-r1\src\cursor.c", line 236, in resetcursor
CorruptError: CorruptError: database disk image is malformed

So, although I don't know for sure, seems like everything is pointing to corruption on the kobo side.

Thank you for the extra clarification on factory reset VS signing out. I understand a lot better what my options are now.

Here's what I will try to do:

I will install the latest update manually, then I will sign out/sign back into my account on the device.

If that doesn't work, and everything is still corrupted, I will do a factory reset and start everything again from scratch.

Hopefully I won't have to do the factory reset, but I know that it's always an option if step 1 fails.

Thanks again for taking the time to explain everything, I feel much more confident about what to do!

[lealla]

Originally Posted by El Duderino

Absolutely what I was referring to. I had just side-loaded my whole collection and spent hours manually sorting them on the device into bookshelves before the corruption occurred while updating to 4.x. I couldn't restore the database backup because it was a different SQLite version, so ended up performing a factory reset.

@ lealla, If I can offer any advice, resolve to manage every aspect of your Kobo library in Calibre including bookshelves with regular external backups, so restoration isn't a painful process. A healthy respect for the Kobo Utilities plugin by not pushing every button to see what it does is also advised.

Best of luck on this.

Thank you for the tips! I used some of the advice here on mobileread about automating the collections via the tag tools in Calibre, so although it will be time consuming to restore, it won't be too agonizing! As I said, I'm sad to lose my annotations, but not as sad as I am without my ereader right now!
However, I feel your pain because I used to have one of the older kindles and had spent hours sorting my books into groups, only to accidentally ruin everything when it became damaged. I try to be very careful now!

(I have also learned too not to be too button happy with plugins - especially after it took me ages to properly catalogue everything after my first gung-ho attempts!)

[davidfor]

Originally Posted by El Duderino

Absolutely what I was referring to. I had just side-loaded my whole collection and spent hours manually sorting them on the device into bookshelves before the corruption occurred while updating to 4.x. I couldn't restore the database backup because it was a different SQLite version, so ended up performing a factory reset.

From memory, the device checks the version of the database each time it restarts. Restoring one from a version or two ago should be OK. Plus, the database version doesn't change with every firmware version. I'll have to do some tests to check.

@ lealla, If I can offer any advice, resolve to manage every aspect of your Kobo library in Calibre including bookshelves with regular external backups, so restoration isn't a painful process. A healthy respect for the Kobo Utilities plugin by not pushing every button to see what it does is also advised.

What did you press? I thought everything was reasonably well explained. It's not hard to do something like reset the reading status of a lot of books, but, you have to do a bit of work to achieve that.

[El Duderino]

Originally Posted by lealla

Thank you for the tips! I used some of the advice here on mobileread about automating the collections via the tag tools in Calibre, so although it will be time consuming to restore, it won't be too agonizing! As I said, I'm sad to lose my annotations, but not as sad as I am without my ereader right now!

However, I feel your pain because I used to have one of the older kindles and had spent hours sorting my books into groups, only to accidentally ruin everything when it became damaged. I try to be very careful now!

(I have also learned too not to be too button happy with plugins - especially after it took me ages to properly catalogue everything after my first gung-ho attempts!)

Yes, very unfortunately re: your lost annotations. I've also experienced losing annotations, but had only created them for 3 books before the loss, and figured the process too cumbersome to bother anymore on Kobo's sluggish firmware.

You might want to look into KOReader. I hear you can connect EverNote, so maybe your e-reader annotations can live across multiple devices and avoid losing them again? Plus the benefits of PDF reflow seem outstanding. I'm planning on installing KOReader after I get firmware 4.5 OTA.

Indeed, sorting books on the device is now a thing of the past, good riddance!

Originally Posted by davidfor

From memory, the device checks the version of the database each time it restarts. Restoring one from a version or two ago should be OK. Plus, the database version doesn't change with every firmware version. I'll have to do some tests to check.

What did you press? I thought everything was reasonably well explained. It's not hard to do something like reset the reading status of a lot of books, but, you have to do a bit of work to achieve that.

Thanks for trying to figure this out davidfor. Honestly, it's a hazy memory, but I'm almost positive my database got corrupted during the update/reboot phase, and was unable to restore the database due to an incompatibility issue brought about by the upgrade process. It may have been the move to journaling...

Kobo Utilities offers a good surface explanation of what a function does, even to a newb like me, but I wish there was a warning dialogue for potential consequences of some actions.

I may have clicked "Refresh the list of books on the device", or "Update metadata in device library". These sound pretty innocuous, but I really have no idea of negative consequences by clicking those. Now that I manage everything in Calibre, there's less fear of a SNAFU.

These days I simply avoid everything in Kobo Utilities except to backup the database. Maybe one day I'll find the Wiki...

[davidfor]

Originally Posted by El Duderino

Thanks for trying to figure this out davidfor. Honestly, it's a hazy memory, but I'm almost positive my database got corrupted during the update/reboot phase, and was unable to restore the database due to an incompatibility issue brought about by the upgrade process. It may have been the move to journaling...

It should have worked. As a test, I just put a backup from 4.3 on my Touch. It was migrated with no problem. The change in the journal mode shouldn't is unlikely to cause a direct problem. It changes how the database is opened from then on. But, it's a bit hard to test and prove.

Kobo Utilities offers a good surface explanation of what a function does, even to a newb like me, but I wish there was a warning dialogue for potential consequences of some actions.

I may have clicked "Refresh the list of books on the device", or "Update metadata in device library". These sound pretty innocuous, but I really have no idea of negative consequences by clicking those. Now that I manage everything in Calibre, there's less fear of a SNAFU.

The first of those is simply to reread the device and get the book list again. The latter is to update the metadata on the device from the calibre library. The latter can cause a problem but only if the metadata is wrong. Or you use the options to change the reading status when you don't really want to.

These days I simply avoid everything in Kobo Utilities except to backup the database. Maybe one day I'll find the Wiki...

Well, you could try the help in the top right corner of each dialog.

[El Duderino]

Thanks again davidfor, you've got me on the ropes and now I have no idea what could have caused my issue, if it's likely not the upgrade.

Did you create Kobo Utilities? If so, bravo sir, it's an indispensable utility for many happy users, and apologies for my ignorance!

I was confused by your instruction to "try the help in the top right corner of each dialog". When I clicked the drop-down, I don't see a help link next to each item, then realized you might have meant clicking the Kobo Utilities icon/button instead of the drop-down, no? If so, I see it top right of "Kobo Utilities Options"... Funny, I've never clicked it, always used the drop-down.

Some people's kids.

[davidfor]

Originally Posted by El Duderino

Thanks again davidfor, you've got me on the ropes and now I have no idea what could have caused my issue, if it's likely not the upgrade.

From what I can see of the upgrade process, a database corruption during that is only a slightly more likely than at other times. Most changes to the database are simply adding a column to the table or a new table. Very few have real data changes. Yes, there is work happening, but very controlled.

Did you create Kobo Utilities? If so, bravo sir, it's an indispensable utility for many happy users, and apologies for my ignorance!

Yes, that's mine. Glad you like it.

I was confused by your instruction to "try the help in the top right corner of each dialog". When I clicked the drop-down, I don't see a help link next to each item, then realized you might have meant clicking the Kobo Utilities icon/button instead of the drop-down, no? If so, I see it top right of "Kobo Utilities Options"... Funny, I've never clicked it, always used the drop-down.

All the dialogs should have a help link in the top right corner. It's not on the menu as it broke things on Macs and I couldn't work out how to fix it. I should revisit that.

[El Duderino]

Originally Posted by davidfor

All the dialogs should have a help link in the top right corner. It's not on the menu as it broke things on Macs and I couldn't work out how to fix it. I should revisit that.

Wish I was a programmer to offer the answer, but whether you figure it out or not you're still

I'll dive into the help and get familiar with it.

[DNSB]

Originally Posted by lealla

Thanks DNSB - I suspected as much, but was hoping that an update might just refresh everything and sort itself out. I can see that's unlikely given the likelyhood of a corrupted database.

I have downloaded the link you supplied but am wondering if I should factory reset first, and then install the update - or install update and then factory reset. Which order do you think makes most sense?

Thanks for your help

My personal choice would be a signout/signin first. If that doesn't clear up the database corruption, then do the factory reset. After the signout/signin or factory reset, then do the firmware update on a clean database.

[lealla]

Thanks to everyone here!
To conclude - I did a log in/log out, followed by a manual update.
Unfortunately, none of it worked, so I had to do a factory reset afterall.
Still, that seems to have fixed the corruption, and I have recopied all my books across.
I also took the opportunity to patch the firmware, something I'd been meaning to do for a while.

I am having a couple of issues - I don't know if they are due to the patch or due to the new firmware, or due to something else. I'll do some research and you might see me back here again with new questions, or asking over in the developers corner