Aura H2O Critical Errors on DB restore

[youngmax]

Hi all, been a lurker for years, great forum here. I've searched a lot but haven't been able to find an answer for my issue.

I had a Kobo H2O running 3.19.5613 with Metazoa's patch. This device was working great with my sideloaded books until it developed a hardware problem. I used the Kindle Utilities calibre plugin to backup the DB and reading positions, as well as copying the entire contents of the device's storage to a folder on my computer. Then I got a warranty refund.

Kobo sent me a new H2O, and although I haven't been able to find an exact procedure for restoring the database (davidfor references it in this thread https://www.mobileread.com/forums/sho...d.php?t=254512 but doesn't actually describe what to do) and several things I tried (replacing the koboreader.sqlite, replacing the kobo folder, setting up the device with my kobo login again and replacing the entire storage contents with my backup copy) gave me a version of the following error message:

"Critical Error: any highlights, notes, or reading progress since your last sync on 3/17/16 will be lost." There was a mandatory button to sign out of the device, which would then reset.

Note that at this point I was having a brainfart and hadn't patched the firmware on the new device. Not sure if that was a problem. In any event, I was able to solve this by telling the reset device I had no wifi, connecting to my computer, and copying the entire storage backup to the kobo. This worked, and I had a functioning device. I continued to back up my reading progress along with the database using Kobo Utilities.

That brings us to now, when I tried to sync the kobo using the main page button. Bad idea. That updated the firmware, but also brought the return of the Critical Error, dumping me back to the Set up Kobo screen and deleting all my reading progress. (However, my books are still present.) Despite the error message claiming I will have reading progress/annotations to 3/17/16, even that is gone. Help?

I've tried copying over the .kobo folder from my 'old' backup, the koboreader.sqlite file from that backup, and the .kobo folder from my most recent backup in the Calibre library's Kobo Backups folder (each of these done after a Critical Error forced sign out). I'm getting critical errors on each - most recently, using the Calibre backup, the slightly different "Critical Error: Your eReader hasn't synchronized with the server yet. Any highlights, notes, or reading progress you've made on the device will be lost."

Any help is appreciated. I just want my reading positions back, I'll probably never update my firmware again based on how much trouble this has caused. Thanks for reading my wall of text.

[davidfor]

It sounds like you are doing all the right things, but, it is not guaranteed to work. And is definitely not supported by Kobo. There is at least one entry in the database that is generated based on the device. Because of this, moving the database from one device to another, might not work. Some people have done this with no problems, for others it doesn't work. And something was changed in the last firmware version or two that made it less likely to work.

What might work is to setup the new device and then extract the data in the users table out of the database. Then replace the database with the backup and replace the entries in the users table with the extracted data. I haven't had the need try this, so don't know if it will work or not.

Whether you have patched the firmware or not won't affect this. I don't remember any patches that affect the login process.

As to there not being a automatic method to restore the backup that the Kobo Utilities plugin produces, that's because it is really just a file copy to replace the database on the device with the backup. But, it isn't intended for a circumstance like this. It is intended to be used to replace a corrupt database on the device it came from. The closer the match of the backup database with what is on the device, the more likely it is that it will work.

[GeoffR]

Originally Posted by youngmax

I had a Kobo H2O running 3.19.5613 with Metazoa's patch.

The old deviice was still on 3.19.5613 for some reason, but I assume the new device would be on the latest version 3.19.5761?

If so then you might try downgrading to 3.19.5613, restoring the backup from the old device, and then allowing the device to upgrade to 3.19.5761.

I've never done a restore from backup myself though, and if it wasn't for others reporting that they had done it successfully I wouldn't have guessed it was possible to restore a backup to a different device.

[PeterT]

Was your device registered with Kobo or had you bypassed it by inserting data into the database?

[youngmax]

Thanks for the help, here are some more answers.

Originally Posted by GeoffR

The old deviice was still on 3.19.5613 for some reason, but I assume the new device would be on the latest version 3.19.5761?

If so then you might try downgrading to 3.19.5613, restoring the backup from the old device, and then allowing the device to upgrade to 3.19.5761.

I don't actually have a list of when the various firmware updates came out, but the old device was using the most recent firmware as of March, and I got the new device shipped on May 3rd. I don't recall the notification of an updated Kobo Utilities plugin in that time (it just happened in the past two weeks, right?) so I assumed they were both on 3.19.5613. The DB restore did eventually work on that version, for weeks, until I allowed the device to upgrade to the latest version which immediately broke it.

Unless there is a way to transfer reading positions to a new database (I'd survive losing my annotations), it sounds like a firmware downgrade and reattempting the DB copy is my best approach. Does this require cracking open the case?

Originally Posted by PeterT

Was your device registered with Kobo or had you bypassed it by inserting data into the database?

The original device was registered normally. The replacement was registered with Kobo at one point, but was only working after I reset the device and replaced the entire DB with that from the original, so I suppose that means bypassed. In retrospect it was pretty dumb of me to update the firmware at all, I should have just been happy with the working setup I had.

I'll also download an sqlite tool and start poking around the DB.

[youngmax]

Okay, I must have been confused about what firmware version I was using. There hasn't been a firmware update since January? Why did I just receive a notification about the new firmware version this week - is it because I updated Calibre to 2.57.1 this week, but had been using 2.55.0 since April?

I wonder if this is the source of the problem. I guess I'm not sure what versions of the firmware I was using at all, I was basing all those numbers on the presumption that the firmware update was this week, and until then I had the most recent firmware. Apparently not true? Is there a way to pull that information from the backed up DB's? I wonder if something crazy happened like I was using an old firmware version on the original device (not sure how, since it displayed a successful wifi sync in february) and that DB was copied to a device that was new in box with a different version, possibly the latest, but tried to upgrade itself incorrectly because the device firmware and DB were not in agreement?

There's probably a good reason David suggested to copy the user info from a new DB to the old backup DB, but with this much confusion I wonder if it might not be better to start a new DB and copy the reading positions over. Of course, if something mysterious in the firmware is rejecting data from other devices, this might not work either...

[GeoffR]

Originally Posted by youngmax

I don't actually have a list of when the various firmware updates came out, but the old device was using the most recent firmware as of March, and I got the new device shipped on May 3rd. I don't recall the notification of an updated Kobo Utilities plugin in that time (it just happened in the past two weeks, right?) so I assumed they were both on 3.19.5613. The DB restore did eventually work on that version, for weeks, until I allowed the device to upgrade to the latest version which immediately broke it.

Firmware 3.19.5613 was out by early December 2015, version 3.19.5761 was out by.early January 2016.

Is it possible the new device was runing 3.19.5761 but you applied a patch for 3.19.5613? I don't know what would happen in that case, usually the device wouldn't boot, but if it did there would certainly be problems at some point.

Unless there is a way to transfer reading positions to a new database (I'd survive losing my annotations), it sounds like a firmware downgrade and reattempting the DB copy is my best approach. Does this require cracking open the case?

I haven't done a downgrade for a very long time so I don't know for sure. It should be possible to downgrade without opening the case, but it might depend on which firmware was on the device to start with. If the device comes with an older firmware version then it is easy enough to upgrade from there to another later version instead of the very latest version. It is not always possible to cleanly downgrade to an earlier version than the one the device comes with from the factory. Others have reported that it works, but you don't know for sure if it has really worked or if there was a problem that you just haven't come across yet.

[GeoffR]

You can find the firmware version in the file .kobo/version on the device, if you backed up this file with the rest of the data then you can check it to see which version the old device was running.

[davidfor]

Originally Posted by youngmax

Okay, I must have been confused about what firmware version I was using. There hasn't been a firmware update since January? Why did I just receive a notification about the new firmware version this week - is it because I updated Calibre to 2.57.1 this week, but had been using 2.55.0 since April?

What sort of notification? One the device? Or in calibre? Calibre displays a warning if the firmware version on the device is later than the firmware version programmed in the driver. As there hasn't been a firmware version released since January, you would need to be using a version of calibre from before this for this to happen. The Kobo Utilities plugin has a firmware check that can be run automatically. This will trigger if you haven't upgraded the device.

I wonder if this is the source of the problem. I guess I'm not sure what versions of the firmware I was using at all, I was basing all those numbers on the presumption that the firmware update was this week, and until then I had the most recent firmware. Apparently not true?

If you were syncing with the Kobo server, you had the latest version. The sync always checks for an update and downloads and installs it before the rest of the sync. You can't stop this. Unless there is a patch that turns it of.

Is there a way to pull that information from the backed up DB's?

Not really. The table "DBVersion" has the database version. But, that doesn't change for every firmware version. If you look, the value will probably be 125.

I wonder if something crazy happened like I was using an old firmware version on the original device (not sure how, since it displayed a successful wifi sync in february) and that DB was copied to a device that was new in box with a different version, possibly the latest, but tried to upgrade itself incorrectly because the device firmware and DB were not in agreement?

There's probably a good reason David suggested to copy the user info from a new DB to the old backup DB, but with this much confusion I wonder if it might not be better to start a new DB and copy the reading positions over. Of course, if something mysterious in the firmware is rejecting data from other devices, this might not work either...

No, the reason I suggested copying the user info is that there is at least one key in the table that is generated on the device using information that is unique to the device. The firmware is probably validating this information against the device. Or it is sending it to the Kobo server to validate during the sync. I don't know exactly what is happening as I don't have access to the source. It is an informed guess based on inspection of the tables and the behaviour people are seeing.

If the point of all this is to get the reading statuses back on the device, then there is a way to do it. Use the Kobo Utilities plugin "Store/Restore reading status" function (can't remember exactly what I used for the menu). The quick and dirty steps are:

1. Backup you calibre library (just in case).
2. Configure the store/restore custom columns.
3. Close calibre.
4. Connect the device.
5. Restore your full backup of the old device. This is everything: the database, the books, everything.
6. DO NOT DISCONNECT THE DEVICE.
7. Start calibre.
8. When the connect jobs are finished, select all the books that are on the device and do a store of the reading status.
9. Disconnect the device and let it redo the setup.
10. Reconnect the device.
11. Select all the books on the device and do a restore of the reading status.
12. Disconnect the device and restart it.

If all goes right, then you will have the reading status set for all the books on the device. But, you won't have the reading stats or the bookmarks. If you use calibre to manage the collections, they should be back.

[youngmax]

Originally Posted by davidfor

What sort of notification? One the device? Or in calibre?

Originally Posted by GeoffR

Is it possible the new device was runing 3.19.5761 but you applied a patch for 3.19.5613?

Although the notification about the firmware was originally in Calibre (pretty sure it was the Kobo Utilities notification of an available update for the device, though not 100%), I disconnected the device and updated by touching the sync tile on the home screen. Unfortunately the firmware status of the replacement device is unknown since it could have been sitting in its box since November for all I know. I don't have a backup of the new device's .kobo folder before I overwrote it.

Possibly of note, while setting up the new device I first tried logging into my kobo account and got to the home screen. Would that have been enough to trigger a check for available firmware updates? If so, there seems to have been no reason for the device to download an update, which I saw it doing. If a bare initial login without hitting the sync tile even once isn't enough to trigger an update check, it sounds like I was running a database from the most recent firmware on a 'new' device with older firmware.

Here's the contents of the version file from my backups of the original device:

N250553006736,2.6.35.3,3.19.5761,2.6.35.3,2.6.35.3,00000000-0000-0000-0000-000000000370

Here's the version file off the new device the day I received it, but I'm not sure if this is from before or after I overwrote all the files:

N2505B3901229,2.6.35.3,3.19.5761,2.6.35.3,2.6.35.3,00000000-0000-0000-0000-000000000370

Here it is from two days ago, just before I hit the sync tile and everything went to hell.

N2505B3901229,2.6.35.3,3.16.17,2.6.35.3,2.6.35.3,00000000-0000-0000-0000-000000000370

David, thanks for the restoration procedure. It seems my options right now are:

-Restore reading positions, RIP bookmarks
-Attempt a downgrade to 3.16.17? Not frequently done, might brick the kobo.
-Reset device, export user table from fresh DB to a CSV file, then load a backed-up DB and replace the user table from CSV. sqlitebrowser doesn't seem to have the ability to open two databases at once, which I suppose makes sense with the embedded focus of sqlite, so this is the best I could come up with.

[davidfor]

Originally Posted by youngmax

Although the notification about the firmware was originally in Calibre (pretty sure it was the Kobo Utilities notification of an available update for the device, though not 100%), I disconnected the device and updated by touching the sync tile on the home screen. Unfortunately the firmware status of the replacement device is unknown since it could have been sitting in its box since November for all I know. I don't have a backup of the new device's .kobo folder before I overwrote it.

Possibly of note, while setting up the new device I first tried logging into my kobo account and got to the home screen. Would that have been enough to trigger a check for available firmware updates?

The firmware should be updated during the setup process. It is usually done after entering the WiFi password and before being asked to login to the Kobo account. If there is an update, it will be downloaded, installed and the device rebooted. Then you will be taken through the rest of the setup.

If so, there seems to have been no reason for the device to download an update, which I saw it doing. If a bare initial login without hitting the sync tile even once isn't enough to trigger an update check, it sounds like I was running a database from the most recent firmware on a 'new' device with older firmware.

The upgrade check should be done:

• When setting up the device either new, or after a factory reset.
• When doing a sync. This can be the automatic daily sync if you haven't turned that off.
• When connecting to the Kobo desktop app.
• From the Kobo Utilities plugin in calibre. This is not officially supported

There is a dependency on the affiliate set on the device. In general, everywhere gets the same firmware version, but it can take a month for it to reach all affiliates and countries. Japan is the exception to this. Kobo seem to only release occasional firmware versions to Japan. I haven't seen a pattern except when a new device is launched there.

You can get different results from the utilities plugin. It can use the affiliate from the device, or an "early adopters" affiliate.The latter should be one of the affiliates that get the update first.

Here's the contents of the version file from my backups of the original device:

N250553006736,2.6.35.3,3.19.5761,2.6.35.3,2.6.35.3,00000000-0000-0000-0000-000000000370

Here's the version file off the new device the day I received it, but I'm not sure if this is from before or after I overwrote all the files:

N2505B3901229,2.6.35.3,3.19.5761,2.6.35.3,2.6.35.3,00000000-0000-0000-0000-000000000370

Both of those are from the current firmware. If you are intending to use your backup, you will probably need to be on this firmware.

Here it is from two days ago, just before I hit the sync tile and everything went to hell.

N2505B3901229,2.6.35.3,3.16.17,2.6.35.3,2.6.35.3,00000000-0000-0000-0000-000000000370

3.16.17 is a strange release. It isn't listed in the firmware links thread. I know it was released as a beta, but I don't remember it being released elsewhere. Unless it was a Japanese release and withdrawn or replaced quickly. It might have been a version with sickel enabled and quickly replaced by 3.17.0. And then 3.17.3 and then 3.18.0 without sickel.

David, thanks for the restoration procedure. It seems my options right now are:

-Restore reading positions, RIP bookmarks
-Attempt a downgrade to 3.16.17? Not frequently done, might brick the kobo.

1. Downgrading from 3.19.x to 3.16.17 is probably a risk. The way to do it is via factory reset. The steps are:
2. Install the Kobo desktop application.
3. Do the factory reset.
4. When prompted to setup the device, choose to use the PC.
5. Start the Kobo desktop application and connect the device.
6. Let the desktop application do the firmware update and do the setup, but DO NOT eject the device.
7. Open the drive for the device and find the firmware files in the .kobo directory. Delete them. You are looking for the directory upgrade and the files KoboRoot.tgz and manifest.md5sum.
8. Eject the device.

At this point the device will be running whatever the factory installed firmware was. For my Aura H2O, that is 3.7.0. I don't know if Kobo have updated the firmware on the devices, but they might have for the Japanese release. I don't remember when that was, so can't guess.

You can now connect to the PC and manually install any desired firmware version. You can also do that before ejecting the device above, but seeing the factory firmware is probably a useful data point.

-Reset device, export user table from fresh DB to a CSV file, then load a backed-up DB and replace the user table from CSV. sqlitebrowser doesn't seem to have the ability to open two databases at once, which I suppose makes sense with the embedded focus of sqlite, so this is the best I could come up with.

I don't normally use SQLiteBrowser, but if you want to open two databases at the same time, you can run two instances of it. They are then independent and can open different databases. But, extracting to CSV is probably the best bet and then importing that to the other database.

[youngmax]

This was a huge help! Constructing a hybrid database by moving the user table values worked great, my annotations are back. There's a little bit of glitchiness but nothing that interferes with my reading so far: a couple dozen 'new books' appeared with the name index by unknown author, composed of chunks from books I've read before. I suspect they're related to old annotations.

Deleting the table and using a CSV to hold the values ended up messing with the table settings (I don't think sqlitebrowser really understand the BIT field) but since there are less than a dozen values I ended up just editing the table with copy/paste. 100% fixed, I'm going to stay away from firmware updates in the future unless there's some stupendous new feature I can't live without.

Thanks again.

[algieba]

For future reference in case someone has issues with this in the future, I couldn't get it to work, the way I read the above, what did work however was this:

Using SQLiteBrowser:

1. Export user table from working database (what the device has after you click the Critical error away and log in).

2. Delete entry from user table (Should only be one) from your backup database.

3. Import the entry from the working database.

This restored all progress, bookmarks, font settings and all such things for me.

[davidfor]

Originally Posted by algieba

For future reference in case someone has issues with this in the future, I couldn't get it to work, the way I read the above, what did work however was this:

Using SQLiteBrowser:

1. Export user table from working database (what the device has after you click the Critical error away and log in).

2. Delete entry from user table (Should only be one) from your backup database.

3. Import the entry from the working database.

This restored all progress, bookmarks, font settings and all such things for me.

That sounds a lot like what I suggested in my first post, and what youngmax ended up doing. So, what couldn't you get to work?

[Omnigeek]

I had a similar problem. I was able to solve it by making a copy of the KoboReader.sqlite file on both my old and my new readers. I copied the device ID and pasted it into a copy of the old sqlite file using DBBrowser for SQLite (free Mac program) then saved the new sqlite file onto my new reader. No other changes. I got my old reading stats and bookmarks back without issue.