Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Kobo Reader

Notices

Reply
 
Thread Tools Search this Thread
Old 07-01-2021, 09:56 AM   #16
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,393
Karma: 40030263
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
When connecting my Elipsa to my Mac, upon ejection the Elipsa will reimport all sideloaded books whether I actually sideloaded books in the Finder or just did a sync with the Kobo desktop. Once importing is finished, only sideloaded books remain, not listed in order of import order, but alphabetized instead. All purchased books have been removed from my Elipsa. Then, when I repair my account in Settings, the book thumbnails are returned, but I have to download them all again. This happens EACH time I connect my Kobo to the desktop app.
That is expected behaviour when the database on the device is corrupt. When that happens, the database file cannot be updated. When something happens to force a reread of the database, it is basically reading an old version of the database. That means it doesn't have any changes since the last successful write, so no new books or reading status.

The device can run for a while with a corrupt database. Most of what you see displayed when looking for books is from in-memory structures. If the write to the database fails, the device will generally keep working.

If you are using calibre, it can show an error when getting the list of books. But, the driver only reads a couple of tables, and depending on exactly where the corruption has occurred, it might not see it. You can do a check of the database to see if there are errors using any SQLite database manager. Or my Kobo Utilities plugin has a function to do it.

As to how the the database gets corrupted, the only way I know is not safely ejecting the device when the database is being updated. There has to be other times as people report problems and swear they always eject safely. There are almost definitely some mistakes, but, there are two may reports for it not to happen at some other time.

Signing out of the Kobo account on the device should fix this as a new database is created. And of course, a factory reset will do that as well.


Quote:
Originally Posted by Cootey View Post
Once importing is finished, only sideloaded books remain, not listed in order of import order, but alphabetized instead.
When importing sideloaded books, they should be added in alphabetical order using the file name, which, for me starts with the title. When the books is added to the device, a timestamp is set in the database. For sideloaded books, the same timestamp is used for all books imported in the same batch. The "Recent" sort uses this timestamp or the last read timestamp and it probably sorts by title within these.
davidfor is online now   Reply With Quote
Old 07-05-2021, 07:02 AM   #17
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Quote:
Originally Posted by davidfor View Post
That is expected behaviour when the database on the device is corrupt. When that happens, the database file cannot be updated. When something happens to force a reread of the database, it is basically reading an old version of the database. That means it doesn't have any changes since the last successful write, so no new books or reading status.

The device can run for a while with a corrupt database. Most of what you see displayed when looking for books is from in-memory structures. If the write to the database fails, the device will generally keep working.
Well, *I* know the database was corrupt, but getting Kobo to agree was another matter. They have stopped responding to my emails, and starting today, new notebooks have started to disappear. Four full factory resets later, I thought maybe I had finally got things turned around, but each time some new feature starts to behave erratically. It was never the same corruption.

Quote:
If you are using calibre, it can show an error when getting the list of books. But, the driver only reads a couple of tables, and depending on exactly where the corruption has occurred, it might not see it. You can do a check of the database to see if there are errors using any SQLite database manager. Or my Kobo Utilities plugin has a function to do it.
I didn't realize that was built into your plug-in. I'll try it out. I do recall seeing calibre errors in previous incarnations. The current one works fine with calibre.

Öand a check of the database provided the following data:

Spoiler:
Result of running 'PRAGMA integrity_check' on database on the Kobo device:

*** in database main ***
On tree page 16290 cell 0: invalid page number 167772160
On tree page 15961 cell 2: 2nd reference to page 16598
On tree page 46691 cell 2: 2nd reference to page 16539
On tree page 419 cell 3: 2nd reference to page 50752
Page 690 is never used
Page 1049 is never used
Page 1367 is never used
row 1 missing from index sqlite_autoindex_content_1
row 2 missing from index sqlite_autoindex_content_1
row 3 missing from index sqlite_autoindex_content_1
row 4 missing from index sqlite_autoindex_content_1
row 5 missing from index sqlite_autoindex_content_1
row 6 missing from index sqlite_autoindex_content_1
row 7 missing from index sqlite_autoindex_content_1
row 8 missing from index sqlite_autoindex_content_1
row 9 missing from index sqlite_autoindex_content_1
row 10 missing from index sqlite_autoindex_content_1
row 11 missing from index sqlite_autoindex_content_1
row 12 missing from index sqlite_autoindex_content_1
row 13 missing from index sqlite_autoindex_content_1
row 14 missing from index sqlite_autoindex_content_1
row 15 missing from index sqlite_autoindex_content_1
row 16 missing from index sqlite_autoindex_content_1
row 17 missing from index sqlite_autoindex_content_1
row 18 missing from index sqlite_autoindex_content_1
row 19 missing from index sqlite_autoindex_content_1
row 20 missing from index sqlite_autoindex_content_1
row 21 missing from index sqlite_autoindex_content_1
row 22 missing from index sqlite_autoindex_content_1
row 23 missing from index sqlite_autoindex_content_1
row 24 missing from index sqlite_autoindex_content_1
row 25 missing from index sqlite_autoindex_content_1
row 26 missing from index sqlite_autoindex_content_1
row 27 missing from index sqlite_autoindex_content_1
row 28 missing from index sqlite_autoindex_content_1
row 29 missing from index sqlite_autoindex_content_1
row 30 missing from index sqlite_autoindex_content_1
row 31 missing from index sqlite_autoindex_content_1
row 32 missing from index sqlite_autoindex_content_1
row 33 missing from index sqlite_autoindex_content_1
row 34 missing from index sqlite_autoindex_content_1
row 35 missing from index sqlite_autoindex_content_1
row 36 missing from index sqlite_autoindex_content_1
row 37 missing from index sqlite_autoindex_content_1
row 38 missing from index sqlite_autoindex_content_1
row 39 missing from index sqlite_autoindex_content_1
row 40 missing from index sqlite_autoindex_content_1
row 41 missing from index sqlite_autoindex_content_1
row 42 missing from index sqlite_autoindex_content_1
row 43 missing from index sqlite_autoindex_content_1
row 44 missing from index sqlite_autoindex_content_1
row 45 missing from index sqlite_autoindex_content_1
row 46 missing from index sqlite_autoindex_content_1
row 47 missing from index sqlite_autoindex_content_1
row 48 missing from index sqlite_autoindex_content_1
row 49 missing from index sqlite_autoindex_content_1
row 50 missing from index sqlite_autoindex_content_1
row 51 missing from index sqlite_autoindex_content_1
row 52 missing from index sqlite_autoindex_content_1
row 53 missing from index sqlite_autoindex_content_1
row 54 missing from index sqlite_autoindex_content_1
row 55 missing from index sqlite_autoindex_content_1
row 56 missing from index sqlite_autoindex_content_1
row 57 missing from index sqlite_autoindex_content_1
row 58 missing from index sqlite_autoindex_content_1
row 59 missing from index sqlite_autoindex_content_1
row 60 missing from index sqlite_autoindex_content_1
row 61 missing from index sqlite_autoindex_content_1
row 62 missing from index sqlite_autoindex_content_1
row 63 missing from index sqlite_autoindex_content_1
row 64 missing from index sqlite_autoindex_content_1
row 65 missing from index sqlite_autoindex_content_1
row 66 missing from index sqlite_autoindex_content_1
row 67 missing from index sqlite_autoindex_content_1
row 68 missing from index sqlite_autoindex_content_1
row 69 missing from index sqlite_autoindex_content_1
row 70 missing from index sqlite_autoindex_content_1
row 71 missing from index sqlite_autoindex_content_1
row 72 missing from index sqlite_autoindex_content_1
row 73 missing from index sqlite_autoindex_content_1
row 74 missing from index sqlite_autoindex_content_1
row 75 missing from index sqlite_autoindex_content_1
row 76 missing from index sqlite_autoindex_content_1
row 77 missing from index sqlite_autoindex_content_1
row 78 missing from index sqlite_autoindex_content_1
row 79 missing from index sqlite_autoindex_content_1
row 80 missing from index sqlite_autoindex_content_1
row 81 missing from index sqlite_autoindex_content_1
row 82 missing from index sqlite_autoindex_content_1
row 83 missing from index sqlite_autoindex_content_1
row 84 missing from index sqlite_autoindex_content_1
row 85 missing from index sqlite_autoindex_content_1
row 86 missing from index sqlite_autoindex_content_1
row 87 missing from index sqlite_autoindex_content_1
row 88 missing from index sqlite_autoindex_content_1
row 89 missing from index sqlite_autoindex_content_1
row 90 missing from index sqlite_autoindex_content_1
row 91 missing from index sqlite_autoindex_content_1
row 92 missing from index sqlite_autoindex_content_1
row 93 missing from index sqlite_autoindex_content_1


Quote:
As to how the the database gets corrupted, the only way I know is not safely ejecting the device when the database is being updated. There has to be other times as people report problems and swear they always eject safely. There are almost definitely some mistakes, but, there are two may reports for it not to happen at some other time.

Signing out of the Kobo account on the device should fix this as a new database is created. And of course, a factory reset will do that as well.
In this case, considering that I factory reset from the dev menu four times, there was not an instance of me ejecting the device inappropriately, something I don't do. No amount of repairing the account, or logging out then back in, has made these glitches go away. There is a serious issue with my unit. Perhaps a bad memory module? I really can't say.

Quote:
When importing sideloaded books, they should be added in alphabetical order using the file name, which, for me starts with the title. When the books is added to the device, a timestamp is set in the database. For sideloaded books, the same timestamp is used for all books imported in the same batch. The "Recent" sort uses this timestamp or the last read timestamp and it probably sorts by title within these.
Although I often rely on this alphabetization when importing books to be added to a collection, when ALL books are reimported as if they hadn't been imported before, finding a dozen books alphabetized amidst 400+ books is a chore. This is what I was referring to. All books, including previously imported books, were being reimported without regard to that timestamp. However, I no longer have that problem. A full factory reset fixed it. Now I have disappearing journals, something that is frustrating since I drew on those pages and put a lot of effort into them.

FYI, I ran an undeleting program on the Kobo Elipsa when it was mounted on my Desktop, and it found all the missing *.nebo files. The only problem was that they were all 0 bytes large. They never properly saved to the internal memory. Here I thought I finally fixed it, then the glitch moved yet again. This time to an area I hadn't tested in over a week: notebooks. I have requested a new Elipsa pack from Kobo. Things are complicated since the US store showed that it was sold out earlier today.

Thanks for all your feedback.
Cootey is offline   Reply With Quote
Old 07-05-2021, 08:32 AM   #18
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,393
Karma: 40030263
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
Well, *I* know the database was corrupt, but getting Kobo to agree was another matter. They have stopped responding to my emails, and starting today, new notebooks have started to disappear. Four full factory resets later, I thought maybe I had finally got things turned around, but each time some new feature starts to behave erratically. It was never the same corruption.



I didn't realize that was built into your plug-in. I'll try it out. I do recall seeing calibre errors in previous incarnations. The current one works fine with calibre.

Öand a check of the database provided the following data:

Spoiler:
Result of running 'PRAGMA integrity_check' on database on the Kobo device:

*** in database main ***
On tree page 16290 cell 0: invalid page number 167772160
On tree page 15961 cell 2: 2nd reference to page 16598
On tree page 46691 cell 2: 2nd reference to page 16539
On tree page 419 cell 3: 2nd reference to page 50752
Page 690 is never used
Page 1049 is never used
Page 1367 is never used
row 1 missing from index sqlite_autoindex_content_1
row 2 missing from index sqlite_autoindex_content_1
row 3 missing from index sqlite_autoindex_content_1
row 4 missing from index sqlite_autoindex_content_1
row 5 missing from index sqlite_autoindex_content_1
row 6 missing from index sqlite_autoindex_content_1
row 7 missing from index sqlite_autoindex_content_1
row 8 missing from index sqlite_autoindex_content_1
row 9 missing from index sqlite_autoindex_content_1
row 10 missing from index sqlite_autoindex_content_1
row 11 missing from index sqlite_autoindex_content_1
row 12 missing from index sqlite_autoindex_content_1
row 13 missing from index sqlite_autoindex_content_1
row 14 missing from index sqlite_autoindex_content_1
row 15 missing from index sqlite_autoindex_content_1
row 16 missing from index sqlite_autoindex_content_1
row 17 missing from index sqlite_autoindex_content_1
row 18 missing from index sqlite_autoindex_content_1
row 19 missing from index sqlite_autoindex_content_1
row 20 missing from index sqlite_autoindex_content_1
row 21 missing from index sqlite_autoindex_content_1
row 22 missing from index sqlite_autoindex_content_1
row 23 missing from index sqlite_autoindex_content_1
row 24 missing from index sqlite_autoindex_content_1
row 25 missing from index sqlite_autoindex_content_1
row 26 missing from index sqlite_autoindex_content_1
row 27 missing from index sqlite_autoindex_content_1
row 28 missing from index sqlite_autoindex_content_1
row 29 missing from index sqlite_autoindex_content_1
row 30 missing from index sqlite_autoindex_content_1
row 31 missing from index sqlite_autoindex_content_1
row 32 missing from index sqlite_autoindex_content_1
row 33 missing from index sqlite_autoindex_content_1
row 34 missing from index sqlite_autoindex_content_1
row 35 missing from index sqlite_autoindex_content_1
row 36 missing from index sqlite_autoindex_content_1
row 37 missing from index sqlite_autoindex_content_1
row 38 missing from index sqlite_autoindex_content_1
row 39 missing from index sqlite_autoindex_content_1
row 40 missing from index sqlite_autoindex_content_1
row 41 missing from index sqlite_autoindex_content_1
row 42 missing from index sqlite_autoindex_content_1
row 43 missing from index sqlite_autoindex_content_1
row 44 missing from index sqlite_autoindex_content_1
row 45 missing from index sqlite_autoindex_content_1
row 46 missing from index sqlite_autoindex_content_1
row 47 missing from index sqlite_autoindex_content_1
row 48 missing from index sqlite_autoindex_content_1
row 49 missing from index sqlite_autoindex_content_1
row 50 missing from index sqlite_autoindex_content_1
row 51 missing from index sqlite_autoindex_content_1
row 52 missing from index sqlite_autoindex_content_1
row 53 missing from index sqlite_autoindex_content_1
row 54 missing from index sqlite_autoindex_content_1
row 55 missing from index sqlite_autoindex_content_1
row 56 missing from index sqlite_autoindex_content_1
row 57 missing from index sqlite_autoindex_content_1
row 58 missing from index sqlite_autoindex_content_1
row 59 missing from index sqlite_autoindex_content_1
row 60 missing from index sqlite_autoindex_content_1
row 61 missing from index sqlite_autoindex_content_1
row 62 missing from index sqlite_autoindex_content_1
row 63 missing from index sqlite_autoindex_content_1
row 64 missing from index sqlite_autoindex_content_1
row 65 missing from index sqlite_autoindex_content_1
row 66 missing from index sqlite_autoindex_content_1
row 67 missing from index sqlite_autoindex_content_1
row 68 missing from index sqlite_autoindex_content_1
row 69 missing from index sqlite_autoindex_content_1
row 70 missing from index sqlite_autoindex_content_1
row 71 missing from index sqlite_autoindex_content_1
row 72 missing from index sqlite_autoindex_content_1
row 73 missing from index sqlite_autoindex_content_1
row 74 missing from index sqlite_autoindex_content_1
row 75 missing from index sqlite_autoindex_content_1
row 76 missing from index sqlite_autoindex_content_1
row 77 missing from index sqlite_autoindex_content_1
row 78 missing from index sqlite_autoindex_content_1
row 79 missing from index sqlite_autoindex_content_1
row 80 missing from index sqlite_autoindex_content_1
row 81 missing from index sqlite_autoindex_content_1
row 82 missing from index sqlite_autoindex_content_1
row 83 missing from index sqlite_autoindex_content_1
row 84 missing from index sqlite_autoindex_content_1
row 85 missing from index sqlite_autoindex_content_1
row 86 missing from index sqlite_autoindex_content_1
row 87 missing from index sqlite_autoindex_content_1
row 88 missing from index sqlite_autoindex_content_1
row 89 missing from index sqlite_autoindex_content_1
row 90 missing from index sqlite_autoindex_content_1
row 91 missing from index sqlite_autoindex_content_1
row 92 missing from index sqlite_autoindex_content_1
row 93 missing from index sqlite_autoindex_content_1
That might be recoverable with a VACCUUM. Which is what the compress database function in the my Kobo Utilities plugin does. If the error is just in indexes, this will work, but, the first few errors might prevent it.
Quote:
In this case, considering that I factory reset from the dev menu four times, there was not an instance of me ejecting the device inappropriately, something I don't do. No amount of repairing the account, or logging out then back in, has made these glitches go away. There is a serious issue with my unit. Perhaps a bad memory module? I really can't say.
It feels like a hardware problem. I think you need to get it replaced.
Quote:
Although I often rely on this alphabetization when importing books to be added to a collection, when ALL books are reimported as if they hadn't been imported before, finding a dozen books alphabetized amidst 400+ books is a chore. This is what I was referring to. All books, including previously imported books, were being reimported without regard to that timestamp. However, I no longer have that problem. A full factory reset fixed it. Now I have disappearing journals, something that is frustrating since I drew on those pages and put a lot of effort into them.
When I reload a device, I use set the "Date added" to something useful after the import. I swap between the published date and the "Date" column in calibre. The latter is roughly the date I got the book, so it works fairly well for me.
Quote:
FYI, I ran an undeleting program on the Kobo Elipsa when it was mounted on my Desktop, and it found all the missing *.nebo files. The only problem was that they were all 0 bytes large. They never properly saved to the internal memory. Here I thought I finally fixed it, then the glitch moved yet again. This time to an area I hadn't tested in over a week: notebooks. I have requested a new Elipsa pack from Kobo. Things are complicated since the US store showed that it was sold out earlier today.
Yes, it needs to be replaced.
davidfor is online now   Reply With Quote
Old 07-17-2021, 12:56 AM   #19
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
My new Elipsa arrived. Other people here have complained about database issues. So far, one other person gave up on the Elipsa and returned theirs to the store. I rather love the Elipsa, so I opted to get another one. I'll update this thread if I start to see disappearing books again.

For now, I've setup my Elipsa with a few hundred books, I restored the only notebook that I was able to save from my other Elipsa, and now I'm running my Elipsa through it's paces.

When I first set it up with the Kobo Desktop app, it got stuck rebooting for a few minutes. Kobo Desktop had updated its firmware. When the Elipsa finally booted, the What's New message was there with its sparse details, then I tried to sync to the server and it said it needed to update the firmware again. Then more downloading and booting, and the What's New message with the same sparse details reappeared. I can't say that it was a beginning that inspired confidence, but after all that, I am up and running. Crossing fingers.
Cootey is offline   Reply With Quote
Old 07-17-2021, 05:50 AM   #20
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Well, it didn't take me long. I've run into the corrupt database/deleted downloads issue again. Once the Elipsa is ejected from my Mac, it reimports ALL sideloaded books again, deletes all 242 purchased books, and resets the home screen. Repairing the account restores everything but the downloaded ebooks.

Setting my Elipsa up with Kobo Desktop worked fine.
Manually sideloading books by dragging and dropping in the Finder was fine.
Downloading through Overdrive was fine.
I ran into problems the moment I started to use Calibre to import books. Here's the error:

Code:
calibre, version 5.23.0
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
  File "calibre/gui2/device.py", line 89, in run
  File "calibre/gui2/device.py", line 515, in _books
  File "calibre/devices/kobo/driver.py", line 1974, in books
apsw.CorruptError: CorruptError: database disk image is malformed
I could use some suggestions on how to troubleshoot and identify what is going wrong. I don't want to abandon Calibre and davidfor's plugin. They're too useful. Right now I'm trying to restore from a backup to a state where everything seemed to be working. If that doesn't work, I'll log out of my Elipsa, then log back in. Then factory reset as a last resort.
Cootey is offline   Reply With Quote
Old 07-17-2021, 09:44 AM   #21
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,393
Karma: 40030263
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
Well, it didn't take me long. I've run into the corrupt database/deleted downloads issue again. Once the Elipsa is ejected from my Mac, it reimports ALL sideloaded books again, deletes all 242 purchased books, and resets the home screen. Repairing the account restores everything but the downloaded ebooks.

Setting my Elipsa up with Kobo Desktop worked fine.
Manually sideloading books by dragging and dropping in the Finder was fine.
Downloading through Overdrive was fine.
I ran into problems the moment I started to use Calibre to import books. Here's the error:

Code:
calibre, version 5.23.0
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
  File "calibre/gui2/device.py", line 89, in run
  File "calibre/gui2/device.py", line 515, in _books
  File "calibre/devices/kobo/driver.py", line 1974, in books
apsw.CorruptError: CorruptError: database disk image is malformed
I could use some suggestions on how to troubleshoot and identify what is going wrong. I don't want to abandon Calibre and davidfor's plugin. They're too useful. Right now I'm trying to restore from a backup to a state where everything seemed to be working. If that doesn't work, I'll log out of my Elipsa, then log back in. Then factory reset as a last resort.
The problem is knowing when the database became corrupted. The error is at the start of the loop reading the database. It is probably happening on the first record, but, might be happening after reading some of the book records. And that probably means the database is already corrupted when calibre sees it.

Can you test the database before using calibre? Any SQLite database manager should be able to do it. They might have an option to do the check. But, there isn't, then run "PRAGMA integrity_check". You can try DB Browser for SQLite as that has a MAC version.

If the database is fine, run calibre in debug mode. That will let us see if it is breaking when the database is first opened, or after reading some records. You will need to start calibre with the command "calibre-debug -g" to do this.

You could also check the database before ejecting the device. This is to confirm that calibre did not do anything bad.

Other thoughts are:

Could it be the MAC? If you have another computer to try, it would be good to do soo.

Are you ever powering off?

Are there any log files being produced in the .kobo directory? I'm mainly wondering about stack dumps which will have the name "stack_xx.log" where xx is two numbers. If these exist, they are a sign of a crash happening somewhere.

Does the partition show as being dirty when you connect? I don't know if the MAC will show this. Windows would offer to check the partition if it is dirty.
davidfor is online now   Reply With Quote
Old 07-17-2021, 08:10 PM   #22
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Quote:
Originally Posted by davidfor View Post
The problem is knowing when the database became corrupted. The error is at the start of the loop reading the database. It is probably happening on the first record, but, might be happening after reading some of the book records. And that probably means the database is already corrupted when calibre sees it.

Can you test the database before using calibre? Any SQLite database manager should be able to do it. They might have an option to do the check. But, there isn't, then run "PRAGMA integrity_check". You can try DB Browser for SQLite as that has a MAC version.
I had DB Browser for SQLite already installed, so I copied the Elipsa KoboReader.sqlite and ran the "PRAGMA integrity_check" on it.

Code:
Execution finished without errors.
Result: 1 rows returned in 3144ms
At line 1:
PRAGMA integrity_check;
Because I backed up to an earlier version before things went kablooie, my database is fine.

Quote:
Originally Posted by davidfor View Post
If the database is fine, run calibre in debug mode. That will let us see if it is breaking when the database is first opened, or after reading some records. You will need to start calibre with the command "calibre-debug -g" to do this.
You could also check the database before ejecting the device. This is to confirm that calibre did not do anything bad.
Running in debug mode now. Thanks.

I used Calibre to copy one book over and checked the database before ejecting. Everything is A-OK.

Quote:
Originally Posted by davidfor View Post
Other thoughts are:

Could it be the MAC? If you have another computer to try, it would be good to do soo.

Are you ever powering off?

Are there any log files being produced in the .kobo directory? I'm mainly wondering about stack dumps which will have the name "stack_xx.log" where xx is two numbers. If these exist, they are a sign of a crash happening somewhere.

Does the partition show as being dirty when you connect? I don't know if the MAC will show this. Windows would offer to check the partition if it is dirty.
1) I don't have problems like this with either of our Kobo Clara HDs, or either Kobo Aura One in the past. I wasn't exactly paying attention to what I was doing just prior to getting the database error in Calibre, but IIRC, the error popped up when I plugged the Elipsa in after an evening of setup. That might indicate the problem happened on the Elipsa. I'll watch for it.
2) Powering off my Mac while the Elipsa is plugged in? No. I keep my Mac on at all times. I don't even let it sleep.
3) No log files as you described.
4) From what I've read, Mac tools for dealing with ExFat dirty drives is minimal. I'll have to see if that is the case if the corruption should occur again.

Thanks!

Last edited by Cootey; 07-17-2021 at 08:26 PM.
Cootey is offline   Reply With Quote
Old 07-17-2021, 10:08 PM   #23
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,393
Karma: 40030263
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
I had DB Browser for SQLite already installed, so I copied the Elipsa KoboReader.sqlite and ran the "PRAGMA integrity_check" on it.

Code:
Execution finished without errors.
Result: 1 rows returned in 3144ms
At line 1:
PRAGMA integrity_check;
Because I backed up to an earlier version before things went kablooie, my database is fine.
Unfortunately, you will need to do this check each time. We don't know when the corruption is happening. I'm trying to eliminate calibre as the cause. Doing the check a few times should help with that.
Quote:
Running in debug mode now. Thanks.

I used Calibre to copy one book over and checked the database before ejecting. Everything is A-OK.



1) I don't have problems like this with either of our Kobo Clara HDs, or either Kobo Aura One in the past. I wasn't exactly paying attention to what I was doing just prior to getting the database error in Calibre, but IIRC, the error popped up when I plugged the Elipsa in after an evening of setup. That might indicate the problem happened on the Elipsa. I'll watch for it.
2) Powering off my Mac while the Elipsa is plugged in? No. I keep my Mac on at all times. I don't even let it sleep.
No I meant powering off the Elipsa. If it doesn't get shutdown properly, that could be a cause. And when the database is read, that can be when problems are noticed such as missing books and reading status.
Quote:
3) No log files as you described.
4) From what I've read, Mac tools for dealing with ExFat dirty drives is minimal. I'll have to see if that is the case if the corruption should occur again.
I have no idea what tools are available on the MAC for this, but I had a feeling that there wasn't much. Kobo did update dosfstools last year which solved some problems with the PC seeing the drive as dirty. I don't really expect this to be the problem, but it would be good to eliminate.
davidfor is online now   Reply With Quote
Old 07-22-2021, 12:07 AM   #24
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Hereís an update:

Iíve enabled debug mode for Calibre, and I check the database upon mounting and before ejecting. Iíve also checked the Elipsa database with SQLite software. No problems have been found. I also constantly backup my Elipsa so I donít lose data.

With all the database issues Iíve run into, I can report that many ebook annotations and page locations have not been lost, but the Elipsa has forgotten that Iím reading quite a few books. Those ebooks still retain their annotations, but not current page location. Fortunately, bookmarks are intact, too.

Kobo backs up our stylus written annotations so that they are restored when logging into a reset or new Kobo. Notebooks are goners, unfortunately.

I have had one close call that didnít require restoring the database. After ejecting my Elipsa using Calibre at the end of last week, the Elipsa powered off completely. When I tried to turn it back on, it didnít respond. I pushed the on button a few times, waiting for the Elipsa to respond after each press, but only after the fourth time did it power up, then acted as it had meant to do that, everything was perfectly fine, and would you please stop staring at me?

Iím watching it with a suspicious side eye now, but the Elipsa has been behaving without issue since then.


Sent from my iPhone using Tapatalk Pro
Cootey is offline   Reply With Quote
Old 07-22-2021, 08:44 AM   #25
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Alright. I've got a corrupt database again. I was working with Kobo tech over a different issue (Elipsa doesn't prompt to review or rate purchased books when they are finished). I was asked to connect my Elipsa to Kobo Desktop on my Mac. It synced fine. No books were added. I cmd-shift-i'd my log file. Then I got busy working on something else. When I ejected my Elipsa properly, it suddenly reimported EVERYTHING all over again. All my purchased downloads have disappeared, again.

I connected it to the Mac again, this time with Calibre. I immediately got the ERROR: Error dialog letting me know "Error communicating with device".

Here is the clipboard error output:

Code:
calibre, version 5.23.0
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
  File "calibre/gui2/device.py", line 89, in run
  File "calibre/gui2/device.py", line 515, in _books
  File "calibre/devices/kobo/driver.py", line 1974, in books
apsw.CorruptError: CorruptError: database disk image is malformed
If I repair the account on the Kobo, I know it will be fixed. Then I have to redownload everything again. Instead, I have opted to try to check the database using Calibre.

Result of running 'PRAGMA integrity_check' on database on the Kobo device:


Spoiler:
*** in database main ***
On tree page 16876 cell 1: Rowid 393 out of order
On tree page 44734 cell 1: Rowid 348 out of order
On tree page 44734 cell 0: 2nd reference to page 49801
On tree page 51788 cell 1: 2nd reference to page 51790
On tree page 529 cell 2: Rowid 40093 out of order
Page 50211: btreeInitPage() returns error code 11
On tree page 29598 cell 103: Child page depth differs
On tree page 50155 cell 0: invalid page number 167850496
Page 1150: btreeInitPage() returns error code 11
Page 543: btreeInitPage() returns error code 11
Page 520: btreeInitPage() returns error code 11
Page 463: btreeInitPage() returns error code 11
Page 457: btreeInitPage() returns error code 11
Page 497: btreeInitPage() returns error code 11
On tree page 526 cell 3: invalid page number 218103808
Page 505: btreeInitPage() returns error code 11
On tree page 481 cell 0: invalid page number 218103808
Page 50171: btreeInitPage() returns error code 11
Page 971: btreeInitPage() returns error code 11
Page 964: btreeInitPage() returns error code 11
Page 50176: btreeInitPage() returns error code 11
Page 50204: btreeInitPage() returns error code 11
On tree page 130 cell 3: Rowid 39762 out of order
On tree page 130 cell 3: 2nd reference to page 50274
On tree page 50166 cell 39: Rowid 39672 out of order
On tree page 1260 cell 0: Rowid 41643 out of order
Page 50157: btreeInitPage() returns error code 11
Page 50215: btreeInitPage() returns error code 11
On tree page 1165 cell 5: invalid page number 218103808
On tree page 1165 cell 0: invalid page number 218103808
Page 319: btreeInitPage() returns error code 11
On tree page 978 cell 4: Rowid 41636 out of order
Page 539: btreeInitPage() returns error code 11
Page 8888: btreeInitPage() returns error code 11
Page 1134: btreeInitPage() returns error code 11
On tree page 50190 cell 5: Rowid 41611 out of order
On tree page 50190 cell 4: 2nd reference to page 51806
On tree page 50190 cell 3: 2nd reference to page 50269
On tree page 50262 cell 0: 2nd reference to page 579
Page 1129: btreeInitPage() returns error code 11
On tree page 50170 cell 5: Rowid 41605 out of order
On tree page 50170 cell 0: 2nd reference to page 640
On tree page 50260 cell 4: invalid page number 218103808
On tree page 50260 cell 1: invalid page number 218103808
On tree page 979 cell 2: Rowid 41572 out of order
On tree page 979 cell 2: 2nd reference to page 1290
On tree page 979 cell 1: 2nd reference to page 50171
Page 469: btreeInitPage() returns error code 11
Page 931: btreeInitPage() returns error code 11
Page 232: btreeInitPage() returns error code 11
Page 8893: btreeInitPage() returns error code 11
Page 566: btreeInitPage() returns error code 11
On tree page 50159 cell 2: Rowid 41559 out of order
On tree page 50159 cell 1: 2nd reference to page 566
Page 50172: btreeInitPage() returns error code 11
On tree page 50259 cell 0: 2nd reference to page 1485
On tree page 10131 cell 3: Rowid 41437 out of order
On tree page 10131 cell 3: invalid page number 218103808
Page 547: btreeInitPage() returns error code 11
On tree page 50258 cell 4: invalid page number 218103808
On tree page 50258 cell 3: 2nd reference to page 482
On tree page 50258 cell 1: 2nd reference to page 366
Page 230: btreeInitPage() returns error code 11
Page 2303: btreeInitPage() returns error code 11
On tree page 957 cell 4: Rowid 41536 out of order
Page 1198: btreeInitPage() returns error code 11
Page 1434: btreeInitPage() returns error code 11
Page 518: btreeInitPage() returns error code 11
Page 1348: btreeInitPage() returns error code 11
Page 509: btreeInitPage() returns error code 11
On tree page 476 cell 3: invalid page number 218103808
Page 1302: btreeInitPage() returns error code 11
Page 18616: btreeInitPage() returns error code 11
Page 461: btreeInitPage() returns error code 11
On tree page 29598 cell 102: Child page depth differs
On tree page 459 cell 5: Rowid 41661 out of order
On tree page 459 cell 3: 2nd reference to page 1161
On tree page 459 cell 2: invalid page number 167884544
On tree page 501 cell 3: Rowid 41502 out of order
On tree page 501 cell 2: 2nd reference to page 18616
On tree page 501 cell 0: 2nd reference to page 523
Page 618: btreeInitPage() returns error code 11
On tree page 363 cell 2: 2nd reference to page 612
On tree page 363 cell 1: 2nd reference to page 1573
On tree page 363 cell 0: 2nd reference to page 1398
On tree page 44802 cell 79: 2nd reference to page 45116
On tree page 44802 cell 52: Rowid 35248 out of order
On tree page 44802 cell 52: 2nd reference to page 45112
On tree page 44802 cell 51: 2nd reference to page 45105
On tree page 44802 cell 50: 2nd reference to page 45103
On tree page 44802 cell 49: 2nd reference to page 45101
On tree page 44802 cell 48: 2nd reference to page 45091
On tree page 44802 cell 47: 2nd reference to page 45090
On tree page 44802 cell 46: 2nd reference to page 45085
On tree page 44802 cell 45: 2nd reference to page 45081
On tree page 44802 cell 44: 2nd reference to page 45079
On tree page 44802 cell 43: 2nd reference to page 45076
On tree page 44802 cell 42: 2nd reference to page 45071
On tree page 44802 cell 41: 2nd reference to page 45070
On tree page 44802 cell 40: 2nd reference to page 45058


This is an annoying bug. Since the Elipsa was mounted on my desktop for an hour, I wonder if Spotlight's search indexing caused a problem. I have now excluded KOBOeReader from being indexed. But this could be a Kobo Desktop problem, too. Any suggestions on how to isolate who is at fault?

How fortunate for me that I just did a backup half an hour ago before this happened.

Last edited by Cootey; 07-22-2021 at 05:48 PM.
Cootey is offline   Reply With Quote
Old 07-22-2021, 09:37 AM   #26
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,393
Karma: 40030263
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
The act of reimporting everything suggests the database was corrupted when the device was ejected. Whether that was the Kobo desktop or something else running, I don't know. If something else was accessing the database when the Kobo desktop was, that would be a likely culprit. If that was also running during your calibre sessions, that could also explain the other problems you have been seeing,
davidfor is online now   Reply With Quote
Old 07-22-2021, 06:12 PM   #27
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Quote:
Originally Posted by davidfor View Post
The act of reimporting everything suggests the database was corrupted when the device was ejected. Whether that was the Kobo desktop or something else running, I don't know. If something else was accessing the database when the Kobo desktop was, that would be a likely culprit. If that was also running during your calibre sessions, that could also explain the other problems you have been seeing,
I posted the clipboard error message and the 'PRAGMA integrity_check' results for you above. Was there anything else you'd need in the future?

Since I began bumping into these issues, I don't run Kobo desktop and Calibre at the same time. Also, there are no active programs operating on the Kobo when I eject it. The Mac won’t allow it. I can’t even eject it if the Elipsa is simply the current directory in Terminal. I also plug the Elipsa directly into the back of my Mac instead of using a USB hub to eliminate potential issues there. I never have these problems with my Clara. I haven't treated my Elipsa any different, except now I treat it as if its made of handblown Egyptian glass when I bring it near my Mac.

I usually check the database first in Calibre before ejecting it. I may have forgotten to do that this time, but I’ve ejected directly from within Kobo Desktop before with no issues.

That all being said, since I last posted above I had to restore my Kobo. Due to our past conversation in another thread, I’ve been experimenting with using rsync instead of dd to make a backup. So I used cp to create the backup archive, then planned to rsync the archive back onto the Kobo in the event of a database corruption. I had a full disk image backup at the ready, so this was my first time to experiment using rsync to restore the Kobo.

While it was running, it deleted a ton of ._ files off the Elipsa. Usually, the Mac creates those files every time you access a directory from the Finder. However, there were ._ files in directories I had never accessed from Finder. To be clear, these were on the Kobo and didn’t exist in the backup. That’s why rsync removed them. That’s probably obvious to you, but I’m trying to be thorough. They were created between the time I made the backup and the time I ejected the Kobo.

So what process is accessing the Kobo? Prior to your suggestion, I was already puzzling over this. TimeMachine apparently can store data in ._ files, but I don't use TimeMachine on USB external drives. I blocked Spotlight from indexing the Kobo whenever it is mounted. Spotlight only indexes for searching, however. It doesn’t create ._ files. I have never seen this issue with any USB mounted device before, and I can’t be certain this is the issue that caused the corruption. The Kobo tends to ignore ._ files in my experience.

This is a lot of text, but as I said above, I'm trying to be thorough. I know a couple of people have bumped into corrupt Elipsa databases. One person gave up and returned his Elipsa to the store. We should be finding out if they use Macs, Kobo Desktop, and Calibre to find commonalities. And the big question is: Why is only my Elipsa having these issues and not either of our Claras?

I can't imagine somebody who is not a geek trying to deal with this.

Last edited by Cootey; 07-22-2021 at 06:21 PM.
Cootey is offline   Reply With Quote
Old 07-23-2021, 03:02 AM   #28
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,393
Karma: 40030263
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
I posted the clipboard error message and the 'PRAGMA integrity_check' results for you above. Was there anything else you'd need in the future?
There isn't much I can do with the results of the integrity check. I can tell from a quick look if they are likely to be fixed by doing a VACUUM. That will be the case if the errors are only in the indexes or with free space. With the results you have, I doubt it can be fixed.
Quote:
Since I began bumping into these issues, I don't run Kobo desktop and Calibre at the same time. Also, there are no active programs operating on the Kobo when I eject it. The Mac wonít allow it. I canít even eject it if the Elipsa is simply the current directory in Terminal. I also plug the Elipsa directly into the back of my Mac instead of using a USB hub to eliminate potential issues there. I never have these problems with my Clara. I haven't treated my Elipsa any different, except now I treat it as if its made of handblown Egyptian glass when I bring it near my Mac.

I usually check the database first in Calibre before ejecting it. I may have forgotten to do that this time, but Iíve ejected directly from within Kobo Desktop before with no issues.

That all being said, since I last posted above I had to restore my Kobo. Due to our past conversation in another thread, Iíve been experimenting with using rsync instead of dd to make a backup. So I used cp to create the backup archive, then planned to rsync the archive back onto the Kobo in the event of a database corruption. I had a full disk image backup at the ready, so this was my first time to experiment using rsync to restore the Kobo.

While it was running, it deleted a ton of ._ files off the Elipsa. Usually, the Mac creates those files every time you access a directory from the Finder. However, there were ._ files in directories I had never accessed from Finder. To be clear, these were on the Kobo and didnít exist in the backup. Thatís why rsync removed them. Thatís probably obvious to you, but Iím trying to be thorough. They were created between the time I made the backup and the time I ejected the Kobo.

So what process is accessing the Kobo? Prior to your suggestion, I was already puzzling over this. TimeMachine apparently can store data in ._ files, but I don't use TimeMachine on USB external drives. I blocked Spotlight from indexing the Kobo whenever it is mounted. Spotlight only indexes for searching, however. It doesnít create ._ files. I have never seen this issue with any USB mounted device before, and I canít be certain this is the issue that caused the corruption. The Kobo tends to ignore ._ files in my experience.
The corruption could happen from:
  • An application crashing doing during the write.
  • Two applications attempting to access the database at the same time.
  • Failing or otherwise damaged storage.
  • File system corruption by something.
  • Something making changes at the file system level.
  • Device being disconnected before all writes have been completed.

Unfortunately, I just don't know when it happens. A possibility is a background task on the MAC clashing. But, you should be able to see that.

Another thing to try is to not eject from within the applications, either calibre or the Kobo desktop. Close these and wait a couple of minutes. That should make sure nothing that you are doing trying to update, and it should let any caches flush as well. I don't know the MAC, but, Windows has, or at least had, an option for "fast eject" or something like that. Turning it on, meant that writes to removable media were immediate and not cached.

Quote:
This is a lot of text, but as I said above, I'm trying to be thorough. I know a couple of people have bumped into corrupt Elipsa databases. One person gave up and returned his Elipsa to the store. We should be finding out if they use Macs, Kobo Desktop, and Calibre to find commonalities. And the big question is: Why is only my Elipsa having these issues and not either of our Claras?
Unfortunately, there are always corrupt databases. But, the number with the Elipsa seems to be a lot worse. It might be something in the firmware. If it is, it will show when Kobo roll these updates out to other devices. Or it could be something about the hardware. There is enough different hardware in them (USB-C, multi-core CPU etc) that there might be something that they have missed. If it is caused by something during the connection to the PC, maybe it is one of the processes on the device not stopping cleanly and contending with the database access from the PC. The obvious culprit would be something in the note taking software. That is new and not on the other devices. And the only way to test that is to not use it for a while.
davidfor is online now   Reply With Quote
Old 07-23-2021, 07:36 AM   #29
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Quote:
Originally Posted by davidfor View Post
Another thing to try is to not eject from within the applications, either calibre or the Kobo desktop. Close these and wait a couple of minutes. That should make sure nothing that you are doing trying to update, and it should let any caches flush as well. I don't know the MAC, but, Windows has, or at least had, an option for "fast eject" or something like that. Turning it on, meant that writes to removable media were immediate and not cached.
Good response. Thank you. There IS a lot new with the Elipsa compared to other Kobo eReaders, and since I'm using the same software and computer with my Clara as I do my Elipsa, and the Clara soldiers on without incident, I'm thinking the Elipsa has an issue. I am crossing my fingers that it's something they can fix in firmware, unlike the backlighting issues with the Forma that people just had to live with. I wish I could isolate what is triggering this issue, but I'll keep a watchful eye. Restoring my Elipsa so often is a pain in the back.

However, I'll follow your suggestion and quit the apps before ejecting the Elipsa. If Time Machine, Spotlight, Kobo Desktop, Terminal, and Calibre are not accessing the mounted Elipsa, I'll look into how to spot what other process could be poking around in the Elipsa's SD Card before ejecting. Still, the Mac is VERY stubborn about not allowing disks to be ejected if a process is running that is using the drive. I'm thinking there is something going wrong when the Elipsa recognizes it is no longer connected and runs through a check that detects changes before opening to Home. Next time the corruption happens, I'll try restoring one SQLite database at a time to see which one is the problem.

Kobo should be paying us. LOL
Cootey is offline   Reply With Quote
Old 08-14-2021, 11:03 PM   #30
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 633
Karma: 5764395
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Alright. I'm back with another database corruption. For the record, I never have these issues with my Kobo Clara.

I believe that the recent firmware updates released for the Kobo have improved things. I've gone 22 days without a database corruption—until earlier this morning. Unfortunately, I did not remember to quit Calibre before ejecting as you had suggested (something I had been forgetting to do recently), but I do have more information that might help us.

My notes on my latest Elipsa glitch:

1. Checked database using Kobo Utilities in Calibre. ✔
2. Imported about 21 manga through Calibre. Database ✔
3. Imported another 9 manga. Checked database again. It was corrupt.
4. On a whim, I deleted all 30 manga manually off the Elipsa before ejecting it from desktop. This avoided the import/database destruction I’ve noticed in the past. All purchased books remained downloaded. No re-importation of all previously imported books occurred.
5. Repaired Kobo account. Plugged Elipsa back into Mac. Checked database. It remained corrupt.
6. Backed up my .nebo journal files, then I restored from older backup via Terminal. This has ALWAYS worked before, but this time…
7. …Elipsa .jpg & .svg files (the stylus notations and their page screenshots) from the .kobo/markups directory were now loaded as individual files. Several Kobo pages worth in My Books. Even repairing the account didn’t properly categorize them. I thought perhaps they might be newer out of sync annotations that weren't part of the older database, but the rsync string I use to restore a backup deletes everything off the Elipsa that isn't present in the backup.
8. Logged out of my account on my Kobo, then logged back in. Markup files were still listed individually as "books". I checked the database with Kobo Utils plug-in in Calibre, and the database had been repaired as expected. Since I wasn’t sure which markup files were being categorized correctly and which were not, I…
9. Did a factory reset (with current firmware).
10. Annotations were restored. I don't know if I lost any, but I keep WiFi on all the time, so it's not likely. (Turns out I lost annotations from just the past 24 hours before corruption.) Database is not corrupt. I'm ready to rebuild my setup again. But I've lost time, I'll be losing more time getting things back up and running, and I'm grumpy about it. Why is this happening to my Elipsa? ����*♂️

Here are my observations. I’ve used Calibre to import manga before since the last Elipsa database corruption with no issues. Yet whenever the database is corrupted, Calibre seems to be involved. (I run Calibre in debug mode, so please tell me what you folks need to see to help identify what is happening. I'll send it to you since it likely contains names of my local drives and network). When I sideload hundreds of books by dragging and dropping onto the mounted Elipsa, I have ZERO database issues.

I did notice, however that the Elipsa was constantly being ejected this morning. THIS behavior is consistent with corruption events. Whether I was plugged into a hub or directly into the back of my 2014 Mac Mini, the Elipsa would connect, then suddenly eject itself. This is very irregular behavior, and the Elipsa was the only device having this issue. The Mac did not display any warning dialogues as it usually does if you unplug a USB device without ejecting through the Finder first. That seems to indicate that it was rejected properly. When the Elipsa seemed to be stably connected for a few minutes, I imported the manga through Calibre. The Elipsa was not ejected during this process. However, after the transfer was complete (done in two steps BTW with me checking the database after each batch) the database was corrupt after the second, smaller batch. Then I did step 4 above to minimize the damage, but to no avail.

After resetting the firmware, my Elipsa was connected for half a day without being ejected from my Mac. I can make sure that I quit Calibre and Kobo Desktop in the future. I have put in an order for a different USB hub. But neither of those things may be the problem. In the meantime, something became corrupt on that second Calibre batch. The Elipsa hadn't technically imported anything yet since it was still mounted to my desktop. I'd like to figure this out. I'm not the only person running into corrupt database issues with the Elipsa.

The following is the PRAGMA integrity check when I discovered the database was corrupt. I don't think this is any different than any other integrity check dump that I've shared. It just means the database is now corrupt:
Spoiler:
Result of running 'PRAGMA integrity_check' on database on the Kobo device:


*** in database main ***
On tree page 54734 cell 0: invalid page number 167909888
Page 52133: btreeInitPage() returns error code 11
Page 52134: btreeInitPage() returns error code 11
Page 52132: btreeInitPage() returns error code 11
On tree page 52116 cell 0: Rowid 47480 out of order
On tree page 52116 cell 0: invalid page number 84037120
On tree page 52171 cell 1: Rowid 47546 out of order
On tree page 52171 cell 1: invalid page number 167772160
On tree page 52171 cell 0: invalid page number 167836416
On tree page 54521 cell 35: Rowid 45054 out of order
Page 52177: btreeInitPage() returns error code 11
On tree page 29598 cell 91: 2nd reference to page 50413
On tree page 29598 cell 91: Child page depth differs
On tree page 29598 cell 90: Child page depth differs
On tree page 759 cell 7: Rowid 28237 out of order
On tree page 761 cell 1: Rowid 47659 out of order
On tree page 762 cell 1: Rowid 47644 out of order
On tree page 11685 cell 10: 2nd reference to page 15476
On tree page 2578 cell 2: Child page depth differs
On tree page 2578 cell 1: Child page depth differs
On tree page 12703 cell 3: 2nd reference to page 20888
On tree page 51982 cell 3: 2nd reference to page 7442
On tree page 51982 cell 2: 2nd reference to page 52151
On tree page 51982 cell 1: 2nd reference to page 52153
On tree page 51982 cell 0: 2nd reference to page 684
Page 52009: btreeInitPage() returns error code 11
On tree page 42540 cell 0: 2nd reference to page 46433
On tree page 42540 cell 5: 2nd reference to page 46348
On tree page 42540 cell 4: 2nd reference to page 46308
On tree page 42540 cell 3: Child page depth differs
On tree page 7572 cell 2: invalid page number 218103808
On tree page 2978 cell 1: Rowid 1526 out of order
On tree page 54756 cell 1: Rowid 47600 out of order
On tree page 54756 cell 1: 2nd reference to page 1099
On tree page 54756 cell 0: 2nd reference to page 54764
On tree page 2304 cell 2: invalid page number 218103808
On tree page 2309 cell 1: Rowid 1221 out of order
On tree page 51860 cell 1: 2nd reference to page 1179
On tree page 50331 cell 0: Rowid 1438 out of order
Page 2310: btreeInitPage() returns error code 11
Page 50332: btreeInitPage() returns error code 11
On tree page 3874 cell 1: Rowid 1135 out of order
On tree page 3873 cell 0: Rowid 1133 out of order
On tree page 3872 cell 0: Rowid 1132 out of order
On tree page 3871 cell 1: Rowid 1131 out of order
On tree page 52237 cell 4: invalid page number 218103808
Page 54754: btreeInitPage() returns error code 11
Page 370 is never used
Page 498 is never used
Page 529 is never used
Page 615 is never used
Page 630 is never used
Page 631 is never used
Page 1227 is never used
Page 2301 is never used
Page 2812 is never used
Page 2972 is never used
Page 2974 is never used
Page 2975 is never used
Page 2976 is never used
Page 2977 is never used
Page 3038 is never used
Page 3321 is never used
Page 6416 is never used
Page 6519 is never used
Page 6525 is never used
Page 6527 is never used
Page 6528 is never used
Page 6529 is never used
Page 6530 is never used
Page 6532 is never used
Page 6534 is never used
Page 6535 is never used
Page 6536 is never used
Page 6537 is never used
Page 6538 is never used
Page 6540 is never used
Page 6543 is never used
Page 6544 is never used
Page 6546 is never used
Page 6547 is never used
Page 6549 is never used
Page 6550 is never used
Page 6551 is never used
Page 6552 is never used
Page 6553 is never used
Page 6577 is never used
Page 6825 is never used
Page 11226 is never used
Page 11399 is never used
Page 11407 is never used
Page 11551 is never used
Page 49819 is never used
Page 49824 is never used
Page 49826 is never used
Page 49832 is never used
Page 49837 is never used
Page 49844 is never used
Page 49845 is never used
Page 49849 is never used

Last edited by Cootey; 08-18-2021 at 02:00 AM.
Cootey is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Need help in Australia (ebook purchase) heymae News 14 01-21-2013 10:33 PM
Young Adult Novel (eBook) - Ghost of Doors WrongDayToQuit Self-Promotions by Authors and Publishers 0 11-16-2012 07:56 PM
First ebook purchase! gumleaf Introduce Yourself 5 10-10-2010 06:22 PM
How to tell if an eBook is Topaz before purchase? JSWolf Amazon Kindle 42 11-13-2009 08:59 AM
Can't Purchase eBook outside of US? Jakodai Kindle Formats 2 01-15-2009 08:12 PM


All times are GMT -4. The time now is 03:07 AM.


MobileRead.com is a privately owned, operated and funded community.