Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Kobo Reader

Notices

Reply
 
Thread Tools Search this Thread
Old 07-01-2021, 09:56 AM   #16
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,088
Karma: 40026751
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
When connecting my Elipsa to my Mac, upon ejection the Elipsa will reimport all sideloaded books whether I actually sideloaded books in the Finder or just did a sync with the Kobo desktop. Once importing is finished, only sideloaded books remain, not listed in order of import order, but alphabetized instead. All purchased books have been removed from my Elipsa. Then, when I repair my account in Settings, the book thumbnails are returned, but I have to download them all again. This happens EACH time I connect my Kobo to the desktop app.
That is expected behaviour when the database on the device is corrupt. When that happens, the database file cannot be updated. When something happens to force a reread of the database, it is basically reading an old version of the database. That means it doesn't have any changes since the last successful write, so no new books or reading status.

The device can run for a while with a corrupt database. Most of what you see displayed when looking for books is from in-memory structures. If the write to the database fails, the device will generally keep working.

If you are using calibre, it can show an error when getting the list of books. But, the driver only reads a couple of tables, and depending on exactly where the corruption has occurred, it might not see it. You can do a check of the database to see if there are errors using any SQLite database manager. Or my Kobo Utilities plugin has a function to do it.

As to how the the database gets corrupted, the only way I know is not safely ejecting the device when the database is being updated. There has to be other times as people report problems and swear they always eject safely. There are almost definitely some mistakes, but, there are two may reports for it not to happen at some other time.

Signing out of the Kobo account on the device should fix this as a new database is created. And of course, a factory reset will do that as well.


Quote:
Originally Posted by Cootey View Post
Once importing is finished, only sideloaded books remain, not listed in order of import order, but alphabetized instead.
When importing sideloaded books, they should be added in alphabetical order using the file name, which, for me starts with the title. When the books is added to the device, a timestamp is set in the database. For sideloaded books, the same timestamp is used for all books imported in the same batch. The "Recent" sort uses this timestamp or the last read timestamp and it probably sorts by title within these.
davidfor is offline   Reply With Quote
Old 07-05-2021, 07:02 AM   #17
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 613
Karma: 3596061
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Quote:
Originally Posted by davidfor View Post
That is expected behaviour when the database on the device is corrupt. When that happens, the database file cannot be updated. When something happens to force a reread of the database, it is basically reading an old version of the database. That means it doesn't have any changes since the last successful write, so no new books or reading status.

The device can run for a while with a corrupt database. Most of what you see displayed when looking for books is from in-memory structures. If the write to the database fails, the device will generally keep working.
Well, *I* know the database was corrupt, but getting Kobo to agree was another matter. They have stopped responding to my emails, and starting today, new notebooks have started to disappear. Four full factory resets later, I thought maybe I had finally got things turned around, but each time some new feature starts to behave erratically. It was never the same corruption.

Quote:
If you are using calibre, it can show an error when getting the list of books. But, the driver only reads a couple of tables, and depending on exactly where the corruption has occurred, it might not see it. You can do a check of the database to see if there are errors using any SQLite database manager. Or my Kobo Utilities plugin has a function to do it.
I didn't realize that was built into your plug-in. I'll try it out. I do recall seeing calibre errors in previous incarnations. The current one works fine with calibre.

Öand a check of the database provided the following data:

Spoiler:
Result of running 'PRAGMA integrity_check' on database on the Kobo device:

*** in database main ***
On tree page 16290 cell 0: invalid page number 167772160
On tree page 15961 cell 2: 2nd reference to page 16598
On tree page 46691 cell 2: 2nd reference to page 16539
On tree page 419 cell 3: 2nd reference to page 50752
Page 690 is never used
Page 1049 is never used
Page 1367 is never used
row 1 missing from index sqlite_autoindex_content_1
row 2 missing from index sqlite_autoindex_content_1
row 3 missing from index sqlite_autoindex_content_1
row 4 missing from index sqlite_autoindex_content_1
row 5 missing from index sqlite_autoindex_content_1
row 6 missing from index sqlite_autoindex_content_1
row 7 missing from index sqlite_autoindex_content_1
row 8 missing from index sqlite_autoindex_content_1
row 9 missing from index sqlite_autoindex_content_1
row 10 missing from index sqlite_autoindex_content_1
row 11 missing from index sqlite_autoindex_content_1
row 12 missing from index sqlite_autoindex_content_1
row 13 missing from index sqlite_autoindex_content_1
row 14 missing from index sqlite_autoindex_content_1
row 15 missing from index sqlite_autoindex_content_1
row 16 missing from index sqlite_autoindex_content_1
row 17 missing from index sqlite_autoindex_content_1
row 18 missing from index sqlite_autoindex_content_1
row 19 missing from index sqlite_autoindex_content_1
row 20 missing from index sqlite_autoindex_content_1
row 21 missing from index sqlite_autoindex_content_1
row 22 missing from index sqlite_autoindex_content_1
row 23 missing from index sqlite_autoindex_content_1
row 24 missing from index sqlite_autoindex_content_1
row 25 missing from index sqlite_autoindex_content_1
row 26 missing from index sqlite_autoindex_content_1
row 27 missing from index sqlite_autoindex_content_1
row 28 missing from index sqlite_autoindex_content_1
row 29 missing from index sqlite_autoindex_content_1
row 30 missing from index sqlite_autoindex_content_1
row 31 missing from index sqlite_autoindex_content_1
row 32 missing from index sqlite_autoindex_content_1
row 33 missing from index sqlite_autoindex_content_1
row 34 missing from index sqlite_autoindex_content_1
row 35 missing from index sqlite_autoindex_content_1
row 36 missing from index sqlite_autoindex_content_1
row 37 missing from index sqlite_autoindex_content_1
row 38 missing from index sqlite_autoindex_content_1
row 39 missing from index sqlite_autoindex_content_1
row 40 missing from index sqlite_autoindex_content_1
row 41 missing from index sqlite_autoindex_content_1
row 42 missing from index sqlite_autoindex_content_1
row 43 missing from index sqlite_autoindex_content_1
row 44 missing from index sqlite_autoindex_content_1
row 45 missing from index sqlite_autoindex_content_1
row 46 missing from index sqlite_autoindex_content_1
row 47 missing from index sqlite_autoindex_content_1
row 48 missing from index sqlite_autoindex_content_1
row 49 missing from index sqlite_autoindex_content_1
row 50 missing from index sqlite_autoindex_content_1
row 51 missing from index sqlite_autoindex_content_1
row 52 missing from index sqlite_autoindex_content_1
row 53 missing from index sqlite_autoindex_content_1
row 54 missing from index sqlite_autoindex_content_1
row 55 missing from index sqlite_autoindex_content_1
row 56 missing from index sqlite_autoindex_content_1
row 57 missing from index sqlite_autoindex_content_1
row 58 missing from index sqlite_autoindex_content_1
row 59 missing from index sqlite_autoindex_content_1
row 60 missing from index sqlite_autoindex_content_1
row 61 missing from index sqlite_autoindex_content_1
row 62 missing from index sqlite_autoindex_content_1
row 63 missing from index sqlite_autoindex_content_1
row 64 missing from index sqlite_autoindex_content_1
row 65 missing from index sqlite_autoindex_content_1
row 66 missing from index sqlite_autoindex_content_1
row 67 missing from index sqlite_autoindex_content_1
row 68 missing from index sqlite_autoindex_content_1
row 69 missing from index sqlite_autoindex_content_1
row 70 missing from index sqlite_autoindex_content_1
row 71 missing from index sqlite_autoindex_content_1
row 72 missing from index sqlite_autoindex_content_1
row 73 missing from index sqlite_autoindex_content_1
row 74 missing from index sqlite_autoindex_content_1
row 75 missing from index sqlite_autoindex_content_1
row 76 missing from index sqlite_autoindex_content_1
row 77 missing from index sqlite_autoindex_content_1
row 78 missing from index sqlite_autoindex_content_1
row 79 missing from index sqlite_autoindex_content_1
row 80 missing from index sqlite_autoindex_content_1
row 81 missing from index sqlite_autoindex_content_1
row 82 missing from index sqlite_autoindex_content_1
row 83 missing from index sqlite_autoindex_content_1
row 84 missing from index sqlite_autoindex_content_1
row 85 missing from index sqlite_autoindex_content_1
row 86 missing from index sqlite_autoindex_content_1
row 87 missing from index sqlite_autoindex_content_1
row 88 missing from index sqlite_autoindex_content_1
row 89 missing from index sqlite_autoindex_content_1
row 90 missing from index sqlite_autoindex_content_1
row 91 missing from index sqlite_autoindex_content_1
row 92 missing from index sqlite_autoindex_content_1
row 93 missing from index sqlite_autoindex_content_1


Quote:
As to how the the database gets corrupted, the only way I know is not safely ejecting the device when the database is being updated. There has to be other times as people report problems and swear they always eject safely. There are almost definitely some mistakes, but, there are two may reports for it not to happen at some other time.

Signing out of the Kobo account on the device should fix this as a new database is created. And of course, a factory reset will do that as well.
In this case, considering that I factory reset from the dev menu four times, there was not an instance of me ejecting the device inappropriately, something I don't do. No amount of repairing the account, or logging out then back in, has made these glitches go away. There is a serious issue with my unit. Perhaps a bad memory module? I really can't say.

Quote:
When importing sideloaded books, they should be added in alphabetical order using the file name, which, for me starts with the title. When the books is added to the device, a timestamp is set in the database. For sideloaded books, the same timestamp is used for all books imported in the same batch. The "Recent" sort uses this timestamp or the last read timestamp and it probably sorts by title within these.
Although I often rely on this alphabetization when importing books to be added to a collection, when ALL books are reimported as if they hadn't been imported before, finding a dozen books alphabetized amidst 400+ books is a chore. This is what I was referring to. All books, including previously imported books, were being reimported without regard to that timestamp. However, I no longer have that problem. A full factory reset fixed it. Now I have disappearing journals, something that is frustrating since I drew on those pages and put a lot of effort into them.

FYI, I ran an undeleting program on the Kobo Elipsa when it was mounted on my Desktop, and it found all the missing *.nebo files. The only problem was that they were all 0 bytes large. They never properly saved to the internal memory. Here I thought I finally fixed it, then the glitch moved yet again. This time to an area I hadn't tested in over a week: notebooks. I have requested a new Elipsa pack from Kobo. Things are complicated since the US store showed that it was sold out earlier today.

Thanks for all your feedback.
Cootey is offline   Reply With Quote
Old 07-05-2021, 08:32 AM   #18
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,088
Karma: 40026751
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
Well, *I* know the database was corrupt, but getting Kobo to agree was another matter. They have stopped responding to my emails, and starting today, new notebooks have started to disappear. Four full factory resets later, I thought maybe I had finally got things turned around, but each time some new feature starts to behave erratically. It was never the same corruption.



I didn't realize that was built into your plug-in. I'll try it out. I do recall seeing calibre errors in previous incarnations. The current one works fine with calibre.

Öand a check of the database provided the following data:

Spoiler:
Result of running 'PRAGMA integrity_check' on database on the Kobo device:

*** in database main ***
On tree page 16290 cell 0: invalid page number 167772160
On tree page 15961 cell 2: 2nd reference to page 16598
On tree page 46691 cell 2: 2nd reference to page 16539
On tree page 419 cell 3: 2nd reference to page 50752
Page 690 is never used
Page 1049 is never used
Page 1367 is never used
row 1 missing from index sqlite_autoindex_content_1
row 2 missing from index sqlite_autoindex_content_1
row 3 missing from index sqlite_autoindex_content_1
row 4 missing from index sqlite_autoindex_content_1
row 5 missing from index sqlite_autoindex_content_1
row 6 missing from index sqlite_autoindex_content_1
row 7 missing from index sqlite_autoindex_content_1
row 8 missing from index sqlite_autoindex_content_1
row 9 missing from index sqlite_autoindex_content_1
row 10 missing from index sqlite_autoindex_content_1
row 11 missing from index sqlite_autoindex_content_1
row 12 missing from index sqlite_autoindex_content_1
row 13 missing from index sqlite_autoindex_content_1
row 14 missing from index sqlite_autoindex_content_1
row 15 missing from index sqlite_autoindex_content_1
row 16 missing from index sqlite_autoindex_content_1
row 17 missing from index sqlite_autoindex_content_1
row 18 missing from index sqlite_autoindex_content_1
row 19 missing from index sqlite_autoindex_content_1
row 20 missing from index sqlite_autoindex_content_1
row 21 missing from index sqlite_autoindex_content_1
row 22 missing from index sqlite_autoindex_content_1
row 23 missing from index sqlite_autoindex_content_1
row 24 missing from index sqlite_autoindex_content_1
row 25 missing from index sqlite_autoindex_content_1
row 26 missing from index sqlite_autoindex_content_1
row 27 missing from index sqlite_autoindex_content_1
row 28 missing from index sqlite_autoindex_content_1
row 29 missing from index sqlite_autoindex_content_1
row 30 missing from index sqlite_autoindex_content_1
row 31 missing from index sqlite_autoindex_content_1
row 32 missing from index sqlite_autoindex_content_1
row 33 missing from index sqlite_autoindex_content_1
row 34 missing from index sqlite_autoindex_content_1
row 35 missing from index sqlite_autoindex_content_1
row 36 missing from index sqlite_autoindex_content_1
row 37 missing from index sqlite_autoindex_content_1
row 38 missing from index sqlite_autoindex_content_1
row 39 missing from index sqlite_autoindex_content_1
row 40 missing from index sqlite_autoindex_content_1
row 41 missing from index sqlite_autoindex_content_1
row 42 missing from index sqlite_autoindex_content_1
row 43 missing from index sqlite_autoindex_content_1
row 44 missing from index sqlite_autoindex_content_1
row 45 missing from index sqlite_autoindex_content_1
row 46 missing from index sqlite_autoindex_content_1
row 47 missing from index sqlite_autoindex_content_1
row 48 missing from index sqlite_autoindex_content_1
row 49 missing from index sqlite_autoindex_content_1
row 50 missing from index sqlite_autoindex_content_1
row 51 missing from index sqlite_autoindex_content_1
row 52 missing from index sqlite_autoindex_content_1
row 53 missing from index sqlite_autoindex_content_1
row 54 missing from index sqlite_autoindex_content_1
row 55 missing from index sqlite_autoindex_content_1
row 56 missing from index sqlite_autoindex_content_1
row 57 missing from index sqlite_autoindex_content_1
row 58 missing from index sqlite_autoindex_content_1
row 59 missing from index sqlite_autoindex_content_1
row 60 missing from index sqlite_autoindex_content_1
row 61 missing from index sqlite_autoindex_content_1
row 62 missing from index sqlite_autoindex_content_1
row 63 missing from index sqlite_autoindex_content_1
row 64 missing from index sqlite_autoindex_content_1
row 65 missing from index sqlite_autoindex_content_1
row 66 missing from index sqlite_autoindex_content_1
row 67 missing from index sqlite_autoindex_content_1
row 68 missing from index sqlite_autoindex_content_1
row 69 missing from index sqlite_autoindex_content_1
row 70 missing from index sqlite_autoindex_content_1
row 71 missing from index sqlite_autoindex_content_1
row 72 missing from index sqlite_autoindex_content_1
row 73 missing from index sqlite_autoindex_content_1
row 74 missing from index sqlite_autoindex_content_1
row 75 missing from index sqlite_autoindex_content_1
row 76 missing from index sqlite_autoindex_content_1
row 77 missing from index sqlite_autoindex_content_1
row 78 missing from index sqlite_autoindex_content_1
row 79 missing from index sqlite_autoindex_content_1
row 80 missing from index sqlite_autoindex_content_1
row 81 missing from index sqlite_autoindex_content_1
row 82 missing from index sqlite_autoindex_content_1
row 83 missing from index sqlite_autoindex_content_1
row 84 missing from index sqlite_autoindex_content_1
row 85 missing from index sqlite_autoindex_content_1
row 86 missing from index sqlite_autoindex_content_1
row 87 missing from index sqlite_autoindex_content_1
row 88 missing from index sqlite_autoindex_content_1
row 89 missing from index sqlite_autoindex_content_1
row 90 missing from index sqlite_autoindex_content_1
row 91 missing from index sqlite_autoindex_content_1
row 92 missing from index sqlite_autoindex_content_1
row 93 missing from index sqlite_autoindex_content_1
That might be recoverable with a VACCUUM. Which is what the compress database function in the my Kobo Utilities plugin does. If the error is just in indexes, this will work, but, the first few errors might prevent it.
Quote:
In this case, considering that I factory reset from the dev menu four times, there was not an instance of me ejecting the device inappropriately, something I don't do. No amount of repairing the account, or logging out then back in, has made these glitches go away. There is a serious issue with my unit. Perhaps a bad memory module? I really can't say.
It feels like a hardware problem. I think you need to get it replaced.
Quote:
Although I often rely on this alphabetization when importing books to be added to a collection, when ALL books are reimported as if they hadn't been imported before, finding a dozen books alphabetized amidst 400+ books is a chore. This is what I was referring to. All books, including previously imported books, were being reimported without regard to that timestamp. However, I no longer have that problem. A full factory reset fixed it. Now I have disappearing journals, something that is frustrating since I drew on those pages and put a lot of effort into them.
When I reload a device, I use set the "Date added" to something useful after the import. I swap between the published date and the "Date" column in calibre. The latter is roughly the date I got the book, so it works fairly well for me.
Quote:
FYI, I ran an undeleting program on the Kobo Elipsa when it was mounted on my Desktop, and it found all the missing *.nebo files. The only problem was that they were all 0 bytes large. They never properly saved to the internal memory. Here I thought I finally fixed it, then the glitch moved yet again. This time to an area I hadn't tested in over a week: notebooks. I have requested a new Elipsa pack from Kobo. Things are complicated since the US store showed that it was sold out earlier today.
Yes, it needs to be replaced.
davidfor is offline   Reply With Quote
Old 07-17-2021, 12:56 AM   #19
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 613
Karma: 3596061
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
My new Elipsa arrived. Other people here have complained about database issues. So far, one other person gave up on the Elipsa and returned theirs to the store. I rather love the Elipsa, so I opted to get another one. I'll update this thread if I start to see disappearing books again.

For now, I've setup my Elipsa with a few hundred books, I restored the only notebook that I was able to save from my other Elipsa, and now I'm running my Elipsa through it's paces.

When I first set it up with the Kobo Desktop app, it got stuck rebooting for a few minutes. Kobo Desktop had updated its firmware. When the Elipsa finally booted, the What's New message was there with its sparse details, then I tried to sync to the server and it said it needed to update the firmware again. Then more downloading and booting, and the What's New message with the same sparse details reappeared. I can't say that it was a beginning that inspired confidence, but after all that, I am up and running. Crossing fingers.
Cootey is offline   Reply With Quote
Old 07-17-2021, 05:50 AM   #20
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 613
Karma: 3596061
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Well, it didn't take me long. I've run into the corrupt database/deleted downloads issue again. Once the Elipsa is ejected from my Mac, it reimports ALL sideloaded books again, deletes all 242 purchased books, and resets the home screen. Repairing the account restores everything but the downloaded ebooks.

Setting my Elipsa up with Kobo Desktop worked fine.
Manually sideloading books by dragging and dropping in the Finder was fine.
Downloading through Overdrive was fine.
I ran into problems the moment I started to use Calibre to import books. Here's the error:

Code:
calibre, version 5.23.0
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
  File "calibre/gui2/device.py", line 89, in run
  File "calibre/gui2/device.py", line 515, in _books
  File "calibre/devices/kobo/driver.py", line 1974, in books
apsw.CorruptError: CorruptError: database disk image is malformed
I could use some suggestions on how to troubleshoot and identify what is going wrong. I don't want to abandon Calibre and davidfor's plugin. They're too useful. Right now I'm trying to restore from a backup to a state where everything seemed to be working. If that doesn't work, I'll log out of my Elipsa, then log back in. Then factory reset as a last resort.
Cootey is offline   Reply With Quote
Old 07-17-2021, 09:44 AM   #21
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,088
Karma: 40026751
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
Well, it didn't take me long. I've run into the corrupt database/deleted downloads issue again. Once the Elipsa is ejected from my Mac, it reimports ALL sideloaded books again, deletes all 242 purchased books, and resets the home screen. Repairing the account restores everything but the downloaded ebooks.

Setting my Elipsa up with Kobo Desktop worked fine.
Manually sideloading books by dragging and dropping in the Finder was fine.
Downloading through Overdrive was fine.
I ran into problems the moment I started to use Calibre to import books. Here's the error:

Code:
calibre, version 5.23.0
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
  File "calibre/gui2/device.py", line 89, in run
  File "calibre/gui2/device.py", line 515, in _books
  File "calibre/devices/kobo/driver.py", line 1974, in books
apsw.CorruptError: CorruptError: database disk image is malformed
I could use some suggestions on how to troubleshoot and identify what is going wrong. I don't want to abandon Calibre and davidfor's plugin. They're too useful. Right now I'm trying to restore from a backup to a state where everything seemed to be working. If that doesn't work, I'll log out of my Elipsa, then log back in. Then factory reset as a last resort.
The problem is knowing when the database became corrupted. The error is at the start of the loop reading the database. It is probably happening on the first record, but, might be happening after reading some of the book records. And that probably means the database is already corrupted when calibre sees it.

Can you test the database before using calibre? Any SQLite database manager should be able to do it. They might have an option to do the check. But, there isn't, then run "PRAGMA integrity_check". You can try DB Browser for SQLite as that has a MAC version.

If the database is fine, run calibre in debug mode. That will let us see if it is breaking when the database is first opened, or after reading some records. You will need to start calibre with the command "calibre-debug -g" to do this.

You could also check the database before ejecting the device. This is to confirm that calibre did not do anything bad.

Other thoughts are:

Could it be the MAC? If you have another computer to try, it would be good to do soo.

Are you ever powering off?

Are there any log files being produced in the .kobo directory? I'm mainly wondering about stack dumps which will have the name "stack_xx.log" where xx is two numbers. If these exist, they are a sign of a crash happening somewhere.

Does the partition show as being dirty when you connect? I don't know if the MAC will show this. Windows would offer to check the partition if it is dirty.
davidfor is offline   Reply With Quote
Old 07-17-2021, 08:10 PM   #22
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 613
Karma: 3596061
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Quote:
Originally Posted by davidfor View Post
The problem is knowing when the database became corrupted. The error is at the start of the loop reading the database. It is probably happening on the first record, but, might be happening after reading some of the book records. And that probably means the database is already corrupted when calibre sees it.

Can you test the database before using calibre? Any SQLite database manager should be able to do it. They might have an option to do the check. But, there isn't, then run "PRAGMA integrity_check". You can try DB Browser for SQLite as that has a MAC version.
I had DB Browser for SQLite already installed, so I copied the Elipsa KoboReader.sqlite and ran the "PRAGMA integrity_check" on it.

Code:
Execution finished without errors.
Result: 1 rows returned in 3144ms
At line 1:
PRAGMA integrity_check;
Because I backed up to an earlier version before things went kablooie, my database is fine.

Quote:
Originally Posted by davidfor View Post
If the database is fine, run calibre in debug mode. That will let us see if it is breaking when the database is first opened, or after reading some records. You will need to start calibre with the command "calibre-debug -g" to do this.
You could also check the database before ejecting the device. This is to confirm that calibre did not do anything bad.
Running in debug mode now. Thanks.

I used Calibre to copy one book over and checked the database before ejecting. Everything is A-OK.

Quote:
Originally Posted by davidfor View Post
Other thoughts are:

Could it be the MAC? If you have another computer to try, it would be good to do soo.

Are you ever powering off?

Are there any log files being produced in the .kobo directory? I'm mainly wondering about stack dumps which will have the name "stack_xx.log" where xx is two numbers. If these exist, they are a sign of a crash happening somewhere.

Does the partition show as being dirty when you connect? I don't know if the MAC will show this. Windows would offer to check the partition if it is dirty.
1) I don't have problems like this with either of our Kobo Clara HDs, or either Kobo Aura One in the past. I wasn't exactly paying attention to what I was doing just prior to getting the database error in Calibre, but IIRC, the error popped up when I plugged the Elipsa in after an evening of setup. That might indicate the problem happened on the Elipsa. I'll watch for it.
2) Powering off my Mac while the Elipsa is plugged in? No. I keep my Mac on at all times. I don't even let it sleep.
3) No log files as you described.
4) From what I've read, Mac tools for dealing with ExFat dirty drives is minimal. I'll have to see if that is the case if the corruption should occur again.

Thanks!

Last edited by Cootey; 07-17-2021 at 08:26 PM.
Cootey is offline   Reply With Quote
Old 07-17-2021, 10:08 PM   #23
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,088
Karma: 40026751
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
I had DB Browser for SQLite already installed, so I copied the Elipsa KoboReader.sqlite and ran the "PRAGMA integrity_check" on it.

Code:
Execution finished without errors.
Result: 1 rows returned in 3144ms
At line 1:
PRAGMA integrity_check;
Because I backed up to an earlier version before things went kablooie, my database is fine.
Unfortunately, you will need to do this check each time. We don't know when the corruption is happening. I'm trying to eliminate calibre as the cause. Doing the check a few times should help with that.
Quote:
Running in debug mode now. Thanks.

I used Calibre to copy one book over and checked the database before ejecting. Everything is A-OK.



1) I don't have problems like this with either of our Kobo Clara HDs, or either Kobo Aura One in the past. I wasn't exactly paying attention to what I was doing just prior to getting the database error in Calibre, but IIRC, the error popped up when I plugged the Elipsa in after an evening of setup. That might indicate the problem happened on the Elipsa. I'll watch for it.
2) Powering off my Mac while the Elipsa is plugged in? No. I keep my Mac on at all times. I don't even let it sleep.
No I meant powering off the Elipsa. If it doesn't get shutdown properly, that could be a cause. And when the database is read, that can be when problems are noticed such as missing books and reading status.
Quote:
3) No log files as you described.
4) From what I've read, Mac tools for dealing with ExFat dirty drives is minimal. I'll have to see if that is the case if the corruption should occur again.
I have no idea what tools are available on the MAC for this, but I had a feeling that there wasn't much. Kobo did update dosfstools last year which solved some problems with the PC seeing the drive as dirty. I don't really expect this to be the problem, but it would be good to eliminate.
davidfor is offline   Reply With Quote
Old 07-22-2021, 12:07 AM   #24
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 613
Karma: 3596061
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Hereís an update:

Iíve enabled debug mode for Calibre, and I check the database upon mounting and before ejecting. Iíve also checked the Elipsa database with SQLite software. No problems have been found. I also constantly backup my Elipsa so I donít lose data.

With all the database issues Iíve run into, I can report that many ebook annotations and page locations have not been lost, but the Elipsa has forgotten that Iím reading quite a few books. Those ebooks still retain their annotations, but not current page location. Fortunately, bookmarks are intact, too.

Kobo backs up our stylus written annotations so that they are restored when logging into a reset or new Kobo. Notebooks are goners, unfortunately.

I have had one close call that didnít require restoring the database. After ejecting my Elipsa using Calibre at the end of last week, the Elipsa powered off completely. When I tried to turn it back on, it didnít respond. I pushed the on button a few times, waiting for the Elipsa to respond after each press, but only after the fourth time did it power up, then acted as it had meant to do that, everything was perfectly fine, and would you please stop staring at me?

Iím watching it with a suspicious side eye now, but the Elipsa has been behaving without issue since then.


Sent from my iPhone using Tapatalk Pro
Cootey is offline   Reply With Quote
Old 07-22-2021, 08:44 AM   #25
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 613
Karma: 3596061
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Alright. I've got a corrupt database again. I was working with Kobo tech over a different issue (Elipsa doesn't prompt to review or rate purchased books when they are finished). I was asked to connect my Elipsa to Kobo Desktop on my Mac. It synced fine. No books were added. I cmd-shift-i'd my log file. Then I got busy working on something else. When I ejected my Elipsa properly, it suddenly reimported EVERYTHING all over again. All my purchased downloads have disappeared, again.

I connected it to the Mac again, this time with Calibre. I immediately got the ERROR: Error dialog letting me know "Error communicating with device".

Here is the clipboard error output:

Code:
calibre, version 5.23.0
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
  File "calibre/gui2/device.py", line 89, in run
  File "calibre/gui2/device.py", line 515, in _books
  File "calibre/devices/kobo/driver.py", line 1974, in books
apsw.CorruptError: CorruptError: database disk image is malformed
If I repair the account on the Kobo, I know it will be fixed. Then I have to redownload everything again. Instead, I have opted to try to check the database using Calibre.

Result of running 'PRAGMA integrity_check' on database on the Kobo device:


Spoiler:
*** in database main ***
On tree page 16876 cell 1: Rowid 393 out of order
On tree page 44734 cell 1: Rowid 348 out of order
On tree page 44734 cell 0: 2nd reference to page 49801
On tree page 51788 cell 1: 2nd reference to page 51790
On tree page 529 cell 2: Rowid 40093 out of order
Page 50211: btreeInitPage() returns error code 11
On tree page 29598 cell 103: Child page depth differs
On tree page 50155 cell 0: invalid page number 167850496
Page 1150: btreeInitPage() returns error code 11
Page 543: btreeInitPage() returns error code 11
Page 520: btreeInitPage() returns error code 11
Page 463: btreeInitPage() returns error code 11
Page 457: btreeInitPage() returns error code 11
Page 497: btreeInitPage() returns error code 11
On tree page 526 cell 3: invalid page number 218103808
Page 505: btreeInitPage() returns error code 11
On tree page 481 cell 0: invalid page number 218103808
Page 50171: btreeInitPage() returns error code 11
Page 971: btreeInitPage() returns error code 11
Page 964: btreeInitPage() returns error code 11
Page 50176: btreeInitPage() returns error code 11
Page 50204: btreeInitPage() returns error code 11
On tree page 130 cell 3: Rowid 39762 out of order
On tree page 130 cell 3: 2nd reference to page 50274
On tree page 50166 cell 39: Rowid 39672 out of order
On tree page 1260 cell 0: Rowid 41643 out of order
Page 50157: btreeInitPage() returns error code 11
Page 50215: btreeInitPage() returns error code 11
On tree page 1165 cell 5: invalid page number 218103808
On tree page 1165 cell 0: invalid page number 218103808
Page 319: btreeInitPage() returns error code 11
On tree page 978 cell 4: Rowid 41636 out of order
Page 539: btreeInitPage() returns error code 11
Page 8888: btreeInitPage() returns error code 11
Page 1134: btreeInitPage() returns error code 11
On tree page 50190 cell 5: Rowid 41611 out of order
On tree page 50190 cell 4: 2nd reference to page 51806
On tree page 50190 cell 3: 2nd reference to page 50269
On tree page 50262 cell 0: 2nd reference to page 579
Page 1129: btreeInitPage() returns error code 11
On tree page 50170 cell 5: Rowid 41605 out of order
On tree page 50170 cell 0: 2nd reference to page 640
On tree page 50260 cell 4: invalid page number 218103808
On tree page 50260 cell 1: invalid page number 218103808
On tree page 979 cell 2: Rowid 41572 out of order
On tree page 979 cell 2: 2nd reference to page 1290
On tree page 979 cell 1: 2nd reference to page 50171
Page 469: btreeInitPage() returns error code 11
Page 931: btreeInitPage() returns error code 11
Page 232: btreeInitPage() returns error code 11
Page 8893: btreeInitPage() returns error code 11
Page 566: btreeInitPage() returns error code 11
On tree page 50159 cell 2: Rowid 41559 out of order
On tree page 50159 cell 1: 2nd reference to page 566
Page 50172: btreeInitPage() returns error code 11
On tree page 50259 cell 0: 2nd reference to page 1485
On tree page 10131 cell 3: Rowid 41437 out of order
On tree page 10131 cell 3: invalid page number 218103808
Page 547: btreeInitPage() returns error code 11
On tree page 50258 cell 4: invalid page number 218103808
On tree page 50258 cell 3: 2nd reference to page 482
On tree page 50258 cell 1: 2nd reference to page 366
Page 230: btreeInitPage() returns error code 11
Page 2303: btreeInitPage() returns error code 11
On tree page 957 cell 4: Rowid 41536 out of order
Page 1198: btreeInitPage() returns error code 11
Page 1434: btreeInitPage() returns error code 11
Page 518: btreeInitPage() returns error code 11
Page 1348: btreeInitPage() returns error code 11
Page 509: btreeInitPage() returns error code 11
On tree page 476 cell 3: invalid page number 218103808
Page 1302: btreeInitPage() returns error code 11
Page 18616: btreeInitPage() returns error code 11
Page 461: btreeInitPage() returns error code 11
On tree page 29598 cell 102: Child page depth differs
On tree page 459 cell 5: Rowid 41661 out of order
On tree page 459 cell 3: 2nd reference to page 1161
On tree page 459 cell 2: invalid page number 167884544
On tree page 501 cell 3: Rowid 41502 out of order
On tree page 501 cell 2: 2nd reference to page 18616
On tree page 501 cell 0: 2nd reference to page 523
Page 618: btreeInitPage() returns error code 11
On tree page 363 cell 2: 2nd reference to page 612
On tree page 363 cell 1: 2nd reference to page 1573
On tree page 363 cell 0: 2nd reference to page 1398
On tree page 44802 cell 79: 2nd reference to page 45116
On tree page 44802 cell 52: Rowid 35248 out of order
On tree page 44802 cell 52: 2nd reference to page 45112
On tree page 44802 cell 51: 2nd reference to page 45105
On tree page 44802 cell 50: 2nd reference to page 45103
On tree page 44802 cell 49: 2nd reference to page 45101
On tree page 44802 cell 48: 2nd reference to page 45091
On tree page 44802 cell 47: 2nd reference to page 45090
On tree page 44802 cell 46: 2nd reference to page 45085
On tree page 44802 cell 45: 2nd reference to page 45081
On tree page 44802 cell 44: 2nd reference to page 45079
On tree page 44802 cell 43: 2nd reference to page 45076
On tree page 44802 cell 42: 2nd reference to page 45071
On tree page 44802 cell 41: 2nd reference to page 45070
On tree page 44802 cell 40: 2nd reference to page 45058


This is an annoying bug. Since the Elipsa was mounted on my desktop for an hour, I wonder if Spotlight's search indexing caused a problem. I have now excluded KOBOeReader from being indexed. But this could be a Kobo Desktop problem, too. Any suggestions on how to isolate who is at fault?

How fortunate for me that I just did a backup half an hour ago before this happened.

Last edited by Cootey; 07-22-2021 at 05:48 PM.
Cootey is offline   Reply With Quote
Old 07-22-2021, 09:37 AM   #26
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,088
Karma: 40026751
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
The act of reimporting everything suggests the database was corrupted when the device was ejected. Whether that was the Kobo desktop or something else running, I don't know. If something else was accessing the database when the Kobo desktop was, that would be a likely culprit. If that was also running during your calibre sessions, that could also explain the other problems you have been seeing,
davidfor is offline   Reply With Quote
Old 07-22-2021, 06:12 PM   #27
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 613
Karma: 3596061
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Quote:
Originally Posted by davidfor View Post
The act of reimporting everything suggests the database was corrupted when the device was ejected. Whether that was the Kobo desktop or something else running, I don't know. If something else was accessing the database when the Kobo desktop was, that would be a likely culprit. If that was also running during your calibre sessions, that could also explain the other problems you have been seeing,
I posted the clipboard error message and the 'PRAGMA integrity_check' results for you above. Was there anything else you'd need in the future?

Since I began bumping into these issues, I don't run Kobo desktop and Calibre at the same time. Also, there are no active programs operating on the Kobo when I eject it. The Mac won’t allow it. I can’t even eject it if the Elipsa is simply the current directory in Terminal. I also plug the Elipsa directly into the back of my Mac instead of using a USB hub to eliminate potential issues there. I never have these problems with my Clara. I haven't treated my Elipsa any different, except now I treat it as if its made of handblown Egyptian glass when I bring it near my Mac.

I usually check the database first in Calibre before ejecting it. I may have forgotten to do that this time, but I’ve ejected directly from within Kobo Desktop before with no issues.

That all being said, since I last posted above I had to restore my Kobo. Due to our past conversation in another thread, I’ve been experimenting with using rsync instead of dd to make a backup. So I used cp to create the backup archive, then planned to rsync the archive back onto the Kobo in the event of a database corruption. I had a full disk image backup at the ready, so this was my first time to experiment using rsync to restore the Kobo.

While it was running, it deleted a ton of ._ files off the Elipsa. Usually, the Mac creates those files every time you access a directory from the Finder. However, there were ._ files in directories I had never accessed from Finder. To be clear, these were on the Kobo and didn’t exist in the backup. That’s why rsync removed them. That’s probably obvious to you, but I’m trying to be thorough. They were created between the time I made the backup and the time I ejected the Kobo.

So what process is accessing the Kobo? Prior to your suggestion, I was already puzzling over this. TimeMachine apparently can store data in ._ files, but I don't use TimeMachine on USB external drives. I blocked Spotlight from indexing the Kobo whenever it is mounted. Spotlight only indexes for searching, however. It doesn’t create ._ files. I have never seen this issue with any USB mounted device before, and I can’t be certain this is the issue that caused the corruption. The Kobo tends to ignore ._ files in my experience.

This is a lot of text, but as I said above, I'm trying to be thorough. I know a couple of people have bumped into corrupt Elipsa databases. One person gave up and returned his Elipsa to the store. We should be finding out if they use Macs, Kobo Desktop, and Calibre to find commonalities. And the big question is: Why is only my Elipsa having these issues and not either of our Claras?

I can't imagine somebody who is not a geek trying to deal with this.

Last edited by Cootey; 07-22-2021 at 06:21 PM.
Cootey is offline   Reply With Quote
Old 07-23-2021, 03:02 AM   #28
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 22,088
Karma: 40026751
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by Cootey View Post
I posted the clipboard error message and the 'PRAGMA integrity_check' results for you above. Was there anything else you'd need in the future?
There isn't much I can do with the results of the integrity check. I can tell from a quick look if they are likely to be fixed by doing a VACUUM. That will be the case if the errors are only in the indexes or with free space. With the results you have, I doubt it can be fixed.
Quote:
Since I began bumping into these issues, I don't run Kobo desktop and Calibre at the same time. Also, there are no active programs operating on the Kobo when I eject it. The Mac wonít allow it. I canít even eject it if the Elipsa is simply the current directory in Terminal. I also plug the Elipsa directly into the back of my Mac instead of using a USB hub to eliminate potential issues there. I never have these problems with my Clara. I haven't treated my Elipsa any different, except now I treat it as if its made of handblown Egyptian glass when I bring it near my Mac.

I usually check the database first in Calibre before ejecting it. I may have forgotten to do that this time, but Iíve ejected directly from within Kobo Desktop before with no issues.

That all being said, since I last posted above I had to restore my Kobo. Due to our past conversation in another thread, Iíve been experimenting with using rsync instead of dd to make a backup. So I used cp to create the backup archive, then planned to rsync the archive back onto the Kobo in the event of a database corruption. I had a full disk image backup at the ready, so this was my first time to experiment using rsync to restore the Kobo.

While it was running, it deleted a ton of ._ files off the Elipsa. Usually, the Mac creates those files every time you access a directory from the Finder. However, there were ._ files in directories I had never accessed from Finder. To be clear, these were on the Kobo and didnít exist in the backup. Thatís why rsync removed them. Thatís probably obvious to you, but Iím trying to be thorough. They were created between the time I made the backup and the time I ejected the Kobo.

So what process is accessing the Kobo? Prior to your suggestion, I was already puzzling over this. TimeMachine apparently can store data in ._ files, but I don't use TimeMachine on USB external drives. I blocked Spotlight from indexing the Kobo whenever it is mounted. Spotlight only indexes for searching, however. It doesnít create ._ files. I have never seen this issue with any USB mounted device before, and I canít be certain this is the issue that caused the corruption. The Kobo tends to ignore ._ files in my experience.
The corruption could happen from:
  • An application crashing doing during the write.
  • Two applications attempting to access the database at the same time.
  • Failing or otherwise damaged storage.
  • File system corruption by something.
  • Something making changes at the file system level.
  • Device being disconnected before all writes have been completed.

Unfortunately, I just don't know when it happens. A possibility is a background task on the MAC clashing. But, you should be able to see that.

Another thing to try is to not eject from within the applications, either calibre or the Kobo desktop. Close these and wait a couple of minutes. That should make sure nothing that you are doing trying to update, and it should let any caches flush as well. I don't know the MAC, but, Windows has, or at least had, an option for "fast eject" or something like that. Turning it on, meant that writes to removable media were immediate and not cached.

Quote:
This is a lot of text, but as I said above, I'm trying to be thorough. I know a couple of people have bumped into corrupt Elipsa databases. One person gave up and returned his Elipsa to the store. We should be finding out if they use Macs, Kobo Desktop, and Calibre to find commonalities. And the big question is: Why is only my Elipsa having these issues and not either of our Claras?
Unfortunately, there are always corrupt databases. But, the number with the Elipsa seems to be a lot worse. It might be something in the firmware. If it is, it will show when Kobo roll these updates out to other devices. Or it could be something about the hardware. There is enough different hardware in them (USB-C, multi-core CPU etc) that there might be something that they have missed. If it is caused by something during the connection to the PC, maybe it is one of the processes on the device not stopping cleanly and contending with the database access from the PC. The obvious culprit would be something in the note taking software. That is new and not on the other devices. And the only way to test that is to not use it for a while.
davidfor is offline   Reply With Quote
Old 07-23-2021, 07:36 AM   #29
Cootey
Absentminded Reader
Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.Cootey ought to be getting tired of karma fortunes by now.
 
Cootey's Avatar
 
Posts: 613
Karma: 3596061
Join Date: Apr 2017
Device: 2 Kobo Claras, Kobo Elipsa, Kindle Paperwhite 3 & 4, iPad 6 & iPhoneXS
Quote:
Originally Posted by davidfor View Post
Another thing to try is to not eject from within the applications, either calibre or the Kobo desktop. Close these and wait a couple of minutes. That should make sure nothing that you are doing trying to update, and it should let any caches flush as well. I don't know the MAC, but, Windows has, or at least had, an option for "fast eject" or something like that. Turning it on, meant that writes to removable media were immediate and not cached.
Good response. Thank you. There IS a lot new with the Elipsa compared to other Kobo eReaders, and since I'm using the same software and computer with my Clara as I do my Elipsa, and the Clara soldiers on without incident, I'm thinking the Elipsa has an issue. I am crossing my fingers that it's something they can fix in firmware, unlike the backlighting issues with the Forma that people just had to live with. I wish I could isolate what is triggering this issue, but I'll keep a watchful eye. Restoring my Elipsa so often is a pain in the back.

However, I'll follow your suggestion and quit the apps before ejecting the Elipsa. If Time Machine, Spotlight, Kobo Desktop, Terminal, and Calibre are not accessing the mounted Elipsa, I'll look into how to spot what other process could be poking around in the Elipsa's SD Card before ejecting. Still, the Mac is VERY stubborn about not allowing disks to be ejected if a process is running that is using the drive. I'm thinking there is something going wrong when the Elipsa recognizes it is no longer connected and runs through a check that detects changes before opening to Home. Next time the corruption happens, I'll try restoring one SQLite database at a time to see which one is the problem.

Kobo should be paying us. LOL
Cootey is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Need help in Australia (ebook purchase) heymae News 14 01-21-2013 10:33 PM
Young Adult Novel (eBook) - Ghost of Doors WrongDayToQuit Self-Promotions by Authors and Publishers 0 11-16-2012 07:56 PM
First ebook purchase! gumleaf Introduce Yourself 5 10-10-2010 06:22 PM
How to tell if an eBook is Topaz before purchase? JSWolf Amazon Kindle 42 11-13-2009 08:59 AM
Can't Purchase eBook outside of US? Jakodai Kindle Formats 2 01-15-2009 08:12 PM


All times are GMT -4. The time now is 10:34 AM.


MobileRead.com is a privately owned, operated and funded community.