![]() |
#16 | |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 24,905
Karma: 47303824
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
The device can run for a while with a corrupt database. Most of what you see displayed when looking for books is from in-memory structures. If the write to the database fails, the device will generally keep working. If you are using calibre, it can show an error when getting the list of books. But, the driver only reads a couple of tables, and depending on exactly where the corruption has occurred, it might not see it. You can do a check of the database to see if there are errors using any SQLite database manager. Or my Kobo Utilities plugin has a function to do it. As to how the the database gets corrupted, the only way I know is not safely ejecting the device when the database is being updated. There has to be other times as people report problems and swear they always eject safely. There are almost definitely some mistakes, but, there are two may reports for it not to happen at some other time. Signing out of the Kobo account on the device should fix this as a new database is created. And of course, a factory reset will do that as well. When importing sideloaded books, they should be added in alphabetical order using the file name, which, for me starts with the title. When the books is added to the device, a timestamp is set in the database. For sideloaded books, the same timestamp is used for all books imported in the same batch. The "Recent" sort uses this timestamp or the last read timestamp and it probably sorts by title within these. |
|
![]() |
![]() |
![]() |
#17 | ||||
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
Quote:
Quote:
…and a check of the database provided the following data: Spoiler:
Quote:
Quote:
FYI, I ran an undeleting program on the Kobo Elipsa when it was mounted on my Desktop, and it found all the missing *.nebo files. The only problem was that they were all 0 bytes large. They never properly saved to the internal memory. Here I thought I finally fixed it, then the glitch moved yet again. This time to an area I hadn't tested in over a week: notebooks. I have requested a new Elipsa pack from Kobo. Things are complicated since the US store showed that it was sold out earlier today. Thanks for all your feedback. |
||||
![]() |
![]() |
Advert | |
|
![]() |
#18 | ||||
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 24,905
Karma: 47303824
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Quote:
Quote:
Quote:
|
||||
![]() |
![]() |
![]() |
#19 |
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
My new Elipsa arrived. Other people here have complained about database issues. So far, one other person gave up on the Elipsa and returned theirs to the store. I rather love the Elipsa, so I opted to get another one. I'll update this thread if I start to see disappearing books again.
For now, I've setup my Elipsa with a few hundred books, I restored the only notebook that I was able to save from my other Elipsa, and now I'm running my Elipsa through it's paces. When I first set it up with the Kobo Desktop app, it got stuck rebooting for a few minutes. Kobo Desktop had updated its firmware. When the Elipsa finally booted, the What's New message was there with its sparse details, then I tried to sync to the server and it said it needed to update the firmware again. Then more downloading and booting, and the What's New message with the same sparse details reappeared. I can't say that it was a beginning that inspired confidence, but after all that, I am up and running. Crossing fingers. |
![]() |
![]() |
![]() |
#20 |
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
Well, it didn't take me long. I've run into the corrupt database/deleted downloads issue again. Once the Elipsa is ejected from my Mac, it reimports ALL sideloaded books again, deletes all 242 purchased books, and resets the home screen. Repairing the account restores everything but the downloaded ebooks.
Setting my Elipsa up with Kobo Desktop worked fine. Manually sideloading books by dragging and dropping in the Finder was fine. Downloading through Overdrive was fine. I ran into problems the moment I started to use Calibre to import books. Here's the error: Code:
calibre, version 5.23.0 ERROR: Error: Error communicating with device CorruptError: database disk image is malformed Traceback (most recent call last): File "calibre/gui2/device.py", line 89, in run File "calibre/gui2/device.py", line 515, in _books File "calibre/devices/kobo/driver.py", line 1974, in books apsw.CorruptError: CorruptError: database disk image is malformed |
![]() |
![]() |
Advert | |
|
![]() |
#21 | |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 24,905
Karma: 47303824
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Can you test the database before using calibre? Any SQLite database manager should be able to do it. They might have an option to do the check. But, there isn't, then run "PRAGMA integrity_check". You can try DB Browser for SQLite as that has a MAC version. If the database is fine, run calibre in debug mode. That will let us see if it is breaking when the database is first opened, or after reading some records. You will need to start calibre with the command "calibre-debug -g" to do this. You could also check the database before ejecting the device. This is to confirm that calibre did not do anything bad. Other thoughts are: Could it be the MAC? If you have another computer to try, it would be good to do soo. Are you ever powering off? Are there any log files being produced in the .kobo directory? I'm mainly wondering about stack dumps which will have the name "stack_xx.log" where xx is two numbers. If these exist, they are a sign of a crash happening somewhere. Does the partition show as being dirty when you connect? I don't know if the MAC will show this. Windows would offer to check the partition if it is dirty. |
|
![]() |
![]() |
![]() |
#22 | |||
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
Quote:
Code:
Execution finished without errors. Result: 1 rows returned in 3144ms At line 1: PRAGMA integrity_check; Quote:
I used Calibre to copy one book over and checked the database before ejecting. Everything is A-OK. Quote:
2) Powering off my Mac while the Elipsa is plugged in? No. I keep my Mac on at all times. I don't even let it sleep. 3) No log files as you described. 4) From what I've read, Mac tools for dealing with ExFat dirty drives is minimal. I'll have to see if that is the case if the corruption should occur again. Thanks! Last edited by Cootey; 07-17-2021 at 08:26 PM. |
|||
![]() |
![]() |
![]() |
#23 | |||
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 24,905
Karma: 47303824
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Quote:
Quote:
|
|||
![]() |
![]() |
![]() |
#24 |
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
Here’s an update:
I’ve enabled debug mode for Calibre, and I check the database upon mounting and before ejecting. I’ve also checked the Elipsa database with SQLite software. No problems have been found. I also constantly backup my Elipsa so I don’t lose data. With all the database issues I’ve run into, I can report that many ebook annotations and page locations have not been lost, but the Elipsa has forgotten that I’m reading quite a few books. Those ebooks still retain their annotations, but not current page location. Fortunately, bookmarks are intact, too. Kobo backs up our stylus written annotations so that they are restored when logging into a reset or new Kobo. Notebooks are goners, unfortunately. I have had one close call that didn’t require restoring the database. After ejecting my Elipsa using Calibre at the end of last week, the Elipsa powered off completely. When I tried to turn it back on, it didn’t respond. I pushed the on button a few times, waiting for the Elipsa to respond after each press, but only after the fourth time did it power up, then acted as it had meant to do that, everything was perfectly fine, and would you please stop staring at me? I’m watching it with a suspicious side eye now, but the Elipsa has been behaving without issue since then. Sent from my iPhone using Tapatalk Pro |
![]() |
![]() |
![]() |
#25 |
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
Alright. I've got a corrupt database again. I was working with Kobo tech over a different issue (Elipsa doesn't prompt to review or rate purchased books when they are finished). I was asked to connect my Elipsa to Kobo Desktop on my Mac. It synced fine. No books were added. I cmd-shift-i'd my log file. Then I got busy working on something else. When I ejected my Elipsa properly, it suddenly reimported EVERYTHING all over again. All my purchased downloads have disappeared, again.
I connected it to the Mac again, this time with Calibre. I immediately got the ERROR: Error dialog letting me know "Error communicating with device". Here is the clipboard error output: Code:
calibre, version 5.23.0 ERROR: Error: Error communicating with device CorruptError: database disk image is malformed Traceback (most recent call last): File "calibre/gui2/device.py", line 89, in run File "calibre/gui2/device.py", line 515, in _books File "calibre/devices/kobo/driver.py", line 1974, in books apsw.CorruptError: CorruptError: database disk image is malformed Result of running 'PRAGMA integrity_check' on database on the Kobo device: Spoiler:
This is an annoying bug. Since the Elipsa was mounted on my desktop for an hour, I wonder if Spotlight's search indexing caused a problem. I have now excluded KOBOeReader from being indexed. But this could be a Kobo Desktop problem, too. Any suggestions on how to isolate who is at fault? How fortunate for me that I just did a backup half an hour ago before this happened. Last edited by Cootey; 07-22-2021 at 05:48 PM. |
![]() |
![]() |
![]() |
#26 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 24,905
Karma: 47303824
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
The act of reimporting everything suggests the database was corrupted when the device was ejected. Whether that was the Kobo desktop or something else running, I don't know. If something else was accessing the database when the Kobo desktop was, that would be a likely culprit. If that was also running during your calibre sessions, that could also explain the other problems you have been seeing,
|
![]() |
![]() |
![]() |
#27 | |
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
Quote:
Since I began bumping into these issues, I don't run Kobo desktop and Calibre at the same time. Also, there are no active programs operating on the Kobo when I eject it. The Mac won’t allow it. I can’t even eject it if the Elipsa is simply the current directory in Terminal. I also plug the Elipsa directly into the back of my Mac instead of using a USB hub to eliminate potential issues there. I never have these problems with my Clara. I haven't treated my Elipsa any different, except now I treat it as if its made of handblown Egyptian glass when I bring it near my Mac. I usually check the database first in Calibre before ejecting it. I may have forgotten to do that this time, but I’ve ejected directly from within Kobo Desktop before with no issues. That all being said, since I last posted above I had to restore my Kobo. Due to our past conversation in another thread, I’ve been experimenting with using rsync instead of dd to make a backup. So I used cp to create the backup archive, then planned to rsync the archive back onto the Kobo in the event of a database corruption. I had a full disk image backup at the ready, so this was my first time to experiment using rsync to restore the Kobo. While it was running, it deleted a ton of ._ files off the Elipsa. Usually, the Mac creates those files every time you access a directory from the Finder. However, there were ._ files in directories I had never accessed from Finder. To be clear, these were on the Kobo and didn’t exist in the backup. That’s why rsync removed them. That’s probably obvious to you, but I’m trying to be thorough. They were created between the time I made the backup and the time I ejected the Kobo. So what process is accessing the Kobo? Prior to your suggestion, I was already puzzling over this. TimeMachine apparently can store data in ._ files, but I don't use TimeMachine on USB external drives. I blocked Spotlight from indexing the Kobo whenever it is mounted. Spotlight only indexes for searching, however. It doesn’t create ._ files. I have never seen this issue with any USB mounted device before, and I can’t be certain this is the issue that caused the corruption. The Kobo tends to ignore ._ files in my experience. This is a lot of text, but as I said above, I'm trying to be thorough. I know a couple of people have bumped into corrupt Elipsa databases. One person gave up and returned his Elipsa to the store. We should be finding out if they use Macs, Kobo Desktop, and Calibre to find commonalities. And the big question is: Why is only my Elipsa having these issues and not either of our Claras? I can't imagine somebody who is not a geek trying to deal with this. Last edited by Cootey; 07-22-2021 at 06:21 PM. |
|
![]() |
![]() |
![]() |
#28 | |||
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 24,905
Karma: 47303824
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Quote:
Unfortunately, I just don't know when it happens. A possibility is a background task on the MAC clashing. But, you should be able to see that. Another thing to try is to not eject from within the applications, either calibre or the Kobo desktop. Close these and wait a couple of minutes. That should make sure nothing that you are doing trying to update, and it should let any caches flush as well. I don't know the MAC, but, Windows has, or at least had, an option for "fast eject" or something like that. Turning it on, meant that writes to removable media were immediate and not cached. Quote:
|
|||
![]() |
![]() |
![]() |
#29 | |
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
Quote:
However, I'll follow your suggestion and quit the apps before ejecting the Elipsa. If Time Machine, Spotlight, Kobo Desktop, Terminal, and Calibre are not accessing the mounted Elipsa, I'll look into how to spot what other process could be poking around in the Elipsa's SD Card before ejecting. Still, the Mac is VERY stubborn about not allowing disks to be ejected if a process is running that is using the drive. I'm thinking there is something going wrong when the Elipsa recognizes it is no longer connected and runs through a check that detects changes before opening to Home. Next time the corruption happens, I'll try restoring one SQLite database at a time to see which one is the problem. Kobo should be paying us. LOL ![]() |
|
![]() |
![]() |
![]() |
#30 |
Absentminded Reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,324
Karma: 7387565
Join Date: Apr 2017
Device: Kobo Mini, Libra Colour & Elipsa; Kindle Paperwhite 3 & 4; iOS apps
|
Alright. I'm back with another database corruption. For the record, I never have these issues with my Kobo Clara.
I believe that the recent firmware updates released for the Kobo have improved things. I've gone 22 days without a database corruption—until earlier this morning. Unfortunately, I did not remember to quit Calibre before ejecting as you had suggested (something I had been forgetting to do recently), but I do have more information that might help us. My notes on my latest Elipsa glitch: 1. Checked database using Kobo Utilities in Calibre. ✔ 2. Imported about 21 manga through Calibre. Database ✔ 3. Imported another 9 manga. Checked database again. It was corrupt. 4. On a whim, I deleted all 30 manga manually off the Elipsa before ejecting it from desktop. This avoided the import/database destruction I’ve noticed in the past. All purchased books remained downloaded. No re-importation of all previously imported books occurred. 5. Repaired Kobo account. Plugged Elipsa back into Mac. Checked database. It remained corrupt. 6. Backed up my .nebo journal files, then I restored from older backup via Terminal. This has ALWAYS worked before, but this time… 7. …Elipsa .jpg & .svg files (the stylus notations and their page screenshots) from the .kobo/markups directory were now loaded as individual files. Several Kobo pages worth in My Books. Even repairing the account didn’t properly categorize them. I thought perhaps they might be newer out of sync annotations that weren't part of the older database, but the rsync string I use to restore a backup deletes everything off the Elipsa that isn't present in the backup. 8. Logged out of my account on my Kobo, then logged back in. Markup files were still listed individually as "books". I checked the database with Kobo Utils plug-in in Calibre, and the database had been repaired as expected. Since I wasn’t sure which markup files were being categorized correctly and which were not, I… 9. Did a factory reset (with current firmware). 10. Annotations were restored. I don't know if I lost any, but I keep WiFi on all the time, so it's not likely. (Turns out I lost annotations from just the past 24 hours before corruption.) Database is not corrupt. I'm ready to rebuild my setup again. But I've lost time, I'll be losing more time getting things back up and running, and I'm grumpy about it. Why is this happening to my Elipsa? *♂️ Here are my observations. I’ve used Calibre to import manga before since the last Elipsa database corruption with no issues. Yet whenever the database is corrupted, Calibre seems to be involved. (I run Calibre in debug mode, so please tell me what you folks need to see to help identify what is happening. I'll send it to you since it likely contains names of my local drives and network). When I sideload hundreds of books by dragging and dropping onto the mounted Elipsa, I have ZERO database issues. I did notice, however that the Elipsa was constantly being ejected this morning. THIS behavior is consistent with corruption events. Whether I was plugged into a hub or directly into the back of my 2014 Mac Mini, the Elipsa would connect, then suddenly eject itself. This is very irregular behavior, and the Elipsa was the only device having this issue. The Mac did not display any warning dialogues as it usually does if you unplug a USB device without ejecting through the Finder first. That seems to indicate that it was rejected properly. When the Elipsa seemed to be stably connected for a few minutes, I imported the manga through Calibre. The Elipsa was not ejected during this process. However, after the transfer was complete (done in two steps BTW with me checking the database after each batch) the database was corrupt after the second, smaller batch. Then I did step 4 above to minimize the damage, but to no avail. After resetting the firmware, my Elipsa was connected for half a day without being ejected from my Mac. I can make sure that I quit Calibre and Kobo Desktop in the future. I have put in an order for a different USB hub. But neither of those things may be the problem. In the meantime, something became corrupt on that second Calibre batch. The Elipsa hadn't technically imported anything yet since it was still mounted to my desktop. I'd like to figure this out. I'm not the only person running into corrupt database issues with the Elipsa. The following is the PRAGMA integrity check when I discovered the database was corrupt. I don't think this is any different than any other integrity check dump that I've shared. It just means the database is now corrupt: Spoiler:
Last edited by Cootey; 08-18-2021 at 02:00 AM. |
![]() |
![]() |
![]() |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Need help in Australia (ebook purchase) | heymae | News | 14 | 01-21-2013 10:33 PM |
Young Adult Novel (eBook) - Ghost of Doors | WrongDayToQuit | Self-Promotions by Authors and Publishers | 0 | 11-16-2012 07:56 PM |
First ebook purchase! | gumleaf | Introduce Yourself | 5 | 10-10-2010 06:22 PM |
How to tell if an eBook is Topaz before purchase? | JSWolf | Amazon Kindle | 42 | 11-13-2009 08:59 AM |
Can't Purchase eBook outside of US? | Jakodai | Kindle Formats | 2 | 01-15-2009 08:12 PM |