Elipsa Ghost eBook Purchase on an Elipsa

[Cootey]

I tried to change the title of this thread since the Elipsa database corruption issue can go beyond missing and suddenly deleted eBooks. However, I may lack authority (or know how) to do so.

[davidfor]

Originally Posted by Cootey

Alright. I'm back with another database corruption. For the record, I never have these issues with my Kobo Clara.

I believe that the recent firmware updates released for the Kobo have improved things. I've gone 22 days without a database corruption—until earlier this morning. Unfortunately, I did not remember to quit Calibre before ejecting as you had suggested (something I had been forgetting to do recently), but I do have more information that might help us.

My notes on my latest Elipsa glitch:

1. Checked database using Kobo Utilities in Calibre. ✔
2. Imported about 21 manga through Calibre. Database ✔
3. Imported another 9 manga. Checked database again. It was corrupt.

How much time was between these? It shouldn't matter, but, see my comment on item 10.

4. On a whim, I deleted all 30 manga manually off the Elipsa before ejecting it from desktop. This avoided the import/database destruction I’ve noticed in the past. All purchased books remained downloaded. No re-importation of all previously imported books occurred.

That makes sense. If the last books added before the database corruption, then removing them any added afterwards will mean the import is not triggered. Though there is a dependency on where in the database the problem is.

5. Repaired Kobo account. Plugged Elipsa back into Mac. Checked database. It remained corrupt.

That is expected. The repair is fixing the differences between the data the device knows about your purchased books and what the Kobo server knows. While it should update the database, it won't actually attempt to repair the corruption.

6. Backed up my .nebo journal files, then I restored from older backup via Terminal. This has ALWAYS worked before, but this time…
7. …Elipsa .jpg & .svg files (the stylus notations and their page screenshots) from the .kobo/markups directory were now loaded as individual files. Several Kobo pages worth in My Books. Even repairing the account didn’t properly categorize them. I thought perhaps they might be newer out of sync annotations that weren't part of the older database, but the rsync string I use to restore a backup deletes everything off the Elipsa that isn't present in the backup.

I don't know about this. I don't have an Elipsa to play with, so I don't know what to expect. The images being imported as "books" means that the device saw them as books. Is it possible you got the directory name wrong? Nothing under ".kobo/markups" should be imported. If you made a typo with the name, it would explain it.

8. Logged out of my account on my Kobo, then logged back in. Markup files were still listed individually as "books". I checked the database with Kobo Utils plug-in in Calibre, and the database had been repaired as expected. Since I wasn’t sure which markup files were being categorized correctly and which were not, I…

That is consistent with point 7. It really does sound like the files were in the wrong place.

9. Did a factory reset (with current firmware).
10. Annotations were restored. I don't know if I lost any, but I keep WiFi on all the time, so it's not likely. (Turns out I lost annotations from just the past 24 hours before corruption.) Database is not corrupt. I'm ready to rebuild my setup again. But I've lost time, I'll be losing more time getting things back up and running, and I'm grumpy about it. Why is this happening to my Elipsa? ����*♂️

The lost annotations should be those made after the last sync. But, the annotations made while reading should be synced shortly after you made them. But, it is possible that would happen at most every 30 minutes. By that I mean if you made an annotation, it will probably sync. But, the next might not be done for 30 minutes and all made in that time period will then be synced. I don't know for sure as I have never tested this.

But, I suspect that annotations sync stopped when the database became corrupted. That's why I asked about the time between notes 2 and 3. If there was some time between them, the corruption might have happened shortly after the first books were imported. But, you didn't see it until the trying to import the next set of books.

Here are my observations. I’ve used Calibre to import manga before since the last Elipsa database corruption with no issues. Yet whenever the database is corrupted, Calibre seems to be involved. (I run Calibre in debug mode, so please tell me what you folks need to see to help identify what is happening. I'll send it to you since it likely contains names of my local drives and network). When I sideload hundreds of books by dragging and dropping onto the mounted Elipsa, I have ZERO database issues.

But, how do you know? The reason you know about the database corruption is that calibre discovers it. All that tells you is that between the last time and this time, something happened. And even that isn't accurate. Unless the problem is really bad, calibre will only report a problem if the parts it accesses have problems. Getting the list of books reads two tables. If there are no problems with those tables, then no error will be reported. But, if there are problem elsewhere, things could be going wrong elsewhere. And maybe they will affect the part that calibre usually sees later. This is why I sometimes tell people to check the database.

But, the basic problem is that calibre does not treat the Elipsa differently to any other Kobo device. If calibre is the cause, why aren't we seeing the same problem with all devices?

I did notice, however that the Elipsa was constantly being ejected this morning. THIS behavior is consistent with corruption events. Whether I was plugged into a hub or directly into the back of my 2014 Mac Mini, the Elipsa would connect, then suddenly eject itself. This is very irregular behavior, and the Elipsa was the only device having this issue. The Mac did not display any warning dialogues as it usually does if you unplug a USB device without ejecting through the Finder first. That seems to indicate that it was rejected properly. When the Elipsa seemed to be stably connected for a few minutes, I imported the manga through Calibre. The Elipsa was not ejected during this process. However, after the transfer was complete (done in two steps BTW with me checking the database after each batch) the database was corrupt after the second, smaller batch. Then I did step 4 above to minimize the damage, but to no avail.

What else was going on? Did you have calibre or something else running that would have been accessing the device?

After resetting the firmware, my Elipsa was connected for half a day without being ejected from my Mac. I can make sure that I quit Calibre and Kobo Desktop in the future. I have put in an order for a different USB hub. But neither of those things may be the problem. In the meantime, something became corrupt on that second Calibre batch. The Elipsa hadn't technically imported anything yet since it was still mounted to my desktop. I'd like to figure this out. I'm not the only person running into corrupt database issues with the Elipsa.

The following is the PRAGMA integrity check when I discovered the database was corrupt. I don't think this is any different than any other integrity check dump that I've shared. It just means the database is now corrupt:

Spoiler:

Result of running 'PRAGMA integrity_check' on database on the Kobo device:


*** in database main ***
On tree page 54734 cell 0: invalid page number 167909888
Page 52133: btreeInitPage() returns error code 11
Page 52134: btreeInitPage() returns error code 11
.
.
.

That really just tells me that the database is corrupted an it is unlikely to be recoverable. The only times I have found that corrupt databases have been recoverable is when the problems are only in the indexes or they are only freespace errors. It won't harm to try a VACUUM, but, I expect you will get an error during this and it won't be able to complete.

[Cootey]

Originally Posted by davidfor

How much time was between these? It shouldn't matter, but, see my comment on item 10.

Immediately after each step. I didn't wait. I cannot recall how long it took for each upload job.

Originally Posted by davidfor

I don't know about this. I don't have an Elipsa to play with, so I don't know what to expect. The images being imported as "books" means that the device saw them as books. Is it possible you got the directory name wrong? Nothing under ".kobo/markups" should be imported. If you made a typo with the name, it would explain it.
…That is consistent with point 7. It really does sound like the files were in the wrong place.

I don't know what to tell you. I was as surprised as you are. It should not have happened. I never meddled with the contents of the .kobo folder. I would never arbitrarily change the name of a folder inside .kobo. Until I get my Elipsa to behave, I haven't even installed NickelMenu. The only additions I have made have been screensaver photos, custom dictionaries, and fonts.

Originally Posted by davidfor

But, I suspect that annotations sync stopped when the database became corrupted. That's why I asked about the time between notes 2 and 3. If there was some time between them, the corruption might have happened shortly after the first books were imported. But, you didn't see it until the trying to import the next set of books.

Since I checked the database between importation jobs, not pausing long between jobs, it seemed clear to me that the corruption occurred during or after the second batch.

Originally Posted by davidfor

But, how do you know? The reason you know about the database corruption is that calibre discovers it. All that tells you is that between the last time and this time, something happened. And even that isn't accurate. Unless the problem is really bad, calibre will only report a problem if the parts it accesses have problems. Getting the list of books reads two tables. If there are no problems with those tables, then no error will be reported. But, if there are problem elsewhere, things could be going wrong elsewhere. And maybe they will affect the part that calibre usually sees later. This is why I sometimes tell people to check the database.

This is why I have been asking for how to share the debug logs with you—so that we can better determine which process (Mac, Calibre, Kobo, etc) is causing this database corruption.

Originally Posted by davidfor

But, the basic problem is that calibre does not treat the Elipsa differently to any other Kobo device. If calibre is the cause, why aren't we seeing the same problem with all devices?

What else was going on? Did you have calibre or something else running that would have been accessing the device?

I use the same setup with my Clara and have never had this problem. This has only been an issue for my Elipsa (both the original and its replacement).

Originally Posted by davidfor

That really just tells me that the database is corrupted an it is unlikely to be recoverable. The only times I have found that corrupt databases have been recoverable is when the problems are only in the indexes or they are only freespace errors. It won't harm to try a VACUUM, but, I expect you will get an error during this and it won't be able to complete.

Yes, but why is it corrupting? That's what I need help identifying. (I don't see "VACUUM" in the database submenu of KoboUtils). I know that other people in this forum have had database corruption issues with their Elipsas. At least one or two returned their Elipsa because of it. Where would I find the debug logs that you need to see?

[davidfor]

@Cootey: Reading through that, there is only one thing I can say: I have absolutely no idea what is happening. None of it makes sense as it works fine for the other devices. It seems to be something different about the Elipsa. Whether it is related to how the device connects to the PC or something else running that correctly connect or disconnect from the database I don't know.

For debugging this, I gave all my ideas earlier. At best it tells you what sort of action is triggering it. The calibre debug logs are not going to tell us anything more than what you have shown. They are basically showing an error when it happens, and what happens before that is exactly what is expected. You can produce logs on the device, but, they are encrypted and I have no way to decrypt them to see what is in them. You can redirect the log (I think there is a NickelMenu function to do it) but whether that will show anything useful I don't know. Before Kobo started encrypting the logs, I never saw anything that would give a clue for why the corruption happened. But, I don't know if I ever saw logs from when it happened.

I don't believe it it calibre. If it was, we would be seeing a similar rate of database corruption for other devices. The only thing calibre does differently for each device is the sizes for covers if you send them to the device. And the name displayed when the device is connected. Nothing else is different.


As to doing a "VACUUM", that is the command you would use in a SQLite management tool or command-line SQLite. In the Kobo Utilities plugin, it is the compress option as that is what I am used to for other databases.

[NiLuJe]

FWIW, I was just looking into weirdly missing unmounts on the Elipsa, and, yeah, the lazy unmount is really sweeping a lot of stuff under the rug, stuff that might actually be problematic.

As in, quite a few Nickel threads *still* have fds open'ed on (at least?) book files, which means the filesystem is never actually unmounted. Which is obviously how Really Bad Stuff Happens (TM).

I happened to be testing a CBZ here, and hwmon, QProcessManager, eventengine, QEvdevTouch, QTBearerThread, imageprovider and QtHTTPThread all had a couple of fds opened on the last book.

Sidebar: I haven't looked at this stuff in a *long* time, so, this may not be specific to the Elipsa, so this isn't excluding the possibility that the Elipsa's storage solution might just be more prone to this kind of badness .

(for instance, I can almost reliably trash my ZSH history (which lives in onboard) if I *properly* soft-reboot the Elipsa with the shell open. That never happens anywhere else).

EDIT: This is fairly easy to double-check with ForceWifi and a real lsof binary. fds from MNT_DETACHED fs end up relocated to / (with the rest of their path, relative to the original mountpoint, preserved) in the listing.

(I happened to notice the missing unmounts because I track 'em in KFMon & NanoClock ^^).

EDIT²: On the upside, quickly tested with a notebook, a PDF, an ePub, a KePub, and a KePub w/ annotations, and I can only reproduce a consistent failure with a CBZ .

EDIT³: Issue persists as long as a CBZ has been open'ed during that boot session, though :/.

[davidfor]

Originally Posted by NiLuJe

FWIW, I was just looking into weirdly missing unmounts on the Elipsa, and, yeah, the lazy unmount is really sweeping a lot of stuff under the rug, stuff that might actually be problematic.

As in, quite a few Nickel threads *still* have fds open'ed on (at least?) book files, which means the filesystem is never actually unmounted. Which is obviously how Really Bad Stuff Happens (TM).

I happened to be testing a CBZ here, and hwmon, QProcessManager, eventengine, QEvdevTouch, QTBearerThread, imageprovider and QtHTTPThread all had a couple of fds opened on the last book.

Sidebar: I haven't looked at this stuff in a *long* time, so, this may not be specific to the Elipsa, so this isn't excluding the possibility that the Elipsa's storage solution might just be more prone to this kind of badness .

(for instance, I can almost reliably trash my ZSH history (which lives in onboard) if I *properly* soft-reboot the Elipsa with the shell open. That never happens anywhere else).

EDIT: This is fairly easy to double-check with ForceWifi and a real lsof binary. fds from MNT_DETACHED fs end up relocated to / (with the rest of their path, relative to the original mountpoint, preserved) in the listing.

(I happened to notice the missing unmounts because I track 'em in KFMon & NanoClock ^^).

EDIT²: On the upside, quickly tested with a notebook, a PDF, an ePub, a KePub, and a KePub w/ annotations, and I can only reproduce a consistent failure with a CBZ .

EDIT³: Issue persists as long as a CBZ has been open'ed during that boot session, though :/.

Now that's interesting. I wouldn't even have thought of CBZ as being a problem as I assumed it would have been a low user of the database. But, it does put in an entry for each image into the database.

And it would seem to explain @Cootey's problems. That's assuming the mentioned Manga are in CBZ or CBR (I'm assuming these are handled the same) and not epub or kepub.

Anyway, I'll have a play and see what I can see.

[davidfor]

I have done some playing and can reproduce the database corruption when CBZs or CBRs are involved. Basically, if I have read a comic and connect to a PC and then do anything that updates the database while connected, I can end up with a corrupted database. It doesn't happen all the time. As it is happening when the database is updated, using calibre will do it. But, so will using the Kobo desktop application. Both can update the database and that clashes with whatever hasn't stopped using the database when the device was connected.

There is nothing that can be done from calibre. I don't think there is any way to detect if something is still using the database in this way. There are two operating system involved, and no way for them to communicate.

[cpimentel]

I had the same experience as davidfor. All problems occurred after reading a comic or after updating its metadata (I am not sure if this is really related).

At 1st I thought was something related to the collections/series metadata being updated. I disabled and moved on. Things were nice, I got confident and enabled it again, and boom, database corrupted. At same time I was sideloading CBRs/CBZs and reading them too.

I was following this thread and yesterday I decided to reset the device, sideload everything without the Collection/Series enabled and no CBR/CBZ.

Profit! Enabled the Collection/Series. Profit! Sideloaded converted CBZ/CBR as EPUB and later reconverted to PDF (because of the zoom). Profit!

I side loaded all my books, comics in PDF format and have all of them on their respective collections and series. Perfect.

And here is the list of problems I faced:
- WIFI losing passwords after crashs;
- Files being in the device and Calibre not able to match them with my Library;
- Ghost files on home and collection;
- Crashes reading CBR/CBZ followed by database broken;
- Device with its mass storage marked as dirty (needs to rescan);

Thank you all for the effort on this, if it was not for this thread I would just gave up and started sideloading books using dropbox or Calibre web.

[Haesslich]

Originally Posted by davidfor

Now that's interesting. I wouldn't even have thought of CBZ as being a problem as I assumed it would have been a low user of the database. But, it does put in an entry for each image into the database.

And it would seem to explain @Cootey's problems. That's assuming the mentioned Manga are in CBZ or CBR (I'm assuming these are handled the same) and not epub or kepub.

Anyway, I'll have a play and see what I can see.

That explains the inability to sync after the last firmware update. I had like 50 CBZ on my Forma, and I had to do a factory reset to get everything working again....

[Cootey]

Originally Posted by davidfor

Now that's interesting. I wouldn't even have thought of CBZ as being a problem as I assumed it would have been a low user of the database. But, it does put in an entry for each image into the database.

And it would seem to explain @Cootey's problems. That's assuming the mentioned Manga are in CBZ or CBR (I'm assuming these are handled the same) and not epub or kepub.

Anyway, I'll have a play and see what I can see.

They are indeed CBZ files.

I haven't plugged my Elipsa into my Mac since I last posted. Just plugged it in a few moments ago. Calibre was running. And I received an instant error message as soon as the device mounted into Calibre. I hadn't even added anything through the Finder nor Calibre. That was new.

I see that with NiLuJe's input (who can speak the lingo), there are some new developments. I'll post my error log here, and catch up.

Spoiler:

calibre, version 5.26.0
ERROR: Unhandled exception: <b>CorruptError</b>:CorruptError: database disk image is malformed

calibre 5.26 embedded-python: True is64bit: True
macOS-10.15.7-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '19.6.0', 'Darwin Kernel Version 19.6.0: Tue Jun 22 19:49:55 PDT 2021; root:xnu-6153.141.35~1/RELEASE_X86_64')
Python 3.8.5
OSX: ('10.15.7', ('', '', ''), 'x86_64')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 42, 0) && DeDRM (7, 2, 1) && Package KFX (from KFX Input) (1, 42, 0) && Embed Comic Metadata (1, 6, 1) && KFX metadata reader (from KFX Input) (1, 42, 0) && KFX Input (1, 42, 0) && Set KFX metadata (from KFX Output) (1, 55, 0) && KFX Output (1, 55, 0) && KePub Input (3, 4, 3) && KePub Metadata Reader (3, 4, 3) && KePub Metadata Writer (3, 4, 3) && KePub Output (3, 5, 3) && Kobo Books (1, 8, 2) && Kobo Utilities (2, 14, 1) && Obok DeDRM (7, 2, 1)
Traceback (most recent call last):
File "calibre_plugins.koboutilities.action", line 1737, in check_device_database
check_result = self._check_device_database()
File "calibre_plugins.koboutilities.action", line 3711, in _check_device_database
return check_device_database(self.device_database_path())
File "calibre_plugins.koboutilities.action", line 6073, in check_device_database
result = cursor.fetchall()
apsw.CorruptError: CorruptError: database disk image is malformed

[Cootey]

Originally Posted by davidfor

I have done some playing and can reproduce the database corruption when CBZs or CBRs are involved. Basically, if I have read a comic and connect to a PC and then do anything that updates the database while connected, I can end up with a corrupted database. It doesn't happen all the time. As it is happening when the database is updated, using calibre will do it. But, so will using the Kobo desktop application. Both can update the database and that clashes with whatever hasn't stopped using the database when the device was connected.

There is nothing that can be done from calibre. I don't think there is any way to detect if something is still using the database in this way. There are two operating system involved, and no way for them to communicate.

So this is a Kobo issue where something is left open in the database when comics are involved? It's not closing access to the database properly? I've been ripping my hair out over here, and talking to tech support doesn't help. They just have me wipe everything and start over which "fixes" the database issue until I randomly read a comic while the moon is over Mars while the wind blows southwesterly. I'm glad you two have been able to quantify better what is happening. I suppose the problem could happen with the Kobo desktop, but I haven't noticed it. It only happens with Calibre for me. Reading manga on my Elipsa was one of the reasons I got it. I'll have to remove them all in the interim.

Will you be filing a report with Kobo? I can use their website to leave feedback, but as a customer, my experience is that they just assume I'm an idiot who doesn't know one end of the Kobo from the other.

[Cootey]

Originally Posted by cpimentel

I had the same experience as davidfor. All problems occurred after reading a comic or after updating its metadata (I am not sure if this is really related).

At 1st I thought was something related to the collections/series metadata being updated. I disabled and moved on. Things were nice, I got confident and enabled it again, and boom, database corrupted. At same time I was sideloading CBRs/CBZs and reading them too.

I was following this thread and yesterday I decided to reset the device, sideload everything without the Collection/Series enabled and no CBR/CBZ.

Profit! Enabled the Collection/Series. Profit! Sideloaded converted CBZ/CBR as EPUB and later reconverted to PDF (because of the zoom). Profit!

I side loaded all my books, comics in PDF format and have all of them on their respective collections and series. Perfect.

And here is the list of problems I faced:
- WIFI losing passwords after crashs;
- Files being in the device and Calibre not able to match them with my Library;
- Ghost files on home and collection;
- Crashes reading CBR/CBZ followed by database broken;
- Device with its mass storage marked as dirty (needs to rescan);

Thank you all for the effort on this, if it was not for this thread I would just gave up and started sideloading books using dropbox or Calibre web.

I'm sorry to hear that you've been running into the same issues, but I'm also a bit glad I'm not alone. I've been keeping up this thread despite risking people thinking I'm bonkers. LOL So I'm glad somebody else found it helpful. Now let's figure out a solution.

My first move is to restore my Kobo AGAIN, then refrain from side-loading any comics onto it. That's not much of a solution, but it's better than factory restoring all the time.

[Cootey]

Originally Posted by Haesslich

That explains the inability to sync after the last firmware update. I had like 50 CBZ on my Forma, and I had to do a factory reset to get everything working again....

It's possible you ran into a different issue than we did here. I could still sync with Kobo desktop and transfer files from Calibre even with a corrupt database, but my Elipsa started acting possessed afterwards. Please skim from the begging of this thread and see if you noticed similar issues. And could you describe what you mean by "sync" and with which app you were using?

David, were you able to reproduce the database corruption with other Kobo devices? I'm curious

[davidfor]

Originally Posted by Cootey

So this is a Kobo issue where something is left open in the database when comics are involved? It's not closing access to the database properly? I've been ripping my hair out over here, and talking to tech support doesn't help. They just have me wipe everything and start over which "fixes" the database issue until I randomly read a comic while the moon is over Mars while the wind blows southwesterly. I'm glad you two have been able to quantify better what is happening. I suppose the problem could happen with the Kobo desktop, but I haven't noticed it. It only happens with Calibre for me. Reading manga on my Elipsa was one of the reasons I got it. I'll have to remove them all in the interim.

With the Kobo desktop, it didn't happen every time. And it didn't happen every time with calibre, but, it was more often. I was also able to reproduce it with DB Browser for SQLite if I updated something.

Will you be filing a report with Kobo? I can use their website to leave feedback, but as a customer, my experience is that they just assume I'm an idiot who doesn't know one end of the Kobo from the other.

It's always a problem reporting problems like this in ways they can understand. I'm not sure how I would. But, I discussed it with one of the people on the beta list and he was able to see what was happening. I expect it will be fixed in the next release, but we will have to wait and see.

[Cootey]

Originally Posted by davidfor

With the Kobo desktop, it didn't happen every time. And it didn't happen every time with calibre, but, it was more often. I was also able to reproduce it with DB Browser for SQLite if I updated something.

That’s fascinating that you were able to reproduce the issue with DB Browser. Is this a problem with just the Elipsa, or with all Kobos running the 4.28.x firmware?

Since the database is being left open by the Kobo, would we be able to avoid corruption by power cycling the Kobo before connecting to a computer?

Originally Posted by davidfor

It's always a problem reporting problems like this in ways they can understand. I'm not sure how I would. But, I discussed it with one of the people on the beta list and he was able to see what was happening. I expect it will be fixed in the next release, but we will have to wait and see.

I’ll probably give Kobo a call and update one of the cases I had open with them. It couldn’t hurt.


Sent from my iPhone using Tapatalk Pro