Calibre seems to reset metadata.calibre everytime I connect

[i_am_not_a_duck]

Hi Everyone!

I'm having a strange problem with my Kobo Libra H2O and Calibre.

When I plug in my Kobo, Calibre detects it and can write kepubs to it without any problems. I have KoboTouchExtended and KoboUtilities installed. If I open the Kobo in the windows file explorer, I can see that metadata.calibre has been updated. I eject the device nicely and the Kobo will import the files. They show up in my library correctly, with covers, etc.

So everything works relatively well up to this point. Occasionally if I'm trying to upload a LOT of books at once, the Kobo will fail to import some of them and I'm not sure why, but I'll deal with that after this problem has been resolved.

Now: with Kobo disconnected and Calibre not currently running, if I connect the Kobo to the computer and browse to it, I see the metadata.calibre file is still intact. I open Calibre, and as soon as it detects the device, it overwrites metadata.calibre with a fresh empty one and therefore cannot see what is already on the device. It does not pop up any errors.

Does anyone know what's wrong here? I've reinstalled Calibre, making sure to delete everythiung in appdata and anywhere else I can find but that did not work. I've tried every setting I could find, but haven't found anything that will control that behavior. I must be missing something. This used to work.


Please Help, I beg you!

[DNSB]

The only time I've seen that issue was years back and happened when the Kobo was forcibly disconnected (i.e. it was not safely ejected).

As for uploading a lot of books, my suggestion is that your ebooks need to be checked with epubcheck (you can use epubcheck as a plugin in calibre's ebook editor or the Sigil editor) and correcting any errors. My last pass at transferring a large back of ebooks was 12,000 epubs sent to a Clara HD (it has an 128GB µSD card) and while it took quite a while to transfer and import, the process completed successfully.

[i_am_not_a_duck]

I make sure that all jobs have completed and then eject via Calibre. I click the arrow next to the device icon and click eject this device, so I do try to be careful and kind to it.

I just found the setting about Manual/Automatic/Only on Send metadata management and I thought I had it. I switched from manual to Automatic and the first time I unplugged and plugged back in it worked! Calibre read the file and was able to show which books were already on the device. But then I added more, ejected, waited for import, reconnected and this time, the metadata.calibre file reverted to when I had just the first batch. ughhh.

I tried again with it set to Only on Send, but same issue.

I'll give epubcheck a try though. Thanks for the suggestions!

[ownedbycats]

Originally Posted by DNSB

My last pass at transferring a large back of ebooks was 12,000 epubs sent to a Clara HD (it has an 128GB µSD card) and while it took quite a while to transfer and import, the process completed successfully.

Wait, Kobo can reliably handle that many books? I'd expect it to start acting weird after a few thousand, if only because of the database size.

[Quoth]

So far Sage is fine with about 7,000 and I've had no database corruption issue on current FW. Careful to select My Books and wait for Sage etc.

Normally if setting up a new ereader I'd only transfer about 200 per connection.

[i_am_not_a_duck]

I supposed I should have quantified what I meant by a lot. I was just trying to upload ~90 books at once. I'm fairly new to ereaders and ebooks. I've had mine for a while but just recently made it a point to really try to primary it.

[PeterT]

I think what would be very useful if you reproduce this issue while running calibre in debug mode and posting the log.

The calibre metadata file is populated with the list of books on the device and that list is obtained when the Kobo driver reads the database from the device. I believe that it is only written when the device disconnects from calibre.

Sent from my Pixel 7a using Tapatalk

[JSWolf]

If calibre was to overwrite metadata.calibre, it would scan for all the content on your Kobo. So you would still see which books are on your Kobo regardless.

As to why it's happening, my guess could be some of the errors in the eBooks are causing it.

[DNSB]

Originally Posted by ownedbycats

Wait, Kobo can reliably handle that many books? I'd expect it to start acting weird after a few thousand, if only because of the database size.

My curiosity was originally about using a SDXC card in the Clara HD. Once I had the 128GB card in place, I was led to testing storing a large library on device. It worked but as mentioned some operations slowed since the database was larger than available RAM so no caching the entire database.

I don't know about reliably but I read about 80 books on the Clara HD and didn't see any issues.

[i_am_not_a_duck]

UPDATE: So I'm not sure if this is the solution or a coincidence. I tried uploading 90 kepubs in one go. All showed up in the metadata file once they were uploaded before I disconnected the device. However, the Kobo only imported ~45 of them. I read in another thread that logging out and logging back in would make the device import the remainder of them so I decided to try that before reconnecting the Kobo to Calibre. It worked! But now I'm a little confused as to why that worked. Why didn't they all go through the first time around? How does logging out and back in fix it?

Originally Posted by PeterT

The calibre metadata file is populated with the list of books on the device and that list is obtained when the Kobo driver reads the database from the device. I believe that it is only written when the device disconnects from calibre.

I can see the file is written before I eject it and it is populated with all the books I've uploaded. When I connect the Kobo to the computer I can see that the file is still intact before I start Calibre. It isn't until I start Calibre and it detects the device that it immediately- and I truly mean immediately, resets metadata.calibre.
If what you're writing is true, why bother writing a file at all? Why not just read the contents from the database every time instead of creating another file?

Let me know how to put Calibre in debug mode and I will share logs next time I upload a batch of books. I don't want to do it right now because I may not be able to connect for a while (which is the reason for the 90 books in the first place). But next time I do, I will share whatever I find.

Originally Posted by JSWolf

If calibre was to overwrite metadata.calibre, it would scan for all the content on your Kobo. So you would still see which books are on your Kobo regardless.

As to why it's happening, my guess could be some of the errors in the eBooks are causing it.

But the thing is, every book's metadata gets written to metadata.calibre in the first place. I can see the files on the Kobo. And from my update above, I now know that every book CAN successfully work.

[ownedbycats]

There's a bug with the current firmware, Kobo is looking into it, that occasionally causes database corruption and causes various issues like books failing to import. I suspect maybe you bumped into it.

Logging out and logging back in basically resets the database which fixes it. It also resets all reading positions and annotations.

[i_am_not_a_duck]

Originally Posted by ownedbycats

There's a bug with the current firmware, Kobo is looking into it, that occasionally causes database corruption and causes various issues like books failing to import. I suspect maybe you bumped into it.

Logging out and logging back in basically resets the database which fixes it. It also resets all reading positions and annotations.

Losing reading positions is not too bad, but losing my annotations is huge for me. I've already went through losing them for three books. I don't trust the Kobo anymore and generally reverted to keeping notes on my laptop instead.

Just wanted to mention I'm on software version 4.38.21908 from Aug 22, 2023.

[DNSB]

If you have NickelMenu installed, you can set up an entry that reboots your Kobo. I find this keeps the database corruption from happening since the databases will be properly closed before the USB connect. I used 3 entries so I could reboot from the home screen, books or reading.

menu_item : main    : Reboot              : power          : reboot
menu_item : library : Reboot              : power          : reboot
menu_item : reader  : Reboot              : power          : reboot

[PeterT]

To enter debug mode from the GUI, right shift on the Preferences menu icon (or click the drop down arrow besides it) and select the Restart in debug mode option. Alternatively Ctrl+Shift+R

[JeffElkins]

Originally Posted by DNSB

My curiosity was originally about using a SDXC card in the Clara HD. Once I had the 128GB card in place, I was led to testing storing a large library on device. It worked but as mentioned some operations slowed since the database was larger than available RAM so no caching the entire database.

I don't know about reliably but I read about 80 books on the Clara HD and didn't see any issues.

I just bought my wife a Clara 2E, but I decided to pull the trigger on an ebay Clara HD for myself, just so I could do this very thing. So far, I've loaded up about 80 books (my Heinlein collection and The Expanse series), I'm looking forward to testing it out. 128gb is insane overkill, but what the hell ...