koreader deletes calibre metadata after sending book via USB to device

[Zoasa]

When I send a book from calibre to my Kobo via USB, the book gets copied to the device and the metadata gets updated, but after ejecting, the meta data file gets reverted.
Next time I connect the device, calibre doesn't see the new book anymore. With Nickel, the same workflow does work correctly.

Scenario 1: Sending book from calibre to Kobo via USB when koreader is open

• koreader is running
• Connect USB to computer
• "Share storage via USB?" -> Share
• Send new book from Calibre to main memory
• Check file on root of kobo: metadata.calibre: The new book gets an new entry there
• Eject Kobo from Calibre
• On Kobo, go to file explorer. New book is there as a file
• Connect Kobo again via USB

• Not OK: Check metadata.calibre: The entry for the new book is not there anymore.
• Not OK: In Calibre the book is not shown as existing on device in library (Green mark)

Scenario 2: Sending book from calibre to Kobo via USB when nickel is open

• Exit koreader
• Nickel is running
• Connect USB
• Send new book from Calibre to main memory
• Check file on root of kobo: metadata.calibre: The new book gets an new entry there
• Eject Kobo from Calibre
• Nickel shows import dialog
• Open Koreder
• Go to file explorer: New book is there as a file
• Connect Kobo again via USB

• OK: Check metadata.calibre: The entry for the new book is still there.
• OK: In Calibre the book is shown as existing on device in library (Green check mark)

Why is scenario 2 with nickel working correctly but scenario 1 with koreader not? Is this a bug in koreader or am I missing something?

Device: Kobo Clara HD
Versions:
Original Firmware 4.26.16704
koreader v2021.05

[NiLuJe]

It's perfectly normal, and probably related to the Kobo driver's automatic metadata management: the driver expects to be talking to nickel, that means it bases most of its sanity checking on the state of the *nickel* database. If a book isn't in there, it assumes it doesn't exist anymore (e.g., was deleted from within Nickel). We, of course, never ever touch said db.

The good news is, you magically recover everything if you actually let nickel scan your books once in a while (and then let calibre pick the changes up again).

The quirky bit is that if you chain multiple USBMS session within KOReader without a Nickel scan in between, you "lose" the previous session's on-device metadata cache changes (which basically only means that those books "disappear" from calibre metadata searches, as that's the only thing in KOReader that actually uses it).

TL;DR:

* Either follow a strict "KOReader USBMS -> Nickel scan" workflow
* Or setup your calibre to *not* use the Kobo driver, and use the plain dumb USBMS driver instead.

(Disabling calibre's automatic metadata management and/or the metadata refresh on connect in the Kobo driver may also help, but I haven't tested that, because it kinda ruins most of what I use it for in the first place ^^).

[NiLuJe]

Books imported another way won't suffer from this quirk, e.g.,

* The Calibre Wireless plugin warns you if you set your inbox to a folder already handled by another calibre driver, to avoid such metadata shenanigans (among other things).
* Anything else (e.g., OPDS) doesn't generate calibre metadata. (But calibre may then still manage to match it automatically in most cases (e.g., sane in-book metadata; the perfect example being the OPDS catalog from the same calibre library ^^)).

[Zoasa]

Many thanks for your reply. That explains a lot now. I'm trying out your proposals now. Regarding that Kobo driver, I've found only the settings under Plugins in Calibre. I've disabled the Kobo plugin, but somehow this has no effect. Where can I find this setting which you mentioned? Googling "calibre USBMS" leads me some python code for that driver, but no documentation unfortunately. Does this maybe has another name in the application or documentation?

[NiLuJe]

Disabling both the Kobo & Kobo touch device drivers should make Calibre gracefully downgrade to the stock USBMS driver (I don't actually know how it presents itself in the UI, as I've never used it).

Keep in mind that the same warnings I mentioned about mixing drivers apply here, too: this will most likely screw with the Kobo driver if you ever decide to re-enable it.

If you want to avoid that, you can make it use a different subfolder as the (on-device) Library root.

(Again, never done it myself, refer to the calibre docs for more details).