Calibre failing to transfer books to Android device

[BetterRed]

Strange - did you try using Windows File Explorer->Send to->Xxxxx phone, that's how i move stuff onto my phone and tablet, including files stored in calibre library book folders.

I only keep a very limited amount of stuff on them - a few reference texts and what I need right now. Hence I don't use any elaborate shelving or scaffolding schemes, an old version of ES File Explorer is adequate.

BR

[DMcCunney]

Originally Posted by DMcCunney

I may later try plugging in the device, connecting through MTP, and dropping an eBook file of two directly into the Books folder from Windows Explorer rather than using Calibre. I use FBReader on device, and telling ti to Rescan after will pick up the new volumes.

I may also try the suggestion of having Calibre send to a different location. That's a good thought.

And this gets curiouser and curiouser, as Alice said.

I can tell Calibre to use a different directory on device and successfully transfer a test file to it.

I can export volumes from Calibre to a directory (to preserve the metadata). I can connect the device via USB cable, and drop them into the Books directory from Windows Explorer via MTP. I can run FBReader after, tell it to Rescan, and have it pick up the new additions.

But when I connect to the device via Calibre and it detects volumes already on device, it doesn't see them. An assortment of books I had previously placed on device via Calibre simply do not show in the On Device column, even though they are on device.

In the attached screenshot, the ones that are not checked in the On Device column are on device, but are not being detected by Calibre, and this is after I manually removed the driveinfo.calibre and metadata.calibre files to force Calibre to rebuild its metadata from scratch.

(I've opened a terminal on device and poked around on the Linux level, too. Books is under /storage/emulated/0, and I don't see any obvious oddities or permissions issues.)

I am very confused. I'm not quite annoyed enough to hardware reset the device to factory stock and recreate my configuration, but I can see that approaching in my rear view mirror.
______
Dennis

[DMcCunney]

Originally Posted by BetterRed

Strange - did you try using Windows File Explorer->Send to->Xxxxx phone, that's how i move stuff onto my phone and tablet, including files stored in calibre library book folders.

No. For the current issue, I was just manually dropping files onto device from a directory on the host to a directory on device. I wasn't using Send To.

I only keep a very limited amount of stuff on them - a few reference texts and what I need right now. Hence I don't use any elaborate shelving or scaffolding schemes, an old version of ES File Explorer is adequate.

We differ. I keep rather a lot on device (about 1.3GB at the moment.) I use either Amaze or Solid Explorer, but no matter.

I have enough stuff on device that I want Calibre to manage it, and I want Calibre to accurately list what's actually on the device. Right now, it isn't. See my most recent post.
______
Dennis

[DNSB]

One question. If you switch to device view (click on the Device icon dropdown and then Show books for where ever they are stored), do any of the books that don't have a checkmark in Library view show as being present? If you see those books, can you try right-clicking on a book that doesn't have a check mark and use the Match to library function.

[DMcCunney]

Originally Posted by DNSB

One question. If you switch to device view (click on the Device icon dropdown and then Show books for where ever they are stored), do any of the books that don't have a checkmark in Library view show as being present? If you see those books, can you try right-clicking on a book that doesn't have a check mark and use the Match to library function.

I've looked at device view previously. I did it again, and a quick scan did not show any that did not appear in Library's On Device column even though actually present appearing in Device view. (I assume both views are generated from the same underlying data, so I'd have been startled if any did.)

I prefer ePub, but use FBReader as viewer, and it can handle Mobi and PDF among others. I did find a Mobi file, which I had previously deleted (I thought) using FBReader's delete function. I deleted it via Calibre from Device View, then converted the Mobi file via Calibre to ePub, and could transfer it to device using the normal Calibre function.

Attempts to transfer other files in that session failed with the same transfer error I saw before.
______
Dennis

[DNSB]

Originally Posted by DMcCunney

I've looked at device view previously. I did it again, and a quick scan did not show any that did not appear in Library's On Device column even though actually present appearing in Device view. (I assume both views are generated from the same underlying data, so I'd have been startled if any did.)

The library view shows books in your calibre library, the device view shows books that are present on your device -- two separate data sets. The checkmark indicates where the title/author match.

I get quite a few ebooks that show as on the device but not in the calibre library. Most commonly when I have synchronized ebooks from Kobo's servers that have not been imported into calibre (very often these are library books borrowed through the Overdrive integration * ) but I have also seen this when I have been mucking with metadata (title, author) and the book on the device has not been replaced/updated. I also get books that show as present twice when the synchronized and sideloaded versions both match the library.

* There are quirks in my handling of ebooks. Books that I buy are happily deDRMed, format shifted, etc.—they are mine and I will keep accessible backup copies. Library books are not mine and get the it's here for a week and I am not going to tempt myself to keep a copy treatment.

[DMcCunney]

Originally Posted by DNSB

The library view shows books in your calibre library, the device view shows books that are present on your device -- two separate data sets. The checkmark indicates where the title/author match.

So the views aren't generated from the same underlying data. Thank you, that's a missing piece.

I get quite a few ebooks that show as on the device but not in the calibre library. Most commonly when I have synchronized ebooks from Kobo's servers that have not been imported into calibre (very often these are library books borrowed through the Overdrive integration * ) but I have also seen this when I have been mucking with metadata (title, author) and the book on the device has not been replaced/updated. I also get books that show as present twice when the synchronized and sideloaded versions both match the library.

Library books aren't in this mix. Noted on duplication when matching volumes are synced and sideloaded.

* There are quirks in my handling of ebooks. Books that I buy are happily deDRMed, format shifted, etc.—they are mine and I will keep accessible backup copies. Library books are not mine and get the it's here for a week and I am not going to tempt myself to keep a copy treatment.

And I don't get stuff with DRM. I do muck about with metadata, but try to do that before adding them to any device.

I'd do the same with library books. If I'm borrowing it from the library in the first place, it's likely to be something I want to read but probably not keep.

I'm still no closer to understanding why Calibre is suddenly failing on transferring books via USB cable but Wifi transfers via content server work.

And I'd really like Calibre's list of what is on device to match what's actually there.

But I'm learning things and looking more closely at Calibre Companion and how to integrate it into my eBook workflow, so I won't count this all as a total loss.
______
Dennis

[DMcCunney]

Originally Posted by DMcCunney

Library books aren't in this mix. Noted on duplication when matching volumes are synced and sideloaded.

I'm still no closer to understanding why Calibre is suddenly failing on transferring books via USB cable but Wifi transfers via content server work.

And I'd really like Calibre's list of what is on device to match what's actually there.

And I'm getting closer to understanding. As part of setting up Calibre Content Server and using Calibre Companion to transfer volumes via Wifi, I also connected via USB cable to see what Calibre thought was currently on device aside from the stuff it wasn't seeing.

I was able to select and transfer several newly added volume to device via USB just fine. It bombed with the same error I saw previously if I attempted to add the books not shown as On Device in Calibre but actually present. I don't think it's a file name collision, as I've managed to have duplicate volumes on device before, and simply deleted one of them when I discovered it. I'm not sure what Calibre is choking on, but I'll settle for it not doing so.

I think removing the "on device but Calibre doesn't see them" volumes from the device should iet me use Calibre to properly add them and have them visible in Calibre's listing.

The current challenge is learning how to use Calibre Companion for what I bought it to do - provide a way to look at the Calibre metadata and display what is currently on device from the device itself. This is apparently possible, though the FAQ entries I'm looking at seem to say I can't select and read a book from that display. That's fine. I just want to see what's present.

Fun, for suitable values of the term.
______
Dennis

[chaley]

Originally Posted by DMcCunney

The current challenge is learning how to use Calibre Companion for what I bought it to do - provide a way to look at the Calibre metadata and display what is currently on device from the device itself. This is apparently possible, though the FAQ entries I'm looking at seem to say I can't select and read a book from that display. That's fine. I just want to see what's present.

You seem to have questions about how to use CC. For example, I don't understand what you mean when you say you "can't select and read a book from that display" because launching a book reader app on a book is one of CC's primary functions. This isn't the right forum to discuss CC. I suggest you ask in in the CC forum.

[DMcCunney]

Originally Posted by chaley

You seem to have questions about how to use CC. For example, I don't understand what you mean when you say you "can't select and read a book from that display" because launching a book reader app on a book is one of CC's primary functions. This isn't the right forum to discuss CC. I suggest you ask in in the CC forum.

I plan to do just that.

What I am trying to learn is the mechanism. CC is designed to manage Calibre volumes transferred to device via Wifi. Mine, in general, aren't. I transfer volumes via a USB cable connection. My question is how to get CC to see and manage those volumes. It appears possible - I'm learning how.

But yes, this is not the forum for those questions.
______
Dennis

[ilovejedd]

Originally Posted by DMcCunney

What I am trying to learn is the mechanism. CC is designed to manage Calibre volumes transferred to device via Wifi. Mine, in general, aren't. I transfer volumes via a USB cable connection. My question is how to get CC to see and manage those volumes. It appears possible - I'm learning how.

I believe via USB/SD card is somewhat all or nothing.

CC can read a Calibre library stored on the phone but you have to mirror/copy the structure of your Calibre library exactly (the folder that contains metadata.db).

I don't believe using Send to Device or Save to Disk will work if you want these to work with CC.

[theducks]

Originally Posted by ilovejedd

I believe via USB/SD card is somewhat all or nothing.

CC can read a Calibre library stored on the phone but you have to mirror/copy the structure of your Calibre library exactly (the folder that contains metadata.db).

I don't believe using Send to Device or Save to Disk will work if you want these to work with CC.

True
I think it is because they use different tracking methods.
Calibre writes metadata.calibre when it did the initial scan, then keeps it up-to-date withe each USB transfer

CC maintains its own DB as it transfers (CS) /receives (wdc)
CC is more complex, with mull metadata and covers

[stumped]

Originally Posted by theducks

True
I think it is because they use different tracking methods.
Calibre writes metadata.calibre when it did the initial scan, then keeps it up-to-date withe each USB transfer

But Can calibre, via usb, cope if some books are added or removed from the device by other means. I.e. does it rescan at each connect or run any integrity checks. Would mixing multiple transfer mechanisms account for the woes being discussed here?

[ilovejedd]

Originally Posted by stumped

But Can calibre, via usb, cope if some books are added or removed from the device by other means. I.e. does it rescan at each connect or run any integrity checks. Would mixing multiple transfer mechanisms account for the woes being discussed here?

Dunno about Android but on Kindle or using "Connect to folder" (I use the latter for KOReader), it rescans at each connect.

[ilovejedd]

Originally Posted by theducks

Calibre writes metadata.calibre when it did the initial scan, then keeps it up-to-date withe each USB transfer

CC maintains its own DB as it transfers (CS) /receives (wdc)
CC is more complex, with mull metadata and covers

metadata.calibre contains full metadata as well (although obviously, no covers). Requires a lot of RAM, though. I once tried sending ~5K books to SD card and Calibre Portable (32-bit Windows 7) just crashed repeatedly. Haven't tried yet on 64-bit Windows.