Calibre failing to transfer books to Android device

[DMcCunney]

Originally Posted by stumped

But Can calibre, via usb, cope if some books are added or removed from the device by other means. I.e. does it rescan at each connect or run any integrity checks. Would mixing multiple transfer mechanisms account for the woes being discussed here?

Calibre rescans at each connect here, and picks up changes in the underlying files on device. I have deleted stuff from the device and Calibre sees it's gone next time I do file transfers.

Part of my reason for this thread was Calibre failing to pick up stuff on device that was there, and figuring out why.

Mixing transfer mechanisms might be a cause of the issues I'm seeing, but I can't yet be certain.
______
Dennis

[DMcCunney]

Originally Posted by theducks

True
I think it is because they use different tracking methods.
Calibre writes metadata.calibre when it did the initial scan, then keeps it up-to-date withe each USB transfer

CC maintains its own DB as it transfers (CS) /receives (wdc)
CC is more complex, with mull metadata and covers

I'm closer to where I'd like to be. I told CC where books live on the device I'm currently dealing with. I can start Calibre Content Server, connect to it via Wifi from the device, and select and download volumes to device from CC.

Disconnect, run FBReader, and it sees the new volumes in the directory where I configured it to look for books.

Plug the device into the desktop via USB cable, run Calibre, and it rescans and sees the volumes on device I added through CC.

I can live with that, though I'd still like to have a tool that will show me what all is on the device in the same way Calibre on the desktop can show what's in the master library.

CC appears to to that for stuff it transfers, but the vast majority of volumes on any of my devices get there through either USB cable or popping out the external microSD card, putting it in an adapter, and using Connect to Folder to add volumes, then putting the card back in the device and telling FReader to rescan so it sees them. That's simply the fastest way to do it, and I'm not changing that workflow.
______
Dennis

[DMcCunney]

Originally Posted by ilovejedd

metadata.calibre contains full metadata as well (although obviously, no covers). Requires a lot of RAM, though. I once tried sending ~5K books to SD card and Calibre Portable (32-bit Windows 7) just crashed repeatedly. Haven't tried yet on 64-bit Windows.

I use 64 bit Calibre under Win10, though I have Calibre Portable as well.

For stuff in an external SD card, I pop the card from the device, put in in an adapter, and plug it into a USB port on the desktop. It appears as a drive in Windows, and I can tell Calibre Connect to Folder, and select the folder on the card that is the base of the eBook storage hierarchy. I haven't tried to copy 5K of books like that, but I suspect I could without crashing. It would just take a while.
______
Dennis

[ilovejedd]

Originally Posted by DMcCunney

For stuff in an external SD card, I pop the card from the device, put in in an adapter, and plug it into a USB port on the desktop. It appears as a drive in Windows, and I can tell Calibre Connect to Folder, and select the folder on the card that is the base of the eBook storage hierarchy. I haven't tried to copy 5K of books like that, but I suspect I could without crashing. It would just take a while.

If you use 32-bit, I highly doubt it. The problem is Calibre runs out of RAM (I think 32-bit Windows limits RAM usage on a single program to 2GB). I've actually tried with copying the books separately (via Save to Disk) and generating metadata.calibre via "Update cached metadata on device" and updating cache with 5K books always crashes after a while.

I have no problems copying a smaller number of books via Connect to Folder and haven't really tested where the upper limit might be on 32-bit. I expect there wouldn't be any issues with 64-bit Windows and 64-bit Calibre either.

[chaley]

Since the discussion seems to be happening here: CC does not see books transferred via cable. It sees books that it transfers via one of its connection types. If you use the cable then you shouldn't use CC.

CC can see a complete calibre library stored locally. Note that the local library is a source for books, not a CC library.

[DMcCunney]

Originally Posted by ilovejedd

If you use 32-bit, I highly doubt it. The problem is Calibre runs out of RAM (I think 32-bit Windows limits RAM usage on a single program to 2GB). I've actually tried with copying the books separately (via Save to Disk) and generating metadata.calibre via "Update cached metadata on device" and updating cache with 5K books always crashes after a while.

32 bit Windows has a 4GB address space limitation because it is 32 bit, and 32 bit Windows itself can only see and use about 3.2GB if you have 4GB RAM installed. (On my old 32 bit desktop, I had a RAMdisk driver that could see and use that extra RAM, and had a 768MB ramdisk seen as Z: I found uses for.) IIRC, you're correct about the 2GB program limit.

I have Calibre Portable on a thumbdrive to handle cases where I might want to bring it up on a laptop or netbook where full Calibre wasn't installed while traveling (and I have a subset of my full Calibre Library on a 256GB thumbdrive.)

I would not try to use Calibre Portable on the 64 bit desktop.

I have no problems copying a smaller number of books via Connect to Folder and haven't really tested where the upper limit might be on 32-bit. I expect there wouldn't be any issues with 64-bit Windows and 64-bit Calibre either.

Out of curiosity, why aren't you using 64bit Calibre? Why restrict yourself to Calibre Portable?
______
Dennis

[ilovejedd]

Out of curiosity, why aren't you using 64bit Calibre? Why restrict yourself to Calibre Portable?

Because 32-bit Windows can't run 64-bit programs and I need 32-bit for some legacy devices.

I prefer portable apps for, well, portability. Also for ease of backup and restore. If one doesn't need multi-platform support, it's much faster and easier to just copy the Calibre Portable folder than use the import/export function in Calibre when switching computers.

Mind, one can actually convert Calibre Portable to 64-bit. Just install 64-bit on a PC then replace the app folder on Calibre Portable with the 64-bit install folder. Obviously, then Calibre Portable wouldn't work on 32-bit Windows.

[DMcCunney]

Originally Posted by chaley

Since the discussion seems to be happening here: CC does not see books transferred via cable. It sees books that it transfers via one of its connection types. If you use the cable then you shouldn't use CC.

CC can see a complete calibre library stored locally. Note that the local library is a source for books, not a CC library.

I planned to open a thread in the CC support forum for this, but since you're here...

I bought CC in the first place as a tool I could run on device that might let me examine what was on my device, in the same way I can run Calibre on the desktop to see what is on it. The inherent limitation is that CC doesn't see stuff transferred by cable, and most of mine is.

I've found and am examining the FAQ you linked to. I tried setting Cloud Connection to point to a local library. On my device, that's /sdcard/Books. It doesn't work because there isn't a valid metadata.db file, which is what I expected.

CC is still useful as an alternate way of getting Calibre content on device and working for that purpose.

But I'd still like to find a program that can read the existing Calibre metadata and display what I have on the device. Are you aware of one?

Thanks in advance for any advice you might offer.
______
Dennis

[DMcCunney]

Originally Posted by ilovejedd

Because 32-bit Windows can't run 64-bit programs and I need 32-bit for some legacy devices.

Well, 64 bit Windows will run 32 bit Windows programs. What sort of legacy devices are the problems?

I prefer portable apps for, well, portability. Also for ease of backup and restore. If one doesn't need multi-platform support, it's much faster and easier to just copy the Calibre Portable folder than use the import/export function in Calibre when switching computers.

I'm in favor of portability, but the limitation you are encountering would be a showstopper here.

In terms of portability, I also have Calibre under bit Linux, and the fact that it behaves identically to the Windows version is a boon.

Mind, one can actually convert Calibre Portable to 64-bit. Just install 64-bit on a PC then replace the app folder on Calibre Portable with the 64-bit install folder. Obviously, then Calibre Portable wouldn't work on 32-bit Windows.

I have an older Windows device or two that are still 32 bit, and why I got Calibre Portable in the first place. I can count on the fingers of one hand the times I've used it.
______
Dennis

[ilovejedd]

Originally Posted by DMcCunney

I'm in favor of portability, but the limitation you are encountering would be a showstopper here.

Ah, that wasn't actually normal use case for me so certainly not a showstopper.

It was merely a test on the number of files in a single folder that FAT32 can handle and how responsive KOReader on Kindle is when browsing directories with thousands of files. It just so happened Calibre crashed when using Connect to Folder/Send to Device and I decided to investigate why.

I usually just download my books wirelessly (Dropbox, Calibre-PHP server via browser or COPS OPDS depending on the device/app). Normal use, I haven't had any issues with Calibre Portable at all. Thus, I don't see any pressing need to switch to 64-bit Calibre.

[chaley]

Originally Posted by DMcCunney

But I'd still like to find a program that can read the existing Calibre metadata and display what I have on the device. Are you aware of one?

No. There are some apps like CC that can read a metadata.db in the cloud. As you said, CC can read a local metadata.db. CC stores all calibre metadata locally for books that it transfers.

You seem to be asking for an app that can extract all the calibre metadata from a set of books on the device. Some reader apps can extract some metadata, but I have never seen one that can extract all metadata. In fact, I haven't seen one that can handle custom columns at all.

[DMcCunney]

Originally Posted by chaley

No. There are some apps like CC that can read a metadata.db in the cloud. As you said, CC can read a local metadata.db. CC stores all calibre metadata locally for books that it transfers.

But CC creates its own metadata store that is not compatible with the one Calibre creates, and can hence only be used for material CC transfers.

If I were starting from scratch, using CC and only doing wireless transfers would be tempting. But I have multiple devices, Wifi is problematic on a couple, and a wired connection or dropping things directly onto an sdcard mounted as a drive is simply faster.

You seem to be asking for an app that can extract all the calibre metadata from a set of books on the device. Some reader apps can extract some metadata, but I have never seen one that can extract all metadata. In fact, I haven't seen one that can handle custom columns at all.

I'm think what I'm asking for is something that can read metadata.calibre on device and display it. Poking around, it looks like it contains the basic data I want.

My use case is wanting to know what books are on device without opening my eBook viewer. I can look at /sdcard/Books in a file manager to get a files list, but that's insufficient.

I don't need all metadata just to see if a book exists on device. Title and Author would do in a pinch, but Size, Date added, Series, and Tags for classification would be icing on that cake. I don't need covers, custom columns and the like.

But either way such a thing doesn't seem to exist. (There are SQLite viewers/editors for Android and I've looked at several, but they don't do what I'd like.)

I find that lack mystifying, but I appear to be stuck with it.

Thanks for chiming in.
______
Dennis

[ilovejedd]

Originally Posted by DMcCunney

But CC creates its own metadata store that is not compatible with the one Calibre creates, and can hence only be used for material CC transfers.

That's likely for speed. I remember seeing stats for CC and some people had 50,000 books there. On Android, a JSON with that many books is likely going to be insupportable.

Originally Posted by DMcCunney

I'm think what I'm asking for is something that can read metadata.calibre on device and display it. Poking around, it looks like it contains the basic data I want.

My use case is wanting to know what books are on device without opening my eBook viewer. I can look at /sdcard/Books in a file manager to get a files list, but that's insufficient.

I don't need all metadata just to see if a book exists on device. Title and Author would do in a pinch, but Size, Date added, Series, and Tags for classification would be icing on that cake. I don't need covers, custom columns and the like.

But either way such a thing doesn't seem to exist. (There are SQLite viewers/editors for Android and I've looked at several, but they don't do what I'd like.)

I find that lack mystifying, but I appear to be stuck with it.

Yep, none exist.

Not really mystifying. I'm guessing most who use Calibre and Android just use Calibre Companion rather than USB. Either that or they just browse the books via their preferred reading app without need for a third-party listing/browsing app.

That said, why do you wish to avoid using your ebook viewer in order to view books stored on device? Does it take a really long time to load or something?

[DMcCunney]

Originally Posted by ilovejedd

That's likely for speed. I remember seeing stats for CC and some people had 50,000 books there. On Android, a JSON with that many books is likely going to be insupportable.

My complete Calibre library is in that range. But only a small subset exists on mobile devices. For instance, I don't normally try to read PDFs on a device, as they tend to be created assuming larger screen sizes than a mobile device will have. By preference, I don't get PDFs unless no other format is available.

On the device I'm currently wrestling with, there are about 1,700 volumes. That's a supportable amount.

And I prefer ePub, and convert other formats to ePub when possible. It's a better for devices, and a better underlying storage format. (I got Calibre in the first place for conversion - I had a Palm OS device that handled Mobi via the first Mobipocket viewer to be made, because Palm devices were Mobi's first target. It didn't handle ePub because that format did not then exist. Using Calibre, I could convert ePub to Mobi and read it on that device.)

Yep, none exist.

As I discovered.

Not really mystifying. I'm guessing most who use Calibre and Android just use Calibre Companion rather than USB. Either that or they just browse the books via their preferred reading app without need for a third-party listing/browsing app.

That may be the case. I've explained my practice above.

That said, why do you wish to avoid using your ebook viewer in order to view books stored on device? Does it take a really long time to load or something?

I use FBReader. I began using it in the days it was written in C. It ran on Windows and Linux, and could display eBooks in supported formats on my desktop. It handled ePub, Mobi, FB2 and an assortment of other things, and not having to care what format a book used and have to use more than one viewer was a huge win.

It also supported the Plucker format, used by the Plucker offline HTML viewer for Palm OS devices. The Plucker desktop could convert HTML to a format that could be displayed by the Plucker viewer on the Palm device. Plucker supported hyperlinks, embedded illustrations, text attributes, color (on color devices) and custom fonts (on OS5 devices.)

The first target was documentation for stuff I dealt with as a sysadmin, much of which was in HTML. I could carry a documentation library in my pocket.

It was a hop, skip, and jump to converting things like Project Gutenberg files. I wound up with 4,000+ converted Plucker files, and I could use FBReader on the desktop to view them. Some stuff I only had as converted Plucker files because the original HTML got lost. (I've subsequently discovered Calibre handles Palm PDB files as an input format, and happily converted some stuff I only had a Plucker files. To my delight, the conversion produced working ePub ToCs with no intervention required by me.)

But while I could read just about anything on the PDA, I needed to have half a dozen different viewers, and remember which book was in which format viewed by what program. That was just nuts.

When I began swimming in the Android pool, FBReader for Android, written in Java, was the obvious choice for eBook viewer, because I again didn't have to care what format a book was in. It handles it native, or via a plugin. It doesn't handle DRM, but I don't get books with DRM and don't care.

It comes up fast enough, thank you.

The issue with using FBReader to see what's on device is the way it displays things. The Library view divides into sub-folders:
Favorites
Recent
By Author
By Title
By Series
By Tag
File tree

And By Title further sub-divides by first letter of Title name (but it ignores things like A or The if it's the first word in a Title). (It's a feature request outstanding for a while that FBReader use the same practice for By Author, with the first letter of the Author's last name the selector.)

What it doesn't offer is an All Books display, and that's what I'm specifically trying to get. It has a Search function I can use to drill down and find things, assuming I recall the correct title or author, but that's not the problem I'm trying to solve.

Ultimately, it's about "What's the fastest way to do this?", where "this" is scroll though a list of books on device to see what's there.

That question is also why I use a USB cable or plug an external SDcard in an adapter plugged into a USB port where it shows as a drive, and use Connect/share Connect to Folder in Calibre to add books. It's simply faster than Wifi. If I didn't keep as many books on device, or want to add/remove the number that I often do in a session, CC would be fine. Given the size of my library, the number of files I keep in devices, and the amount I transfer in a session, CC simply isn't a good fit for me. It's a splendid piece of work, but I'm not the kind of user it's aimed at.
______
Dennis

[ilovejedd]

Originally Posted by DMcCunney

I use FBReader.

The issue with using FBReader to see what's on device is the way it displays things. The Library view divides into sub-folders:
Favorites
Recent
By Author
By Title
By Series
By Tag
File tree

And By Title further sub-divides by first letter of Title name (but it ignores things like A or The if it's the first word in a Title). (It's a feature request outstanding for a while that FBReader use the same practice for By Author, with the first letter of the Author's last name the selector.)

What it doesn't offer is an All Books display, and that's what I'm specifically trying to get. It has a Search function I can use to drill down and find things, assuming I recall the correct title or author, but that's not the problem I'm trying to solve.

Ultimately, it's about "What's the fastest way to do this?", where "this" is scroll though a list of books on device to see what's there.

Perhaps use a different reader just for browsing then?

I'm not sure what formats Moon+ Reader supports but it has an All Books display mode (I always use EPUB on Android and iOS).

I'm not overly familiar with options on Android as I'm primarily on iOS and my preferred reading app (Marvin) already displays all books by default.