[DavidTC]

Note that you can already "fetch all books not on device" in CC's cloud connection by tapping "Newest" and then "Download All". CC queues all the books, skips books that are already on the device (a book has a matching UUID), and complains if a book doesn't have a usable format. This method has two problems. 1) It isn't obvious that it can be used for this purpose. I don't think you have twigged to it and I didn't remember to mention it. 2) It queues all the books even if the books cannot be downloaded (no acceptable format) or are already downloaded. In the second case the processing to determine whether a download is necessary happens while the queue is being processed.

Oh, I've discovered that, and also that you can filter books to just be the ones not on the device, and download just those. (I don't have any books not in acceptable format.)

The problem *there* is, as I said back where I started with all this, my original suggestion #1: This requires keeping two copies of everything on the device, which is why I asked for the ability to 'download' a book by just pointing the record at the existing file, instead of making a copy. This method also, from what I understand now, won't correctly get calculated fields.


I fear this converstation is getting oddly bogged down in weird details, including talking about a lot of things I was wrong about or didn't actually know,and also bogged down in my thoughts about how CC is somewhat too liberal in the files it accepts, which isn't relevant to anything.

So let me summarize what I am thinking *now* about adding books from the cloud:

1) Currently, CC ends up having to use a mix of metadata.db and internal book metadata, and can *still* get it wrong when getting it from a cloud connection. (Right? If I understand correctly, calculated fields are only in the book metadata, so won't be there usually.)

2) The internal book metadata *is* correct (For books that Calibre supports) after Send to Device books.

3) Also CC does needs metadata.db to exist (Even if it is incomplete in some fields), so it can download that to show a listing of books from the cloud.

4) But there is also a metadata 'database' that follows 'Send to Device' folders around, called metadata.calibre. This appears to be...not actually a database, but instead a text file with all that information. In fact, looking at it, it has information that even metadata.db doesn't have, including calculated fields! (Which also nicely solves the problem of non-supported metadata files!)

So let me pause here and make a suggestion:

Perhaps the problem with cloud libraries is that CC is only supporting *the wrong type of cloud*. CC supports putting the entire library up there *as is*, with wrong metadata and everything.

However, it is almost as easy for people to put books in the cloud using 'Send to Device' folder in the cloud concept.

This would require CC to read metadata.calibre to show books, in addition to metadata.db.

And now two obvious objections arise: Not only is metadata.calibre a new format for CC to support, it's *huge*. My library is ~5000, and my metadata.db is 25 megs. But when putting those books in a folder device, metadata.calibre is 124 megs! Which causes obvious problems if the intent is to download it from the cloud every time someone wants a book.

There's an easy solution to the size: That file should be zlib compressed. When I compressed that file using the fastest compression level, it was only 12 megs. Considering the speed of the compression (4 seconds for me), vs the likely speed difference transferring of 12 megs vs. 124 megs over a USB connection, it should be doing that anyway! (Not sure if any third party software uses metadata.calibre, but if so, an option to additionally generating the uncompressed file would solve that.)

But the first point still holds, and is work. This would requires writing code in CC to *read* the format of metadata.calibre. (It appears to be json?)

I am not sure of the level of required work with *that*. But it seems like, in addition to fixing the metadata, it would solve a lot of problems that cloud connections currently have to work around.

It stops the 'How do I do more than one library?' questions. It allows multiple libraries, and even multiple *computers*, to put files there, as long people can point Calibre at that synced-to-cloud directory. It allows people to put only certain books there, instead of their entire library.

Some people, of course, will keep using the old way, either because they don't know how to work Reading List and they want their whole library, or they're lazy. And that's fine. They just risk bad metadata for calculated fields, exactly as already is true

To continue:

4) It would be best if CC could support plugging the device into Calibre and have files sent that way.

5) Likewise, the current 'cloud connection' results in duplicate files when the 'cloud' is local.

Second suggestion:

If CC can parse metadata.calibre, couldn't it just notice (Either automatically or manually) a new or updated metadata.calibe.gz in its *own* folder, and adding files it doesn't have which are listed in that, using the paths that are in that?

This metadata.calibre.gz could come from either a synced Send-to-Device folder, or from plugging the device into a computer.

And this also short-circuits any problem of marking such books in the metadata, like I was talking about above. The only hypothetical problematic situation here is that pirate are insanely producing pirated book folders with calibre.metadata.gz files in them and people are copying those entire folders straight to devices without using Calibre at all. Which is so obviously silly and delibarate it's not worth worry about.