Suggestions for the CC cloud connection

[DavidTC]

Here's are a few suggestions. I am not sure how easy they are.

First, to explain: I keep wanting to use the 'local folder' cloud storage instead of the CC connection. All my devices have enough room for my entire library, and I already have cloud synchronization of various other directories set up. I would love to just be able to add things and they'd magically get on my device after a slight delay, or I push the widget labeled 'Sync' if I want them sooner.

Yes, I know there's a 'connect at specific times', and I know I can set up Reading Lists to get always make sure all books are on the device...except that assumes I have Calibre always running, and that it syncs before I leave me house. I would much prefer to use Cloud storage...or, rather, synced cloud storage to local folders, like I do for *everything else* on my Androids.

What keeps stopping me is that the Cloud interface is incredibly annoying. First, I can't filter on multiple things, or sort, and the display is not the same, and I have to 'download' the book, and now it's in the *other* interface, etc, etc.

I though, for the longest time, there was a reason this had to be. I mean, it's how the Calibre server works, right? Except...that's not the Calibre server. And I can't figure out any good reasons this limitation makes sense.

Let me dissect my thoughts:

First, sometimes the cloud is a local path.

Suggestion #1: Why not have the option to use the local cloud path *as* the local book path, and instead of 'downloading', the book merely gets added to the database with the path it already has? (And also need to make sure that file doesn't get deleted upon book deletion.) Why does it need to be put *somewhere else*?

Suggestion #2: And then that leads to the obvious idea that if 'downloading' is just adding a database record, why not just automatically do it for *all* the books when 'connecting' to the local cloud folder? CC reads metadata.db, it fills up a database with all those books, tada.

Suggestion #3: And then this leads into my confusion of why CC doesn't do this for *actual* cloud storage, too. CC already downloads metadata.db, so just fill up CC's database with all those book records, and put cloud:blah/whatever.epub as a 'virtual path' in the path field.

And now it's in the *real* CC interface. Not only is this better, but whatever code is generating the 'pseudo-calibre-server interface from metadata.db' that Cloud users get can be removed.

Then when they 'Read' the book, download it and change the path in CC's database.

Possible issues:

1) Sometimes people just want some of their books 'on their device', but frankly, I suspect any complaints in that direction are just people not paying attention to filtering options. (And if it's some sort of 'I want to hide these books in my library from other people', uh, those books should be in a different library, not in one they put in the cloud and hooked CC to!)

2) Virtual paths are going to create some odd interaction with direct connection to Calibre, if people are trying to do both. A book could be listed as on the device, but not actually on the device.

3) The entire library directory synced to CC has some odd dangers, especially if it's then connected directly to Calibre. These can that either can be dealt with...or 99% of them can be avoided by telling users that such a sync should be one-way. (Which is sad, because it sorta excludes the obvious feature of 'marking books read'.)

[chaley]

Originally Posted by DavidTC

Here's are a few suggestions. I am not sure how easy they are.

First, to explain: I keep wanting to use the 'local folder' cloud storage instead of the CC connection. All my devices have enough room for my entire library, and I already have cloud synchronization of various other directories set up. I would love to just be able to add things and they'd magically get on my device after a slight delay, or I push the widget labeled 'Sync' if I want them sooner.

[...]

Moderator Notice
Discussion moved to its own thread.

[chaley]

Originally Posted by DavidTC

Here's are a few suggestions. I am not sure how easy they are.

Thanks for the well-written suggestions.

The problem: doing what you ask for requires a complete rewrite of CC. CC is designed to be an offline book manager that integrates with calibre in various ways. Offline means that CC must have its own copies of books and metadata, and the user must be able to trust that the information is in fact available offline. In addition, there are a zillion technical problems related to integration with readers, read/write data, multiple book formats (extensions), integration with Amazon SW on the Fires, and the like. I am not going to go there.

You might be happier with one of CC's "competitors". For example, Hofferic's CalibreBox directly interfaces with libraries and uses that data. I don't know if he supports local libraries, but if not he might be willing to consider it. Calibre Cloud Pro supports local libraries, but the app seems to be no longer maintained. Leger Calibre does (did) what you want (I think) but it too is no longer maintained.

What keeps stopping me is that the Cloud interface is incredibly annoying. First, I can't filter on multiple things, or sort, and the display is not the same

This isn't correct. You can easily filter on two things by long-pressing the first one then using the navigation to examine what is left. You can do arbitrary filtering using CC's search. Finally, you can sort most book list displays by one of Title, Authors, or Series.

First, sometimes the cloud is a local path.

And much more often it is not.

Suggestion #2: And then that leads to the obvious idea that if 'downloading' is just adding a database record, why not just automatically do it for *all* the books when 'connecting' to the local cloud folder? CC reads metadata.db, it fills up a database with all those books, tada.

Others have asked for a variant of this, specifically to be able to automatically update CC's database and book files with information from the cloud. I am still considering the idea. And yes, I know that isn't what you are asking for.

Suggestion #3: And then this leads into my confusion of why CC doesn't do this for *actual* cloud storage, too. CC already downloads metadata.db, so just fill up CC's database with all those book records, and put cloud:blah/whatever.epub as a 'virtual path' in the path field.

Because the majority of users don't want all their books on the device, and because that would require maintaining a connection to fetch the book files and covers.

And now it's in the *real* CC interface. Not only is this better, but whatever code is generating the 'pseudo-calibre-server interface from metadata.db' that Cloud users get can be removed.

And this is one reason why it is a total rewrite of CC. The existing wireless device connection and all of its support would have to change. Something equivalent but non-cloud would be necessary to transfer the metadata.db and book files, because *lots* of users do not have their library in the cloud. And so on.

Possible issues:

1) Sometimes people just want some of their books 'on their device', but frankly, I suspect any complaints in that direction are just people not paying attention to filtering options. (And if it's some sort of 'I want to hide these books in my library from other people', uh, those books should be in a different library, not in one they put in the cloud and hooked CC to!)

Your suggestion both raises complexity and requires a permanent internet connection. The complexity: one must peruse the library in advance of being off-line (or on a limited data plan or paid roaming) to ensure that the books shown in the library are actually on the device. People won't remember to do this (I wouldn't) so they must have a connection or won't be able to read the book.

2) Virtual paths are going to create some odd interaction with direct connection to Calibre, if people are trying to do both. A book could be listed as on the device, but not actually on the device.

The number of weird interactions with the wireless device connection is enormous.

In an early version of CC we allowed downloading metadata without downloading books. Things didn't go well. Not only did calibre get confused about what is where (formats, covers, metadata, etc), but the users (at that point "testers") did as well.

3) The entire library directory synced to CC has some odd dangers, especially if it's then connected directly to Calibre. These can that either can be dealt with...or 99% of them can be avoided by telling users that such a sync should be one-way. (Which is sad, because it sorta excludes the obvious feature of 'marking books read'.)

There are lots of users who sync their entire library to CC using the wireless device connection. Some have in excess of 15,000 books.

Finally, my apologies for coming across as negative. I think that an app based on your well-written analysis would be something that many people would want to use. Unfortunately, CC is not that app.

[DavidTC]

Offline means that CC must have its own copies of books and metadata, and the user must be able to trust that the information is in fact available offline.

It seems like it would be a trivial toggle to show all books vs. only local books.

This isn't correct. You can easily filter on two things by long-pressing the first one then using the navigation to examine what is left. You can do arbitrary filtering using CC's search. Finally, you can sort most book list displays by one of Title, Authors, or Series.

Okay. Although I do note that you don't seem to be able to filter *twice*, although I guess you could just type it in, and additionally the listing and sorting is unconfigurable, unlik all the custom mods over in the main interface.

And that sorta proves my point. You're duplicating everything again in that interface, instead of just letting us use the real interface.

Others have asked for a variant of this, specifically to be able to automatically update CC's database and book files with information from the cloud. I am still considering the idea. And yes, I know that isn't what you are asking for.

It sorta *is* what I'm asking for. I'm just not sure as to why it should be limited to 'books already on the device'.

Because the majority of users don't want all their books on the device, and because that would require maintaining a connection to fetch the book files and covers.

I don't understand this objection. The book covers, yes, but that seems like an obvious fix of just having a placeholder image. (Which would seem to be a nice visual indicator the book isn't downloaded yet!)

And obviously they have to download the book to read the book. That's always true.

And this is one reason why it is a total rewrite of CC. The existing wireless device connection and all of its support would have to change. Something equivalent but non-cloud would be necessary to transfer the metadata.db and book files, because *lots* of users do not have their library in the cloud. And so on.

Huh? Why do you think something non-cloud would be needed? Why would you transfer the metadata.db differently? How would any of my suggestion even *apply* if the library wasn't in the cloud?

Okay, I clearly am not explaining this correctly:

Upon connecting to the cloud, what *currently* happens is CC downloads metadata.db, and presents a listing. When users click 'Download', the ebook and cover image files are transferred, and a record is added to the internal CC database. (If I am wrong about any of this, please tell me.)

Here is what I am suggesting: Upon connecting to the cloud, CC downloads metadata.db and adds all those records to the internal CC database, or updates existing records, making the file path be a pointer back to the cloud (Or, in the case of a local 'cloud', just pointing at where they already exist.) Then the user can use all the existing stuff built into CC to locate whatever book they want, and download the book (and cover) when they want them.

I have no idea why you think this is a big rewrite. It really seems like almost all the code required here already exists in the CC code. CC can already downloads and parses metadata.db, and it already adds books records from that, and it already downloads book files and covers from the cloud.

The sole thing needed is 'What a second, I don't actually have this book yet, let me connect to the cloud and get it and the cover before I launch it.'.

The number of weird interactions with the wireless device connection is enormous.

I thought this at first, but not really. From what I understand, Calibre *isn't* reading CC's interal database, but rather just getting a list of files, like it does with all other devices. So it doesn't really matter what's in the CC database.

The one cavet seems to be that CC needs to be smart enough to notice when it's being handed a book it already has a record for, but no file for, and change that existing record to point at the new file instead of adding a new record.

There are lots of users who sync their entire library to CC using the wireless device connection. Some have in excess of 15,000 books.

Yes. I have 5000. The problem is not 'how long syncing takes', it is 'When I get new books, remember to track down all my ereaders, and connect them to Calibre, and make it send new books'. Or, alternately, try to locate that stuff in a completely different interface from the cloud.

Everyone who uses the cloud server wants to be able to access (in some way), all their books somewhere where they don't have Calibre with them. (Or they wouldn't use the cloud server, they'd just use a Calibre connection, or the Calibre server.)

Assuming that sort of user, there are, basically, several combinations possible devices they can have:
1) Devices with enough space that someone can put their entire library on them.
2) Devices without that much space, or a user who is unwilling to dedicate that space.
a) Devices have a permanent free (or very cheap) internet connection.
b) Devices that are metered.
c) Devices that only have wifi.

People who are 1a, 1b, or 1c have an ideal solution. They can put their library in a cloud or a local shared drive, and get something like Foldersync and tell that to run on a schedule, possibly only on wifi. Then hook up CC to that folder. (This, I believe, the entire point of a 'local cloud' folder, unless there's some subset of people manually copying calibre library folders to devices.) This is what I would do.

Except CC makes this strangely complicated and makes people 'download' books from the local storage into their library, instead of just importing everything. Change that, and suddenly everything works magically. Entire libraries would just *be* on our devices.

People who are 2a would also probably like CC to transparently present their entire library to them in one interface, and it doesn't matter to them if the book is already there or not, except it's going to take one second longer. I don't see any problem at all there, this makes things much better for them.

People who are 2b, right now, are in a poor situation right now. First, they browse to where it should be, and when it's not there they have to connect to the cloud, download the *entire* metadata.db (Mine is twenty megs.), and then browse to the file there, all to download a 100k book!

With my suggestion, all those records would be local (They'd have to connect to the cloud occasionally to get a new metadata.db, but much rarer, and could do that when they're on a non-metered connection.) And so they'd only having to pay when they download a book.

So that is five, out of six possible combinations, that would operate much better if the CC database would just fill itself from metadata.db.

People who are 2c...well, they might, in theory, get confused as to what is going on, when they have books show up that they cannot access at all. Like I said, a toggle to only show local books would fix that. And, here's an interesting problem, the way it is currently, they literally have no way to find out if they have a specific book or not in their library when they can't get on wifi.

[DavidTC]

Oh, and if you really, really don't want to do 'Showing books that aren't on the device', I'll pull back and only ask for the first two suggestions. I will repeat them:

Specifically, when it's a *local* cloud folder, can we have the option to automatically 'Download' *all* books if they are not in CC already?

(Or, heck, give everyone using the cloud that option! Might take a while from an actual cloud, but some people have small libraries or really fast connections.)


And, while we're at it, so we don't waste space, can we have the option of having the storage location of books be the existing 'local cloud' folder, and the books (and covers) just stay in place instead of being copied around?

This is already something that is 99% percent possible...nothing is stopping us from setting the default folder to that path, and making a filename template that exactly matches how Calibre stores books...except for the fact it would be trying to copy the book on top of itself, which is obviously not a clever plan. (It might technically work, though!) Just put a toggle in the Cloud config that locks both the default path and the filename template config to that, and make CC not try to copy the file upon 'Download'.

EDIT: Of course, when the paths are configured this way, they have essentially dedicated their local storage to being their Calibre library, and hence they shouldn't be allowed to add books in *other* ways, or delete them. I don't think this will confuse people.

[DavidTC]

Well, my last idea didn't fly, but since then, I've discovered something very interesting: CC reads existing book in its directory when it connects to Calibre, and pulls the metadata from Calibre about those books. This almost lets me do what I want, in that I can point CC's 'Default folder' at a cloud-synced copy of the Calibre database directory, and then initiate a Calibre connection, and tada, it magically finds all the books and gives them records in CC. Eventually. After a really long time.

But this is only for a Calibre connection, and not a local cloud connection.

This shouldn't be too hard for a cloud connection. Please note I'm not asking to update all metadata, which you seem to have concerns about, just *adding* records when the book file is there without a record, exactly like the Calibre wireless device connection.

And obviously add covers the normal way that the Cloud connection adds covers, by grabbing cover.jpg. (Which is why I said this is for *local* cloud connections. Remote connections probably need a prompt, because that is additional data and could be costly.)

Of course, the question is, would anyone but me use this? I think, yes. A good deal of people wish to keep all their books on their device already, and while it might not have occurred to them, the amount of sync tools on Android are pretty large, with all sorts of configuration. No more worries about updating.

Additionally, reading the wishlist, there appear to be a lot of people who want CC to read existing books in its directory, and the solution is always 'connect to Calibre'. I suspect at least a few of those people are using the old 'Set up a folder device inside dropbox, dump all the books in it' trick to get books out (Which is what a lot of people did pre-CC.), at which point they'd benefit to switching to what I described above.

In fact, that is actually where I am coming from. I had a rooted Nook Simple Touch, I had a Dropbox folder as my Device, I had a syncing tool that kept it in sync on my Nook, and all my stuff was magically there whenever I picked up the device. If for some reason it hadn't synced, like I had left it unplugged so the wifi was off, I could find any wifi or even tether it to my phone and quickly sync it.

Now I have a Boox with Android 4 which can run CC, which is *massively* better at sorting and organizing the books (You wouldn't believe the plugboards and crazy templates I had set up for my Nook.) but the fact I have to remember to a) have my computer on and Calibre started, b) get my Boox, c) Fire up CC and connect, d) wait for it to finish, before I can walk out the door with new stuff, is incredibly annoying. (Or my current hack of syncing to a local folder and a local cloud connection to 'Download all', which results in two copies of everything! Heck, I even considering just living with that and making a script to hardlink the files, but then remembered sd cards are FAT32.)

I don't *need* a way to get my library to my device, I've had that covered for almost a decade...I need a program that understands all the Calibre metadata and organizes everything. And CC is the best program to do that by *several orders of magnitude* (Seriously, there is nothing even close. There is basically one other program that can even *conceptually* do this, eLibrary Manager, and it requires building specific queries in advance.), with the only annoyance of having to use non-ideal ways to get books *into* the program. Which I sorta understand, you want the Calibre metadata instead of scanning it from the epub...but I have the Calibre metadata *right there*! You even are already reading it!

[chaley]

I may seem like I am ignoring you. I'm not.

Originally Posted by DavidTC

Well, my last idea didn't fly, but since then, I've discovered something very interesting: CC reads existing book in its directory when it connects to Calibre, and pulls the metadata from Calibre about those books. This almost lets me do what I want, in that I can point CC's 'Default folder' at a cloud-synced copy of the Calibre database directory, and then initiate a Calibre connection, and tada, it magically finds all the books and gives them records in CC. Eventually. After a really long time.

And it really doesn't work very well.

Auto-scan is an old feature that depends on the fact that books in CC's folder were put there by CC's wireless device. It was added because for 3 years or so we didn't have any way to backup and restore CC's database.

Why put there by CC? Because CC reads the metadata out of the books (epubs only). When a book is sent from calibre to CC via the WD connection, the metadata in the book is updated to what is in calibre at that moment. The most important bit is the book's UUID, which permits CC and calibre to agree later on which book in the calibre library matches which book in CC's library.

If the book "appears out of the sky" (as they do in the cloud connection) then the metadata in the book is almost certainly wrong. They only way it is up-to-date is if the user did something like "polish" in calibre on all the books so that the metadata is updated in the copies in calibre's library. If a book is not up-to-date then there is a very good chance that the book will not match the correct book in calibre's library. Or perhaps not match any book at all.

In addition, it works only for epubs. All other formats are uploaded to calibre for analysys.

This shouldn't be too hard for a cloud connection. Please note I'm not asking to update all metadata, which you seem to have concerns about, just *adding* records when the book file is there without a record, exactly like the Calibre wireless device connection.

First, a bit of a peeve. Don't tell me what is hard and what isn't hard. I know my app (4 years of development) and calibre (6 years of development) inside out, and I am capable of estimating the complexity and work to do X.

What you are asking for is very similar to what we have already discussed and that I said I am considering. The cloud connection (no matter which one) has calibre's metadata.db in hand, so there is no reason at all to look in the books. Instead, CC would need to traverse the two databases (CC and calibre), find all books in calibre's db, then add them to CC, copy and all. This is a bit easier than updating metadata because one need not mess around with last-modified dates and the like.

One problem that has raised its head from time to time: the cloud connection cannot get the values of calibre custom columns "built from other columns". The values are not in calibre's metadata.db. People who use such columns will need to connect as a wireless device so that calibre and CC can fetch/update the values.

Of course, the question is, would anyone but me use this? I think, yes. A good deal of people wish to keep all their books on their device already, and while it might not have occurred to them, the amount of sync tools on Android are pretty large, with all sorts of configuration. No more worries about updating.

This is the fundamental question. At the moment it appears that less than 5% of CC users use *any* cloud connection. One hard statistic I have: the number of devices connected to Dropbox is less than 2% of the devices actively running CC. My guess is that the number of people using the local cloud connection is less than 100, but I don't know that.

I need data to decide what to do. For this reason and many others, I have added gathering of some statistics to CC V5. With this information I will have a better idea of how many people use what and do what. Of course, people can opt out, skewing the stats. People who opt out in effect won't exist.

Additionally, reading the wishlist, there appear to be a lot of people who want CC to read existing books in its directory, and the solution is always 'connect to Calibre'. I suspect at least a few of those people are using the old 'Set up a folder device inside dropbox, dump all the books in it' trick to get books out (Which is what a lot of people did pre-CC.), at which point they'd benefit to switching to what I described above.

There are three basic requests that CC doesn't handle, and won't handle. If you look through the reviews, several 1-star reviews make mention of these.

• Divorce CC from calibre. Let me import my books from where ever, edit metadata, and the like. This is in effect duplicating calibre. I have no interest in going there.
• Make CC a R/W client of calibre, with the ability to update calibre's metadata. This I would love to do, but the complexities are such that I wouldn't finish before entropy consumes the universe.
• And one irrelevant to the discussion, add a reader app to CC. I see no reason at all to do this, given the number of apps to choose from.

My bottom line: I am not going to start some hacking adventure until I am convinced that there is a large enough audience. I have been surprised at how few people use the cloud connection. At the moment I am not at all convinced that even 0.1% of CC users have libraries on an SD card, but I willing to be wrong. With this level of take up, doing anything special with all the attendant risk doesn't make much sense.

[DavidTC]

First, a bit of a peeve. Don't tell me what is hard and what isn't hard. I know my app (4 years of development) and calibre (6 years of development) inside out, and I am capable of estimating the complexity and work to do X.

Sorry. When I say things are easy, I just mean, 'It appears CC is already doing most of this in some manner'. But I don't know anything about the internals of CC, and very very little about the internals of Calibre, and certainly didn't want to imply I knew better than you.

Why put there by CC? Because CC reads the metadata out of the books (epubs only). When a book is sent from calibre to CC via the WD connection, the metadata in the book is updated to what is in calibre at that moment. The most important bit is the book's UUID, which permits CC and calibre to agree later on which book in the calibre library matches which book in CC's library.

First, you said you were considering something, and I'm not quite sure what it is, so if you have better ideas, nevermind.

And at this point, I've pretty much got a working system cobbled together from leaving Calibre running all day and getting CC to connect once a day, which it *usually* manages. I don't supposed I could at least get Tasker support to trigger a connection, which would let me do it when the device was actually *on* the correct network? Tasker can trigger an Activity in an app, so if you'd just expose 'Connect' as one...

But it's clear I've been wrong about some stuff, namely, I thought that CC *couldn't* read the metadata of epubs, or at least that was something you seriously wanted to avoid, so I was proposing hacks using the Calibre db. I thought reading book metadata was flatly something you wouldn't do.

But as CC *does* turn out read them, which is how it does the 'link them up with Calibre books'. Which means all my 'Get the metadata somehow' ideas were pointless.

That means, right now, 'Scan all books and add them using their internal metadata' could be possible, in theory, and the only reason CC doesn't have it as a menu item is that it would cause problems with people who just drop non-Calibre books in there?

Or is CC only reading the Calibre UUID, and then gets all the other information from Calibre?

[chaley]

Originally Posted by DavidTC

Why put there by CC? Because CC reads the metadata out of the books (epubs only). When a book is sent from calibre to CC via the WD connection, the metadata in the book is updated to what is in calibre at that moment. The most important bit is the book's UUID, which permits CC and calibre to agree later on which book in the calibre library matches which book in CC's library.

First, you said you were considering something, and I'm not quite sure what it is, so if you have better ideas, nevermind.

Actually, I did. See post #3, where I said

Originally Posted by chaley

Others have asked for a variant of this, specifically to be able to automatically update CC's database and book files with information from the cloud. I am still considering the idea. And yes, I know that isn't what you are asking for.

There are two variants.

• The first: update metadata in CC's library from the calibre database in the cloud. This makes me nervous because the metadata isn't complete. The values of custom columns built from other columns would be lost. I don't like operations that arguably corrupt the CC library.
• The second: find all books in the cloud library but not in CC's library and add them to CC. This is a more "acceptable" operation because there isn't any data loss; the results would be the same as manually adding the books. It is less acceptable in that the processing required to match all the books in calibre's library to all the books in CC's library would be slow and possibly memory consumptive. The only practical (performance-wise) way I can see to do it is to build two sets of UUIDs, one of the books in CC and the other of the books in calibre, then do a set difference (calibre - cc). The result will be the books that must be added to CC. And BTW: such a process would work for all cloud libraries, not just local libraries.

And of course I must decide whether the time spent building something like either one of these will be justified by its use. I am not convinced yet.

I don't supposed I could at least get Tasker support to trigger a connection, which would let me do it when the device was actually *on* the correct network? Tasker can trigger an Activity in an app, so if you'd just expose 'Connect' as one...

Auto-connection is triggered by a broadcast. There isn't an activity involved. I don't have a problem with exposing the broadcast receiver in CC V5 (in fact I just changed the manifest), but I don't know if "Tasker" can send broadcast events. The receiver name is com.multipie.cclibrary.ConnectionAlarmReceiver. It takes no extras.

But it's clear I've been wrong about some stuff, namely, I thought that CC *couldn't* read the metadata of epubs, or at least that was something you seriously wanted to avoid, so I was proposing hacks using the Calibre db. I thought reading book metadata was flatly something you wouldn't do.

But as CC *does* turn out read them, which is how it does the 'link them up with Calibre books'. Which means all my 'Get the metadata somehow' ideas were pointless.

No, they aren't. CC can read metadata from epubs, but the results are very iffy. As I said before, there is no reason to believe that the book has correct embedded metadata. Calibre does not update metadata in books in the library unless the user takes specific action. I do not wish to build new features that depend on a process that is known not to be reliable.

That means, right now, 'Scan all books and add them using their internal metadata' could be possible, in theory, and the only reason CC doesn't have it as a menu item is that it would cause problems with people who just drop non-Calibre books in there?

Or "calibre books" that have not had their embedded metadata updated, which is the normal state of books in a calibre library. Or books that are not epub. Or books that are in the wrong folder.

Embedded metadata is updated when calibre sends books via a "device" connection such as CC's wireless device. This is why extracting metadata from books (epubs) sent by calibre has a much better chance of working than books in cloud libraries or that come out of the sky.

Or is CC only reading the Calibre UUID, and then gets all the other information from Calibre?

CC extracts what metadata it can from epubs. It then constructs a metadata packet and sends that to calibre. If that packet contains a UUID then calibre will use it to match the book to a book in calibre's library. If there isn't a UUID or if the UUID is not in calibre's library then calibre goes through a process of "book matching", comparing the title and author of the book to books in calibre's library. Matching works if a close-to-exact match is found. This process very often matches nothing or the wrong book. The user must clean up any remaining mess.

If a book is matched, calibre sends the metadata from the library book to CC. This metadata will have the correct UUID in it, meaning that in the future CC and calibre will agree on the match. Of course, it could have incorrect metadata, in which case things aren't ideal.

For non-epub, CC sends the entire book to calibre for analysis. This is very slow.

[DavidTC]

The first: update metadata in CC's library from the calibre database in the cloud. This makes me nervous because the metadata isn't complete. The values of custom columns built from other columns would be lost. I don't like operations that arguably corrupt the CC library.

If you can't get custom columns, I entirely agree that's a bad idea. That would certainly screw me up. Almost all my 'Top, Middle Left, Middle Right, and Bottom' additional lines in the book listing are custom columns.

The second: find all books in the cloud library but not in CC's library and add them to CC. This is a more "acceptable" operation because there isn't any data loss; the results would be the same as manually adding the books. It is less acceptable in that the processing required to match all the books in calibre's library to all the books in CC's library would be slow and possibly memory consumptive.

I think most people are sorta thinking of this as a background operation, so the speed really isn't an issue as long as the rest of the interface can be used, and the new books just show up all at once at the end...and I have no idea how easy that would be.

And of course I must decide whether the time spent building something like either one of these will be justified by its use. I am not convinced yet.

You surely know the users of CC better than I do. I myself have trouble conceiving of why anyone wouldn't want their entire library on their device. (Except for the sole exception of not enough space.) But from the way you're talking, it seems like *most* people don't put their entire library on their device.

Auto-connection is triggered by a broadcast. There isn't an activity involved. I don't have a problem with exposing the broadcast receiver in CC V5 (in fact I just changed the manifest), but I don't know if "Tasker" can send broadcast events. The receiver name is com.multipie.cclibrary.ConnectionAlarmReceiver. It takes no extras.

Tasker can send 'Intents', if that's what that is, and has a 'Target' field with the option of 'Broadcast Receiver'. It even has a few blanks to specify 'Extras', so it sounds like what you're talking about.

I am not sure if you mean you have to expose it via some method and haven't done it yet, but just in case, I tried it now and it doesn't do anything currently. (Or I'm doing it wrong.)

If you get that working, I'll gladly provide some step-by-step instructions how to make this happen in Tasker, and when people show up and say 'I want to schedule this to run more than once a day' and 'I want to make it run immediately on connection to certain networks' and 'I want to turn wifi on first', etc, you can say 'Get Tasker and you can do it'.

Embedded metadata is updated when calibre sends books via a "device" connection such as CC's wireless device. This is why extracting metadata from books (epubs) sent by calibre has a much better chance of working than books in cloud libraries or that come out of the sky.

I was actually building to a suggestion in that comment, but deleted it. Here we go:

For those of us with some technical know-how who couldn't run CC, and didn't want to deal with USB connections, our obvious solution was was usually a Dropbox 'folder device' that we'd send books too. Either all books, or just some of them. Those got the correct metadata. (And a way I'd be happy to switch back to, if CC would pick up that metadata.)

Likewise, there people, occasionally showing up in this very forum, who use CC, but also plug their device in and send books that way, and are confused as to why those books don't show up in CC despite the fact they've carefully told CC to use the same directory that their Calibre device driver uses. (And then the books *do* show up after they've connecting once via CC!) Those books have the correct metadata also.

So the idea I was going to suggest was: Have Calibre put a specific metadata tag on sending the book to a device, a date saying 'This is the right metadata as of {date}'. A tag that *isn't* allowed on books in the library (As in, Calibre strips it from content.odf during 'Add book'.), and is also unlikely to be found on books in the wild.

And CC then only reads and imports books with that tag. CC knows the book has been through Calibre and exported, so it knows the book has somewhat sane metadata. Since the field is a date, CC can even stop from taking the metadata backwards on a refresh, even if the file time has accidentally changed.

The reason I didn't mention it was that you seem to be going somewhere else, so whatever.

CC extracts what metadata it can from epubs. It then constructs a metadata packet and sends that to calibre. If that packet contains a UUID then calibre will use it to match the book to a book in calibre's library. If there isn't a UUID or if the UUID is not in calibre's library then calibre goes through a process of "book matching", comparing the title and author of the book to books in calibre's library. Matching works if a close-to-exact match is found. This process very often matches nothing or the wrong book. The user must clean up any remaining mess.

Ugh. That's way more work spent on matching books than I would have bothered with, both on CC's end and Calibre's end. I would have just said 'If this doesn't have a Calibre UUID, it ain't a Calibre book and I'm not going to try to match it to one.'.

I actually think this situation is why I started off a bit confused...I find it very strange that someone would try using CC without all books they are trying to manage going through Calibre. That's almost incomprehensible to me that anyone would think that would work at all. (I just wanted my books to take the *long* way around.)

I find it even odder that they'd have the book on their ereader *and* in Calibre but the book on the ereader somehow didn't go through Calibre! Huh?

Are people downloading them on a computer, plugging in their ereader, blindly copying the files over using Explorer, and *also* importing the files into Calibre? And then trying to sync them up later? What spore of madness is this?

[PeterT]

It's a lot easier to follow threads if you make use of the [quote] ... [/quote] tags.

So

Originally Posted by chaley

Actually, I did. See post #3, where I said
There are two variants.

• The first: update metadata in CC's library from the calibre database in the cloud. This makes me nervous because the metadata isn't complete. The values of custom columns built from other columns would be lost. I don't like operations that arguably corrupt the CC library.
• The second: find all books in the cloud library but not in CC's library and add them to CC. This is a more "acceptable" operation because there isn't any data loss; the results would be the same as manually adding the books. It is less acceptable in that the processing required to match all the books in calibre's library to all the books in CC's library would be slow and possibly memory consumptive. The only practical (performance-wise) way I can see to do it is to build two sets of UUIDs, one of the books in CC and the other of the books in calibre, then do a set difference (calibre - cc). The result will be the books that must be added to CC. And BTW: such a process would work for all cloud libraries, not just local libraries.

Your message then looks like

Originally Posted by chaley

• The first: update metadata in CC's library from the calibre database in the cloud. This makes me nervous because the metadata isn't complete. The values of custom columns built from other columns would be lost. I don't like operations that arguably corrupt the CC library.

If you can't get custom columns, I entirely agree that's a bad idea. That would certainly screw me up. Almost all my 'Top, Middle Left, Middle Right, and Bottom' additional lines in the book listing are custom columns.

Originally Posted by chaley

• The second: find all books in the cloud library but not in CC's library and add them to CC. This is a more "acceptable" operation because there isn't any data loss; the results would be the same as manually adding the books. It is less acceptable in that the processing required to match all the books in calibre's library to all the books in CC's library would be slow and possibly memory consumptive.

I think most people are sorta thinking of this as a background operation, so the speed really isn't an issue as long as the rest of the interface can be used, and the new books just show up all at once at the end...and I have no idea how easy that would be.

[chaley]

Some quick comments:

Originally Posted by DavidTC

You surely know the users of CC better than I do. I myself have trouble conceiving of why anyone wouldn't want their entire library on their device. (Except for the sole exception of not enough space.) But from the way you're talking, it seems like *most* people don't put their entire library on their device.

Fully 25% of CC's users run the demo version, limited to 20 books. Of the people running the full version, using a very unreliable sample it seems that at least half have less than 50 books on their device. That means that either their libraries are quite small or they are transferring a subset.

So yes, given what we know today, a large majority of people do not send their entire library to CC. I will know somewhat more once V5 is out, noting that the V5 statistics do not attempt to record the calibre library size.

Auto-connection is triggered by a broadcast. There isn't an activity involved. I don't have a problem with exposing the broadcast receiver in CC V5 (in fact I just changed the manifest), but I don't know if "Tasker" can send broadcast events. The receiver name is com.multipie.cclibrary.ConnectionAlarmReceiver. It takes no extras.

[...]

I am not sure if you mean you have to expose it via some method and haven't done it yet, but just in case, I tried it now and it doesn't do anything currently. (Or I'm doing it wrong.)

The change is in CC's code. You won't see it until you are running a CC V5 later than V5.0.1.6.

So the idea I was going to suggest was: Have Calibre put a specific metadata tag on sending the book to a device, a date saying 'This is the right metadata as of {date}'. A tag that *isn't* allowed on books in the library (As in, Calibre strips it from content.odf during 'Add book'.), and is also unlikely to be found on books in the wild.

The UUID element can serve that purpose.

CC extracts what metadata it can from epubs. It then constructs a metadata packet and sends that to calibre. If that packet contains a UUID then calibre will use it to match the book to a book in calibre's library. If there isn't a UUID or if the UUID is not in calibre's library then calibre goes through a process of "book matching", comparing the title and author of the book to books in calibre's library. Matching works if a close-to-exact match is found. This process very often matches nothing or the wrong book. The user must clean up any remaining mess.

Ugh. That's way more work spent on matching books than I would have bothered with, both on CC's end and Calibre's end. I would have just said 'If this doesn't have a Calibre UUID, it ain't a Calibre book and I'm not going to try to match it to one.'.

I actually think this situation is why I started off a bit confused...I find it very strange that someone would try using CC without all books they are trying to manage going through Calibre. That's almost incomprehensible to me that anyone would think that would work at all. (I just wanted my books to take the *long* way around.)

I am not sure that you have internalized that books in libraries synchronized by a cloud provider are almost certainly not "calibre books". Their internal metadata is not updated. Also note that calibre must deal with devices that present themselves as simple disk drives. The CC wireless device driver goes to some trouble to make itself look like a disk.

I find it even odder that they'd have the book on their ereader *and* in Calibre but the book on the ereader somehow didn't go through Calibre! Huh

Happens with some frequency. The usual path is something like a book store where one is strongly encouraged to download books directly to the device and separately to calibre. Another case arises when people use their reader app to download books. A related case happens when people reset their device or clear app data because of some problem. And finally, and as I said above, books sync'ed to the device directly from the calibre library (no "send to device") usually don't contain (correct) calibre metadata.

It is also worth noting that many (most?) pirated book collections are generated using calibre. Sometimes the books *do* contain calibre metadata, but it is metadata for the pirate's library not the user's library. I can't get on my high horse and tell these putative pirates to pound sand because there is at least one legitimate reason (I am not saying "legal") for grabbing pirated ebooks: fair use, where the user owns a paper copy. In that case getting the ebook is arguably a format shift, not dissimilar to ripping an album on cassette to mp3.

[kaufman]

The only suggestion of David that I would use would be to use the cloud connection to automatically download all new books into CC once a day. I haven't suggested it, because it seemed like a lot of work for a little benefit. As a middle ground, perhaps once a cloud connection has been established, maybe have an option to download everything new.

David: If you can keep your entire Calibre library on your device, you can keep the entire CC library too. Once you have gone through the initial pain, its not that much effort to upgrade it with new books.

[DavidTC]

So yes, given what we know today, a large majority of people do not send their entire library to CC.

Fair enough. I don't understand why, but fair enough.

I am not sure that you have internalized that books in libraries synchronized by a cloud provider are almost certainly not "calibre books". Their internal metadata is not updated.

I do not understand what you mean by 'Their internal metadata is not updated.'. Any book that has been in Calibre has its internal metadata 'updated' at least once, when it was added. (Assuming Calibre supports the metadata, but, if not, all this is moot.) When it was added, at minimum, a UUID was put in, allowing an easy re-match to that book if discovered elsewhere.

That is what I meant when I said 'I would stop there'....if that UUID doesn't exist, the file has never been in Calibre. (A UUID that doesn't match anything also shouldn't be farther matched, especially since a likely setup is that it's from *another* Calibre library the user has.)

Anyway, stepping back from book matching a second, my point is that books can, indeed, have their correct metadata, and it is possible to know when that is.

There are several possible situations non-standard 'cloud' (aka 'books exist in the Default path that CC does not know of') situations:

1) Pointing that path directly at a copied (however) Calibre library. This can't work easily for importing books, because the books have crappy metadata. This is what I started with doing, and I'm slightly worried you keep basing what I'm talking about off it...except that I have no problem being #3 instead. In fact, I like #3 better.

2) Pointing that path at...a folder that has random files that have never been in Calibre. This...is an entirely different thing that I am not making any suggestions for at all. (I personally would just shrug at them, but that's me.)

3) Books sent via the cloud 'Folder device', or via direct device connection, and ending up in CC's Default folder. This always have correct metadata, because they are sent via 'Send to Device'. They're just sent in a *different* way than the wireless device connection.

It is #3 that I was going to make a case for supporting. (With a flag set to indicate they actually were 'Sent to Device' like that.)

The usual path is something like a book store where one is strongly encouraged to download books directly to the device and separately to calibre.

That requires them to have their CC Default folder pointed at the book path their store uses. That, *by itself*, is pretty broken:

What happens if they remove the book from the device using CC? Will the device's 'store' handle that? What happens if CC puts it back, but in the path that CC wants? Will the store find it? And now what if they decide to 'archive' that file in the store (Aka, delete the file off the device)...now CC has files that don't exist and they have to delete books with missing files.

They've basically got two different programs reading and writing to that directory, one of which (CC) *definitely* won't see additions by the other, and needs to be prompted to see deletions...until it does see additions, semi-randomly, because it was connected to a computer. And the other program (Being a proprietary store) is probably the same way, with the added fact it probably has no 'matching' ability at all! That is the ultimate setup for confusion.

That's one of the reasons I said I'd throw my hands up at any files that don't have a Calibre UUID.

And that's just what's going on *on the device*. Running around additionally downloading books on a computer is just adding to the situation.

Another case arises when people use their reader app to download books.

That would just make one copy, which doesn't need to be synced with any Calibre book at all...the user just needs to copy it off the device into their library. My point was more the oddity of having two independent copies of the book. I understand the copy starting on the book.

...oh, wait, what happens if they copy the book up, but they have automatic metadata sync turned off? I guess they'd have to *rematch* the book...again and again...on every connection. No, surely that can't be right.

A related case happens when people reset their device or clear app data because of some problem. And finally, and as I said above, books sync'ed to the device directly from the calibre library (no "send to device") usually don't contain (correct) calibre metadata.

Well, no, but in both those cases the files would contain a Calibre UUIDs in their metadata.

It is also worth noting that many (most?) pirated book collections are generated using calibre. Sometimes the books *do* contain calibre metadata, but it is metadata for the pirate's library not the user's library.

Heh. I literally deleted a section on pirates in my last two posts addressing exactly that.

To start with, this situation is hard to get to. I suspect that pirates use 'Save to Disk' instead of 'Send to Device', and that's why I *didn't* suggest having 'Save to Disk' add that flag, despite that metadata being correct also. Next, the person who *downloaded* the copy must, instead of importing it into Calibre, would have to somehow get it into CC Default folder directly, which is odd behavior.

But, assuming all that, they now have the file imported into CC without it being in Calibre.

And this...isn't a real problem, as far as I can see. Books can already be in CC without being in Calibre. It's called 'deleting them from Calibre' or 'Using multiple libraries'. (And that second even allows entirely different metadata fields! Although you can't actually use them in CC.) It doesn't seem to cause any problems.

If they want to edit the metadata, the solution is, as always, that people to put the book in Calibre! (As the pirated book file already has a Calibre UUID, they can even just copy that into Calibre and it will magically match the file already on their device.)

Again, I'm not saying allowing this import is actually worth supporting. I'm just pointing out it there is some subset of books where you *could* safely read the metadata from them, if Calibre added a metadata tag like that during Send to Device. The pirate situation would be pretty rare and odd and the answer to 'How do I edit this wrong metadata on my pirated books?' is 'Like you always do...in Calibre. If the book is, somehow, not in Calibre, put it there.'.

Meanwhile, *most* pirated books *wouldn't* be importable via that means, and in this hypothetical CC probably popped up something saying "For CC to import books directly, they must leave Calibre via 'Send to Device'. You cannot copy non-Calibre books files directly here, or even use them straight from the Calibre library.". So I'm unsure why people would even *continue* to attempt directly copy downloaded books there on the off-chance it would work.

Actually, frankly, having that flag in 'Send to Device' books (And a different one in 'Save to disk' books) wouldn't be a bad idea even if there were no changes in CC.

I can't get on my high horse and tell these putative pirates to pound sand because there is at least one legitimate reason (I am not saying "legal") for grabbing pirated ebooks: fair use, where the user owns a paper copy.

Sometimes it's even more legit than that, like when the ebook hasn't even come out and people have disabilities that prevent them from reading paper books. This is usually obscure things, but for a very long time Harry Potter wasn't out in ebook.

[DavidTC]

The only suggestion of David that I would use would be to use the cloud connection to automatically download all new books into CC once a day. I haven't suggested it, because it seemed like a lot of work for a little benefit. As a middle ground, perhaps once a cloud connection has been established, maybe have an option to download everything new.

That middle ground already exists, with the Reading List plugin. You tell it to make a list consisting of all books, and that new books on the list should go to the device. Whenever CC connects, all my books download.

That is not the problem.

The problem is getting the connection to happen. I really don't want to run Calibre all the time (Although I've pretty much decided I have to), and the wifi on my device (A Boox Afterglow 2) is a bit wonky (CC can't turn it on, although I don't know if it even tries to do that with a scheduled connection anyway.) and often disconnects when I have no data transfer, despite the fact I told it not to do that.

At this point I have CC on a timer for each evening, I deliberately leave my computer on, and Tasker flips the wifi off ten minutes before the CC scheduled time and then on one minute before it. It...seems to mostly work. If I remember to leave things on, and I'm actually at home. (If I am not at home, I have my ereader with me!)

The reason I am unsatisfied is that...I have a half a dozen things that sync offline, on both my ereader and my Android phone, and *every single one* of them is completely automatic and less work than dealing with CC. All of them just stay synced, period, thanks to timers turning my wifi on and off so they can do that.

And my books *also* used to work this way, where they all got exported, via Folder Device, to a folder in Dropbox that magically, with no effort on my part, got to my Nook STR.

Worst case scenario is that I whisk my reader and phone out of my house before things sync, at which point I either have to find wifi or use some data/tether it to my phone...neither of which I can do with CC.

And, hey, CC supports a cloud connection, right? We can use that to sync, or, better yet, just sync the entire dir offline, so I set that up...oh, now I have to track down stuff that isn't on the device and add it. Also, I've now got two entire copies of the library taking up space on my device, nice. Well, this *almost* works.