CC and the cloud: discussion

[chaley]

Originally Posted by kaufman

This is somewhat more work for you, but perhaps you can do it as a separate app that completely is controlled via IPC from CC. If a person tried to run it standalone it would just popup a message saying that it can't be run that way.

Another advantage of this is that you could leverage some of the searching/sorting capabilities of CC to add more functionality at little cost. So, for example, if you are grouping by a specific author in CC you could have a menu item to show all the other books by that author that are stored in the cloud (or even better all the books by that author that are not already part of the CC database). Executing that function would send a message (or however android handles IPC) to the the new app to actually get the list of matching books, which could then be shown in CC. And so on....

Actually, this is more work than option 1, adding cloud access directly to CC. First, the slave app would need to understand completely calibre's database so it could answer questions about anything CC can group on, an amount of work more-or-less equivalent to building a new DB layer into CC. In addition, CC would need to implement the client side of some protocol to ask the questions in the way you describe, and in addition implement the user interface to be able to formulate questions about books that share no metadata with the books in the current library (browsing the cloud).

My feeling is that you are objecting to the a second app (second UI etc), and are really arguing for option 1, implemented differently to deal with our objections to IAP. I can understand this position. Clearly the combination of CC's import and the second app is not as elegant as CC itself directly accessing the cloud. On the other hand, the combination offers more functionality because it is not tied to any N (for whatever N) cloud storage companies. On the third hand, the combination is more difficult to explain. and so on.

[kaufman]

Originally Posted by chaley

My feeling is that you are objecting to the a second app (second UI etc), and are really arguing for option 1, implemented differently to deal with our objections to IAP. I can understand this position. Clearly the combination of CC's import and the second app is not as elegant as CC itself directly accessing the cloud. On the other hand, the combination offers more functionality because it is not tied to any N (for whatever N) cloud storage companies. On the third hand, the combination is more difficult to explain. and so on.

My concern about a second app is that its going to be harder to convince people to use two apps for functionality that is pretty tightly intertwined with their use of CC. The more functionality you give the app, the more you will be doing things that are kind of CC actions.

As far as funtionality goes, at one end you have a file manager app that does nothing but browse the file folders in the cloud and allow users to download them. This replicates nothing in CC. At the other end you have a full metadata reader/browser/searcher that does everything CC except using a calibre metadata database and a cloud folder for its books.

Some users like me keep their entire databases on their phones. Our use of the new app will be for the occasional times we want to grab a few new books that we not have synced after adding them to the library. We will be satisfied with a minimal app that perhaps just allows people to see which books are new (or perhaps missing).

At the other end of the spectrum, some users may keep only a few books on their phone. They will want a full searching/sorting solution that helps them pick new books to download.

If all your users were like me, you could do the simplest app and move on. If your users are all like the second type, then a simple app isn't going to cut it, and you are going to have to do more if you want a usable solution.

I don't know what the bell curve of your users looks like, so I am just trying to come up with ideas that might start small and be extensible.

I also figured an app that looked like it was part of CC would initially raise less expectations among users than a full new app. A standalone app is going to have to be fairly full featured right from the start or people whose needs are more than mine may just look at it once and never come back.

[Dopedangel]

I would like calibre companion to have an in app purchase that adds the ability to compare the metadata from calibre database in my dropbox and then download the missing books and update the metadata in cc for books that have changes. It should also have the ability to set the formats to download so only epubs or the format of choice are downloaded.
The problem with this approach is that calibre companion will have to download the whole metadata.db every time it is updated unless dropbox or the cloud service of choice supports incremental changes where only the changes in the file are downloaded.

[chaley]

Originally Posted by Dopedangel

I would like calibre companion to have an in app purchase that adds the ability to compare the metadata from calibre database in my dropbox and then download the missing books and update the metadata in cc for books that have changes.

In effect, you want the sync functionality of connecting as a wireless device, plus a library compare. The first is interesting and in theory is possible, but it requires I go with option 1 and it would not work with up-syncing is-read information. The second makes me rather nervous. I need to think about it.

It seems to me that option 3, the entire calibre library duplicated onto the device with CC able to read/use the calibre database, is a better fit for what you want. There would be no "sync" or missing books because CC would be directly using the complete library.

It should also have the ability to set the formats to download so only epubs or the format of choice are downloaded.

You have this today. Use CC's folder settings to allow only epub.

The problem with this approach is that calibre companion will have to download the whole metadata.db every time it is updated unless dropbox or the cloud service of choice supports incremental changes where only the changes in the file are downloaded.

Yes, you are right. We have no control over how the cloud service does downloads. In addition to metadata.db, the other bit of option 1, browsing, requires that cover files be downloaded.

The encouraging side is that calibre cloud must do the same thing and, for me at least, the performance is acceptable. Not good by any means, but acceptable. I hesitate to generalize because I have a gigabit internet connection, much faster than the norm.

[cybmole]

metadata.db is not a large file.
the app could probably stash it and re-download it only if it's date/time stamp has changed since last time.
some inspection of tablet storage will reveal if that is what c cloud already does. I find that app to be pretty quick at getting stuff from dropbox. I am 99% sure it keeps a cache of all cover thumbnails.

the problem with adding all of calibre library to tablet is one of overall size.
my modest collection is only 2.5 Gb but I expect the real power users have much larger libraries, and won't want to commit most of a 16GB tablet to storing a copy.

& its gets silly to have one or more copies on local PC hard drives, plus one in a cloud, plus yet another one on each tablet.

[kaufman]

Originally Posted by chaley

It seems to me that option 3, the entire calibre library duplicated onto the device with CC able to read/use the calibre database, is a better fit for what you want. There would be no "sync" or missing books because CC would be directly using the complete library.

This would certainly be a workable solution for me. Perhaps a first step towards this would be to have CC change the way it stores the books to be the way Calibre does it on the computer (this would include the directory structure as well as the opf and cover files). Then the next step might be to scan for new files when requested and add them to your internal database. A last step might be to read the metadata.db file directory, But I am not convinced you ever need to do that.

You could then (if you wanted to) write an external program to grab the files from the cloud, but I am not convinced you would need to since programs like dropsync could do that for you. Some people would work that way, and others would just continue to do the wireless syncs.

[chaley]

Originally Posted by cybmole

metadata.db is not a large file.

Mine for a 2300 book library is 7 megabytes. That is a significant download. Smaller than a book to be sure, but not free.

the app could probably stash it and re-download it only if it's date/time stamp has changed since last time.
some inspection of tablet storage will reveal if that is what c cloud already does. I find that app to be pretty quick at getting stuff from dropbox. I am 99% sure it keeps a cache of all cover thumbnails.

The cloud APIs all contain the ability to say "get me this if what you have is newer". The more interesting question is whether the app will do incremental downloads.

As for covers, yes, calibre cloud does keep a cache. It has a command to invalidate the cache. However, unless it ignores changes in the cloud it will be keeping covers, not thumbnails. For most people these are substantially larger than thumbs.

the problem with adding all of calibre library to tablet is one of overall size.
my modest collection is only 2.5 Gb but I expect the real power users have much larger libraries, and won't want to commit most of a 16GB tablet to storing a copy.

Granted. However, both DopedAngel and Kaufman have asked for auto-transfer of books in the cloud but not on the device, which implies that they do want all the books on the device. In this case, cloud sync is a distraction.

& its gets silly to have one or more copies on local PC hard drives, plus one in a cloud, plus yet another one on each tablet.

This I disagree with. One cannot have too many backups.

[chaley]

Originally Posted by kaufman

This would certainly be a workable solution for me. Perhaps a first step towards this would be to have CC change the way it stores the books to be the way Calibre does it on the computer (this would include the directory structure as well as the opf and cover files). Then the next step might be to scan for new files when requested and add them to your internal database. A last step might be to read the metadata.db file directory, But I am not convinced you ever need to do that.

Why change how CC stores data when syncing as a wireless device?

Almost all of your arguments have strongly implied (I think "required") that CC be able to read and process calibre's metadata.db. For CC that means a different DB layer that understands metadata.db instead of the current db. If it can do that then it is a very small step to using the rest of the library folder structure. After all, the paths to the books are in the db in both scenarios.

It is worth saying that replacing CC's data layer is an enormous amount of work. That is why option 1 is not attractive to me unless I can find a way to charge for it, etc etc as has been said many times in this discussion. Having the complete calibre library locally on the device would have to be a chargeable option, as would embedded cloud sync.

Of course, having IAB to charge for the above opens the door to having all four sync types be separately chargeable (wireless device, content server, local calibre library, cloud sync), but I think down that road lies perdition. We certainly can't take functionality away from current purchasers and we can't "fork" CC to have two apps. We can't use in-app flags because they are so easy to pirate, unless we just throw our hands up and admit that the slime buckets will defeat anything we do so it doesn't matter.

Another thing to deal with: having IAB for connection options reopens the free/pro question that we (hope we) just closed with the demo version of CC.

You could then (if you wanted to) write an external program to grab the files from the cloud, but I am not convinced you would need to since programs like dropsync could do that for you. Some people would work that way, and others would just continue to do the wireless syncs.

Assuming that CC (or a variant of CC) could process calibre's db, then Dropsync would do everything needed. All that would be required is to have a consistent copy of calibre's library on the device.

[kaufman]

Originally Posted by chaley

Almost all of your arguments have strongly implied (I think "required") that CC be able to read and process calibre's metadata.db.

I wasn't trying to imply that. I'm all about just adding the books in a way so that they get in the CC database in some usable way and that it is trivial to fix the missing metadata on the next wireless connect. That's why I started with just the epub and then moved to the epub plus opf and so on.

Originally Posted by chaley

All that would be required is to have a consistent copy of calibre's library on the device.

What I was trying with my last post was to find a way to avoid having a second copy of all of the books on the device. I don't mind the overheard of the opf/covers which adds about 20% to the size on the device. What I don't want to do is to have to have two copies of all of the epubs which would add a lot more space.

So, if your storage format was the same as Calibre's, then you could store the two files together and the only thing you would have to process would be the opf's of new books.


For people like me who keep their entire libraries on the device, there are really only two use cases for all this stuff:

1) I have added a new book to my library, and I forgot or wasn't able to sync it to my device before I moved away from my computer. This has happened to me occasionally, but since I always have my phone with me, it doesn't happen very often.

2) I get a book from some location while on the road (email, web, etc) and I want to add to to CC, in preparation to moving it from CC to Calibre. This is somewhat more common, but given your mistrust of the metadata in bare ePubs, is probably not going to addressed. Which is not that big a deal, because it doesn't happen that often, and because I can still read the books on the device, I just can't put them in CC.

There is a sort of third use case of

3) I want my books to automagically show up in CC after I add them to Calibre, but I think that that is a huge amount of work for you, and probably not worth your trouble.

For people who don't keep their all their books on their devices, there is also the usecase of:

4) I am disconnected, and I want help in getting a new book into CC. But the problem with what you are proposing is that people who don't keep all their books on their device are the last group who is going to want to keep a copy of their Calibre library on their device.

[chaley]

Originally Posted by kaufman

I wasn't trying to imply that. I'm all about just adding the books in a way so that they get in the CC database in some usable way and that it is trivial to fix the missing metadata on the next wireless connect. That's why I started with just the epub and then moved to the epub plus opf and so on.

Browsing the cloud from inside CC implies understanding calibre's db. Of course, one could offer very limited browsing capability which would reduce the amount of the db that CC needs to be understand, but a) that would invite very negative comparisons with the current content server and CC functionality, and b) once NxM tables like tags and authors are "understood", the remaining tables are easy.

What I was trying with my last post was to find a way to avoid having a second copy of all of the books on the device. I don't mind the overheard of the opf/covers which adds about 20% to the size on the device. What I don't want to do is to have to have two copies of all of the epubs which would add a lot more space.

The "entire calibre library" option would not require two copies of anything. There would only be the calibre library. There wouldn't be a CC library.

So, if your storage format was the same as Calibre's, then you could store the two files together and the only thing you would have to process would be the opf's of new books.

Unfortunately CC does not control the file paths for the books, calibre does. In addition, for several reasons it is very difficult to come up with a template that duplicates calibre's folder naming algorithms (many people have tried and failed, including me). Thus, to duplicate calibre's structure I would have to do fairly significant damage to calibre's device driver system to import calibre's internal names, something I am not willing to do.

For people like me who keep their entire libraries on the device, there are really only two use cases for all this stuff:

1) I have added a new book to my library, and I forgot or wasn't able to sync it to my device before I moved away from my computer. This has happened to me occasionally, but since I always have my phone with me, it doesn't happen very often.

2) I get a book from some location while on the road (email, web, etc) and I want to add to to CC, in preparation to moving it from CC to Calibre. This is somewhat more common, but given your mistrust of the metadata in bare ePubs, is probably not going to addressed. Which is not that big a deal, because it doesn't happen that often, and because I can still read the books on the device, I just can't put them in CC.

There is a sort of third use case of

3) I want my books to automagically show up in CC after I add them to Calibre, but I think that that is a huge amount of work for you, and probably not worth your trouble.

This third use case is what the "entire library on the device" would do, assuming that the library is synced through the cloud.

For people who don't keep their all their books on their devices, there is also the use case of:

4) I am disconnected, and I want help in getting a new book into CC. But the problem with what you are proposing is that people who don't keep all their books on their device are the last group who is going to want to keep a copy of their Calibre library on their device.

I don't follow. First, I assume that the new book is in calibre. There are already two ways to get the new book, content server and wireless device, and the "cloud sync" (the point of this thread) would be a third. The "entire library" is a fourth option.

[kaufman]

Got it now. I misunderstood a basic part of what you were proposing. Now I need to go back and reread 100 threads to see what I missed.

[cybmole]

Originally Posted by kaufman

This would certainly be a workable solution for me. Perhaps a first step towards this would be to have CC change the way it stores the books to be the way Calibre does it on the computer ....

sorry but I think that is a really BAD idea. 3rd party reader apps are happy to have a single defined directory for their auto-import function, but a calibre style set of nested folders could be problematic,

I much prefer pointing C companion, C cloud, Mantano, moon reader... all at the same single tablet directory.

KISS

& let's not even get into the migration issue for folks who'd have all their books suddenly in the "wrong" place on their phones/tablets, when the y installed the upgrade

I'm just wondering when someone will jump to where this thread seems to be going & say , lets just write a calibre for android & be done with it

[kaufman]

Originally Posted by cybmole

sorry but I think that is a really BAD idea. 3rd party reader apps are happy to have a single defined directory for their auto-import function, but a calibre style set of nested folders could be problematic,

I much prefer pointing C companion, C cloud, Mantano, moon reader... all at the same single tablet directory.

No need to be sorry, but I have to disagree with your idea of having three unrelated programs store their data in the same directory. What happens if one of the programs starts storing unrelated data that confuses the other two? What happens if there are filename collisions? What happens if you want to store your books on the SD card and only one of the three programs can write to the file? But most of all, I keep about 5 thousand books in my CC library. The only ones I want showing up in MoonReader are the ones I have specifically asked to be there.

Having the same file structure between CC and Calibre makes more sense because they share a lot of the same developers, and because it simplifies a lot of the cloud sync effort. It may not be a good idea for other reasons, but "I want to use the current folder for a bunch of other programs" is probably not the biggest among them.

Originally Posted by cybmole

& let's not even get into the migration issue for folks who'd have all their books suddenly in the "wrong" place on their phones/tablets, when the y installed the upgrade

I bet if you polled the typical CC user and asked them, they would have no idea if their books were stored in one directory or a whole bunch of subdirectories. And if they were moved all around, they would have no idea about that either. Why would they care?

Originally Posted by cybmole

I'm just wondering when someone will jump to where this thread seems to be going & say , lets just write a calibre for android & be done with it

I think that's been brought up and rejected by Chaley.

Having said that, if someone came out with a complete and perfect implementation of Calibre for Android that perfectly synced across the cloud, I would not switch from CC to it. Calibre is a great tool for editing and maintaining a book library. But I don't need to do that on my phone. CC is a far better tool for browsing an existing database on a phone or tablet.

[chaley]

This is a good time to summarize what I think I am hearing.

The options:

1. Connect to the library in the cloud via its local app. This means that the calibre db must be downloaded to the device, as well as all the covers. The implication is that CC must be able to browse the DB in some way. Two obvious choices: 1) present the information as if it was a content server, or 2) present the information as if it were a CC library. Both require CC to have a deep understanding of calibre's database.
   
   The arguments for this option center around usability. There is one app. There isn't any "weird" folder. If done correctly, the user interface would match an existing user interface.
   
   One person suggested an extension to this option where CC could update metadata from the calibre db in the cloud instead of via a wireless device connection.
   
   The argument against is that this option is the most difficult to "monetize", in the sense that it is a lot of work and CC contains no in-app purchasing ability.
2. Add a function to CC that can import a book from a calibre-style folder on the device, which is one that contains formats, metadata.opf, and cover.jpg. The principle advantage is that from CC's point of view, there is no cloud service involved. The folder appears and can be imported. The disadvantages are that the folder must somehow get to the device, and that the metadata.opf file in the folder must be up-to-date.
   
   This option was modified to include a second app that could browse (somehow) the calibre db in the cloud, download calibre folders, and ask CC to import them. The app would be pay-for, obviating the problem of adding in-app payments to CC.
   
   The arguments for include ease of implementation, not being tied to a fixed set of cloud services, and straightforward "monetization". The arguments against include a more difficult user experience and potential unhappiness if the second app isn't as capable as the user wants.
3. Connect to a local calibre library downloaded from the cloud to the device. This has the advantage of performance, because everything is local. It appears that this option would be popular only with people who keep their entire library on the device, and therefore should not be considered as an alternative to #1 or #2. This option would need to be pay-for, somehow.
   
   The arguments for include ease of use for people who want their entire library on the device. The arguments against are that it is of no use at all to someone who wants a subset of their library on the device, and (yet again) monetization.

Several people have expressed preference for #1, others for #2. What isn't clear to me is whether the people expressing preference for #2 really prefer it or were indicating that they could accept it under some set of conditions.

I understand that getting compensated for our work is not the end user's problem -- not *your* problem. That is as it should be. It is up to us to choose whether or not to offer functionality, and if offered how. I further understand that it is sometimes better to do nothing than to do something half assed. This understanding argues for doing nothing or doing option 1, based on user experience.

Note that no option would permit modifying the calibre database. There would be no way to change metadata or to add/delete books or formats. This is a downside of option #3 as described above; the is_read syncing would go away.

[Purple Lady]

As far as compensation goes, can't you sell a CC Pro license instead of using an in-app purchase? This is how I got the Titanium backup pro and Supersu pro versions.