CC and the cloud: discussion

[chaley]

I have been thinking for some time about whether and how CC might work with calibre libraries stored in the cloud, such as in Dropbox. As far as I can tell, there are three possible directions:

1. Connect to the library in the cloud via its local app. This means that the calibre db must be downloaded to the device, as well as all the covers. It opens the issue of detecting obsolescence, and will have "interesting" performance characteristics.
2. Add a function to CC that can import a book from a calibre-style folder on the device, which is one that contains formats, metadata.opf, and cover.jpg. The disadvantages are that the folder must somehow get to the device, and that the metadata.opf file in the folder must be up-to-date.
3. Connect to a local calibre library downloaded from the cloud to the device. This has the advantage of performance, because everything is local. It solves the obsolescence problem, at least if the underlying cloud sync works. It has the disadvantage of space; all the book formats are stored locally.

Calibre Cloud free does #1. Calibre Cloud Pro does #1 and #3, as does Leger Calibre and perhaps others. No one does #2 as far as I know.

My problems with #1 are (in no particular order):

• By necessity it would need to be a part of CC, because downloaded books need to be managed locally.
• It would be slow, certainly no faster than Calibre Cloud.
• It would require us understanding the API of several cloud storage providers.
• It would be difficult to monetize. It is a lot of work both to build and maintain, and we need to get something out of such a significant development effort.

Number 3 doesn't have any of the above problems. There is no need at all for it to be integrated into the current CC, permitting it to be a separate app (lets call it CClocal), chargeable separately. CCLocal's performance would be similar to CC, in some cases faster and in some cases slower. There is no need to understand any API, because the user would choose the synchronization method. CCLocal should have the same functionality as the current CC, with the exception of the "Connect" operations. The grouping drawer, grid + list views, book details, settings, etc would work as now, with the exception of "columns built from other columns" (see below). One exception to functional equivalence might be the read/date read information. It isn't clear how to sync this information with calibre. We cannot change the DB because that would certainly create conflicted copies.

Number 2 has the advantage of simplicity from CC's point of view. A folder appears on the device by magic and CC imports that folder. The downside is that there is no assistance to find the folder to download. The user must find that folder manually then download it.

Both #1 and #3 will suffer from not being able to use values in "columns built from other columns". The value is not stored in calibre's database and there is no way that CC or CCLocal will implement calibre's template language processor. They could in theory get the values from the metadata.opf files in the book folders, but doing so would be glacially slow.

An argument against doing CCLocal is that it is difficult to see any significant advantage offered by CCLocal over simply sending one's entire library to the current CC. And of course, sending one's entire library obviates the need to do any of #1, #2, and #3.

It seems to me that #3 is superior to #1. The drawbacks are the space required to hold a copy of calibre's library on the device and the possibility of losing the ability to set is_read info, and I am not convinced that these are enough of a problem to force #1 over #3.

I wonder about #2. It would be rather straightforward to build into CC, removing the need to monetize it. It would be part of CC, thus supporting is_read and the like. Calibre tries hard to keep the metadata.opf files up-to-date. The values for columns built from other columns are available in the metadata.opf file. Of course the major problem, and potentially a killer problem, is the requirement to find a book by navigating the calibre library folder structure and downloading the desired folder.

Comments? When commenting, please assume you get to choose only one option. For me, "Do them all" is equivalent to "Do none of the above".

[eschwartz]

I admire the simplicity of #2.

Perhaps some simple search function to match {author/title (***)} as a folder search?

It would support both syncing the entire library to device via dropbox/whatever, and saving to disk then exporting that somehow -- USB or dropboxing that for transfer -- serving as a way to add books even when wirelessly connecting is not an option for whatever reason.

Much superior to #3 I think.

Perhaps it can also integrate into cloud storage apps to search for and download those folders, which would steal the only advantage that #1 has too.

I am NOT saying import the database and stuff -- just search à la #2 in cloud apps too. Obsolescence can stuff itself, since we are treating it just like an imported save-to-disk. Covers can be downloaded if you choose the right folder -- but you'd do that anyway, together with the books itself.

I don't know about the difficulties of supporting cloud APIs, although the idea does come to mind that you do not necessarily need to support every provider out there. I imagine just dropbox would be enough, really.

Then again, I can just see people saying "you support Dropbox (and even every other popular provider out there), why don't you support provider $weird also!!!" Maybe it's best not to support any no matter what.

[theducks]

Call it CCC (Calibre Cloud Client) or C cubed

[cybmole]

my single vote is for option 1 though I get that it's a lot of work for no reward and that the other app already does it for free or for cheap.

[chaley]

Thanks for the comments. They help.

@eschwartz: your suggestion to use #2 and "Perhaps it can also integrate into cloud storage apps to search for and download those folders" opens a possibility I hadn't considered. One could build #2 into CC and offer it "included in the price". Those who know what to do can use it as it is. In addition, we could build a "search app" that knows how to use some cloud APIs to download the calibre database file and do searches on it in a way similar to CC's grouping drawer or calibre's tag browser. When the desired book is found, the app would download its folder to a place where CC would automatically see it and import it. This second app would be "pay for", dealing with the monetization problem of #1. We could even consider a "free version" of this app that searches only on title.

[cybmole]

the dropbox android app does not support folder download. it's 1 file at a time only.
and it is not officially available for Kindles -
but if you can work around those things, go for it !

[chaley]

Originally Posted by cybmole

the dropbox android app does not support folder download. it's 1 file at a time only.
and it is not officially available for Kindles -
but if you can work around those things, go for it !

The free app Folder Downloader for Dropbox downloads individual folders. It works well.

[cybmole]

Originally Posted by chaley

The free app Folder Downloader for Dropbox downloads individual folders. It works well.

if you are willing to grant full read AND WRITE access to your dropbox, to a realatively obscure app, then yes

I am quite happy for a book related app to have read access to all books in my dropbox account - but not to the whole account, and certainly not write access.

[chaley]

Originally Posted by cybmole

if you are willing to grant full read AND WRITE access to your dropbox, to a realatively obscure app, then yes

I am quite happy for a book related app to have read access to all books in my dropbox account - but not to the whole account, and certainly not write access.

I just checked my dropbox account. *Every app* has full access, including Moon+, calibre cloud, dropsync, and folder downloader. I am not sure that it is possible for an app to get restricted access. So you already have given read/write access to relatively obscure apps.

[cybmole]

there may be 2 sets of permissions, one in android one in dropbox,

when I was trying out different reader apps, I am sure I saw in one of them that it was restricted to epub and txt files, and maybe also limited to calibre folder.... checking.... it may have been mantano lite which I have since uninstalled.

you are right that C Cloud has full RW access & you say you've checked moon reader... So maybe it was only for Mantano that i saw it, but as I no longer have that or moon linked I cannot easily check. It could be that there's a a screen of info that you see once only , when you give the app permission to link

update: it was Mantano - whereas Moon says that it want full permissions ( on the link to dropbox screen ) Mantano says:
would like access to ebooks and txt fiiles in your dropbox I took "access " to mean permission to read, but I accept that it is ambiguous

[eschwartz]

Originally Posted by chaley

Thanks for the comments. They help.

@eschwartz: your suggestion to use #2 and "Perhaps it can also integrate into cloud storage apps to search for and download those folders" opens a possibility I hadn't considered. One could build #2 into CC and offer it "included in the price". Those who know what to do can use it as it is. In addition, we could build a "search app" that knows how to use some cloud APIs to download the calibre database file and do searches on it in a way similar to CC's grouping drawer or calibre's tag browser. When the desired book is found, the app would download its folder to a place where CC would automatically see it and import it. This second app would be "pay for", dealing with the monetization problem of #1. We could even consider a "free version" of this app that searches only on title.

I am more than happy to offer them.

I don't actually use or have Calibre Companion, because I do all my reading on my E-ink Kindle, but I still love your app just because it sounds awesome and enhances calibre and ereading in such an awesome way. Keep up the good work!

[cybmole]

Originally Posted by chaley

I just checked my dropbox account. *Every app* has full access, including Moon+, calibre cloud, dropsync, and folder downloader. I am not sure that it is possible for an app to get restricted access. So you already have given read/write access to relatively obscure apps.

additional response on that.

the way that C cloud works makes it unable to see any epub outside of the calibre library folder. It uses metadata.db as i'ts index of what it available. ( as I'd expect from an app with calibre in the name).So although it has wider permissions in theory, it cannot actually see any book or text file placed in dropbox root folder or placed elsewhere in dropbox; and I don't think it has any upload facility, or even any delete facility it is just a dumb db browser.

but the other epub reader apps present a file/ folder explorer, via a dropbox tab, and they will open any file from anywhere in dropbox, if it is in a format they understand.
I assume the do something similar for google drive, but as amazon ban me from accessing that via my Kindle tablet I'm only guessing
This again is reasonable - they are advertising access to dropbox / google drive not access to calibre backups!
But what they ( probably ) lack - as I've not spent much time with then, - is the ability to preview all of the calibre metadata, like your app and the C cloud app can do.

Of course, if you were a crazed capitalist megalomaniac, and not a normal nice guy, (albeit one who uses the word "monetize" ) you could make a hostile takeover bid for C cloud & bundle that into a super "triple C " branded app ( icons of cowboys, rodeos, horses optional)

[chaley]

Originally Posted by cybmole

the way that C cloud works makes it unable to see any epub outside of the calibre library folder. is uses metadata.db as i'ts index of what it available. ( as Id expect from an app with calibre in the name).So although it has wider permissions in theory, it cannot see any book or text file placed in drop box root folder or placed elsewhere in dropbox. and I don't think it has any upload facility, or even any delete facility it is just a dumb db browser.

This may be what it does, but it isn't what it has permissions to do. Clearly it can wander around in your dropbox because it does when you tell it where the library is. There is nothing stopping it from snooping wherever it wants.

To be sure, I uninstalled and reinstalled it, and when it linked to dropbox the Android dropbox app told me it has full permissions on my entire dropbox. These are the same permissions asked for by Folder Copy and Moon+. Images of the Dropbox permissions screens are attached.

[cybmole]

accepted.

so that case for havng just one app that needs to be be trusted with accessing your cloud storage books becomes stronger....

I unlinked my other apps because I got twitchy about the various aide-memoirs I keep in itsy bitsy dropbox text files - for when i forget how to access stuff....

[chaley]

Originally Posted by cybmole

I unlinked my other apps because I got twitchy about the various aide-memoirs I keep in itsy bitsy dropbox text files - for when i forget how to access stuff....

I keep this sort of stuff in a program that encrypts the information, PassKeep. The password I use for the keeper a good one that I can remember and is used nowhere else. I currently have 135 different passwords in the program, most of them auto-generated using random letters, numbers, and if permitted, punctuation.