CC Cloud

[chaley]

I have finally started work on adding a "cloud connection" to CC. In the process of prototyping such a thing I have run into some issues that raise questions. I am posting this in hopes of getting some information that will guide me as I "finish" this feature.

Some basics:

• I decided not to try to monitize it. The cloud connection will arrive in CC for anyone and everyone.
• The connection will appear in the same way as the existing content server. The first page will show you more-or-less what the top content server page shows. and so on.
• "Downloading" a book will add it to CC's library.
• I assume that providing more cloud providers than Dropbox would be a good thing.

Now to the questions:

• The cloud APIs are a mess. Example: if I choose to use the Dropbox API then I am more-or-less stuck with it, The impact: developing a more generic API will require the user to authenticate using the web browser and not the local app, meaning you will be required from time-to-time to sign in to your provider on a web page. "Time-to-time" might be as often as every connection. Is this OK? Note that if the answer is "no", the result will be that CC will support an extremely limited number of cloud providers, specifically those who provide a software library. Currently that is Dropbox, perhaps Box, and even less perhaps Onedrive. Copy.com is not on the list. I am refusing to support Google Drive because it can destroy a calibre library.
• The cloud providers are *extremely slow". For example, it can take 5 to 10 seconds to download a cover from Dropbox. Should we even try to cache covers when browsing categories? Is consuming the space to store the cached covers worth it?
• There seems to be no way to use the account information stored on the device. This means that when you connect to the cloud you might be required to log in. Dropbox does not suffer from this problem because it gives back a token that never expires. The other providers provide login tokens that expire after a time. Is the benefit of allowing other providers worth the user hassle of having to re-login on (I think) each connection? Note that I cannot ask for and store credentials (usernames and passwords) because the underlying programming interfaces do not provide a way for an app to log itself in.
• Searching is extraordinarily slow, 3 to 5 books per second. Is searching really required, given that you will be able to navigate the item hierarchy just as you can in the current content server interface?
• Would it be useful for this connector to access a complete calibre library downloaded to the device? CC would copy desired books from this library copy into its database. It would be the responsibility of the cloud provider/program to keep the calibre library copy up-to-date. It seems to me to be a bit strange that the entire library is copied to the device but not to CC.

On the good side, once the calibre metadata.db is downloaded the interface is very fast (ignoring covers). That raises a question: would it be useful for CC to accept manually-downloaded databases from "unusual" cloud providers? Going this route would require the user manually to download a copy of calibre's db, then later to manually download the desired books after CC gives you the file path information. Cover thumbnails would not appear except (perhaps) in book details and then only after the user intervened and downloaded the cover jpg. My feeling is that this is too much of a pain in the ^% for users to justify the development.

EDIT: some updates:

• The cover slowness problem has been resolved. It now takes around 1.5 seconds per cover. If more than one cover is needed they are fetched in parallel. If one cover takes 1 second, 10 covers takes around 2 seconds.
• I have now using the base-level APIs so supporting multiple cloud providers is possible. I have working implementations for Dropbox and Onedrive.
• For the two implemented providers, authentication is a one-time thing. It happens the first time you connect to a given provider.

[cybmole]

Calibre cloud caches all covers, as small thumbnails I think. Browsing those seems instant, once they are cached. I can check how much android storage is used....146mb on my kindle for a library of say 1500 titles
I vote for dropbox only, no browser sign in.

[Purple Lady]

I use Titanium Backup with Box on some devices and Dropbox on others. Titanium signs me in to each with no interaction from me. Dropbox is listed as an account in Android settings and I think that's how access was requested, but Box isn't an account there. I may have had to sign in to Box once but I haven't had to since.

[cybmole]

PS the initial dropbox authentication may cause you extra hassle in Kindle devices. I had much trouble getting mantano on KIndle Fire to access dropbox folders, but no trouble at all on google tablet. I did solve it eventually but did not write down the answer- from memory though it need to do some one-off token thingie via browser and the default silk browser fails to get and then hand the token to Mantano I had to do somethign like copy the failing URL and post into chrome browser on tablet... just a heads up

[Purple Lady]

Originally Posted by cybmole

PS the initial dropbox authentication may cause you extra hassle in Kindle devices. I had much trouble getting mantano on KIndle Fire to access dropbox folders, but no trouble at all on google tablet. I did solve it eventually but did not write down the answer- from memory though it need to do some one-off token thingie via browser and the default silk browser fails to get and then hand the token to Mantano I had to do somethign like copy the failing URL and post into chrome browser on tablet... just a heads up

My Kindle is rooted so it may be different, but on all my devices I installed the Dropbox app and logged in before doing anything else. When I needed to add Dropbox functionality to another app it just asks me to confirm allowing the app to have access to Dropbox.

[DoctorOhh]

I use Dropbox and I just installed calibre cloud and searching is very quick and as soon as I go to an author the covers are shown very quickly. These are not cached as it was not on this device until just now and all of the dates for everything is current. Additionally it is only showing 961 items so I know it is grabbing the covers out of the cloud. When I fling it to a certain area it fills the 15 covers showing from Dropbox in about 3 seconds.

I only give you this info because the cover grabs from the cloud seem to be quicker then your estimates.

That said, I would love a cloud option. It would allow me to dump the majority of my library from my device and still be able to grab any book quickly without my home PC being on.

[cybmole]

Originally Posted by Purple Lady

My Kindle is rooted so it may be different, but on all my devices I installed the Dropbox app and logged in before doing anything else. When I needed to add Dropbox functionality to another app it just asks me to confirm allowing the app to have access to Dropbox.

that is what i saw. My KF is not rooted but I had dropbox sideloaded.
Mantano app asks for one-off permission to access dropbox and pops up a screen to input dropbox credentials. it must then hand those off to silk browser for some one off process and silk pops ups with a gobbledygook URL problem.

copy/paste of that or temp change of default browser works around the issue, but I suspect that the way that amazon cache all silk browser traffic gets in the way of the token processing.
It could be that silk cannot directly access dropbox but I worked around the problem, rather than spend more time on it.
( I didn't really need mantano to dropbox link anyway, I was only testing it out of curiosity! ; I would use CC to dropbox though hence the heads up for specific Kindle device testing )

[Purple Lady]

Originally Posted by cybmole

that is what i saw. My KF is not rooted but I had dropbox sideloaded.
Mantano app asks for one-off permission to access dropbox and pops up a screen to input dropbox credentials. it must then hand those off to silk browser for some one off process and silk pops ups with a gobbledygook URL problem.

copy/paste of that or temp change of default browser works around the issue, but I suspect that the way that amazon cache all silk browser traffic gets in the way of the token processing.
It could be that silk cannot directly access dropbox but I worked around the problem, rather than spend more time on it.
( I didn't really need mantano to dropbox link anyway, I was only testing it out of curiosity! ; I would use CC to dropbox though hence the heads up for specific Kindle device testing )

For me I didn't have to input anything. It just asked for permission and I said yes. I'm assuming it got the details from the Dropbox account data that Android stores.

[cybmole]

Originally Posted by DoctorOhh

I use Dropbox and I just installed calibre cloud and searching is very quick and as soon as I go to an author the covers are shown very quickly. These are not cached as it was not on this device until just now and all of the dates for everything is current..

i think you will find it caches as it goes, i.e. as you scroll through the library it is generating thumbnails ans storing them, as the app setting have an option to clear cache

[DoctorOhh]

Originally Posted by cybmole

i think you will find it caches as it goes, i.e. as you scroll through the library it is generating thumbnails ans storing them, as the app setting have an option to clear cache

Without a doubt that is how it works. Since it was a first time install the covers were all pulled from the cloud and then cached for future use.

I never used it because it shows my entire library, including the 3000+ books I have without proper covers, metadata or editing. Calibre itself allows me to start in a default virtual library where all of my finished books reside. Calibre Companion easily synced this library to my device.

[cybmole]

Originally Posted by DoctorOhh

I never used it because it shows my entire library, including the 3000+ books I have without proper covers, metadata or editing. ...

i use it rarely, and only to retrieve a book without getting out of bed, if the calibre library PC is off.
it also lacks some other useful search faciity - could be series, could be tags, I forget which... I tried contacting the author re that but no response.

[chaley]

Originally Posted by DoctorOhh

I use Dropbox and I just installed calibre cloud and searching is very quick and as soon as I go to an author the covers are shown very quickly. These are not cached as it was not on this device until just now and all of the dates for everything is current. Additionally it is only showing 961 items so I know it is grabbing the covers out of the cloud. When I fling it to a certain area it fills the 15 covers showing from Dropbox in about 3 seconds.

I only give you this info because the cover grabs from the cloud seem to be quicker then your estimates.

I found a few cases where I was doing the same work twice and fixed them. Here are some new numbers. These come from my Galaxy Nexus, a rather slow phone, so I think they are on the high side. Someday I will try it on my N5.

Downloading 5 covers:
Cover 1: 1160 milliseconds
Cover 2:  491 milliseconds
Cover 3:  587 milliseconds
Cover 4:  476 milliseconds
Cover 5:  571 milliseconds

In all cases CC requires around 200 milliseconds to resize the covers and store them into its cache.

EDIT: I tried it on my N5. The cover download times didn't improve much, but the 200 milliseconds went to 50.

This performance of around 1.5 covers per second clearly does not match what you are seeing from CalibreCloud, approximately 5 covers per second. My guess is that it is using Dropbox's thumbnail feature, something I cannot do because I need the real cover to insert into CC's database.

As for searching, the reason I am concerned about performance is that for consistency I would need to use the existing CC search engine, which is similar to calibre's. CC's search engine works on 'calibre-processed' information that does not exist in calibre's database (calibre is memory-based). I think that it will take around 250 milliseconds to generate that information for a book, meaning that search will run at around 4 books per second. I have no idea if this is acceptable. I know that it won't be for some number of people.

@Cybmole: thanks for the heads-up on possible problems with Fires. I will try it on mine once I get further along in development.

@Everyone: so far I haven't heard any problems with this feature being dropbox-only. That really would simplify things!

[DoctorOhh]

Dropbox sounds fine. Being ignorant in what you're facing I wonder if search couldn't be sped up with the metadata.db file being transferred to the device?

[chaley]

Originally Posted by DoctorOhh

Being ignorant in what you're facing I wonder if search couldn't be sped up with the metadata.db file being transferred to the device?

It might if I wrote another search engine to use directly the db. Unfortunately, calibre's search language can make that less than optimal. In particular, naked (no prefix) searches would be horrible because I would need to retrieve *all* the fields just to check if the string is in any of them. Partial matching would also be a pain, but less so. Parenthesized expressions with AND, OR, and NOT would also present difficulties to a system based on DB queries. If I gave up on using calibre's (and CC's) search language and allowed only single-question prefixed queries (queries of the form series:x or title:y) then I could quite easily search using metadata.db, but even here there is a potential problem with things like series_index.

I still remain unconvinced that search is needed, given that I am using a content-server tree-based interface. Here are some screen captures from the top-level down to a series list. Unless you are looking for something that is in the middle of an item (e.g., I want the book with "pirate" somewhere in the title) then the tree index will get you there.

[DoctorOhh]

I have never used the content server and CC either. What you display is perfectly fine for what is expected. Looking forward to trying it out.