MobileRead Forums - View Single Post - New Cloud Storage + OPDS Plugin needs testers

chaley · 08-11-2015, 03:44 AM

@QuietThyme: Thanks for the answer.

Below are a few random thoughts. I haven't looked at the code so they might be totally off base, and it is highly likely that you have already thought of/dealt with most of them.

Metadata. The OPDS server must have accurate metadata for the books in order to function. You can't reliably get it from the books because some formats don't have metadata and for those that do, plugboards can change it. The device driver is given the correct metadata. I imagine that you would store the accurate metadata somewhere in the user's cloud so you can regenerate the ODPS tree when the user sends or deletes a book. Alternatively you could store it yourself. In any event, you will have an consistency problem because users can (and will) change what is in their cloud without telling you.

Another issue: calibre content server users are accustomed to the metadata (the navigation tree) being correct even after metadata changes. It might be necessary to set expectations appropriately, because (I think) in your system the metadata will be updated only when a book is sent. You might end up needing to do what we do in Calibre Companion (CC) and calibre's wireless device driver, which is syncing calibre metadata separate from sending books.
Security and credentials. Users will be (or should be) unwilling to give you their cloud credentials. If cloud providers support OAUTH then this a manageable problem because you never see the actual credentials (or shouldn't), but if the provider does not then you must convince people that your credential store is secure and you are trustworthy.

Also, the credentials to access the OPDS server must be separate from the credentials to access the cloud provider. Users will certainly want to share access to each other's servers, and they will certainly not want to share access to their cloud provider.
Piracy. Given the indirection between the OPDS users and the books it will be tempting for some people to use your system to distribute pirated material. The "problem" arises because the cloud provider thinks that the books are private and so doesn't (usually) scan for copyright violations. Your server turns the private cloud into a public one behind the provider's back and (perhaps) provides a level of anonymity to the pirate. This could lead to you having to deal with DCMA takedowns or worse.
Caching. Our experience with CC is that cloud providers are very slow, painfully slow at times. For example, it can easily take a half-second to fetch a cover. We work around this by doing several in parallel and caching the results on the user's device. In your case, the cache would need to be on your server, opening (again) the data consistency problem and also consuming space on your servers. Of course, this performance problem mostly goes away if you are storing the entire OPDS tree including thumbnails on your servers, but that increases the amount of storage you must provide.
I imagine that if this works out and becomes popular then you might need to charge something to cover the costs of your VPS farm. If you go that route then you might also consider providing the storage for the books. You could even consider shadowing the user's library and running a calibre content server, although that is obviously different from what you are proposing today.

08-11-2015, 03:44 AM	#10
chaley Grand Sorcerer Posts: 12,475 Karma: 8025702 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	@QuietThyme: Thanks for the answer. Below are a few random thoughts. I haven't looked at the code so they might be totally off base, and it is highly likely that you have already thought of/dealt with most of them. Metadata. The OPDS server must have accurate metadata for the books in order to function. You can't reliably get it from the books because some formats don't have metadata and for those that do, plugboards can change it. The device driver is given the correct metadata. I imagine that you would store the accurate metadata somewhere in the user's cloud so you can regenerate the ODPS tree when the user sends or deletes a book. Alternatively you could store it yourself. In any event, you will have an consistency problem because users can (and will) change what is in their cloud without telling you. Another issue: calibre content server users are accustomed to the metadata (the navigation tree) being correct even after metadata changes. It might be necessary to set expectations appropriately, because (I think) in your system the metadata will be updated only when a book is sent. You might end up needing to do what we do in Calibre Companion (CC) and calibre's wireless device driver, which is syncing calibre metadata separate from sending books. Security and credentials. Users will be (or should be) unwilling to give you their cloud credentials. If cloud providers support OAUTH then this a manageable problem because you never see the actual credentials (or shouldn't), but if the provider does not then you must convince people that your credential store is secure and you are trustworthy. Also, the credentials to access the OPDS server must be separate from the credentials to access the cloud provider. Users will certainly want to share access to each other's servers, and they will certainly not want to share access to their cloud provider. Piracy. Given the indirection between the OPDS users and the books it will be tempting for some people to use your system to distribute pirated material. The "problem" arises because the cloud provider thinks that the books are private and so doesn't (usually) scan for copyright violations. Your server turns the private cloud into a public one behind the provider's back and (perhaps) provides a level of anonymity to the pirate. This could lead to you having to deal with DCMA takedowns or worse. Caching. Our experience with CC is that cloud providers are very slow, painfully slow at times. For example, it can easily take a half-second to fetch a cover. We work around this by doing several in parallel and caching the results on the user's device. In your case, the cache would need to be on your server, opening (again) the data consistency problem and also consuming space on your servers. Of course, this performance problem mostly goes away if you are storing the entire OPDS tree including thumbnails on your servers, but that increases the amount of storage you must provide. I imagine that if this works out and becomes popular then you might need to charge something to cover the costs of your VPS farm. If you go that route then you might also consider providing the storage for the books. You could even consider shadowing the user's library and running a calibre content server, although that is obviously different from what you are proposing today.