MobileRead Forums - View Single Post - New Cloud Storage + OPDS Plugin needs testers

QuietThyme · 08-12-2015, 03:36 AM

@kovidgoyal
Thanks, I ended up just ripping out fromEncoded calls. I don't think I acutally needed them. At one point I was doing some debugging and I thought I was seeing an error due to encoding problems.

While I have you here, I'm looking at setting up a basic test suite of some sort for the plugin. It doesn't seem like there's an easy way to do that though. The only way I can think of is by making my methods accessable via the calibre CLI and then testing them that way. Is there some other way that I missed completely?

@chaley
Thanks for the thoughtful questions. I've been working on Quietthyme off and on for a while. It originally started as a calibre SAAS, but I recently decided to reduce the scope of the project and just leverage calibre as the metadata editor.

Metadata. Believe me, in QuietThyme's SAAS incarnation I had written parsers that let me retrieve metadata from PDFs, EPubs and Mobi books, and then try to automatically retrieve missing metadata using Amazon and Goodreads API. The pain of getting that whole cycle working "properly" gave me a whole new respect for what you calibre guys have done. In its current form, the QuietThyme plugin gets the current metadata of the book, and uploads it along with the ebook data. The metadata gets stored in a database, while the book gets uploaded to your chosen cloud storage. If the user deletes the book using the QuietThyme plugin, the book and the metadata get destroyed, but if the user manually deletes the book in the cloud storage, the OPDS can get out of sync. I have a todo list item to handle that case using webhooks and event feeds -- most cloud services offer one or the other.

OPDS metadata consistency. Hopefully this won't be much of a problem. Only the books which the user manually selected to be uploaded to their cloud storage will be visible in the OPDS feed. Second, calibre has a handy button called "Update cached metadata on device" which I can use to update the metadata in the OPDS catalog. It may require a bit of user education, but hopefully it wont be too confusing. Metadata updating is already separate from ebook uploading.
Security and credentials. Yep, I don't want to have to deal with credentials at all. Only cloud services that support OAuth are used. Unfortunately thats not the only trust problem. Dropbox has a pretty granular security model, and they have a sandbox mode that lets developers only read/write to files in a specific folder. However not all services are like that. For box/onedrive/google I have read and write permissions to their whole cloud drive. I'm obviously only modifying files in a QuietThyme folder that I create, but the application does have permissions to everything because those cloud services are stupid (or I just cant RTFM). In any case, I'm hoping the open source nature of the web application will alleviate some concerns.

Credentials to access OPDS. There are no credentials to access the OPDS catalogs right now. Its a bit of security though obscurity. I initially thought of using a random UUID to differentiate each catalog, and make it hard/time consuming to access them. I've since moved to a XKCD password style catalog url, which has a similar level of security, while balancing ease of use as these catalog urls will most likely be entered on a mobile device.
https://www.quietthyme.com/catalog/c...battery-staple
In the future if there's a request for it, I can even give users the ability to regenerate their catalog urls on demand.
Piracy. I've thought a bit about this. At present theres a scare message when you visit the catalog url directly, informing you that your IP is logged and the catalog url could be disabled at any time if abused. I've also added a DCMA email address for notices. I don't know how much anonymity QuietThyme would really add, as at the end of the day, the books are still hosted on your cloud storage, and theres a direct link from your catalog url to your dropbox/google/box account. I don't think theres going to be more piracy from QuietThyme links than there would be from a calibre content server.
Caching. Yep, I can definitely attest to the speed issues for pulling files out of cloud storage. Cover art is actually hosted on a cdn, and will not need to be pulled from cloud storage. The metadata and cover thumbnails don't take up much space, and if the need arises I can look into denormalizing the database, compressing the cover thumbs and potentially moving from parse to AWS or some other cloud platform. For the end user, the hosted assets in the CDN and the OPDS metadata coming directly from a database means the catalog will be reasonably fast, while the book download might take a bit longer than it would from the calibre content server. Even that delay should be balanced out by the fact that the books will be served to the users over corporate tier bandwidth.
Costs. Hah, if it gets to that point, QuietThyme would have gone full circle. The initial version was built around the premise of charging for QuietThyme storage while allowing users to connect their own cloud storage. But honestly I didn't think that it was a viable business strategy, dropbox and google are always going to be able to provide a better storage price point, which is why I moved away from that idea. It also seems a bit disingenuous to charge for QuietThyme the way it is now, as I make use of the user's cloud storage, and the metadata is all from Calibre (I don't even need to run my custom metadata parsers anymore). The VPS farm I'm running now was used for some fairly large docker based projects, and QuietThyme only uses a small percentage of it, even under load testing. If it does get popular, I do have some monetization ideas: such as ebook content searching, recommendations using affiliate links to ebook stores, mobile apps with annotation support that could feed back to calibre, and a few others. I have no real immediate plans to monetize though I'm open if you have some ideas.

Sorry for the wall of text

-Jason

08-12-2015, 03:36 AM	#11
QuietThyme Junior Member Posts: 6 Karma: 475632 Join Date: Jan 2013 Device: IOS	@kovidgoyal Thanks, I ended up just ripping out fromEncoded calls. I don't think I acutally needed them. At one point I was doing some debugging and I thought I was seeing an error due to encoding problems. While I have you here, I'm looking at setting up a basic test suite of some sort for the plugin. It doesn't seem like there's an easy way to do that though. The only way I can think of is by making my methods accessable via the calibre CLI and then testing them that way. Is there some other way that I missed completely? @chaley Thanks for the thoughtful questions. I've been working on Quietthyme off and on for a while. It originally started as a calibre SAAS, but I recently decided to reduce the scope of the project and just leverage calibre as the metadata editor. Metadata. Believe me, in QuietThyme's SAAS incarnation I had written parsers that let me retrieve metadata from PDFs, EPubs and Mobi books, and then try to automatically retrieve missing metadata using Amazon and Goodreads API. The pain of getting that whole cycle working "properly" gave me a whole new respect for what you calibre guys have done. In its current form, the QuietThyme plugin gets the current metadata of the book, and uploads it along with the ebook data. The metadata gets stored in a database, while the book gets uploaded to your chosen cloud storage. If the user deletes the book using the QuietThyme plugin, the book and the metadata get destroyed, but if the user manually deletes the book in the cloud storage, the OPDS can get out of sync. I have a todo list item to handle that case using webhooks and event feeds -- most cloud services offer one or the other. OPDS metadata consistency. Hopefully this won't be much of a problem. Only the books which the user manually selected to be uploaded to their cloud storage will be visible in the OPDS feed. Second, calibre has a handy button called "Update cached metadata on device" which I can use to update the metadata in the OPDS catalog. It may require a bit of user education, but hopefully it wont be too confusing. Metadata updating is already separate from ebook uploading. Security and credentials. Yep, I don't want to have to deal with credentials at all. Only cloud services that support OAuth are used. Unfortunately thats not the only trust problem. Dropbox has a pretty granular security model, and they have a sandbox mode that lets developers only read/write to files in a specific folder. However not all services are like that. For box/onedrive/google I have read and write permissions to their whole cloud drive. I'm obviously only modifying files in a QuietThyme folder that I create, but the application does have permissions to everything because those cloud services are stupid (or I just cant RTFM). In any case, I'm hoping the open source nature of the web application will alleviate some concerns. Credentials to access OPDS. There are no credentials to access the OPDS catalogs right now. Its a bit of security though obscurity. I initially thought of using a random UUID to differentiate each catalog, and make it hard/time consuming to access them. I've since moved to a XKCD password style catalog url, which has a similar level of security, while balancing ease of use as these catalog urls will most likely be entered on a mobile device. https://www.quietthyme.com/catalog/c...battery-staple In the future if there's a request for it, I can even give users the ability to regenerate their catalog urls on demand. Piracy. I've thought a bit about this. At present theres a scare message when you visit the catalog url directly, informing you that your IP is logged and the catalog url could be disabled at any time if abused. I've also added a DCMA email address for notices. I don't know how much anonymity QuietThyme would really add, as at the end of the day, the books are still hosted on your cloud storage, and theres a direct link from your catalog url to your dropbox/google/box account. I don't think theres going to be more piracy from QuietThyme links than there would be from a calibre content server. Caching. Yep, I can definitely attest to the speed issues for pulling files out of cloud storage. Cover art is actually hosted on a cdn, and will not need to be pulled from cloud storage. The metadata and cover thumbnails don't take up much space, and if the need arises I can look into denormalizing the database, compressing the cover thumbs and potentially moving from parse to AWS or some other cloud platform. For the end user, the hosted assets in the CDN and the OPDS metadata coming directly from a database means the catalog will be reasonably fast, while the book download might take a bit longer than it would from the calibre content server. Even that delay should be balanced out by the fact that the books will be served to the users over corporate tier bandwidth. Costs. Hah, if it gets to that point, QuietThyme would have gone full circle. The initial version was built around the premise of charging for QuietThyme storage while allowing users to connect their own cloud storage. But honestly I didn't think that it was a viable business strategy, dropbox and google are always going to be able to provide a better storage price point, which is why I moved away from that idea. It also seems a bit disingenuous to charge for QuietThyme the way it is now, as I make use of the user's cloud storage, and the metadata is all from Calibre (I don't even need to run my custom metadata parsers anymore). The VPS farm I'm running now was used for some fairly large docker based projects, and QuietThyme only uses a small percentage of it, even under load testing. If it does get popular, I do have some monetization ideas: such as ebook content searching, recommendations using affiliate links to ebook stores, mobile apps with annotation support that could feed back to calibre, and a few others. I have no real immediate plans to monetize though I'm open if you have some ideas. Sorry for the wall of text -Jason Last edited by QuietThyme; 08-12-2015 at 03:39 AM. Reason: english.