New Cloud Storage + OPDS Plugin needs testers

[QuietThyme]

New Cloud Storage + OPDS Plugin needs testers

Hi,
I've been working on an open source device driver plugin + web server for Calibre.

Features:
- It allows you to connect many of the most common cloud storage providers (Dropbox, Google Drive, etc.)
- Clouds are connected as a device drives in Calibre (manage them as you would books on your eReader/iPad)
- Automatically creates a web accessible OPDS server (you don't have to keep your computer on to run Calibre Content Server, or open ports on your router to access your library)
- There's no limit to how many books you can store (you're only limited by your aggregate dropbox/google drive storage space)

Screenshots:

Configuration Screen (Dropbox + Google Drive Connected)



Standard Calibre Device Management


Storage Providers:
Here's a list of the storage providers that I currently have support for:
- Dropbox
- Google Drive
- OneDrive (SkyDrive)
- Box

Here's a list of storage providers I want to add support for if there's enough appeal:
- iCloud
- Bittorrent Sync
- OwnCloud

Requests:
- I need developers who are willing to be beta testers for the plugin. Once we've ironed out any bugs I've missed I'll repost to the plugins forumn.
- I'm trying to figure out how to customize the UI to display custom icons and labels for the Card and Device buttons, I couldnt really figure out how to do that from the device driver api. Any help here would be appreciated.

- I'm trying to gauge interest in the plugin and see if there's other storage providers I should support. If you have any storage requests, please leave a comment.

Please share this with any developers you think might be interested in beta testing or contributing. The plugin and webserver are both open source, and I would appreciate any help

-Jason


Installation Instructions

• Download the QuietThyme Device Plugin.zip file
• Open Calibre
• Open Preferences
• Under Advanced, click "Plugins"
• Click "Load Plugin from File"
• Select the zip file and click "Open"
• In the Preferences - Plugin list, select "QuietThyme Device Plugin"
• Click "Customize plugin"
• Record your Catalog URL (This is your OPDS url)
• Connect your Cloud Storage provider by clicking the "Connect" button under either Dropbox, Google Drive, etc.
• Approve the connection by signing into your service
• Once your Storage provider is connected, click Ok
• Close all Preferences windows

Usage Instructions
With QuietThyme installed, you'll see a few extra menu items. It is incredibly simple to store your books in your cloud storage.

• Right click on any book in your library and select "Send to device"
• Send to "Main Memory"
• On your OPDS compatible ebook reader, open up your Catalog URL
• You should see your new book available for download with all its metadata and cover art.

Beta Updates
Oct 22, 2015 - Misc. cleanup, web service api changes, added tests
Aug 09, 2015 - Initial release

[kovidgoyal]

IIRC you can customize the icon used for the device, by setting

icon = 'path to some icon file'

in the device plugin.

I dont think you can customize the card icons or any of the text.

[eschwartz]

I tried installing the plugin and after restarting calibre it keeps on connecting and disconnecting to a "device".

Attempting to configure it through the plugin preferences yields the following error message:

calibre, version 2.34.0
ERROR: Unhandled exception: <b>TypeError</b>:QUrl.fromEncoded(QByteArray, QUrl.ParsingMode mode=QUrl.TolerantMode): argument 1 has unexpected type 'unicode'

calibre 2.34  isfrozen: False is64bit: False
Linux-4.1.4-1-ARCH-i686-with-glibc2.0 Linux ('32bit', 'ELF')
('Linux', '4.1.4-1-ARCH', '#1 SMP PREEMPT Mon Aug 3 21:53:10 UTC 2015')
Python 2.7.10
Linux: ('', '', '')
Successfully initialized third party plugins: DeDRM && Quality Check && Kindle Collections && QuietThyme Device Plugin && Open With && View Manager && FanFicFare && KindleUnpack - The Plugin && Diaps Editing Toolbag && Overdrive Link && Goodreads Sync && Annotations && Modify ePub && Overdrive Link
Traceback (most recent call last):
  File "/home/eschwartz/git/calibre/src/calibre/gui2/preferences/plugins.py", line 289, in double_clicked
    self.modify_plugin(op='customize')
  File "/home/eschwartz/git/calibre/src/calibre/gui2/preferences/plugins.py", line 370, in modify_plugin
    if plugin.do_user_config(self.gui):
  File "/home/eschwartz/git/calibre/src/calibre/customize/__init__.py", line 154, in do_user_config
    config_widget = self.config_widget()
  File "calibre_plugins.quietthyme.__init__", line 735, in config_widget
  File "calibre_plugins.quietthyme.config", line 47, in __init__
TypeError: QUrl.fromEncoded(QByteArray, QUrl.ParsingMode mode=QUrl.TolerantMode): argument 1 has unexpected type 'unicode'

Still some work, I'm afraid...


The idea sounds really interesting. Looking forward to seeing it in action.

[QuietThyme]

Hey eschwartz,
Yep, looks like your right. I only did testing on my machine using the latest Calibre, and I haven't done any testing against QT4.

I'll take another look at it and release a version this evening.

In case anyone's wondering, the source for the plugin is located here:
https://github.com/AnalogJ/quietthyme.plugin


Thanks for the feedback

[chaley]

It would be interesting to know how this improves on calibre's connect to folder where the folder is in the cloud provider's sync directory (answer: opds server?), and where the OPDS server is running as it seems not to be on a computer controlled by the user.

[eschwartz]

Originally Posted by QuietThyme

Hey eschwartz,
Yep, looks like your right. I only did testing on my machine using the latest Calibre, and I haven't done any testing against QT4.

I'll take another look at it and release a version this evening.

In case anyone's wondering, the source for the plugin is located here:
https://github.com/AnalogJ/quietthyme.plugin


Thanks for the feedback

Nope -- take another look at my traceback: calibre 2.34.0
No Qt4 here.

In fact, I am probably one of the only people anywhere who actually bothers to run calibre from a live checkout of the code, updated daily. But I doubt the fourteen commits since Friday broke compatibility.

OTOH, I am using ArchLinux and building calibre from source -- which means e.g. my pyqt5 is more up-to-date than the official calibre builds. So I catch a few bugs that haven't made its way into official releases yet.

Something like:

http://pyqt.sourceforge.net/Docs/PyQt5/gotchas.html

If Qt expects a QByteArray then PyQt5 will accept a unicode that contains only Latin-1 characters. This is deprecated in PyQt v5.4 and will be removed in PyQt v5.5.)

[eschwartz]

Originally Posted by chaley

It would be interesting to know how this improves on calibre's connect to folder where the folder is in the cloud provider's sync directory (answer: opds server?), and where the OPDS server is running as it seems not to be on a computer controlled by the user.

I believe this plugin does not require an active Dropbox et al. client syncing, which could be nice. Esp. for GDrive, which is currently languishing doing nothing for me, since it is far too awkward to use on linux.

[QuietThyme]

@eschwartz

Originally Posted by eschwartz

If Qt expects a QByteArray then PyQt5 will accept a unicode that contains only Latin-1 characters. This is deprecated in PyQt v5.4 and will be removed in PyQt v5.5.)

I'm still not completely familiar with the way that PyQt works internally, but that does look like the problem. I've switched the QUrl.fromEncoded call to use the QUrl constructor directly (it supports QString). I've made the change but I won't make a new release just yet, as there's probably other v5.5 issues that I've missed. I'm going to try to come up with a plugin test suite that I can run inside a docker container that will test the plugin out in multiple versions of calibre.

@chaley
As eschwartz mentioned, the plugin doesn't require you to have any client software installed on your machine, meaning you can store your books on dropbox or google drive with out their sync applications installed, or even (in the future) sync you books to your iCloud on a windows machine. The main difference is definitely the OPDS support though. It's not a very complicated system but I feel like it merges the best parts of 'Connect to Folder' and 'Content Server' without opening up any ports or setting up any dynamic dns services.

Since we're all developers here, I feel like I should get into the gritty details a bit. Like chaley mentioned, the web server is on a computer that the user doesn't control, for the reasons I specified above. That said, the server is completely open source and the code can be viewed here:
https://github.com/AnalogJ/quietthyme.web

The web server component is currently baked into a docker image, and then deployed on a cluster of VPS's that I manage. The web server is written in node (using the sails framework) and its only role is to upload/download files from cloud storage and generate the OPDS catalog.

[kovidgoyal]

If you want to use fromEncoded in PyQt 5.5, you just need to do this

QUrl.fromEncoded(x if isinstance(x, bytes) else x.encode('utf-8'))

[chaley]

@QuietThyme: Thanks for the answer.

Below are a few random thoughts. I haven't looked at the code so they might be totally off base, and it is highly likely that you have already thought of/dealt with most of them.

• Metadata. The OPDS server must have accurate metadata for the books in order to function. You can't reliably get it from the books because some formats don't have metadata and for those that do, plugboards can change it. The device driver is given the correct metadata. I imagine that you would store the accurate metadata somewhere in the user's cloud so you can regenerate the ODPS tree when the user sends or deletes a book. Alternatively you could store it yourself. In any event, you will have an consistency problem because users can (and will) change what is in their cloud without telling you.
  
  Another issue: calibre content server users are accustomed to the metadata (the navigation tree) being correct even after metadata changes. It might be necessary to set expectations appropriately, because (I think) in your system the metadata will be updated only when a book is sent. You might end up needing to do what we do in Calibre Companion (CC) and calibre's wireless device driver, which is syncing calibre metadata separate from sending books.
• Security and credentials. Users will be (or should be) unwilling to give you their cloud credentials. If cloud providers support OAUTH then this a manageable problem because you never see the actual credentials (or shouldn't), but if the provider does not then you must convince people that your credential store is secure and you are trustworthy.
  
  Also, the credentials to access the OPDS server must be separate from the credentials to access the cloud provider. Users will certainly want to share access to each other's servers, and they will certainly not want to share access to their cloud provider.
• Piracy. Given the indirection between the OPDS users and the books it will be tempting for some people to use your system to distribute pirated material. The "problem" arises because the cloud provider thinks that the books are private and so doesn't (usually) scan for copyright violations. Your server turns the private cloud into a public one behind the provider's back and (perhaps) provides a level of anonymity to the pirate. This could lead to you having to deal with DCMA takedowns or worse.
• Caching. Our experience with CC is that cloud providers are very slow, painfully slow at times. For example, it can easily take a half-second to fetch a cover. We work around this by doing several in parallel and caching the results on the user's device. In your case, the cache would need to be on your server, opening (again) the data consistency problem and also consuming space on your servers. Of course, this performance problem mostly goes away if you are storing the entire OPDS tree including thumbnails on your servers, but that increases the amount of storage you must provide.
• I imagine that if this works out and becomes popular then you might need to charge something to cover the costs of your VPS farm. If you go that route then you might also consider providing the storage for the books. You could even consider shadowing the user's library and running a calibre content server, although that is obviously different from what you are proposing today.

[QuietThyme]

@kovidgoyal
Thanks, I ended up just ripping out fromEncoded calls. I don't think I acutally needed them. At one point I was doing some debugging and I thought I was seeing an error due to encoding problems.

While I have you here, I'm looking at setting up a basic test suite of some sort for the plugin. It doesn't seem like there's an easy way to do that though. The only way I can think of is by making my methods accessable via the calibre CLI and then testing them that way. Is there some other way that I missed completely?

@chaley
Thanks for the thoughtful questions. I've been working on Quietthyme off and on for a while. It originally started as a calibre SAAS, but I recently decided to reduce the scope of the project and just leverage calibre as the metadata editor.

• Metadata. Believe me, in QuietThyme's SAAS incarnation I had written parsers that let me retrieve metadata from PDFs, EPubs and Mobi books, and then try to automatically retrieve missing metadata using Amazon and Goodreads API. The pain of getting that whole cycle working "properly" gave me a whole new respect for what you calibre guys have done. In its current form, the QuietThyme plugin gets the current metadata of the book, and uploads it along with the ebook data. The metadata gets stored in a database, while the book gets uploaded to your chosen cloud storage. If the user deletes the book using the QuietThyme plugin, the book and the metadata get destroyed, but if the user manually deletes the book in the cloud storage, the OPDS can get out of sync. I have a todo list item to handle that case using webhooks and event feeds -- most cloud services offer one or the other.
  
  
  OPDS metadata consistency. Hopefully this won't be much of a problem. Only the books which the user manually selected to be uploaded to their cloud storage will be visible in the OPDS feed. Second, calibre has a handy button called "Update cached metadata on device" which I can use to update the metadata in the OPDS catalog. It may require a bit of user education, but hopefully it wont be too confusing. Metadata updating is already separate from ebook uploading.
  
• Security and credentials. Yep, I don't want to have to deal with credentials at all. Only cloud services that support OAuth are used. Unfortunately thats not the only trust problem. Dropbox has a pretty granular security model, and they have a sandbox mode that lets developers only read/write to files in a specific folder. However not all services are like that. For box/onedrive/google I have read and write permissions to their whole cloud drive. I'm obviously only modifying files in a QuietThyme folder that I create, but the application does have permissions to everything because those cloud services are stupid (or I just cant RTFM). In any case, I'm hoping the open source nature of the web application will alleviate some concerns.
  
  Credentials to access OPDS. There are no credentials to access the OPDS catalogs right now. Its a bit of security though obscurity. I initially thought of using a random UUID to differentiate each catalog, and make it hard/time consuming to access them. I've since moved to a XKCD password style catalog url, which has a similar level of security, while balancing ease of use as these catalog urls will most likely be entered on a mobile device.
  https://www.quietthyme.com/catalog/c...battery-staple
  In the future if there's a request for it, I can even give users the ability to regenerate their catalog urls on demand.
  
• Piracy. I've thought a bit about this. At present theres a scare message when you visit the catalog url directly, informing you that your IP is logged and the catalog url could be disabled at any time if abused. I've also added a DCMA email address for notices. I don't know how much anonymity QuietThyme would really add, as at the end of the day, the books are still hosted on your cloud storage, and theres a direct link from your catalog url to your dropbox/google/box account. I don't think theres going to be more piracy from QuietThyme links than there would be from a calibre content server.
  
• Caching. Yep, I can definitely attest to the speed issues for pulling files out of cloud storage. Cover art is actually hosted on a cdn, and will not need to be pulled from cloud storage. The metadata and cover thumbnails don't take up much space, and if the need arises I can look into denormalizing the database, compressing the cover thumbs and potentially moving from parse to AWS or some other cloud platform. For the end user, the hosted assets in the CDN and the OPDS metadata coming directly from a database means the catalog will be reasonably fast, while the book download might take a bit longer than it would from the calibre content server. Even that delay should be balanced out by the fact that the books will be served to the users over corporate tier bandwidth.
  
• Costs. Hah, if it gets to that point, QuietThyme would have gone full circle. The initial version was built around the premise of charging for QuietThyme storage while allowing users to connect their own cloud storage. But honestly I didn't think that it was a viable business strategy, dropbox and google are always going to be able to provide a better storage price point, which is why I moved away from that idea. It also seems a bit disingenuous to charge for QuietThyme the way it is now, as I make use of the user's cloud storage, and the metadata is all from Calibre (I don't even need to run my custom metadata parsers anymore). The VPS farm I'm running now was used for some fairly large docker based projects, and QuietThyme only uses a small percentage of it, even under load testing. If it does get popular, I do have some monetization ideas: such as ebook content searching, recommendations using affiliate links to ebook stores, mobile apps with annotation support that could feed back to calibre, and a few others. I have no real immediate plans to monetize though I'm open if you have some ideas.

Sorry for the wall of text

-Jason

[kovidgoyal]

You can run any arbitrary python script, with full access to plugins, like this:

calibre-debug file.py

and in file.py

from calibre_plugins.whatever import whatever
...

[QuietThyme]

Sorry for the long delay. I've been traveling a lot for work.

The latest version of the plugin is now available. I've attached it below, but I'll also update the main post.

If you run into any issues while testing the plugin, please post a response or create an issue on github:

https://github.com/AnalogJ/quietthyme.plugin/issues

-Jason

[DerBusIsVoll]

Hello,

I tried to install the plugin, but when I try to configute it, I get:

{"success":false,"error_msg":"No Authorization header was found"}

as response. What am I missing?

Brgds DerBusIsVoll

[mrbrimi]

Installed plugin but when I try to connect to OneDrive get error message about forbidden. When I try the other services the log in pages appear properly.