I am in active discussion with Kovid about replacing calibre's back end with a plugin architecture. Kovid brought the up when I asked him about refactoring the device interface. He is not driven to give access to book files, but instead to support diverse storage possibilities. The ones he mentioned are cloud storage, non-relational DBs, and pure filesystem storage. I add virtual file systems, devices as libraries, and client-server setups. We have talked about it supporting multiple open libraries simultaneously. Discussions have halted while 0.7 goes through its growing pains, but I expect them to restart in the not far future.
There are some downsides:
1) It will be an incredible amount of work, almost a complete rewrite of the non-gui code and a substantial rewrite of the gui. By incredible, I mean thousands of hours. Fortunately, with V7 we have (almost completely) moved to a dictionary-based metadata structure, which will make the transition easier. (No, this wasn't an accident. What remains is to migrate the devices to the more general structure.)
2) It will almost certainly be behavior-inconsistent with the current calibre. It is also likely that some features will disappear. We will know only when we have arrived.
3) Such an architecture will give users enough rope to hang themselves. The support challenges are formidable now, and having random plugins floating around running on the some variant of the Glasnost 87 slate will make it near impossible. It is likely that only calibre-supplied back ends in specific configurations will be supported.
Current thoughts are to begin with migrating the device interface, with the goal of looking for the gotchas. Kovid will take the lead on this, outlining what he wants to see. Once that is semi-stable, then moving the library will be considered.
Staging has not been discussed. Kovid will work that out as time passes. That said, I will be surprised if anything appears within 6 months, and a year is more likely.