Storing plugin data for multi-machine usage

[kiwidude]

A post today on the Reading List plugin has prompted me to rethink about how I store per library plugin data in particular, and I am interested to know what options exist in calibre today and what might be considered adding for it.

Right now every plugin I have written at least uses a local json file in the plugins folder to store all its data. It is the highest performance, most convenient to develop with, not impacted by people rebuilding their libraries and allows library-agnostic setting storage. Given that calibre also serializes a lot of configuration style data into the same directory area it seemed a reasonable approach at the time at least.

However with Kovid changing the back-end to one day support calibre being used from multiple machines, to an end-user this approach becomes a rather inconvenient if not downright flawed one. Certainly there can be some settings which you would keep machine specific, for which the json file remains a good choice. But there are also plugin settings a user will want to share across their machines. Some may still be library agnostic (such as configuring Search the Internet menus or Generate Cover settings) - the user can use features I have written within the plugins to import/export between their machines as a workaround. However others have library specific data such as reading lists which screams out to be stored in the library database.

Right now the data storage options from a plugin I am aware of are:

• private file storage, such as the json files, local resource images etc
• in-memory storage ("marked data"), only for temporary per book data, used by many plugins such as Quality Check
• library database per book storage in a custom column.
• library database per book storage ("custom book data"), used by Find Duplicates to store computed hashes for performance reasons.

Anything else I have forgotten/don't know about? Specifically there are two more kinds of storage that might be desirable or else have to be simulated using the above:

• library database storage that is not per book, allowing a plugin to serialise a dict of values. For instance an ordered list of book ids for the reading list plugin, or custom column names assigned to specific functions.
• cross-library storage. Today calibre does this using files in its configuration directory, which will become a potential problem for users with multiple computers. They can put their library in a shared folder, but what about sharing their calibre settings and the implications thereof? Perhaps my import/export approach is the only "practical" solution.

Appreciate any thoughts...

[kovidgoyal]

I just committed some code you can use for a per library preference store:

db.prefs.set_namespaced(namespace, key, val)
db.prefs.get_namespaced(namespace, key, default=None)

use your plugin import name as the namespace (though you can use anything you like).

For the second question, there is no mechanism for that, and the development of such a mechanism will have to wait for the development of multi-client calibre

[kiwidude]

Awesome, thanks Kovid, I was hoping you might do something like that.

There are still some other considerations that plugins will have to be wary of when the multi-client things rolls around of course - the standard concurrency type of issues. Such as where stale data in a plugin on one running instance of calibre will overwrite that being used in memory of another instance. I will be interested to see which approach you choose when you get there...

[chaley]

@kiwidude: avoid using colons in the namespace and key names. Kovid's code uses a colon between these fields to generate the database key, which could in pathological cases lead to collisions. For example, (namespace="foo", key="bar:mumble") will collide with (namespace="foo:bar", key="mumble").

@kovid: you might want to forbid use of colons in both the namespace and the key to avoid the possibility of collisions. Not that such a collision would ever happen, of course.

[kovidgoyal]

I've got set_namespaced() checking for colons in key and namespace, however, collisions can still happen given that people could be using direct access. But not a lot that can be done about that.

[kiwidude]

I've had theducks in my ear with a suggestion which prompted something I would rather discuss here...

What about backup/restore of such settings if they are stored in the library? I've not personally had a need to to rebuild my library in 2 years of calibre, but there are users who have, either via disk corruption or doing silly things like killing calibre processes / rebooting while it is writing to disk etc.

Putting these settings in the calibre library will potentially put them at a higher risk of corruption. And unlike rebuilding from calibre's .opf files, there would (I assume?) be no means of restoring them, other than a user being disciplined enough to have a recent and uncorrupted backup of the metadata.db?

Any thoughts on this? So unlikely as to not be an issue? Or could there be some mechanism added to calibre's backup/restore functionality to similarly backup the settings into a file in the root of the library folder alongside the library?

[theducks]

I am copying a PM I had done offline to Kiwidude.

Storing settings with Library

Originally Posted by kiwidude

That is an interesting suggestion. Though you should just make it on the thread I put in the dev forum though . Kovid's change he has just made will suffice for most of the plugins that come to mind, particularly since there is no per user settings (excepting Goodreads Sync which has a specific mechanism to support them).

It is however not the complete solution - cross library settings are not solved. And a user deleting/corrupting their metadata.db will lose their settings, which your suggestion could circumvent (though disk corruption could just happen in that folder too - just perhaps less likely given less access). Maybe Kovid could add to the metadata backup/restore mechanism to also backup the table that will contain this settings data to a file. Guess it depends on how likely it is that the metadata.db gets corrupted - in two years of calibre I have not once had cause to rebuild my database (and would go to a backup as a preferred choice) but plenty of people have done so it seems.

[kovidgoyal]

In my experience db corruption is rare enough and most preferences are not important enough (after all in the worst case they can just be re-created) that this is not worth doing. But patches are welcome, if you feel differently.

[chaley]

I pushed code to save db preferences when the GUI exits and to restore them when the db is restored. Preferences are saved to the file metadata_db_prefs.json in the same directory as metadata.db.

Preferences are not saved by command line tools. I don't think any of them modify preferences, so this shouldn't be a problem. Ahh ... perhaps saved searches can be created and deleted by command line tools. These changes will be lost if the gui hasn't run between the command line invocation and the restore.

There is another possible hole. If we have the sequence a) the preferences file came from db version N, b) calibre has been updated to db version N+1 and only a command line tool has run, and c) the upgrade function modifies preferences, then the restored preferences may be incorrect. Up to now, all db preference modifications have been done outside the db upgrade functions, so this isn't currently an issue.

[kiwidude]

Thanks chaley

[chaley]

The restore code is now in trunk.

A side-benefit of the change is that because field_metadata is stored in the prefs, we have a more accurate set of custom column definitions. In particular, composite column and formatting templates should reflect the latest changes. Before, the opf backups had to catch up, leaving a potentially long period of inconsistency. Also, now custom template functions, saved searches, and a lot of other "stuff" are now restored.

[hakan42]

I'm just now reading through the source of "Count Pages" plugin to understand how this [get|set]_library_config works.

Am I correct to assume that I can pass whole dicts into those methods, as I would with JSONConfig? And if so, is there a practical limit on the size of data I push?

As of now, my plugin data (OPDS Client for Beam EBooks) is about 800 bytes, but with a list of harvested URLs I intend to keep track of, it could grow to 4-5 kilobytes.

[kovidgoyal]

I would recommend against storing gigantic chunks of data in sqlite, instead use the plugins config folder for that. Also the only reason to store something in the database is if it is library psecific, otherwise use the config folder.

That said, storing a few MB in the db should not cause any problems.

[hakan42]

Originally Posted by kovidgoyal

I would recommend against storing gigantic chunks of data in sqlite, instead use the plugins config folder for that. Also the only reason to store something in the database is if it is library psecific, otherwise use the config folder.

Actually, my use case asks for shared data. During the daylight hours, I am using calibre on my laptop, doing all the things one does with it. My OPDS client can (optionally, of course) download new episodes by invoking it with "--run-plugin" from cron on my basement Linux server. Both installations share the same library, but not necessarily the same plugins config directory.

Originally Posted by kovidgoyal

That said, storing a few MB in the db should not cause any problems.

I will try to keep my data below 10 - 15 kb, this should not hurt too much