MobileRead Forums - View Single Post - New database backend

kovidgoyal · 07-28-2013, 12:56 AM

Hi all,

After a year and a half of development, the new calibre database backend is finally ready for testing.

The major goals of this new backend are:

Improve performance
Be thread safe (thereby eventually enabling multi-user calibre)
Maintain compatibility of database format with the previous backend. This means that you can freely switch between the old and new backends.
Also, there is a new cover grid based on the new backend, see attached screenshot. You can turn it on by clicking the button in the bottom right corner of the calibre main window that has an icon of a grid on it.

In order to test the new backend, do the following steps:

WARNING: Only do the following if you have backed up your calibre libraries and are willing to encounter and help debug bugs.

Update to calibre 0.9.44
Go to Preferences->Tweaks and click the button "Plugin Tweaks" in the bottom left corner
In the text box, type the following
Code:
```
use_new_db = True
```
Click Apply
Restart calibre

You should now see [newdb] in the status bar at the bottom of the main calibre window. This indicates you are successfully running with the new db backend.

You can switch back to the old backend at any time by simply changing that tweak to False and restarting calibre.

The database backend in calibre is an extremely complex bit of code, so while there is an extensive test suite for the new backend and I have been running it myself for a couple of weeks, there will be bugs. So please backup your calibre libraries before embarking on this adventure

Some details on the performance improvements:

The startup time for calibre with my 22,000 book test library has gone down from 11.5 to 6.4 seconds which is almost twice as fast.
The time taken to refresh the Tag Browser after an edit is also greatly reduced.
Searches are cached, so if you do a search (or use a virtual library) you should notice a speedup when re-doing the same search or re-applying the same virtual library
The code path for evaluating composite columns (based on templates) has been greatly optimized

A note for plugin developers: I have tried hard to maintain API compatibility between the old and new backends, so hopefully most plugins should continue to work, without modification. However, given the complexity of the code involved there may well be regressions, if so I will be happy to help with fixing them.

07-28-2013, 12:56 AM	#1
kovidgoyal creator of calibre Posts: 45,415 Karma: 27757236 Join Date: Oct 2006 Location: Mumbai, India Device: Various	New database backend - testers needed Hi all, After a year and a half of development, the new calibre database backend is finally ready for testing. The major goals of this new backend are: Improve performance Be thread safe (thereby eventually enabling multi-user calibre) Maintain compatibility of database format with the previous backend. This means that you can freely switch between the old and new backends. Also, there is a new cover grid based on the new backend, see attached screenshot. You can turn it on by clicking the button in the bottom right corner of the calibre main window that has an icon of a grid on it. In order to test the new backend, do the following steps: WARNING: Only do the following if you have backed up your calibre libraries and are willing to encounter and help debug bugs. Update to calibre 0.9.44 Go to Preferences->Tweaks and click the button "Plugin Tweaks" in the bottom left corner In the text box, type the following Code: use_new_db = True Click Apply Restart calibre You should now see [newdb] in the status bar at the bottom of the main calibre window. This indicates you are successfully running with the new db backend. You can switch back to the old backend at any time by simply changing that tweak to False and restarting calibre. The database backend in calibre is an extremely complex bit of code, so while there is an extensive test suite for the new backend and I have been running it myself for a couple of weeks, there will be bugs. So please backup your calibre libraries before embarking on this adventure Some details on the performance improvements: The startup time for calibre with my 22,000 book test library has gone down from 11.5 to 6.4 seconds which is almost twice as fast. The time taken to refresh the Tag Browser after an edit is also greatly reduced. Searches are cached, so if you do a search (or use a virtual library) you should notice a speedup when re-doing the same search or re-applying the same virtual library The code path for evaluating composite columns (based on templates) has been greatly optimized A note for plugin developers: I have tried hard to maintain API compatibility between the old and new backends, so hopefully most plugins should continue to work, without modification. However, given the complexity of the code involved there may well be regressions, if so I will be happy to help with fixing them. Attached Thumbnails Last edited by kovidgoyal; 08-17-2013 at 07:29 AM.