New database backend - testers needed

kovidgoyal · 07-28-2013, 12:56 AM

Hi all,

After a year and a half of development, the new calibre database backend is finally ready for testing.

The major goals of this new backend are:

Improve performance
Be thread safe (thereby eventually enabling multi-user calibre)
Maintain compatibility of database format with the previous backend. This means that you can freely switch between the old and new backends.
Also, there is a new cover grid based on the new backend, see attached screenshot. You can turn it on by clicking the button in the bottom right corner of the calibre main window that has an icon of a grid on it.

In order to test the new backend, do the following steps:

WARNING: Only do the following if you have backed up your calibre libraries and are willing to encounter and help debug bugs.

Update to calibre 0.9.44
Go to Preferences->Tweaks and click the button "Plugin Tweaks" in the bottom left corner
In the text box, type the following
Code:
```
use_new_db = True
```
Click Apply
Restart calibre

You should now see [newdb] in the status bar at the bottom of the main calibre window. This indicates you are successfully running with the new db backend.

You can switch back to the old backend at any time by simply changing that tweak to False and restarting calibre.

The database backend in calibre is an extremely complex bit of code, so while there is an extensive test suite for the new backend and I have been running it myself for a couple of weeks, there will be bugs. So please backup your calibre libraries before embarking on this adventure

Some details on the performance improvements:

The startup time for calibre with my 22,000 book test library has gone down from 11.5 to 6.4 seconds which is almost twice as fast.
The time taken to refresh the Tag Browser after an edit is also greatly reduced.
Searches are cached, so if you do a search (or use a virtual library) you should notice a speedup when re-doing the same search or re-applying the same virtual library
The code path for evaluating composite columns (based on templates) has been greatly optimized

A note for plugin developers: I have tried hard to maintain API compatibility between the old and new backends, so hopefully most plugins should continue to work, without modification. However, given the complexity of the code involved there may well be regressions, if so I will be happy to help with fixing them.

PeterT · 07-28-2013, 01:24 AM

Just trying. Connected my Kobo with the new database and received this error:

Spoiler:

PeterT · 07-28-2013, 01:28 AM

Some functions seem a lot slower; for instance DavidFor's Kobo Utilities / store location is crawling as it updates the database. I will do more benchmarking tomorrow but it is glacially slow compared to what it was.

* Updating 170 books with readling locations, the storing of data took close to 5 minutes.

* Reverting to old DB engine; storing of data was less than 30 seconds

kovidgoyal · 07-28-2013, 01:48 AM

The plugin should be changed to use db.has_id() rather than db.data.has_id() (the API of the db object is almost fully ported, db.data, not so much) I have added has_id() to the new backend db.data API, so that issue should go away with the next build.

As for the speed issues, it's hard for me to comment as I dont have akobo to test with. If you stick some print statements intot he driver to measure execution speed to find out what is slower i can help with suggesting ways to make it faster.

BetterRed · 07-28-2013, 02:21 AM

will do what I can

BR

davidfor · 07-28-2013, 02:37 AM

Quote:

Originally Posted by PeterT

Some functions seem a lot slower; for instance DavidFor's Kobo Utilities / store location is crawling as it updates the database. I will do more benchmarking tomorrow but it is glacially slow compared to what it was.

* Updating 170 books with readling locations, the storing of data took close to 5 minutes.

* Reverting to old DB engine; storing of data was less than 30 seconds

Were you running it in the foreground or background version? The two ways use different ways to update the library.

PeterT · 07-28-2013, 06:55 AM

Quote:

Originally Posted by davidfor

Were you running it in the foreground or background version? The two ways use different ways to update the library.

Background methid

chaley · 07-28-2013, 07:53 AM

If you are using the new DB then you also have access to a new template function that returns a list of virtual libraries that a book is in. The function is virtual_libraries(). A (template program mode) template that uses it is

Code:

{0:'virtual_libraries()'}

The performance of this function should be adequate if you don't have "lots" of virtual libraries. One question: is this true? Second question: how many is "lots"? I think that "lots" is more than 10 or so, and is certainly more than 50.

Note that if you sort on this column then calibre must compute the values for every book in the db, which might be slow for large libraries, although with the composite column evaluation optimizations in the new DB this effect might be not as noticeable. If you don't sort on it then the values are computed when displayed.

This function is not available in the old DB.

mbovenka · 07-28-2013, 08:05 AM

Startup time old DB: 1:30
Startup time new DB: 36 seconds(!)

For me it's almost 3 times faster...

mbovenka · 07-28-2013, 08:36 AM

Bad news: the Kobo Touch Extended driver doesn't seem to like the new DB.When I connect my Aura, Calibre detects it, but never completes the scanning job and I have to kill it using Task Manager. Switching back to the old DB and all is well.

Same with the Kindle driver; connecting my DX gets it recognized, but the device job never ends and killing it with Task Manager is all that I can do.

This is on WinXP Pro 32-bit.

jgoguen · 07-28-2013, 10:23 AM

Can you open calibre in debug mode, try this again, and open a bug report at https://bugs.launchpad.net/calibre-kobo-driver with all the details?

I will also try the new backend myself

EDIT: I tried the new backend with my Glo and calibre went from detecting to scanning to done in 3 seconds flat. Now, I'm running 64-bit calibre from a RPM package I built myself from source, but that shouldn't make this much of a difference. Perhaps this is yet another Aura-specific bug... looking forward to getting that debug log from you

kovidgoyal · 07-28-2013, 10:33 AM

My Kindle PW works with the new db, can you post the debug log (start calibre in debug mode by right clicking the preferences button).

kovidgoyal · 07-28-2013, 11:08 AM

Also, how many books do you have on your device, it may be that some code path used by the device drivers is slower in the new backend and I never noticed as I dont tend to put a lot of books on my devices.

mbovenka · 07-28-2013, 11:34 AM

Quote:

Originally Posted by kovidgoyal

Also, how many books do you have on your device, it may be that some code path used by the device drivers is slower in the new backend and I never noticed as I dont tend to put a lot of books on my devices.

This is the debug log from startup -> connect Aura (with 67 books on it):

Spoiler:

I left it for 15 minutes or so on an earlier try, so it's not just slow. Calibre eats one full core of CPU at that point (50% CPU on my Core2Duo.

jgoguen · 07-28-2013, 12:31 PM

Can you unplug your device, disable the KoboTouchExtended driver, enable the main KoboTouch driver, restart calibre, and see if this still happens?

07-28-2013, 12:56 AM	#1
kovidgoyal creator of calibre Posts: 43,856 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	New database backend - testers needed Hi all, After a year and a half of development, the new calibre database backend is finally ready for testing. The major goals of this new backend are: Improve performance Be thread safe (thereby eventually enabling multi-user calibre) Maintain compatibility of database format with the previous backend. This means that you can freely switch between the old and new backends. Also, there is a new cover grid based on the new backend, see attached screenshot. You can turn it on by clicking the button in the bottom right corner of the calibre main window that has an icon of a grid on it. In order to test the new backend, do the following steps: WARNING: Only do the following if you have backed up your calibre libraries and are willing to encounter and help debug bugs. Update to calibre 0.9.44 Go to Preferences->Tweaks and click the button "Plugin Tweaks" in the bottom left corner In the text box, type the following Code: use_new_db = True Click Apply Restart calibre You should now see [newdb] in the status bar at the bottom of the main calibre window. This indicates you are successfully running with the new db backend. You can switch back to the old backend at any time by simply changing that tweak to False and restarting calibre. The database backend in calibre is an extremely complex bit of code, so while there is an extensive test suite for the new backend and I have been running it myself for a couple of weeks, there will be bugs. So please backup your calibre libraries before embarking on this adventure Some details on the performance improvements: The startup time for calibre with my 22,000 book test library has gone down from 11.5 to 6.4 seconds which is almost twice as fast. The time taken to refresh the Tag Browser after an edit is also greatly reduced. Searches are cached, so if you do a search (or use a virtual library) you should notice a speedup when re-doing the same search or re-applying the same virtual library The code path for evaluating composite columns (based on templates) has been greatly optimized A note for plugin developers: I have tried hard to maintain API compatibility between the old and new backends, so hopefully most plugins should continue to work, without modification. However, given the complexity of the code involved there may well be regressions, if so I will be happy to help with fixing them. Attached Thumbnails Last edited by kovidgoyal; 08-17-2013 at 07:29 AM.

07-28-2013, 01:24 AM	#2
PeterT Grand Sorcerer Posts: 12,166 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	Just trying. Connected my Kobo with the new database and received this error: Spoiler: calibre, version 0.9.41 ERROR: Unhandled exception: <b>AttributeError</b>:'View' object has no attribute 'has_id' Traceback (most recent call last): File "calibre_plugins.reading_list.action", line 943, in _on_device_metadata_available File "calibre_plugins.reading_list.action", line 1058, in sync_now File "calibre_plugins.reading_list.action", line 1238, in _rebuild_auto_device_list File "calibre_plugins.reading_list.config", line 219, in get_book_list AttributeError: 'View' object has no attribute 'has_id'

07-28-2013, 01:28 AM	#3
PeterT Grand Sorcerer Posts: 12,166 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	Some functions seem a lot slower; for instance DavidFor's Kobo Utilities / store location is crawling as it updates the database. I will do more benchmarking tomorrow but it is glacially slow compared to what it was. * Updating 170 books with readling locations, the storing of data took close to 5 minutes. * Reverting to old DB engine; storing of data was less than 30 seconds Last edited by PeterT; 07-28-2013 at 01:36 AM.

07-28-2013, 01:48 AM	#4
kovidgoyal creator of calibre Posts: 43,856 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The plugin should be changed to use db.has_id() rather than db.data.has_id() (the API of the db object is almost fully ported, db.data, not so much) I have added has_id() to the new backend db.data API, so that issue should go away with the next build. As for the speed issues, it's hard for me to comment as I dont have akobo to test with. If you stick some print statements intot he driver to measure execution speed to find out what is slower i can help with suggesting ways to make it faster. Last edited by kovidgoyal; 07-28-2013 at 02:28 AM.

07-28-2013, 07:53 AM	#8
chaley Grand Sorcerer Posts: 11,741 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	If you are using the new DB then you also have access to a new template function that returns a list of virtual libraries that a book is in. The function is virtual_libraries(). A (template program mode) template that uses it is Code: {0:'virtual_libraries()'} The performance of this function should be adequate if you don't have "lots" of virtual libraries. One question: is this true? Second question: how many is "lots"? I think that "lots" is more than 10 or so, and is certainly more than 50. Note that if you sort on this column then calibre must compute the values for every book in the db, which might be slow for large libraries, although with the composite column evaluation optimizations in the new DB this effect might be not as noticeable. If you don't sort on it then the values are computed when displayed. This function is not available in the old DB. Last edited by chaley; 07-28-2013 at 08:23 AM. Reason: typo

07-28-2013, 02:21 AM	#5
BetterRed null operator (he/him) Posts: 20,568 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	will do what I can BR

07-28-2013, 08:05 AM	#9
mbovenka Wizard Posts: 2,018 Karma: 13471689 Join Date: Oct 2007 Location: Almere, The Netherlands Device: Kobo Sage	Startup time old DB: 1:30 Startup time new DB: 36 seconds(!) For me it's almost 3 times faster...

07-28-2013, 08:36 AM	#10
mbovenka Wizard Posts: 2,018 Karma: 13471689 Join Date: Oct 2007 Location: Almere, The Netherlands Device: Kobo Sage	Bad news: the Kobo Touch Extended driver doesn't seem to like the new DB.When I connect my Aura, Calibre detects it, but never completes the scanning job and I have to kill it using Task Manager. Switching back to the old DB and all is well. Same with the Kindle driver; connecting my DX gets it recognized, but the device job never ends and killing it with Task Manager is all that I can do. This is on WinXP Pro 32-bit. Last edited by mbovenka; 07-28-2013 at 10:27 AM.

07-28-2013, 10:23 AM	#11
jgoguen Generally Awesome Person Posts: 1,061 Karma: 2178845 Join Date: Jan 2013 Location: /dev/kmem Device: Kobo Clara HD, Kindle Oasis	Can you open calibre in debug mode, try this again, and open a bug report at https://bugs.launchpad.net/calibre-kobo-driver with all the details? I will also try the new backend myself EDIT: I tried the new backend with my Glo and calibre went from detecting to scanning to done in 3 seconds flat. Now, I'm running 64-bit calibre from a RPM package I built myself from source, but that shouldn't make this much of a difference. Perhaps this is yet another Aura-specific bug... looking forward to getting that debug log from you Last edited by jgoguen; 07-28-2013 at 10:40 AM.

07-28-2013, 10:33 AM	#12
kovidgoyal creator of calibre Posts: 43,856 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	My Kindle PW works with the new db, can you post the debug log (start calibre in debug mode by right clicking the preferences button).

07-28-2013, 11:08 AM	#13
kovidgoyal creator of calibre Posts: 43,856 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Also, how many books do you have on your device, it may be that some code path used by the device drivers is slower in the new backend and I never noticed as I dont tend to put a lot of books on my devices.

07-28-2013, 12:31 PM	#15
jgoguen Generally Awesome Person Posts: 1,061 Karma: 2178845 Join Date: Jan 2013 Location: /dev/kmem Device: Kobo Clara HD, Kindle Oasis	Can you unplug your device, disable the KoboTouchExtended driver, enable the main KoboTouch driver, restart calibre, and see if this still happens?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
MTP support: Testers needed	kovidgoyal	Calibre	68	09-21-2012 03:54 AM
shared backend database?	perler	Calibre	4	01-26-2012 05:37 AM
Windows testers needed	kovidgoyal	Calibre	51	11-21-2010 05:58 AM
OS X testers needed	kovidgoyal	Calibre	23	11-18-2010 12:11 PM
calibre ereader output testers needed	user_none	Other formats	24	05-11-2009 12:00 AM

Advert

Advert