New database backend - testers needed

[GRiker]

Originally Posted by kovidgoyal

A note for plugin developers: I have tried hard to maintain API compatibility between the old and new backends, so hopefully most plugins should continue to work, without modification. However, given the complexity of the code involved there may well be regressions, if so I will be happy to help with fixing them.

Kovid,

I have a plugin which fetches .ondevice_col from the Metadata object for a given calibre id. It appears that ondevice_col is no longer available in the object. Is there some other way for me to find that property using the new db?

G

[kovidgoyal]

mi._proxy_metadata.ondevice_col this will work with both the olddb and the newdb from 0.9.41 onwards.

[kovidgoyal]

The new builds with the fixes are available here: http://calibre-ebook.com/downloads/betas/

The version number remains the same, just install as normal. For linux binary users, the way to install is to go to /opt/calibre (or wherever you chose to install calibre) and untar the appropriate (32 bit/64 bit) tarball.

[CRussel]

Kovid -- when I enabled the new_db, and restarted, it appeared to uninstall other plugins? I had to reload the official ones. (OTOH, the unofficial filetype plugin I run had no issues.)

Overall, no issues other than this yet. Load times are WAY better. I don't have timings, but it's at least an order of magnitude better. (This on a slow, single core, 64bit machine running Win7 64bit.) I'll try to get some timings on it later.

[gbm]

Originally Posted by kovidgoyal

The new builds with the fixes are available here: http://calibre-ebook.com/downloads/betas/

The version number remains the same, just install as normal. For linux binary users, the way to install is to go to /opt/calibre (or wherever you chose to install calibre) and untar the appropriate (32 bit/64 bit) tarball.

Installed useing:

sudo rm -rf /opt/calibre/* && sudo tar xvjf /replace_with_your_download_path/calibre-0.9.41-i686.tar.bz2 -C /opt/calibre

works.

Spoiler:

with old backend:

calibre 0.9.41  isfrozen: True is64bit: False
Linux-3.2.0-51-generic-i686-with-debian-wheezy-sid Linux ('32bit', 'ELF')
('Linux', '3.2.0-51-generic', '#77-Ubuntu SMP Wed Jul 24 20:21:10 UTC 2013')
Python 2.7.5
Linux: ('debian', 'wheezy/sid', '')
bernie@gbmoore:~$ Starting up...

macmenuhack file_path:/home/bernie/.config/calibre/plugins/fanfictiondownloader_macmenuhack.txt
Started up in 3.53 seconds with 2527 books

with new:

calibre 0.9.41  isfrozen: True is64bit: False
Linux-3.2.0-51-generic-i686-with-debian-wheezy-sid Linux ('32bit', 'ELF')
('Linux', '3.2.0-51-generic', '#77-Ubuntu SMP Wed Jul 24 20:21:10 UTC 2013')
Python 2.7.5
Linux: ('debian', 'wheezy/sid', '')
Starting up...

macmenuhack file_path:/home/bernie/.config/calibre/plugins/fanfictiondownloader_macmenuhack.txt
Started up in 2.74 seconds with 2527 books

bernie

System Details:

Ubuntu 12.04 LTS 32 bit
memory 2gb
processor: Intel® Core™2 CPU 6600 @ 2.40GHz × 2
graphics: Intel® 965G x86/MMX/SSE2

[kovidgoyal]

@gbm: Glad to hear it

@CRussel: As far as I know the new db should have no effect on plugins at all, they are not stored in the db. You can get startup time by running

calibre-debug -g

which will print the time taken to startup calibre.

[CRussel]

On a fast machine (NOT that ancient Acer Ferrari laptop), startup time is 1.35 seconds with new, 1.83 seconds with old. This on 2core/4 logical processor HP laptop with 16 GB of RAM and an SSD, connecting to a shared network mounted Calibre database.

On the plugins not showing after restart. I wonder if it's related to updating the Count Pages plugin in the same pass as enabling the new db. Since that was the only one that showed after I restarted? I did not see that behaviour on the fast machine where I didn't have any that needed updating.

[CRussel]

On slow machine (Acer Ferrari laptop, 5400 RPM HD, 2 GB RAM, 64-bit Win7), I get 3.84 seconds on old db, 2.93 on new db. But both feel faster than I usually see, which probably means that the db is cached on the Windows Storage Server at this point and being served up from RAM.

[Divingduck]

Kovid, I made some test with Windows Vista 64bit and a PRS-650. No problems up to now (did only normal things for testing). The feeling is a bit faster in opening Calibre, browsing in book covers.

Creating Catalogs seems to be approximately 8 to 10% faster


Edit: Saved searches are not working.

Edit once more: My mistake, I had change the GUI-language and forgot to reverse this.

[mbovenka]

Originally Posted by kovidgoyal

The new builds with the fixes are available here: http://calibre-ebook.com/downloads/betas/

The version number remains the same, just install as normal. For linux binary users, the way to install is to go to /opt/calibre (or wherever you chose to install calibre) and untar the appropriate (32 bit/64 bit) tarball.

YAY! I don't know what it was you fixed, but with the beta my problem (Calibre hanging with 100% CPU use on one core when connecting a device) is gone too.

[BetterRed]

Custom Column Dates are showing blank cell when date is Undefined
Find Duplicates error is gone, Right Arrow is working

BR

[mbovenka]

Sorting seems buggered with the new backend. If I sort, say, on series and then on author, normally I end up with my book list sorted on author and within the author on series (this is my startup sort)

What happens now is that I end up with my last sort (author, in this case) and within that sorted on the date the book was added to the library.

[theducks]

Confirming on XP

Sort on date or series
Sort on Title
Sort on Authors

Titles for an author should be alphabetical, they are {Date} ordered


My tweak: maximum_resort_levels = 3

[PeterT]

Originally Posted by davidfor

Peter, I'm not seeing this here. On my Touch, I have about 1100 books. The calibre library has 447 book of which 127 are on the Touch and have a reading postition. Selecting all the books in the library and doing a store took 31 seconds old DB engine and 37 with the new.

David; I just retried my tests using the current "beta" version of calibre.

Both tests done selecting 184 books (all the calibre managed material on the Kobo).

Old DB: Batch Mode: Updating metadata stage 00:13 seconds
New DB: Batch Mode: Updating metadata stage 2:49 seconds

Now I wonder if it's significant how I have things configured.

Current Reading Location:	#koboreadloc		Text, column shown in the tag browser
Percent Read Column:		#koboreadpct		Integers
Rating Column:			#koborating		Integers
Last Read Column:		#kobolastread		Date

Other related custom columns
#readstatus:						Column built from other columns
{#koboreadpct:'cmp($, 100, cmp($, 0, '', '', 'Reading'), 'Finished', '')'}
							Sort/Search column by Text
							Show in Tags browser

[kovidgoyal]

@davidfor: From Peter's description it looks to me like updating the database for a large number of books is slower with the new backend in your plugin.

This is because the new backend operates in auto-commit mode. That is after every change, the database is committed. So if you are making changes to a lot of database records, you should do them in bulk. For example, to change the the #xyz field in for a number of records, in the old db you would do:

for book_id in book_ids:
   db.set_custom(book_id, val, label='xyz', commit=False)
db.commit()

With the new db you should instead do:

if hasattr(db, 'new_api'):
   db.new_api.set_field('#xyz', {book_id:val for book_id in book_ids})
else:
    # do it the old way