Calibre database backend

[q4brk]

I've been trying to find out of a while now if there are any plans (or work in progress) for allowing Calibre to use a different database than SQLite; I didn't have much success , maybe someone can answer this for me?

I might be able/willing to help such an effort.

As a side note, for me, the database is one of the major issues of Calibre: I have a large collection of about 15000 books, *lots* of tags, authors, etc. and almost all operations involving the database are slow: searching, adding new books, e.g. right now calibre is frozen for >15 minutes after trying to delete about 1000 tags at once... (and no, my computer is not old/slow, has plenty of ram, disks defragged, etc. )

[kovidgoyal]

No, there are no plans, and I highly doubt your speed issues are anything to do with the database. Search a little on this forum and you will find plenty of tips on how to improve your performance in calibre. The most common being:

1) exclude your calibre library folder from antivirus
2) hide the tag browser
3) Dont use custom columns that are built from other columns using templates
4) Restrict what columns unprefixed searches search via preferences->Searching

Just anecdotally, on my machine adding a book to a library containing 50,000 books completes in under five seconds, and adding a book involves searching through all existing books for duplicates.

[q4brk]

Originally Posted by kovidgoyal

No, there are no plans, and I highly doubt your speed issues are anything to do with the database. Search a little on this forum and you will find plenty of tips on how to improve your performance in calibre. The most common being:

1) exclude your calibre library folder from antivirus
2) hide the tag browser
3) Dont use custom columns that are built from other columns using templates
4) Restrict what columns unprefixed searches search via preferences->Searching

Just anecdotally, on my machine adding a book to a library containing 50,000 books completes in under five seconds, and adding a book involves searching through all existing books for duplicates.

I don't know what to say... I've been using SQLite for some projects in the past and I know it wasn't designed to handle operations as efficient as a full-blown database like Postgres, for example. Not sure if that changed lately, but anyway...

• I've excluded the e-book folder from the antivirus; I'm not sure that was a problem, since my on-access scanner only scans executable files.
• My default view doesn't show any tags at all.
• Not using any custom columns at all.

My gut feeling is that the slowness simply comes from the large amount of books, authors and tags . At this moment my collection has 14689 books, 6426 authors and 5315 tags. I'm trying to clean the tags up and I delete them in batches of 100-200 using the Manage Tags feature; each time the whole GUI freezes up for 1-2 minutes (not complaining about the fact that the operation seems to take place on the GUI thread, since Calibre shows up as (not responding), but about the fact that it takes long to make changes on the database). As for imports, I import folders at once and it seems to take long for PDF files. I'll try to do some profiling, maybe I can figure out what's going on.

As a side note, any idea why Calibre is doing these operations while importing books?

[kovidgoyal]

It's not a question of showing tags, it is closing the tag browser, the entire left panel in calibre.

And trust me, your problems have nothing to do with the database. There's no way I can convince you of that in plain text, so feel free to instrument the code, calibre is open source

[kovidgoyal]

And as for deleting tags in bulk being slow, that is probably because the manage tags dialog deletes them one at a time rather than in bulk. (It was written a long time ago before the bulk manipulation APIs were available in the new calibre database backend).

[kovidgoyal]

This commit: https://github.com/kovidgoyal/calibr...1e7237fc50c454

should greatly improve the performance of deleting/renaming large numbers of items via the Manage dialog

[kovidgoyal]

Originally Posted by q4brk

As a side note, any idea why Calibre is doing these operations while importing books?

Here you go: https://github.com/kovidgoyal/calibr...bf611828c218c3

[q4brk]

Originally Posted by kovidgoyal

Here you go: https://github.com/kovidgoyal/calibr...bf611828c218c3

[lbik]

You can speedup Calibre when you put the Library on an SSD.