Testing Calibre's limits (not a question!)

[wishingstar]

I wanted to test Calibre's limits in terms of handling an absurdly large number of books. I'm an engineer and can't really help it.

I created a new Calibre library and kept adding a set of ~1500 books again and again, ignoring duplicates, until I started seeing a noticeable slowdown in Calibre. This is by no means a typical situation, no one has that many books. Ultimately, Calibre begins to struggle around the 30,000 book mark.

Now I know, and now you know too!

[BetterRed]

@wishingstar - but I know for fact that my experience is otherwise.

One of my libraries recently went though the 100,000 book barrier. Another similar library has ~3,500 titles, and there's no significant difference in performance between them - unless I use something like Save to Disk for the entire library.

Have you considered the possibility that having every book duplicated 20 times might impact on performance? I don't know that it does, but I wouldn't be surprised if it did.

BR

[wishingstar]

Interesting, so you think that when you're performing database-related functions (correct me if I'm wrong about assuming this), Calibre's responsiveness is affected by how many duplicate entries it has to parse through?

Presumably when you have a huge library you have to perform a lot of searches to find specific books, that's where I saw Calibre choking and even the window title showing "Not Responding" for a few seconds before the search results show up.

[BetterRed]

Originally Posted by wishingstar

Interesting, so you think that when you're performing database-related functions (correct me if I'm wrong about assuming this), Calibre's responsiveness is affected by how many duplicate entries it has to parse through?

As I said before - "I don't know that it does, but I wouldn't be surprised if it did." The only person who can say for certain is Kovid Goyal, or maybe chaley - I'm sure one of them will drop by in the next 24 hours.

Originally Posted by wishingstar

Presumably when you have a huge library you have to perform a lot of searches to find specific books, that's where I saw Calibre choking and even the window title showing "Not Responding" for a few seconds before the search results show up.

I just did a search for 'after' in the 8,533 book library I happen to have open - 31 books of 8,533 were found more or less instantly - all bar one had 'after' in the title, the exception had it in a long text custom column. I did three similar searches where I knew I'd have more matches, 700+, 140+ and 4,500+ (a publisher name) - all three gave a similar response. The library is on a WD 2TB Caviar Black drive.

I sometimes see 'Not Responding' when I add a chunky video as a new book, but never in a search. IIRC there are custom composite column templates that can slow things down, displaying file system properties, such as size, date modified, etc for format files, come to mind.

BR

[kovidgoyal]

As far as I know searching is almost never a bottleneck, unless you have either

1) Very large amounts if text in some metadata (for instance I know some people who put the entire book text in the comments field)

2) Use custom columns built from other columns, this means that doing a search causes the column to be evaluated for every single book at once, instead of on demand as is normal.

Both these cases can be mitigated by reducing which columns a prefixless search searches in Preferences->Searching

As far as I know, on commodity hardware, calibre can handle libraries of about 50K to 100K books with reasonable performance. If you search these forums you will find many tips on improving performance for large libraries.

[BetterRed]

On reflection and after peeking in the library database I'm wondering if the presence of duplicates would affect search times - might affect add times.

Which file system and what sort of drive are you using - if its FAT on USB2.0 I would expect that to be considerably slower. If its a network device then you'd better read this ==>> I am getting errors with my calibre library on a networked drive/NAS?

BR

[wishingstar]

@BetterRed: My Library is on a USB 3.0 external drive, so that's not the issue. You are right that add times are affected by duplicates though. The custom column this is the reason though, see my response to kovidgoyal below.

@kovidgoyal: thanks for responding to my crazy thread attempting to test the limits of your program!
Regarding your first suggestion, I add metadata from goodreads, so the book descriptions are not long, usually a paragraph or two at the most.
Your second suggestion is exactly it though! The reason search was choking is because of custom columns built from other columns, I changed the setting you mentioned and search was snappy again, even with ~30,000 books in the library, most of which are duplicates. Amazing!

Thanks for entertaining my idea and helping me find a way to improve search in general!