01-14-2017, 07:46 PM | #1 |
Enthusiast
Posts: 33
Karma: 1032
Join Date: Nov 2016
Device: Kobo Aura Edition 2, Lenovo Android 10
|
Testing Calibre's limits (not a question!)
I wanted to test Calibre's limits in terms of handling an absurdly large number of books. I'm an engineer and can't really help it.
I created a new Calibre library and kept adding a set of ~1500 books again and again, ignoring duplicates, until I started seeing a noticeable slowdown in Calibre. This is by no means a typical situation, no one has that many books. Ultimately, Calibre begins to struggle around the 30,000 book mark. Now I know, and now you know too! |
01-14-2017, 08:33 PM | #2 |
null operator (he/him)
Posts: 20,579
Karma: 26954694
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
@wishingstar - but I know for fact that my experience is otherwise.
One of my libraries recently went though the 100,000 book barrier. Another similar library has ~3,500 titles, and there's no significant difference in performance between them - unless I use something like Save to Disk for the entire library. Have you considered the possibility that having every book duplicated 20 times might impact on performance? I don't know that it does, but I wouldn't be surprised if it did. BR |
Advert | |
|
01-14-2017, 08:47 PM | #3 |
Enthusiast
Posts: 33
Karma: 1032
Join Date: Nov 2016
Device: Kobo Aura Edition 2, Lenovo Android 10
|
Interesting, so you think that when you're performing database-related functions (correct me if I'm wrong about assuming this), Calibre's responsiveness is affected by how many duplicate entries it has to parse through?
Presumably when you have a huge library you have to perform a lot of searches to find specific books, that's where I saw Calibre choking and even the window title showing "Not Responding" for a few seconds before the search results show up. |
01-14-2017, 10:08 PM | #4 | ||
null operator (he/him)
Posts: 20,579
Karma: 26954694
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Quote:
Quote:
I sometimes see 'Not Responding' when I add a chunky video as a new book, but never in a search. IIRC there are custom composite column templates that can slow things down, displaying file system properties, such as size, date modified, etc for format files, come to mind. BR |
||
01-14-2017, 10:30 PM | #5 |
creator of calibre
Posts: 43,860
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
As far as I know searching is almost never a bottleneck, unless you have either
1) Very large amounts if text in some metadata (for instance I know some people who put the entire book text in the comments field) 2) Use custom columns built from other columns, this means that doing a search causes the column to be evaluated for every single book at once, instead of on demand as is normal. Both these cases can be mitigated by reducing which columns a prefixless search searches in Preferences->Searching As far as I know, on commodity hardware, calibre can handle libraries of about 50K to 100K books with reasonable performance. If you search these forums you will find many tips on improving performance for large libraries. |
Advert | |
|
01-14-2017, 10:40 PM | #6 |
null operator (he/him)
Posts: 20,579
Karma: 26954694
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
On reflection and after peeking in the library database I'm wondering if the presence of duplicates would affect search times - might affect add times.
Which file system and what sort of drive are you using - if its FAT on USB2.0 I would expect that to be considerably slower. If its a network device then you'd better read this ==>> I am getting errors with my calibre library on a networked drive/NAS? BR |
01-14-2017, 11:33 PM | #7 |
Enthusiast
Posts: 33
Karma: 1032
Join Date: Nov 2016
Device: Kobo Aura Edition 2, Lenovo Android 10
|
@BetterRed: My Library is on a USB 3.0 external drive, so that's not the issue. You are right that add times are affected by duplicates though. The custom column this is the reason though, see my response to kovidgoyal below.
@kovidgoyal: thanks for responding to my crazy thread attempting to test the limits of your program! Regarding your first suggestion, I add metadata from goodreads, so the book descriptions are not long, usually a paragraph or two at the most. Your second suggestion is exactly it though! The reason search was choking is because of custom columns built from other columns, I changed the setting you mentioned and search was snappy again, even with ~30,000 books in the library, most of which are duplicates. Amazing! Thanks for entertaining my idea and helping me find a way to improve search in general! |
Tags |
library books, library size, testing |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Calibre stopped working in Debian testing | cokowiek | Devices | 9 | 11-29-2016 08:00 PM |
calibre V0.9.41 released, includes new database backend for testing | Alexander Turcic | Calibre | 0 | 07-28-2013 02:47 AM |
[Android App] Calibre Companion 1.7 Beta testing | Adoby | Devices | 31 | 09-07-2012 10:13 PM |
iPad Testing .epub Files from Calibre on iPad | Fabe | Apple Devices | 46 | 06-08-2012 04:37 AM |
Calibre + Instapaper Limits | feelsgoodman | Calibre | 3 | 11-27-2010 02:40 AM |