04-12-2011, 04:43 PM | #1 |
Member
Posts: 18
Karma: 10
Join Date: Feb 2011
Device: Nook
|
Profiling Large Libraries
I have a large library (39,000 entries over 12,000 Authors) That is pretty much unusable due to how long it takes to do anything. Is there any hooks to run cProfile against it and see where it's spending it's time?
One thought outright is over 12,000 entries just doing an fstat can take a very long time. When I used to own an ISP we wrote some custom Linux kernel hacks just to avoid the inode lookup on the news server. Is there any way to have a module specify the way to convert a book entry into the path? IE: rather than being library root/Aadam Johnson/His First Book (ID) it may help on large libraries to go library root/Aa/Aadam Johnson/His First Book (ID) so that the initial lookups are from a directory with ~676 max entries and not the 12,000 I have now. |
04-12-2011, 04:50 PM | #2 |
creator of calibre
Posts: 44,337
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Hide the Tag browser.
|
Advert | |
|
04-13-2011, 01:01 AM | #3 |
Member
Posts: 18
Karma: 10
Join Date: Feb 2011
Device: Nook
|
Tag browser hidden, 39270 books going from filter of 2720 to filter of 2792 books took 70 seconds. Filter was y/n field and a user category made up of authors. (@PaperAuthors:True OR #gr_read:True) If I apply the same search as the filter itself (IE: Every book in the filter should match) it took 50 seconds once the filter was applied.
Any way to profile what that 70 seconds was spent doing? I thought maybe it would be creating/destroying (or adding/removing) the books from the grid view, that took a long time, so I tested the filtered view instead and had the same issues. The machine is a Duo Core 3Ghz 6 gigs memory, windows 7 64 bit, calibre 0.7.54. |
04-13-2011, 01:44 AM | #4 |
US Navy, Retired
Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
I'm curious is the library on a local drive or external on the network or a USB drive?
|
04-13-2011, 06:08 AM | #5 |
Grand Sorcerer
Posts: 11,939
Karma: 7219261
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
I suspect you are seeing the same issue as was discussed in ticket https://bugs.launchpad.net/calibre/+bug/757445.
The improvements will be in the next release, or you can run from source to get them now. |
Advert | |
|
04-13-2011, 10:06 AM | #6 | |
Member
Posts: 18
Karma: 10
Join Date: Feb 2011
Device: Nook
|
dwanthny: Local drive 2 TB 7200 RPM...
Quote:
Which I guess just goes back to my original question, is there an easy way to get cProfile on the main code to find these type of things? I saw that some of the support tools have cProfile already set up, but didn't see it in the main gui2. |
|
04-13-2011, 10:47 AM | #7 |
creator of calibre
Posts: 44,337
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Running calibre from source is trivial. Just do it and import cProfile wherever you like.
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
UK libraries | feenix1363 | General Discussions | 96 | 05-06-2012 07:26 AM |
For those with large (500+) libraries... | ficbot | General Discussions | 60 | 12-13-2011 01:51 PM |
Multiple Libraries Possible? | KeithO | Calibre | 38 | 02-19-2010 10:35 AM |
Performance with large Libraries? | itimpi | Calibre | 3 | 12-14-2008 02:46 PM |
Where are there ebook libraries?? | slayda | News | 29 | 10-10-2008 03:05 PM |