Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 04-12-2011, 04:43 PM   #1
Doug-W
Member
Doug-W began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Feb 2011
Device: Nook
Profiling Large Libraries

I have a large library (39,000 entries over 12,000 Authors) That is pretty much unusable due to how long it takes to do anything. Is there any hooks to run cProfile against it and see where it's spending it's time?

One thought outright is over 12,000 entries just doing an fstat can take a very long time. When I used to own an ISP we wrote some custom Linux kernel hacks just to avoid the inode lookup on the news server. Is there any way to have a module specify the way to convert a book entry into the path? IE: rather than being library root/Aadam Johnson/His First Book (ID) it may help on large libraries to go library root/Aa/Aadam Johnson/His First Book (ID) so that the initial lookups are from a directory with ~676 max entries and not the 12,000 I have now.
Doug-W is offline   Reply With Quote
Old 04-12-2011, 04:50 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,771
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Hide the Tag browser.
kovidgoyal is offline   Reply With Quote
Advert
Old 04-13-2011, 01:01 AM   #3
Doug-W
Member
Doug-W began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Feb 2011
Device: Nook
Quote:
Originally Posted by kovidgoyal View Post
Hide the Tag browser.
Tag browser hidden, 39270 books going from filter of 2720 to filter of 2792 books took 70 seconds. Filter was y/n field and a user category made up of authors. (@PaperAuthors:True OR #gr_read:True) If I apply the same search as the filter itself (IE: Every book in the filter should match) it took 50 seconds once the filter was applied.

Any way to profile what that 70 seconds was spent doing? I thought maybe it would be creating/destroying (or adding/removing) the books from the grid view, that took a long time, so I tested the filtered view instead and had the same issues.

The machine is a Duo Core 3Ghz 6 gigs memory, windows 7 64 bit, calibre 0.7.54.
Doug-W is offline   Reply With Quote
Old 04-13-2011, 01:44 AM   #4
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
I'm curious is the library on a local drive or external on the network or a USB drive?
DoctorOhh is offline   Reply With Quote
Old 04-13-2011, 06:08 AM   #5
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,703
Karma: 6658935
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
I suspect you are seeing the same issue as was discussed in ticket https://bugs.launchpad.net/calibre/+bug/757445.

The improvements will be in the next release, or you can run from source to get them now.
chaley is offline   Reply With Quote
Advert
Old 04-13-2011, 10:06 AM   #6
Doug-W
Member
Doug-W began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Feb 2011
Device: Nook
dwanthny: Local drive 2 TB 7200 RPM...


Quote:
Originally Posted by chaley View Post
I suspect you are seeing the same issue as was discussed in ticket https://bugs.launchpad.net/calibre/+bug/757445.

The improvements will be in the next release, or you can run from source to get them now.
That's an intriguing possibility, I do have an ISBN column displayed.

Which I guess just goes back to my original question, is there an easy way to get cProfile on the main code to find these type of things? I saw that some of the support tools have cProfile already set up, but didn't see it in the main gui2.
Doug-W is offline   Reply With Quote
Old 04-13-2011, 10:47 AM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,771
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Running calibre from source is trivial. Just do it and import cProfile wherever you like.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
UK libraries feenix1363 General Discussions 96 05-06-2012 07:26 AM
For those with large (500+) libraries... ficbot General Discussions 60 12-13-2011 01:51 PM
Multiple Libraries Possible? KeithO Calibre 38 02-19-2010 10:35 AM
Performance with large Libraries? itimpi Calibre 3 12-14-2008 02:46 PM
Where are there ebook libraries?? slayda News 29 10-10-2008 03:05 PM


All times are GMT -4. The time now is 07:50 AM.


MobileRead.com is a privately owned, operated and funded community.