Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 08-18-2012, 12:29 AM   #1
SilverAce
Junior Member
SilverAce began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Aug 2012
Device: Kindle Fire
Calibre opens very slowly with large library?

I have a fairly large library with 10k+ documents. Calibre takes a LONG time to load, easily more than 10 minutes after I open it to the time I can start using it. Is this normal, and is there any way to make it faster?
SilverAce is offline   Reply With Quote
Old 08-18-2012, 04:07 AM   #2
Thasaidon
Hedge Wizard
Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.Thasaidon ought to be getting tired of karma fortunes by now.
 
Thasaidon's Avatar
 
Posts: 105
Karma: 624406
Join Date: May 2011
Location: Doncaster UK
Device: Kobo Touch, Nook Simple
This is not normal. I have an old PC (pentium 4) and with a library of over 23k books Caliber loads less than one minute.
Thasaidon is offline   Reply With Quote
 
Enthusiast
Old 08-19-2012, 01:48 AM   #3
pchrist7
Addict
pchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animalspchrist7 is kind to children and small, furry animals
 
pchrist7's Avatar
 
Posts: 385
Karma: 6514
Join Date: Aug 2010
Location: Denmark
Device: Kindle 3 3G+Wifi
Custom columns ?

Quote:
Originally Posted by SilverAce View Post
I have a fairly large library with 10k+ documents. Calibre takes a LONG time to load, easily more than 10 minutes after I open it to the time I can start using it. Is this normal, and is there any way to make it faster?
Hi SilverAce
Only have 2500 books in my library, but I did notice that adding custom columns with "instructions" to search each book entry and then do "something" definately could slow down Calibre startup.
Unfortunately that was more than a year ago, so I'm afraid I don't remember the specific details.
Have you tried starting Calibre from cmd using calibre-debug -g ?
As far as I remember this has been discussed before, with lots of tips to improve startup speed. Antivirus settings etc.
A bit more information could help us help you
Cheers, Per
pchrist7 is offline   Reply With Quote
Old 08-21-2012, 10:08 PM   #4
SilverAce
Junior Member
SilverAce began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Aug 2012
Device: Kindle Fire
Thanks for the responses. I should mention I'm running OS X 10.7.4 -- can I still start Calibre from the command line?

I have not created any special columns to search for anything...

Calibre is also pretty slow, when, for example, I add a book. It's noticeably laggy -- it takes say, 20 or 30 seconds to add it. Maybe that slowness is related?
SilverAce is offline   Reply With Quote
Old 08-22-2012, 08:40 AM   #5
atjnjk
Zealot
atjnjk will become famous soon enoughatjnjk will become famous soon enoughatjnjk will become famous soon enoughatjnjk will become famous soon enoughatjnjk will become famous soon enoughatjnjk will become famous soon enough
 
Posts: 105
Karma: 554
Join Date: Oct 2008
Device: none
Maybe the tips in this thread will help you speed calibre up.
You can use command line to start calibre easily in Windows, Linux and OS X. Note (from calibre's manual):
Quote:
On OS X you have to go to Preferences->Advanced->Miscellaneous and click install command line tools to make the command line tools available. On other platforms, just start a terminal and type the command.
atjnjk is offline   Reply With Quote
Old 08-22-2012, 09:41 AM   #6
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 6,678
Karma: 43902502
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
I've always found it easiest just to go the Preferences option menu (select the drop down arrow besides Preferences) and select the Restart in Debug mode command or use the keyboard short cut for that (Ctrl+Shift+R).
PeterT is online now   Reply With Quote
Old 08-24-2012, 06:34 PM   #7
SilverAce
Junior Member
SilverAce began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Aug 2012
Device: Kindle Fire
You know what? I strongly suspect that the slowness is due to my external hard drive, a Western Digital My Passport, being formatted as NTFS and being accessed through a third-party driver. This benchmark shows that such drivers are FAR slower than HFS+ in dealing with folders with lots of files and subfolders.

So that may be the difference right there.
SilverAce is offline   Reply With Quote
Old 08-25-2012, 01:35 AM   #8
Dopedangel
Wizard
Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.
 
Dopedangel's Avatar
 
Posts: 1,090
Karma: 8499999
Join Date: Dec 2006
Location: Singapore
Device: Coolreader(Nexus 5)\Coolreader(Nook Touch)
Quote:
Originally Posted by SilverAce View Post
You know what? I strongly suspect that the slowness is due to my external hard drive, a Western Digital My Passport, being formatted as NTFS and being accessed through a third-party driver. This benchmark shows that such drivers are FAR slower than HFS+ in dealing with folders with lots of files and subfolders.

So that may be the difference right there.
Another thing could be you might need to defrag your device. On Mac it is not needed but now that you say you are suing an ntfs portable device. I think you might need to defrag your harddrive
Dopedangel is offline   Reply With Quote
Old 08-25-2012, 05:08 AM   #9
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,113
Karma: 1904752
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by Dopedangel View Post
Another thing could be you might need to defrag your device. On Mac it is not needed but now that you say you are suing an ntfs portable device. I think you might need to defrag your harddrive
To do a defrag I think SilverAce will need to mount the drive on a Windows system. IMO defrag on NTFS has always been overrated as a solution to any problem, its a hang over from FAT and something with which to kick the can down the road - maybe Draghi should defrag the Euro

The 'drivers' from Paragon etc should be properly called called emulators. I use file system emulators on a Solaris system for NTFS and HFS+; performance of both is poor compared to native NFS over USB.

@SilverAce - if you've not done so already, I suggest you replicate the library to an HFS+ volume and point Calibre at it, if performance is OK then the problem is almost certainly the use of NTFS, although it could also be a bad drive, there may be OS/X utils available from WD to 'measure' drive health.

If you have access to a 'friendly' windows box, you could put Calibre Portable on it, and point it at the library on your WD USB drive - if its performance is acceptable then its 99.9 % sure its the NTFS on a Mac issue, if its slow then you may have a bad drive - WD have Windows utils for measuring drive health.

On reflection the data is being written to the NTFS volume via one of these so-called 'drivers', so a defrag may have a bigger effect than I would normally expect. Who knows how they manage the drive space.

BR

Last edited by BetterRed; 08-25-2012 at 05:10 AM.
BetterRed is offline   Reply With Quote
Old 09-18-2012, 04:51 AM   #10
electronicfur
Enthusiast
electronicfur is on a distinguished road
 
Posts: 26
Karma: 50
Join Date: Aug 2011
Device: Samsung Note FBReader, N900 FBReader, Nook Simple Touch (Kindle3 died)
SilverAce,

I am also having a slow startup problem, and I've so far traced the issue to Calibre reading the attributes of all the books in my library on startup.

I dont yet know why this is happening. I've even got this happening if I use the Calibre portable version and point it at my library, so I know it's not a plugin causing the issue.

I used Process Monitor to discover what Calibre was doing on startup. See

http://technet.microsoft.com/en-us/s.../bb896645.aspx

Process Monitor for me shows lots of CreateFile, QueryNetwork, CloseFile entries, and when I look at the details it is requesting the Read Attributes of the files. It seems to do this for all books, so my startup is extremely slow.

I dont use any composite columns, so not sure why this is happening. I only have one custom Genre column. And although I have excluded the calibre directories, I've completely disabled my virus scanner whilst testing this to be sure it is not the cause.

Last edited by electronicfur; 09-18-2012 at 04:54 AM.
electronicfur is offline   Reply With Quote
Old 09-18-2012, 06:55 AM   #11
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,043
Karma: 3824376
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
If you have custom columns that depends on other columns, then I believe all values have to be initiated when you start Calibre. I have my metadata.db on a SSD drive and still noticed this very clearly. When I deleted those columns startup was fast again.

I have my metadata.db on SSD but the Calibre library on a NAS. I use a symlink in the Calibre library to the real file on the SSD, to speed up things. Calibre became a little bit more responsive that way.

Make sure you have enough memory in the computer.

You say you have no custom columns, so that may not be the problem, unless they are hidden?

I don't believe Calibre even access the actual books when starting. Only metadata.db.

Last edited by Adoby; 09-18-2012 at 07:09 AM.
Adoby is offline   Reply With Quote
Old 09-18-2012, 07:32 AM   #12
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,258
Karma: 821512
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by Adoby View Post
If you have custom columns that depends on other columns, then I believe all values have to be initiated when you start Calibre. I have my metadata.db on a SSD drive and still noticed this very clearly. When I deleted those columns startup was fast again.
Not quite. The columns are evaluated only when displayed. Sorting on a column is equivalent to displaying all values for that column.
Quote:
I don't believe Calibre even access the actual books when starting. Only metadata.db.
Several versions ago calibre did look at attributes of books (modtime, etc) when starting. This was changed. I don't remember the version, but it was a couple of months ago at least. The current version does not look at book attributes unless you have a custom column that contains them (a "formats" column).
chaley is offline   Reply With Quote
Old 09-18-2012, 09:15 AM   #13
electronicfur
Enthusiast
electronicfur is on a distinguished road
 
Posts: 26
Karma: 50
Join Date: Aug 2011
Device: Samsung Note FBReader, N900 FBReader, Nook Simple Touch (Kindle3 died)
Quote:
Originally Posted by chaley View Post
Several versions ago calibre did look at attributes of books (modtime, etc) when starting. This was changed. I don't remember the version, but it was a couple of months ago at least. The current version does not look at book attributes unless you have a custom column that contains them (a "formats" column).
I'm using the latest calibre 0.8.69 (portable to avoid any plugin issues) and it does seem to be reading attributes of all books in my library.

In order to test further I created a subset of my large > 10k book library. I need a subset to test because it's so long to open otherwise.

If I copy a subset of the library directory (eg only some of the books and the metadata.db), switch to the library, delete the one custom Genre column (just to be sure), and do a Database Maintenance->Restore Library, then when opening the resulting library, Calibre still read the attributes of all the books.

If however I create a new empty library, and then import the same subset of books into it, then when switching to that library, Calibre does not read the attributes of all the books, it only reads the first book attributes, and the next 6 book cover files.

So if I can figure out what the difference between the two test libraries is, I can work out where the issue lies. Does the first way of creating the subset test library still retain the preferences of the original library that are stored in the metadata.db? I'm guessing it does, and something in the preferences is causing Calibre to read all files.

Cheers,
EF
electronicfur is offline   Reply With Quote
Old 09-18-2012, 09:36 AM   #14
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,809
Karma: 12535517
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by electronicfur View Post
So if I can figure out what the difference between the two test libraries is, I can work out where the issue lies.
I'm curious, do the two libraries have the same set of columns listed in preferences? I attached the default set of columns, does your troubled library have the same columns?
Attached Thumbnails
Click image for larger version

Name:	default_columns.jpg
Views:	96
Size:	93.3 KB
ID:	92546  
DoctorOhh is offline   Reply With Quote
Old 09-18-2012, 09:36 AM   #15
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,258
Karma: 821512
Join Date: Jan 2010
Location: France
Device: Many android devices
The only code I can find in calibre that looks at the attributes of a format file is called when a template referring to a format is evaluated. Are you certain that you don't have a hidden composite column, and in particular a "formats" column? Or perhaps you have installed a plugin that walks through your books?

Copying a library folder will keep many of the preferences, and in particular all the custom columns, hidden or not. There are a large number of preferences stored outside the db.
chaley is offline   Reply With Quote
Reply

Tags
open slow

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre Portable - Which Library Opens? nynaevelan Calibre 5 09-21-2011 12:58 PM
Best way to get a large Calibre library into the PE library? Filark enTourage Archive 0 04-20-2011 10:18 PM
Kindle and Calibre user with problem importing large library into Calibre pleabargain Calibre 1 12-07-2010 10:19 AM
Large library/calibre interface StanByk Calibre 12 10-19-2009 10:20 AM
How large is your Calibre library? radius Calibre 28 10-12-2009 01:14 AM


All times are GMT -4. The time now is 10:16 AM.


MobileRead.com is a privately owned, operated and funded community.