Calibre opens very slowly with large library?

SilverAce · 08-18-2012, 12:29 AM

I have a fairly large library with 10k+ documents. Calibre takes a LONG time to load, easily more than 10 minutes after I open it to the time I can start using it. Is this normal, and is there any way to make it faster?

Thasaidon · 08-18-2012, 04:07 AM

This is not normal. I have an old PC (pentium 4) and with a library of over 23k books Caliber loads less than one minute.

pchrist7 · 08-19-2012, 01:48 AM

Quote:

Originally Posted by SilverAce

I have a fairly large library with 10k+ documents. Calibre takes a LONG time to load, easily more than 10 minutes after I open it to the time I can start using it. Is this normal, and is there any way to make it faster?

Hi SilverAce
Only have 2500 books in my library, but I did notice that adding custom columns with "instructions" to search each book entry and then do "something" definately could slow down Calibre startup.
Unfortunately that was more than a year ago, so I'm afraid I don't remember the specific details.
Have you tried starting Calibre from cmd using calibre-debug -g ?
As far as I remember this has been discussed before, with lots of tips to improve startup speed. Antivirus settings etc.
A bit more information could help us help you

Cheers, Per

SilverAce · 08-21-2012, 10:08 PM

Thanks for the responses. I should mention I'm running OS X 10.7.4 -- can I still start Calibre from the command line?

I have not created any special columns to search for anything...

Calibre is also pretty slow, when, for example, I add a book. It's noticeably laggy -- it takes say, 20 or 30 seconds to add it. Maybe that slowness is related?

atjnjk · 08-22-2012, 08:40 AM

Maybe the tips in this thread will help you speed calibre up.
You can use command line to start calibre easily in Windows, Linux and OS X. Note (from calibre's manual):

Quote:

On OS X you have to go to Preferences->Advanced->Miscellaneous and click install command line tools to make the command line tools available. On other platforms, just start a terminal and type the command.

PeterT · 08-22-2012, 09:41 AM

I've always found it easiest just to go the Preferences option menu (select the drop down arrow besides Preferences) and select the Restart in Debug mode command or use the keyboard short cut for that (Ctrl+Shift+R).

SilverAce · 08-24-2012, 06:34 PM

You know what? I strongly suspect that the slowness is due to my external hard drive, a Western Digital My Passport, being formatted as NTFS and being accessed through a third-party driver. This benchmark shows that such drivers are FAR slower than HFS+ in dealing with folders with lots of files and subfolders.

So that may be the difference right there.

Dopedangel · 08-25-2012, 01:35 AM

Quote:

Originally Posted by SilverAce

You know what? I strongly suspect that the slowness is due to my external hard drive, a Western Digital My Passport, being formatted as NTFS and being accessed through a third-party driver. This benchmark shows that such drivers are FAR slower than HFS+ in dealing with folders with lots of files and subfolders.

So that may be the difference right there.

Another thing could be you might need to defrag your device. On Mac it is not needed but now that you say you are suing an ntfs portable device. I think you might need to defrag your harddrive

BetterRed · 08-25-2012, 05:08 AM

Quote:

Originally Posted by Dopedangel

Another thing could be you might need to defrag your device. On Mac it is not needed but now that you say you are suing an ntfs portable device. I think you might need to defrag your harddrive

To do a defrag I think SilverAce will need to mount the drive on a Windows system. IMO defrag on NTFS has always been overrated as a solution to any problem, its a hang over from FAT and something with which to kick the can down the road - maybe Draghi should defrag the Euro

The 'drivers' from Paragon etc should be properly called called emulators. I use file system emulators on a Solaris system for NTFS and HFS+; performance of both is poor compared to native NFS over USB.

@SilverAce - if you've not done so already, I suggest you replicate the library to an HFS+ volume and point Calibre at it, if performance is OK then the problem is almost certainly the use of NTFS, although it could also be a bad drive, there may be OS/X utils available from WD to 'measure' drive health.

If you have access to a 'friendly' windows box, you could put Calibre Portable on it, and point it at the library on your WD USB drive - if its performance is acceptable then its 99.9 % sure its the NTFS on a Mac issue, if its slow then you may have a bad drive - WD have Windows utils for measuring drive health.

On reflection the data is being written to the NTFS volume via one of these so-called 'drivers', so a defrag may have a bigger effect than I would normally expect. Who knows how they manage the drive space.

BR

electronicfur · 09-18-2012, 04:51 AM

SilverAce,

I am also having a slow startup problem, and I've so far traced the issue to Calibre reading the attributes of all the books in my library on startup.

I dont yet know why this is happening. I've even got this happening if I use the Calibre portable version and point it at my library, so I know it's not a plugin causing the issue.

I used Process Monitor to discover what Calibre was doing on startup. See

http://technet.microsoft.com/en-us/s.../bb896645.aspx

Process Monitor for me shows lots of CreateFile, QueryNetwork, CloseFile entries, and when I look at the details it is requesting the Read Attributes of the files. It seems to do this for all books, so my startup is extremely slow.

I dont use any composite columns, so not sure why this is happening. I only have one custom Genre column. And although I have excluded the calibre directories, I've completely disabled my virus scanner whilst testing this to be sure it is not the cause.

Adoby · 09-18-2012, 06:55 AM

If you have custom columns that depends on other columns, then I believe all values have to be initiated when you start Calibre. I have my metadata.db on a SSD drive and still noticed this very clearly. When I deleted those columns startup was fast again.

I have my metadata.db on SSD but the Calibre library on a NAS. I use a symlink in the Calibre library to the real file on the SSD, to speed up things. Calibre became a little bit more responsive that way.

Make sure you have enough memory in the computer.

You say you have no custom columns, so that may not be the problem, unless they are hidden?

I don't believe Calibre even access the actual books when starting. Only metadata.db.

chaley · 09-18-2012, 07:32 AM

Quote:

Originally Posted by Adoby

If you have custom columns that depends on other columns, then I believe all values have to be initiated when you start Calibre. I have my metadata.db on a SSD drive and still noticed this very clearly. When I deleted those columns startup was fast again.

Not quite. The columns are evaluated only when displayed. Sorting on a column is equivalent to displaying all values for that column.

Quote:

I don't believe Calibre even access the actual books when starting. Only metadata.db.

Several versions ago calibre did look at attributes of books (modtime, etc) when starting. This was changed. I don't remember the version, but it was a couple of months ago at least. The current version does not look at book attributes unless you have a custom column that contains them (a "formats" column).

electronicfur · 09-18-2012, 09:15 AM

Quote:

Originally Posted by chaley

Several versions ago calibre did look at attributes of books (modtime, etc) when starting. This was changed. I don't remember the version, but it was a couple of months ago at least. The current version does not look at book attributes unless you have a custom column that contains them (a "formats" column).

I'm using the latest calibre 0.8.69 (portable to avoid any plugin issues) and it does seem to be reading attributes of all books in my library.

In order to test further I created a subset of my large > 10k book library. I need a subset to test because it's so long to open otherwise.

If I copy a subset of the library directory (eg only some of the books and the metadata.db), switch to the library, delete the one custom Genre column (just to be sure), and do a Database Maintenance->Restore Library, then when opening the resulting library, Calibre still read the attributes of all the books.

If however I create a new empty library, and then import the same subset of books into it, then when switching to that library, Calibre does not read the attributes of all the books, it only reads the first book attributes, and the next 6 book cover files.

So if I can figure out what the difference between the two test libraries is, I can work out where the issue lies. Does the first way of creating the subset test library still retain the preferences of the original library that are stored in the metadata.db? I'm guessing it does, and something in the preferences is causing Calibre to read all files.

Cheers,
EF

DoctorOhh · 09-18-2012, 09:36 AM

Quote:

Originally Posted by electronicfur

So if I can figure out what the difference between the two test libraries is, I can work out where the issue lies.

I'm curious, do the two libraries have the same set of columns listed in preferences? I attached the default set of columns, does your troubled library have the same columns?

chaley · 09-18-2012, 09:36 AM

The only code I can find in calibre that looks at the attributes of a format file is called when a template referring to a format is evaluated. Are you certain that you don't have a hidden composite column, and in particular a "formats" column? Or perhaps you have installed a plugin that walks through your books?

Copying a library folder will keep many of the preferences, and in particular all the custom columns, hidden or not. There are a large number of preferences stored outside the db.

08-18-2012, 12:29 AM	#1
SilverAce Junior Member Posts: 5 Karma: 10 Join Date: Aug 2012 Device: Kindle Fire	Calibre opens very slowly with large library? I have a fairly large library with 10k+ documents. Calibre takes a LONG time to load, easily more than 10 minutes after I open it to the time I can start using it. Is this normal, and is there any way to make it faster?

09-18-2012, 04:51 AM	#10
electronicfur Connoisseur Posts: 54 Karma: 30682 Join Date: Aug 2011 Device: Samsung Note FBReader, Nook Simple Touch FBReader, (Kindle3 died)	SilverAce, I am also having a slow startup problem, and I've so far traced the issue to Calibre reading the attributes of all the books in my library on startup. I dont yet know why this is happening. I've even got this happening if I use the Calibre portable version and point it at my library, so I know it's not a plugin causing the issue. I used Process Monitor to discover what Calibre was doing on startup. See http://technet.microsoft.com/en-us/s.../bb896645.aspx Process Monitor for me shows lots of CreateFile, QueryNetwork, CloseFile entries, and when I look at the details it is requesting the Read Attributes of the files. It seems to do this for all books, so my startup is extremely slow. I dont use any composite columns, so not sure why this is happening. I only have one custom Genre column. And although I have excluded the calibre directories, I've completely disabled my virus scanner whilst testing this to be sure it is not the cause. Last edited by electronicfur; 09-18-2012 at 04:54 AM.

09-18-2012, 06:55 AM	#11
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	If you have custom columns that depends on other columns, then I believe all values have to be initiated when you start Calibre. I have my metadata.db on a SSD drive and still noticed this very clearly. When I deleted those columns startup was fast again. I have my metadata.db on SSD but the Calibre library on a NAS. I use a symlink in the Calibre library to the real file on the SSD, to speed up things. Calibre became a little bit more responsive that way. Make sure you have enough memory in the computer. You say you have no custom columns, so that may not be the problem, unless they are hidden? I don't believe Calibre even access the actual books when starting. Only metadata.db. Last edited by Adoby; 09-18-2012 at 07:09 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre Portable - Which Library Opens?	nynaevelan	Calibre	5	09-21-2011 12:58 PM
Best way to get a large Calibre library into the PE library?	Filark	enTourage Archive	0	04-20-2011 10:18 PM
Kindle and Calibre user with problem importing large library into Calibre	pleabargain	Calibre	1	12-07-2010 10:19 AM
Large library/calibre interface	StanByk	Calibre	12	10-19-2009 10:20 AM
How large is your Calibre library?	radius	Calibre	28	10-12-2009 01:14 AM

08-18-2012, 04:07 AM	#2
Thasaidon Hedge Wizard Posts: 800 Karma: 19999999 Join Date: May 2011 Location: UK/Philippines Device: Kobo Touch, Nook Simple	This is not normal. I have an old PC (pentium 4) and with a library of over 23k books Caliber loads less than one minute.

08-21-2012, 10:08 PM	#4
SilverAce Junior Member Posts: 5 Karma: 10 Join Date: Aug 2012 Device: Kindle Fire	Thanks for the responses. I should mention I'm running OS X 10.7.4 -- can I still start Calibre from the command line? I have not created any special columns to search for anything... Calibre is also pretty slow, when, for example, I add a book. It's noticeably laggy -- it takes say, 20 or 30 seconds to add it. Maybe that slowness is related?

08-22-2012, 09:41 AM	#6
PeterT Grand Sorcerer Posts: 12,166 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	I've always found it easiest just to go the Preferences option menu (select the drop down arrow besides Preferences) and select the Restart in Debug mode command or use the keyboard short cut for that (Ctrl+Shift+R).

08-24-2012, 06:34 PM	#7
SilverAce Junior Member Posts: 5 Karma: 10 Join Date: Aug 2012 Device: Kindle Fire	You know what? I strongly suspect that the slowness is due to my external hard drive, a Western Digital My Passport, being formatted as NTFS and being accessed through a third-party driver. This benchmark shows that such drivers are FAR slower than HFS+ in dealing with folders with lots of files and subfolders. So that may be the difference right there.

09-18-2012, 09:36 AM	#15
chaley Grand Sorcerer Posts: 11,741 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	The only code I can find in calibre that looks at the attributes of a format file is called when a template referring to a format is evaluated. Are you certain that you don't have a hidden composite column, and in particular a "formats" column? Or perhaps you have installed a plugin that walks through your books? Copying a library folder will keep many of the preferences, and in particular all the custom columns, hidden or not. There are a large number of preferences stored outside the db.