Calibre Running Really Slowly

[polly]

Hi,

I've been using Calibre since about version 0.5. I usually update to the newest version each week. About version 1.0 I noticed that Calibre was starting to run more slowly. The problem didn't resolve with newer versions, so I tried uninstalling and going back to a version that had worked for me in the past. Unfortunately, that didn't solve the problem.

Many tasks come with a noticeable lag. Switching from one record to another takes 30 seconds to a minute. Closing the window after downloading metadata can take several minutes, as can opening a book or closing a book within Calibre. The last book that I tried to add (drag and drop) took 35 minutes. Clicking on anything else during the wait makes Calibre grey out and become unstable.

Calibre itself is mostly running normally, until I ask it to do something with a book. I can use the tag browser, search for books, and update metadata without problems, but when I try to select a book or save the changes, it lags again. I does everything that I ask, but it's like trying to run Windows 7 with 1GB of RAM.

I tried checking the library under Library Maintenance and didn't get any errors back. I didn't really expect any, since I have several libraries, all with this problem.

I ran Calibre in debug and went through some of the tasks that lag. The log below is what it spit out. I would welcome any suggestions to solve this.

calibre Debug log
calibre 0.9.40 isfrozen: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.4
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Starting up...
Started up in 3028.98 seconds with 112 books
Traceback (most recent call last):
File "site-packages\calibre\gui2\metadata\single_download.py" , line 1091, in ok_clicked
File "site-packages\calibre\gui2\metadata\single_download.py" , line 1086, in next_clicked
File "site-packages\calibre\gui2\metadata\single_download.py" , line 314, in get_result
File "site-packages\calibre\gui2\metadata\single_download.py" , line 309, in select_index
AttributeError: 'NoneType' object has no attribute 'index'

[kovidgoyal]

The slowness you describe is not normal and likely has nothing to do with calibre itself. It's caused by something in the software environment on your machine, unfortunately there's no easy way to determine what that may be.

General tips, that may or may not help:

1) Defragment your hard disk
2) Disable your antivirus
3) Try to eliminate as many programs running int he background as possible
4) Run a registry cleaner
5) Update the drivers on your computer

If nothing works, you are left with the old panacea of re-installing windows.

[Sabardeyn]

Kovid is, of course, absolutely correct. I did want to add something for yourself and other users to consider.

If calibre is running on an older computer system, with limited memory or a less capable cpu chip, it will run slowly, in just the manner you have described. However, calibre's ability to process actions and perform tasks/conversions does not generally change. Over the course of several updates calibre should maintain the same relative speed. It should not noticeably slow down.

There are some exceptions: poorly crafted ebooks with too complex of an internal structure; or a copy of an ebook that has been converted too many times (conversion of a conversion of a conversion situation). Other possibilities exist too. For instance, if you download metadata and have thousands of tags, or series names - many of which are spelling or alternate wording of other entries. Or just a humongous library with tens of thousands of books.

All of these things can cause calibre to seem to respond slowly, but are usually temporary delays until the specific task is done. You can improve calibre's response time somewhat by cleaning up your metadata.

It also helps to keep the Tag Browser and Book Panel (left and right sides of the default view, respectively) closed until needed. Use them as needed and then re-close them.

[theducks]

I will toss in:
(multiple) Use of any 'Calculated' objects (fields, colors, icons) in the main GUI (Convert and complex Send templates are a different places to slow other use down)

[BetterRed]

Originally Posted by polly

Many tasks come with a noticeable lag. Switching from one record to another takes 30 seconds to a minute. Closing the window after downloading metadata can take several minutes, as can opening a book or closing a book within Calibre. The last book that I tried to add (drag and drop) took 35 minutes. Clicking on anything else during the wait makes Calibre grey out and become unstable.

@polly - here are my additional thoughts to the previous suggestions

The above seems to indicate that tasks that are confined to reading and writing to the database are OK, whereas tasks involving reading and writing to the book folders are the ones running slowly. That suggests to me the problem is file system related.

The trace reports <<Started up in 3028.98 seconds with 112 books>> That's 51 minutes!!! Perhaps you can confirm that it really took that long, and let us know how long it took to start Calibre before this slowness started.

Do you have a back up of your libraries - if not then I suggest you take a backup of them ASAP. My hunch is that the disk drive on which you have your libraries is on its last legs and generating massive numbers of hardware interrupts - this can really slow things down.

To 'prove' its not something wrong with your existing calibre installation or configuration you could install the portable version and trying opening that 112 book library and doing a couple of task that run slow - if it also runs slowly then we can be almost certain that your problem has nothing to do with calibre.

What sort of device are the libraries on - a regular spinning drive disk inside the computer, an external spinning drive on USB or eSata or a thumb drive - or maybe a cloud drive.

Do you have other data on the drive where your libraries are - eg photos, videos, documents and are you able to access them and add/rename and delete new items.

When calibre is NOT running you could try to copy that 112 book library (the entire folder) with Windows Explorer to another disk drive - and rename it to have a distinguishing name. Then start calibre, have breakfast, and then get it to open the library you just copied to another drive and see what happens when you do things in calibre - if they're OK then its something to do with the disk drive.

BTW if it really takes 51 minutes to start then where can I download some of the patience you apparently have <grin>

Addendum : are the metadata.db databases a) actually in your library folders, or b) are they on another drive with symbolic links in the library folders - if you don't know what I'm talking about then the answer is a)

BR

[polly]

Thank you for all of the suggestions. I do routinely defrag the hard drive, use ccCleaner, etc. Everything else on this computer is working normally, at least for now. The system is a quad core processor with 8GB of RAM.

The metadata.db file is with the library folders on a separate drive from the program. I have a 2 TB hard drive where all of my data lives, including my Calibre libraries. I can access everything normally on this drive, add and rename files, and play videos with no problems or lags. My Calibre libraries are backed up, both to the cloud and to my Raid 0 NAS.

I guess the next step is to close Calibre, move the libraries to a different hard drive, and then point Calibre to the new location.

[Sabardeyn]

Polly,
With a quad core cpu and 8Gb of RAM you shouldn't be slowing down at all. I just loaded a library of 2400 books in about a minute and you are running a more capable computer system than I am.

What I said before is valid in and of itself, but obviously not applicable to your circumstances.

Are you running any disk intensive activities? Perhaps you have PerfectDisk (a defragmenter) which has an optimized writing feature which moves stuff around silently? Or you have changed the frequency of antivirus updates? Performing AV scans? Are you using Indexing? When is the last time you did a full system AV scan?

[theducks]

Originally Posted by polly

Thank you for all of the suggestions. I do routinely defrag the hard drive, use ccCleaner, etc. Everything else on this computer is working normally, at least for now. The system is a quad core processor with 8GB of RAM.

The metadata.db file is with the library folders on a separate drive from the program. I have a 2 TB hard drive where all of my data lives, including my Calibre libraries. I can access everything normally on this drive, add and rename files, and play videos with no problems or lags. My Calibre libraries are backed up, both to the cloud and to my Raid 0 NAS.

I guess the next step is to close Calibre, move the libraries to a different hard drive, and then point Calibre to the new location.

Polly
having the Library on external rotating drives is not that unusual .
Try to avoid using it on a USB Hub. The throughput can suffer with (cpu side)port traffic

Be sure to exclude the Library folder from a A/V scan (this may require you to assign a permanent drive letter. I used L: )

[BetterRed]

For what file system has the drive been formatted. And can you confirm that calibre start up really takes 51 minutes

Yes its worth moving the libraries to another drive to see what happens. Have you run chkdsk on that 2TB drive - nothing you have posted has removed my suspicion that its a hardware issue.

If you install Process Explorer (get it from MS) and have a look at the hardware interrupts entry then if its much more than 1% CPU usage then it probably indicates something is failing, see attachment. also download the disk manufacturers diagnostics tools and run those.

BTW if its an NTFS formatted drive you don't need to defrag, Windows 7 does it on the fly - if you use a 3rd party defragger then I suggest you turn off the Windows on the fly defragging - Google will find you plenty of help with that.

Is it possible that you are running multiple AV's - I recently stumbled on a system that was running MSE, McAfee and AVG - calibre was sluggish but so was everything else - 10 minutes to boot and 20 minutes to shut down.

I run calibre on a lesser system than you are using - 3.2GHz i5, 6GB Ram and a 2TB internal Sata 2 drive, I allow MSE AV to scan the calibre libraries, the entire drive is indexed, and the drive is compressed - I have no performance issues.

are you running an image manager that sucks up any images it can find - such as Picasa or similar? You may not be running the main program but they can hide their scanners - Process Explorer or Task Manager should help you spot anything like that.

Task Manager also has some useful graphs for monitoring resource usage - ctrl/shift/esc - click Performance Tab, click Resource Usage button.

BR

[polly]

It's an NTFS formatted SATA drive that occupies a second drive bay and has a SATA connection to the motherboard. According to CHKDSK, there are no file system errors or bad sectors.

My only antivirus is AVG and ZoneAlarm is my firewall. I exclude my entire data drive from scanning. AVG automatically scans files that I download and CDs/DVDs that I insert.

Process Explorer and Task Manager agree that I'm using about half of my RAM and 60% of my CPU. Nothing resource intensive in the background.

I haven't timed Calibre opening, but would believe 51 minutes. It tends to be ask it to do something, go load the dishwasher, come back for the next step, etc.

[theducks]

Originally Posted by polly

It's an NTFS formatted SATA drive that occupies a second drive bay and has a SATA connection to the motherboard. According to CHKDSK, there are no file system errors or bad sectors.

My only antivirus is AVG and ZoneAlarm is my firewall. I exclude my entire data drive from scanning. AVG automatically scans files that I download and CDs/DVDs that I insert.

Process Explorer and Task Manager agree that I'm using about half of my RAM and 60% of my CPU. Nothing resource intensive in the background.

I haven't timed Calibre opening, but would believe 51 minutes. It tends to be ask it to do something, go load the dishwasher, come back for the next step, etc.

Something is seriously amiss.

Have you run any 'disk drive performance speed tests' on that drive?

That is a 32bit system. 'Half my RAM' is kind a vague.
http://msdn.microsoft.com/en-us/libr...mits_windows_7
The slots may accept more than can be used by W7

[BetterRed]

Originally Posted by polly

It's an NTFS formatted SATA drive that occupies a second drive bay and has a SATA connection to the motherboard. According to CHKDSK, there are no file system errors or bad sectors.

My only antivirus is AVG and ZoneAlarm is my firewall. I exclude my entire data drive from scanning. AVG automatically scans files that I download and CDs/DVDs that I insert.

Process Explorer and Task Manager agree that I'm using about half of my RAM and 60% of my CPU. Nothing resource intensive in the background.

I haven't timed Calibre opening, but would believe 51 minutes. It tends to be ask it to do something, go load the dishwasher, come back for the next step, etc.

Like the 51 minutes, 60% CPU usage is extraordinary on a quad core system.

What % of the CPU is being used before starting calibre, and what's contributing to that 60% usage - is it calibre or something else, eg interrupts & DPCs.

I think the Calibre debug is reporting that you are running 32 bit windows,

Originally Posted by polly from Post #1

calibre Debug log
calibre 0.9.40 isfrozen: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')('Windows', '7', '6.1.7601')

Which given what you've said about your system configuration surprises me. What does Control Panel->System and Security->System report - and if you are running 32bit Windows I'm not sure that it can address more than 4GB of the 8GB of installed RAM.

BR

[polly]

I'm running 64-bit Windows 7 with an AMD Phenom 2.8GHz processor. I can't explain why Calibre is reporting it as 32 bit. The interrupts and system usage don't appear to change when Calibre is running. The Interrupts range from 0.6 to 0.8. They went as high as 1.5 when I started Calibre, but returned to normal when Calibre was running, even when Calibre isn't responding (most of the time). Task Manager says that I have 8.183 GB of physical memory, with around 4.7 GB available. The RAM seems really stable. The CPU usage is around 20% now. I agree that there's something wrong with the system, but can't explain why it only affects Calibre.

[gbm]

Originally Posted by polly

I'm running 64-bit Windows 7 with an AMD Phenom 2.8GHz processor. I can't explain why Calibre is reporting it as 32 bit. The interrupts and system usage don't appear to change when Calibre is running. The Interrupts range from 0.6 to 0.8. They went as high as 1.5 when I started Calibre, but returned to normal when Calibre was running, even when Calibre isn't responding (most of the time). Task Manager says that I have 8.183 GB of physical memory, with around 4.7 GB available. The RAM seems really stable. The CPU usage is around 20% now. I agree that there's something wrong with the system, but can't explain why it only affects Calibre.

I would run a root kit scanner/remover. Malwarebytes Anti-Rootkit

good luck

bernie

[theducks]

Originally Posted by polly

I'm running 64-bit Windows 7 with an AMD Phenom 2.8GHz processor. I can't explain why Calibre is reporting it as 32 bit. The interrupts and system usage don't appear to change when Calibre is running. The Interrupts range from 0.6 to 0.8. They went as high as 1.5 when I started Calibre, but returned to normal when Calibre was running, even when Calibre isn't responding (most of the time). Task Manager says that I have 8.183 GB of physical memory, with around 4.7 GB available. The RAM seems really stable. The CPU usage is around 20% now. I agree that there's something wrong with the system, but can't explain why it only affects Calibre.

You installed the 32bit version of Calibre (allowed)

Install (alongside. both can exist, but only use one or the other at any time) Calibre64 It will use everything, just use the correct shortcut to start