Performance when adding and moving - Page 2

kovidgoyal · 10-08-2014, 09:11 AM

@abuck: You are welcome to spend time profiling. Adding is not multicore and never will be, adding large numbers of books is not acommon enough occurrence to spend the time on making it multicore, I have higher priority things to do.

As for where the code is, look in gui2/add.py

ms233 · 10-18-2014, 11:42 AM

Quote:

Originally Posted by abuck

+1 for the performance issue. I think it seems to be a recurrent problem.

I've checked my ressource and like other said, my cpu/memory/disk usage does not go higher than 15% and I do have a core i7.

I thought I posted a reply to this last night, but I either screwed something up or it got deleted. Anyway.

This is also what I'm seeing. CPU utilization obviously isn't spiked so I looked everywhere else I could think of. Memory paging doesn't look out of the ordinary, disk IO is reasonable, processor interrupts don't spike, and processor queue length doesn't seem to be the culprit either. I don't think there's a hardware bottleneck causing this dreadfully slow performance. Is it possible something in the code is entering a wait state during each copy?

kovidgoyal · 10-18-2014, 11:48 AM

@ms233: Anything is possible, but fortunately calibre is open source, feel free to instrument the code to find out if it is indeed waiting where it shouldn't be. Once again, the code is in gui2/add.py

ioconnor · 05-04-2015, 05:14 AM

I'm at about 1500 books now. I have an i7-4910, 32GBs of RAM, and two SSDs each half full. Now when I go to add a new book it takes minutes, regardless of the size, as if the database has no indexes on it. If I add say 10 more books I now expect it to take hours!

So I thought maybe I had done something wrong. So I installed a new OS, linux mint, and installed calibre on it and copied over the calibre directory. Everything started up fine with no delays. Then I added a batch of books. Same problem. The screen doesn't refresh. It is like hung and unresponsive. (Though the rest of the computer is fine and I can do work.) I ended up going to bed. This morning the books are in calibre this morning. However now I know the delay is part of a default installation with calibre.

Any chance of adding indexes to the database or doing whatever it takes to let calibre handle large libraries?

chaley · 05-04-2015, 05:43 AM

I have a library containing 4,000 books (and that is rather small). It takes less than 10 seconds to add a book. My test library contains 25,000 books and takes the same amount of time.

ioconnor · 05-04-2015, 07:12 AM

Quote:

Originally Posted by chaley

I have a library containing 4,000 books (and that is rather small). It takes less than 10 seconds to add a book. My test library contains 25,000 books and takes the same amount of time.

So what can possibly be wrong?! Is there a diagnostics mode that I can run and send you the results with?

BetterRed · 05-04-2015, 07:47 AM

@ioconnor - My rig is an i5 processor with 6GB RAM running Windows 7 Pro, it has an Hitachi 256GB SDD system drive. My libraries are on a SATA2 WD 2TB Caviar Black, it is indexed by Windows, it is compressed by Windows and it is NOT excluded from Windows Security Essentials and Malwarebytes scanning

I just added a a 400MB AVI to a 8,000+ book library - ~9 secs, I then added a 200KB EPUB to same library - ~2 secs

Each morning I add 10-20 books (50-250KB each) to a 65,000 book library, takes a minute or two for all of them

Things you could try

rebuild the database,
hide the libraries from indexers, malware scanners etc,
run diagnostics on the hard drive,
run a chkdsk on the drive,
run a system file check

I have no doubt the problem lies with your computer. Most of time taken by adding a book is copying the file into the library folders - not updating the db. I just copied that same 400MB AVI to another location on the same drive with my file manager, which has a timer - 8.46 secs

BR

chaley · 05-04-2015, 07:53 AM

Also ensure that your library is not on a network drive/NAS, and that the file systems on the SSDs are "native", not something esoteric.

Beyond that, perhaps eschwartz can make some suggestions when he checks in. IIRC he is a Linux Mint user.

BetterRed · 05-04-2015, 07:55 AM

Quote:

Originally Posted by ioconnor

Is there a diagnostics mode that I can run and send you the results with?

@ioconnor - yep - restart in debug mode (ctrl/shift/R) - add book, and post debug log here

But try other things I suggested first.

Also try running in Safe mode - maybe there's something interfering

Added : Did you copy a library from Windows to Linux?

BR

ioconnor · 05-04-2015, 10:50 AM

Thanks. I'll try those things this evening. (When I get back.)

eschwartz · 05-04-2015, 08:53 PM

Quote:

Originally Posted by chaley

Also ensure that your library is not on a network drive/NAS, and that the file systems on the SSDs are "native", not something esoteric.

Beyond that, perhaps eschwartz can make some suggestions when he checks in. IIRC he is a Linux Mint user.

ArchLinux actually, but same difference.

I'm afraid I can't imagine what the problem might be either. Unless perhap the library is on a secondary partition that is formatted with FAT32...
Modern filesystems (non-networked, too) should work fine with calibre. And from calibre's end nothing should be running slow.

LadyKate · 05-08-2015, 12:51 AM

Is it possible the antivirus you are using is scanning each file as it is added?

eschwartz · 05-08-2015, 11:35 AM

That might be the case on Windows, but if the problem shows up on linux as well, then it can't be.

Linux has no antiviruses (they aren't needed).

ioconnor · 05-09-2015, 09:24 PM

Ok, problem solved. Following BetterRed's suggestions I discovered my SSD data drive, less than a month old, was having problems. (I did not suspect a new SSD drive less than half full and only really used for documents would fail.) Ordered a new SSD and just finished testing. Everything is now working fine. Thank you all for your input.

05-04-2015, 07:47 AM	#22
BetterRed null operator (he/him) Posts: 21,769 Karma: 30237628 Join Date: Mar 2012 Location: Sydney Australia Device: none	@ioconnor - My rig is an i5 processor with 6GB RAM running Windows 7 Pro, it has an Hitachi 256GB SDD system drive. My libraries are on a SATA2 WD 2TB Caviar Black, it is indexed by Windows, it is compressed by Windows and it is NOT excluded from Windows Security Essentials and Malwarebytes scanning I just added a a 400MB AVI to a 8,000+ book library - ~9 secs, I then added a 200KB EPUB to same library - ~2 secs Each morning I add 10-20 books (50-250KB each) to a 65,000 book library, takes a minute or two for all of them Things you could try rebuild the database, hide the libraries from indexers, malware scanners etc, run diagnostics on the hard drive, run a chkdsk on the drive, run a system file check I have no doubt the problem lies with your computer. Most of time taken by adding a book is copying the file into the library folders - not updating the db. I just copied that same 400MB AVI to another location on the same drive with my file manager, which has a timer - 8.46 secs BR

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
touchscreen performance	sebigbos	PocketBook	6	03-21-2012 01:02 PM
PRS-650 Sluggish performance	SeaBookGuy	Sony Reader	11	12-12-2011 10:26 PM
Scalability and performance	blither	Calibre	1	08-10-2010 08:31 PM
DR800 Performance	jjansen	iRex	18	08-03-2010 09:10 PM
Feature Request: Moving and not copying when adding files...	Chuwiey	Calibre	7	07-03-2010 05:35 AM

10-08-2014, 09:11 AM	#16
kovidgoyal creator of calibre Posts: 45,398 Karma: 27756918 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@abuck: You are welcome to spend time profiling. Adding is not multicore and never will be, adding large numbers of books is not acommon enough occurrence to spend the time on making it multicore, I have higher priority things to do. As for where the code is, look in gui2/add.py

10-18-2014, 11:48 AM	#18
kovidgoyal creator of calibre Posts: 45,398 Karma: 27756918 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@ms233: Anything is possible, but fortunately calibre is open source, feel free to instrument the code to find out if it is indeed waiting where it shouldn't be. Once again, the code is in gui2/add.py

05-04-2015, 05:14 AM	#19
ioconnor Junior Member Posts: 4 Karma: 10 Join Date: Dec 2014 Device: laptop	I'm at about 1500 books now. I have an i7-4910, 32GBs of RAM, and two SSDs each half full. Now when I go to add a new book it takes minutes, regardless of the size, as if the database has no indexes on it. If I add say 10 more books I now expect it to take hours! So I thought maybe I had done something wrong. So I installed a new OS, linux mint, and installed calibre on it and copied over the calibre directory. Everything started up fine with no delays. Then I added a batch of books. Same problem. The screen doesn't refresh. It is like hung and unresponsive. (Though the rest of the computer is fine and I can do work.) I ended up going to bed. This morning the books are in calibre this morning. However now I know the delay is part of a default installation with calibre. Any chance of adding indexes to the database or doing whatever it takes to let calibre handle large libraries?

05-04-2015, 05:43 AM	#20
chaley Grand Sorcerer Posts: 12,450 Karma: 8012886 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	I have a library containing 4,000 books (and that is rather small). It takes less than 10 seconds to add a book. My test library contains 25,000 books and takes the same amount of time.

05-04-2015, 07:53 AM	#23
chaley Grand Sorcerer Posts: 12,450 Karma: 8012886 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	Also ensure that your library is not on a network drive/NAS, and that the file systems on the SSDs are "native", not something esoteric. Beyond that, perhaps eschwartz can make some suggestions when he checks in. IIRC he is a Linux Mint user.

05-04-2015, 10:50 AM	#25
ioconnor Junior Member Posts: 4 Karma: 10 Join Date: Dec 2014 Device: laptop	Thanks. I'll try those things this evening. (When I get back.)

05-08-2015, 12:51 AM	#27
LadyKate Fanatic Posts: 515 Karma: 1470724 Join Date: Jul 2013 Location: Quebec CA Device: android 4 (samsung tablet and asus tablet)	Is it possible the antivirus you are using is scanning each file as it is added?

05-08-2015, 11:35 AM	#28
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	That might be the case on Windows, but if the problem shows up on linux as well, then it can't be. Linux has no antiviruses (they aren't needed).

05-09-2015, 09:24 PM	#29
ioconnor Junior Member Posts: 4 Karma: 10 Join Date: Dec 2014 Device: laptop	Ok, problem solved. Following BetterRed's suggestions I discovered my SSD data drive, less than a month old, was having problems. (I did not suspect a new SSD drive less than half full and only really used for documents would fail.) Ordered a new SSD and just finished testing. Everything is now working fine. Thank you all for your input.