Performance when adding and moving

Motomaggot · 02-21-2014, 06:36 PM

I've read countless posts on this forum and around the web on performance and so far, nothing has really worked. I have an i7 4770 processor with 16Gb of RAM with an SSD boot and RAID 1 4Tb drives for data.

Its seems that adding books to a library is painfully slow, and the copy and delete is just as bad.

Any suggestions on what I can adjust to make Calibre run faster no matter how much resources it eats up on the machine.

Matthew

PeterT · 02-21-2014, 07:28 PM

Maybe some description of the custom columns you have; whether or not you have any that display the formats material is in.

Adoby · 02-21-2014, 08:40 PM

Move the calibre library to SSD.

But don't expect too much...

I have a pretty sweet setup with a hybrid filesystem using bcache in Ubuntu 13.10 with 200 GB SATA 3 SSD write-back cache for two 3TB 7200RPM SATA 3 harddrives. So the entire library can be cached on SSD at once. It is a relatively new setup, I am still tweaking it. I may test to move the metadata.db - files to pure SSD.

While I copy 100 books to another large library, on the same partition, the CPU-usage by calibre is 6-7% on average (8 core AMD Bulldozer) and the CPU never even clocks up to full speed. If I copy to a new and empty library even less, around 4% CPU-usage.

But it still takes close to 5 minutes to copy 100 books. (3 custom text columns, no calculated columns, several virtual libraries, if that matter). If I copy to a new empty library it is one or two minutes faster.

If I copy the folder with the library with 100 books to another place using the filemanager, it is instantaneous. It is only around 40MB after all. So I assume that it is all the database updating and creation of folders and so on, that takes time?

Edit: I tested with both source and destionation metadata.db on pure SSD. It was a little faster, around 2 1/2 minutes. Not enough for me to bother. It is fast enough as it is...

kovidgoyal · 02-21-2014, 09:02 PM

Adding books has to:

1) read metadata from the files, which for some file formats can be ver slow, for example, epub can require rendering of html in a browser to generate covers.

2) Search the entire existing library for duplicates

3) Update lots of fields in the database

There is no way that adding books is every going to come close to the performance of copying files in the filesystem.

Adoby · 02-21-2014, 09:14 PM

Yes, I wouldn't expect it to either.

But considering that the actual filecopying is almost instantaneous, it is surprising that the CPU usage isn't higher. Well below 10%. But then I only used top to gauge the CPU-usage...

Correction. When copying to a large (+6000 books) destination library the CPU usage goes up to 16%.

theducks · 02-21-2014, 09:53 PM

How often does one add a BIG glob of books to a Library?

Nowadays my big batch is when , I add my (almost) monthly Baen bundle of 6-9 books and that is done before I get back from the Kitchen coffee pot.

Motomaggot · 02-21-2014, 10:45 PM

Quote:

Originally Posted by PeterT

Maybe some description of the custom columns you have; whether or not you have any that display the formats material is in.

I'm not doing anything special with columns at all. I guess the level of performance I'm seeing is just normal.

BetterRed · 02-21-2014, 10:48 PM

Quote:

Originally Posted by Motomaggot

I've read countless posts on this forum and around the web on performance and so far, nothing has really worked. I have an i7 4770 processor with 16Gb of RAM with an SSD boot and RAID 1 4Tb drives for data.

Its seems that adding books to a library is painfully slow, and the copy and delete is just as bad.

Any suggestions on what I can adjust to make Calibre run faster no matter how much resources it eats up on the machine.

Matthew

@Matthew - could you to quantify "painfully slow"

I did some tests on a 6GB I5 system, running windows 7-sp1, it has two 2TB SATA2, NTFS internal drives C: & E:, both WD Caviar Blacks drive E: is compressed. At the time I did the tests MSE was running (no exclusions), and I was rebuilding the search indexes (including calibre libraries) with the Indexer Backoff feature OFF - that means the indexer is running at normal priority. I used a large file (1.3GB AVI) because book files are small and file operations on a small files are difficult to time with a stop watch.

Using standard Windows I copied the avi from E: to C: in 15.5 seconds. I renamed the avi (author - title) on Drive C and added it to Library A on E: in 17.5 seconds. A 2 second overhead to create the author and book folders, and add the necessary rows to the database tables etc, seems reasonable to me.

I then moved the 'book' from Library A to Library B, also on drive E in 41.4 secs; libraries A and B are clones, so same columns etc. A similar operation (ie move a folder from one parent to another) using the same data on the same drive in Windows its too fast to time. The difference is that Windows makes use of hardlinks to effect the move, whereas Calibre does a copy+delete. When I copied the same data from folder A to folder B on drive E with Windows it took 44.2 secs - 2.8 seconds slower, probably because of head movement and cache differences.

If I change the Title on that book, it's also too fast to time. I think that's because calibre does the move from folder "Wrong Title (20)" to folder "Right Title (20)" using hardlinks rather than copy/delete. Its a bit tricky to do that on a library move, because the two libraries may be on different drives and hardlinks don't work across drives. Kovid would have to implement two types of move books - one that works between different drives and one that works on the same drive - then there's the problem that not all file systems support hardlinking.

BR

Vortex · 02-22-2014, 01:29 PM

I have an i3 3.3Ghz, 8Gbs ram, 7Tb's of HDD. I tried adding 155,000 files into Calibre, it was managing about 1% an hour.

DoctorOhh · 02-23-2014, 02:44 AM

Quote:

Originally Posted by Vortex

I have an i3 3.3Ghz, 8Gbs ram, 7Tb's of HDD. I tried adding 155,000 files into Calibre, it was managing about 1% an hour.

It is nuts to try and do these in a giant batch. Try lots of 1000, 2000 or 5000 instead. You may find a magic level that maximizes the speed at which books will be added.

Personally I would recommend a 1000 at a time, followed by fixing the metadata for that batch before moving on to the next batch. Fixing the metadata is far from automatic, it requires eyes on the data and books. Once the batch is complete move onto the next.

Good Luck.

Motomaggot · 07-08-2014, 04:04 PM

I've played around with sizes of transfers and using books that have both epub and mobi formats it takes from 7 to 10 seconds per book in a copy. Blocks of a 1000 books just crawl. With 32Gb of memory and a super fact processor, I still think we should have a way to speed this up a bit.

BetterRed · 07-08-2014, 05:53 PM

@Motomaggot - might be worth looking at this thread, Deleting duplicate folders. In particular I draw your attention to Kovid's statement in post #19.

It would be interesting to know what the difference would be if you turned off the settings relating to duplicates in the Adding Books->The Add Process tab (assuming they're set)

BR

theducks · 07-08-2014, 09:56 PM

Remember: Quad cores gain you little with Calibre, its Single threaded.
Fast IO helps

Inferno · 07-20-2014, 02:15 PM

I find it best to do a large add of about 5000 to 10000 overnight. I just have to look through the duplicate list when I wake up.

abuck · 10-08-2014, 08:32 AM

+1 for the performance issue. I think it seems to be a recurrent problem.

I've checked my ressource and like other said, my cpu/memory/disk usage does not go higher than 15% and I do have a core i7.

I understand that I'm sure there is a lot to do in the background and lot of reading from metadata and writting to db. However, do we know exactly what's the slowest operation ?

@kovidgoyal

1) It seems that we should need to do a bit of profiling to really target what's the bottleneck. One thing that come to my mind is maybe trying to make a better use of the multi-threading.

2) Is all the import operation through python scripts or it's in the core calibre ? I currently have a windows box so might need a linux distro if want to look at the core (base in the installation dev comments).

3) I'm willing to give some of my time to look at this issue, if you can maybe point me out at where the add books take place ?

02-21-2014, 08:40 PM	#3
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Move the calibre library to SSD. But don't expect too much... I have a pretty sweet setup with a hybrid filesystem using bcache in Ubuntu 13.10 with 200 GB SATA 3 SSD write-back cache for two 3TB 7200RPM SATA 3 harddrives. So the entire library can be cached on SSD at once. It is a relatively new setup, I am still tweaking it. I may test to move the metadata.db - files to pure SSD. While I copy 100 books to another large library, on the same partition, the CPU-usage by calibre is 6-7% on average (8 core AMD Bulldozer) and the CPU never even clocks up to full speed. If I copy to a new and empty library even less, around 4% CPU-usage. But it still takes close to 5 minutes to copy 100 books. (3 custom text columns, no calculated columns, several virtual libraries, if that matter). If I copy to a new empty library it is one or two minutes faster. If I copy the folder with the library with 100 books to another place using the filemanager, it is instantaneous. It is only around 40MB after all. So I assume that it is all the database updating and creation of folders and so on, that takes time? Edit: I tested with both source and destionation metadata.db on pure SSD. It was a little faster, around 2 1/2 minutes. Not enough for me to bother. It is fast enough as it is... Last edited by Adoby; 02-21-2014 at 09:21 PM.

02-21-2014, 09:14 PM	#5
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Yes, I wouldn't expect it to either. But considering that the actual filecopying is almost instantaneous, it is surprising that the CPU usage isn't higher. Well below 10%. But then I only used top to gauge the CPU-usage... Correction. When copying to a large (+6000 books) destination library the CPU usage goes up to 16%. Last edited by Adoby; 02-21-2014 at 09:18 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
touchscreen performance	sebigbos	PocketBook	6	03-21-2012 01:02 PM
PRS-650 Sluggish performance	SeaBookGuy	Sony Reader	11	12-12-2011 10:26 PM
Scalability and performance	blither	Calibre	1	08-10-2010 08:31 PM
DR800 Performance	jjansen	iRex	18	08-03-2010 09:10 PM
Feature Request: Moving and not copying when adding files...	Chuwiey	Calibre	7	07-03-2010 05:35 AM

02-21-2014, 06:36 PM	#1
Motomaggot I'll always be a newb Posts: 20 Karma: 10 Join Date: Feb 2012 Device: Kindle Paperwhite & Touch	Performance when adding and moving I've read countless posts on this forum and around the web on performance and so far, nothing has really worked. I have an i7 4770 processor with 16Gb of RAM with an SSD boot and RAID 1 4Tb drives for data. Its seems that adding books to a library is painfully slow, and the copy and delete is just as bad. Any suggestions on what I can adjust to make Calibre run faster no matter how much resources it eats up on the machine. Matthew

02-21-2014, 07:28 PM	#2
PeterT Grand Sorcerer Posts: 12,158 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	Maybe some description of the custom columns you have; whether or not you have any that display the formats material is in.

02-21-2014, 09:02 PM	#4
kovidgoyal creator of calibre Posts: 43,843 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Adding books has to: 1) read metadata from the files, which for some file formats can be ver slow, for example, epub can require rendering of html in a browser to generate covers. 2) Search the entire existing library for duplicates 3) Update lots of fields in the database There is no way that adding books is every going to come close to the performance of copying files in the filesystem.

02-21-2014, 09:53 PM	#6
theducks Well trained by Cats Posts: 29,782 Karma: 54830978 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	How often does one add a BIG glob of books to a Library? Nowadays my big batch is when , I add my (almost) monthly Baen bundle of 6-9 books and that is done before I get back from the Kitchen coffee pot.

02-22-2014, 01:29 PM	#9
Vortex Groupie Posts: 171 Karma: 10 Join Date: Dec 2008 Device: Likebook Mars	I have an i3 3.3Ghz, 8Gbs ram, 7Tb's of HDD. I tried adding 155,000 files into Calibre, it was managing about 1% an hour.

07-08-2014, 04:04 PM	#11
Motomaggot I'll always be a newb Posts: 20 Karma: 10 Join Date: Feb 2012 Device: Kindle Paperwhite & Touch	I've played around with sizes of transfers and using books that have both epub and mobi formats it takes from 7 to 10 seconds per book in a copy. Blocks of a 1000 books just crawl. With 32Gb of memory and a super fact processor, I still think we should have a way to speed this up a bit.

07-08-2014, 05:53 PM	#12
BetterRed null operator (he/him) Posts: 20,553 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	@Motomaggot - might be worth looking at this thread, Deleting duplicate folders. In particular I draw your attention to Kovid's statement in post #19. It would be interesting to know what the difference would be if you turned off the settings relating to duplicates in the Adding Books->The Add Process tab (assuming they're set) BR

07-08-2014, 09:56 PM	#13
theducks Well trained by Cats Posts: 29,782 Karma: 54830978 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Remember: Quad cores gain you little with Calibre, its Single threaded. Fast IO helps

07-20-2014, 02:15 PM	#14
Inferno Connoisseur Posts: 60 Karma: 2016606 Join Date: Apr 2012 Device: android tablets	I find it best to do a large add of about 5000 to 10000 overnight. I just have to look through the duplicate list when I wake up.

10-08-2014, 08:32 AM	#15
abuck Member Posts: 17 Karma: 10 Join Date: Sep 2013 Device: playbook	+1 for the performance issue. I think it seems to be a recurrent problem. I've checked my ressource and like other said, my cpu/memory/disk usage does not go higher than 15% and I do have a core i7. I understand that I'm sure there is a lot to do in the background and lot of reading from metadata and writting to db. However, do we know exactly what's the slowest operation ? @kovidgoyal 1) It seems that we should need to do a bit of profiling to really target what's the bottleneck. One thing that come to my mind is maybe trying to make a better use of the multi-threading. 2) Is all the import operation through python scripts or it's in the core calibre ? I currently have a windows box so might need a linux distro if want to look at the core (base in the installation dev comments). 3) I'm willing to give some of my time to look at this issue, if you can maybe point me out at where the add books take place ?

Advert

Advert