02-21-2014, 06:36 PM | #1 |
I'll always be a newb
Posts: 20
Karma: 10
Join Date: Feb 2012
Device: Kindle Paperwhite & Touch
|
Performance when adding and moving
I've read countless posts on this forum and around the web on performance and so far, nothing has really worked. I have an i7 4770 processor with 16Gb of RAM with an SSD boot and RAID 1 4Tb drives for data.
Its seems that adding books to a library is painfully slow, and the copy and delete is just as bad. Any suggestions on what I can adjust to make Calibre run faster no matter how much resources it eats up on the machine. Matthew |
02-21-2014, 07:28 PM | #2 |
Grand Sorcerer
Posts: 12,162
Karma: 73448616
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
|
Maybe some description of the custom columns you have; whether or not you have any that display the formats material is in.
|
02-21-2014, 08:40 PM | #3 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Move the calibre library to SSD.
But don't expect too much... I have a pretty sweet setup with a hybrid filesystem using bcache in Ubuntu 13.10 with 200 GB SATA 3 SSD write-back cache for two 3TB 7200RPM SATA 3 harddrives. So the entire library can be cached on SSD at once. It is a relatively new setup, I am still tweaking it. I may test to move the metadata.db - files to pure SSD. While I copy 100 books to another large library, on the same partition, the CPU-usage by calibre is 6-7% on average (8 core AMD Bulldozer) and the CPU never even clocks up to full speed. If I copy to a new and empty library even less, around 4% CPU-usage. But it still takes close to 5 minutes to copy 100 books. (3 custom text columns, no calculated columns, several virtual libraries, if that matter). If I copy to a new empty library it is one or two minutes faster. If I copy the folder with the library with 100 books to another place using the filemanager, it is instantaneous. It is only around 40MB after all. So I assume that it is all the database updating and creation of folders and so on, that takes time? Edit: I tested with both source and destionation metadata.db on pure SSD. It was a little faster, around 2 1/2 minutes. Not enough for me to bother. It is fast enough as it is... Last edited by Adoby; 02-21-2014 at 09:21 PM. |
02-21-2014, 09:02 PM | #4 |
creator of calibre
Posts: 43,850
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Adding books has to:
1) read metadata from the files, which for some file formats can be ver slow, for example, epub can require rendering of html in a browser to generate covers. 2) Search the entire existing library for duplicates 3) Update lots of fields in the database There is no way that adding books is every going to come close to the performance of copying files in the filesystem. |
02-21-2014, 09:14 PM | #5 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Yes, I wouldn't expect it to either.
But considering that the actual filecopying is almost instantaneous, it is surprising that the CPU usage isn't higher. Well below 10%. But then I only used top to gauge the CPU-usage... Correction. When copying to a large (+6000 books) destination library the CPU usage goes up to 16%. Last edited by Adoby; 02-21-2014 at 09:18 PM. |
02-21-2014, 09:53 PM | #6 |
Well trained by Cats
Posts: 29,791
Karma: 54830978
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
How often does one add a BIG glob of books to a Library?
Nowadays my big batch is when , I add my (almost) monthly Baen bundle of 6-9 books and that is done before I get back from the Kitchen coffee pot. |
02-21-2014, 10:45 PM | #7 |
I'll always be a newb
Posts: 20
Karma: 10
Join Date: Feb 2012
Device: Kindle Paperwhite & Touch
|
|
02-21-2014, 10:48 PM | #8 | |
null operator (he/him)
Posts: 20,565
Karma: 26954694
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Quote:
I did some tests on a 6GB I5 system, running windows 7-sp1, it has two 2TB SATA2, NTFS internal drives C: & E:, both WD Caviar Blacks drive E: is compressed. At the time I did the tests MSE was running (no exclusions), and I was rebuilding the search indexes (including calibre libraries) with the Indexer Backoff feature OFF - that means the indexer is running at normal priority. I used a large file (1.3GB AVI) because book files are small and file operations on a small files are difficult to time with a stop watch. Using standard Windows I copied the avi from E: to C: in 15.5 seconds. I renamed the avi (author - title) on Drive C and added it to Library A on E: in 17.5 seconds. A 2 second overhead to create the author and book folders, and add the necessary rows to the database tables etc, seems reasonable to me. I then moved the 'book' from Library A to Library B, also on drive E in 41.4 secs; libraries A and B are clones, so same columns etc. A similar operation (ie move a folder from one parent to another) using the same data on the same drive in Windows its too fast to time. The difference is that Windows makes use of hardlinks to effect the move, whereas Calibre does a copy+delete. When I copied the same data from folder A to folder B on drive E with Windows it took 44.2 secs - 2.8 seconds slower, probably because of head movement and cache differences. If I change the Title on that book, it's also too fast to time. I think that's because calibre does the move from folder "Wrong Title (20)" to folder "Right Title (20)" using hardlinks rather than copy/delete. Its a bit tricky to do that on a library move, because the two libraries may be on different drives and hardlinks don't work across drives. Kovid would have to implement two types of move books - one that works between different drives and one that works on the same drive - then there's the problem that not all file systems support hardlinking. BR Last edited by BetterRed; 02-21-2014 at 10:58 PM. |
|
02-22-2014, 01:29 PM | #9 |
Groupie
Posts: 171
Karma: 10
Join Date: Dec 2008
Device: Likebook Mars
|
I have an i3 3.3Ghz, 8Gbs ram, 7Tb's of HDD. I tried adding 155,000 files into Calibre, it was managing about 1% an hour.
|
02-23-2014, 02:44 AM | #10 | |
US Navy, Retired
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
Personally I would recommend a 1000 at a time, followed by fixing the metadata for that batch before moving on to the next batch. Fixing the metadata is far from automatic, it requires eyes on the data and books. Once the batch is complete move onto the next. Good Luck. |
|
07-08-2014, 04:04 PM | #11 |
I'll always be a newb
Posts: 20
Karma: 10
Join Date: Feb 2012
Device: Kindle Paperwhite & Touch
|
I've played around with sizes of transfers and using books that have both epub and mobi formats it takes from 7 to 10 seconds per book in a copy. Blocks of a 1000 books just crawl. With 32Gb of memory and a super fact processor, I still think we should have a way to speed this up a bit.
|
07-08-2014, 05:53 PM | #12 |
null operator (he/him)
Posts: 20,565
Karma: 26954694
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
@Motomaggot - might be worth looking at this thread, Deleting duplicate folders. In particular I draw your attention to Kovid's statement in post #19.
It would be interesting to know what the difference would be if you turned off the settings relating to duplicates in the Adding Books->The Add Process tab (assuming they're set) BR |
07-08-2014, 09:56 PM | #13 |
Well trained by Cats
Posts: 29,791
Karma: 54830978
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
Remember: Quad cores gain you little with Calibre, its Single threaded.
Fast IO helps |
07-20-2014, 02:15 PM | #14 |
Connoisseur
Posts: 60
Karma: 2016606
Join Date: Apr 2012
Device: android tablets
|
I find it best to do a large add of about 5000 to 10000 overnight. I just have to look through the duplicate list when I wake up.
|
10-08-2014, 08:32 AM | #15 |
Member
Posts: 17
Karma: 10
Join Date: Sep 2013
Device: playbook
|
+1 for the performance issue. I think it seems to be a recurrent problem.
I've checked my ressource and like other said, my cpu/memory/disk usage does not go higher than 15% and I do have a core i7. I understand that I'm sure there is a lot to do in the background and lot of reading from metadata and writting to db. However, do we know exactly what's the slowest operation ? @kovidgoyal 1) It seems that we should need to do a bit of profiling to really target what's the bottleneck. One thing that come to my mind is maybe trying to make a better use of the multi-threading. 2) Is all the import operation through python scripts or it's in the core calibre ? I currently have a windows box so might need a linux distro if want to look at the core (base in the installation dev comments). 3) I'm willing to give some of my time to look at this issue, if you can maybe point me out at where the add books take place ? |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
touchscreen performance | sebigbos | PocketBook | 6 | 03-21-2012 01:02 PM |
PRS-650 Sluggish performance | SeaBookGuy | Sony Reader | 11 | 12-12-2011 10:26 PM |
Scalability and performance | blither | Calibre | 1 | 08-10-2010 08:31 PM |
DR800 Performance | jjansen | iRex | 18 | 08-03-2010 09:10 PM |
Feature Request: Moving and not copying when adding files... | Chuwiey | Calibre | 7 | 07-03-2010 05:35 AM |