Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 02-21-2014, 06:36 PM   #1
Motomaggot
I'll always be a newb
Motomaggot began at the beginning.
 
Motomaggot's Avatar
 
Posts: 20
Karma: 10
Join Date: Feb 2012
Device: Kindle Paperwhite & Touch
Performance when adding and moving

I've read countless posts on this forum and around the web on performance and so far, nothing has really worked. I have an i7 4770 processor with 16Gb of RAM with an SSD boot and RAID 1 4Tb drives for data.

Its seems that adding books to a library is painfully slow, and the copy and delete is just as bad.

Any suggestions on what I can adjust to make Calibre run faster no matter how much resources it eats up on the machine.

Matthew
Motomaggot is offline   Reply With Quote
Old 02-21-2014, 07:28 PM   #2
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 6,992
Karma: 44005669
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
Maybe some description of the custom columns you have; whether or not you have any that display the formats material is in.
PeterT is offline   Reply With Quote
Old 02-21-2014, 08:40 PM   #3
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,107
Karma: 5186678
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
Move the calibre library to SSD.

But don't expect too much...

I have a pretty sweet setup with a hybrid filesystem using bcache in Ubuntu 13.10 with 200 GB SATA 3 SSD write-back cache for two 3TB 7200RPM SATA 3 harddrives. So the entire library can be cached on SSD at once. It is a relatively new setup, I am still tweaking it. I may test to move the metadata.db - files to pure SSD.

While I copy 100 books to another large library, on the same partition, the CPU-usage by calibre is 6-7% on average (8 core AMD Bulldozer) and the CPU never even clocks up to full speed. If I copy to a new and empty library even less, around 4% CPU-usage.

But it still takes close to 5 minutes to copy 100 books. (3 custom text columns, no calculated columns, several virtual libraries, if that matter). If I copy to a new empty library it is one or two minutes faster.

If I copy the folder with the library with 100 books to another place using the filemanager, it is instantaneous. It is only around 40MB after all. So I assume that it is all the database updating and creation of folders and so on, that takes time?

Edit: I tested with both source and destionation metadata.db on pure SSD. It was a little faster, around 2 1/2 minutes. Not enough for me to bother. It is fast enough as it is...

Last edited by Adoby; 02-21-2014 at 09:21 PM.
Adoby is offline   Reply With Quote
Old 02-21-2014, 09:02 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,135
Karma: 5381911
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Adding books has to:

1) read metadata from the files, which for some file formats can be ver slow, for example, epub can require rendering of html in a browser to generate covers.

2) Search the entire existing library for duplicates

3) Update lots of fields in the database

There is no way that adding books is every going to come close to the performance of copying files in the filesystem.
kovidgoyal is online now   Reply With Quote
Old 02-21-2014, 09:14 PM   #5
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,107
Karma: 5186678
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
Yes, I wouldn't expect it to either.

But considering that the actual filecopying is almost instantaneous, it is surprising that the CPU usage isn't higher. Well below 10%. But then I only used top to gauge the CPU-usage...

Correction. When copying to a large (+6000 books) destination library the CPU usage goes up to 16%.

Last edited by Adoby; 02-21-2014 at 09:18 PM.
Adoby is offline   Reply With Quote
Old 02-21-2014, 09:53 PM   #6
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,870
Karma: 5654321
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
How often does one add a BIG glob of books to a Library?

Nowadays my big batch is when , I add my (almost) monthly Baen bundle of 6-9 books and that is done before I get back from the Kitchen coffee pot.
theducks is offline   Reply With Quote
Old 02-21-2014, 10:45 PM   #7
Motomaggot
I'll always be a newb
Motomaggot began at the beginning.
 
Motomaggot's Avatar
 
Posts: 20
Karma: 10
Join Date: Feb 2012
Device: Kindle Paperwhite & Touch
Quote:
Originally Posted by PeterT View Post
Maybe some description of the custom columns you have; whether or not you have any that display the formats material is in.
I'm not doing anything special with columns at all. I guess the level of performance I'm seeing is just normal.
Motomaggot is offline   Reply With Quote
Old 02-21-2014, 10:48 PM   #8
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,629
Karma: 2183656
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by Motomaggot View Post
I've read countless posts on this forum and around the web on performance and so far, nothing has really worked. I have an i7 4770 processor with 16Gb of RAM with an SSD boot and RAID 1 4Tb drives for data.

Its seems that adding books to a library is painfully slow, and the copy and delete is just as bad.

Any suggestions on what I can adjust to make Calibre run faster no matter how much resources it eats up on the machine.

Matthew
@Matthew - could you to quantify "painfully slow"

I did some tests on a 6GB I5 system, running windows 7-sp1, it has two 2TB SATA2, NTFS internal drives C: & E:, both WD Caviar Blacks drive E: is compressed. At the time I did the tests MSE was running (no exclusions), and I was rebuilding the search indexes (including calibre libraries) with the Indexer Backoff feature OFF - that means the indexer is running at normal priority. I used a large file (1.3GB AVI) because book files are small and file operations on a small files are difficult to time with a stop watch.

Using standard Windows I copied the avi from E: to C: in 15.5 seconds. I renamed the avi (author - title) on Drive C and added it to Library A on E: in 17.5 seconds. A 2 second overhead to create the author and book folders, and add the necessary rows to the database tables etc, seems reasonable to me.

I then moved the 'book' from Library A to Library B, also on drive E in 41.4 secs; libraries A and B are clones, so same columns etc. A similar operation (ie move a folder from one parent to another) using the same data on the same drive in Windows its too fast to time. The difference is that Windows makes use of hardlinks to effect the move, whereas Calibre does a copy+delete. When I copied the same data from folder A to folder B on drive E with Windows it took 44.2 secs - 2.8 seconds slower, probably because of head movement and cache differences.

If I change the Title on that book, it's also too fast to time. I think that's because calibre does the move from folder "Wrong Title (20)" to folder "Right Title (20)" using hardlinks rather than copy/delete. Its a bit tricky to do that on a library move, because the two libraries may be on different drives and hardlinks don't work across drives. Kovid would have to implement two types of move books - one that works between different drives and one that works on the same drive - then there's the problem that not all file systems support hardlinking.

BR

Last edited by BetterRed; 02-21-2014 at 10:58 PM.
BetterRed is offline   Reply With Quote
Old 02-22-2014, 01:29 PM   #9
Vortex
Connoisseur
Vortex began at the beginning.
 
Posts: 81
Karma: 10
Join Date: Dec 2008
Device: Panasonic Toughbook CF18 + uBook
I have an i3 3.3Ghz, 8Gbs ram, 7Tb's of HDD. I tried adding 155,000 files into Calibre, it was managing about 1% an hour.
Vortex is offline   Reply With Quote
Old 02-23-2014, 02:44 AM   #10
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,861
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by Vortex View Post
I have an i3 3.3Ghz, 8Gbs ram, 7Tb's of HDD. I tried adding 155,000 files into Calibre, it was managing about 1% an hour.
It is nuts to try and do these in a giant batch. Try lots of 1000, 2000 or 5000 instead. You may find a magic level that maximizes the speed at which books will be added.

Personally I would recommend a 1000 at a time, followed by fixing the metadata for that batch before moving on to the next batch. Fixing the metadata is far from automatic, it requires eyes on the data and books. Once the batch is complete move onto the next.

Good Luck.
DoctorOhh is offline   Reply With Quote
Old 07-08-2014, 04:04 PM   #11
Motomaggot
I'll always be a newb
Motomaggot began at the beginning.
 
Motomaggot's Avatar
 
Posts: 20
Karma: 10
Join Date: Feb 2012
Device: Kindle Paperwhite & Touch
I've played around with sizes of transfers and using books that have both epub and mobi formats it takes from 7 to 10 seconds per book in a copy. Blocks of a 1000 books just crawl. With 32Gb of memory and a super fact processor, I still think we should have a way to speed this up a bit.
Motomaggot is offline   Reply With Quote
Old 07-08-2014, 05:53 PM   #12
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,629
Karma: 2183656
Join Date: Mar 2012
Location: NSW Australia
Device: none
@Motomaggot - might be worth looking at this thread, Deleting duplicate folders. In particular I draw your attention to Kovid's statement in post #19.

It would be interesting to know what the difference would be if you turned off the settings relating to duplicates in the Adding Books->The Add Process tab (assuming they're set)

BR
BetterRed is offline   Reply With Quote
Old 07-08-2014, 09:56 PM   #13
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,870
Karma: 5654321
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Remember: Quad cores gain you little with Calibre, its Single threaded.
Fast IO helps
theducks is offline   Reply With Quote
Old 07-20-2014, 02:15 PM   #14
Inferno
Connoisseur
Inferno began at the beginning.
 
Posts: 58
Karma: 10
Join Date: Apr 2012
Device: android tablets
I find it best to do a large add of about 5000 to 10000 overnight. I just have to look through the duplicate list when I wake up.
Inferno is offline   Reply With Quote
Old 10-08-2014, 08:32 AM   #15
abuck
Member
abuck began at the beginning.
 
Posts: 17
Karma: 10
Join Date: Sep 2013
Device: playbook
+1 for the performance issue. I think it seems to be a recurrent problem.

I've checked my ressource and like other said, my cpu/memory/disk usage does not go higher than 15% and I do have a core i7.

I understand that I'm sure there is a lot to do in the background and lot of reading from metadata and writting to db. However, do we know exactly what's the slowest operation ?

@kovidgoyal

1) It seems that we should need to do a bit of profiling to really target what's the bottleneck. One thing that come to my mind is maybe trying to make a better use of the multi-threading.

2) Is all the import operation through python scripts or it's in the core calibre ? I currently have a windows box so might need a linux distro if want to look at the core (base in the installation dev comments).

3) I'm willing to give some of my time to look at this issue, if you can maybe point me out at where the add books take place ?
abuck is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
touchscreen performance sebigbos PocketBook 6 03-21-2012 01:02 PM
PRS-650 Sluggish performance SeaBookGuy Sony Reader 11 12-12-2011 10:26 PM
Scalability and performance blither Calibre 1 08-10-2010 08:31 PM
DR800 Performance jjansen iRex 18 08-03-2010 09:10 PM
Feature Request: Moving and not copying when adding files... Chuwiey Calibre 7 07-03-2010 05:35 AM


All times are GMT -4. The time now is 02:03 AM.


MobileRead.com is a privately owned, operated and funded community.