Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 07-09-2013, 01:52 AM   #1
mikeloeven
Junior Member
mikeloeven began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Jul 2013
Device: windows 7
why is calibre slow on batch operations?

can someone tell me how to speed up calibres add to library operation. it should not take 10 minutes to add 20 books totaling 12 MB the sheer speed and power of modern computers can move this amount of data in under a second so why is calibre dragging ass?
mikeloeven is offline   Reply With Quote
Old 07-09-2013, 07:25 AM   #2
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
If you have Preferences->Add Books->Automatically convert books to the current output format checked then its converting each book - and conversions can sometimes be slow - especially on large PDFs.

BR
BetterRed is offline   Reply With Quote
Old 07-09-2013, 08:12 AM   #3
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Calibre not only has to move the data, depending on your settings calibre also has to explode/unzip the books an examine the contents. And perhaps convert and zip again. And it has to do this in a way that ensures that the calibre library doesn't loose data.

Calibre sometimes automatically does a lot of very heavy lifting, depending on the preferences set.

It is without any doubt the most demanding program I have installed on my computer, perhaps with the exception of Handbrake for video conversion. But Handbrake doesn't put a load on the I/O like calibre does, mainly cpu.
Adoby is offline   Reply With Quote
Old 07-09-2013, 10:11 AM   #4
mbovenka
Wizard
mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.
 
Posts: 2,013
Karma: 13471689
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Sage
I don't find book import all that slow. But what I have wondered about is why batch metadata updates are so much slower than single-book metadata updates.

Say, correcting a title or author sort for a few books. Do it 1 book at a time, and they take less than a second each. Do even two at a time (thus using the batch metadata update screen) and it takes far longer than (N x <1s).

I can't really explain that one.
mbovenka is offline   Reply With Quote
Old 07-09-2013, 12:15 PM   #5
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Try to minimize the tag browser to the left. I have have it minimized all the time, except when I actually need it.

Usually when you update metadata only some records in the database is changed. But when you update authors or title, then the books have to be moved to a new folder in the calibre library.

Also when the books are moved they are really first copied to the new place and then deleted at the old. Not just renamed. The reason is that this reduces the chance of data corruption if the process is interrupted for some reason. If only one book is updated, then the disc caches can most likely be fully used to speed it up. But maybe not as likely if several books are updated.

More RAM will most likely help.
Adoby is offline   Reply With Quote
Old 07-09-2013, 01:22 PM   #6
mikeloeven
Junior Member
mikeloeven began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Jul 2013
Device: windows 7
so if it is all the extra work calibre is doing how do i dumb it down a bit to streamline the process because i really don't care what format the book is in as long as it is supported.

on a side note will copying an entire calibre library from one computer to another fresh install bring the preferences or is that stored in appdata seperatly
mikeloeven is offline   Reply With Quote
Old 07-09-2013, 01:26 PM   #7
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Originally Posted by mikeloeven View Post
so if it is all the extra work calibre is doing how do i dumb it down a bit to streamline the process because i really don't care what format the book is in as long as it is supported.

on a side note will copying an entire calibre library from one computer to another fresh install bring the preferences or is that stored in appdata seperatly
Some preference are stored in the library db, global ones are stored in the appdata directory.

Best to copy both.

Helen
speakingtohe is offline   Reply With Quote
Old 07-09-2013, 01:30 PM   #8
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Originally Posted by mbovenka View Post
I don't find book import all that slow. But what I have wondered about is why batch metadata updates are so much slower than single-book metadata updates.

Say, correcting a title or author sort for a few books. Do it 1 book at a time, and they take less than a second each. Do even two at a time (thus using the batch metadata update screen) and it takes far longer than (N x <1s).

I can't really explain that one.
Wish my metadata individual downloads took less than a second. More like 20 to 50 seconds for me, and it is not connection speed or hardware I am pretty sure. Probably I have too may sources configured.

Helen
speakingtohe is offline   Reply With Quote
Old 07-09-2013, 01:46 PM   #9
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by speakingtohe View Post
Quote:
Originally Posted by mbovenka View Post
I don't find book import all that slow. But what I have wondered about is why batch metadata updates are so much slower than single-book metadata updates.

Say, correcting a title or author sort for a few books. Do it 1 book at a time, and they take less than a second each. Do even two at a time (thus using the batch metadata update screen) and it takes far longer than (N x <1s).

I can't really explain that one.
Wish my metadata individual downloads took less than a second. More like 20 to 50 seconds for me, and it is not connection speed or hardware I am pretty sure. Probably I have too may sources configured.
Look a little closer. he was referring to metadata updates not metadata downloads.
DoctorOhh is offline   Reply With Quote
Old 07-09-2013, 02:31 PM   #10
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Originally Posted by DoctorOhh View Post
Look a little closer. he was referring to metadata updates not metadata downloads.
Ah I understand. Turning of the tag browser does indeed make a big difference in this regard, in the GUI at least although I love the tag browser.

Kovid Goyal explained to me that the tag browser took a lot of time to update every time a field was changed. Even with the tag browser off though it takes a long time to do a bulk metadata change of even one field in two records, so there is something else in play here.

Helen
speakingtohe is offline   Reply With Quote
Old 07-09-2013, 02:37 PM   #11
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by speakingtohe View Post
Ah I understand. Turning of the tag browser does indeed make a big difference in this regard, in the GUI at least although I love the tag browser.

Kovid Goyal explained to me that the tag browser took a lot of time to update every time a field was changed. Even with the tag browser off though it takes a long time to do a bulk metadata change of even one field in two records, so there is something else in play here.
Sounds like good advice.
DoctorOhh is offline   Reply With Quote
Old 07-10-2013, 04:37 AM   #12
mbovenka
Wizard
mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.
 
Posts: 2,013
Karma: 13471689
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Sage
Quote:
Originally Posted by Adoby View Post
Try to minimize the tag browser to the left. I have have it minimized all the time, except when I actually need it.

Usually when you update metadata only some records in the database is changed. But when you update authors or title, then the books have to be moved to a new folder in the calibre library.

Also when the books are moved they are really first copied to the new place and then deleted at the old. Not just renamed. The reason is that this reduces the chance of data corruption if the process is interrupted for some reason. If only one book is updated, then the disc caches can most likely be fully used to speed it up. But maybe not as likely if several books are updated.

More RAM will most likely help.
I'm already running with the tag browser minimized. And, yeah, I know what Calibre does when I update an author or a title, so I expect it to be slower than say, adding a tag or something like that which only needs to update the Calibre database. But that is not it, because actions like that are a lot slower as well when done in batch as compared to doing them one at a time.

It does seem database size related, because the effect is a lot less when I do it on my small library (which is about a sixth the size of my main one)
mbovenka is offline   Reply With Quote
Old 07-10-2013, 04:48 AM   #13
mbovenka
Wizard
mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.
 
Posts: 2,013
Karma: 13471689
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Sage
Quote:
Originally Posted by speakingtohe View Post
Even with the tag browser off though it takes a long time to do a bulk metadata change of even one field in two records, so there is something else in play here.
Yup, that's exactly what I'm seeing. Pick two books, do a metadata update through the individual update screen and it's poof, poof, done. Do it with the bulk update screen and it's wait...wait....wait....done. Far more than twice as long as a single update.

It's database size related, too; the effect is a lot less if your library is smaller.
mbovenka is offline   Reply With Quote
Old 07-10-2013, 06:04 AM   #14
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Single updates do not cause a resort, bulk updates do. Sorting in large libraries is slow.
kovidgoyal is offline   Reply With Quote
Old 07-10-2013, 08:06 AM   #15
mbovenka
Wizard
mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.
 
Posts: 2,013
Karma: 13471689
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Sage
Quote:
Originally Posted by kovidgoyal View Post
Single updates do not cause a resort, bulk updates do. Sorting in large libraries is slow.
Aha. Yeah, that would explain it. Is there a way around it, or is it just a question of 'Sorry, you gotta live with it'?
mbovenka is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Batch Converting Ebooks with Calibre and Python jmeb Conversion 17 10-20-2018 11:16 AM
Batch transfer books from Kindle to Calibre engt Library Management 7 08-21-2014 06:54 AM
PRS-T1 questions on collections and slow operations Jade Aislin Sony Reader 9 08-14-2012 01:49 PM
Calibre Portable batch file. soleilos Calibre 2 08-20-2011 11:52 AM
calibre - very slow conversion, very slow on PRS cremofix Calibre 3 06-10-2009 04:21 PM


All times are GMT -4. The time now is 02:59 AM.


MobileRead.com is a privately owned, operated and funded community.