why is calibre slow on batch operations?

mikeloeven · 07-09-2013, 01:52 AM

can someone tell me how to speed up calibres add to library operation. it should not take 10 minutes to add 20 books totaling 12 MB the sheer speed and power of modern computers can move this amount of data in under a second so why is calibre dragging ass?

BetterRed · 07-09-2013, 07:25 AM

If you have Preferences->Add Books->Automatically convert books to the current output format checked then its converting each book - and conversions can sometimes be slow - especially on large PDFs.

BR

Adoby · 07-09-2013, 08:12 AM

Calibre not only has to move the data, depending on your settings calibre also has to explode/unzip the books an examine the contents. And perhaps convert and zip again. And it has to do this in a way that ensures that the calibre library doesn't loose data.

Calibre sometimes automatically does a lot of very heavy lifting, depending on the preferences set.

It is without any doubt the most demanding program I have installed on my computer, perhaps with the exception of Handbrake for video conversion. But Handbrake doesn't put a load on the I/O like calibre does, mainly cpu.

mbovenka · 07-09-2013, 10:11 AM

I don't find book import all that slow. But what I have wondered about is why batch metadata updates are so much slower than single-book metadata updates.

Say, correcting a title or author sort for a few books. Do it 1 book at a time, and they take less than a second each. Do even two at a time (thus using the batch metadata update screen) and it takes far longer than (N x <1s).

I can't really explain that one.

Adoby · 07-09-2013, 12:15 PM

Try to minimize the tag browser to the left. I have have it minimized all the time, except when I actually need it.

Usually when you update metadata only some records in the database is changed. But when you update authors or title, then the books have to be moved to a new folder in the calibre library.

Also when the books are moved they are really first copied to the new place and then deleted at the old. Not just renamed. The reason is that this reduces the chance of data corruption if the process is interrupted for some reason. If only one book is updated, then the disc caches can most likely be fully used to speed it up. But maybe not as likely if several books are updated.

More RAM will most likely help.

mikeloeven · 07-09-2013, 01:22 PM

so if it is all the extra work calibre is doing how do i dumb it down a bit to streamline the process because i really don't care what format the book is in as long as it is supported.

on a side note will copying an entire calibre library from one computer to another fresh install bring the preferences or is that stored in appdata seperatly

speakingtohe · 07-09-2013, 01:26 PM

Quote:

Originally Posted by mikeloeven

so if it is all the extra work calibre is doing how do i dumb it down a bit to streamline the process because i really don't care what format the book is in as long as it is supported.

on a side note will copying an entire calibre library from one computer to another fresh install bring the preferences or is that stored in appdata seperatly

Some preference are stored in the library db, global ones are stored in the appdata directory.

Best to copy both.

Helen

speakingtohe · 07-09-2013, 01:30 PM

Quote:

Originally Posted by mbovenka

I don't find book import all that slow. But what I have wondered about is why batch metadata updates are so much slower than single-book metadata updates.

Say, correcting a title or author sort for a few books. Do it 1 book at a time, and they take less than a second each. Do even two at a time (thus using the batch metadata update screen) and it takes far longer than (N x <1s).

I can't really explain that one.

Wish my metadata individual downloads took less than a second. More like 20 to 50 seconds for me, and it is not connection speed or hardware I am pretty sure. Probably I have too may sources configured.

Helen

DoctorOhh · 07-09-2013, 01:46 PM

Quote:

Originally Posted by speakingtohe

Quote:

Originally Posted by mbovenka

I don't find book import all that slow. But what I have wondered about is why batch metadata updates are so much slower than single-book metadata updates.

Say, correcting a title or author sort for a few books. Do it 1 book at a time, and they take less than a second each. Do even two at a time (thus using the batch metadata update screen) and it takes far longer than (N x <1s).

I can't really explain that one.

Wish my metadata individual downloads took less than a second. More like 20 to 50 seconds for me, and it is not connection speed or hardware I am pretty sure. Probably I have too may sources configured.

Look a little closer. he was referring to metadata updates not metadata downloads.

speakingtohe · 07-09-2013, 02:31 PM

Quote:

Originally Posted by DoctorOhh

Look a little closer. he was referring to metadata updates not metadata downloads.

Ah I understand. Turning of the tag browser does indeed make a big difference in this regard, in the GUI at least although I love the tag browser.

Kovid Goyal explained to me that the tag browser took a lot of time to update every time a field was changed. Even with the tag browser off though it takes a long time to do a bulk metadata change of even one field in two records, so there is something else in play here.

Helen

DoctorOhh · 07-09-2013, 02:37 PM

Quote:

Originally Posted by speakingtohe

Ah I understand. Turning of the tag browser does indeed make a big difference in this regard, in the GUI at least although I love the tag browser.

Kovid Goyal explained to me that the tag browser took a lot of time to update every time a field was changed. Even with the tag browser off though it takes a long time to do a bulk metadata change of even one field in two records, so there is something else in play here.

Sounds like good advice.

mbovenka · 07-10-2013, 04:37 AM

Quote:

Originally Posted by Adoby

Try to minimize the tag browser to the left. I have have it minimized all the time, except when I actually need it.

Usually when you update metadata only some records in the database is changed. But when you update authors or title, then the books have to be moved to a new folder in the calibre library.

Also when the books are moved they are really first copied to the new place and then deleted at the old. Not just renamed. The reason is that this reduces the chance of data corruption if the process is interrupted for some reason. If only one book is updated, then the disc caches can most likely be fully used to speed it up. But maybe not as likely if several books are updated.

More RAM will most likely help.

I'm already running with the tag browser minimized. And, yeah, I know what Calibre does when I update an author or a title, so I expect it to be slower than say, adding a tag or something like that which only needs to update the Calibre database. But that is not it, because actions like that are a lot slower as well when done in batch as compared to doing them one at a time.

It does seem database size related, because the effect is a lot less when I do it on my small library (which is about a sixth the size of my main one)

mbovenka · 07-10-2013, 04:48 AM

Quote:

Originally Posted by speakingtohe

Even with the tag browser off though it takes a long time to do a bulk metadata change of even one field in two records, so there is something else in play here.

Yup, that's exactly what I'm seeing. Pick two books, do a metadata update through the individual update screen and it's poof, poof, done. Do it with the bulk update screen and it's wait...wait....wait....done. Far more than twice as long as a single update.

It's database size related, too; the effect is a lot less if your library is smaller.

kovidgoyal · 07-10-2013, 06:04 AM

Single updates do not cause a resort, bulk updates do. Sorting in large libraries is slow.

mbovenka · 07-10-2013, 08:06 AM

Quote:

Originally Posted by kovidgoyal

Single updates do not cause a resort, bulk updates do. Sorting in large libraries is slow.

Aha. Yeah, that would explain it. Is there a way around it, or is it just a question of 'Sorry, you gotta live with it'?

07-09-2013, 01:52 AM	#1
mikeloeven Junior Member Posts: 4 Karma: 10 Join Date: Jul 2013 Device: windows 7	why is calibre slow on batch operations? can someone tell me how to speed up calibres add to library operation. it should not take 10 minutes to add 20 books totaling 12 MB the sheer speed and power of modern computers can move this amount of data in under a second so why is calibre dragging ass?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Batch Converting Ebooks with Calibre and Python	jmeb	Conversion	17	10-20-2018 11:16 AM
Batch transfer books from Kindle to Calibre	engt	Library Management	7	08-21-2014 06:54 AM
PRS-T1 questions on collections and slow operations	Jade Aislin	Sony Reader	9	08-14-2012 01:49 PM
Calibre Portable batch file.	soleilos	Calibre	2	08-20-2011 11:52 AM
calibre - very slow conversion, very slow on PRS	cremofix	Calibre	3	06-10-2009 04:21 PM

07-09-2013, 07:25 AM	#2
BetterRed null operator (he/him) Posts: 21,616 Karma: 29710338 Join Date: Mar 2012 Location: Sydney Australia Device: none	If you have Preferences->Add Books->Automatically convert books to the current output format checked then its converting each book - and conversions can sometimes be slow - especially on large PDFs. BR

07-09-2013, 08:12 AM	#3
Adoby Handy Elephant Posts: 1,737 Karma: 26785684 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Samsung Galaxy Tab S8 Ultra	Calibre not only has to move the data, depending on your settings calibre also has to explode/unzip the books an examine the contents. And perhaps convert and zip again. And it has to do this in a way that ensures that the calibre library doesn't loose data. Calibre sometimes automatically does a lot of very heavy lifting, depending on the preferences set. It is without any doubt the most demanding program I have installed on my computer, perhaps with the exception of Handbrake for video conversion. But Handbrake doesn't put a load on the I/O like calibre does, mainly cpu.

07-09-2013, 10:11 AM	#4
mbovenka Wizard Posts: 2,075 Karma: 14079267 Join Date: Oct 2007 Location: Almere, The Netherlands Device: Kobo Sage	I don't find book import all that slow. But what I have wondered about is why batch metadata updates are so much slower than single-book metadata updates. Say, correcting a title or author sort for a few books. Do it 1 book at a time, and they take less than a second each. Do even two at a time (thus using the batch metadata update screen) and it takes far longer than (N x <1s). I can't really explain that one.

07-09-2013, 12:15 PM	#5
Adoby Handy Elephant Posts: 1,737 Karma: 26785684 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Samsung Galaxy Tab S8 Ultra	Try to minimize the tag browser to the left. I have have it minimized all the time, except when I actually need it. Usually when you update metadata only some records in the database is changed. But when you update authors or title, then the books have to be moved to a new folder in the calibre library. Also when the books are moved they are really first copied to the new place and then deleted at the old. Not just renamed. The reason is that this reduces the chance of data corruption if the process is interrupted for some reason. If only one book is updated, then the disc caches can most likely be fully used to speed it up. But maybe not as likely if several books are updated. More RAM will most likely help.

07-09-2013, 01:22 PM	#6
mikeloeven Junior Member Posts: 4 Karma: 10 Join Date: Jul 2013 Device: windows 7	so if it is all the extra work calibre is doing how do i dumb it down a bit to streamline the process because i really don't care what format the book is in as long as it is supported. on a side note will copying an entire calibre library from one computer to another fresh install bring the preferences or is that stored in appdata seperatly

07-10-2013, 06:04 AM	#14
kovidgoyal creator of calibre Posts: 45,182 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Single updates do not cause a resort, bulk updates do. Sorting in large libraries is slow.

Advert

Advert