Importing 4GB of books requires 8GB+ of space. Why?

MelBr · 03-12-2013, 10:04 PM

I've noticed a peculiar thing. Calibre, when importing, first copies all the files to a temp folder (on my machine it's /var/folders/ys/mk3r41nj2jv20t0cw7_f_tvw0010gn/T/calibre_0.9.22_tmp_fOK9I5/ ) and then it copies files again to Calibre Library folder (it doesn't do a mv). Then it deletes files from temp. Why is it doing this? It's so inefficient and slow. It took me forever to import these books and majority of time was spent on I/O operations.

Adoby · 03-13-2013, 01:19 PM

It makes it less likely that you end up with a partially successful import. Either everything will be imported or nothing. Calibre also works with a temporary copy of the database file. This may save you a lot of headache if the import failed halfway.

Calibre also works the same way when for instance bulk converting books or sending to device.

It is an approach that favour data integrity, reliability and safety over performance. I suspect that it is intentional.

Import fewer books in several installments and you should be fine. Optimally no more books than what can be held in the disk caches. Install more RAM to increase that amount. Make sure that the temp folder is on a fast disk with plenty of free space. I've actually put mine on a 8 GB RAM disk using the tempfs feature in Linux. Works fine, but plenty of RAM for disk caches should also help.

And seriously, how often do you import huge amounts of books? Go do something else during the import or take a nap. Perhaps read a book?

theducks · 03-13-2013, 01:32 PM

Quote:

Originally Posted by Adoby

Perhaps read a book?

Big jobs (of any kind) are why I have more than 1 computer. God invented KVM's for a reason

audeojude · 03-18-2013, 11:02 PM

lol... god invented the i7-3770 and 32 gigs of ram for multiple virtual systems andbig jobs at the same time and still running smoothly for everything else

I need to put my tempfs out on a ram disk.. bet it would speed stuff up some.

MelBr · 03-31-2013, 03:44 PM

Quote:

Originally Posted by Adoby

....
It is an approach that favour data integrity, reliability and safety over performance. I suspect that it is intentional.

How does copying something twice improve data integrity?

Quote:

And seriously, how often do you import huge amounts of books? Go do something else during the import or take a nap. Perhaps read a book?

Often.

And it's not the issue of time but of space. Who said anything about time? I'm using SSD disk for most of the imports and don't have much space.

At one point in time in Calibre's import procedure, there are three copies of a file on the disk. If that file is large, you'll need a ton of space for absolutely no gain in anything.

Adoby · 03-31-2013, 06:08 PM

Before the library is updated, all the books are fully converted/processed and waiting in the temp folder. Same with the database file. So normally either all books are converted/imported or none. Even if calibre suddenly crash, the power is broken, the drive is full or memory runs out during a batch process, the library is still most likely fully intact.

I use a RAM disk for temp, and have even less space, but manage without any problems with 8GB temp storage.

03-12-2013, 10:04 PM	#1
MelBr Zealot Posts: 105 Karma: 414068 Join Date: Feb 2013 Device: iPad Pro, Kobo Aura One	Importing 4GB of books requires 8GB+ of space. Why? I've noticed a peculiar thing. Calibre, when importing, first copies all the files to a temp folder (on my machine it's /var/folders/ys/mk3r41nj2jv20t0cw7_f_tvw0010gn/T/calibre_0.9.22_tmp_fOK9I5/ ) and then it copies files again to Calibre Library folder (it doesn't do a mv). Then it deletes files from temp. Why is it doing this? It's so inefficient and slow. It took me forever to import these books and majority of time was spent on I/O operations.

03-13-2013, 01:19 PM	#2
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	It makes it less likely that you end up with a partially successful import. Either everything will be imported or nothing. Calibre also works with a temporary copy of the database file. This may save you a lot of headache if the import failed halfway. Calibre also works the same way when for instance bulk converting books or sending to device. It is an approach that favour data integrity, reliability and safety over performance. I suspect that it is intentional. Import fewer books in several installments and you should be fine. Optimally no more books than what can be held in the disk caches. Install more RAM to increase that amount. Make sure that the temp folder is on a fast disk with plenty of free space. I've actually put mine on a 8 GB RAM disk using the tempfs feature in Linux. Works fine, but plenty of RAM for disk caches should also help. And seriously, how often do you import huge amounts of books? Go do something else during the import or take a nap. Perhaps read a book? Last edited by Adoby; 03-13-2013 at 01:29 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
I am buying some USB (4GB ~ 8GB), does the brand of the USB matters much ?	bookmeal	Lounge	1	01-10-2012 12:48 PM
DR800 Only 4GB of 8GB SD-Card useable?	DeReader	iRex	6	05-18-2010 06:16 PM
Nokia N95 8GB VS iPhone Apple 8GB	ryanjone	Which one should I buy?	27	12-12-2009 05:33 AM
Lexar Memory Stick Pro Duo 4GB - $3 (AR) & 8GB $13 (AR) @ Fry's	chumbucket	Sony Reader	0	08-26-2008 02:08 PM
new CF card (Transcend 8gb) 0mb free space	zerospinboson	iRex	4	04-18-2008 04:39 AM

03-18-2013, 11:02 PM	#4
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	lol... god invented the i7-3770 and 32 gigs of ram for multiple virtual systems andbig jobs at the same time and still running smoothly for everything else I need to put my tempfs out on a ram disk.. bet it would speed stuff up some.

03-31-2013, 06:08 PM	#6
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Before the library is updated, all the books are fully converted/processed and waiting in the temp folder. Same with the database file. So normally either all books are converted/imported or none. Even if calibre suddenly crash, the power is broken, the drive is full or memory runs out during a batch process, the library is still most likely fully intact. I use a RAM disk for temp, and have even less space, but manage without any problems with 8GB temp storage.

Advert

Advert