03-12-2013, 10:04 PM | #1 |
Zealot
Posts: 105
Karma: 414068
Join Date: Feb 2013
Device: iPad Pro, Kobo Aura One
|
Importing 4GB of books requires 8GB+ of space. Why?
I've noticed a peculiar thing. Calibre, when importing, first copies all the files to a temp folder (on my machine it's /var/folders/ys/mk3r41nj2jv20t0cw7_f_tvw0010gn/T/calibre_0.9.22_tmp_fOK9I5/ ) and then it copies files again to Calibre Library folder (it doesn't do a mv). Then it deletes files from temp. Why is it doing this? It's so inefficient and slow. It took me forever to import these books and majority of time was spent on I/O operations.
|
03-13-2013, 01:19 PM | #2 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
It makes it less likely that you end up with a partially successful import. Either everything will be imported or nothing. Calibre also works with a temporary copy of the database file. This may save you a lot of headache if the import failed halfway.
Calibre also works the same way when for instance bulk converting books or sending to device. It is an approach that favour data integrity, reliability and safety over performance. I suspect that it is intentional. Import fewer books in several installments and you should be fine. Optimally no more books than what can be held in the disk caches. Install more RAM to increase that amount. Make sure that the temp folder is on a fast disk with plenty of free space. I've actually put mine on a 8 GB RAM disk using the tempfs feature in Linux. Works fine, but plenty of RAM for disk caches should also help. And seriously, how often do you import huge amounts of books? Go do something else during the import or take a nap. Perhaps read a book? Last edited by Adoby; 03-13-2013 at 01:29 PM. |
03-13-2013, 01:32 PM | #3 |
Well trained by Cats
Posts: 29,803
Karma: 54830978
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
|
03-18-2013, 11:02 PM | #4 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
lol... god invented the i7-3770 and 32 gigs of ram for multiple virtual systems andbig jobs at the same time and still running smoothly for everything else
I need to put my tempfs out on a ram disk.. bet it would speed stuff up some. |
03-31-2013, 03:44 PM | #5 | ||
Zealot
Posts: 105
Karma: 414068
Join Date: Feb 2013
Device: iPad Pro, Kobo Aura One
|
Quote:
Quote:
And it's not the issue of time but of space. Who said anything about time? I'm using SSD disk for most of the imports and don't have much space. At one point in time in Calibre's import procedure, there are three copies of a file on the disk. If that file is large, you'll need a ton of space for absolutely no gain in anything. |
||
03-31-2013, 06:08 PM | #6 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Before the library is updated, all the books are fully converted/processed and waiting in the temp folder. Same with the database file. So normally either all books are converted/imported or none. Even if calibre suddenly crash, the power is broken, the drive is full or memory runs out during a batch process, the library is still most likely fully intact.
I use a RAM disk for temp, and have even less space, but manage without any problems with 8GB temp storage. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
I am buying some USB (4GB ~ 8GB), does the brand of the USB matters much ? | bookmeal | Lounge | 1 | 01-10-2012 12:48 PM |
DR800 Only 4GB of 8GB SD-Card useable? | DeReader | iRex | 6 | 05-18-2010 06:16 PM |
Nokia N95 8GB VS iPhone Apple 8GB | ryanjone | Which one should I buy? | 27 | 12-12-2009 05:33 AM |
Lexar Memory Stick Pro Duo 4GB - $3 (AR) & 8GB $13 (AR) @ Fry's | chumbucket | Sony Reader | 0 | 08-26-2008 02:08 PM |
new CF card (Transcend 8gb) 0mb free space | zerospinboson | iRex | 4 | 04-18-2008 04:39 AM |