Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 03-12-2013, 10:04 PM   #1
MelBr
Connoisseur
MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.
 
Posts: 82
Karma: 405216
Join Date: Feb 2013
Device: iPad
Importing 4GB of books requires 8GB+ of space. Why?

I've noticed a peculiar thing. Calibre, when importing, first copies all the files to a temp folder (on my machine it's /var/folders/ys/mk3r41nj2jv20t0cw7_f_tvw0010gn/T/calibre_0.9.22_tmp_fOK9I5/ ) and then it copies files again to Calibre Library folder (it doesn't do a mv). Then it deletes files from temp. Why is it doing this? It's so inefficient and slow. It took me forever to import these books and majority of time was spent on I/O operations.
MelBr is offline   Reply With Quote
Old 03-13-2013, 01:19 PM   #2
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,033
Karma: 3824376
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
It makes it less likely that you end up with a partially successful import. Either everything will be imported or nothing. Calibre also works with a temporary copy of the database file. This may save you a lot of headache if the import failed halfway.

Calibre also works the same way when for instance bulk converting books or sending to device.

It is an approach that favour data integrity, reliability and safety over performance. I suspect that it is intentional.

Import fewer books in several installments and you should be fine. Optimally no more books than what can be held in the disk caches. Install more RAM to increase that amount. Make sure that the temp folder is on a fast disk with plenty of free space. I've actually put mine on a 8 GB RAM disk using the tempfs feature in Linux. Works fine, but plenty of RAM for disk caches should also help.

And seriously, how often do you import huge amounts of books? Go do something else during the import or take a nap. Perhaps read a book?

Last edited by Adoby; 03-13-2013 at 01:29 PM.
Adoby is online now   Reply With Quote
Old 03-13-2013, 01:32 PM   #3
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,278
Karma: 5495472
Join Date: Aug 2009
Location: The (original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by Adoby View Post
Perhaps read a book?


Big jobs (of any kind) are why I have more than 1 computer. God invented KVM's for a reason
theducks is online now   Reply With Quote
Old 03-18-2013, 11:02 PM   #4
audeojude
Enthusiast
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 25
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
lol... god invented the i7-3770 and 32 gigs of ram for multiple virtual systems andbig jobs at the same time and still running smoothly for everything else

I need to put my tempfs out on a ram disk.. bet it would speed stuff up some.
audeojude is offline   Reply With Quote
Old 03-31-2013, 03:44 PM   #5
MelBr
Connoisseur
MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.MelBr ought to be getting tired of karma fortunes by now.
 
Posts: 82
Karma: 405216
Join Date: Feb 2013
Device: iPad
Quote:
Originally Posted by Adoby View Post
....
It is an approach that favour data integrity, reliability and safety over performance. I suspect that it is intentional.
How does copying something twice improve data integrity?


Quote:
And seriously, how often do you import huge amounts of books? Go do something else during the import or take a nap. Perhaps read a book?
Often.

And it's not the issue of time but of space. Who said anything about time? I'm using SSD disk for most of the imports and don't have much space.

At one point in time in Calibre's import procedure, there are three copies of a file on the disk. If that file is large, you'll need a ton of space for absolutely no gain in anything.
MelBr is offline   Reply With Quote
Old 03-31-2013, 06:08 PM   #6
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,033
Karma: 3824376
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
Before the library is updated, all the books are fully converted/processed and waiting in the temp folder. Same with the database file. So normally either all books are converted/imported or none. Even if calibre suddenly crash, the power is broken, the drive is full or memory runs out during a batch process, the library is still most likely fully intact.

I use a RAM disk for temp, and have even less space, but manage without any problems with 8GB temp storage.
Adoby is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
I am buying some USB (4GB ~ 8GB), does the brand of the USB matters much ? bookmeal Lounge 1 01-10-2012 12:48 PM
DR800 Only 4GB of 8GB SD-Card useable? DeReader iRex 6 05-18-2010 06:16 PM
Nokia N95 8GB VS iPhone Apple 8GB ryanjone Which one should I buy? 27 12-12-2009 05:33 AM
Lexar Memory Stick Pro Duo 4GB - $3 (AR) & 8GB $13 (AR) @ Fry's chumbucket Sony Reader 0 08-26-2008 02:08 PM
new CF card (Transcend 8gb) 0mb free space zerospinboson iRex 4 04-18-2008 04:39 AM


All times are GMT -4. The time now is 02:10 PM.


MobileRead.com is a privately owned, operated and funded community.