Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 05-22-2009, 12:34 PM   #1
snowwraith
Junior Member
snowwraith began at the beginning.
 
Posts: 2
Karma: 10
Join Date: May 2009
Device: Sony PRS-505
*incredibly* slow importing of Ebooks

Hello everyone.

I had hoped that my first posting on this forum regarding Calibre would be a positive one since, based on the software brief, this piece of software should be amazing and everything that the Sony's software isn't.

However, 3 *days* in and my collection of approximately 13,000 Ebooks is *still* being imported into Calibre (version 5.12) under Windows XP (SP3). After the initial import "finished" this afternoon I was presented with a prompt regarding duplicate books. I know these aren't duplicates and were probably due to naming conventions used in some of the titles, so I've allowed Calibre to add these. Now Calibre is presumably importing this unknown quantity of titles to the database, only this time with no progress bar due to the screen not updating - leaving just a white hole where the previous window had been.

To fend off any questions regarding harddrive speed I can copy the source files from 1 fixed permanent drive to the destination drive, another fixed drive, in 8 minutes. By all accounts the PDF files that my books are formatted as are not being converted (I did a binary file comparison of source and destination files of some books already done by Calibre) and therefore the importing process seems to be hung up somewhere in the creation of the database.

So, is this importing process normal? Why does it take so long to create entries in the database? I daren't kill off Calibre now through the task manager since I can see it's still grinding away via the CPU load against its task and I'd hate to lose the 3 days already lost via the import.

Perhaps as a future update, rather than show the % of import completed it should show a number of books completed out of the total identified by the initial scan? And the importing process needs some *serious* optimising, otherwise all the fine work that everyone has contributed to the package will be for nought since potential users won't get past the initial importing stage.

I'll keep waiting to see what happens in the hope Calibre finally finishes doing its importing and that I can then use the software.

Thank you for reading.

Last edited by snowwraith; 05-22-2009 at 12:37 PM.
snowwraith is offline   Reply With Quote
Old 05-22-2009, 01:04 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,352
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
This will be fixed in the 0.6 release that is around the corner
kovidgoyal is offline   Reply With Quote
Old 05-22-2009, 05:04 PM   #3
snowwraith
Junior Member
snowwraith began at the beginning.
 
Posts: 2
Karma: 10
Join Date: May 2009
Device: Sony PRS-505
In the end I decided to kill the Calibre task since the not knowing how long the process of adding the duplicate entries would take - due to no status bar showing after leaving it for a couple hours.

After a reboot I restarted Calibre and had a look to see if it still ran. Loaded up for around 14,000 ebooks. I had a little fiddle with it since my initial and primary task was to tag all the free books I'd acquired that had no meta tags on them. It seems tagging takes roughly a minute to apply once the OK button is selected and this leads me to believe this is why the import takes so long. Part of the importing process is that it picks up the basic meta tags from the filename and writes this info to the database. It stands to reason that if the meta tag writing takes approximately 1 minute and you've got a large number of books then this will take a monstrous amount of time which explains why my PC was taking so long to do this.

For the moment I've regrettably uninstalled Calibre since the re-tagging tasks I have in mind can't be done quickly. Imagine marking up 60 books with 1 author; that's an hours worth of time taken up.

As I said in my original posting it's a shame I couldn't be more positive about Calibre since it looks like it has a promising future. But something as important as the initial importing of books and their meta tagging is a feature that should have been fixed right from the get-go.

However, I will keep my eyes open on the updates and hope for a future release that will be useable.

Thank you for your response and for taking the plunge into writing a solution that the big guns (e.g. Sony) have clearly failed to provide to their end users. I look forward to future releases.
snowwraith is offline   Reply With Quote
Old 05-22-2009, 07:16 PM   #4
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,022
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
I can tag around 100 books with a new author in a few seconds. Also, I am not sure why your import is slow - yesterday I imported 50 books in about 2 minutes which seems to be an order of magnitude different to what you are seeing.

What I have found is that Calibre is VERY slow in such actions if you are using a network drive. In that case it can take minutes per book rather than seconds. In otherwords it take around 100 times as long as doing a similar action with the Calibre database on a local drive. That is a lot more than one would expect from the different performance of network drives compared to local drives. I am guessing t is something to do with the way that sqlite performs over a network.
itimpi is offline   Reply With Quote
Old 05-23-2009, 04:05 AM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,352
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The slowdown only happens for the import of large numbers of books > 1000 IIRC in a single import.
kovidgoyal is offline   Reply With Quote
Old 05-23-2009, 05:08 AM   #6
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,022
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Just to clarify - my comment about network drives being very slow applies even to single book imports or metadata changes. About 2 orders of magnitude slower than doing the same actions against a local drive. In these cases, the front-end running Calibre is Windows XP Pro, and the back-end is a Samba 2 networked file server, with a Gigabit LAN connection between them.

I have been meaning for some time to run from source with timing statements added so I can see exactly which statements are running slowly when using network drives, but I have not yet got around to it. I am sure if they are identified it would be easy to apply some significant tuning.

I think I will wait until the first 0.6 release comes out as you may already have solved this issue for that release. If not I will try and find the time to look further.

Last edited by itimpi; 05-23-2009 at 05:17 AM.
itimpi is offline   Reply With Quote
Old 05-23-2009, 06:08 AM   #7
sirbruce
Provocateur
sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.
 
sirbruce's Avatar
 
Posts: 1,859
Karma: 505847
Join Date: Feb 2009
Location: Columbus, OH
Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS
My first guess is the problem is Samba, which sucks. But that doesn't mean Calibre can't be made more efficient to negate that to some extent.
sirbruce is offline   Reply With Quote
Old 05-23-2009, 07:55 AM   #8
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,022
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by sirbruce View Post
My first guess is the problem is Samba, which sucks. But that doesn't mean Calibre can't be made more efficient to negate that to some extent.
Actually, my suspicion is sqlite. I have heard that it is hard to get it to perform well over a network.

I will do a check on that by trying to run from a Windows share to see how that performs.
itimpi is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
mass-importing ebooks in multiple formats? hakan42 Calibre 18 02-28-2010 08:06 PM
Incredibly poor Terry Pratchett Epub offering Alfy News 29 08-12-2009 09:17 PM
calibre - very slow conversion, very slow on PRS cremofix Calibre 3 06-10-2009 04:21 PM
importing ebooks iconeo Calibre 4 05-05-2009 03:35 AM
Scrolling - incredibly awful dso371 Bookeen 33 02-21-2008 07:08 AM


All times are GMT -4. The time now is 08:06 PM.


MobileRead.com is a privately owned, operated and funded community.