*incredibly* slow importing of Ebooks

[snowwraith]

Hello everyone.

I had hoped that my first posting on this forum regarding Calibre would be a positive one since, based on the software brief, this piece of software should be amazing and everything that the Sony's software isn't.

However, 3 *days* in and my collection of approximately 13,000 Ebooks is *still* being imported into Calibre (version 5.12) under Windows XP (SP3). After the initial import "finished" this afternoon I was presented with a prompt regarding duplicate books. I know these aren't duplicates and were probably due to naming conventions used in some of the titles, so I've allowed Calibre to add these. Now Calibre is presumably importing this unknown quantity of titles to the database, only this time with no progress bar due to the screen not updating - leaving just a white hole where the previous window had been.

To fend off any questions regarding harddrive speed I can copy the source files from 1 fixed permanent drive to the destination drive, another fixed drive, in 8 minutes. By all accounts the PDF files that my books are formatted as are not being converted (I did a binary file comparison of source and destination files of some books already done by Calibre) and therefore the importing process seems to be hung up somewhere in the creation of the database.

So, is this importing process normal? Why does it take so long to create entries in the database? I daren't kill off Calibre now through the task manager since I can see it's still grinding away via the CPU load against its task and I'd hate to lose the 3 days already lost via the import.

Perhaps as a future update, rather than show the % of import completed it should show a number of books completed out of the total identified by the initial scan? And the importing process needs some *serious* optimising, otherwise all the fine work that everyone has contributed to the package will be for nought since potential users won't get past the initial importing stage.

I'll keep waiting to see what happens in the hope Calibre finally finishes doing its importing and that I can then use the software.

Thank you for reading.

[kovidgoyal]

This will be fixed in the 0.6 release that is around the corner

[snowwraith]

In the end I decided to kill the Calibre task since the not knowing how long the process of adding the duplicate entries would take - due to no status bar showing after leaving it for a couple hours.

After a reboot I restarted Calibre and had a look to see if it still ran. Loaded up for around 14,000 ebooks. I had a little fiddle with it since my initial and primary task was to tag all the free books I'd acquired that had no meta tags on them. It seems tagging takes roughly a minute to apply once the OK button is selected and this leads me to believe this is why the import takes so long. Part of the importing process is that it picks up the basic meta tags from the filename and writes this info to the database. It stands to reason that if the meta tag writing takes approximately 1 minute and you've got a large number of books then this will take a monstrous amount of time which explains why my PC was taking so long to do this.

For the moment I've regrettably uninstalled Calibre since the re-tagging tasks I have in mind can't be done quickly. Imagine marking up 60 books with 1 author; that's an hours worth of time taken up.

As I said in my original posting it's a shame I couldn't be more positive about Calibre since it looks like it has a promising future. But something as important as the initial importing of books and their meta tagging is a feature that should have been fixed right from the get-go.

However, I will keep my eyes open on the updates and hope for a future release that will be useable.

Thank you for your response and for taking the plunge into writing a solution that the big guns (e.g. Sony) have clearly failed to provide to their end users. I look forward to future releases.

[itimpi]

I can tag around 100 books with a new author in a few seconds. Also, I am not sure why your import is slow - yesterday I imported 50 books in about 2 minutes which seems to be an order of magnitude different to what you are seeing.

What I have found is that Calibre is VERY slow in such actions if you are using a network drive. In that case it can take minutes per book rather than seconds. In otherwords it take around 100 times as long as doing a similar action with the Calibre database on a local drive. That is a lot more than one would expect from the different performance of network drives compared to local drives. I am guessing t is something to do with the way that sqlite performs over a network.

[kovidgoyal]

The slowdown only happens for the import of large numbers of books > 1000 IIRC in a single import.

[itimpi]

Just to clarify - my comment about network drives being very slow applies even to single book imports or metadata changes. About 2 orders of magnitude slower than doing the same actions against a local drive. In these cases, the front-end running Calibre is Windows XP Pro, and the back-end is a Samba 2 networked file server, with a Gigabit LAN connection between them.

I have been meaning for some time to run from source with timing statements added so I can see exactly which statements are running slowly when using network drives, but I have not yet got around to it. I am sure if they are identified it would be easy to apply some significant tuning.

I think I will wait until the first 0.6 release comes out as you may already have solved this issue for that release. If not I will try and find the time to look further.

[sirbruce]

My first guess is the problem is Samba, which sucks. But that doesn't mean Calibre can't be made more efficient to negate that to some extent.

[itimpi]

Originally Posted by sirbruce

My first guess is the problem is Samba, which sucks. But that doesn't mean Calibre can't be made more efficient to negate that to some extent.

Actually, my suspicion is sqlite. I have heard that it is hard to get it to perform well over a network.

I will do a check on that by trying to run from a Windows share to see how that performs.