Adding books to a large database

[Student1]

I was wondering if there is no easier way to update the db. i have about 30000 books and i want to add 16000 more... it is really hard to do it. It keeps crashing, not responding and when it finally adds the books it often goes into not responding mode and it never comes back. When i manually stop it then i realise some books did not make into the db...

Is there an easier way than to use the gui to add new books, it is just so long to do (over 1 week to add all those books) and the crashes keeps me restarting over and over.

I decided to cut the books into various packs so its easier to do, but even with that some crashes keep preventing me from continuing.

Any tips on doing this are welcome, as i am at wits end!

thanks!

N.b: i'm wondering why is it so long? Isn t there a way to just read the metainfo and simply copy each book into a directory? Updating a mysql db is extremely fast, and its not like the books are being entered entirely in the db, just the metainfo so i'm wondering why its taking so long to just enter the title, author etc into the db. Just a question not criticism .

[sirbruce]

Calibre seems to do a linear check for duplicates for every file, and the time it takes to do this seems to grow exponentially. If there were a way to turn off check for duplicates, the import might go faster, but otherwise, no there's no easy way to add new books to a large database.

[Student1]

Originally Posted by sirbruce

Calibre seems to do a linear check for duplicates for every file, and the time it takes to do this seems to grow exponentially. If there were a way to turn off check for duplicates, the import might go faster, but otherwise, no there's no easy way to add new books to a large database.

Guess i'll keep trying with smaller batches... but boy its frustrating to come back after 6 hours and see that calibre has crashed or that it found some duplicates, you click on "ok" and it goes into none responsive mode to never come back...

I thin you are right, the check for duplicates is causing some problems, turning it off might be a good solution. In any case it really doesn t matter as at the end it asks you if you want to include the duplicates... i always say yes as i deal with various format, so its kind of redundant.

[darkmonk]

While my book collection is not quite as large, I have had the same problem. I just split them into folders by their first letter. I have not yet finished them all - I'm currently on L. It does take a while. And it would be nice if we could turn off the check for duplicates until they are all added - then do a single check for them all. I would also like a way to download the metadata and cover with an "I'm feeling Lucky" Button. Getting so many books at once can take a while, eh?

[Student1]

You can say that again, lol ages is putting it mildly ! I split them into 21 sets... i m at set 3... boys its going to take time lol

[kovidgoyal]

patches are welcome

[roger64]

Could some standard defragmentation software help in some way ?

[kovidgoyal]

Not really, it's a pretty simple problem, say you have 20000 books in your database, and are adding 100 new books

Then calibre has to make 20,000 * 100 comparisons to check for duplicates. That's presumably what's slowing it down.

[Xenophon]

Originally Posted by kovidgoyal

Not really, it's a pretty simple problem, say you have 20000 books in your database, and are adding 100 new books

Then calibre has to make 20,000 * 100 comparisons to check for duplicates. That's presumably what's slowing it down.

Surely it need only make log2(20,000) * 100 comparisons, right? And you could optimize even that by hashing over whatever you are using for the comparison first. And...

Care to exchange a bit of email?

Xenophon

[sirbruce]

I was just getting ready to suggest hashing myself. But in the meantime an option to turn off check for duplicates would be good enough.

[kovidgoyal]

Originally Posted by Xenophon

Surely it need only make log2(20,000) * 100 comparisons, right? And you could optimize even that by hashing over whatever you are using for the comparison first. And...

Care to exchange a bit of email?

Xenophon

Like I said, patches are welcome This is not a high enough priority for me to spend time optimizing.

[BinaryTB]

Thought about maybe switching to an SQLite database?

[Student1]

My programming days are like a distant memory... last time i touched some code was on mangos server 2 years ago ! What language is calibre being programmed on just for fun? Who knows might check it if someone doesn't beat me to it!

[pilotbob]

Originally Posted by Spooon69

Thought about maybe switching to an SQLite database?

I think that is what he is already using.

BOb

[BinaryTB]

Originally Posted by pilotbob

I think that is what he is already using.

BOb

Ah, you're right, I didn't bother checking the program directory.