Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 04-03-2009, 09:38 PM   #1
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
Adding books to a large database

I was wondering if there is no easier way to update the db. i have about 30000 books and i want to add 16000 more... it is really hard to do it. It keeps crashing, not responding and when it finally adds the books it often goes into not responding mode and it never comes back. When i manually stop it then i realise some books did not make into the db...

Is there an easier way than to use the gui to add new books, it is just so long to do (over 1 week to add all those books) and the crashes keeps me restarting over and over.

I decided to cut the books into various packs so its easier to do, but even with that some crashes keep preventing me from continuing.

Any tips on doing this are welcome, as i am at wits end!

thanks!

N.b: i'm wondering why is it so long? Isn t there a way to just read the metainfo and simply copy each book into a directory? Updating a mysql db is extremely fast, and its not like the books are being entered entirely in the db, just the metainfo so i'm wondering why its taking so long to just enter the title, author etc into the db. Just a question not criticism .

Last edited by Student1; 04-03-2009 at 09:42 PM.
Student1 is offline   Reply With Quote
Old 04-03-2009, 09:40 PM   #2
sirbruce
Provocateur
sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.
 
sirbruce's Avatar
 
Posts: 1,859
Karma: 505847
Join Date: Feb 2009
Location: Columbus, OH
Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS
Calibre seems to do a linear check for duplicates for every file, and the time it takes to do this seems to grow exponentially. If there were a way to turn off check for duplicates, the import might go faster, but otherwise, no there's no easy way to add new books to a large database.
sirbruce is offline   Reply With Quote
Advert
Old 04-03-2009, 09:50 PM   #3
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
Quote:
Originally Posted by sirbruce View Post
Calibre seems to do a linear check for duplicates for every file, and the time it takes to do this seems to grow exponentially. If there were a way to turn off check for duplicates, the import might go faster, but otherwise, no there's no easy way to add new books to a large database.
Guess i'll keep trying with smaller batches... but boy its frustrating to come back after 6 hours and see that calibre has crashed or that it found some duplicates, you click on "ok" and it goes into none responsive mode to never come back...

I thin you are right, the check for duplicates is causing some problems, turning it off might be a good solution. In any case it really doesn t matter as at the end it asks you if you want to include the duplicates... i always say yes as i deal with various format, so its kind of redundant.

Last edited by Student1; 04-03-2009 at 09:53 PM.
Student1 is offline   Reply With Quote
Old 04-04-2009, 11:50 AM   #4
darkmonk
Connoisseur
darkmonk began at the beginning.
 
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
While my book collection is not quite as large, I have had the same problem. I just split them into folders by their first letter. I have not yet finished them all - I'm currently on L. It does take a while. And it would be nice if we could turn off the check for duplicates until they are all added - then do a single check for them all. I would also like a way to download the metadata and cover with an "I'm feeling Lucky" Button. Getting so many books at once can take a while, eh?
darkmonk is offline   Reply With Quote
Old 04-04-2009, 01:45 PM   #5
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
You can say that again, lol ages is putting it mildly ! I split them into 21 sets... i m at set 3... boys its going to take time lol
Student1 is offline   Reply With Quote
Advert
Old 04-04-2009, 04:26 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
patches are welcome
kovidgoyal is offline   Reply With Quote
Old 04-04-2009, 05:10 PM   #7
roger64
Wizard
roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.roger64 ought to be getting tired of karma fortunes by now.
 
Posts: 2,608
Karma: 3000161
Join Date: Jan 2009
Device: Kindle PW3 (wifi)
Could some standard defragmentation software help in some way ?
roger64 is offline   Reply With Quote
Old 04-04-2009, 05:17 PM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Not really, it's a pretty simple problem, say you have 20000 books in your database, and are adding 100 new books

Then calibre has to make 20,000 * 100 comparisons to check for duplicates. That's presumably what's slowing it down.
kovidgoyal is offline   Reply With Quote
Old 04-04-2009, 05:29 PM   #9
Xenophon
curmudgeon
Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.
 
Xenophon's Avatar
 
Posts: 1,481
Karma: 5748190
Join Date: Jun 2006
Location: Redwood City, CA USA
Device: Kobo Aura HD, (ex)nook, (ex)PRS-700, (ex)PRS-500
Quote:
Originally Posted by kovidgoyal View Post
Not really, it's a pretty simple problem, say you have 20000 books in your database, and are adding 100 new books

Then calibre has to make 20,000 * 100 comparisons to check for duplicates. That's presumably what's slowing it down.
Surely it need only make log2(20,000) * 100 comparisons, right? And you could optimize even that by hashing over whatever you are using for the comparison first. And...

Care to exchange a bit of email?

Xenophon
Xenophon is offline   Reply With Quote
Old 04-04-2009, 05:54 PM   #10
sirbruce
Provocateur
sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.
 
sirbruce's Avatar
 
Posts: 1,859
Karma: 505847
Join Date: Feb 2009
Location: Columbus, OH
Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS
I was just getting ready to suggest hashing myself. But in the meantime an option to turn off check for duplicates would be good enough.
sirbruce is offline   Reply With Quote
Old 04-04-2009, 05:57 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by Xenophon View Post
Surely it need only make log2(20,000) * 100 comparisons, right? And you could optimize even that by hashing over whatever you are using for the comparison first. And...

Care to exchange a bit of email?

Xenophon
Like I said, patches are welcome This is not a high enough priority for me to spend time optimizing.
kovidgoyal is offline   Reply With Quote
Old 04-05-2009, 04:22 AM   #12
BinaryTB
Connoisseur
BinaryTB is on a distinguished road
 
BinaryTB's Avatar
 
Posts: 73
Karma: 74
Join Date: Mar 2009
Location: Boston, MA
Device: Kindle 3
Thought about maybe switching to an SQLite database?
BinaryTB is offline   Reply With Quote
Old 04-05-2009, 01:42 PM   #13
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
My programming days are like a distant memory... last time i touched some code was on mangos server 2 years ago ! What language is calibre being programmed on just for fun? Who knows might check it if someone doesn't beat me to it!

Last edited by Student1; 04-05-2009 at 01:55 PM.
Student1 is offline   Reply With Quote
Old 04-05-2009, 01:54 PM   #14
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by Spooon69 View Post
Thought about maybe switching to an SQLite database?
I think that is what he is already using.

BOb
pilotbob is offline   Reply With Quote
Old 04-05-2009, 01:57 PM   #15
BinaryTB
Connoisseur
BinaryTB is on a distinguished road
 
BinaryTB's Avatar
 
Posts: 73
Karma: 74
Join Date: Mar 2009
Location: Boston, MA
Device: Kindle 3
Quote:
Originally Posted by pilotbob View Post
I think that is what he is already using.

BOb
Ah, you're right, I didn't bother checking the program directory.
BinaryTB is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
"Database Disk Image is malformed" when connecting to Calibre after adding new books djorkboy Kobo Reader 13 10-07-2010 08:47 PM
"Database Disk Image is malformed" when connecting to Calibre after adding new books djorkboy Calibre 1 10-06-2010 08:59 PM
how to remove database names with no books? setherd Calibre 3 02-23-2010 09:26 PM
6.11 problem with adding large book collection acekyoto Calibre 25 10-01-2009 12:25 PM
Adding "Wanted" books to Calibre database? itimpi Calibre 6 11-26-2008 03:14 PM


All times are GMT -4. The time now is 08:16 AM.


MobileRead.com is a privately owned, operated and funded community.