04-03-2009, 09:38 PM | #1 |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
Adding books to a large database
I was wondering if there is no easier way to update the db. i have about 30000 books and i want to add 16000 more... it is really hard to do it. It keeps crashing, not responding and when it finally adds the books it often goes into not responding mode and it never comes back. When i manually stop it then i realise some books did not make into the db...
Is there an easier way than to use the gui to add new books, it is just so long to do (over 1 week to add all those books) and the crashes keeps me restarting over and over. I decided to cut the books into various packs so its easier to do, but even with that some crashes keep preventing me from continuing. Any tips on doing this are welcome, as i am at wits end! thanks! N.b: i'm wondering why is it so long? Isn t there a way to just read the metainfo and simply copy each book into a directory? Updating a mysql db is extremely fast, and its not like the books are being entered entirely in the db, just the metainfo so i'm wondering why its taking so long to just enter the title, author etc into the db. Just a question not criticism . Last edited by Student1; 04-03-2009 at 09:42 PM. |
04-03-2009, 09:40 PM | #2 |
Provocateur
Posts: 1,859
Karma: 505847
Join Date: Feb 2009
Location: Columbus, OH
Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS
|
Calibre seems to do a linear check for duplicates for every file, and the time it takes to do this seems to grow exponentially. If there were a way to turn off check for duplicates, the import might go faster, but otherwise, no there's no easy way to add new books to a large database.
|
Advert | |
|
04-03-2009, 09:50 PM | #3 | |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
Quote:
I thin you are right, the check for duplicates is causing some problems, turning it off might be a good solution. In any case it really doesn t matter as at the end it asks you if you want to include the duplicates... i always say yes as i deal with various format, so its kind of redundant. Last edited by Student1; 04-03-2009 at 09:53 PM. |
|
04-04-2009, 11:50 AM | #4 |
Connoisseur
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
|
While my book collection is not quite as large, I have had the same problem. I just split them into folders by their first letter. I have not yet finished them all - I'm currently on L. It does take a while. And it would be nice if we could turn off the check for duplicates until they are all added - then do a single check for them all. I would also like a way to download the metadata and cover with an "I'm feeling Lucky" Button. Getting so many books at once can take a while, eh?
|
04-04-2009, 01:45 PM | #5 |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
You can say that again, lol ages is putting it mildly ! I split them into 21 sets... i m at set 3... boys its going to take time lol
|
Advert | |
|
04-04-2009, 04:26 PM | #6 |
creator of calibre
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
patches are welcome
|
04-04-2009, 05:10 PM | #7 |
Wizard
Posts: 2,608
Karma: 3000161
Join Date: Jan 2009
Device: Kindle PW3 (wifi)
|
Could some standard defragmentation software help in some way ?
|
04-04-2009, 05:17 PM | #8 |
creator of calibre
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Not really, it's a pretty simple problem, say you have 20000 books in your database, and are adding 100 new books
Then calibre has to make 20,000 * 100 comparisons to check for duplicates. That's presumably what's slowing it down. |
04-04-2009, 05:29 PM | #9 | |
curmudgeon
Posts: 1,481
Karma: 5748190
Join Date: Jun 2006
Location: Redwood City, CA USA
Device: Kobo Aura HD, (ex)nook, (ex)PRS-700, (ex)PRS-500
|
Quote:
Care to exchange a bit of email? Xenophon |
|
04-04-2009, 05:54 PM | #10 |
Provocateur
Posts: 1,859
Karma: 505847
Join Date: Feb 2009
Location: Columbus, OH
Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS
|
I was just getting ready to suggest hashing myself. But in the meantime an option to turn off check for duplicates would be good enough.
|
04-04-2009, 05:57 PM | #11 |
creator of calibre
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Like I said, patches are welcome This is not a high enough priority for me to spend time optimizing.
|
04-05-2009, 04:22 AM | #12 |
Connoisseur
Posts: 73
Karma: 74
Join Date: Mar 2009
Location: Boston, MA
Device: Kindle 3
|
Thought about maybe switching to an SQLite database?
|
04-05-2009, 01:42 PM | #13 |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
My programming days are like a distant memory... last time i touched some code was on mangos server 2 years ago ! What language is calibre being programmed on just for fun? Who knows might check it if someone doesn't beat me to it!
Last edited by Student1; 04-05-2009 at 01:55 PM. |
04-05-2009, 01:54 PM | #14 |
Grand Sorcerer
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
|
|
04-05-2009, 01:57 PM | #15 |
Connoisseur
Posts: 73
Karma: 74
Join Date: Mar 2009
Location: Boston, MA
Device: Kindle 3
|
|
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
"Database Disk Image is malformed" when connecting to Calibre after adding new books | djorkboy | Kobo Reader | 13 | 10-07-2010 08:47 PM |
"Database Disk Image is malformed" when connecting to Calibre after adding new books | djorkboy | Calibre | 1 | 10-06-2010 08:59 PM |
how to remove database names with no books? | setherd | Calibre | 3 | 02-23-2010 09:26 PM |
6.11 problem with adding large book collection | acekyoto | Calibre | 25 | 10-01-2009 12:25 PM |
Adding "Wanted" books to Calibre database? | itimpi | Calibre | 6 | 11-26-2008 03:14 PM |