04-05-2009, 02:10 PM | #16 |
hopeless n00b
Posts: 5,111
Karma: 19597086
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9
|
|
04-06-2009, 04:24 PM | #17 |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
|
Advert | |
|
04-06-2009, 06:48 PM | #18 |
Connoisseur
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
|
Just a tip, I've got 15 000 books added on mine. Took bloody well forever - and when I went over them again, I found 62 unique books calibre didn't add the first time.
|
04-06-2009, 11:55 PM | #19 | |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
Quote:
When it times out do you leave it alone? Or did you cancel it? Found out after it finds duplicates and ask you to press ok it takes 3 to 4 hours more to insert the books... now i just leave it be until its finished. Anyway still have a way to go... at 28576 books and still a whole lot to go... Last edited by Student1; 04-07-2009 at 12:07 AM. |
|
04-07-2009, 01:49 AM | #20 |
Connoisseur
Posts: 73
Karma: 74
Join Date: Mar 2009
Location: Boston, MA
Device: Kindle 3
|
Just throwing out ideas here since I'm not familiar with SQLite (I'm a Python/Oracle guy myself), but does SQLite have merge and/or upsert capabilities? That's pretty quick usually.
Better yet, bulk (single transaction) inserts into a stage table which would then merge into the main table would by far be the quickest I think. I'm sure these ideas have been already thought of (they're the norm for this kind of stuff), but like I said, thinking out loud for any patchers out there. |
Advert | |
|
04-07-2009, 12:02 PM | #21 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Spooon69's comment made me realize that there is a simple optimization (3 lines of code) I can do, that should help with this (basically calibre maintains an in memory cache of title, authors and other basic metadata which I can use instead of making an SQL query).
So try the next release. |
04-07-2009, 04:11 PM | #22 |
curmudgeon
Posts: 1,481
Karma: 5748190
Join Date: Jun 2006
Location: Redwood City, CA USA
Device: Kobo Aura HD, (ex)nook, (ex)PRS-700, (ex)PRS-500
|
If python has reasonable collections libraries available (surely it does!), you can also switch that in-memory cache to be a balanced tree or other nicely optimized structure. That'd give n-log-n performance -- another factor-of-14 speedup for Student1's case. If that takes more than a couple more lines of code, I'd be really surprised.
|
04-07-2009, 04:19 PM | #23 | |
Grand Sorcerer
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
|
Quote:
BOb |
|
04-07-2009, 04:31 PM | #24 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You can have either speed or decreased memory consumption, not both. I vote for speed since the size of a typical cached record is about 300 bytes on average (it of course depends on average title/author/publisher/tags lengths) which gives about 5MB of RAM for 20000 records.
|
04-07-2009, 04:35 PM | #25 | |
Grand Sorcerer
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
|
Quote:
BOb |
|
04-07-2009, 04:53 PM | #26 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Actually I just ran a test on a library of 20k books with random metadata, adding a 100 books (also with random metadata to it) and the check for duplicates is definitely not causing the slowdown. Indeed the cumulative time spent in the check for dupliactes function is not even in the top 10.
As further corroboration, the adding of the 100 books took less than a minute. So the slowdown is elsewhere. |
04-07-2009, 05:03 PM | #27 | |
Grand Sorcerer
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
|
Quote:
Also, everyone should DISABLE on-access Virus scanning on their calibre folder... that sucker will REALLY slow things down. BOb |
|
04-07-2009, 05:09 PM | #28 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
As best as I can tell the performance for adding a single book to a library is not significantly degraded, there is a long wait because after adding, the library has to be re-sorted, but that's about it.
As for adding multiple books, there may be some other problem coming into play, but as I've already spent more time than can afford on this... |
04-07-2009, 05:44 PM | #29 | |
Provocateur
Posts: 1,859
Karma: 505847
Join Date: Feb 2009
Location: Columbus, OH
Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS
|
Quote:
The slowdown seems clearly exponential to me; if I add a lot of books, the progress bar moves rapidly at first, then slower and slower. |
|
04-07-2009, 05:45 PM | #30 |
Grand Sorcerer
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
"Database Disk Image is malformed" when connecting to Calibre after adding new books | djorkboy | Kobo Reader | 13 | 10-07-2010 08:47 PM |
"Database Disk Image is malformed" when connecting to Calibre after adding new books | djorkboy | Calibre | 1 | 10-06-2010 08:59 PM |
how to remove database names with no books? | setherd | Calibre | 3 | 02-23-2010 09:26 PM |
6.11 problem with adding large book collection | acekyoto | Calibre | 25 | 10-01-2009 12:25 PM |
Adding "Wanted" books to Calibre database? | itimpi | Calibre | 6 | 11-26-2008 03:14 PM |