Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 04-05-2009, 02:10 PM   #16
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 5,111
Karma: 19597086
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9
Quote:
Originally Posted by Student1 View Post
What language is calibre being programmed on just for fun?
Python.
ilovejedd is offline   Reply With Quote
Old 04-06-2009, 04:24 PM   #17
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
Quote:
Originally Posted by ilovejedd View Post
Python.
Hmm python... thats a language i've never even touched... don't expect any patch any time soon but i might still check it up when i have time!
Student1 is offline   Reply With Quote
Advert
Old 04-06-2009, 06:48 PM   #18
darkmonk
Connoisseur
darkmonk began at the beginning.
 
Posts: 58
Karma: 12
Join Date: Jan 2009
Device: none
Just a tip, I've got 15 000 books added on mine. Took bloody well forever - and when I went over them again, I found 62 unique books calibre didn't add the first time.
darkmonk is offline   Reply With Quote
Old 04-06-2009, 11:55 PM   #19
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
Quote:
Originally Posted by darkmonk View Post
Just a tip, I've got 15 000 books added on mine. Took bloody well forever - and when I went over them again, I found 62 unique books calibre didn't add the first time.
You were lucky to notice them! At 15 000 its not easy to look it up! I know i ll have to do it... and re arrange the titles and series... THATS going to take forever...

When it times out do you leave it alone? Or did you cancel it? Found out after it finds duplicates and ask you to press ok it takes 3 to 4 hours more to insert the books... now i just leave it be until its finished.

Anyway still have a way to go... at 28576 books and still a whole lot to go...

Last edited by Student1; 04-07-2009 at 12:07 AM.
Student1 is offline   Reply With Quote
Old 04-07-2009, 01:49 AM   #20
BinaryTB
Connoisseur
BinaryTB is on a distinguished road
 
BinaryTB's Avatar
 
Posts: 73
Karma: 74
Join Date: Mar 2009
Location: Boston, MA
Device: Kindle 3
Just throwing out ideas here since I'm not familiar with SQLite (I'm a Python/Oracle guy myself), but does SQLite have merge and/or upsert capabilities? That's pretty quick usually.

Better yet, bulk (single transaction) inserts into a stage table which would then merge into the main table would by far be the quickest I think.

I'm sure these ideas have been already thought of (they're the norm for this kind of stuff), but like I said, thinking out loud for any patchers out there.
BinaryTB is offline   Reply With Quote
Advert
Old 04-07-2009, 12:02 PM   #21
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Spooon69's comment made me realize that there is a simple optimization (3 lines of code) I can do, that should help with this (basically calibre maintains an in memory cache of title, authors and other basic metadata which I can use instead of making an SQL query).

So try the next release.
kovidgoyal is offline   Reply With Quote
Old 04-07-2009, 04:11 PM   #22
Xenophon
curmudgeon
Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.
 
Xenophon's Avatar
 
Posts: 1,481
Karma: 5748190
Join Date: Jun 2006
Location: Redwood City, CA USA
Device: Kobo Aura HD, (ex)nook, (ex)PRS-700, (ex)PRS-500
If python has reasonable collections libraries available (surely it does!), you can also switch that in-memory cache to be a balanced tree or other nicely optimized structure. That'd give n-log-n performance -- another factor-of-14 speedup for Student1's case. If that takes more than a couple more lines of code, I'd be really surprised.
Xenophon is offline   Reply With Quote
Old 04-07-2009, 04:19 PM   #23
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by Xenophon View Post
If python has reasonable collections libraries available (surely it does!), you can also switch that in-memory cache to be a balanced tree or other nicely optimized structure. That'd give n-log-n performance -- another factor-of-14 speedup for Student1's case. If that takes more than a couple more lines of code, I'd be really surprised.
Of course, I am not sure I want 20k book records cached in memory. I prefer a smaller memory footprint.

BOb
pilotbob is offline   Reply With Quote
Old 04-07-2009, 04:31 PM   #24
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by pilotbob View Post
Of course, I am not sure I want 20k book records cached in memory. I prefer a smaller memory footprint.

BOb
You can have either speed or decreased memory consumption, not both. I vote for speed since the size of a typical cached record is about 300 bytes on average (it of course depends on average title/author/publisher/tags lengths) which gives about 5MB of RAM for 20000 records.
kovidgoyal is offline   Reply With Quote
Old 04-07-2009, 04:35 PM   #25
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by kovidgoyal View Post
You can have either speed or decreased memory consumption, not both. I vote for speed since the size of a typical cached record is about 300 bytes on average (it of course depends on average title/author/publisher/tags lengths) which gives about 5MB of RAM for 20000 records.
I guess that's not too bad, considering that Firefox has about a 1GB working set.

BOb
pilotbob is offline   Reply With Quote
Old 04-07-2009, 04:53 PM   #26
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Actually I just ran a test on a library of 20k books with random metadata, adding a 100 books (also with random metadata to it) and the check for duplicates is definitely not causing the slowdown. Indeed the cumulative time spent in the check for dupliactes function is not even in the top 10.

As further corroboration, the adding of the 100 books took less than a minute. So the slowdown is elsewhere.
kovidgoyal is offline   Reply With Quote
Old 04-07-2009, 05:03 PM   #27
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by kovidgoyal View Post
As further corroboration, the adding of the 100 books took less than a minute. So the slowdown is elsewhere.
Yes, I always thought when adding alot of books the slowdown was in reading the metadata of each file.

Also, everyone should DISABLE on-access Virus scanning on their calibre folder... that sucker will REALLY slow things down.

BOb
pilotbob is offline   Reply With Quote
Old 04-07-2009, 05:09 PM   #28
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
As best as I can tell the performance for adding a single book to a library is not significantly degraded, there is a long wait because after adding, the library has to be re-sorted, but that's about it.

As for adding multiple books, there may be some other problem coming into play, but as I've already spent more time than can afford on this...
kovidgoyal is offline   Reply With Quote
Old 04-07-2009, 05:44 PM   #29
sirbruce
Provocateur
sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.sirbruce ought to be getting tired of karma fortunes by now.
 
sirbruce's Avatar
 
Posts: 1,859
Karma: 505847
Join Date: Feb 2009
Location: Columbus, OH
Device: Kindle Touch, Kindle 2, Kindle DX, iPhone 3GS
Quote:
Originally Posted by kovidgoyal View Post
As best as I can tell the performance for adding a single book to a library is not significantly degraded, there is a long wait because after adding, the library has to be re-sorted, but that's about it.
Is it resorting *after every add*? Perhaps that's the issue.

The slowdown seems clearly exponential to me; if I add a lot of books, the progress bar moves rapidly at first, then slower and slower.
sirbruce is offline   Reply With Quote
Old 04-07-2009, 05:45 PM   #30
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by sirbruce View Post
Is it resorting *after every add*? Perhaps that's the issue.
It resorts 1 after the add. It doesn't resort after each book is added. I think this was something Kovid changed no to long ago.

BOb
pilotbob is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
"Database Disk Image is malformed" when connecting to Calibre after adding new books djorkboy Kobo Reader 13 10-07-2010 08:47 PM
"Database Disk Image is malformed" when connecting to Calibre after adding new books djorkboy Calibre 1 10-06-2010 08:59 PM
how to remove database names with no books? setherd Calibre 3 02-23-2010 09:26 PM
6.11 problem with adding large book collection acekyoto Calibre 25 10-01-2009 12:25 PM
Adding "Wanted" books to Calibre database? itimpi Calibre 6 11-26-2008 03:14 PM


All times are GMT -4. The time now is 09:35 PM.


MobileRead.com is a privately owned, operated and funded community.