Very long startup time - Page 2

kovidgoyal · 11-03-2010, 01:13 PM

With that test, I get

Code:

RSS            Number of books

41.1            0

41.6            23

41.8            86

44.4            422


RSS - 0 size (MB)   Number of books    Size per book
0.5                   23                  22.3 KB
0.7                   86                  8.3  KB
3.3                   422                 8.0  KB (production library)

Which tells me that per book consumption is under 10KB and that subtracting the zero size doesn't remove some overhead that is not per book.

chaley · 11-03-2010, 01:38 PM

The numbers I see are somewhat consistent with yours. They are smaller, but I think that is because I am not counting python.

Code:


MB
47.8     0 books 
48.1    22 books 14.1KB/book
58.0   980 books 10.3KB/book (many cust cols, almost no data)
49.9  1400 books  1.5KB/book (few cust cols, lots of data, production)
54.5  4995 books  4.8KB/book (more cust cols, replicated books)

I don't understand why the production library is so much smaller per book. I ran the test several times and always came up with the same number. My guess is the overhead for custom columns?

Of course, this shows that the GUI accounts for a lot of the memory use. No real surprise, that.

kovidgoyal · 11-03-2010, 01:42 PM

If you have the time, try copying all custom column data into the comments field (at least the string variety) and deleting the cust cols.

I really don't see why cust cols should have a greater overhead than normal cols.

chaley · 11-03-2010, 02:04 PM

Quote:

Originally Posted by kovidgoyal

If you have the time, try copying all custom column data into the comments field (at least the string variety) and deleting the cust cols.

No point in copying, really. 95% of the fields are empty.

Numbers for the same library, as it was + with most cust cols removed + with all cust cols removed.

Code:

MB
58.0   980 books 10.3KB/book (many cust cols, almost no data)
52.7   980 books  5.0KB/book (one text/mult cust col, same 5 tags each)
50.1   980 books  2.4KB/book (no cust cols)

Quote:

I really don't see why cust cols should have a greater overhead than normal cols.

They probably don't. There are just more of them, which means more indices, more table records, more columns in the cache, etc.

itimpi · 11-03-2010, 02:06 PM

I have a library of around 10,000 books with an average of about 4 formats per book. Nearly all books have comments attached. Even on this library Calibre never seems to use more than about 250MB of RAM which is nothing to most modern machines. I am running Windows 7 Pro. I am very sceptical that total RAM is therefore much of a limit on performance.

kovidgoyal · 11-03-2010, 02:09 PM

Just for completeness, here are my figures with measuring only heap size, which means that the size of python etc. are not included

Code:


Heap (KB)    Number of books
35604           0
36054           23
36388           85
38948           422


Heap - 0 size (KB)   Number of books    Size per book
450                   23                  19.57 KB
784                   85                  9.2  KB
3344                  422                 7.9  KB (production library)

The numbers haven't changed significantly which means you have a lot more metadata per book than I do

You are on a 64 bit machine, right?

EDIT: Oops, I meant I have a lot more metadata per book than you do

chaley · 11-03-2010, 02:17 PM

The real question becomes 'do people who have performance problems have "modern machines"'?

Calibre total memory consumption for my production library of 1,400 books on Win7Pro is 280mb, of which some 140mb seems to vary with the db size. Even assuming minimum scaling up, a 35,000 book library (the OP's library size) would at least triple the variable size (probably more), taking total consumption to at least some 550MB. With those numbers, if he has a 1GB machine (these are still common), and especially if he is running other apps like mail, calibre will thrash. On a 2GB system, it likely will not.

chaley · 11-03-2010, 02:18 PM

Quote:

Originally Posted by kovidgoyal

You are on a 64 bit machine, right?

No. I am running in 32-bit mode, 4GB ram.

kovidgoyal · 11-03-2010, 02:19 PM

Quote:

Originally Posted by chaley

No. I am running in 32-bit mode, 4GB ram.

Ah that would explain the difference in per book sizes.

kovidgoyal · 11-03-2010, 02:43 PM

Hmm itimpi's 10k library uses 250MB and chaley's 1k library uses 280MB and both on W7??

chaley · 11-03-2010, 02:49 PM

Quote:

Originally Posted by kovidgoyal

Hmm itimpi's 10k library uses 250MB and chaley's 1k library uses 280MB and both on W7??

Apples & oranges.

My 280MB number is total, with programs & parallel processes. The comparable number is probably the 140MB one.

jbcohen · 11-03-2010, 02:53 PM

I feel your pain, I had a pandigital novel before it got stollen from my home and it had probably a 3 minute boot time.

Manichean · 11-03-2010, 05:50 PM

FWIW:
53 MB, 336 Books. System is Win XP 32, 1 GB RAM

Because there's no such thing as too much data.

theducks · 11-03-2010, 08:08 PM

1225 books Win XP SP3 1.5G RAM

Process manager says 282M Virt. Size, 101M working set

11-03-2010, 01:13 PM	#16
kovidgoyal creator of calibre Posts: 45,569 Karma: 28548962 Join Date: Oct 2006 Location: Mumbai, India Device: Various	With that test, I get Code: RSS Number of books 41.1 0 41.6 23 41.8 86 44.4 422 RSS - 0 size (MB) Number of books Size per book 0.5 23 22.3 KB 0.7 86 8.3 KB 3.3 422 8.0 KB (production library) Which tells me that per book consumption is under 10KB and that subtracting the zero size doesn't remove some overhead that is not per book.

11-03-2010, 01:38 PM	#17
chaley Grand Sorcerer Posts: 12,524 Karma: 8065948 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	The numbers I see are somewhat consistent with yours. They are smaller, but I think that is because I am not counting python. Code: MB 47.8 0 books 48.1 22 books 14.1KB/book 58.0 980 books 10.3KB/book (many cust cols, almost no data) 49.9 1400 books 1.5KB/book (few cust cols, lots of data, production) 54.5 4995 books 4.8KB/book (more cust cols, replicated books) I don't understand why the production library is so much smaller per book. I ran the test several times and always came up with the same number. My guess is the overhead for custom columns? Of course, this shows that the GUI accounts for a lot of the memory use. No real surprise, that.

11-03-2010, 02:09 PM	#21
kovidgoyal creator of calibre Posts: 45,569 Karma: 28548962 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Just for completeness, here are my figures with measuring only heap size, which means that the size of python etc. are not included Code: Heap (KB) Number of books 35604 0 36054 23 36388 85 38948 422 Heap - 0 size (KB) Number of books Size per book 450 23 19.57 KB 784 85 9.2 KB 3344 422 7.9 KB (production library) The numbers haven't changed significantly which means you have a lot more metadata per book than I do You are on a 64 bit machine, right? EDIT: Oops, I meant I have a lot more metadata per book than you do Last edited by kovidgoyal; 11-03-2010 at 02:12 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
STARTUP TIME, WORKING TIME	jomaweb	Calibre	2	02-13-2010 11:17 AM
Long time stalker, first time poster	yellowfeather	Introduce Yourself	8	09-08-2009 08:08 PM
Hi - First time poster long time ereader	werdegast	Introduce Yourself	5	11-15-2008 01:15 AM
Long time listener, first time caller	reddots	Introduce Yourself	2	01-23-2008 08:40 PM

11-03-2010, 01:42 PM	#18
kovidgoyal creator of calibre Posts: 45,569 Karma: 28548962 Join Date: Oct 2006 Location: Mumbai, India Device: Various	If you have the time, try copying all custom column data into the comments field (at least the string variety) and deleting the cust cols. I really don't see why cust cols should have a greater overhead than normal cols.

11-03-2010, 02:06 PM	#20
itimpi Wizard Posts: 4,553 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	I have a library of around 10,000 books with an average of about 4 formats per book. Nearly all books have comments attached. Even on this library Calibre never seems to use more than about 250MB of RAM which is nothing to most modern machines. I am running Windows 7 Pro. I am very sceptical that total RAM is therefore much of a limit on performance.

11-03-2010, 02:17 PM	#22
chaley Grand Sorcerer Posts: 12,524 Karma: 8065948 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	The real question becomes 'do people who have performance problems have "modern machines"'? Calibre total memory consumption for my production library of 1,400 books on Win7Pro is 280mb, of which some 140mb seems to vary with the db size. Even assuming minimum scaling up, a 35,000 book library (the OP's library size) would at least triple the variable size (probably more), taking total consumption to at least some 550MB. With those numbers, if he has a 1GB machine (these are still common), and especially if he is running other apps like mail, calibre will thrash. On a 2GB system, it likely will not.

11-03-2010, 02:43 PM	#25
kovidgoyal creator of calibre Posts: 45,569 Karma: 28548962 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Hmm itimpi's 10k library uses 250MB and chaley's 1k library uses 280MB and both on W7??

11-03-2010, 02:53 PM	#27
jbcohen Wizard Posts: 3,033 Karma: 11196738 Join Date: Oct 2010 Location: Where am I? Device: Kindle Paperwhite Signature edition and a Samsung S24 Ultra	I feel your pain, I had a pandigital novel before it got stollen from my home and it had probably a 3 minute boot time.

11-03-2010, 05:50 PM	#28
Manichean Wizard Posts: 3,130 Karma: 91256 Join Date: Feb 2008 Location: Germany Device: Cybook Gen3	FWIW: 53 MB, 336 Books. System is Win XP 32, 1 GB RAM Because there's no such thing as too much data.

11-03-2010, 08:08 PM	#29
theducks Well trained by Cats Posts: 31,206 Karma: 60406678 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	1225 books Win XP SP3 1.5G RAM Process manager says 282M Virt. Size, 101M working set