Library Size

Tanjamuse · 10-07-2015, 06:10 AM

Hi.

Does anyone have any experiences with having very large libraries. I'm talking 500.000+ stories.

Thanks

eschwartz · 10-07-2015, 08:47 AM

Yes, people have had experience with very large libraries.

...

...

...

Was there a question somewhere in that?

Spoiler:

Tanjamuse · 10-07-2015, 08:50 AM

Yes. Can it be done? (without having the computer crash)

eschwartz · 10-07-2015, 08:53 AM

Number one, the worst that would happen is that calibre crashed. Crashing the OS is somewhat more unlikely.
(You can do it though -- if your %TEMP% is on a ramdisk, say, which is the case on linux, and you try to convert hundreds of books, they all unpack at once and then you run out of RAM unusually fast, making it impossible to do anything without rebooting first. Not sure if that counts.)

Number two, calibre is not going to crash from having a lot of books. Once you hit the hundreds of thousands, you may find however that certain things slow down.
And it really matters how big the metadata.db file is, more than the accumulated sizes of the books.

Tanjamuse · 10-07-2015, 08:55 AM

What would make the metadata.db large?

eschwartz · 10-07-2015, 08:57 AM

A ton of book records. But it happens much more slowly.

My metadata.db is 10MB per 1000 books. But I have a lot of custom columns.

Tanjamuse · 10-07-2015, 08:59 AM

I have a library with about 9.000 books the metadata.db is 14.297 mb.

I would use the same set-up in that library in one where I would have more than 500.000 books.

Is that doable?

eschwartz · 10-07-2015, 10:29 AM

Well, if the size of the metadata.db scales with the number of books, you would probably have a metadata.db that was 700MB. Note: I am not sure it does, precisely.

Even loading the whole database in RAM all at once, on an ancient computer with only 2GB of RAM, I could handle that -- although I might not be able to run anything at the same time.

The only likely bottleneck is RAM. I assume most newer computers have plenty of RAM anyway.

Best way to tell if it'll work is to try it.

Tanjamuse · 10-07-2015, 10:31 AM

Computer specs:

12 gb RAM (updated to be able to run a better Calibre)
2.30 GHz
windows 64

I'm down to considering library of 250.000 instead

theducks · 10-07-2015, 11:11 AM

Keep your Library (DB) SIMPLE. No extras...
Custom (plain) columns add more size
Custom (Calculated) gobble up CPU power and RAM

Keep your metadata Neat and Trim. It is Lean or Mean (to your system HW).
Less tags, regularized Author names (same Author, but 16 variant spellings

J B Smith, James Smith, J. B. Smith...)

The actual books (formats) mostly use Disk space, But

Every Format is another (path to file) entry in a DB table (See #1

)

Currently, lots of CPU cores gains you nothing (basic Library browsing). Fast Disk, Large RAM (you got this covered) and CPU speed are the gains

Tanjamuse · 10-07-2015, 11:15 AM

I only have one format in the libraries (I download fanfiction) and I can remove my "Build from other columns" from the library I download to. They can always be redone (as far as I know) as long as the columns where the info stems from is there. (Please correct me if I'm wrong).

theducks · 10-07-2015, 11:30 AM

Quote:

Originally Posted by Tanjamuse

I only have one format in the libraries (I download fanfiction) and I can remove my "Build from other columns" from the library I download to. They can always be redone (as far as I know) as long as the columns where the info stems from is there. (Please correct me if I'm wrong).

If things get tight, rebuilding the column might push you over the edge.

I am

RDB
If it is built from data, don't store the answer you just use in a report.

but... Developers please correct me:

A 'built from' column is NOT stored in the DB, but recalculated every Time for the GUI.
If this is true, storing the result might be the lesser of the evils.
(

Static data: Use the metadata S&R to calculate and Store the result in a Plain column. rebuild those, when needed )

Tanjamuse · 10-07-2015, 11:40 AM

Quote:

Originally Posted by theducks

(

Static data: Use the metadata S&R to calculate and Store the result in a Plain column. rebuild those, when needed )

I'm sorry, I don't know what you mean.

Terisa de morgan · 10-07-2015, 11:44 AM

Quote:

Originally Posted by theducks

(

Static data: Use the metadata S&R to calculate and Store the result in a Plain column. rebuild those, when needed )

Very nice... if you don't have to calculate every day (as you add books every day

)

theducks · 10-07-2015, 11:45 AM

Quote:

Originally Posted by Tanjamuse

I'm sorry, I don't know what you mean.

Static Data: data that does not change. eg Author name, book title

Dynamic data: subject to frequent change. eg Reading progress
Includes Calculated data: (Living) Authors Age

10-07-2015, 08:47 AM	#2
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Yes, people have had experience with very large libraries. ... ... ... Was there a question somewhere in that? Spoiler: Maybe you should just ask your question, rather than, you know, asking whether it makes sense to to think people may have ideal testing circumstances for answering the question you haven't asked. Last edited by eschwartz; 10-07-2015 at 08:50 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Library Size and Performance	Hampshire Nanny	Library Management	7	06-12-2014 09:10 AM
Library size limits?	BeccaPrice	Library Management	3	09-14-2012 09:38 AM
Pratical Library Size	Motomaggot	Library Management	23	03-02-2012 09:41 AM
Maximum size of library?	crossi	Library Management	1	02-22-2012 04:35 PM
Library size and performance	MrFinch	Calibre	4	01-01-2009 04:11 AM

10-07-2015, 06:10 AM	#1
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	Library Size Hi. Does anyone have any experiences with having very large libraries. I'm talking 500.000+ stories. Thanks

10-07-2015, 08:50 AM	#3
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	Yes. Can it be done? (without having the computer crash)

10-07-2015, 08:53 AM	#4
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Number one, the worst that would happen is that calibre crashed. Crashing the OS is somewhat more unlikely. (You can do it though -- if your %TEMP% is on a ramdisk, say, which is the case on linux, and you try to convert hundreds of books, they all unpack at once and then you run out of RAM unusually fast, making it impossible to do anything without rebooting first. Not sure if that counts.) Number two, calibre is not going to crash from having a lot of books. Once you hit the hundreds of thousands, you may find however that certain things slow down. And it really matters how big the metadata.db file is, more than the accumulated sizes of the books.

10-07-2015, 08:55 AM	#5
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	What would make the metadata.db large?

10-07-2015, 08:57 AM	#6
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	A ton of book records. But it happens much more slowly. My metadata.db is 10MB per 1000 books. But I have a lot of custom columns.

10-07-2015, 08:59 AM	#7
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	I have a library with about 9.000 books the metadata.db is 14.297 mb. I would use the same set-up in that library in one where I would have more than 500.000 books. Is that doable?

10-07-2015, 10:29 AM	#8
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Well, if the size of the metadata.db scales with the number of books, you would probably have a metadata.db that was 700MB. Note: I am not sure it does, precisely. Even loading the whole database in RAM all at once, on an ancient computer with only 2GB of RAM, I could handle that -- although I might not be able to run anything at the same time. The only likely bottleneck is RAM. I assume most newer computers have plenty of RAM anyway. Best way to tell if it'll work is to try it.

10-07-2015, 10:31 AM	#9
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	Computer specs: 12 gb RAM (updated to be able to run a better Calibre) 2.30 GHz windows 64 I'm down to considering library of 250.000 instead

10-07-2015, 11:11 AM	#10
theducks Well trained by Cats Posts: 30,909 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Keep your Library (DB) SIMPLE. No extras... Custom (plain) columns add more size Custom (Calculated) gobble up CPU power and RAM Keep your metadata Neat and Trim. It is Lean or Mean (to your system HW). Less tags, regularized Author names (same Author, but 16 variant spellings J B Smith, James Smith, J. B. Smith...) The actual books (formats) mostly use Disk space, But Every Format is another (path to file) entry in a DB table (See #1 ) Currently, lots of CPU cores gains you nothing (basic Library browsing). Fast Disk, Large RAM (you got this covered) and CPU speed are the gains

10-07-2015, 11:15 AM	#11
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	I only have one format in the libraries (I download fanfiction) and I can remove my "Build from other columns" from the library I download to. They can always be redone (as far as I know) as long as the columns where the info stems from is there. (Please correct me if I'm wrong).

Advert

Advert