Can calibre handle close to 1million files?

[mirthen]

simple question, most ive seen mentioned is around 200,000 ebooks. Looking at around 800,000 and climbing and need to know if calibre will havea hear attack if i try to add that many books, also the fact i dont want to mess with trying if theres a chance it will damage the collection in any way.

[kovidgoyal]

calibre does not care how many files you have. You simply need a powerful enough computer to handle them. That said you should probably split up your collection into a calibre few libraries to keep performance at an acceptable level.

And when you import books into calibre it creates copies, it does not touch the originals.

[JSWolf]

To be serious, what the heck would you do with 1 million eBooks?

[kacir]

Originally Posted by kovidgoyal

calibre does not care how many files you have. You simply need a powerful enough computer to handle them.

Calibre still creates a subdirectory for every single author in the "Calibre library" directory. This might be too much for an older version of an operating system to handle in a reasonable manner.
If you have a million books, you might have 300000 authors, so there will be that many subdirectories in the Calibre Library directory. If you make a mistake of opening this directory in a file manager on your computer, you might have to wait some time before being able to do anything else, especially if you do not use SSD disk for the library.

One possible solution would be creating 26 or so subdirectories in Calibre Library directory and placing all authors beginning with letter A there in the "A" subdirectory and so on. Or creating one numbered subdirectory for each 100 authors. But I think that would require way too many changes just to support those *very* few users that have such libraries.

You also have to use special tools for making backups, because with a million books you will have several hundred thousand authors and some 3 million files. A file with a book itself, a backup copy of metadata in metadata.opf file and a cover for each book.
If you do not have an ssd and use standard file manager to copy files it might take 4 hours for a file manager just to compute needed space *before* it starts copying of files themselves.
For windows you might want to use something like robocopy program (available from Microsoft), for Linux there is, for example, rsync (with a graphical GUI called grsync).

[stumped]

Originally Posted by JSWolf

To be serious, what the heck would you do with 1 million eBooks?

& is there a medical condition for that. sort of like e-kleptomania ?

[BetterRed]

Originally Posted by stumped

& is there a medical condition for that. sort of like e-kleptomania ?

Calibre can and is used to manage public domain texts, and to manage audio, video, and image collections, and there plugins to support doing that, e.g. look for Zotero and Media in the PI index.

BR

[stumped]

ok but the #1 post says books, TWICE, so clearly someone want to hoard 1M books

" most ive seen mentioned is around 200,000 ebooks. Looking at around 800,000 and climbing and need to know if calibre will havea hear attack if i try to add that many books,"

not our business of course, we should just answer the question and not take the e - p*ss

[mbovenka]

Originally Posted by stumped

not our business of course, we should just answer the question and not take the e - p*ss

Quite. The answer is: yes, Calibre will handle it, if you use the 64bit version (the 32bit or the portable version will run into the 32bit process memory limit) and throw enough hardware at it (SSD, lots of RAM, fast CPU).

Given that Calibre uses ~2GB of RAM with my 220K library, I'm guessing it will use roughly 8-10GB wit a million items.

[Fiat_Lux]

Originally Posted by stumped

ok but the #1 post says books, TWICE, so clearly someone want to hoard 1M books

That might be because it is easier to say books, than documents.

Offhand, the most probable candidate for needing to handle that many documents is somebody who works in the legal industry, in a firm that either lacks the cash, or the inclination to use the industry standard software for handling documents. Even an simple open and shut case can generate 100 or more documents, each of which has to be retained, and findable upon demand.

Calibre makes an excellent research management system, if one is willing to add the appropriate metadata to each item, as one imports it.

From a performance point of view, it is better to go with ten libraries of 100,000 items each, than one library of 1,000,000 items. But if the hardware is up to it, Calibre won't care.

Assuming Kovid hasn't done anything strange in programming Calibre, the hardware will cause issues, long before the database engine causes issues.

The theoretical maximum size of an SQLite database is in the neighbourhood of 140TB, which implies a library of around 1,400,000,000,000 items.