@Tanjamuse:
Not a half-million, but sufficiently large enough to give me a lot of personal experience that I can share with you in the form of suggestions derived from what I have had to do:
(1) Create a new empty "pattern" Calibre Library that has all of the custom columns and plugin configurations that you can foresee needing or wanting.
(2) Create new Calibre Libraries using your "pattern" metadata.db and its associated .json.
(3) Create one new Calibre Library for each forseeable high-level "Genre". For example, as a minimum, 3: Fiction; Factual; Poetry.
(4) Based on your anticipated book-count by high-level "Genre", create more detailed new Calibre Libraries using your "pattern" metadata.db and its associated .json.
(5) Move your books in your original 9,000 book library into their new homes, and then eliminate duplicates.
(6) Count what remains in each Library, and create new "pattern" Libraries as needed to further break down the "Genre" represented in a particular Library if it is too voluminous. Example: You may need to create separate Libraries for "Factual:History", "Factual:Autobiographies&Biographies" etc.
(7) Create new Virtual Libraries based on the final metadata in each of your pattern Libraries. Example: VLs of "Factual:Autobiographies" and "Factual:Biographies" for the "Factual:Autobiographies&Biographies Library.
(8) You will want to compress the metadata.db files frequently. I suggest using .bat files that invoke the calibredb command-line utility to "check_library", which not only tells you of any corruption, but also vacuums/compresses/defragments a Library. You can create a single .bat file that sequentially executes calibredb for each of your libraries in turn. See:
http://manual.calibre-ebook.com/cli/calibredb.html
Personally, I would not put all of the libraries on a single physical hard-drive unless it was a reliable RAID device. Too risky. Backing up 500,000 metadata.opf files plus 500,000 cover.jpg files, which are "small" files, takes a
very long time. Obviously you need to back up everything initially. I back up all of my metadata.db files daily to multiple external devices, since "small" files take much too long to do, and Calibre generates new metadata.opf files automatically regardless of whether you want it to or not.
Good luck.
DaltonST