Request for Ideas: Adding Indexed Files

[mc3]

So, I won't open the Calibre folder structure question...

But I do need some ideas.

Apart from my normal ebook collection, I have a technical document collection (mostly PDF's) that I have not yet added to my Calibre Library. There are over 100,000 files and they take over 150GB. I use a full-text search and retrieval system to be able to instantly find what I want in the collection. Here's the rub:

Both the creation of indexes and the search/retrieval speed is faster if the indexes are hierarchical (so an index can be rebuilt without re-processing all 150GB). The hierarchy for indexing, however, is the folder structure.

So hear is my quandary. I want to be able to use Calibre for its metadata features and to allow selective downloading of the docs. However, I also need to have full-text search and retrieval.

A possibility (short of having Calbre change its folder structure), would be for Calibre to use symbolic links to document files. This way the files appear to be in Calibre's folders, but the OS open's the file from another location. Non source files (like opf's, images, and formats Caibre generates) could co-exist alongside the symlink files.

But I'm asking for ideas, and I'm open to any possible solutions.

Thanks,

mike