That's a nice concept. I see, you understood from where I came from.
Performance within calibre isn't a problem for me too (up to now). I leave big files outside calibre. This works with links perfect for me.
(I like calibre very much but to be honest, I can't imagine to manage huge audio, pictures or video content and collections within calibre in a way I like to use it. For these cases I have better solutions - but more often I integrate a "cross over" access via calibre.)