Which part is slow? By file manager or by Calibre metadata search?
If it's the file manager, add some more hierarchies to the directory structure. A folder on FAT32 with thousands of entries in it doesn't make for good performance.
For example:
Code:
{author_sort:.1}/{author_sort}/{title}
If I remembered the syntax correctly, that should give you:
Code:
/L/Last Name, First Name/Title