The system is slow on a huge number of files for a number of reasons:
1. Traversing a deep and large folder structure on FAT is slow. The indexer recurses over the complete SD card to find new and changed files. A possible speed up is to use a faster file system (ext3?) for the SD card, and/or have a separate interface to trigger re-indexing.
2. Extracting meta data (title, author, thumbnails) currently requires opening of the file through UDS and its plugins. A possible speed up is to use dedicated tool(s) to extract meta data from documents.
3. The global.db which has the details of all files on the card can grow to serious size which slow operations down. Again, a quicker file system may help, limiting the (number of) thumbnails will too. As yuri_b said, indexing the database might as well.
4. Content Browser currently prepares a full list of the content for a category (Books, Images...). With a large list, preparing it per page will speed it up considerably.
Hope this helps a bit.