I used to have a bunch of books loaded on my SD card but again, I stopped using it. This was mainly because I didn't really need it.
I very rarely have problems with the speed though since I never really turn the device off - I only put it in standby.
The only time when loading is an issue is after I have added some books or hooked up to the USB for charging. Even then it doesn't bother me since this is done during a time I wouldn't be reading anyway!
I have however recently found that using the 505 as a Card Reader is great when I need to put something on my Archos and haven't got the device specific USB cable with me