I understand what you want to do. I decided recently that when reading a book that was in a series, I would have the rest of the series on the device as well. Or at least the earlier books. But, I only have about 200 books on it, so manually doing that in calibre isn't that hard.
For what you are doing, I would look at the Reading List plugin. With this you can add books to a list and then easily sync that list to the device.
The processing of the SD card seems to be in two phases. The first is reading the card and checking for changes - books added or deleted. Any deletions seem to be removed from the database at this time. If there are additions, then the process screen comes up for the second phase. How long that takes depends on how many new books there are. The last time I tried 500 books, it took less than 10 minutes, so eight hours for 5000 seems to long. That was with FW2.4.0. Older firmware were slower. I haven't tried with 2.5.0.
Recent firmware do not store anything for the books on the SD card in the main memory except for details in the database. Some of the older firmware did store the cover images in the main memory. You will need to look at it when connected to the PC to see what is taking the space.
And as speakingtohe said, the time to open the shelf list is based on how many shelves and how many books are on them. One large shelf tends to affect the time as much as adding a lot of empty shelves.