USB ports are more flexible than you think. Just measure the voltage on a USB AC wall charger and you'll know it's seldom exactly 5V.
Connecting the solar panel to the battery might indeed be a better idea, as the battery will work like a capacitor.
The only thing I'd be worrying about is when the sun isn't shining, doesn't the panel consume some battery instead of providing power?
One thing I've thought of,is having a 6V panel, pasted on the back of the reader.
As long as the sun isn't shining fully on the panel, it'll never reach 6V.
Another thing is, that every device has a power LED, as well as this has an LED screen.
Both will dampen over-voltage (with over-voltage they perform better, and prevent the voltage from reaching too high). Before the power led would become unusually bright, or decolored, you'd be running at least 25-33% higher voltage. Also most of the internal CPU and stuff can handle a bit more than 10% overvoltage, especially with such low heat devices.