A couple of my degrees are in mathematics I fully understand the desire for a reader that will help with the learning process or research process, heck, even just reading papers for the fun if it, but...in a word, there is not a single reader device that operates in what I feel is a useful fashion as a learning/research tool for the given subject. The closest would be the Entourage Edge, maybe an iPad or other slatePC, or in general a 9.7" reader. I had hopes for the iRex 1000S (or whatever the designation of the 1000 was) but never got around to asking HarryT.
A lot also depends on the format of the books you have access to, if those books make use of color to indicate equation changes as a proof progresses or to delimit important segments of the equations in question.
For many older, say pre-1970s era texts, odds are any 9"-10" device should be OK, but again it comes down to file format handling. I don't know any readers which handle any of the file formats, beyond PDF of course, from Mathematica/MathCAD or other programs, which, for those who don't know, are in-part the word processors of the math world.
I gotta say that today's modern texts and papers for mathematics make such wonderful use of color, you are likely going to end up with an LCD based device as your best option. Of course if all you want is to read, even if the inset graphics might be pretty disjointed from where they should be in the text and can handle a fair amount of zooming around the screen, I would suggest a device with at least a 9.7" panel...however a 12" panel is far better as a "Mendoza Line"...

I have seriously looked around and none of the device makers have anything right now that I see as viable...
We I to need one, and at this point in life I don't, I would have to look at dealing with the cost of a true slatePC of a convertible tabletPC. I like any of the TabletKiosk Sahara 400 family of devices, probably the i440D would be the choice but be prepared for a $3000 cost. But it has the power to run the underlying software to support the documents as well as work with multiple docs at the same time, including a Wacom digitizer so you could actually take notes and even work problems on the device...combine that with OneNote or Evernote or other similar note management app. Plus the i440D is fully readable in sunlight which is nice when you want out of the library/office/home to study.
I guess the ultimate answer depends on what you want to do with the tool...and what shortcomings are worth spending the funds on for a device you likely will not find all that transparent in your study work-flow.
Wish my take on the options was better but right now, I don't see anything...yet...I wish one of the device makers would consider working with Wolfram Research to embed some version of they one of they math oriented apps, maybe only the reader, paper writing parts because obviously the modeling portions would never run on the current hardware used in these readers.
http://www.wolfram.com/
But nothing is really ready for other than a PDF and even then the lack of color will be at best bothersome and at worst discouraging from using the device at all for the purpose you intend.