I've used a variety of devices over the years to read fiction and technical material. For reading fiction I have found that the kind of display is much more of a factor than the size of display. Displays with backlights such as LCD and LED are far more tiring on the eyes than the more paper like displays on the dedicated eReaders. Once at 5" or above I found the display size is no longer an issue, certainly compared to PDAs, and don't think about how a book compares with the larger paperback.
Technical books, papers and notetaking are a different matter and I have not yet found a good alternative to the 12" display on my tablet pc as size does matter with this material. I have yet to have a proper look at the 9.7" devices such as the Kindle DX and I'm looking forward to trying out the upcoming ASUS devices, the DR 950 especially.
My advice is not to worry too much about the size of display initially, buy an eReader and start using it. As you get used to reading on it the differences with a paper book will become less of an issue but the advantages will remain. Over time you will also become aware of what capabilities of the reader and software are important for you and so be able to make a fully informed choice when the cheaper, faster, better devices become available.