You can get used to just about any size, I think. I did a little e-book reading, and quite a bit of forum reading, on my old <2.5" dumbphone. Anymore, I don't find myself in too many situations where I want to read and have my phone (4") but not my Nook Color(7"). When I do, though, the interface disappears pretty quickly, page turning becomes automatic, and it's just me and the text.
On the quantity-vs-quality front, my old 10.1" netbook was my e-reader for a while, and while it served me well, the Nook Color's smaller screen at the same 1024x600 resolution is so much sharper, which especially comes through in comics and PDFs. The wide IPS viewing angles are also more tolerant of the odd contortions some of us engage in while reading
So yes, I'd much rather read on a 4" 800x480 device than a 7-10" device of the same resolution.