When looking at potential marriage material, yes a lack of books would be a big problem. (and I have mentioned this before, but my now-husband actually tricked me in this way! When I first went to his apartment, he had TONS of books, literary and otherwise. It turns out, he has finished hardly any of them. Ha ha.)
HOWEVER, it doesn't sound like you were looking for marriage-material in your rendezvous. I can't say that a lack of books would bother me, in that case? Maybe you have higher standards than me.
eP