Well, let me jump in here - I've been a huge fan of lovecraft and weird fiction generally since my teens. My favorite Lovecraft works are "Dream-Quest of the Unknown Kadath" and "The White Ship." I'm also a fan of Clark Ashton Smith and William Hope Hodgson. Like Dr. Drib, I have also collected a nearly full set of the Ballantine Adult Fantasy series - I first read William Morris, George McDonald and Lord Dunsany through that imprint.
That being said, I've also developed a real taste for the New Weird - current writers of strange fiction, often influenced by Lovecraft and his peers. A great website for online new weird fiction is Fantastic Metropolis (
http://tinyurl.com/46bmnu), which is sadly dormant at the moment, but still has a remarkable collection of new weird tales (I recommend anything by Jeff Vandermeer or Jeffery Ford). Jeff Vandermeer's City of Saints and Madmen is a tremendous work of weird fiction and is available at Books on Board, among other ebook sites.
I also highly recommend anything by Thomas Ligotti, who is perhaps the most Lovecraftian writer going. Two of his short stories are available online at
http://tinyurl.com/6jpnda. His work can be hard to find, but the collections Teatro Grottesco is currently available in hard cover at Amazon (
http://tinyurl.com/6yak9z), with a trade paperback edition available for pre-order. Otherwise, ebay, alibris and Amazon's used books section are probably your best bets.