I've only really started on the WH40K universe myself (I've been going a while, but considering the size of the project...), but here are my thoughts in regards to the books I've put away so far:
The
Horus Heresy storyline is pretty good up to volume 9, where I'm at right now. The quality is slightly variable, but no more so than in any other long series and it hasn't strayed below "good" so far. In my opinion of course.
Horus Heresy also deals with the background to why the WH40K universe is "presently" such a dismal place, but with a current count of 20 or so books it's probably too optimistic to think "I'm just gonna read HH first, to get the history and background out of the way". Which is what I did.
The
Space Wolf series is quite good and gives you a good insight into the selection and training of a Space Marine, albeit in a very non-standard Space Marine Chapter. The last, or last two books (I forget) are not quite up to the standard of the original three (or four), but they're all easy reads so if you begin you may as well finish.
Another book to pick up early is
Space Marine, which also gives you an "insight" into how a Space Marine comes into existence. This time in a more "standard" setting.
The first book I started was
Space Hulk, because I have fond memories of the computer game from back in the day. Bad choice - it's the only book so far which has been just plain bad.
Grey Knights is another good series, dealing with one of the chapters charged with hunting down daemons and chaotic forces.
Other series which are widely recommended are, amongst others,
Gaunt's Ghosts,
Ciaphas Cain,
Eisenhorn and
Ravenor.
For the actual books,
The Black Library is where to get them. Which you obviously already know.
For more information on all things WH40K,
Warhammer 40K Wiki and
Lexicanum are the best places to go and if you have obscure questions, there are plenty nerds over at the
Warhammer 40K Forums who will be able to enlighten you.