I prefer reading on eInk over a tablet myself.
But your question on what you should keep is really going to be a personal decision. There are different reasons to like all the various readers - so it just depends on which of the features fit you best.
If you take the various readers at face-value - rather than the more technically involved folks who are hacking their readers! - your Paperwhite will read your Amazon ebooks. Some, but not all, books from Overdrive eBook libraries are also available for the Kindle. If you buy eBooks from the smaller e-publishers, you might also be able to get "mobi" formatted books that the Kindle will read.
Kobo will read ePub books purchased from everywhere but B&N (last I knew) and Apple. The Nook will read ePubs purchased from everywhere but Apple. There's a lot of speculation that the Nook as a brand is dying, so keep that in mind if it concerns you. I actually recently bought a Nook Glow because my ebooks are all ePub and I like the light. And for now at least, I have local help with it if I have problems (unlike Kobo).
On the other hand, if you decide to keep the Paperwhite, you can learn how to remove the encryption (DRM) from your ePub ebooks and use a program called Calibre to convert them to mobi formatted books.
We can't give instructions on how to do that specifically. We always suggest you google "Apprentice Alf" for details.
For most people, it's likely not legal, so you have to decide how it fits in your ethics. (Most of us figure if we're just stripping the DRM for our own use, it's worth learning how to do - we're not offering our books up to be pirated!)