I use Kindle for PC and it stores all the books I get in My Documents under a folder which I think is called My Kindle Content - all you need to do is copy this folder elsewhere for a backup.
I back all mine up onto a hard-drive and it makes no odds about whether they have drm or not - they are backed up.
So no need to go back to buying print books because once you know where the books are stored it's very easy