Only the copyright holder could sue you as others mentioned, it's a civil offence (at least afaik in the UK). However, no copyright holder would be daft enough to sue someone who just strips DRM to read the book on another device or as a backup even if they ever found out about it.
They gain nothing by suing you, unlike suing pirates. You've caused them no damage, you paid for the book, you're not sharing it with anyone who shouldn't have access. At best you're perhaps avoiding buying the book a 2nd time in a kindle rather than nook format, maybe they can claim $5 in damages, but there's no way a case like that would be even remotely worth pursuing and I can only imagine the PR disaster it would cause.
Unlike people who distribute their books, where there is an incentive for holders to go after them even if it costs more to do so and the damages can be way higher.
IANAL, but chances are it is illegal in most countries.If you're truly doing it for personal use imho you'll never have a problem. In time it will likely be a non-issue anyway. I expect more and more publishers to realise they're been scammed into paying per book DRM fees when the DRM is not in anyway reducing the levels of piracy. Either that or
laws will hopefully change to make it legal