Perhaps some references would be useful to the discussion of what is or is not legal.
In the USA, the relevant legislation is the
Digital Millennium Copyright Act
In Canada, it's the
amended Copyright Act of 1985
In the UK it's the
amended Copyright, Designs and Patents Act 1988
In the EU it's the relevant national government implementation of
Directive 2001/29/EC of 22 May 2001.
In all these cases, circumvention of technological measures for personal use is a civil infringement, with many exceptions. The copyright holder would need to take the infringer to court, and show that none of the many exceptions applied, and (I think) that there were actual damages.
Ethically, I see nothing wrong in removing DRM for personal use.