One format isn't much of an issue. That's not what's bothering people. We have wonderful tools like Calibre to do format conversion. The problem is DRM.
DRM only blocks legitimate people from access to their books. Every DRM is broken, and even without that, folks can (and do) scan books. That's why the Harry Potter books are available even though not legitimately.
As long as publishers make getting legitimate books more difficult than legitimate books, they will lose more to piracy than they gain in market protection.