I've been giving this some more thought and it occurs to me that maybe the best approach would be to build better tools and use the best format for each type of book.
e.g.
For linear books like fiction, use DTBook or Fictionbook2
For technical books, use DocBook
For topic/map books like encyclopedias and recipe books, use DITA
For complex office documents, use OpenDoc.
For complex books that don't fit into any other category, use TEI
A single toolchain could convert all of these into any output format, or readers could even handle them directly. Though this might be a bit pie in the sky
I've heard that the Rosetta tools go even further by storing the document in a database format from which it can be exported in any format, but I've been unable to find any further information on that.