MobileRead Forums - View Single Post - The fight for a reflowable e-document standard

Bob Russell · 09-03-2006, 04:22 PM

If you've ever wondered what all the fuss is about with regard to e-book formats and the new standards, and so forth, this is your chance to get a quick look into the situation without having to spend your whole day on it. I have to admit that I haven't really paid much attention to this topic until recently, and am only beginning to get familiar with it myself. But it's an important topic, so it's about time to start getting up to speed.

My attention has been caught today by two recent shots that have been fired in the fight for a document standard. But before we address that, we need to get caught up a bit with regard to the landscape.

What are we talking about here? The driving need is for a widely used format which allows for documents to be successfully presented on a wide range of display sizes, such as handheld devices. Adobe's pdf, for example, is a widely successful document standard, but does not excel at handling presentation on multiple screen sizes. Other formats may be sufficient, but if there are a dozen of them and they are proprietary, then you still don't have any level of assurance you can use the format on future devices. Even worse, publishers could spend a fortune trying to prepare and support e-books for all those formats.

The OpenReader Camp

So, in one corner of this fight for e-readability, is David Rothman, Director of Strategy and External Relations for OpenReader. You are probably already familiar with David through the excellent TeleRead Blog. The OpenReader Consortium describes themselves as "a nonprofit organization developing open digital publication standards. We serve publishers, consumers, e-book retailers, librarians, and other key stakeholders in socially responsible ways."

David has coined the phrase "the Tower of e-Babel" to describe the current situation of incompatible e-book formats. The OpenReader site describes the problem this way: "Any Compact Disc will play on most any CD player, and the VHS vs. Betamax battle is long over. Why can’t electronic books, newspapers, journals and other publications be as easy? Consumers are confused right now. They don’t need different eyeglasses to read a paperback rather than a hardback. Why should consumers have to worry about the software they use to read downloads of electronic publications? Just in e-books alone, more than 20 different formats exist."

OpenReader has some specs in development, including a draft version of the Binder Document Specification (which appears to be an XML specification for a way to collect all the files that make up a publication), and a draft version of the Basic Content Document Specification (which appears to be an XML specification for the actual contents of a document). More than just an abstract specification, OpenReader says that "dotReader will be the first OpenReader-compatible commercial program [and] FBReader will be also be compatible." FBReader is an e-book reader for Linux PDAs that supports, for example, the Nokia 770 and the Sharp Zaurus.

The IDPF Camp

In the other corner of this fight, there is another standards group working on the tower of e-Babel, and it's the IDPF - the International Digital Publishing Forum. The IDPF describes itself this way: "The International Digital Publishing Forum (IDPF), formerly the Open eBook Forum (OeBF), is the trade and standards association for the digital publishing industry. Our members consist of academic, trade and professional publishers, hardware and software companies, digital content retailers, libraries, educational institutions, accessibility advocates and related organizations whose common goals are to advance the competitiveness and exposure of digital publishing." While I don't know if there is a public membership list, it does boast a remarkable list of members as can be gleaned from the published presentations, if nothing else. Among those worth noting are Adobe, MobiPocket, Motricity, iRex and eBook Technologies. As I don't have a full list, I'm sure I've left out many other prominent names.

As quoted from PublishersWeekly, here is another very succinct description of the problem we face. "At present, there is no common standard used by producers and manufacturers. As a result, customers can’t read a Palm e-book on a Microsoft Reader, noted Nick Bogaty, executive director of the IDPF. If companies adopt the new standards, not only will customers be able to read e-books on different devices, but e-books will be cheaper and easier to produce, which should lead to more titles being available, said Bogaty. 'We’re looking to create the MP3 for e-books,” Bogaty said about the goal of attaining file flexibility.'"

I should point out that the IDPF is much more than a standards group. Some of the guiding values are:

Providing a forum for the discussion of issues and technologies related to electronic books.
Developing, publishing, and maintaining common specifications relating to electronic books and promoting the successful adoption of these specifications.
Promoting industry-wide participation of electronic publishing through training sessions, guidelines, and demonstrations of proven technology.
Identifying, evaluating and recommending standards created by other bodies related to electronic books.
Encouraging interoperable implementations of electronic book related systems and providing a forum for resolution of interoperability issues.
Accommodating differences in language, culture, reading and learning styles, and individual abilities.

But the IDPF has done a remarkable job of involving the publishing industry at many levels, as well as e-book sellers. And they have come up with a draft of a container format is based on zip archive technology. It supports publishers while working on document production, distribution to the sales channels, and for delivery of the final result to the end user. It will "allow publishers to release only a single standard file into their sales and distribution channels instead of the multiple proprietary files that they currently produce. The new ZIP-based proposed standard, OEBPS Container Format (OCF), is an extension of the packaging format defined by the OpenDocument OASIS Standard (ISO/IEC 26300)."

And in addition to this container format (which is evidently analogous to the Binder format from OpenReader), there is another working group developing the document standard itself. This second specification is the next generation of OEBPS (Open eBook Publication Structure). It is an XML based e-book format from the IDPF (International Digital Publishing Forum). It is being developed to "focus on detailed control of content rendering, navigation and accessibility, and alignment with other standards efforts."

There have been announcements that OSoft will support the IDPF standard in DotReader, as well as support for these IDPF standards by Motricity (eReader, but it's not clear they've made any actual product plans to support the standards) and iRex ("We expect to support the IDPF standards for use in the iLiad e-reader in the future.")

Other Players?

Most certainly there are other efforts to develop document and container formats. And there may be more related document standards that come closer to solving these issues than I'm currently aware. But this gives a basic lay of the land as I understand it so far. And it gives you a basic framework from which to understand the two recent shots fired in the battle. Let's take a look.

Shot 1: OpenReader is the solution

David Rothman, in his article published by Publishers Weekly, points out that the only DRM content you can read on a Pepper Pad is from MobiPocket. That means you can't read anything from the Fairfax County Public Library's digital offerings, which are in pdf format.

He reminds us that the landscape is full of broken promises. "The big companies' unkept promises go back at least as far as 1998, when Microsoft and others said they would avoid a VHS/Beta–style war. Unsurprisingly, the agreement ended up a joke among the e-book cognoscenti."

He gives credit to the IDPF as the main e-book trade group, but says that the IDPF is dominated by big corporations, whose interests then dominate the organization and most publishers who "remained ignorant of the nuances of e-book formats."

In what is probably a bit of posturing, he says that the IDPF "has unintentionally stymied the popularity of e-books by failing to adopt consumer-friendly standards." What he is saying is that the IDPF has not so far created that successful and widely adopted standard that would prevent the current tower of e-babel. But as they are one of the main forces trying to solve the problem, it might be a bit much to put the blame for our current state on them unless one thinks they are a genuine obstacle to progress.

Then having established the problem, and arguing that the IDPF is not the solution, he argues that they should leave the standards business altogether. (Leaving it to OpenReader, of course.) He does argue that standards bodies like OASIS should be more involved in the standards. But there has been indication that the IDPF is interested in doing just that, and I could be wrong, but I don't believe we've seen a lot of success to this point from OpenReader on the OASIS front yet.

Basically, I think David argues that the OpenReader specs, being more technology focused rather than publishing industry focused, is producing a better spec and therefore should become the based standard for e-books. Then with adoption by standards bodies and by the publishing industry as it becomes more tech savvy, a firm foundation will be set. In his words, "As a starting point, standards setters could consider the OpenReader format, which is now in draft form. (I cofounded the standards-minded OpenReader Consortium.) With the right software features in place, the OpenReader standard will allow forums, blogs and annotations to be visible within books, including even copy-protected titles, making them competitive with the Web."

And then, of course, he points out that DRM needs to also become a part of the e-book standards. I wholeheartedly agree. The biggest obstacle for e-book compatibility is not the container or even the document format, but the DRM mechanism. Even HTML or text documents are useless if there is proprietary DRM protection on them.

So there you have it. Together with the support for their format in DotReader, the basic argument for OpenReader is given succinctly and clearly.

Probably David has done a better job so far of getting the word out to the technical crowd. He has an excellent and active blog which is a forum for presenting his ideas. And he has been very eloquent and persuasive in his arguments. But there's always another side to every issue, isn't there?

Shot 2: IDPF is the solution

While the IDPF is likely to become more and more active in publicity in the tech world as time passes, they present a more "official" face and we see more from them in places like presentations from their annual conference and press releases, and the occasional publication.

On the other hand, Bill McCoy from Adobe is a big supporter of a solution for the tower of e-Babel, and has written about the topic in places including his own Adobe blog. Let's look at what he has had to say in his latest entry on the topic.

He directly addresses David's article, and starts by setting the common ground. We all have the same end goal. It's how to achieve those goals that we dispute. "I agree with his main points - we have to make digital reading simple and compelling for end users, and the plethora of proprietary eBook formats has created consumer confusion. Where I disagree is that I see the answer at hand."

Also as a prelude, he points out that we have a successful and widely used format for e-books that are fixed format and page-oriented. "[F]or paginated final-form 'ePaper' there is no Tower of e-Babel - PDF is the answer, it's game over." It's hard to argue that point, no matter how frustratingly slow it may be to get Acrobat Reader to load in order to read a pdf document. I know that in many cases, I avoid following a link to a .pdf document because it's often not worth the wait. I know others feel the same way because there are often warning labels on links to alert the reader that they will end up waiting on a pdf document. But, imperfect as it may be, it gets the job done, and most importantly it's almost universally adopted. As Bill says, "Game over."

So how should we address the reflow problem for various sizes of displays? Far from considering the IDPF an obstacle to progress, Bill says that "this problem is now being rapidly and effectively addressed by the IDPF. Publishers, vendors (including Adobe), and library and educational groups have come together in IDPF to create the open standards that will end the Tower of eBabel."

While David Rothman's criticism of the IDPF hinged on it's dominance by publisher interests and big companies, together with the lack of a firm technical foundation, Bill McCoy suggests that David's alternative is troubling. I.e. to let OpenReader do all the technical specifications and then get them "rubber-stamped" by OASIS without effectively bringing the publishing industry into the picture.

Bill doesn't pull any punches. He says of David's recommendations that, "Well, I think this is elitist, egocentric nonsense that hasn't a prayer of working. No doubt it's messier to have publishers, competing vendors, and nonprofits all coming together. True, not every publisher has industry-leading technologists on staff. But many do, and this kind of participative process is part of building successful (by which I mean broadly adopted) solutions." He suggests that much of the battle comes down to egos. And he goes on to point out that e-book document standards are not likely to be much of a priority for large standards bodies like OASIS anyway.

Bill basically concludes that the IDPF is a very effective organization to address the tower of e-Babel. And that, in fact, it's the best approach we have. "I believe that we're all a lot better off with publishing industry working together in an architecture of participation, and IDPF is where this is happening now. The results of this, including the IDPF OCF container format standard, are already bearing fruit and the Tower of e-Babel that plagues reflow-centric eBook formats will start falling down over the coming months. There's always room for more contributors in the IDPF."

Conclusions

What we have seen is just a snapshot of the battle for e-book standards. As I said in the beginning, I'm not an expert. But I do feel this is an important discussion, and one that has previously been fairly inaccessible and incomprehensible to most people interested in e-books.

If you expect me to choose a side, then you will be disappointed. My greatest desire and hope in writing this article is that it will prove helpful in making this whole topic more inclusive, by making it more accessible and interesting to a much broader audience.

Where I have incompletely or inaccurately represented any of the positions, there is always room for further discussion and input. But what we should all remember is that, outside of a few special interests, we all want the same end goal of compatible e-books. The people fighting for that goal are our heroes. And one way or another, the e-book industry is going to explode. It's going to be an exciting ride, and a lot of fun to watch!

Update:As this is published, I notice that there's another response over at TeleBlog to Bill McCoy's comment. I haven't read it yet, but as I indicated, this is an ongoing discussion, and I hope that now you are in a better position to enjoy the battle.

09-03-2006, 04:22 PM	#1
Bob Russell Recovering Gadget Addict Posts: 5,381 Karma: 676161 Join Date: May 2004 Location: Pittsburgh, PA Device: iPad	The fight for a reflowable e-document standard If you've ever wondered what all the fuss is about with regard to e-book formats and the new standards, and so forth, this is your chance to get a quick look into the situation without having to spend your whole day on it. I have to admit that I haven't really paid much attention to this topic until recently, and am only beginning to get familiar with it myself. But it's an important topic, so it's about time to start getting up to speed. My attention has been caught today by two recent shots that have been fired in the fight for a document standard. But before we address that, we need to get caught up a bit with regard to the landscape. What are we talking about here? The driving need is for a widely used format which allows for documents to be successfully presented on a wide range of display sizes, such as handheld devices. Adobe's pdf, for example, is a widely successful document standard, but does not excel at handling presentation on multiple screen sizes. Other formats may be sufficient, but if there are a dozen of them and they are proprietary, then you still don't have any level of assurance you can use the format on future devices. Even worse, publishers could spend a fortune trying to prepare and support e-books for all those formats. The OpenReader Camp So, in one corner of this fight for e-readability, is David Rothman, Director of Strategy and External Relations for OpenReader. You are probably already familiar with David through the excellent TeleRead Blog. The OpenReader Consortium describes themselves as "a nonprofit organization developing open digital publication standards. We serve publishers, consumers, e-book retailers, librarians, and other key stakeholders in socially responsible ways." David has coined the phrase "the Tower of e-Babel" to describe the current situation of incompatible e-book formats. The OpenReader site describes the problem this way: "Any Compact Disc will play on most any CD player, and the VHS vs. Betamax battle is long over. Why can’t electronic books, newspapers, journals and other publications be as easy? Consumers are confused right now. They don’t need different eyeglasses to read a paperback rather than a hardback. Why should consumers have to worry about the software they use to read downloads of electronic publications? Just in e-books alone, more than 20 different formats exist." OpenReader has some specs in development, including a draft version of the Binder Document Specification (which appears to be an XML specification for a way to collect all the files that make up a publication), and a draft version of the Basic Content Document Specification (which appears to be an XML specification for the actual contents of a document). More than just an abstract specification, OpenReader says that "dotReader will be the first OpenReader-compatible commercial program [and] FBReader will be also be compatible." FBReader is an e-book reader for Linux PDAs that supports, for example, the Nokia 770 and the Sharp Zaurus. The IDPF Camp In the other corner of this fight, there is another standards group working on the tower of e-Babel, and it's the IDPF - the International Digital Publishing Forum. The IDPF describes itself this way: "The International Digital Publishing Forum (IDPF), formerly the Open eBook Forum (OeBF), is the trade and standards association for the digital publishing industry. Our members consist of academic, trade and professional publishers, hardware and software companies, digital content retailers, libraries, educational institutions, accessibility advocates and related organizations whose common goals are to advance the competitiveness and exposure of digital publishing." While I don't know if there is a public membership list, it does boast a remarkable list of members as can be gleaned from the published presentations, if nothing else. Among those worth noting are Adobe, MobiPocket, Motricity, iRex and eBook Technologies. As I don't have a full list, I'm sure I've left out many other prominent names. As quoted from PublishersWeekly, here is another very succinct description of the problem we face. "At present, there is no common standard used by producers and manufacturers. As a result, customers can’t read a Palm e-book on a Microsoft Reader, noted Nick Bogaty, executive director of the IDPF. If companies adopt the new standards, not only will customers be able to read e-books on different devices, but e-books will be cheaper and easier to produce, which should lead to more titles being available, said Bogaty. 'We’re looking to create the MP3 for e-books,” Bogaty said about the goal of attaining file flexibility.'" I should point out that the IDPF is much more than a standards group. Some of the guiding values are: Providing a forum for the discussion of issues and technologies related to electronic books. Developing, publishing, and maintaining common specifications relating to electronic books and promoting the successful adoption of these specifications. Promoting industry-wide participation of electronic publishing through training sessions, guidelines, and demonstrations of proven technology. Identifying, evaluating and recommending standards created by other bodies related to electronic books. Encouraging interoperable implementations of electronic book related systems and providing a forum for resolution of interoperability issues. Accommodating differences in language, culture, reading and learning styles, and individual abilities. But the IDPF has done a remarkable job of involving the publishing industry at many levels, as well as e-book sellers. And they have come up with a draft of a container format is based on zip archive technology. It supports publishers while working on document production, distribution to the sales channels, and for delivery of the final result to the end user. It will "allow publishers to release only a single standard file into their sales and distribution channels instead of the multiple proprietary files that they currently produce. The new ZIP-based proposed standard, OEBPS Container Format (OCF), is an extension of the packaging format defined by the OpenDocument OASIS Standard (ISO/IEC 26300)." And in addition to this container format (which is evidently analogous to the Binder format from OpenReader), there is another working group developing the document standard itself. This second specification is the next generation of OEBPS (Open eBook Publication Structure). It is an XML based e-book format from the IDPF (International Digital Publishing Forum). It is being developed to "focus on detailed control of content rendering, navigation and accessibility, and alignment with other standards efforts." There have been announcements that OSoft will support the IDPF standard in DotReader, as well as support for these IDPF standards by Motricity (eReader, but it's not clear they've made any actual product plans to support the standards) and iRex ("We expect to support the IDPF standards for use in the iLiad e-reader in the future.") Other Players? Most certainly there are other efforts to develop document and container formats. And there may be more related document standards that come closer to solving these issues than I'm currently aware. But this gives a basic lay of the land as I understand it so far. And it gives you a basic framework from which to understand the two recent shots fired in the battle. Let's take a look. Shot 1: OpenReader is the solution David Rothman, in his article published by Publishers Weekly, points out that the only DRM content you can read on a Pepper Pad is from MobiPocket. That means you can't read anything from the Fairfax County Public Library's digital offerings, which are in pdf format. He reminds us that the landscape is full of broken promises. "The big companies' unkept promises go back at least as far as 1998, when Microsoft and others said they would avoid a VHS/Beta–style war. Unsurprisingly, the agreement ended up a joke among the e-book cognoscenti." He gives credit to the IDPF as the main e-book trade group, but says that the IDPF is dominated by big corporations, whose interests then dominate the organization and most publishers who "remained ignorant of the nuances of e-book formats." In what is probably a bit of posturing, he says that the IDPF "has unintentionally stymied the popularity of e-books by failing to adopt consumer-friendly standards." What he is saying is that the IDPF has not so far created that successful and widely adopted standard that would prevent the current tower of e-babel. But as they are one of the main forces trying to solve the problem, it might be a bit much to put the blame for our current state on them unless one thinks they are a genuine obstacle to progress. Then having established the problem, and arguing that the IDPF is not the solution, he argues that they should leave the standards business altogether. (Leaving it to OpenReader, of course.) He does argue that standards bodies like OASIS should be more involved in the standards. But there has been indication that the IDPF is interested in doing just that, and I could be wrong, but I don't believe we've seen a lot of success to this point from OpenReader on the OASIS front yet. Basically, I think David argues that the OpenReader specs, being more technology focused rather than publishing industry focused, is producing a better spec and therefore should become the based standard for e-books. Then with adoption by standards bodies and by the publishing industry as it becomes more tech savvy, a firm foundation will be set. In his words, "As a starting point, standards setters could consider the OpenReader format, which is now in draft form. (I cofounded the standards-minded OpenReader Consortium.) With the right software features in place, the OpenReader standard will allow forums, blogs and annotations to be visible within books, including even copy-protected titles, making them competitive with the Web." And then, of course, he points out that DRM needs to also become a part of the e-book standards. I wholeheartedly agree. The biggest obstacle for e-book compatibility is not the container or even the document format, but the DRM mechanism. Even HTML or text documents are useless if there is proprietary DRM protection on them. So there you have it. Together with the support for their format in DotReader, the basic argument for OpenReader is given succinctly and clearly. Probably David has done a better job so far of getting the word out to the technical crowd. He has an excellent and active blog which is a forum for presenting his ideas. And he has been very eloquent and persuasive in his arguments. But there's always another side to every issue, isn't there? Shot 2: IDPF is the solution While the IDPF is likely to become more and more active in publicity in the tech world as time passes, they present a more "official" face and we see more from them in places like presentations from their annual conference and press releases, and the occasional publication. On the other hand, Bill McCoy from Adobe is a big supporter of a solution for the tower of e-Babel, and has written about the topic in places including his own Adobe blog. Let's look at what he has had to say in his latest entry on the topic. He directly addresses David's article, and starts by setting the common ground. We all have the same end goal. It's how to achieve those goals that we dispute. "I agree with his main points - we have to make digital reading simple and compelling for end users, and the plethora of proprietary eBook formats has created consumer confusion. Where I disagree is that I see the answer at hand." Also as a prelude, he points out that we have a successful and widely used format for e-books that are fixed format and page-oriented. "[F]or paginated final-form 'ePaper' there is no Tower of e-Babel - PDF is the answer, it's game over." It's hard to argue that point, no matter how frustratingly slow it may be to get Acrobat Reader to load in order to read a pdf document. I know that in many cases, I avoid following a link to a .pdf document because it's often not worth the wait. I know others feel the same way because there are often warning labels on links to alert the reader that they will end up waiting on a pdf document. But, imperfect as it may be, it gets the job done, and most importantly it's almost universally adopted. As Bill says, "Game over." So how should we address the reflow problem for various sizes of displays? Far from considering the IDPF an obstacle to progress, Bill says that "this problem is now being rapidly and effectively addressed by the IDPF. Publishers, vendors (including Adobe), and library and educational groups have come together in IDPF to create the open standards that will end the Tower of eBabel." While David Rothman's criticism of the IDPF hinged on it's dominance by publisher interests and big companies, together with the lack of a firm technical foundation, Bill McCoy suggests that David's alternative is troubling. I.e. to let OpenReader do all the technical specifications and then get them "rubber-stamped" by OASIS without effectively bringing the publishing industry into the picture. Bill doesn't pull any punches. He says of David's recommendations that, "Well, I think this is elitist, egocentric nonsense that hasn't a prayer of working. No doubt it's messier to have publishers, competing vendors, and nonprofits all coming together. True, not every publisher has industry-leading technologists on staff. But many do, and this kind of participative process is part of building successful (by which I mean broadly adopted) solutions." He suggests that much of the battle comes down to egos. And he goes on to point out that e-book document standards are not likely to be much of a priority for large standards bodies like OASIS anyway. Bill basically concludes that the IDPF is a very effective organization to address the tower of e-Babel. And that, in fact, it's the best approach we have. "I believe that we're all a lot better off with publishing industry working together in an architecture of participation, and IDPF is where this is happening now. The results of this, including the IDPF OCF container format standard, are already bearing fruit and the Tower of e-Babel that plagues reflow-centric eBook formats will start falling down over the coming months. There's always room for more contributors in the IDPF." Conclusions What we have seen is just a snapshot of the battle for e-book standards. As I said in the beginning, I'm not an expert. But I do feel this is an important discussion, and one that has previously been fairly inaccessible and incomprehensible to most people interested in e-books. If you expect me to choose a side, then you will be disappointed. My greatest desire and hope in writing this article is that it will prove helpful in making this whole topic more inclusive, by making it more accessible and interesting to a much broader audience. Where I have incompletely or inaccurately represented any of the positions, there is always room for further discussion and input. But what we should all remember is that, outside of a few special interests, we all want the same end goal of compatible e-books. The people fighting for that goal are our heroes. And one way or another, the e-book industry is going to explode. It's going to be an exciting ride, and a lot of fun to watch! Update:As this is published, I notice that there's another response over at TeleBlog to Bill McCoy's comment. I haven't read it yet, but as I indicated, this is an ongoing discussion, and I hope that now you are in a better position to enjoy the battle.