Ebook Formating Tool Wishlist

[Dave Berk]

Originally Posted by aapezzuto

As I have been going over the basic design concept I keep finding little holes in epub, that make me want to consider making a native structural xml file. Perhaps something like a striped down version of DocBook or some other only structural base format. If a good number of input and output formats are provided at release, does anyone care what format the program "thinks in" is?

What's the problem with using as epub as intermediate format? What are these holes?

[kovidgoyal]

I would strongly urge against creating a new format. That would mean that users would be stuck with your program for editing the "Master" file and application developers would have to code for yet another format. Interoperability is about more than just the source and sink formats.

[aapezzuto]

Originally Posted by Dave Berk

What's the problem with using as epub as intermediate format? What are these holes?

I'm not happy with having all elements needing to be reachable in a linear way, This makes footnotes still a problem. But the real problem is that epub supports so many things that I don't want to support (Im looking for a structural markup + css) and by the time that I normalize epub its another format, it just happens to be epub compliant.

Originally Posted by kovidgoyal

I would strongly urge against creating a new format. That would mean that users would be stuck with your program for editing the "Master" file and application developers would have to code for yet another format. Interoperability is about more than just the source and sink formats.

This is my main worry with this. I hate the current format proliferation, and I would hate to add to it. At the same time, the generation of new formats comes because of needs or purposeful incompatibility... It was just another question to have some consideration on...

[kovidgoyal]

Originally Posted by aapezzuto

I'm not happy with having all elements needing to be reachable in a linear way, This makes footnotes still a problem. But the real problem is that epub supports so many things that I don't want to support (Im looking for a structural markup + css) and by the time that I normalize epub its another format, it just happens to be epub compliant.

The best solution for footnotes once you get rid of the concept of a page is hyperlinks to the end of the file (with a working back button that's almost as convenient as "bottom of page" footnotes).

[Dave Berk]

First, what I'm asking may be stupid, I simply don't know enough. I'm just trying to help .

Originally Posted by aapezzuto

But the real problem is that epub supports so many things that I don't want to support

Can't you use just a subset of epub, say, xtml (not all tags) + css? that should be plenty enough.

Originally Posted by aapezzuto

I'm not happy with having all elements needing to be reachable in a linear way, This makes footnotes still a problem.

In what way?
Also what do you mean "having all elements needing to be reachable in a linear way", regarding the xml/xhtml tree structure, regarding file reading/writing?

[llasram]

Originally Posted by aapezzuto

I'm not happy with having all elements needing to be reachable in a linear way, This makes footnotes still a problem.

You appear to have missed the //spine/itemref[@linear] attribute. It allows one to specify sections which are not part of the linear document flow and which the reader may choose to make accessible only via links to those sections.

Originally Posted by aapezzuto

But the real problem is that epub supports so many things that I don't want to support (Im looking for a structural markup + css) and by the time that I normalize epub its another format, it just happens to be epub compliant.

What don't you want to support exactly? The XML islands and fall-back support for non-standard media types? An epub reader must implement the entire standard to be compliant, but there's no reason an authoring tool can't deal with only a well-defined subset.

[kfarmer]

Originally Posted by tompe

Well, for an open source project one use of it is to reuse the code. Only seeing code reuse as a convenience for the developer is a mistake.

And where is a user going to encounter source code? Most non-developers don't care, and those that do, frankly, you should watch carefully (either they lied and really are developers, or they're apt to break something).

Back to the original point I was trying to make, which is why I think the choice of canonical format being epub is immaterial -- I've rethought, and I agree that it *is* important. It's important in the sense that I think it would be *better* to use a new format. The reason I think this way is to ensure freedom to extend the system.

Let's say they did use epub internally. This means there are going to be dependencies in a variety of places:

- parsers/generators of other (non-epub) formats;
- the application's manipulation of the internal (epub) document

Let's say further that a new format comes out, very compelling, that includes support for animated vector graphics (probably *not* on a PRS-505). They now have (assuming epub doesn't have animated vector graphics -- if it does I'm impressed) a rather *painful* opportunity to add support for this format without a rearchitecture of the system. They at least have to branch off of epub, and deal with any breakages, on a piece of software that has, for all intents, shipped.

Consider instead that from the beginning they anticipate this possibility, and instead branched off of some format, or created their own, before all these dependencies were established. Then, they can have the following model:

(Editor) -> (canonical document) <-> (document-foo translator) <-> (foo-file parser/generator)

If a new file format foo2 comes out, they just add the parser/generator to convert, say, foo2 to their canonical form. If foo3 comes out and requires extending their canonical model and adding editor support, they can do so without impacting any of the other translators. The file parsers are never impacted, since they translate to fixed forms. Just add some metadata to the formats to list their feature support, and they can detect whether you'll lose formatting at all going from one type to another.

So this leads to the idea that the canonical form, whatever it is, should be entirely under our control. They should own it, it should be theirs free to modify. In exchange, they offer the ability to export to a variety of formats, potentially losing formatting that isn't supported at the destination.

The canonical form could be a true *branch* of epub, but it should *not* be epub itself.

[kovidgoyal]

@kfarmer
I'm afraid you don't really get open source development. Open source development is all about interoperability and re-use. The end users of open source development are other open source developers, not users. That does not mean that making applications easy to use for conventional end users is not important, but it does mean that it is *more* important to make an application easy to use for other developers. That is how open source thrives.

An important part of the process is using standards wherever possible. Mutating formats arbitrarily to support today's hot new feature is the way to make your application the Internet Explorer of whatever field it is in.

Incidentally, epub as a markup specification can support any image format. Rendering the images is of course dependent on support for that format in reader software.

[paulnjean]

Hey all, I sure would like to have a program that converts, hassle free of course, all of the free pdf files I have to my Reb 1100. My RCA reader still works and I still enjoy it. Sure there are problems with it but an easy to use program would extend its usefulness. If I could reformat my work rulebooks and such and carry them on my reader I would be able to get rid of about 17lb. of books that I carry with me every day. If any one knows of such a program I would like to get it. Paul

[DaleDe]

Originally Posted by paulnjean

Hey all, I sure would like to have a program that converts, hassle free of course, all of the free pdf files I have to my Reb 1100. My RCA reader still works and I still enjoy it. Sure there are problems with it but an easy to use program would extend its usefulness. If I could reformat my work rulebooks and such and carry them on my reader I would be able to get rid of about 17lb. of books that I carry with me every day. If any one knows of such a program I would like to get it. Paul

If images will work for you then try:
https://www.mobileread.com/forums/showthread.php?t=21906

Dale

[kfarmer]

Originally Posted by kovidgoyal

@kfarmer
I'm afraid you don't really get open source development. Open source development is all about interoperability and re-use. The end users of open source development are other open source developers, not users. That does not mean that making applications easy to use for conventional end users is not important, but it does mean that it is *more* important to make an application easy to use for other developers. That is how open source thrives.

(I spend 26 years programming, 10 of them professionally, all but 2 of those in startups, and I don't "get" sharing source code with others... I'm amazed!)

But let's be serious. Applications exist to benefit the end-user. Without a user, an application has little reason to exist beyond academic experimentation (itself a fine past-time, but hardly worth advertising as a product). What I described was a widely used -- consider multi-format editors in general -- means to maximize the potential benefit to users, while buffering against the types of changes seen in the real world.

Originally Posted by kovidgoyal

An important part of the process is using standards wherever possible. Mutating formats arbitrarily to support today's hot new feature is the way to make your application the Internet Explorer of whatever field it is in.

Standards make sense for externals, and on that we can agree. But internals can (should) be using whatever gives them the advantage. What I was describing was the internals -- what the OP spoke of as what the application would "think in". You're focussing (apprently) on externals, and nothing I've suggested prevents the architecture so described from emitting pure, clean epub if it wanted. But pure, clean epub is just as much a lock-in as any other model developed by any third party.

And I'll state it: standards are meant to be abandoned when they become a hinderence. That's why there *is* a ratification process to get better standards, and every so-called standard started out as a non-standard.

Originally Posted by kovidgoyal

Incidentally, epub as a markup specification can support any image format. Rendering the images is of course dependent on support for that format in reader software.

Lacking the epub spec I'll accept that. So, does it (like pretty much every other container format) just deal with it as a blob? In which case you still face the problem of deserializing, editting, and serializing new formats. My suggestion still stands.

Gotta run to a late lunch and a long meeting. TTYL

[kovidgoyal]

Originally Posted by kfarmer

But let's be serious. Applications exist to benefit the end-user. Without a user, an application has little reason to exist beyond academic experimentation (itself a fine past-time, but hardly worth advertising as a product). What I described was a widely used -- consider multi-format editors in general -- means to maximize the potential benefit to users, while buffering against the types of changes seen in the real world.

Yes but an open source application has a more complex user base than a commercial one. While the so called "average end users" may far outnumber "developer end users" for the long term success of an open source project it has to appeal to both constituencies.

Originally Posted by kfarmer

Standards make sense for externals, and on that we can agree. But internals can (should) be using whatever gives them the advantage. What I was describing was the internals -- what the OP spoke of as what the application would "think in". You're focussing (apprently) on externals, and nothing I've suggested prevents the architecture so described from emitting pure, clean epub if it wanted. But pure, clean epub is just as much a lock-in as any other model developed by any third party.

No the OP is talking about what file format to use as a master format. What that means is his program will accept input in a number of formats, convert to the master format, edit the master format, save it for later re-edit and output to a number of other formats from the master format on-demand. That means that only the master format will contain the full representation of all information about the document. That can only be considered an internal format if he intends never to let anybody else write code to process it. And that means that end users will be locked into his app.

[kfarmer]

Originally Posted by kovidgoyal

Yes but an open source application has a more complex user base than a commercial one. While the so called "average end users" may far outnumber "developer end users" for the long term success of an open source project it has to appeal to both constituencies.

And the better tact on that is to not limit yourself to someone else's view of documents. There have been too many go-nowhere projects that were popular only among the small group of developers congratulating themselves on not using anything original and coding themselves into corners. I know, because I've been involved in a few (yes, I actually *do* know something about open development, and if they'd only listened to me... ).

Originally Posted by kovidgoyal

No the OP is talking about what file format to use as a master format. What that means is his program will accept input in a number of formats, convert to the master format, edit the master format, save it for later re-edit and output to a number of other formats from the master format on-demand. That means that only the master format will contain the full representation of all information about the document. That can only be considered an internal format if he intends never to let anybody else write code to process it. And that means that end users will be locked into his app.

I read through the thread, and I think you're reading a bit much into it. His stated hopes are that there would be a nearly infinite flexibility to support different file formats. He himself *suggested* using an xml file format, though I think that may have been putting the file before the memory, so to speak. Even you suggested using an internal data structure, just as I have. (If you've been interpreting my statements to mean that I think he should create a new markup language, my apologies -- I'm talking about canonical structures *in memory*. Why thrash the disk all the time -- it wastes cycles for no real benefit given today's RAM capacities.)

Given what I described, it would be a simple matter to convert between formats using that code as a library. There is no lock to be had, any more than there is lock to be had in any converter.

[kovidgoyal]

ah well we seem to be talking past each other. I am indeed concerned with the on disk format. The in memory format could be anything he likes, but from the perspective of plugin development, I'd vote for a nice object model that can be exposed in a scripting language.

And I apologize for saying you don't get open source development, that was a little too harsh.

[aapezzuto]

Originally Posted by kovidgoyal

Open source development is all about interoperability and re-use. The end users of open source development are other open source developers, not users. That does not mean that making applications easy to use for conventional end users is not important, but it does mean that it is *more* important to make an application easy to use for other developers. That is how open source thrives.

This is less of an issue here than in other cases. This is open source because we believe that any work we do should be protected against isolation and commercial mercenary practices. Not that I don't love the Darwinian nature of capitalism, but this market is still so young that a few bones need to be thrown to it. I hope that the nature of the code documents A way that documents can be converted, and ONE model of how that data can be viewed structurally. If the actual code is useful to someone great... I just want to make sure that the code logic is available... But that is just another purpose of open source, Free as in speech, and as in beer.

I do want to have a way to save a file in a format that is as similar as how the program thinks as possible... In my mind, if i use just a small and well defined set of epub, than i feel i have made a new format, it just happens to be compliant with another format. I strongly doubt I will make anything up from scratch, and it will not be a compiled or binary format.

After this now almost 90 posts I look forward to Greg getting back into the country (he is in Ireland proposing to his girlfriend of 3+ years) and seeing what he things, and where his views are mutable. At this point continuing to focus on just simple feature lists, perhaps in order of importance, would be most helpful... so that I don't forget anything obvious during the first round of design. Don't get me wrong, I love talking/arguing/ranting/etc... about somethings true/higher purpose, ethical imparitives and all that, but with as much new stuff as is comming into my head (just read the docbook spec and the epub specs [3 of them] again, to make sure I understood them) and I have the twins still :-)

oh speaking of epub and images... for any non text media, you can list a new file format to display, and dedicate it space in the object model (sort of a media box/blob), but you must specify a fall back chain that eventually hits one of the required formats. Just throwing that out there sense there was a question about it. And yes, I know that the media evolution was just an example of goals shifting.

Im going to bed