PDF is not an eBook format

[frabjous]

cmdahler's post is wonderful, but it isn't really an argument for PDF as an ebook format, but .tex... if the .tex compiling is done by the reader itself, it obviously doesn't matter whether it does it to PDF or to DVI or to some future unknown format so long as it can do so quickly and displays the same.

LaTeX of course is evolving, with LaTeX 3 under development... branches like XeLaTeX are probably already more suitable for ebook readers. Probably what we'd want is a suitably tweaked new branch.

...or, what's almost the same, a tweaked version of HTML or XHTML that has the same power, along with a rendering algorithm that properly obeys good rules of typography, but can adapt those rules somewhat to the preferences of the reader (font size, and so on).

"Tagged PDF" is also a promising new direction, but I think the kinds of tags involved would have to expand dramatically... before long you'd be back to a purely semantic mark-up language.

We shouldn't have to give up kerning, font choice, ligatures, properly displayed and spaced mathematics, and all the other things professional typesetting provides but currently impossible with anything but a PDF on our ebook readers. And this should come from a open source format that it is easily to understand and manipulate (which makes .tex far preferable, IMHO to things like MathML, which looks like a nightmare to manually edit).

ADE already does fairly decent widow/orphan control. It won't allow single lines of a paragraph to stand alone at the end of a page or at the beginning of a page. The result of this is that you may end up with 2 blank lines at the end of a page (since the bumped paragraph would consist of 3 lines). Something they could do is modulate the leading a tiny bit to spread everything out to make those 2 missing lines less noticeable.

I wouldn't call that "decent". It's too restrictive. TeX has a "badness" score that can be manually set to various levels for various undesirable features. It'll normally avoid a widow or orphan, but if avoiding badness there creates enough badness elsewhere, it will do an orphan or widow instead of creating that badness. Sometimes, it's very ugly to prevent widows and, especially, orphans, when the space becomes to great. TeX also does "feathering" to space out pages to make them the same height... most of the time. Again, there's a badness score associated with not doing so, and if doing so creates more badness elsewhere it won't.

And the document can customize its own badness weights if it wants to, or leave them at their defaults depending on its particular style and demands.

[WillAdams]

There are several things which are needed and it should be admitted up front that a couple of problems are intractable and will not be generally solved:

- First, .pdfs need to be available at size(s) which are optimized for screen size(s) / proportions

- second, .pdfs need to have internal tagging so that _under user control_ they can be re-flowed at need, or read audibly by a screen reader &c. --- note that this includes additional manual formatting to indicate break points which must not be allowed, as well as suggestions on where breaks are allowable

- third, viewing programs need to have more sophisticated algorithms for re-flowing text, considering not just an entire paragraph, but an entire page (until such time as there're widely available dual-screen readers at which point one will need to consider a spread)

- while widows and orphans can be precluded by appropriate settings in a tools like TeX, other aspects of line-breaking / hyphenation and paragraph and page building / composition are more intractable --- it's incredibly difficult to anticipate and prevent stacks for example, and if there're figures or tables w/ matching references, showing these on a page together can sometimes require re-laying an entire chapter

Granted, for straight text, w/ a simpler format, one can achieve a very usable result just by tying together words strategically w/ non-breaking spaces and setting sensible defaults, but it would be terribly limiting if ebooks were to adhere to the limitations imposed by such --- there was a time when writers would naturally switch from writing to drawing and back in a given text and ebooks should allow such flexibility.

William

[Jellby]

Originally Posted by Abecedary

ADE already does fairly decent widow/orphan control. It won't allow single lines of a paragraph to stand alone at the end of a page or at the beginning of a page. The result of this is that you may end up with 2 blank lines at the end of a page (since the bumped paragraph would consist of 3 lines). Something they could do is modulate the leading a tiny bit to spread everything out to make those 2 missing lines less noticeable.

Note that CSS2 has "widows" and "orphans" properties (both default to 2):
http://www.w3.org/TR/CSS2/page.html#break-inside

[WillAdams]

CSS2 having the property is only the first step --- readers / viewing program implementations need to honour it --- has anyone put together a test suite or compiled a list of programs and their behaviour regarding this?

William

[pdurrant]

Originally Posted by Abecedary

And aren't the processors in most readers about the same as what you'd find in a PDA?

Perhaps in a modern PDA. But the Mobipocket reader (for example) was designed at a time when the processor in a PDA was a 16MHz or 32MHz processor. Not the 200MHz or 400MHz processors in dedicated ebook readers.

There's a /lot/ more processor power available now than there was. It just isn't being used properly.

[EowynCarter]

Originally Posted by WillAdams

- second, .pdfs need to have internal tagging so that _under user control_ they can be re-flowed at need, or read audibly by a screen reader &c. --- note that this includes additional manual formatting to indicate break points which must not be allowed, as well as suggestions on where breaks are allowable

Reflow ? it's more of an hack. Pdf is not meant to be reflowed. If you do that, pdf won't be pdf anymore.

Granted, for straight text, w/ a simpler format, one can achieve a very usable result just by tying together words strategically w/ non-breaking spaces and setting sensible defaults, but it would be terribly limiting if ebooks were to adhere to the limitations imposed by such --- there was a time when writers would naturally switch from writing to drawing and back in a given text and ebooks should allow such flexibility.

ePub can use images, different fonts, tables. Html and css allows for almost anything.

- First, .pdfs need to be available at size(s) which are optimized for screen size(s) / proportions

Once again, the publishers won't proofread one e-book. So asking them to proffed that many version would be suicide.

[Tuna]

Originally Posted by frabjous

LaTeX of course is evolving, with LaTeX 3 under development... branches like XeLaTeX are probably already more suitable for ebook readers. Probably what we'd want is a suitably tweaked new branch.

...or, what's almost the same, a tweaked version of HTML or XHTML that has the same power, along with a rendering algorithm that properly obeys good rules of typography, but can adapt those rules somewhat to the preferences of the reader (font size, and so on).

Um.. ePub is a tweaked version of HTML. Rendering engines for epub are improving.

The issue with rendering engines is that the processors on most ebooks are not that powerful in order to give good battery life. They work by waking up, doing the minimal amount of work and going back to sleep again. Doing a full layout of a book as it is opened would shorten battery life from weeks to hours.

[Jellby]

I have a dream...

Programs like Calibre could be used to turn an ePUB (or whatever) into a PDF customized for a given page and font size, with the best possible automatic layout, and even with user input in difficult cases (please mark acceptable hyphenation points for the following words). Then we'd have good things from both worlds (short of hand-tailored PDFs): ePUB for distribution, PDF for reading.

[frabjous]

Originally Posted by Tuna

Um.. ePub is a tweaked version of HTML. Rendering engines for epub are improving.

ePub can do almost nothing that HTML can't do; that was the point. It needs to be tweaked further. I don't care if something HTML or XML based is tweaked to come closer something like TeX, or the tweaking is in the other direction, though from what I've seen TeX, or at least LaTeX, is much more user-friendly to edit directly than, e.g., MathML.

I'm not an expert about what drains batteries on the readers, but I would have guessed that processing is not what's draining the battery. LaTeX is a fast and efficient program; the core TeX algorithm is more than 30 years old, and runs on hardware from the 80s. If the hardware for readers isn't there yet, I can't imagine it's too far off.

Originally Posted by Jellby

I have a dream...

Programs like Calibre could be used to turn an ePUB (or whatever) into a PDF customized for a given page and font size, with the best possible automatic layout, and even with user input in difficult cases (please mark acceptable hyphenation points for the following words). Then we'd have good things from both worlds (short of hand-tailored PDFs): ePUB for distribution, PDF for reading.

How hard would it be to combine the open source epub2html, html2tex and pdflatex tools out there to work something like this up? It might not be possible under the guise of calibre, but for a system that already had a full LaTeX distribution installed, this could be scripted using existing resources, couldn't it?

[EowynCarter]

Originally Posted by Jellby

I have a dream...

Programs like Calibre could be used to turn an ePUB (or whatever) into a PDF customized for a given page and font size, with the best possible automatic layout, and even with user input in difficult cases (please mark acceptable hyphenation points for the following words). Then we'd have good things from both worlds (short of hand-tailored PDFs): ePUB for distribution, PDF for reading.

Calibre already does pdf output right ?
And i wouldn't bother converting

[WillAdams]

I don't see how tagging a .pdf is a ``hack'' --- it's a natural extension of the format, one which has been long overdue.

``html and css allow for almost anything''

The corollary of which is ``there are things which can't be done in html and css''

Show me a mathml implementation done using html and css or an optimized h&j algorithm and I'll be interested --- until then I'll find tagged .pdf a better solution for my needs --- if your needs are satisfied by html and css's simplistic capabilities that's fine, for you. Doesn't work for me, I need better capabilities and more control.

William

[EowynCarter]

Originally Posted by WillAdams

I don't see how tagging a .pdf is a ``hack'' --- it's a natural extension of the format, one which has been long overdue.

``html and css allow for almost anything''

The corollary of which is ``there are things which can't be done in html and css''

Show me a mathml implementation done using html and css or an optimized h&j algorithm and I'll be interested --- until then I'll find tagged .pdf a better solution for my needs --- if your needs are satisfied by html and css's simplistic capabilities that's fine, for you. Doesn't work for me, I need better capabilities and more control.

William

Mathml ? you mean displaying formula ?

Edit : ah, right. (Tanks you wikipedia).
As mathml is Xml, shoudn't be that hard to have it included into ePub. The main job there would be adobe's.
No formula support is a bit annoying sometimes.

[Jellby]

Originally Posted by EowynCarter

Calibre already does pdf output right ?
And i wouldn't bother converting

But it does not have "high typographic quality" output. The important part of my message was letting the computer doing the computing-intensive formatting once, and have the reader just display it. Of course, if the formatting is going to be just your default browser-like formatting, it's not worth it.

Those who like ePUB, can use plain ePUB.
Those who like PDF, can use plain PDF.
Those who would like ePUB if the reader had a better quality renderer, can convert it to PDF using a better quality renderer

[EowynCarter]

You're trying to fix the old, rather than improve the new.
ePub is not perfect, far from it, but it have advantage pdf will never have (openness and ease of editing). And it can still be improved.
And an other thing, when doing a mobi -> ePub conversion, it works fine. With pdf, it get a mess because pdf actually hard write headers and footer.

Right now, anything pdf is totaly unreadable on my reader. And even with reflow, still won't be. I wish some publishers with give up pdf so i can buy their e-books.

The only way pdf can think of working is having multiple version with different font size and scren size, and the publishers will never bother themselves with that.

Even Eyrolles have gotten that and they are beginning to experiment with ePub.

[Sonist]

Originally Posted by EowynCarter

... Right now, anything pdf is totaly unreadable on my reader. And even with reflow, still won't be. I wish some publishers with give up pdf so i can buy their e-books....

You are like a guy riling against the widescreen movie format, because your cheap B&W TV watch can't display it properly....

The lowest common denominator may not be good enough for everyone.