PDF is not an eBook format

[DawnFalcon]

Originally Posted by WillAdams

Tuna,

A LaTeX install can easily be in the 10s of MBs (see my earlier post and links)

Without a lot of common dependencies, yes. And I'd point out that 10's of MB is larger than any current firmware size on an Ebook reader. In practice, you're looking at anything larger than a few MB being impossible to squeeze on.

(Heck, remember that they can't put ePuB support on the PRS-500 because they can't find the space)

[tompe]

Originally Posted by DawnFalcon

Without a lot of common dependencies, yes. And I'd point out that 10's of MB is larger than any current firmware size on an Ebook reader. In practice, you're looking at anything larger than a few MB being impossible to squeeze on.

You can put it on the internal memory so this is not a problem.

(Heck, remember that they can't put ePuB support on the PRS-500 because they can't find the space)

This is not the reason.

[DawnFalcon]

So what was the reason then?

And no, you can't count on loading it into the internal memory and have it run at any sort of deacent speed.

[Tuna]

Originally Posted by tompe

You can put it on the internal memory so this is not a problem.

Internal flash is not the same as memory that can be used to run software (RAM).

The PRS-505 only has 64Mb of ram to run software in - about half of that will be used by operating system, and another 10 or so by the standard user interface. Factor in loading the book into memory, and you've got very little left for the renderer.

10's of megabytes IS too big for embedded systems, which is what e-readers are.

[frabjous]

Originally Posted by ahi

If I understand correctly, LaTeX hyphenation patterns are not wordlists, but pattern lists. Which means there is no automatic way of identifying words for which correct hyphenation patterns are not known.

I don't understand why it would make a difference. Whether the rules are stored as complete words or as patterns, they're applied to words at processing time, and there's no way it's not possible for it to recognize when it can't find any appropriate patterns in a word.

All the other examples you've given about other languages just add a level of complexity. If humans can correctly hyphenate, so can a computer. I can't for the life of me imagine what argument there could be for thinking otherwise. Maybe the software isn't quite there yet, but it's getting closer, and we might as well use the best currently available.

Frabjous - A LaTeX install is also, at a minimum, hundreds of meg in size. This is one of the things I'm on about - it's not suitable as a typological processor in a low-resource environment. Typography is demnstrably mostly-solveable, by brute force, but that that soloution is not applicable to low-power devices.

MikTeX portable is about 90 MB in size, and it's a fully functioning and respectable LaTeX distribution -- yes, LaTeX, not TeX -- and widely used even by those who could have dedicated more space. I'm not saying that's tiny by comparison to what's on current readers, but I'd be more than happy to dedicate that much memory even in my 505 to get improved results, and with the next generation of readers, this amount of space will not be too much. What's possible on a PRS-500 hardly matters looking forward. None of those will be sold anymore. RAM is probably a bigger issue, but I really don't think LaTeX requires very much at all--I've admitted all along that hardware might not be there yet, but I haven't seen anything to convince me it won't be soon.

And if not, the conclusion is obvious: ebook sized PDFs (or what amounts to the same) is the only remaining viable option. But I don't think it has to be that long-term.

The Knuth & Plass paper which I cited has a formal proof and discussion of the impossibility of finding the perfect set of breaks for a paragraph.

I haven't seen this proof, but I think you've misunderstood the practical importance of such proofs in general (which is almost nonexistent). At worse it shows that there will be some *theoretical* cases, unlikely ever to be actualized, in which an algorithm will give less than perfect cases. And we are not getting perfect results now even with human-designed books.

It reminds me of a conversation I recently had with a colleague. I teach logic at a large university -- a colleague and I alternate teaching Intro to Logic every other semester. I was discussing with him the possibility of writing some software to check our students' answers. One thing we do is assign translations into first-order predicate logic, but we accept all logically equivalent answers as correct. My colleague tried to convince me that software would be worthless here, since by Church's theorem, first-order logic is recursively undecidable, and thus an algorithm that tells you whether a given formula is logically equivalent to another isn't technically possible (it's not a Turing-computable function). But as I pointed out to him at the time, it's absolutely ridiculous to think that we couldn't write software that checked for equivalence over all models of cardinality of finite cardinality n, and that the chances an introductory student would turn in an answer that was equivalent over a large finite cardinality but not equivalent in some infinite domains, with the problems we give, is astronomical, and that, if by some miracle, they managed it, it would be cruel not to give them full credit.

If an algorithm tries 5 million different layouts for a paragraph and selects the best one of those, but "theoretically" it would be possible to get a better one after 5 million more passes (or just "out there" which it could never find), is not interesting or relevant at all practically speaking. An algorithm that would give as good results as your typical human typographer is certainly still possible.-- and likely you'd want to deliberately make it not seek absolute perfection just to save processing power anyway.

[tompe]

Originally Posted by Tuna

Internal flash is not the same as memory that can be used to run software (RAM).

The PRS-505 only has 64Mb of ram to run software in - about half of that will be used by operating system, and another 10 or so by the standard user interface. Factor in loading the book into memory, and you've got very little left for the renderer.

10's of megabytes IS too big for embedded systems, which is what e-readers are.

The TeX program is not so big. It is all the data files that are not read into RAM at the same time that takes the space and they just need to be stored someplace.

And the claim I objected to was that the firmware size was to small to store the files needed to run LaTeX. RAM size have not been mentioned before.

[DawnFalcon]

...You can do it by paging parts in and out of memory fron the flash disk. However, the result is like to be unacceptably slow.

[WillAdams]

frabjous, I've read the paper, and I work in this field on a daily basis, and stand by my statement.

Automated H&J has significant limitations and in any lengthy work will have multiple instances of breaks which should be manually tweaked to prevent egregious errors such as stacks.

William

[ahi]

Frabjuous, LaTeX cannot find enough correct hyphenation patterns for any given text of meaningful length to avoid overfull boxes... unless you basically include everything in a \begin{sloppypar} ... \end{sloppypar} ... which results in underfull boxes galore.

That's what I do personally... but the moment you do that, the quality takes a giant leap downwards. The fact is, with high quality typography being the goal... LaTeX is totally incapable to find adequate(ly numerous) hyphenation patterns for pretty much any given text of as little as a few pages.

This is a fact. Anything beyond this is just another "but it just has to be good enough" argument.

Which is fine. A poorly done LaTeX PDF is still ten (or a hundred?) times better looking than anything in one of the alternative formats. But the fact remains: quality typography requires fixed layouts and manual intervention... hyphenation being one of the big reasons why.

The fact that we can arbitrarily lower quality expectations to the point where even software-hyphenation will do is handy... but it's no equal value substitute for manual attention + fixed layout.

- Ahi

[frabjous]

Originally Posted by ahi

Frabjuous, LaTeX cannot find enough correct hyphenation patterns for any given text of meaningful length to avoid overfull boxes... unless you basically include everything in a \begin{sloppypar} ... \end{sloppypar} ... which results in underfull boxes galore.

This is true only if you leave things at their default settings, which do assume a relatively normal paper page-size, but all you just need to change the \emergencystretch variable to something like 11pt, and you'll get rid of the overfull boxes.

Depending on the line width, you may get a lot of underfull boxes, though this really this has nothing to do with not "finding" hyphenation. If the line width is too small, there quite simply aren't enough choices to avoid some underfull lines, and it's something you have to put up with. No amount of even manual tweaking is going to fix that. You'll see such result in newsprint all the time.

This is not a problem with LaTeX -- it's a matter of facts of geometry. When line lengths are smaller, there are fewer options with regard to what can be put on a given line, and this increases the possibility that no "acceptable" breaking can be found. Do you really think the problem is not finding the right hyphenation patterns? I think even if hyphenation rules were given individually for every word, you'd still see this phenomenon with shorter lines.

Again, I can't think of any reason at all to think that anything a human can do, a computer couldn't be programmed to do, including identifying stacks and rivers and including them in its determination of what layouts are better than others. If LaTeX isn't now doing what a human would do, give me some reason to think that a better algorithm isn't possible. No reason has been given. (Theoretical arguments about the problem not being NP-complete are irrelevant, and no one has done anything but "cite authority" otherwise.)

Surely what humans do when they hand-tweak is still a rule-governed activity which has serious constraints put on by the nature of the text. When aesthetics are involved, it may be hard to put those rules into words, or code an algorithm to do the same, but I can't think of why it's not possible. Maybe LaTeX isn't there yet, and maybe it'll be awhile until we are, but it just seems to me to be some kind of strange of human-elitism to think it's not even possible. People have made claims of the form "computers will never be able to do X" numerous times in the past and have been proven wrong time and again. Maybe this is not the best place for an argument over artificial intelligence though.

But I'm not sure what we're arguing about. I think we agree that in the meantime, if we want a reflowable format, let's just use the best algorithm available, and put up with the imperfections. I certainly wouldn't be opposed to including, within an ebook, a "default" setting that had a fixed format established by a human, that was deviated from only if the user wanted to reflow the text to customize the font size or margins, etc. I'd be all for that.

[ahi]

Originally Posted by frabjous

Do you really think the problem is not finding the right hyphenation patterns?

Since inserting soft-hyphens at the right places is a more (the most?) elegant way of solving the overfull box problem, yes I do.

Fiddling with emergency stretch is on par with (if, perhaps, not as drastic as) putting everything in sloppypar. No?

- Ahi

[Jellby]

Originally Posted by ahi

Frabjuous, LaTeX cannot find enough correct hyphenation patterns for any given text of meaningful length to avoid overfull boxes... unless you basically include everything in a \begin{sloppypar} ... \end{sloppypar} ... which results in underfull boxes galore.

Note that:

a): You'll get the same thing by just issuing \sloppy at the beginning of your document, though I usually define a \notsosloppy command that does not take such drastic measures. Using the microtype package will also help incresing the quality.

b) Even if the hyphenation points are found, lots of hypenated words degrade the quality just as much as underfull boxes...

[frabjous]

Originally Posted by ahi

Since inserting soft-hyphens at the right places is a more (the most?) elegant way of solving the overfull box problem, yes I do.

The point is, LaTeX already knows where it's appropriate to put soft-hyphens, or if it doesn't, can be extended to. Without help from the book maker, it doesn't for some words, but if a book were properly made, the book maker would include rules for all the words LaTeX doesn't know (proper names, hand-made compounds, etc.). That should remove most overfull boxes, yes, but without hand-tweaking the individual lines. My point was that if the line length is small enough, sometimes it isn't enough, and then the problem isn't hyphenation at all, and there's little choice but to accept a few underfull boxes. LaTeX can be programmed to know all possible hyphenation points -- that won't entirely solve the problem, though.

Fiddling with emergency stretch is on par with (if, perhaps, not as drastic as) putting everything in sloppypar. No?

Yes, but the real effect is only that it's going to put up with more underfull boxes rather than stick things into the margins (create an overfull box), with is LaTeX's default way of annoying the user into hand-tweaking. With a full page size, usually inserting a new hyphenation rule is enough, but where it isn't, I don't see an option but to put up with underfull boxes. If we're going to be using very different line lengths and page sizes than has been typical in past typography, we're simply not going to be able to demand the same things.

And of course, relying on current rendering programs is in effect putting up with underfull boxes galore galore galore...

[ahi]

Originally Posted by Jellby

Note that:

a): You'll get the same thing by just issuing \sloppy at the beginning of your document, though I usually define a \notsosloppy command that does not take such drastic measures. Using the microtype package will also help incresing the quality.

b) Even if the hyphenation points are found, lots of hypenated words degrade the quality just as much as underfull boxes...

Yes, I know about \sloppy, but didn't for a long time, so wanted to put it in simpler terms to avoid causing confusion and/or having to explain this correspondence. Thank you for explaining it though.

An overabundance of hyphenation will degrade quality, yes. But it's not about having a lot of them, but having the right ones.

And I do believe LaTeX is aware in its badness calculation that too much hyphenation is not great, so that's a less likely problem... though certainly a potential one. Or am I wrong?

- Ahi

[ahi]

Originally Posted by frabjous

The point is, LaTeX already knows where it's appropriate to put soft-hyphens, or if it doesn't, can be extended to.

The plausibility of hyphenation software being as good as a human being was discussed already. After some superhuman amount of distributed effort (presumably going on for years, if not decades) it might start to approach.

In the meantime, with an average-sized book, an hour or two worth of human attention will continue to achieve the same thing. We're definitely desperately looking for suboptimal solutions to already solved problems.

Originally Posted by frabjous

If we're going to be using very different line lengths and page sizes than has been typical in past typography, we're simply not going to be able to demand the same things.

I will be demanding a fixed layout and (at least some) manual attention. Given how little this actually requires from the publisher... I'm not sure why others see it as impractically onerous.

- Ahi