Using regex for more elegant hyphenation and word wrap

[Psymon]

I thought this link was rather interesting...

Originally Posted by Tex2002ans

Here, you can see a comparison in justification/hyphenation between Word/InDesign/LaTeX:

https://tex.stackexchange.com/questi...esetting-ligat

...if only because that's what I'd pretty much accomplished. Before I'd coded my book the way that I did, it looked pretty much like the Word example given on that page, with tons of hyphenation all over the place (way more than was needed), but after I'd added "selective hyphenation" to my book -- which I'd been doing manually before, but which would have now been incredibly easy to do using a regex S&R -- it looked pretty much like the LaTex example given there.

I have to admit, though, that while you all have convinced me to just scrap it, and just let hyphenation happen wherever the device (or app) sees fit, and only turn it "off" myself for things like headings, at the same time I can't say that I'm entirely convinced that what I was doing was, indeed, totally such a bad thing to do. I was actually pretty proud of what I'd been doing, I thought it had been working out remarkably well, especially since I'd tested it out at all kinds of different font sizes, etc...

...but on the plus side, it certainly does make things a lot easier for me, coding-wise, to just not even think about it at all anymore. I still can't help but wonder, though, if you've all inadvertently convinced me to scrap something that was, in fact, a really good idea. :/

[Psymon]

Originally Posted by PeterT

Try:

Search: <span class="hyph">(.+?)</span>
Replace: \1

Oh, I forgot to say thank you very much for that, Peter -- worked like a charm! I definitely need to read up and learn more about this regex stuff, but that definitely helped me "clean up" this current issue for the short haul (without having to finish my PhD in regex first).

[Tex2002ans]

Originally Posted by Psymon

...if only because that's what I'd pretty much accomplished. Before I'd coded my book the way that I did, it looked pretty much like the Word example given on that page, with tons of hyphenation all over the place (way more than was needed)[...]

But again, that comparison was showing off differences in the ALGORITHM level.

Your situation, is similar to trying to make the "Word" document look nicer, by adding manual hyphens everywhere. You want to nitpick every single little thing.... but then you change one variable (font size, page size, margins, font), and all those manual hyphens you added will be clogging things up and making things WORSE, compared to if you just left the algorithm alone.

Your "surface" changes will ONLY work in Word, but switch over to a slightly different algorithm (like LibreOffice), and you have indirectly caused yourself a whole host of problems, because you decided to go "too specific" (in my latest terminology).

Each time you tweak a variable, you would have to go back, remove much of your manual fixes, and tweak things in NEW spots (if you wanted to nitpick the thing to death again).

Adding (or disabling) hyphens in certain spots, "solves" the problem on the SURFACE, but not the underlying problem.

Similar with globally denying all auto-hyphenation, and only enabling it on words past 8 characters. You are getting in the way of the algorithm, what if it thinks it is better to split a 6-letter word at position X, but you, in your INFINITE wisdom told it "no." You are indirectly causing inferior hyphenation/spacing in another location. (And again, maybe this inferior spacing problem doesn't appear to your human eyes with font A at font size B with margins C, but it WILL appear if you use font X at font size Y at margins Z).

Note: I started to notice a lot of this stuff while fiddling around with LaTeX, and generating multiple versions of the same book, while tweaking a few variables here and there. A lot of this stuff, I would have never KNOWN, unless I was messing with different font sizes + fonts + margins.

(Some images may be repeats for organizational purposes).

Font Change (Linux Libertine -> Times New Roman):



Note: These look nearly identical, but slight differences in kerning (Times New Roman is a little tighter).

As you can see in the Libertine version, "his-tory" is hyphenated between lines. That is 7 characters. This sort of situation would have been DENIED if you go "8 characters or more".

Side Note: No no, don't come back making a post saying "then I will enable hyphenation on all the words with 7 OR MORE CHARACTERS instead!"

Page Size Changes (5.5"x8.5", 6"x9", 7"x10"):



Note: Minor margin tweaks between them.

Font Size Change (12pt -> 13pt):



Font Size Change (12pt -> 11pt):



And this is keeping nearly all variables CONSTANT, and only tweaking ONE thing at a time. This is not taking into account differences in programs (for example, Word/LibreOffice have slightly different hyphenation/justification algorithms, iBooks/Marvin will hyphenate/justify slightly differently, Nook might justify/hyphenate one way, and Kobo might justify/hyphenate a completely different way). Maybe one says "allow hyphens on every line", and another one says "only allow hyphens twice per paragraph".

Maybe in the future, you (or someone else) wants to change your book into a multi-column format, but now your "denying hyphenation besides 8 characters or more" will really mess that up.

Originally Posted by Psymon

I have to admit, though, that while you all have convinced me to just scrap it, and just let hyphenation happen wherever the device (or app) sees fit, and only turn it "off" myself for things like headings, at the same time I can't say that I'm entirely convinced that what I was doing was, indeed, totally such a bad thing to do. I was actually pretty proud of what I'd been doing, I thought it had been working out remarkably well, especially since I'd tested it out at all kinds of different font sizes, etc...

Anyway, much of this hyphenation talk is abstract, and it is hard to actually SEE with your eyeballs why it is a bad idea. You have to see the logic behind it.

All you have to know is "global denying = bad" + "I should leave it up to the device/user/algorithm" + "I don't want to drive myself into the ground with tons of extra work for no good reason" + "I might be smarter than the hyphenation algorithm in a few specific cases, but the vast majority, I will pick the subpar location".

Similar to the indirect benefits of clean/consistent code over hideous/spaghetti code. Sure, the spaghetti code WORKS, but the problem comes in the future when you want to figure out + maintain the code. It is hard to actually SEE the headaches you are going to cause yourself in the future (or if you go with the clean/maintainable code, it is hard to SEE the time + work you saved, compared to the other route).

Lots of manual intervention "works" if you have a single font size, a single page size, a specific set of margins. But start changing any of those variables, and much of the manual intervention work goes right out the window.

[Psymon]

Originally Posted by Tex2002ans

Your situation, is similar to trying to make the "Word" document look nicer, by adding manual hyphens everywhere. You want to nitpick every single little thing.... but then you change one variable (font size, page size, margins, font), and all those manual hyphens you added will be clogging things up and making things WORSE, compared to if you just left the algorithm alone.

You kinda said something similar before, and I'm left with this feeling that perhaps you don't understand what it was that I was doing (and, indeed, accomplishing). All those variables you mentioned were actually relatively irrelevant, you could change any of them and things still worked out beautifully. Just because I allow hyphenation (if it's necessary) for a specific word doesn't mean that you'll end up with a hyphen in it, when it's not needed (like, if the word is right in the middle of a particular line, not at the end/beginning). And you can change any of those variables you mentioned and everything still came out wonderfully -- the only difference is that instead of having relatively small/short words being hyphenated all over the place, it was only the larger ones that did.

And I did test it out, at all the various font sizes available on the iPad, as well as in both 2-page landscape and 1-page portrait mode -- and please don't yell at me re my using the iPad for testing that out in. By going through all the font sizes, and in particular in 2-page landscape orientation (with its shorter line length), I think that pretty much covered most scenarios one might encounter, regardless of device. If you're imagining bigger margins, well, bumping up the font size pretty much emulates that, for example.

In any case, I have already conceded to give in and just give it up, and not do it the way that I was -- but while I say that, I think you're actually mistaken about the detrimental effect that it has (and also, indeed, mistaken about the positive effect it can potentially have). If you don't believe me, well, just try it on any book of your own, doing what I was doing before -- which, with the appropriate regex S&R, should be a breeze to do.

Like I said, I'm "giving it up" because of all the admonitions that I've gotten to do so -- but I really do have to wonder if, in fact, I'm not making a mistake in doing so. I really can't imagine how changing any of those "variables" you mentioned would end up with disastrous results -- on the contrary, I think what I was doing still would beautifully, regardless of how one change any of those variables.

And that's why I'm wondering, now, if I'm now not making a mistake in giving it up.

[Tex2002ans]

Originally Posted by Psymon

In any case, I have already conceded to give in and just give it up [...]

Well, sometimes I like to beat a dead horse.

Perhaps one of the lurkers, or one of the people who finds this topic in about a year, wasn't convinced yet.

I like to think about all the INDIRECT benefits of making in-depth posts.

Originally Posted by Psymon

Just because I allow hyphenation (if it's necessary) for a specific word doesn't mean that you'll end up with a hyphen in it, when it's not needed (like, if the word is right in the middle of a particular line, not at the end/beginning).

This is exactly what a hyphenation algorithm is already doing. It only adds a hyphen when it thinks it is necessary. (Although admittedly, the specific iBooks algorithm is not to your liking, and it might be a subpar implementation).

This also doesn't get into the insanities of hyphenation.... different languages/countries have different typographical rules/conventions (you probably don't know German or Greek hyphenation rules for example). There are even differences between UK, US, and Austrailian hyphenation.

Edit: For example, this article gives a few examples, "On Hyphenation - Anarchy of Pedantry": http://www.melbpc.org.au/pcupdate/9100/9112article4.htm

"hy-phen-ation" or "hyphena-tion"?
"cryptog-raphy" or "crypto-graphy"?
"ex-actly" or "ex-act-ly" or "exact-ly"?
"ap-pearance" or "appear-ance"?
"di-minish" or "dimin-ish"?

There are people out there who have created giant hyphenation dictionaries to handle many words under the sun. Those that don't have specific rules, then typically they go by some sort of algorithm to split at certain prefixes/suffixes/roots. (For example, when it runs across someone's Last Name, or an obscure word, it needs to figure out how to hyphenate that (if needed)).

Typically, you go dictionary first, algorithm second. (I am making a rough guess, and thinking your typical ereader probably just goes with algorithm only, because it takes up less memory/CPU).

Yes, character count is just one aspect of hyphenation... but it is a heck of a lot more complex. And again, it is closely tied to how good the justification algorithm is at spacing things out (for example, a few spaces here, a few spaces there, might mean the difference between "ex-actly" and "exact-ly". This could reflow the entire thing.)

Other Edit: A lot of this is also a little tough to see, because you have to see the cascade/waterfall effect that happens due to a very minor tweak. Now, while this isn't SPECIFICALLY a "hyphenation" example. You can see what happens in this print book when I make an extremely minor adjustment to the negative indent in the footnote (if I remember correctly, I believe this was 1.2em (or 1.1?) to 1em?).

Page 50 Before/After:



Page 51 Before/After:



As you can see, just that little change on Page 50, two little words "is of", then cascaded, to effect further on in the book. Again, if you need more examples, I can then do a SPECIFIC hyphenation problem. And show what can happen when disabling hyphenation in certain words. It has the potential to completely change the look of a chapter!

And imagine, if I did all of this painstaking manual "fixes". A minor change, could cause me hours of grief in the future. Best to just keep the fixes as "broad" as possible, and stay the heck out of the way.

Originally Posted by Psymon

Like I said, I'm "giving it up" because of all the admonitions that I've gotten to do so -- but I really do have to wonder if, in fact, I'm not making a mistake in doing so. I really can't imagine how changing any of those "variables" you mentioned would end up with disastrous results -- on the contrary, I think what I was doing still would beautifully, regardless of how one change any of those variables.

I will save this post... once you go down your hyphenation-crazy nitpicking path for a few months/years, you will see... you will see!

Originally Posted by Psymon

And that's why I'm wondering, now, if I'm now not making a mistake in giving it up.

Hmmm... were we ever wrong? I mean, we were right about letting fonts do all the fancy footwork! Now go back and read your old topic, you are probably going to think "Who is that crazy guy? Oh wait, his username was Psymon!"

[Psymon]

Originally Posted by Tex2002ans

Well, sometimes I like to beat a dead horse.

Well, I guess you can call me a neigh-sayer, then.

This also doesn't get into the insanities of hyphenation.... different languages/countries have different typographical rules/conventions (you probably don't know German or Greek hyphenation rules for example). There are even differences between UK, US, and Austrailian hyphenation.

Okay, but what you're concerned about here (the language of the text, and whatever rules of hyphenation that that language, or country of origin, might have) are pretty much irrelevant for what I was trying to accomplish. What I was doing had nothing to do with the rules of hyphenation, but rather I was simply trying to minimize the occurrences of hyphenated words to only those places where it was most likely to be necessary, i.e. with larger/longer words.

Apart from that, I could care less if it hy-phenates or hyphen-ates, as long as it hyphenates the word "hyphenates", and doesn't gratuitously also hyphenate shorter words like "shor-ter", simply because it can (and because that word happens to fall at the end of a line).

There are people out there who have created giant hyphenation dictionaries to handle many words under the sun. Those that don't have specific rules, then typically they go by some sort of algorithm to split at certain prefixes/suffixes/roots. (For example, when it runs across someone's Last Name, or an obscure word, it needs to figure out how to hyphenate that (if needed)).

Kewl -- but again, like I've been saying, that's irrelevant (or at least inconsequential) for what I was trying to accomplish.

And all the other things that you mentioned in your previous post(s) are also irrelevant (or, again, at least inconsequential): font size, font face, margin size, page orientation... in my testing I found that there was only improvement of readability (by decreasing the number of hyphenated words), no marked increase in any sort of "disconcerting" white space between words, and overall nicer visual appeal from a typographic design perspective (as exemplified by that Word vs InDesign vs LaTex example you gave before, where one can see how "ugly" the Word example looks with all those gratuitous hyphenated words).

You can see what happens in this print book when I make an extremely minor adjustment to the negative indent in the footnote (if I remember correctly, I believe this was 1.2em (or 1.1?) to 1em?).

Well, as I keep getting reminded about by others, we're not talking about print here, so while I can certainly understand and appreciate the examples you provided (and I've run into similar problems while working with print -- or PDF -- docs, too), I'm not sure how any of that applies, in particular to the subject matter at hand.

I will save this post... once you go down your hyphenation-crazy nitpicking path for a few months/years, you will see... you will see!

I'm saving it, too -- so that if, down the road, the general concensus becomes that I had it right in the first place, then I can actually prove it when I say "Ha ha, told ya so!"

Hmmm... were we ever wrong? I mean, we were right about letting fonts do all the fancy footwork! Now go back and read your old topic, you are probably going to think "Who is that crazy guy? Oh wait, his username was Psymon!"

Well, that's why I gave in and conceded defeat on this so easily, because I've put my faith in you (you all) that you're more experienced than I am, and that I should trust that you know what you're talking about better than I do.

But that doesn't mean that I've thrown out the window the really rather simple and obvious logic behind what I was trying to accomplish, and the fact that in my experience -- including quite a bit of testing, going through my previous books page-by-page (from beginning to end), at different fonts sizes and orientations -- what I was endeavouring to do came out not just successfully, but quite beautifully, quite "elegantly" (as I said in the subject heading for this thread), without any problems at all.

And despite my success, my trust and faith in you apparently has me effectively doubting my own experience, disbelieving what I've seen with my own eyes, and conceding to just up and give it up, and go back and undo what all I've done before.

So yes, save these posts -- because I'm going to, too.

[Psymon]

Originally Posted by Tex2002ans

I will save this post... once you go down your hyphenation-crazy nitpicking path for a few months/years, you will see... you will see!

Hey, Tex, not to promote my book here in the wrong forum, but rather quite in line with what we were discussing in this thread (regarding what I was doing, and suggesting, regarding "selective hyphenation"), although I did take your and others' advice to scrap doing that in the books I'm currently working on, this evening I just uploaded a revised version of the first ebook I ever made (which people here were very helpful with), in which I left that "selective hyphenation" in.

If you'd like to see how that turned out -- and hey, I'm wide open to anyone pointing out to me anything utterly disastrous that's happening as a result (as opposed to something really good, and really nice), I'd be greatly appreciative of it! -- you can snag it here...

https://www.mobileread.com/forums/sho....php?p=2989595

[Notjohn]

There is absolutely nothing wrong with a line of type ending in a single-letter word like "I" or "a". Where in the world did you pick that up? I don't think you found it in the Chicago Manual of Style, which has been the editor's bible for more than a hundred years (108 years, to be exact).

Nor will you (I don't think) find any rule against hyphenating a surname. Indeed, my 14th edition gives a specific example of just such a break: Frederick L. An - / derson (page 207). To be sure, the CMOS recommends against breaking a proper name "if there is any way to avoid it," before going on to show how it ought to be done if necessary.

[Psymon]

Originally Posted by Notjohn

There is absolutely nothing wrong with a line of type ending in a single-letter word like "I" or "a".

What I was trying to avoid wasn't just any instance of "I" or "a", but rather having a line end with a brand new sentence, starting with the word "I" or "A" (capital "A"), i.e. so you don't end up with lines that look like, say, this...

blah blah blah blah blah blah blah. I
blah blah blah blah blah blah blah blah

It's just nicer if that "I" (or "A") wraps over to the next line.

Interestingly, then you wrote...

Nor will you (I don't think) find any rule against hyphenating a surname.

...followed almost right afterwards by...

To be sure, the CMOS recommends against breaking a proper name "if there is any way to avoid it,"

Um, okay.

In any case, with regard to how I did my book (if that's what you're referring to), with all the testing that I did -- in ADE on my computer, and in iBooks on my iPad, in both portrait and two-page landscape orientations, and in all font-size settings -- the effect it had was a great improvement, far less gratuitous hyphenation all over the place, and various other benefits (like the end-of-line "I", etc.).

I'm more than open to hearing about any issues that anyone might find with it, and even open-minded enough to go back and totally scrap what I did completely, and just let the chips (hyphenations) fall where they may -- but the various criticisms that you just made certainly haven't given me a single reason to scrap it.