De-Ligature-ification

[brewt]

Usually I'm the one who can shout in a loud and confident voice "Man, am I ever doing this wrong!" So, here goes. WinXPsp3, calibre uninstalled and reinstalled at 7.1, upgraded yesterday to 7.2.

Attached: Lots of things:

MinionMyriad-source.zip - actual source before calibre 7.2, created in Word 2007, cleaned up in Dreamweaver cs4 and converted to xhtml. Word quickstyles match (close enough, anyway) to the styles.css. Xhtml and css are close to validating - couple lines here and there about colors in hyperlinks and it's there.

Clipboard01.jpg - a picture in firefox showing the top of that source with the ligatures properly in the correct fonts, Minion Pro and MyriadPro-BlackSemiExt.

Basic Set - Brewt Himself.zip - the source zip file created by calibre.

MyriadMinion3_plugin.zip - calibre plugin that embeds fonts in epubs, based on source by Paul Tomashevskyi posted in the sticky "How to Embed Fonts After Calibre" on this forum. Fonts sourced out of C:\Program Files\Calibre2\resources\fonts\MinionMyriad

fontencrypt.zip - font encryption python script written by 'Paul Durrant' posted in the thread "fontencrypt.py - Add Adobe encryption to fonts in ePub" in this forum. I am running it separately; haven't succeeded in making it a plugin yet.

BasicSet.epub - epub created by calibre with embedded fonts embeddeed by above plugin and encrypted by above python script. [Keep ligatures] checked in conversion. Results below are the same before and after encryption.

So. I can see the fonts embedded with ligatures in ADE, minion.
I don't see the ligatures, but it looks like the headline font in ade.

I can see the ligatures in Calibre viewer, minion, but only if I set the default font to minion.
I don't see the ligatures in calibre nor the correct font (myriad) in the headlines, even if I default the sans font to myriad.
No overriding css in viewer.

I see neither the ligatures nor the fonts in B&N/pc.

So. Obviously, I'm doing things, lots of things, wrong because it almost works, but only sometimes. Have at me.

-bjc

[charleski]

As I thought, you've run into encoding issues. 'BasicSet-12.htm' shows no ligatures at all in Firefox, and looking at it with a hex editor shows that your ligatures have been transformed into garbled 3-byte UCS-2 surrogates.

I can only recommend that you follow the advice I gave yesterday and use explicit escape codes rather than trying to paste in characters and hope they remain intact across all those hops.

[brewt]

I'll believe you, but what am I seeing here?

Original-original:

 <p class="MsoBodyTextFirstIndent">st lost / st lost</p>
<p class="MsoBodyTextFirstIndent">ff gruff / î‚® gruî‚®</p>
  <p class="MsoBodyTextFirstIndent">fi finish / î‚« î‚«nish</p>
  <p class="MsoBodyTextFirstIndent">fl flattery / î‚* î‚*attery</p>
  <p class="MsoBodyTextFirstIndent">ft swift / � swi�</p>
  <p class="BodyText4">ffl affluence / î‚° aî‚°uence</p>
  <p class="MsoBodyText">Ligatures in Headline Font: (left plain, right hand
    changed)</p>
  <h1>ff gruff / ff gruff</h1>
  <h1>fi finish / � �nish</h1>
  <h1>fl flattery / fl flattery</h1>
  <h1>ffl affluence / ffl affluence</h1>

Caliber-ified source that rendering is based on:

<p class="MsoBodyTextFirstIndent">st lost / st lost</p>
<p class="MsoBodyTextFirstIndent">ff gruff / î‚® gruî‚®</p>
  <p class="MsoBodyTextFirstIndent">fi finish / î‚« î‚«nish</p>
  <p class="MsoBodyTextFirstIndent">fl flattery / î‚* î‚*attery</p>
  <p class="MsoBodyTextFirstIndent">ft swift / � swi�</p>
  <p class="BodyText4">ffl affluence / î‚° aî‚°uence</p>
  <p class="MsoBodyText">Ligatures in Headline Font: (left plain, right hand
    changed)</p>
  <h1>ff gruff / ff gruff</h1>
  <h1>fi finish / fi finish</h1>
  <h1>fl flattery / fl flattery</h1>
  <h1>ffl affluence / ffl affluence</h1>

-bjc

[JSWolf]

Originally Posted by brewt

My font call in the css has no !Important!'s on it. the font family reads like this:

font-family:"Minion Pro","Dutch801 Rm BT","Times New Roman","Goudy Old Style","Baskerville Old Face","Georgia",serif;

The font being used by Calibre viewer is indeed Times New Roman, despite me having the sony Dutch Roman properly installed on my system; if css rules are being followed, shouldn't they go in order if the first one isn't usable/available?

Clearly (at least to me), Calibre viewer 7.2 is overriding the embedment with it's defaults. And yes, I can set those defaults to Minion and have it work 'better'. I've had minion work out before as embedded (ade picks up the embedded font, even if I remove it from my system fonts directory) as an otf - hain't got a ttf version.

I just got my encrypter off its knees last night; I come up with an intelligible demo epub to post.

Seems odd to me that as font embedding is part of the spec that the implementation of it in the viewers is so spotty. I mis-spake earlier; my B&N PC Viewer doesn't seem to utilize embedded fonts - any body have luck seeing embeddings on a real nook or the viewers?

-bjc

Are these font part or the ePub or are they just installed in your system?

[charleski]

Originally Posted by brewt

st

is an example of the mangled 3-byte code (EFAC86) that has resulted from something in your chain mis-interpreting the 2-byte FB06 code for the st ligature and attempting to convert it. Hard to say if that's Word or Dreamweaver, but Word seems to produce html with the correct escape sequence for a private-use character. It seems calibre has somehow managed to separate this code back into the letters s and t, but can't do the same for the other mangled codes.

Ligatures (like swash caps, text figures and other typographic variants) are not part of the UTF spec* and you can't rely on programs to recognise such font-specific alternative characters. If you want to use them, make sure they're embedded as explicit escape sequences from the start.

*[Edit]Unlike useful stuff like Linear B (which died out around 1100B.C.) and 38 different types of arrow... The lack of UTF codes for text figures is especially annoying.

[brewt]

Ligatures (like swash caps, text figures and other typographic variants)
 are not part of the UTF spec and you can't rely on programs to recognise such font-specific alternative characters. 
If you want to use them, make sure they're embedded as explicit escape sequences from the start.

Ah. Makes sense - finally - thanks. So much for that idea.

	
	
font-family:"Minion Pro","Dutch801 Rm BT","Times New Roman","Goudy Old Style","Baskerville Old Face","Georgia",serif;
Are these font part or the ePub or are they just installed in your system?

That's the ccs I'm using in the epub, which "should" represent a preferred font sequence. I've got some weirdo ideas about compensating for double quotes that the indent numbers only work with certain fonts on, and without embedding, I've got to depend on the idea that someone else has the 'right' font(s) installed on their system.

If I make the call

font-family: serif;

then I am increasing the chance they won't have a font defaulted to on their system that works with my metric design on their system, having the overall look get slaughtered, and run the risk of further complaints and ridicule.

Oh, like I'm not gonna get that anyway for having such a lame idea.

-bjc

[Jellby]

Originally Posted by brewt

BasicSet.epub - epub created by calibre with embedded fonts embeddeed by above plugin and encrypted by above python script. [Keep ligatures] checked in conversion. Results below are the same before and after encryption.

As far as I know, calibre does not support font encryption/obfuscation.

Looking at the differences between your css and the one I use (which works with calibre):

Yours:

@font-face {
  font-style: normal;
  font-family: 'Minion Pro', serif;
  font-weight: normal;
  src: url(MinionPE.otf);
}

Mine:

@font-face {
  font-family: "Carolus FG";
  src: url("../fonts/CAROF___.TTF") format("truetype");
}

You could try something like:

@font-face {
  font-family: 'Minion Pro';
  src: url(MinionPE.otf) format('opentype');
}

and see if that helps.

[Valloric]

Originally Posted by frabjous

But I don't think this is the issue here. Brewt isn't getting any of the fonts in the family to show.

That's another wonderful feature of QtWebKit. I've seen epub books that show the main embedded font variant in Sigil on Windows and Mac just fine, but display with a default font on Linux. I've seen other books that display their embedded fonts + variants on Mac just fine, and fail on Windows and Linux. Also those that work on Linux, but fail on the other two platforms.

Still haven't found any logic to it. The QtWebKit font loading problems are more extensive than just not showing font variants.

This was all with the same version of Sigil and QtWebkit.

So the font issues are directly related to the platform on which QtWebKit is running... Macs seem to have the least problems (but still have quite a few), probably because QtWebKit defers some operations to the system Webkit on Macs (but that's an educated guess).

[WillAdams]

Unicode is set up for scholarship and backwards compatibility.

Variant forms which don't have linguistic significance (and are not needed for backwards compatibility like fi and fl) do not rate and will not get code points (and should not be assigned points in the PUA if one wants the under-lying text to be seen as text)

[brewt]

As far as I know, calibre does not support font encryption/obfuscation.

Really really? 'cause if that's true, the attached shouldn't work.

And tho I didn't take a lot of time on it (quick & dirty, lots of other problems), the ligatures show up just fine in ade and calibre viewer.

Made with Indesign (gak!)

-bjc

[charleski]

Originally Posted by WillAdams

Unicode is set up for scholarship and backwards compatibility.

Luckily, we'll be able to read .txt files from an Ancient Cretian thumb drive.

Variant forms which don't have linguistic significance (and are not needed for backwards compatibility like fi and fl) do not rate and will not get code points (and should not be assigned points in the PUA if one wants the under-lying text to be seen as text)

And this is a failure on the part of the Unicode consortium, simple as that.

Ligatures certainly can have linguistic significance, just look at the ampersand sign, and Unicode honours this for many other languages. I wouldn't be so bothered if if weren't for the fact that the Unicode range contains such a large amount of rubbish.

[Jellby]

Originally Posted by brewt

Really really? 'cause if that's true, the attached shouldn't work.

OK, it works, so calibre supports obfuscation, that's good.

But what is your problem then? Is it only that you don't get it to work when you generate the ePUB with calibre?

Note that your file BasicSet.epub does not pass epubcheck.

[brewt]

Originally Posted by Jellby

But what is your problem then? Is it only that you don't get it to work when you generate the ePUB with calibre?

Note that your file BasicSet.epub does not pass epubcheck.

Maybe I don't know how to do it, but none of my creations out of calibre can pass epubcheck, even if I go to the trouble to fix the last bits of the xhtml to completely validate (along with valid css) and nothing fancy happens in the epub. Same problem with indesign output. And I'm still not finding a deep benefit to validation, given all that I want to do.

And my problem? Where do I start? Lazy, dull, dimwitted, lacking in understanding, frustrated, tired, poor, old, etc., etc. None of which the good folk here can do anything about.

But I do appreciate your considerations, even if I am left with a generalized sense of a lack of direction. Every method leaves large numbers of things left out, making no choice the long term right choice. Ah well, Damn the decisions, full steam ahead. Guess I just want too much to be easier. Or maybe I just want to do more than is allowed, given the current state of he technologies.

No hard feelings?

-bjc

[WillAdams]

charleski, more importantly, scholars are able to unambiguously set Cretan scripts and discuss them unambiguously.

If one wants ligatures it's simple enough to set a run of text and turn them on --- no need for unique code points which aren't practical in any case --- there are simply too many possibilities. I had to do quite a bit of fiddling to manage to encode just the options in Zapfino:

http://www.tug.org/TUGboat/Articles/.../tb77adams.pdf

Assigning codepoints for ligatures would be nightmarish and would quickly exhaust them.

[Jellby]

Originally Posted by brewt

No hard feelings?

None at all.

But at this moment I think if would be beneficial to re-state what you want to attain. At first I thought you just wanted an ePUB with embedded fonts and ligatures that works, but you have already shown a sample that works in ADE and calibre viewer... So, what else do you want? Do you want to understand how to do it by hand? Do you want a workflow to get it from your source files?

Also, when creating sample documents, try to make them simple. To test ligatures and embedded fonts, often a single font is enough, and including several families of different fonts only make the file larger and more cumbersome to analyse.

As a general comment on ligatures I think the "right way" would be to have something like an OTF flag or feature that can be enabled/disabled, either through CSS or through software preferences, the same holds for old-style numbers and other features. Prince works along this lines.