Sigil, UTF-8 and the emdash

[crutledge]

I have just run across a strange occurance with Sigil. I loaded a HTML file to Sigil and noticed all of the emdashs (—) had disappeared. The original file had 250 occurances of the emdash.

I went back to the original HTML file and changed the character set from Windows 1252: Western European to UTF-8 (which Sigil uses) and all of the emdashes disappeared. I then went back to Windows 1252: Western European and replaced all (—) with amp#8212; , converted back to UTF-8 and all emdashes re-appeared. I then loaded to Sigil and all emdashes were present. This appears to be a UTF-8 problem.

As a pre-process, all HTML files will have to be edited prior to loading to Sigil unless someone has come up with a work-around. Are there any other characters to watch for?
Curiouser and curiouser!!

[pholy]

Are there any other characters to watch for?

I would guess that any character that had a numeric value in W-1252 different from its UTF-8 value. Curly quotes, maybe?

According to the epub spec, you should only be using UTF-8 for your text input. If I ran the Sigil world, Sigil wouldn't even try to edit your file if it declared a different encoding.

So yes, you need something that will translate from W-1252 to UTF-8.

[capidamonte]

I use a virtual machine of Win2K for my favorite editor -- as part of the process of creating the text files I run everything through iconv for Windows.

Makes it easy to make batch files and macros that do the conversion to UTF-8. You have to be comfortable with the command line, but it's super-convenient.

[theducks]

Originally Posted by capidamonte

I use a virtual machine of Win2K for my favorite editor -- as part of the process of creating the text files I run everything through iconv for Windows.

Makes it easy to make batch files and macros that do the conversion to UTF-8. You have to be comfortable with the command line, but it's super-convenient.

Notepad++ has plugins that convert a whole lot of text
things from 1 world to another: *nix<->Mac<->Win

Even if Sigil can't (won't) fix dual encoding things, I would like it to Pop up a Stop message, rather than silently throwing characters away or trashing them completely.

[pietvo]

I tried to load a HTML file in Sigil, coded in Windows-1252 with an em dash in it, and there were no problems. The emdash showed as such in Sigil. I checked with a hex editor that the em dash was really coded in windows-1252 (hex 97). I saved it to epub, and Sigil had converted the whole thing to utf-8. The em dash shows perfect in ADE. This is on Mac OS X, so that may be a difference. However, I would suggest to check thoroughly if your HTML file is correct (for example does it really use the Windows-1252 code for the me dash, and does it not state two conflicting encodings). But I also advice, as others have posted above, to use utf-8 for all your files. It is a much better encoding.

[Valloric]

Originally Posted by crutledge

I went back to the original HTML file and changed the character set from Windows 1252: Western European to UTF-8 (which Sigil uses) and all of the emdashes disappeared. I then went back to Windows 1252: Western European and replaced all (—) with amp#8212; , converted back to UTF-8 and all emdashes re-appeared. I then loaded to Sigil and all emdashes were present. This appears to be a UTF-8 problem.

Your file probably did one of two things:

1. It didn't state an encoding; if no encoding is specified in the file, UTF-8 is assumed. You need to specify an encoding in the file. Without it, your playing russian roulette every time you open it in any application.
2. It stated an incorrect encoding. Stating two different encodings also falls under this category.

Originally Posted by pholy

If I ran the Sigil world, Sigil wouldn't even try to edit your file if it declared a different encoding.

Frankly, then it's better that you don't. Sigil automatically converts all files from several dozen different encodings into UTF-16 (and then into UTF-8 on export) as long as the file states the original encoding.

Just saying "sorry, I can't open this" would be silly.

Originally Posted by pietvo

I tried to load a HTML file in Sigil, coded in Windows-1252 with an em dash in it, and there were no problems. The emdash showed as such in Sigil. I checked with a hex editor that the em dash was really coded in windows-1252 (hex 97). I saved it to epub, and Sigil had converted the whole thing to utf-8. The em dash shows perfect in ADE. This is on Mac OS X, so that may be a difference. However, I would suggest to check thoroughly if your HTML file is correct (for example does it really use the Windows-1252 code for the me dash, and does it not state two conflicting encodings). But I also advice, as others have posted above, to use utf-8 for all your files. It is a much better encoding.

Exactly.

BTW it works the same on all platforms.

While I also suggest the use of Unicode encodings whenever possible, users are completely free to use any encoding they wish for their input files as long as the files state the encoding in use. Without that, it's anyone's guess.