View Full Version : What is an illegal multibyte sequence ?


mr ploppy
05-17-2012, 02:13 PM
Trying to turn this:

http://jakonrath.blogspot.co.uk/2012/05/guest-post-by-tom-schreck.html

into an epub with this:

http://web2fb2.net/

but it just says this:

ERROR: 'big5' codec can't decode bytes in position 37378-37379: illegal multibyte sequence

frostschutz
05-17-2012, 02:21 PM
It would be a badly encoded file, except it's trying to use the wrong codec in the first place.

So it's a problem of web2fb2.net

EDIT:

The character in question is in the comment section though ("Yeni �kan"), at least that's where iconv fails when trying to translate from utf8 to latin1 or big5. If you don't want comments in your epub you should probably go for a more selective approach...

pholy
05-17-2012, 03:01 PM
The web site appears to be using UTF-8 encoding, so there are rules about what values can occur in each position of a multi-byte sequence. This is so that you can always find the start of a multi-byte sequence even if you get plopped down in the middle of a file.
According to Table 3-6, in Section 3.9 of the Unicode Book (available from www.unicode.org as a set of pdf files) the first (and only) byte of a single byte code must start with a zero bit -ie 0xxxxxxx. For a two-byte code, the first byte is 110yyyyy and the second is 10xxxxxx. For a three byte code, the first byte is 1110zzzz, the second is 10yyyyyy, and the third is 10xxxxxx. The 16 bit code value is form by concatenating the zzzz as high order, then the yy's and finally the xx's with leading zeroes of course. The chart in the book shows it better :)
You might need a hex editor to find that byte pair, because most text editors deal in lines; but a good unicode editor should point out the problem when it opens the file. Then you can fix it howeveer you choose.
I'm not sure why it is using a 'big5' codec; that's usually used for chinese texts.

edit:
Dang! Beaten to the punch. So much for the long-winded (and still incomplete) explanation.