Character Encoding Question

[mikapanja]

@ kovid

I have a book encoded in ISO 8859-1. When I open it in Sigil, it shows hieroglyphs all over, but both in Calibre viewer and editor it looks just fine.

1) Is Calibre ironing out on-the-fly, with UTF-8 as a default solution?

2) Would it be possible to instruct Check Book to warn about any other encoding than UTF-8 present in the book?

[kovidgoyal]

calibre detects encodings declared in HTML. I have no idea what Sigil does, from your description I'd guess it assumes UTF-8 always.

As for checking encodings, IIRC the editor autoconverts to UTF-8 whenever it processes any HTML, so I dont think checking will be possible or even necessary.

[mikapanja]

Originally Posted by kovidgoyal

calibre detects encodings declared in HTML. I have no idea what Sigil does, from your description I'd guess it assumes UTF-8 always.

As for checking encodings, IIRC the editor autoconverts to UTF-8 whenever it processes any HTML, so I dont think checking will be possible or even necessary.

Well, the Calibre editor may autoconvert to UTF-8 on opening, but the header retains the old encoding:

<meta content="application/xhtml+xml; charset=iso-8859-1" http-equiv="content-type"/>

and I think the e-book readers that do not autoconvert may display text with different strange characters (â, Ã, Â, ¦, etc. instead of “, ”, etc.), as Sigil does.

I know I can use Modify ePub add-on to encode in UTF-8, just would like to know that I need to. The way it is, I must use Sigil to open the book and find out if the encoding is off.

I guess my question is: would it be possible for Check Book to report "There is a non-UTF-8 encoding declared in xyz.html"? That would be really helpful.

[kovidgoyal]

The editor does not do anything on opening. Run any automated tool, such as fix html to make it happen.

[mikapanja]

Originally Posted by kovidgoyal

calibre detects encodings declared in HTML... The editor does not do anything on opening.

The book in question has iso-8859-1 declared throughout the document (no encoding declared in CSS). If the editor detects declared encodings and doesn't autoconvert on opening, how is it possible that all the characters are displayed correctly, as if it used UTF-8?

I'd really like to get to the bottom of this...

P.S. I can send you the book via PM, if it helps.

[kovidgoyal]

The editor detects the encoding, it simply does not make any *changes* to the file until you run a tool.

[kovidgoyal]

Here you go: https://github.com/kovidgoyal/calibr...7aa50e656090ce

[mikapanja]

Originally Posted by kovidgoyal

Here you go: https://github.com/kovidgoyal/calibr...7aa50e656090ce

Thank you so much, this is great

[mikapanja]

This may be irrelevant, but I thought I should report it anyway.

Installed Calibre 1.42. Ran a Check Book on a title with iso-8859-1 encoding. Got the message "This file has its encoding declared as %s". If you feel like it, perhaps you could improve the error report so that it matches the exact encoding. Just a thought...

Also, it would be helpful to add the option to change the encoding throughout the book, not just the particular HTML file.

[eschwartz]

Originally Posted by mikapanja

This may be irrelevant, but I thought I should report it anyway.

Installed Calibre 1.42. Ran a Check Book on a title with iso-8859-1 encoding. Got the message "This file has its encoding declared as %s". If you feel like it, perhaps you could improve the error report so that it matches the exact encoding. Just a thought...

Also, it would be helpful to add the option to change the encoding throughout the book, not just the particular HTML file.

Missing the string formatter (assumed typo).

Anyway, it got fixed: https://github.com/kovidgoyal/calibr...978c08888b9022

[mikapanja]

Originally Posted by eschwartz

Missing the string formatter (assumed typo).

Anyway, it got fixed: https://github.com/kovidgoyal/calibr...978c08888b9022

Thank you, both Kovid and eschwartz

[eschwartz]

Originally Posted by mikapanja

Thank you, both Kovid and eschwartz

All I did was announce it.