Can't match Unicode character

[atordo]

In the good old days when men were men and characters were single bytes, humble ASCII had just two quotes: single (') and double ("). Now we have opening and closing, single and double, low and up, with and without angle, normal and heavy, reversed or not, and their respective combinations. Some of them make recipes fail with the following error:

UnicodeEncodeError: 'charmap' codec can't encode character u'\u201c' in position 30: character maps to <undefined>

The solution seems easy, doesn't it? Just replace the char in question with some HTML entity. So blatantly plagiarizing Danas recipe by Darko Miletic (I must thank Darko and other recipe authors, because I do this all the time), I add the following in preprocess_regexps:

,(re.compile(u'\u201c'), lambda match: '“') # left double quotation mark

Among others because there are several chars which produce the error, but one is enough as example. Problem is the character is never matched. I've tried other forms like '\xe2\x80\x9c' (the three bytes which compose the char in UTF-8), pasting the char directly in the recipe, etc, to no avail. They pass untouched, as can be verified by inspecting the HTML files when running ebook-convert in the command line.

I attach my failed recipe (as a zip to preserve UTF-8) in the hope that somebody can solve it. TIA.

[kovidgoyal]

Try setting encoding to different values in your recipe. utf8, latin1, cp1252, cp1251 are popular.

[atordo]

Thanks for the suggestion, but the page really uses UTF-8. Setting the encoding to other values just adds garbage chars in the text.

I'm afraid this may require more complex solutions. I'll have a look at builtin recipes for more inspiration and report back when/if I make any progress.