Flattening unicode with unidecode() in a plugin

[Phssthpok]

I want to have authors, titles etc. flattened to ASCII, and am trying to use unidecode() to do it. In IDLE, I can say:

from unidecode import unidecode
print(unidecode('Philip José Farmer'))

and the output is "Philip Jose Farmer" (the accented é has been replaced by an unaccented e).

I downloaded the unidecode module and inserted it into my plugin, and at first I got this:

SyntaxError: (unicode error) 'utf8' codec can't decode byte 0xe9 in position 0: unexpected end of data

I re-saved the source code as a UTF-8 file, but then after that exactly the same code as above gives me "Philip Jos Farmer" (the accented é is deleted, rather than being replaced by e).

The only other notable difference is that my IDLE is using Python 3.7, whereas Calibre is using 2.7.

Can anyone tell me what's going on here? And is there anything already in Calibre to do the flattening, rather than having to include the whole unidecode module in my plugin?

[jackie_w]

Originally Posted by Phssthpok

And is there anything already in Calibre to do the flattening, rather than having to include the whole unidecode module in my plugin?

Off the top of my head I can't remember exactly what each of the following does, but if you download a copy of the calibre source code these functions may be helpful:

from calibre.utils.filenames import ascii_text
from calibre import sanitize_file_name_unicode
from calibre.ebooks.oeb.polish.check.parsing import make_filename_safe

I've used all of them at some point in various calibre plugins.

[Hopkins]

Python 2.7 has issues with unicode. This may not help, but try adding the following line to the top of your python code before any imports:

from __future__ import (unicode_literals, division, absolute_import, print_function)

Note: You may only need the unicode_literals part.

[Phssthpok]

Originally Posted by jackie_w

Off the top of my head I can't remember exactly what each of the following does, but if you download a copy of the calibre source code these functions may be helpful:

from calibre.utils.filenames import ascii_text
from calibre import sanitize_file_name_unicode
from calibre.ebooks.oeb.polish.check.parsing import make_filename_safe

I've used all of them at some point in various calibre plugins.

Many thanks, I'll look at those.

Meanwhile a bit of pottering around in unidecode() revealed the problem -- it uses __import__() to dynamically load a conversion module for each 256-char code page, and I just needed to change the module name to "calibre_plugins.foo.unidecode.X" instead of "unidecode.X". But I'd prefer to use something which is already present in Calibre if possible, so I'll check out your suggestions.