Trying to use Textile processing

[ldolse]

Originally Posted by user_none

It's already supposed to...

Bug then? I did a bit of googling and it seems like Python has some quirks in it's handling of BOMs across different unicode encodings. I Didn't look very hard though.

[getajob]

Originally Posted by ldolse

You're mostly correct except for one part of your solution:

UTF-8 is unicode...

Fair point. Perhaps I should have said:

If you have to use Unicode as your 'Save as' option in WordPad, then set Input character encoding to UTF-16

It was not intuitive to me that if I select the 'Unicode' option in the Save As dialog of WordPad then that meant that I needed to set the Input character encoding to UTF-16. UTF-16 doesn't even appear on the dropdown as an option...

so there is no need to use UTF-16 ever.

Probably true.

Regarding both 'ANSI' and 'Unicode', I was actually referring to labels that Microsoft has chosen to use with both WordPad and Notepad, nothing more... I was trying to assist anyone that might fall into the trap that I just fell into.

Thanks everyone who helped me - I learnt a lot.

[user_none]

Originally Posted by ldolse

Bug then? I did a bit of googling and it seems like Python has some quirks in it's handling of BOMs across different unicode encodings. I Didn't look very hard though.

Then it's in chardet which TXT input uses for detecting the encoding. I haven't had time but I need to look at the input file to see if it actually has a BOM. chardet will only detect UTF-16 if it has a valid BE or LE BOM.

[getajob]

Originally Posted by getajob

WordPad has four 'Save As' options: ANSI, Unicode, Unicode big endian and UTF-8. Notepad is the same.

I have no explanation for this but yesterday WordPad had an Encoding dropdown option in the Save As exactly the same as Notepad - but not today.

Notepad still allows me to Save As with Encoding options of ANSI, Unicode, Unicode big endian and UTF-8 but WordPad will no longer co-operate at all.

Damn MS rubbish!

I have switched to Notepad++ - it has much better encoding control.