View Single Post
Old 06-06-2016, 05:33 AM   #23
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,477
Karma: 8025702
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
The two of you did indeed find the root of the problem. It is apparently caused by a bug in python related to locales, or at least how they are used by calibre.

Explanation: columns built from other columns (composite columns) store text, and only text. If a template does any arithmetic then the results are stored as strings representing the number using period as the decimal separator. You can use the format_number function to change the separators to match the locale. When you sort a composite column numerically, calibre must (attempt to) convert the text to a number. This is what is failing.

The bug: windows (at least) provides the locale's separator characters as a 256-bit ascii character instead of unicode. When I switch my locale to France, the thousands separator is reported as a non-breaking space character (0xA0). Unfortunately that character "looks" like it might be unicode but isn't. Unfortunately calibre treats it as unicode, with the result that any attempt to use that character throws an exception. The text-to-number conversion fails because of that exception, which means that the sort fails.

Note that numeric custom columns (float and integer) store the actual number, not the text version of it. This is why the problem doesn't arise with sorting those columns.

I am submitting a calibre change to Kovid to fix this problem. In the meantime (and assuming he accepts the fix), the only workarounds that I see are:
  • Use a locale that does not use the non-breaking space as the thousands separator.
  • Change the settings for your locale so that a simple space is used for the thousands separator instead of the non-breaking space. This could have side effects relating to numbers breaking across lines.
  • Change the thousands separator to some other character such as a dot.
If and when the fix appears you can reset the locale to your default.
chaley is offline   Reply With Quote