Sorting with accented characters

[chaley]

Mixx has suggested in this post that calibre should sort accented characters as equivalent to their non-accented ASCII version.

What the suggestion means is that e, è, and é would be sorted as if they are exactly the same letter. There is no guarantee that one would come in front of the other. The same is true for c and ç, s and ß, A and Å, etc.

I have no idea what it means for non-latin characters such as Greek, Cyrillic or Chinese, but my guess is that they would sort using the letters that are used when creating file system names. It should (at least) be consistent.

Do you, the calibre users, want this?

[Manichean]

I'd like to elaborate beyond just voting: Personally, I don't care either way, as I most often use search to find the book(s) I want. But, as I see it, this would make sorting more consistent for international users (for example I personally feel that "ö" should follow "o", or, at least be sorted in the same area), and that, in my eyes would be a Good Thing. One modifier applies, though: If it's worth the trouble for you developers. If this means you have to rip apart everything and rewrite most of it, my vote would change to no.
Another comment regarding sorting that goes slightly off-topic: Personally, I'd much more like to see an option to include non- english articles in the list that Calibre ignores when sorting, like the german "der, die, das". Optimally, this would be implemented as a tweak, pre-filled with the most common articles, say english, french, spanish and german. For this, the same modifier as above applies.

[Man Eating Duck]

Originally Posted by chaley

[...] A and Å, etc. [...]

I'm all for new features, but please make this one optional, or at least with selectable collation algorithms as you can do in mySQL.

In Norwegian, common collation of oø and aå is wildly incorrect, in fact 'a' and 'å' resides respectively at the beginning and end of our alfabet

For names you might also consider language-specific collation. For instance "van Eeden, Frederik Willem" should be collated under E.

I presume you would use some sort of library for this?

[chaley]

Originally Posted by Man Eating Duck

I'm all for new features, but please make this one optional, or at least with selectable collation algorithms as you can do in mySQL.

In Norwegian, common collation of oø and aå is wildly incorrect, in fact 'a' and 'å' resides respectively at the beginning and end of our alfabet

For names you might also consider language-specific collation. For instance "van Eeden, Frederik Willem" should be collated under E.

It won't be optional. It is, or it isn't. Otherwise the development isn't worth the trouble.

I presume you would use a library for this, which one?

No library. Uses unicode codepoint groupings, the same ones that are currently used for cleaning file names.

I have looked at using unicode sorting libraries. The amount of work to do so is enormous, and I have no desire to do it.

Edit: for names, that is under your control. Enter the sort string using the 'manage authors' function available by right-clicking on any author in the left-hand tag pane.

[Man Eating Duck]

Originally Posted by chaley

It won't be optional. It is, or it isn't. Otherwise the development isn't worth the trouble.

Ok, in that case I'll vote against. This might seem negative, but at least for Scandinavian users it is just plain wrong

[chaley]

Originally Posted by Man Eating Duck

This might seem negative, but at least for Scandinavian users it is just plain wrong

Not in the least negative. I am not at all sure about changing from one (well-understood) wrong thing to some other (not understood) wrong thing.

[chaley]

Originally Posted by Manichean

Another comment regarding sorting that goes slightly off-topic: Personally, I'd much more like to see an option to include non- english articles in the list that Calibre ignores when sorting, like the german "der, die, das". Optimally, this would be implemented as a tweak, pre-filled with the most common articles, say english, french, spanish and german. For this, the same modifier as above applies.

Tweak will be in the next release.

It is 'pre-filled' as today, with A, An, The. You can add what you want, or turn it off completely.

[Manichean]

Originally Posted by chaley

Tweak will be in the next release.

It is 'pre-filled' as today, with A, An, The. You can add what you want, or turn it off completely.

Wow, great, thank you!

[Scott Nielsen]

Originally Posted by Man Eating Duck

Ok, in that case I'll vote against. This might seem negative, but at least for Scandinavian users it is just plain wrong

Agree - it's just plain wrong. Another vote against, here.

[Coleccionista]

The poll is closed but my vote should have been a resounding yes for the very minimum, common sense implementation:

a = á = à
e = é = è
i = í = ì
o = ò = ó
u = ú =ù

I cannot find any reason why the current sorting:

Carlos Fuentes
Cornelia Funke
Alan Furst
Francis Fèvre

should be prefered to the quite more sensible:

Francis Fèvre
Carlos Fuentes
Cornelia Funke
Alan Furst

I understand that there may be some difficulties with other characters and languages but for these vowels it looks a sensible change

[itimpi]

Note that you can already control the sorting of Authors (where I expect it is most important) by setting the Author-sort field to only use the non-accented versions of characters.

[chaley]

Originally Posted by Coleccionista

The poll is closed but my vote should have been a resounding yes for the very minimum, common sense implementation:
...
I understand that there may be some difficulties with other characters and languages but for these vowels it looks a sensible change

This is a non-issue for me, but it seems that you could be the needed 'someone who cares'. I invite you to do the work and submit it. I am sure that Kovid would entertain such a patch.

[Coleccionista]

Originally Posted by chaley

This is a non-issue for me, but it seems that you could be the needed 'someone who cares'. I invite you to do the work and submit it. I am sure that Kovid would entertain such a patch.

Given that I don't have a clue of calibre code and python in general I feel free to make some bold assumptions.

If the grid where calibre is displaying the data is actually the result of a database query (some view query) whe should be able to skip any calibre change and use database functions for collation

Would the following be applicable?

How to sort text in sqlite3 with specified locale?

So in the end we could use the calibre locale to build database queries like:

SELECT * FROM authors ORDER BY name COLLATE POLISH;

[theducks]

Originally Posted by Coleccionista

So in the end we could use the calibre locale to build database queries like:

SELECT * FROM authors ORDER BY name COLLATE POLISH;

So we have a Multi-Lingual Library, What sort is the "Correct" one to use?

Your example works beautiful as long as the data is mono-lingual

Kovid and crew have no easy task making Calibre work across many regional differences
crew

[Coleccionista]

The idea I tried to convey was that you have all locale queries created:

Example:

SELECT * FROM authors ORDER BY name COLLATE POLISH;
SELECT * FROM authors ORDER BY name COLLATE SPANISH;
SELECT * FROM authors ORDER BY name COLLATE FRENCH;
.............

When you install calibre you select a language so from them on the queries launched against the database would use the language collation that you are using in the interface. It seems quite logic.

To summarize, is this SQLite/database-based approach possible?

calibre_lang = German // Or English, or korean, or whatever
db_query = SELEC foo, bar, baz ... whatever COLLATE $calibre_lang