Apostrophes in Full text search

[albell]

Still "playing" with this powerful new feature, I ran into a minor problem that I fear insoluble but I will try anyway

As an Italian-speaking user (but also in French there is this problem) I find myself looking for terms that in my books can be preceded by an article or a preposition followed by an apostrophe. For example: "dall'alto monte" (from the high mountain); "un'ottima cena" (a great dinner) and so on.

Now, searching only for words without an article ("alto monte" / "ottima cena") I expected to find those preceded by an article as well. Instead, the books in which these forms are present do not appear among the results. In practice, it seems that the forms preceded by article and apostrophe ("dall'alto", "un'ottima") are considered as unique terms.

I'm afraid it has to do with the apostrophe not being interpreted by the engine as a limit of a word, or what else?

Is there a way to bypass this problem or is it structural?

Many thanks!

[kovidgoyal]

There's no way to bypass it. Tokenization of text into words id done at indexing time, and once done its done. calibre uses the ICU library to do this tokenization and that uses language sensitive rules, for a number of languages including european ones.