Full-text search can't handle single-quote mark in a phrase (e.g. golfer's elbow)

[droopy]

why can't calibre full-text search for "golfer's elbow" (quotes in search)? I have books with that phrase. Does Calibre full-text search have a hard time with single quote marks in the search?

[kovidgoyal]

full text search is based on words, it doesnt support punctuation of any kind, and that limitataion comes from the sqlite fts engine.

[droopy]

hi kovid,
thanks for your reply.
so how do you propose full-text-searching for the phrase [golfer's elbow] ?

[DNSB]

Why not enclose the search in "s ( i.e. "golfer's elbow" ). That works for me when searching for year's best for example. See attached image.

[droopy]

Originally Posted by kovidgoyal

full text search is based on words, it doesnt support punctuation of any kind, and that limitataion comes from the sqlite fts engine.

Hi Kovid,
Actually, I learned that full-text search, at least in phrases, does support punctuation. But I'm getting ahead of myself. Let's rewind.

FT Search couldn't find any books with
"golfer's elbow"


But it found eight books when it was a curly (a.k.a "smart) apostrophe, as in:
"golfer’s elbow"

[droopy]

Originally Posted by DNSB

Why not enclose the search in "s ( i.e. "golfer's elbow" ). That works for me when searching for year's best for example. See attached image.

Hi DNSB, sorry for not being clear. I did enclose my search in quote marks like you did.

[DNSB]

Originally Posted by droopy

Hi Kovid,
Actually, I learned that full-text search, at least in phrases, does support punctuation. But I'm getting ahead of myself. Let's rewind.

FT Search couldn't find any books with
"golfer's elbow"


But it found eight books when it was a curly (a.k.a "smart) apostrophe, as in:
"golfer’s elbow"

I'm not certain if this would help but one of our devs at work was using tokenchars with the tokenize keyword to add special characters for FTS5 (think it was 5) searches. You might want to check the SQLite FTS5 extension page for more information.

OTOH, I learned some new words from him while he was working on that project.

[kovidgoyal]

Originally Posted by droopy

Hi Kovid,
Actually, I learned that full-text search, at least in phrases, does support punctuation. But I'm getting ahead of myself. Let's rewind.

FT Search couldn't find any books with
"golfer's elbow"


But it found eight books when it was a curly (a.k.a "smart) apostrophe, as in:
"golfer’s elbow"

Did you turn on the option to match on related words?

[droopy]

Originally Posted by kovidgoyal

Did you turn on the option to match on related words?

No, Kovid, I did not turn it on. Please refer to screenshot of the FTS window

[Comfy.n]

Originally Posted by droopy

No, Kovid, I did not turn it on. Please refer to screenshot of the FTS window

I didn't know about this typographic difference, thanks.

For what it's worth, I see a different number of results when full-text searching with each apostrophe's variant:





Could it be the case that you only have books containing results with the smart apostrophe?

Given the sqlite FTS limitation, I often find myself using the Quality Check Plugin Search ePubs tool, as it can run regex searches. For instance:

(?i)\, though\. gives me results containing the segment ", though."

Sqlite FTS can't do that, unfortunately.

[droopy]

Originally Posted by Comfy.n

Could it be the case that you only have books containing results with the smart apostrophe?

This is what I am concluding, Comfy.
Thank you for sharing your thoughts. It seems that Calibre's full-text search is particular when it comes to apostrophes. We must do both smart/curly quotes and "dumb" quotes to get all search hits.

[Comfy.n]

Originally Posted by droopy

This is what I am concluding, Comfy.
Thank you for sharing your thoughts. It seems that Calibre's full-text search is particular when it comes to apostrophes. We must do both smart/curly quotes and "dumb" quotes to get all search hits.

this would need testing, but I believe that the tool I mentioned can do something like...

golfer’s elbow OR golfer's elbow

maybe:

golfer['’]s elbow

[kovidgoyal]

Originally Posted by droopy

No, Kovid, I did not turn it on. Please refer to screenshot of the FTS window

Turn it on and try.

[Quoth]

Originally Posted by droopy

This is what I am concluding, Comfy.
Thank you for sharing your thoughts. It seems that Calibre's full-text search is particular when it comes to apostrophes. We must do both smart/curly quotes and "dumb" quotes to get all search hits.

They are different charaters and there is no such thing as a smart quote. Wordprocessors have “smart quoting”, where
' -> ‘ or ’ and is sometimes wrong, messing up ’tis or ’90s
" -> “ or ”
<< ->«
>> -> »
It’s using very simple context and lookup table.

' Typewriter single quote. Something like it can be used in transliterated words or dictionaries. Otherwise only for strings in programming.
‘ open quote
’ close quote or apostrophe
′ single prime (feet, minutes)

" Typewriter double quote. Only for Strings in programming.
“ Open double quote
” Close double quote
(these will be different in German etc)
″ double prime (inches, seconds)

The ` left of 1 is not a quote, it's diacritic and your keyboard map should have it as dead for à è ì ò ù ỳ

Unfortunately people creating or editing metadata can mess up.

Typewriters omitted ‘ ’ “ ” { } [ ] 1 0 (used I and O) and often ! and currency symbols using l backspace . and S, L, Y etc with l, -, = etc.

[Comfy.n]

Out of curiosity, I tested it this way:

FTS for "year's best" (typewriter's single quote), marked all books, tagged them with year['’]s best

FTS for "year’s best" (apostrophe), marked all books, tagged them with year['’]s best

Then I selected all tagged books and ran these queries with the Search Epubs tool:

year[']s best returned 10 results
year[’]s best returned 68 results

Final step: check if searching for year['’]s best would return the same number, and it found exactly 78 results.

I was under the impression that some books were using both forms in the same edition