Search function not working

[Ruskie_it]

Not sure if I should post here, because I think this is more a thing of the EBOOK, not the editor, however:

I have one book (epub) for which the Search function does not seem to work.
That is, if I try to search for, say, a word, I get "no matches found", even when I have that word right in front of me on the page!
And yes, I am sure I have placed the cursor BEFORE that word with search direction running "down".

I think it's something related to the book however because search works with other ebooks. Additionally, I have tried in Sigil as well, with the same result: no matches found for any word, while it works with others.

So I was wondering: what could cause this problem in an ebook?

Thanks

[Ruskie_it]

Uh-oh, never mind, I think I got it: it has to do with the HypenateThis! plugin. I'll post my question there

[eschwartz]

If you are using the hyphenation plugin, that corresponds to what I would have guessed the problem is... you are searching for

foobar

where the actual text is (unicode hyphens depicted as asterisks)

foo*bar*with*randomly*interspersed*hyphens

And you need to actually match the hyphens as well.

IIRC, calibre's Edit Book should show invisible characters as yellow spaces.

[Ruskie_it]

Yes and no, because... that's what I guessed, actually, and what led me to try and make a test with the un-hypenated ebook.
But, when I open it in Sigil and in Calibre, and I mean html view, the text does not show like that: I can see the words as normal - which also made me wonder how hypenation was rendered; not knowing that much I supposed it was achieved in some way using css.

[eschwartz]

Yeah, well I just tested it. The invisible unicode entity is invisible in Edit Book. I guess only visible characters get yellow highlights (  for example) and invisible ones are not shown at all. I guess that makes sense as well. I didn't bother checking in Sigil because I already know for sure it would be hidden.

You can tell by using the arrow keys to move through the text, paying particular attention to misspelled words (since hyphens where they don't belong are correctly marked as misspelled) and Edit Book will show in the bottom right corner which character is to the right of the cursor.

tl;dr your regex needs to check for U+00AD interspersed at total random.

[Ruskie_it]

Yeah, I guess it makes sense, too, at least with hypenation symbols.
But it would be even better if one could choose to hide or show them.

I don't get your last sentence, anyway it means to me that soft hypenation kinda breaks search functions, isn't it? This, unless your mention to regex expressions meant that it is possible to account for hypenation signs, however even in that case it would mean you're down to only use regex which is not cool at all if you are not used to work with them.
I understand this is not a fault of the editor but still... it's a shame

[eschwartz]

The search function in a plaintext editor shouldn't and won't strip non-displaying characters.

A simple regex like

f@?o@?o@?b@?a@?r

would find the word "foobar" with optional soft hyphens. Note: the at symbols are a placeholder for the aforementioned unicode ­ character.

You will indeed need a regex, because regexes are by definition what you use to find a fragment of text that can appear in multiple forms -- which is the case here.



On the other hand, I would venture to say, the best solution is... don't hyphenate the book until it reaches production and goes on your ereader. Remove all ­ symbols from the book (de-hyphenate it) and searching in ebook-editors will work properly.

Then hyphenate them when they are ready to be pushed to your device. The ereader/app should handle searching just fine, much like the way they don't break on span tags.

[Ruskie_it]

Originally Posted by eschwartz

You will indeed need a regex, because regexes are by definition what you use to find a fragment of text that can appear in multiple forms -- which is the case here.

And yet... wouldn't it be so cool if, since the editor is smart enough not to show "invisible" characters, was able to search ignoring them?

On the other hand, I would venture to say, the best solution is... don't hyphenate the book until it reaches production and goes on your ereader.

Yep, I was already doing that, actually. Since it is so simple (and quick too) to hypenate and dehypenate books, it doesn't really matter that the editor is not able to behave like I wrote above.
For sure, it is easier than using a regex like the one you sampled for every word I need to search!

[eschwartz]

It would be a travesty if the editor utterly failed to do it's job. So... GOD NO FOR HEAVENS SAKE!!!

[Ruskie_it]

Originally Posted by eschwartz

It would be a travesty if the editor utterly failed to do it's job. So... GOD NO FOR HEAVENS SAKE!!!

Well no, come on... it might be switchable!
Something along the line "Account for hidden (or hypenation) in search function: Yes/No".
It wouldn't be THAT bad!
If I were Kovid I would push it very low on the feature development stack, since the workaround is really easy and simple, and yet... I'd consider it a nice to have.

[Toxaris]

As hyphenation with soft-hyphens like that gives issues in almost all ePUB readers, it should not be used at all in my opinion...

[Ruskie_it]

Originally Posted by Toxaris

As hyphenation with soft-hyphens like that gives issues in almost all ePUB readers, it should not be used at all in my opinion...

I don't know in epub readers, but in kindle I've never seen a problem given by soft-hypenation.
On the other hand, Kindle does not support hypenation, it's one of its biggest flaws imho, but it can handle soft-hypenation.
Given the fact the I HATE HATE HATE, and I am not alone I think, the way a book gets displayed when no hypenation is available, with different spaces between words, sometimes with rows shorter than the rest of the text and so on... I am glad that the option to soft-hypenate the book exists.

[arspr]

We discussed this "issue" with non-printable characters (not only soft hyphens but zero length joiners or non joiners, and others, used more or less frequently in some languages). And Kovid said that there's no easy way to make them visible, let's say through specific or even just a general placeholder which meant "there's something hidden here, be careful".

But I really hope he could make some magic... (If MS Word is capable, why is not Calibre going to be?)

[eschwartz]

Originally Posted by arspr

We discussed this "issue" with non-printable characters (not only soft hyphens but zero length joiners or non joiners, and others, used more or less frequently in some languages). And Kovid said that there's no easy way to make them visible, let's say through specific or even just a general placeholder which meant "there's something hidden here, be careful".

But I really hope he could make some magic... (If MS Word is capable, why is not Calibre going to be?)

I knew there must've been some reason.

Now I remember -- and I bet it will indeed happen as soon as someone totally reimplements QPlainTextEdit to fix its support of invisible chars.

[arspr]

And yet another brainstorming about making non-printable characters visible: through named entities.

• Calibre does prefer the usage of unicode characters over named entities. But this case could be the exception to that general rule.
• I propose something like a "white-list" of named entities/"black-list" of their associated unicode characters. (Also fully customizable?)
• Then some options which could cause:
  • Nothing. As Calibre currently beheaves. The default option.
  • Warn me. Whenever a new editor window is opened an automatic search is launched against the black-list. And you get a warning. But nothing else is done.
  • Automatically change when opening but revert on save. The black-listed unicode chars are automatically changed by entities. This procedure is reverted when saving. I mean that the epub/azw3 file on the HDD always contains unicode chars but the opened editor shows entities.
  • Preserve entities. The same as above but entities are preserved when saving.

Of course this scheme needs some polishing and those white-listed entities would also affect other areas of the code, like the main named entities check or the automated change into Unicode when typing the ending ";" ... But I think the general idea is more or less clear.

As I said just brainstorming mode.