regex - issue with spaces?

[mrmikel]

Calibre takes nbsp and turns it into A0!

Since nbsp makes headaches in Sigil, I am not surprised it causes problems in Calibre.

At the fantastic rate this editor has come into being, it is not surprising there are a few problems.

[susan_cassidy]

<p class"something"> is invalid HTML. It should be <p class="somthing">.

[BetterRed]

Originally Posted by theducks

It only applies to (friendly) Named entities. You can still use their entity-numbered ,ISO-8859-1 versions Again & #160 is a NBSP
just like & #62 is a >

It is just a PITA to have to keep a chart handy

Perhaps the editor could have a set of preferences that would allow the user to assign keyboard shortcuts to commonly used 'entities' like non-breaking-space, em-dash etc. I've done it with AHK, but I always prefer using built-in features rather than 3rd party gadgets.

@mrmikel - sorry, I misinterpreted your comment as meaning 'non breaking spaces' should be removed irrespective of their representation - hence 'banished'.

I don't care if its represented by &nbsp;, #160, an A0, or a cube of sugar - as long as I can 'see' it when I'm reading the book.

BR

[arspr]

Hey guys, please let's stay focused or nobody (Kovid less than anyone else) is going to pay attention to the POSSIBLE BUG posted in the thread:

• During the development thread there was a lot of discussion about named entities or unicode equivalents. Kovid opted for the latest one because he explains that option is more robust. (No need for !Doctypes sentences). NEVERTHELESS you can still use named entities if you want, BUT if you use any automated tool like HTML Fix or Beautify, they are going to be automatically converted to their unicode equivalents. If you don't like this behaviour, as Kovid would possibly say, feel free to submit the patches you want, Calibre is opensource.
• IIRC Sigil and Calibre S&R (or F&R, whatever you want to call it) both work in the VERY SAME WAY, but Regex being ALWAYS case sensitive in Sigil but not in Calibre. And that comes from a new version/enhancement in the Regex engine. It's something outside Calibre itself.
• In this way it doesn't matter if you either copy+paste or just Ctrl+f the selected text you want to search, as I previously posted, it should be ALWAYS found either in Sigil or Calibre. (Just be careful about the "forbidden" special characters in Regex which should be escaped, a "\" being put before them, but IIRC it is automatically done when filling the Find text box through both methods).
• So, if some of you, cybmole or DoctorOhh as example, cannot get it working, PLEASE double and triple-check that there's no error in your procedure, (ie. it's not your fault or at least your obvious fault), test with other strings, and post detailed examples, steps, whatever thing you can think of, because there can be a HIDDEN BUG here. (And as I said, I wasn't able to reproduce the issue cybmole firstly posted, as my screenshot showed, but I had already detected other strange behaviour within S&R about Count All).
• But, please, don't mix things or nobody is going to pay any attention to us. (This thread is a whole mess).

[cybmole]

good points.

For the record I tested twice, once before starting the thread then again after jackie posted her method. I gave my detailed steps already, for both sigil & calibre.

my current hypothesis is that when you use windows copy + paste, calibre pre-processes the paste & changes the code on the invisible character(s), but does not do that we you pre-select & then open FIND.

then it can still be true that both S&R work in the very same way. The "bug" or difference is to do with inconsistencies in the various ways in which you can populate the find box with a string containing an invisible character

I have returned & deleted the original library book now but there is still the extract which I attached several posts back, if anyone wishes to continue testing.

[DoctorOhh]

Originally Posted by cybmole

I have returned & deleted the original library book now but there is still the extract which I attached several posts back, if anyone wishes to continue testing.

Did you try every step with the extract? Because using the extract it made no difference which way I populated F&R it didn't work.

[Sunlite]

I did try to reproduce the issue and found the following.

OS: Windows 8 64bit
Calibre: portable 1.17

I can populate the 'find' field by either Ctrl + f or by Ctrl + c & Ctrl + v and the copied string contains the nbsp and the search is successful.

If I copy the string by right click and 'copy', the nbsp goes missing and is replaced by a normal space.

This is also visible by leaving the search out of the equation and pasting into the code itself. With the first way of copying, the nbsp will be marked as the others are. With the second method there is no mark.

Pasting into another application will show the same result. Existing nbsp with keyboard action, normal space with right click copying.

Using the 'copy' function in another application and pasting into ebook-edit does not lose the nbsp.

[arspr]

Originally Posted by Sunlite

I did try to reproduce the issue and found the following.

OS: Windows 8 64bit
Calibre: portable 1.17

I can populate the 'find' field by either Ctrl + f or by Ctrl + c & Ctrl + v and the copied string contains the nbsp and the search is successful.

If I copy the string by right click and 'copy', the nbsp goes missing and is replaced by a normal space.

This is also visible by leaving the search out of the equation and pasting into the code itself. With the first way of copying, the nbsp will be marked as the others are. With the second method there is no mark.

Pasting into another application will show the same result. Existing nbsp with keyboard action, normal space with right click copying.

Using the 'copy' function in another application and pasting into ebook-edit does not lose the nbsp.

GOOD CATCH!!! I confirm the issue.

Right-Click & either Copy or Cut converts the Non-Breaking Space into a common one.

• Ctrl+c (or x) & Ctrl+v >>>> Non breaking space is maintained.
• Ctrl+c (or x) & Right-click Paste >>>> Non breaking space is maintained.
• Right click Copy (or Cut) & Paste however you prefer >>>> Non breaking space is converted into normal space.

So it really seems the issue is in the Copy (Cut) function.

You should post it in the Calibre Bugtracker.

[cybmole]

well done guys - I used right click mouse menu for copy / paste so the above conclusions are consistent with my original experience/ thread starter. will someone who knows exactly how to explain this for Kovid do the bug tracker report please

[theducks]

Great find
I almost always use the keyboard methods for S&R so I never saw that issue.

[cybmole]

Originally Posted by theducks

Great find
I almost always use the keyboard methods for S&R so I never saw that issue.

i much prefer KB to mouse also, but since I have to grab the mouse anyway to select the stuff I want to find..... & I know that CTRL+ V is paste but I can never remember whether CTRL + C is cut or copy...

the most amusing thing about this, in hindsight, is that the book in question was truly awful

[kovidgoyal]

https://github.com/kovidgoyal/calibr...77188256923e0d

[arspr]

A possible related issue (or not)

&nbsp; (in unicode flavour) and normal spaces are not distinguished in the autocompletion dropdown menu in the Find/Search text box.

I mean, in the book I'm reading (Infinite Jest) there are some possessive "’s" for "commented" texts. Let's say "‘Cheers!’ ’s". And I've wanted to change that space in the middle by a &nbsp; (in unicode flavour). But while typing "’&nbsp;’" (through copy/paste, and the GOOD copy/paste through keyboard shortcuts ), the dropdown menu has offered my previous typed "’ ’" with just a normal space. Is this to be expected? Can it be filtered?

And another small possible enhancement. Would it be possible to translate the colours used in the editor to that dropdown previous searched/replaced string matches? I mean there's no way to tell the difference between space types in that dropdown lists.

[DaleDe]

Originally Posted by BetterRed

I like to have two spaces between sentences, especially for foreign (for me) language texts. Currently I achieve this via putting a 'space' followed by a 'nbsp' between sentences.

Some of my colleagues whose first language is not English also like to have double space an the end of sentences in English text. For the same reason as me - it makes reading foreign language texts easier.

If 'nbsp' is banished - because "[its] not part of HTML" - how would I get two spaces between sentences, most (all ?) viewers seem to ignore a second 'space'.

BR

nbsp is not banished in HTML but is not defined in XML. ePub uses XHTML which is a predefined XML implementation, not an HTML implementation no matter what extension you use. It is possible to defined named entities in the file itself but better to use the numeric equivalents. 160 is decimal while A0 is hexadecimal. Either will work if defined properly. You are correct that in XML and HTML multiple regular spaces are reduced to a single space when being interpreted.

Dale


Dale