View Single Post
Old 10-06-2012, 05:41 AM   #57
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by Jellby View Post
What does "tokenise sequence" do? I'd like to know it to find the best translation. Does it just escape special characters? i.e. does it attempt to find the original input (if unmodified afterwards) as a literal string, but with the regexp engine?
To expand on meme's reply as it is a little bit complicated

If you have the Auto-tokenise option on, then it means when you hit Ctrl+F on some text it will:
- Escape any regex characters in the selection, such as periods, parentheses, brackets etc
- Runs a regex replacement of ([\n\s]{2,}) with \s+ to ensure two or more spaces/newlines get substituted with \s+. This is to handle some of the funkiness of the Sigil/Qt editor, where it puts a unicode newline and spaces in as part of the Pretty Print operation. This is handy when you are doing multiline regex to save you typing \s+ in between all the time.

If you choose the "Tokenise selection" context menu option, then it will do the same as above (escape and tokenise spaces), but add a regex find/replace of (\d+) with \d+ in order to replace any numbers in the Find text. This is not done in the "auto tokenise" because it is something you are less likely to do. And it works on your selection in the Find box (the whole text if you have no selection) so if you want to keep a numerical style name intact but substituting for a numerical id you can do so.

Note the next 0.5.907 build adds a slight tweak to Tokenise Selection so that it does not escape the text if it sees a \ in there. This is to allow you to repeatedly apply (or do a Tokenise Selection to do just the numbers while having Auto Tokenise on) without ending up with double escaped \\s+ etc.
kiwidude is offline   Reply With Quote