RegEx & Unicode

[capnm]

I've been using the following regex to abbreviate series names as initialisms:

\s*([a-zA-Z]|\d+\.?\d*)[a-z\']*\.?\s*

\1

Now that more & more of my series include unicode characters, I'm wondering if there is an easy way to either modify the [a-zA-Z] and [a-z'] terms to include appropriate accented characters, or to transliterate (transcode?) the string before regex processing.

Or is my best bet just to manually transcode my series? (yuck)

[Serpentine]

Supply a sample(s) and expected result(s), make life easy.

[DoctorOhh]

Also where are you using this and why?

[capnm]

Originally Posted by Serpentine

Supply a sample(s) and expected result(s), make life easy.

Föô bár
Fb

Though that's pretty irrelevant. I'm not looking for debugging this particular regex, or to start adding tons of individual unicode characters to it.

I'm wondering if calibre's flavor of regex is/can be unicode aware, since I suspect some flavors of regex are, but I've never had occasion to explore the issue before.

Alternatively I thought there might be some calibre template functions that would transliterate a unicode string (though that would have other side effects).

Originally Posted by dwanthny

Also where are you using this and why?

At the moment -- in custom columns and plugboards to abbreviate long series names.

But again, it's more of a general question, since at various times, for various reasons, authors, titles, series, etc., get plugged into regexps, and they all have the occasional unicode character which doesn't fall into the standard [a-zA-Z] or \w range.

[kovidgoyal]

See http://docs.python.org/library/re.html#re.UNICODE

[capnm]

Originally Posted by kovidgoyal

See http://docs.python.org/library/re.html#re.UNICODE

Thanks, I couldn't figure out how to invoke that, so I wasn't sure if it was applicable, but I finally found how to use the (?u) flag.

So I think I'm most of the way there ... but I could still use a little help

I should be able to replace [a-zA-Z] with (?u)[/w/D] (if I ignore _ for now), right?
[edit: of course this doesn't work -- I'm always trying to stick exclusions in a group and it's never worked yet]

But is there an easy equivalent to [a-z] ?

[kovidgoyal]

If you mean only lowercase letters, then no. Though you can use unicode character ranges, like this [\u0028-\u0046] if you know the character ranges you want.

[capnm]

Well, I'm stumped again.

I mean -- [\u0000-\uFFFF]* should match anything, including punctuation and two-part characters, right?

But not only does it not grab accented characters, it doesn't grab v,w,x,y, or z.

[ldolse]

I think re.UNICODE only causes \w & \W to match non-ascii characters, at least practically speaking. Which would be okay except that \w also includes numbers - if you're okay with matching numbers then \w+ should be ok.

I've always wished it would make [a-zA-Z] work the way capnm wants. I suppose you might be able to mix it with an digit exclusion lookahead:

(?u)(?=[^\d]+)(\w+)

But it's going to get tricky.

[kovidgoyal]

Do a bit of googling on how to use unicode char ranges in python regexps. I haven't ever used them myself, so I cannot comment.

[Serpentine]

You can most likely use something like :

(?i)(?:^|\s+)(\d+\.?\d*?|[\D])

To grab all of the interesting first characters/numbers

string = r'Föô bár  šjohka'
>>> regex.findall(string)
[u'F', u'b', u'\xe1']

I'm sure you can work it into a replacement without too much of a problem.

[capnm]

Originally Posted by ldolse

I think re.UNICODE only causes \w & \W to match non-ascii characters, at least practically speaking. Which would be okay except that \w also includes numbers - if you're okay with matching numbers then \w+ should be ok.

I've always wished it would make [a-zA-Z] work the way capnm wants. I suppose you might be able to mix it with an digit exclusion lookahead:

(?u)(?=[^\d]+)(\w+)

But it's going to get tricky.

And what I really want is some form of (?u)[a-z] or (?u)[A-Z] to work, but I think I'm out of luck on that one.

I played/poked around a bit and here's what I found (which may even be correct):

This flavor of python regex supports (?u), which makes \w, \d, \b unicode aware.
It doesn't support \unnnn or \Unnnnnnnn.
It doesn't support upper/lower properties or character classes.


Revising your lookahead idea, I think this will emulate a unicode aware [a-zA-Z]

(?u)\w(?!(?<=[\d_]))

but that doesn't solve my wish ...

Oh, well. This was supposed to be a quick exercise in tweaking some template code. Now I'm just being stubborn

Since I don't forsee any great inspiration on how to make a unicode [a-z], I'll probably settle for adding [à-ÿ] to at least make it Latin-1 aware ...

[capnm]

Originally Posted by Serpentine

You can most likely use something like :

(?i)(?:^|\s+)(\d+\.?\d*?|[\D])

To grab all of the interesting first characters/numbers

Yes ... that's very like one of my variations:

\s*(\d+\.?\d*\w?|\w)[a-z_\']*\.?\s*

But I'm curious -- why the leading (?:^|\s+) instead of \s*
is there a functional difference?

Thanks!

[Serpentine]

Originally Posted by capnm

But I'm curious -- why the leading (?:^|\s+) instead of \s* is there a functional difference?

If you're not using the unicode support or don't have the locale flag set, you will end up with some non-whitespace characters(also punctuation you want to avoid) being seen as a break in a word; If you were to use \s*, this would then mean that the next letter - which has the possibility of being in the middle of a word, will be used as an initial.

By specifying that the starting point either has to be the start of a string (careful of multiline issues), this situation is removed as the word can only be separated by one or more spaces.

If you want to use it for replacement - as you wanted, the pattern would be :

find: (?iu)(?:^|\s+)((?:\d+\.?\d*?)|(?:[\D]))[\w]+
replace: \1

Tho it then uses the unicode flag, a trade off between being robust and easily matching things.

[capnm]

@Serpentine,

Yes -- that makes sense ... and might be a good way to address what led to my latest round of tweaking -- accented chars from the middle of a word popping up in my abbreviations.
Of course it will complicate the other tweaking I've done over time to make the abbreviations more readable/pertinent, like including most punctuation, but not periods and quotes, and including numeric strings and all capital letters, and ...

Hmmm ... if I abandon including all capital letters, the rest will probably fall together -- that's probably the unicode sticking point ...

After several tweaks, these regexps are probably best rewritten from scratch as they've accumulated redundancies and idiosyncrasies, but sometimes I'm lazy

Maybe I'll focus on redoing my {author_sort}{series}-->{author} plugboard template for the Sony, since someone else might find it useful ...

Thanks!