Calibre's XPath implementation for TOC detection

[tram]

I'm converting a few books to go on my kindle, and I'm trying to get the chapters set up the way I want. One thing I'm noticing is that Calibre seems to use a slightly different flavor of xpath than what I'm used to. Where I would write

//p[matches(., '^[A-Z ]+$')]

, Calibre seems to respond to things in the form

//h:p[re:test(., '[A-Z]+')]

re:test seems to behave differently than matches. Is there any documentation on re:test, or anything anyone can tell me that might enhance my googling abilities?

The regular espression I'm trying to use that works with matches but not with re:test follows:

/h:p[re:test(., '^[A-Z]+ [A-Z ]+$')]

[Manichean]

There's documentation on XPath in the manual. The behaviour of the regular expressions and the re.test function should be documented in the python documentation.

[Starson17]

Originally Posted by tram

re:test seems to behave differently than matches. Is there any documentation on re:test, or anything anyone can tell me that might enhance my googling abilities?

The regular espression I'm trying to use that works with matches but not with re:test follows:

/h:p[re:test(., '^[A-Z]+ [A-Z ]+$')]

What makes you think test doesn't work the same as matches? Is it possible that what you are trying to match with test isn't the same as what you are testing the match expression on? Are you trying to match across lines?
re:test()
re:test(src, pattern, flags) returns true if the string src matches the regular expression pattern.

[tram]

I have an XPath visualization tool that I've been using to test expressions, querying against the HTML file that gets left in the debug 'processed' directory. Using that tool, I pull out the elements I'm looking for, but I use matches() instead of re:test. The tool isn't written in python, so that's probably the difference.

I can check the manual, but is there a standard method for testing xpath queries aside from the generate-and-view method? I feel like the regular expression I'm trying to use isn't particularly complicated, and that it should work just fine.

[Manichean]

Originally Posted by tram

I can check the manual, but is there a standard method for testing xpath queries aside from the generate-and-view method? I feel like the regular expression I'm trying to use isn't particularly complicated, and that it should work just fine.

I don't know if XPath comes in several flavors as well, but for regexes, your test tool needs to support the python flavor. Personally, I use python to test expressions, though that may be a bit overkill if you don't actually use it as a language...

[wallcraft]

The XPath Tutorial says:

re:test()
re:test(src, pattern, flags) returns true if the string src matches the regular expression pattern. A particularly useful flag is i, it makes matching case insensitive. A good primer on the syntax for regular expressions can be found at regexp syntax

Python Docs say:

7.2.6.4. search() vs. match()

In a nutshell, match() only attempts to match a pattern at the beginning of a string where search() will match a pattern anywhere in a string.

Is "matching" being used in the Python sense in re:test, i.e. as in re:match?

[janvanmaar]

Originally Posted by tram

I'm converting a few books to go on my kindle, and I'm trying to get the chapters set up the way I want. One thing I'm noticing is that Calibre seems to use a slightly different flavor of xpath than what I'm used to. Where I would write

//p[matches(., '^[A-Z ]+$')]

, Calibre seems to respond to things in the form

//h:p[re:test(., '[A-Z]+')]

Yes, I was also surprised by the XPath syntax. I believe it is Calibre specific thing, Kovid may correct me if I am wrong. But it was very simple to get used to it.

Originally Posted by tram

re:test seems to behave differently than matches. Is there any documentation on re:test, or anything anyone can tell me that might enhance my googling abilities?

The regular espression I'm trying to use that works with matches but not with re:test follows:

/h:p[re:test(., '^[A-Z]+ [A-Z ]+$')]

This should work fine (with two slashes at the beginning of course). I am using regexps like this regularly in Calibre. Are you sure the text is in the p tag and not for instance wrapped in div or some other tag or is not divided between more tags AFTER calibre html processing? You can find out by checking the debug option and looking to the chosen debug directory with HTML file Calibre is really processing.

[kovidgoyal]

The h: prefix is not calibre specific, its XHTML specific. It indicates you are looking for the <p> tag in the XHTML name space. This is neccessary because the conversion pipeline internall converts everything to XHTML in the intermediate stage.

As far as I amaware calibre XPath implementation is bog standard and its regex implementation uses pythons regex implementation.

[Starson17]

Originally Posted by janvanmaar

This should work fine (with two slashes at the beginning of course). I am using regexps like this regularly in Calibre. Are you sure the text is in the p tag and not for instance wrapped in div or some other tag

I use expressions like that, too, and like you, I think his problem has nothing to do with match, search or test differences. It's probably a simple case of not matching the test regex because there's some difference between the string regex sees in test and the string he's running match on.

[janvanmaar]

Thanks for correction and explanation, Kovid, it makes perfect sense now.

[tram]

Hey Kovid, could you tell us which xpath/dom library you're using? I'm lazy and I don't want to check out the source, and I'd like to replicate what I'm seeing in the calibre TOC interface.

[kovidgoyal]

lxml .