using re() and shorten() for custom comments column?

[rebl]

Hello all,

I'm trying to build a custom comments column to display in the library list, that would show text from the comments filed, stripping (removing) any html tags and shortening the string to a predefined length.
I've been looking all over for more information but all I found is not enough.
Especially, I don't understand a few things:
1. When building a custom column from other columns, according to the tooltip in the GUI, "Field template uses the same syntax as save tamplates"
but the first google search for calibre re() function takes me to the calibre documentation (template reference), where the re() function has three arguments: re(val, pattern, replacement)
In the end I've found the correct page: http://manual.calibre-ebook.com/template_lang.html where only two arguments are needed.
But what is the diference betwenn the two? Where is the 3-argument version applied if the custom colums and save templates use the 2-argument one?

2. I suppose there is no (simple) way to combine two functions in a custom field? I've read somewhere here on the forum about the simple mode being the default and about some programing or advanced mode(s) but I haven't figured out how to you use the other two.

If I use

{comments:shorten(50,...,0)}

I get a shortened string but also html tags.

But I would like to remove the HTML tags too, so i'd need a function like:

{comments:re((\<.+\>)?,):shorten(50,...,0)}

Please ignore for the moment how efficiently written is the re() function.
I know the above does not work, but how exactly could I appy two functions to the same input field (comments) for a custom column?

3. I have some strange problem with the re() function and I would really appreciate your help.
If I use

{comments:re((\<.+\>)?,)}

The various html tags (

<hr> <div> </div>

etc) are removed but in some case (when a comma is present) the resulting string is reversed around the comma and I can't figure out why. It's like switching \2 \1 when swapping author surname/name.
Also in some other cases the whole test is removed (like a greedy behavior) and the output is empty.
This behavior is not 100% reproducible and this puzzles me, because when I've done some minor metadata edit in the comment field, the output became the expected one. But other time(s) after a minor edit like adding a space in the html source of the comment the whole output disappeared.
I had a comment similar to this (for a Thomas Mann book in case you're wondering):

<hr> 
A portrait of the German bourgeois society, throughout several decades of the 19th century.

When it was in Romanian (it was like

<hr> Some text 1, some text 2.

) the function's result was:

some text 2., some text 1

...and I have this kind result for most if not all of the books.
But after replacing the comment with the English translation while keeping the tags intact, (just for doing some tests and for pasting here a more meaningful comment ) it suddenly started working and now the results is the expected text without html.
My question(s) are why the editing solved the problem since I've not altered the text pattern and why the re() does not work for the other books and instead it reversed the text fragments around the comma.

Another example of the strange behavior of this custom column:
comments field:

<p class="description">Premiile British SF Association şi John W. Campbell Memorial, 2013.</p>

initial output:

2013., Premiile British SF Association şi John W. Campbell Memorial

after editing the comment (I've just inserted an additional space between W. and C) the result was just EMPTY, no text at all!!!

4. Regarding the (\<.+\>)? part i wanted a "not greedy" matching of strings that start with < and end with > but matching the shortest possible string. I'm not sure it works though.

Thank you all for your time and for any help on this!

[eschwartz]

general program mode is far more flexible and allows multiple functions to be used in arbitrary ways. Unlike single-function mode, it does not assume you are operating on the base template (because there is none) and the field you wish to use must be explicitly stated as the first parameter.

Try:

program:

shorten(
    re(
        field('comments'),
        'some-regex',
        'replacement-text is empty'
    ),
    50,
    '...',
    0
)

Whitespace is optional. I use it to lay out the template and what I am doing more clearly.

[rebl]

Thank you eschwartz, so in program mode the re() function is used with three arguments, while in the single-function mode is used with just two arguments?
But how do I enter program mode? I can't see any optiosn in the add custom column field.

[chaley]

@rebl: You cannot nest functions in single function mode, what you called "simple" mode. It is also difficult to distinguish between an argument containing a comma and the argument separator comma. Because of this, you *really* want to use template program mode (TPM) or general program mode (GPM). Both give you the possibility of nesting function calls. Both distinguish between commas in strings and commas used as argument separators. TPM is better for small expressions needed in larger templates. GPM is better when complexity is high, when you want to be able to more easily read the code, or when constant strings in the template contain { and } characters.

Your example in GPM:

program:
	re(shorten(field('comments'), 50, '...', 50), '<.*?>','')

Your example in TPM:

{comments:'re(shorten($, 50, '...', 50), '<.*?>','')'}

In both cases HTML stripping won't work if the tag contains a string containing a > character. That situation is rather hard to deal with in regular expressions.

EDIT: trumped (somewhat) by eschwartz. To answer your question about how you "get into program mode", you simply begin the template with the word "program:"

[eschwartz]

The dot plus will match all text including tags, thus matching everything from the opening of the first tag to the closing of the last tag.

Use:

<[^<>]*>

to restrict the match using a set which excludes tag opening and closing brackets.

chaley, if the html contains a bracket in the text or class there are other problems bigger than a failed regex...

[chaley]

Originally Posted by rebl

Thank you eschwartz, so in program mode the re() function is used with three arguments, while in the single-function mode is used with just two arguments?

In single function mode the contents of the field referenced in the template are passed as a hidden first parameter. So in the template

{comments:shorten(50,...,50)}

shorten has

field('comments')

as its first parameter,

[chaley]

Originally Posted by eschwartz

The dot plus will match all text including tags, thus matching everything frkm the opening of the first tag to the closing of the last tag.

Use:
[CODE]<[^<>]*>[CODE]
to restrict the match using a set which excludes tag opening and closing brackets.

The dot+? will match up to the first >, not the last one.

And neither pattern will work for something like

<a href="...", alt="demonstrate that a > b">foo</a>

Yes, it will work if you use entities, but not everyone does.

[eschwartz]

Originally Posted by chaley

The dot+? will match up to the first >, not the last one.

And neither pattern will work for something like

<a href="...", alt="demonstrate that a > b">foo</a>

Yes, it will work if you use entities, but not everyone does.

cross-posting wars

bigger problems... nothing I can do about that save to say, "use them. Learn what they are and use them."

Didn't notice the "?"

[rebl]

chaley and eschwartz - Thank you guys, very much! You are awesome.
You've put me on the right track and from now on I think I can manage.
Of course a little trial and error can't hurt
I don't have that many non-empty comments fields and I doubt many of them would contain "un-ampersanded" > or < characters... (&gt; is a html entity right?)
I need the comments filed because that was the only field I could think of, to store the "version" of a book, when reading metadata from the file name

john milton - paradise lost from guttenberg v1.1.rtf

I'm using a modified regex from "quick preferences" plugin:

^(?P<author>((?!\s-\s).)+)\s-\s(?:(?:\[\s*)?(?P<series>[^\[\]\(\)\d]+)(?:\s*\])?\s?(?P<series_index>[\d\.]+)(?:\s*\])?\s-\s)?(?P<title>.+)\s?v\.?(?P<comments>[^\[\]\(\)]+)(?:\s*\]\))?$

to get the version number and put it in the comments field.
The regex above puts 1.1 in the comments field.
Than I just need to display a part of the comments field in a custom column, in order to be able to sort and see which "comment" contain a version number.
But the fact that comments is a html field complicates things.
Since I am not allowed to use custom fields such as #vers I needed to use a built-in one.
But I guess it'd be simpler to use a different predefined filed that I have no use (initially) for temporary storing the "version number" - for example publisher... it doesn't matter which, 'cos I will anyway have to use "search replace" in metadata....

That is assuming I am definitely not allowed to use something like ?P<#version> when adding books...