Templates: various questions not worth their own thread

[ownedbycats]

Can a user category be referenced in a template? e.g. if #column is 'foo' and user category is '@bar' then 'foobar'.

If so, would anything break if I then added 'foobar' to that user category?

[chaley]

Originally Posted by ownedbycats

Can a user category be referenced in a template? e.g. if #column is 'foo' and user category is '@bar' then 'foobar'.

If so, would anything break if I then added 'foobar' to that user category?

I am not sure what question you are asking. Guessing ...

The template function user_categories() returns a list of user categories that the book is in. Using that then this template might return what you might want:

program:
	if 'foo' == $#column && '^bar$' inlist user_categories() then
		'foobar'
	fi

I don't know if the performance is acceptable.

[ownedbycats]

New question: Is there a way to test the performance of a template? I suspect one of my composites is causing slowness, but I'm not sure which one it is.

[chaley]

Originally Posted by ownedbycats

New question: Is there a way to test the performance of a template? I suspect one of my composites is causing slowness, but I'm not sure which one it is.

No, there isn't any profiling for templates. And I am not sure it would be useful in the face of templates that refer to other templates.

In my experience performance problems arise if a composite is used for sorting or searching. This requires calibre to evaluate the column for every book in the (virtual) library. In other cases the composite is evaluated when it is to be displayed.

Also note that most of the time a GPM template is faster than a TPM or Single Function Mode template. Reason: calibre can cache the "compiled" GPM template.

Finally, some functions are just slow, such as any of the format_* functions that must look at the disk to get the answer.

[capink]

Wouldn't measuring the time elapsed (in template tester) provide some kind of rough measure, by which you can compare different templates? e.g.

program:
	start = format_date(today(), 'to_number');
	strcat('a');
	format_date(today(), 'to_number') - start

Edit: A custom function called (time_it) can do the same

def evaluate(self, formatter, kwargs, mi, locals, template):
    from calibre.utils.date import now
    start = now()
    template_output = formatter.__class__().safe_format(template, mi, 'TEMPLATE_ERROR', mi)
    diff = now() - start
    return diff.total_seconds()

program: time_it("program: strcat('a')")

[chaley]

Originally Posted by capink

Wouldn't measuring the time elapsed (in template tester) provide some kind of rough measure, by which you can compare different templates?

One problem is that the work to compute the time could easily exceed the time to compute the template. I think that I would need to add profiling to the formatter itself to avoid this.

Another problem is that the template time depends on the metadata, especially if there are conditionals. For example, the time to split and join tags depends on the number of tags. This argues for some ability to average multiple invocations of the template.

Finally, the execution context matters. By this I mean whether the template is being used in search/sort, and for display purposes whether the column value has been cached. In the latter case the formatter doesn't even know it was used.

My feeling: if templates are complex enough to need this level of profiling then one should consider implementing them as custom functions. Those are compiled python and run at full python speed. You can also avoid extra split/join operations because you know how you will use the information.

[ownedbycats]

I think I identified the misbehaving template.

New question (likely more regex issue): When using a 'contains' on a taglike, how can I make it match when there's multiple options? e.g.

program:
   f = field('#field');
   strcat
   (
    contains(f, "^foo|bar|foobar|test123$", 'icon.png:', ''),
   )

it sometimes breaks if "test123" is in the taglike. It works again if I remove the exact-match modifiers but then it's not exact matching.

[capink]

Originally Posted by chaley

One problem is that the work to compute the time could easily exceed the time to compute the template. I think that I would need to add profiling to the formatter itself to avoid this.

Another problem is that the template time depends on the metadata, especially if there are conditionals. For example, the time to split and join tags depends on the number of tags. This argues for some ability to average multiple invocations of the template.

Finally, the execution context matters. By this I mean whether the template is being used in search/sort, and for display purposes whether the column value has been cached. In the latter case the formatter doesn't even know it was used.

My feeling: if templates are complex enough to need this level of profiling then one should consider implementing them as custom functions. Those are compiled python and run at full python speed. You can also avoid extra split/join operations because you know how you will use the information.

Thanks for the informative and comprehensive reply.

[capink]

Originally Posted by ownedbycats

I think I identified the misbehaving template.

New question (likely more regex issue): When using a 'contains' on a taglike, how can I make it match when there's multiple options? e.g.

program:
   f = field('#field');
   strcat
   (
    contains(f, "^foo|bar|foobar|test123$", 'icon.png:', ''),
   )

it sometimes breaks if "test123" is in the taglike. It works again if I remove the exact-match modifiers but then it's not exact matching.

I think you need to change the regex:

program:
   f = field('#field');
   strcat
   (
    contains(f, "^(foo|bar|foobar|test123)$", 'icon.png:', ''),
   )

[chaley]

Originally Posted by ownedbycats

I think I identified the misbehaving template.

New question (likely more regex issue): When using a 'contains' on a taglike, how can I make it match when there's multiple options? e.g.

program:
   f = field('#field');
   strcat
   (
    contains(f, "^foo|bar|foobar|test123$", 'icon.png:', ''),
   )

it sometimes breaks if "test123" is in the taglike. It works again if I remove the exact-match modifiers but then it's not exact matching.

Are you sure that you want contains() on a tags-like column? The contains() function looks at the string of tags as a single element, not as a list. As such, if the column value is 'foo, zorro' then your expression can never match. It will succeed only if the column contains a single value, which raises the questions "why is it tags-like?".

This expression checks for the existence of any one of the tags, ignoring whether or not other tags also exist.

program:
	f = field('#mytextmult');
	strcat
		(
			if "^(foo|bar|foobar|test123)$" inlist f then 'icon.png:' fi
		)

Alternatively, this, which is closer in spirit to what you did.

program:
	f = field('#mytextmult');
	strcat
		(
			list_contains(f, ',', "^(foo|bar|foobar|test123)$", 'icon.png:', '')
		)

[ownedbycats]

For what it matters, the actual template has quite a few individual entries, e.g.:

program:
   f = field('#fanficcat');
   strcat
   (
    contains(f, "Fallout", 'fallout.png:', ''),
    contains(f, "Half-Life", 'halflife.png:', ''),
    contains(f, "Mass Effect Trilogy|Mass Effect: Andromeda", 'masseffect.png:', ''),
    (at least a dozen more like this)...
    contains(f, "^(2001: A Space Odyssey|Alien|Short Circuit)$", 'Film.png:', ''),
   )

I just combine the miscellaneous one into generic film/book/videogame icons. Since it's a stored template both the column icons and emblems get updated.

Since I want to display all the relevant icons, I'm not sure that an if statement would work too well. But would list_contains be appropriate?

[chaley]

Originally Posted by ownedbycats

For what it matters, the actual template has quite a few individual entries, e.g.:

program:
   f = field('#fanficcat');
   strcat
   (
    contains(f, "Fallout", 'fallout.png:', ''),
    contains(f, "Half-Life", 'halflife.png:', ''),
    contains(f, "Mass Effect Trilogy|Mass Effect: Andromeda", 'masseffect.png:', ''),
    (at least a dozen more like this)...
    contains(f, "^(2001: A Space Odyssey|Alien|Short Circuit)$", 'Film.png:', ''),
   )

I just combine the miscellaneous one into generic film/book/videogame icons. Since it's a stored template both the column icons and emblems get updated.

Since I want to display all the relevant icons, I'm not sure that an if statement would work too well. But would list_contains be appropriate?

Yes, given the above you want list_contains(), not contains().

In this context the if is the same as a function call. Both are expressions that return a value. You can have as many as you want, and you can intermix them. In other words, this works:

program:
	f = $tags;
	strcat(
		if '^Fallout$' in f then 'fallout.png:' fi,
		list_contains(f, ',', '^Half-Life$', 'halflife.png:', ''),
		if "^(Mass Effect Trilogy|Mass Effect: Andromeda)$" in f then 'masseffect.png:' fi
	)

The 'if' is slightly faster because it is directly executed by the formatter. list_contains() is a template function call that is not inlined.

[ownedbycats]

Okay, the ifs are working as expected. However, for some reason I'm having some issue with the exact matching. Did I make a formatting mistake?

Interestingly, it's the $ that seems to make it fail. The ^ alone works.

If it matters, the #fanficcat of this book is Crossover, Fallout, Mass Effect Trilogy

[dunhill]

Hello people could you help me with a template. I want to concatenate the publisher's series with the serial number. What should I put on the note sheet?
Already very grateful

[chaley]

Originally Posted by ownedbycats

Okay, the ifs are working as expected. However, for some reason I'm having some issue with the exact matching. Did I make a formatting mistake?

Interestingly, it's the $ that seems to make it fail. The ^ alone works.

If it matters, the #fanficcat of this book is Crossover, Fallout, Mass Effect Trilogy

Sorry, I led you astray. You should use 'inlist', not 'in'.