python templates (was possible template functions)

[chaley]

EDIT: This thread started as a proposition for some new template functions but has morphed into a discussion of python templates. Most of what is said in this first post is no longer being considered.
------

Several times over the last years I have wanted to have 'real' python-like dictionaries in the template language. By that I mean a variable that holds "key:value" pairs, where both key and value are arbitrary strings. I am thinking about adding this support. Before I spend the time, I am asking "Would anyone else use this?"

The problem: how to do it that makes sense in the context of the template language. What I am considering: a set of "dict_...()" functions as follows. In all cases the parameters are strings. The function explanation code uses python syntax.

• dict_set(dict_name, dict_key, dict_value): Set the key to the value in dict.

  dict_name.set(dict_key, dict_value)

• dict_get(dict_name, dict_key[, default]): return the value from the dict. If default is not provided then the funtion returns the empty string if the key doesn't exist in the dict.

  dict_name.get(dict_key, default)

• dict_keys(dict_name, sep): return a list of keys separated by sep.

  sep.join(dict_name.keys())

• dict_values(dict_name, sep): return a list of values separated by sep.

  sep.join(dict_name.values())

• dict_to_string(dict_name): return the dict JSON encoded. This lets you put the dict into template variables, for example globals or arguments.

  json.dumps(dict_name)

• dict_from_string(dict_name, string_form_of_dict): convert a JSON encoded dict into a template dict.

  dict_name = json.loads(string_form_of_var)

Example: a template search for series with more than one author. The template (that won't work because the dict_() functions don't exist):

program:
# Get the already-computed dict if it exists
	d = globals(series_authors);
	if d then
		dict_from_string('series_authors', d)
	else
# Compute the dict the first time through and save it as a global
		series = book_values('series', 'series:True', ':::', 0);
		for s in series separator ':::':
			authors = book_values('authors', 'series:"""=' & a & '"""', '&', 0);
			for a in authors separator '&':
				dict_set('series_authors', s, list_union(dict_get('series_authors'), a, '&'))
			rof
		rof;
		set_globals(series_authors=dict_to_string('series_authors)
	fi;
# At this point we have a dict keyed by series containing a list of authors of that series.
# Check if the current book has a series, and if so does it have more than one author.
# Return the series name if it does, '' if it doesn't.
	authors = dict_get('series_authors', $series)
	if list_count(authors, '&') ># 1 then
		return $series
	else
		return ''
	fi

Here is a template search that would use the template. It would select books with a series that has more than one author. The authors can be co-authors or single authors of separate books in the same series.

You might ask "How does this differ from using select() on identfier-like strings? Some of the differences:

• Both keys and values are arbitrary strings. There is no assumption about format or content. This avoids the problem that keys and values in identifiers are always separated by a colon.
• The value in a dict can itself be a (json encoded) dict, allowing nesting.
• Performance is much better because the template processer will use real python dicts without converting to/from strings.

[capink]

The dict_get() and dict_set() looks like the best way to go about this. And yes they would be useful.

However, this brings up something I've thinking about for a while now; would it possible to add another mode called python mode to the template language. I don't know how easy to implement this would be. The idea is that it passes mi object were we can process it directly. It would be more handy than defining a template function in python — for every use once scenario — and calling it from other modes. Of-course, if the implementation involves more than just a simple pass-through, it would not be worth the effort.

There are other template things that are only tangentially related to your proposal, so I will raise them in the template thread instead.

[chaley]

Originally Posted by capink

However, this brings up something I've thinking about for a while now; would it possible to add another mode called python mode to the template language. I don't know how easy to implement this would be. The idea is that it passes mi object were we can process it directly. It would be more handy than defining a template function in python — for every use once scenario — and calling it from other modes. Of-course, if the implementation involves more than just a simple pass-through, it would not be worth the effort.

Hmmm... interesting. I can see how to do it using the same syntax as function definition, something like this:

    pydef x(a, b, ...)
        python code goes here
        it returns some string
    fedyp #Yes, this is strange but consistent

The lines between pydef and fedyp must be indented correctly for python. I would convert leading tabs to 4 spaces to avoid problems where tabs and spaces are intermixed such as when using copy/paste.

A pydef() would generate a python class that like this, where "locals" is the formatter local variable dict.

class x(object):
    def do_it(mi, locals_dict, a, b, ...)
        python code goes here
        the code must return a string or None

This compiled class could be stored in a dict named something like python_functions.

A call would look like every other function call, with mi and the locals dict automagically added to the argument list. The actual call to x by the template processor would be something like

    python_functions['x'].do_it(mi, locals, a, b, c)

The biggest problem is avoiding recompiling the function every time the template is used. I think I can deal with the majority of cases by adding to the existing template cache mechanism used for composite columns and template searches. In the db, cache.py and search.py keep a copy of the parsed template, reusing it as required. I could add the python functions there.

The next biggest problem is the db isn't always accessible to the python code. It is available for any mi created by cache.py.get_metadata(). In some cases the mi is built by hand or from a json dict, such as the templates in save_to_disk. These templates will fail if the python code attempts to use the db (cache.py.someting()) because mi._proxy_metadata will be None. Template functions such as book_values() have the same problem so we might be able to ignore it.

I think the above would more useful if it integrated with the proposed dict_() functions. The python code could set or get values from the internal support dictionary passed as a parameter. Any dict_() call template code would operate on the same dict, permitting the two to interact.

[capink]

Originally Posted by chaley

Hmmm... interesting. I can see how to do it using the same syntax as function definition, something like this

I was thinking about a completely seprate mode that would start with a keyword like python: as opposed to program:. But your idea has its merits because it allows mixing both modes.

Originally Posted by chaley

The biggest problem is avoiding recompiling the function every time the template is used. I think I can deal with the majority of cases by adding to the existing template cache mechanism used for composite columns and template searches. In the db, cache.py and search.py keep a copy of the parsed template, reusing it as required. I could add the python functions there.

Yes, I thought it would be cached somewhere.

Originally Posted by chaley

The next biggest problem is the db isn't always accessible to the python code. It is available for any mi created by cache.py.get_metadata(). In some cases the mi is built by hand or from a json dict, such as the templates in save_to_disk. These templates will fail if the python code attempts to use the db (cache.py.someting()) because mi._proxy_metadata will be None. Template functions such as book_values() have the same problem so we might be able to ignore it.

As long as we know exactly where it should fail, and document it properly, I don't think it would be a problem.

Originally Posted by chaley

I think the above would more useful if it integrated with the proposed dict_() functions. The python code could set or get values from the internal support dictionary passed as a parameter. Any dict_() call template code would operate on the same dict, permitting the two to interact.

I think this would be more clear if illustrated with pseudo code.

Also, feedback from others would help crystallize this whole thing even more.

[chaley]

Originally Posted by capink

I was thinking about a completely seprate mode that would start with a keyword like python: as opposed to program:. But your idea has its merits because it allows mixing both modes.

Another good point. I hadn't considered using a mechanism like python formatter functions, but created dynamically instead of statically. The code mostly exists to handle this. The dynamic function could have the same argument list as a stored python function, or perhaps a simpler one like

evaluate(mi, db)

where db might be None. The simpler form wouldn't be usable in the EvalFormatter, used when rendering author links, identifier link rules, and tag browser collapse model templates. It also couldn't use the standard template processor, for example to use formatter functions. Like a stored python template, a synamic python template must return a string.

I would create a class instance for it. The class instances would be cached, so you could store 'stuff' as instance variables and they would persist during the cache lifetime. Templates that are cached: icon and color rules, emblems, composites, and template searches. The caches are cleared whenever something changes the database, or in the template search case on every new search.

This method avoids creating a new function syntax, which makes things easier both to implement and to document. It loses being able to intermix chaley-language templates and python-language templates, but that is probably a small loss.

I would need to make the template editor handle these, probably by removing all syntactic highlighting in "python:" mode.

Finally, this method removes the need for dict_... methods. If you know what a dict is and how to use them then you probably can write in python, or at least understand examples well enough to change them.

[un_pogaz]

Originally Posted by capink

I think this would be more clear if illustrated with pseudo code.

Also, feedback from others would help crystallize this whole thing even more.

I agree. As for pydef it's clear, it's a locally defined function as it already exists in python.
But for the class... I have a hard time seeing how we go from that:

class x(object):
    def do_it(mi, locals_dict, a, b, ...)
        python code goes here
        the code must return a string or None

to this:

    python_functions['x'].do_it(mi, locals, a, b, c)

—

Originally Posted by capink

I was thinking about a completely seprate mode that would start with a keyword like python: as opposed to program:. But your idea has its merits because it allows mixing both modes.

Wow, that could be very powerful indeed. But it would be necessary to define its "main function" as well as its arguments, a bit like it is the case for template functions. This would allow to create "template function for single use".

python:
def main(mi):
    ...
    your python code
    ...

Note that since python accepts local definitions, we can all write in main()

...

Thinking about it, I'm not sure that python: is really useful if you add the pydef.

We just have to define:

program:
pydef __main__(mi) # start your python code
    ...
    your python code
    ...
fedyp # end your python code
main() # run your python code

And now we have our complete python implementation. We just have to define/documente that the function __main__(*args) <> main() are special functions where main() without arguments calls __main__(*args) with the arguments related to the book.

[un_pogaz]

Wait wait wait.
2 seconds. Pause.
Tell me if I'm wrong, but maybe I have a better idea.

Currently, the 'Template functions' and the 'Stored template' are saved a the same place in the DB table "preferences>user_template_functions". In order to be able to sort out what is what, Calibre uses the number of arguments of these: 0 is a 'Stored template' ; 1+ is 'Template functions'.
That's why you can't define 'Template functions' with 0 arguments.
I have the impression that all this thing of python: is a way to circumvent this limitation.
What if we treat the cause rather than the symptoms?

What I suggest is that we revise the way we save this in the DB.
Let's save separately 'Template functions' and 'Stored template' in their own preferences entry. Thus we can define functions with 0 arguments. We just need to do a version check of the DB and a data migration.

So, I agree that this poses a problem of the assenting compatibility (you can't load a +6.7 DB with 5.0 Calibre without a nice mess in your template), but I think, perhaps, that it will be much easier to implement this than python: with the different syntax highlighting changes it announces.
But it would be necessary to see with Kovid.

[kovidgoyal]

I dont particularly mind having new versions create things that dont work in old versions, but I frown on the other way. That is things from old versions should preferably not stop working when going to a new version as much as possible.

[capink]

Originally Posted by chaley

Another good point. I hadn't considered using a mechanism like python formatter functions, but created dynamically instead of statically. The code mostly exists to handle this. The dynamic function could have the same argument list as a stored python function, or perhaps a simpler one like

evaluate(mi, db)

where db might be None. The simpler form wouldn't be usable in the EvalFormatter, used when rendering author links, identifier link rules, and tag browser collapse model templates. It also couldn't use the standard template processor, for example to use formatter functions. Like a stored python template, a synamic python template must return a string.

If the simpler form is not able to do all of the above, it leaves it with only one tiny advantage; the ability to add more arguments to the function definition in the future if a need for this arises.

Originally Posted by chaley

I would create a class instance for it. The class instances would be cached, so you could store 'stuff' as instance variables and they would persist during the cache lifetime. Templates that are cached: icon and color rules, emblems, composites, and template searches. The caches are cleared whenever something changes the database, or in the template search case on every new search.

+1. I even use the instances of the current formatter functions to persist things.

Originally Posted by chaley

This method avoids creating a new function syntax, which makes things easier both to implement and to document. It loses being able to intermix chaley-language templates and python-language templates, but that is probably a small loss.

That is where feedback from others would help. I see un_pogaz has already chimed in.

Originally Posted by chaley

I would need to make the template editor handle these, probably by removing all syntactic highlighting in "python:" mode.

Yes. I also don't see a need to ever re-implement syntax highlighting for python in the template editor. People can simply use their favorite editor/IDE for that, and copy paste into the template editor.

Originally Posted by chaley

Finally, this method removes the need for dict_... methods. If you know what a dict is and how to use them then you probably can write in python, or at least understand examples well enough to change them.

I would say so. Plus all the other features that comes for free with python.

Originally Posted by un_pogaz

Wow, that could be very powerful indeed. But it would be necessary to define its "main function" as well as its arguments, a bit like it is the case for template functions. This would allow to create "template function for single use".

python:
def main(mi):
    ...
    your python code
    ...

Note that since python accepts local definitions, we can all write in main()

The idea is that the code will be wrapped into the function definition — by simply replacing python: with the def whatever() — This way we are not tied to a pre-defined signature. Edit: The call to the function is done automatically as well.

[chaley]

This is getting far into implementation details, but doing so is useful for me because I must think about it so thanks.

Originally Posted by un_pogaz

I agree. As for pydef it's clear, it's a locally defined function as it already exists in python.
But for the class... I have a hard time seeing how we go from that:

class x(object):
    def do_it(mi, locals_dict, a, b, ...)
        python code goes here
        the code must return a string or None

to this:

    python_functions['x'].do_it(mi, locals, a, b, c)

It is a class because that is the easiest way to compile and save a python function (method). python "class x(object):" definitions are in effect executable statements. When executed they define a class type and store it in a local dictionary that is visible. Using this class type you can instantiate the class giving a runtime instance as long as you know the signature for __init__(), which we do because we wrote it. You can then execute a method in that instance as long as you know the name and signature. That is why template functions must always have the signature

evaluate(self, formatter, kwargs, mi, locals, your parameters)

The template processor knows how to call this method.

In the case of a "pydef", if a template calls

some_pydef_function(x)

then the template processor would know to really call

stored_class_instance.do_it(standard_argument_profile, x)

Originally Posted by un_pogaz

Wow, that could be very powerful indeed. But it would be necessary to define its "main function" as well as its arguments, a bit like it is the case for template functions. This would allow to create "template function for single use".

python:
def main(mi):
    ...
    your python code
    ...

Note that since python accepts local definitions, we can all write in main()

It would work like today's python template functions, but in the context of a standalone template. Your code defines a standard entry point such as "evaluate" with the associated code. The template processor wraps that in a class definition and executes that definition, getting a runtime class type. It calls "__init__ in that class to get an actual class instance, and stores that instance in its cache to avoid compiling it again unless something changes.

Whether the class instance is created or comes from the cache, the template processor calls

stored_class_instance.evaluate(standard_argument_profile)

I don't see any need for a more flexible argument scheme. These methods are only called by the template processor, which by definition must know in advance the arguments to provide. This is the same reason that template functions must have a known argument signature.

NB: you can define other "helper" methods that live in the class, as in this example of a python template function.

def evaluate(self, formatter, kwargs, mi, locals, val):
		return val.encode().hex() + self.afunc()

def afunc(self):
		return 'aaa'

Originally Posted by un_pogaz

Wait wait wait.
2 seconds. Pause.
Tell me if I'm wrong, but maybe I have a better idea.

Currently, the 'Template functions' and the 'Stored template' are saved a the same place in the DB table "preferences>user_template_functions". In order to be able to sort out what is what, Calibre uses the number of arguments of these: 0 is a 'Stored template' ; 1+ is 'Template functions'.
That's why you can't define 'Template functions' with 0 arguments.
I have the impression that all this thing of python: is a way to circumvent this limitation.
What if we treat the cause rather than the symptoms?

That isn't how it works. The arg count is irrelevant. If the text of the stored 'whatever' starts with 'def' then it is a python function. If it starts with 'program' then it is a template function. Anything else is illegal.

The requirement that template functions have either -1 (unspecified) or >= 1 arguments is there to help someone avoid writing functions that can't be called in single function mode. I have considered removing that requirement. For example, this template function with no arguments works.

def evaluate(self, formatter, kwargs, mi, locals):
	return 'aaa'

Originally Posted by un_pogaz

What I suggest is that we revise the way we save this in the DB.
Let's save separately 'Template functions' and 'Stored template' in their own preferences entry. Thus we can define functions with 0 arguments. We just need to do a version check of the DB and a data migration.

I don't see the need for this. You can already define functions with zero arguments using -1. If I remove the restriction then you would be able to use an arg count of zero without requiring any migration.

So, I agree that this poses a problem of the assenting compatibility (you can't load a +6.7 DB with 5.0 Calibre without a nice mess in your template), but I think, perhaps, that it will be much easier to implement this than python: with the different syntax highlighting changes it announces.
But it would be necessary to see with Kovid.

One of the reasons I am leaning to the "python:" mode is that it is completely compatible with existing templates and uses existing tested-for-years code. Nothing changes for stored templates and template functions. No migration is required.

[chaley]

Originally Posted by capink

If the simpler form is not able to do all of the above, it leaves it with only one tiny advantage; the ability to add more arguments to the function definition in the future if a need for this arises.

I think the chances of using a python template in an EvalFormatter context are zero. The EvalFormatter isn't given an mi but instead a key:value dict.

The idea is that the code will be wrapped into the function definition — by simply replacing python: with the def whatever() — This way we are not tied to a pre-defined signature.

This would require the template processor to examine the argument signature to figure out what arguments to pass, something like java introspection. By definition this requires knowledge of the semantics of the arguments. I don't see how to do this with reasonable effort.

Here is what I am thinking. You would create a python template that looks something like this:

python:
def evaluate(mi, db):
    # python code as needed, for example, this nonsensical program
    if len(mi.authors) > 5:
        return 'Too many authors!'
    # db is an instance of db.legacy().
    # You can get an instance of db.cache using the attribute db.new_api
    # You can get an instance of db.view using the attribute db.data.
    ids = db.data.search_getting_ids('series:true') 
    if len(ids) > 100:
        return 'Too many books in series!';
    return 'All OK :)'

The template processor would compile this into a class as described above, then execute the "evaluate" method with the correct parameters.

Hmmm ... I could get rid of "python:", instead using "def evaluate(mi, db):" as the key. I am not sure that would is better. It might have some future compatibilty advantages if we want to use a different argument profile. The formatter could directly determine which argument profile the python template is using. On the other hand, using "python:" could permit us to define multiple evaluator methods. I don't see a use for this, but ???. I lean toward "python:".

[chaley]

Originally Posted by capink

The idea is that the code will be wrapped into the function definition — by simply replacing python: with the def whatever() — This way we are not tied to a pre-defined signature. Edit: The call to the function is done automatically as well.

But unless we have some kind of introspection we are tied to a predefined signature. The template processor must know how to call the python template, which requires that it know the arguments and their semantics. Even an introspection technique would need knowledge about what the arguments mean, which without types means by their names.

Although directly using 'def' instead of putting the evaluation method in a class is possible (I think), doing so takes away useful stuff such as class instance variables (in effect globals) and helper methods.

[un_pogaz]

Originally Posted by kovidgoyal

I dont particularly mind having new versions create things that dont work in old versions, but I frown on the other way. That is things from old versions should preferably not stop working when going to a new version as much as possible.

Perfectly understandable.

Originally Posted by chaley

That isn't how it works. The arg count is irrelevant. If the text of the stored 'whatever' starts with 'def' then it is a python function. If it starts with 'program' then it is a template function. Anything else is illegal.

Okay, so its the code itself that defines its type, what about...

Originally Posted by chaley

The requirement that template functions have either -1 (unspecified) or >= 1 arguments is there to help someone avoid writing functions that can't be called in single function mode. I have considered removing that requirement. For example, this template function with no arguments works.

def evaluate(self, formatter, kwargs, mi, locals):
	return 'aaa'

I don't see the need for this. You can already define functions with zero arguments using -1. If I remove the restriction then you would be able to use an arg count of zero without requiring any migration.

Yes, removing this restriction would be nice.
But as there is a workaround easy to implement, detail.
Thanks

But I'm really curious to understand why it should be avoided to create a 0 arg function by default, what would be the risks and problems?
It seems to me absurdly arbitrary , hence my theory 0 'Stored template' / +1 'function', I'm trying to find a way to make sense of it.

[chaley]

Originally Posted by un_pogaz

But I'm really curious to understand why it should be avoided to create a 0 arg function by default, what would be the risks and problems?
It seems to me absurdly arbitrary , hence my theory 0 'Stored template' / +1 'function', I'm trying to find a way to make sense of it.

First, remember that template functions have been around for some 10 years. They were 'invented' when Single Function Mode was the normal way to write a template. When an SFM template calls a template function it *always* passes at least one argument, the value of the cell. The restriction was put in to give a big hint to the function developer that usage in SFM must be considered.

Skip forward some 5 years when General Program Mode has been enhanced enough where it was much more widely used. At this point zero-argument functions were quite acceptable. I never got around to removing the restriction because all the developer had to do was set arg_count to -1.

I haven't decided whether to remove it or make it a warning. I am leaning toward a warning with a "Don't show again" checkbox. Assuming I get around to doing it.

[chaley]

I did a proof of concept implementation using the code from the template functions implementation. It works.

Given the following template:

python:
def evaluate(self, mi, db):
    print(mi.authors)
    s = self.get_default_if_none(mi, 'series', '**no series**')
    return s + ':::' + str(self.get_default_if_none(mi, '#myint', 999))

def get_default_if_none(self, mi, field, default):
    v = mi.get(field, None)
    return default if v is None else v

The screen capture shows what I get in the template tester. The template tester still highlights strings and matching parentheses.

This isn't close to done. For example it doesn't include caching, error handling, or pulling the db out of the mi instance.