python templates (was possible template functions)

[capink]

Originally Posted by chaley

I think the chances of using a python template in an EvalFormatter context are zero. The EvalFormatter isn't given an mi but instead a key:value dict.
This would require the template processor to examine the argument signature to figure out what arguments to pass, something like java introspection. By definition this requires knowledge of the semantics of the arguments. I don't see how to do this with reasonable effort.

Here is what I am thinking. You would create a python template that looks something like this:

python:
def evaluate(mi, db):
    # python code as needed, for example, this nonsensical program
    if len(mi.authors) > 5:
        return 'Too many authors!'
    # db is an instance of db.legacy().
    # You can get an instance of db.cache using the attribute db.new_api
    # You can get an instance of db.view using the attribute db.data.
    ids = db.data.search_getting_ids('series:true') 
    if len(ids) > 100:
        return 'Too many books in series!';
    return 'All OK :)'

The template processor would compile this into a class as described above, then execute the "evaluate" method with the correct parameters.

That is perfectly OK for me. However two more suggestions:

• Add *args, **kwargs to the signature.
• Pass mi as more friendlier name like book. This way, the user can get metadata with book.title, book.authors, .... etc which is more intuitive than mi.title ....etc. Ofcourse for custom columns it would be book.get('#my_custom_column'). Although, if OK with Kovid, we can make the mi object subcriptable to be able to something like book['#my_custom_column']

Originally Posted by chaley

Hmmm ... I could get rid of "python:", instead using "def evaluate(mi, db):" as the key. I am not sure that would is better. It might have some future compatibilty advantages if we want to use a different argument profile. The formatter could directly determine which argument profile the python template is using. On the other hand, using "python:" could permit us to define multiple evaluator methods. I don't see a use for this, but ???. I lean toward "python:".

Future proofing is always a good idea.

[chaley]

Originally Posted by capink

That is perfectly OK for me. However two more suggestions:[*]Add *args, **kwargs to the signature.

Why? There is no way a user can pass extra arguments. Having these implies it is possible, which could lead to confusion and bug reports.

I think what you are looking for is future proofing and simplifying git forking. How about we make the protoype be

def evaluate(**arguments): # This could be **kwargs if that is better for some reason

The template processor would call the method using only keyword arguments, for example:

evaluate(book=book, db=db)

The evaluate() method would do something like

book = arguments['book']
db = arguments['db']

For me this is more clear than args[0] etc.

In the future, more named arguments could be added without breaking existing python templates.

Originally Posted by capink

[*]Pass mi as more friendlier name like book. This way, the user can get metadata with book.title, book.authors, .... etc which is more intuitive than mi.title ....etc.

That makes sense. It is named 'book' when passed to safe_format().

Originally Posted by capink

Of course for custom columns it would be book.get('#my_custom_column'). Although, if OK with Kovid, we can make the mi object subcriptable to be able to something like book['#my_custom_column']

I am not in favor of making mi subscriptable. Reasons:

• It could imply that the book object is really a dict, which it isn't.
• It requires every place that mi is built to implement subscripting. I know of at least 4 places: proxy_metadata, metadata.books, OPF processing, and save_to_disk. I am 90% sure there are more.
• Adding subscripting is a significant change that could break things. Is the risk worth it?
• The construction dict.get('key') is basic to python. It shouldn't be a surprise to the user.
• book.get('title') works, so the user doesn't need to worry about two access methods.

Originally Posted by capink

Also, we can still be tied with a signature, but not make the user type it every time, and go with python: alone without needing the def line.

This is possible but it makes indenting problematic. All defs must be at the outermost indent, while the code must be further indented. Helper functions must be indented at the same level as "def evaluate()".

Adding on to the above suggestion about **arguments, how about adding a line to the context menu of the template editor that inserts

python:
def evaluate(**arguments):
    book = arguments['book']
    db = arguments['db']

This is both a shortcut for typing and aids discoverability.

[capink]

Originally Posted by chaley

I am not in favor of making mi subscriptable. Reasons:

• It could imply that the book object is really a dict, which it isn't.
• It requires every place that mi is built to implement subscripting. I know of at least 4 places: proxy_metadata, metadata.books, OPF processing, and save_to_disk. I am 90% sure there are more.
• Adding subscripting is a significant change that could break things. Is the risk worth it?
• The construction dict.get('key') is basic to python. It shouldn't be a surprise to the user.
• book.get('title') works, so the user doesn't need to worry about two access methods.

I concur.

Originally Posted by chaley

Adding on to the above suggestion about **arguments, how about adding a line to the context menu of the template editor that inserts

python:
def evaluate(**arguments):
    book = arguments['book']
    db = arguments['db']

This is both a shortcut for typing and aids discoverability.

That is more elegant than what I was trying.

[capink]

Can we still access the globals dict with this mode?

[chaley]

Originally Posted by capink

Can we still access the globals dict with this mode?

I have wondered about that and haven't been able to find a use case that class instances (e.g., self.my_data) doesn't cover. Do you have one in mind, taking into consideration of the lifetime of of globals and the class instance? Is there something in action chains that would use it?

If you need it, adding it would trivial as a third argument.

[un_pogaz]

Originally Posted by chaley

python:
def evaluate(**arguments):
    book = arguments['book']
    db = arguments['db']

It's... a bit strange and unusual, but not bad and interesting (just something to learn)

But maybe it would still be great to put an *args.

We have **arguments (which we rename **data) which contains the basic data related to this book, or anything else we want to provide turnkey to the users.
And on the other side we have *args, which allows to pass arguments to the function during its use, to give it a little more versatility and less "single use". For *args, it's up to the user to program the analysis and the use of the arguments he passed to the function.

python_template('authors', 'title', 'tags')
python_template('#original_title', '#translator')

Same code, but different output because different input. And if there is a problem, it's because he mismanaged his *args.

Maybe.

Originally Posted by chaley

Adding on to the above suggestion about **arguments, how about adding a line to the context menu of the template editor that inserts

This is both a shortcut for typing and aids discoverability.

Yep. That would be great.

[capink]

Originally Posted by chaley

I have wondered about that and haven't been able to find a use case that class instances (e.g., self.my_data) doesn't cover. Do you have one in mind, taking into consideration of the lifetime of of globals and the class instance? Is there something in action chains that would use it?

If you need it, adding it would trivial as a third argument.

For example, I do pass event arguments to all the chains invoked by the event using the globals dict (event_args variable). How would I be able to do this self.my_data?

While at it, is there a way to pass the chain instance itself?

[chaley]

Originally Posted by un_pogaz

But maybe it would still be great to put an *args.

We have **arguments (which we rename **data) which contains the basic data related to this book, or anything else we want to provide turnkey to the users.
And on the other side we have *args, which allows to pass arguments to the function during its use, to give it a little more versatility and less "single use". For *args, it's up to the user to program the analysis and the use of the arguments he passed to the function.

python_template('authors', 'title', 'tags')
python_template('#original_title', '#translator')

Same code, but different output because different input. And if there is a problem, it's because he mismanaged his *args.

You can't call a python template with different arguments. Python templates are not callable by another template; they are callable only by the template processor using a fixed argument list. You can't save a python template as a "Stored template"; those must be GPM templates. If you want to store a callable python function usable by other templates then you define it as a "Template function", where you have control over the args list.

Said another way, a python template is like a GPM template used in an advanced icon rule. It doesn't have a name and is not callable from anywhere else.

[chaley]

Originally Posted by capink

For example, I do pass event arguments to all the chains invoked by the event using the globals dict (event_args variable). How would I be able to do this self.my_data?

Not easily. My bad -- I hadn't considered external data that the instance didn't create. I will add the globals dict as a third argument.

While at it, is there a way to pass the chain instance itself?

I am not sure what you mean or what problem you are solving, but if the globals dict is there can you do it with that? Perhaps using an 'odd' key?

[capink]

Originally Posted by chaley

I am not sure what you mean or what problem you are solving, but if the globals dict is there can you do it with that? Perhaps using an 'odd' key?

I was just not sure it is OK to add non string values to the globals dict. Otherwise, I'm fine with using the globals dict for this purpose. This would make my newly added "Run Python Code" action obsolete, and I can phase it out.

[chaley]

Originally Posted by capink

I was just not sure it is OK to add non string values to the globals dict. Otherwise, I'm fine with using the globals dict for this purpose. This would make my newly added "Run Python Code" action obsolete, and I can phase it out.

I thought I better test putting arbitrary things in globals.

This template works:

python:
def evaluate(self, **kwargs):
	mi = kwargs['book']
	db = kwargs['db']
	g = kwargs['globals']
	v = g.get('_mything', dict())
	v['someting'] = {1,2,3,4}
	g['_mything'] = v
	print(g)
	print(mi.authors)
	print('aaa', db.path(mi.id, index_is_id=True))
	s = self.get_default_if_none(mi, 'series', '**no series**')
    return s + ':::' + str(self.get_default_if_none(mi, '#myint', 999))

def get_default_if_none(self, mi, field, default):
	v = mi.get(field, None)
	return default if v is None else v

It prints

{'_mything': {'someting': {1, 2, 3, 4}}}
['The 10 Angry Men']
aaa The 10 Angry Men\_Unknown_ (1444)

The first line shows that globals['_mything'] contains a dict containing a set.

[chaley]

Thank you @capink & @un_pogaz for your comments. I think I have something that will satisfy you both.

What is in this implemention:

• A complete implementation, including caching
• It no longer uses a class, instead a standard function definition. The globals dict provides a better way to persist data than class attributes, and helper functions can still be defined as part of the template.
• In addition to standalone python templates you can create stored python templates. This raised the issue @un_pogaz was discussing: passing arguments to a python template. If you call a stored template (either GPM or python) from a GPM or SFM template then you can pass positional arguments. A standalone python template cannot be given args and the 'arguments' parameter is None.
• Python stored templates are callable in all template language modes.

Python templates have the following header that you can insert using a context menu line in the template editor.

python:
def evaluate(book, db, globals, arguments, **kwargs):
	# book is a calibre metadata object
	# db is a calibre legacy database object
	# globals is the template global variable dictionary
	# arguments is a list of arguments if the template is called by a GPM template, otherwise None
	# kwargs is a dictionary provided for future use

	# Python code goes here
	return 'a string'

As you can see I ended up using positional parameters for the arguments that won't change, and providing a kwargs dict for future proofing.

One sample template:

python:
def evaluate(book, db, globals, arguments, **kwargs):
    print(book.authors)
    print('aaa', db.path(book.id, index_is_id=True))
    s = get_default_if_none(book, 'series', '**no series**')
    return s + ':::' + str(get_default_if_none(book, '#myint', 999))

def get_default_if_none(book, field, default):
    v = book.get(field, None)
    return default if v is None else v

Here is a sample of using a stored python template. First the stored template:

python:
def evaluate(book, db, globals, arguments, **kwargs):
	if arguments:
		if len(arguments != 2):
			return 'incorrect number of arguments'
		return arguments[0] + ':::' + arguments[1]
	return 'no arguments provided'

Here is a screen capture of the stored template definition dialog.


The sample calling GPM template:

program:
	a_python_template_2_args(1, 'aaaa')

I know of one possible incompatibility. Adding python stored templates introduced a third stored object type. The three are user formatter functions, stored GPM templates, and stored python templates. I replaced the existing mechanism, an bool named is_python, with an enumeration. Code that cares about the stored template type must look at the 'object_type' attribute of the FormatterUserFunction instance. For compatibility, is_python is set to True for user formatter functions, false otherwise.

@capink: you use the FormatterFunction class in Action Chains and set is_python=True. This works with the new code to distinguish between python template functions and stored templates. However, if you need to distinguish between stored GPM templates and stored python templates then you must use the object_type attribute. If you want to store python templates then you must use the FormatterUserFunction class instead of the FormatterFunction class.

The current code is attached. There are 5 .py files. If either of you have time and energy to try it before I submit the changes to Kovid that would be great.

EDIT: I removed this zip file. It had the files in the wrong place. A new file is in this post.

[capink]

@chaley: thanks a lot for your efforts and patience. I am more than satisfied with your final implementation. I will test and report back.

Originally Posted by chaley

@capink: you use the FormatterFunction class in Action Chains and set is_python=True. This works with the new code to distinguish between python template functions and stored templates. However, if you need to distinguish between stored GPM templates and stored python templates then you must use the object_type attribute. If you want to store python templates then you must use the FormatterUserFunction class instead of the FormatterFunction class.

The current code is attached. There are 5 .py files. If either of you have time and energy to try it before I submit the changes to Kovid that would be great.

I will take these into consideration for my next version. Thanks again.

[chaley]

Another example python template, this time a bit more complicated and more useful. It produces a list of all the authors for a series. The list is stored in a composite column (#comp2) that shows in the book details "on separate lines", which requires the list be a comma-separated. To make that work the template converts commas in author names to semicolons then builds the comma-separated list. The authors are then sorted, which is why the template uses author_sort.

The template:

python:
def evaluate(book, db, globals, arguments, **kwargs):
	if book.series is None:
		return ''
	ans = set()
	for id_ in db.search_getting_ids(f'series:"={book.series}"', ''):
		ans.update([v.strip() for v in db.new_api.field_for('author_sort', id_).split('&')])
	return ', '.join(v.replace(',', ';') for v in sorted(ans))

The result in book details showing all authors for the series of the current book:

[chaley]

Originally Posted by capink

@chaley: thanks a lot for your efforts and patience. I am more than satisfied with your final implementation. I will test and report back.