python templates (was possible template functions)

[chaley]

Originally Posted by capink

Can be useful for also defining convenience methods in the future, that simplifies repetitive tasks, and can be called using the ctx.convenience_method()

True. For example I frequently write the same code to deal with metadata that is None vs a non-existent attribute. For example, mi.get('#custcol', 'foo') will never return 'foo' if '#custcol' exists. If you want a default for an empty column then you must do something like

    'foo' if mi.get('#custcol') is None else mi.get('#custcol')

One way is a helper method:

def get_default_if_none(book, field, default):
	v = book.get(field, None)
	return default if v is None else v

[chaley]

The 'context' changes are in calibre source.

[chaley]

@capink: do you see a use for constructing your own python formatter context object and passing it to the formatter? One I thought of is what you mentioned in the development thread: adding "helper methods" to the context.

The mechanism would work like globals. If an instance isn't provided (a named parameter) then the formatter would construct one. If one is provided then it will be used, with the lifetime controlled by you.

[capink]

Originally Posted by chaley

@capink: do you see a use for constructing your own python formatter context object and passing it to the formatter? One I thought of is what you mentioned in the development thread: adding "helper methods" to the context.

The mechanism would work like globals. If an instance isn't provided (a named parameter) then the formatter would construct one. If one is provided then it will be used, with the lifetime controlled by you.

Yes. Actually I was thinking about this. Before you mentioned this I thought of two ways:

• Use the chain object that I would smuggle inside the globals dict for defining these methods. Your proposal is much cleaner way of doing this.
• Monkey patching calibre's context class. I am not sure how clean/desirable this would be. But if offers me one thing none of the other options do; the ability to use any plugin defined helper functions in calibre's template search.

[chaley]

Moderator Notice
Moved back to the development forum

Originally Posted by capink

[*]Monkey patching calibre's context class. I am not sure how clean/desirable this would be. But if offers me one thing none of the other options do; the ability to use any plugin defined helper functions in calibre's template search.

I am not sure I understand. Are you suggesting making Action Chains helper functions available to python templates, GPM templates, or both? Mentioning the context class tells me it is for PTM templates.

If you mean both then the only mechanism that exists is the stored templates / template functions. You could add to the base functions but we would need to work out how to control their lifetime.

Assuming you mean PTM templates, I think I see the problem you are solving. The context for template searches is the calibre base context, not one you created, so you have no way to put your helper functions in it. However, there is another way consistent with python and calibre: put these helper functions into their own module or own class and the PTM writer can "import" them. Doing it this way avoids monkey patching and all the lifetime and name collision problems that come with it. It also handles the problem that action chains might not be installed because the user writing the import can ensure that it is. Something like this:

python:

from calibre_plugins.action_chains.templates.functions import SelectionCount

def evaluate(book, ctx):
	return ''

[capink]

Lol. It is funny how simple the solution is. I wanted it for PTM, but I've been looking for solution through the lens of GPM, because that what I was struggling with for the past couple years. Thanks.

The whole PTM thing is going to open up a lot of possibilities. e.g. combining PTM with template search, you can use data from one library (authors library), to search for books in another. Which brings up the question, now that this is not implemented as a method, how can I persist the data, say for the duration of template search.

Edit: The context object or the globals dict can be used for this.

[chaley]

Originally Posted by capink

The whole PTM thing is going to open up a lot of possibilities. e.g. combining PTM with template search, you can use data from one library (authors library), to search for books in another. Which brings up the question, now that this is not implemented as a method, how can I persist the data, say for the duration of template search.

Edit: The context object or the globals dict can be used for this.

The cached template can also be used for this if the lifetime works.

The cached compiled template's lifetime is

• for template searches, the duration of the search.
• For composite columns, until some metadata changes.
• For the template tester, one call
• For most everything else, one call

You can use these by defining attributes in the template at the outermost level.

You can also define persistent attributes in various ways then give a template access to the attributes in some way or another.

As for "other possibilities", yes. A PTM template can do a lot of things, for example interact with the GUI. It can search, add and remove category items, and modify metadata; then tell the GUI to refresh. I am not convinced this is a good idea for lots of reasons, but no matter what I think the templates can do it because they have access to the calibre API,

[chaley]

I added the ability to use custom python context classes. The change is in calibre source.

How to use:

1. Create a subclass of utils.formatter.PythonTemplateContext. Add whatever you want to it.
2. Pass an instance of your class to the formatter using
   

   python_context_object=whatever

   as in
   

   formatter.safe_format(template, {}, 'TEMPLATE ERROR', mi,
                                     python_context_object=CustomContext())

3. If you want something to happen when the object is used, override
   

   set_values(**kwargs)

   This method is called just before executing the template, providing the known attributes db, globals, and arguments. Be sure to call the superclass method.

Here is an example test of the new feature:

        template = '''python:
def evaluate(book, ctx):
    tags = ctx.db.new_api.all_field_names('tags')
    return ','.join(list(ctx.helper_function(tags)))
'''
        from calibre.utils.formatter import PythonTemplateContext
        class CustomContext(PythonTemplateContext):
            def helper_function(self, arg):
                s = set(arg)
                s.add('helper called')
                return s

        v = formatter.safe_format(template, {}, 'TEMPLATE ERROR', mi,
                                  python_context_object=CustomContext())
        self.assertEqual(set(v.split(',')), {'Tag One', 'News', 'Tag Two','helper called'})

[capink]

This will be very useful for me.

[un_pogaz]

Originally Posted by chaley

All this opens the question "What is the difference between a python template and a python formatter function? Good question. Formatter functions can "see" into the formatter and easily use existing formatter builtin functions, but templates cannot.

Actually...
This is no longer the case. At least not on my repositories.
I have created a system that allows to easily call and use the functions currently loaded in the formatter.
Voilà, voilà.

@chaley, @kovidgoyal, could you check this out, thanks. I don't want to make a pull request right now because it's a big change for my knowledge and I don't want to make a big problem. (also if is need to improve the code doc)

So, as it works:
This adds 2 attributes at th context: .formatter and .funcs.
.formatter is the current TemplateFormatter. It's good, but it's not best.
There .funcs enters on stage (that's where the fun start). He will which allows you to easily call any function currently loaded in the TemplateFormatter. You use them like any other function just with the name of the one you want as an attribute (plus an _ 'underscore' at the end to avoid conflicts with Python keywords). And you don't need to handle the special arguments of these (formatter, kwargs, mi, locals | book, context), it does it all by itself, you just need to provide the yours real arguments to your function.

Example: I want to convert formats_sizes() to human_readable(), easy:

python:
def evaluate(book, context):
	formats = {}
	for f in context.funcs.formats_sizes_().split(','):
		f = f.strip().split(':', 1)
		formats[f[0].lower()] = context.funcs.human_readable_(f[1])
	
	return ','.join([k+':'+v for k,v in formats.items()])

That's it, it's all. No need to think or to bother to get the code of the functions already present in calibre, let's use them directly.

And of course, user functions and stored templates can be used in the same way.
(however, what was less fun was to return an exception that made sense)

[chaley]

Originally Posted by un_pogaz

Actually...
This is no longer the case. At least not on my repositories.
I have created a system that allows to easily call and use the functions currently loaded in the formatter.
Voilà, voilà.

Before I spend time on this I need to understand what problem you are solving.

You are adding the ability to call builtin formatter functions from python templates. Calling these functions requires you to accept that the functions return strings, even when that doesn't make sense given the underlying data. For example, by looking at the source for formats_sizes() and human_readable() (easily available), assuming I know python I see how to call the underlying db methods. Using that knowledge I can write your template as

python:
from calibre import human_readable

def evaluate(book, context):
    return ','.join([k+ ':' + human_readable(v['size']) 
                     for k,v in book.get('format_metadata', {}).items()])

which avoids the conversion to/from strings while building the base data.

The same function could be written in TPM as

program:
	sizes = formats_sizes();
	result = '';
	for fmt in sizes:
		list_split(fmt, ':', 'v');
		result = list_join(',', result, ',', v_0 & ':' & human_readable(v_1), ',')
	rof;
	result

The 'for' statement will do the same split() that your PTM template is doing. The list construction using list_join() is less efficient than your sample template, but not outrageously so.

----

Another question: TPM is quite efficient because it is 'compiled' into an abstract syntax tree. A python template calling a sequence of template functions won't be a lot faster than a GPM template doing the same thing. Many of the inefficiencies of TPM come from list handing, which your proposal doesn't address. What do I gain by using PTM? Does the gain justify the ongoing maintenance cost?

Another issue, probably not important: the arguments(), globals(), and set_globals() functions are implemented directly in the formatter. A user cannot call the functions in .funcs.

[un_pogaz]

Originally Posted by chaley

You are adding the ability to call builtin formatter functions from python templates. Calling these functions requires you to accept that the functions return strings, even when that doesn't make sense given the underlying data.

It's a pity but it's "normal" because you use the function as Calibre calls it itself. It's a point to underline in their doc and usage.

Originally Posted by chaley

For example, by looking at the source for formats_sizes() and human_readable() (easily available), assuming I know python I see how to call the underlying db methods. Using that knowledge I can write your template as

python:
from calibre import human_readable

def evaluate(book, context):
    return ','.join([k+ ':' + human_readable(v['size']) 
                     for k,v in book.get('format_metadata', {}).items()])

which avoids the conversion to/from strings while building the base data.

The same function could be written in TPM as

program:
	sizes = formats_sizes();
	result = '';
	for fmt in sizes:
		list_split(fmt, ':', 'v');
		result = list_join(',', result, ',', v_0 & ':' & human_readable(v_1), ',')
	rof;
	result

Yes, of course you could do better if you decided to import the code if you wanted to. My example is simple, but it is a basis for discussion. Other buildin functions can be more cumbersome to import.
The main goal is to have a "zero brain" import of the buildin function. It's not the "best" (I agree) but it's better than not having them at all.

Originally Posted by chaley

Another issue, probably not important: the arguments(), globals(), and set_globals() functions are implemented directly in the formatter. A user cannot call the functions in .funcs.

Oh, I'll look into it.

Originally Posted by chaley

Does the gain justify the ongoing maintenance cost?

The gain is very low, I agree again, but the maintenance cost too because the attributes are automatically created by the initialization of the class on the basis of the functions inside the formatter you used to initialize it (even better, I have push a version that uses __getattribute__, so it is dynamic at the change in formatter.funcs)

Only some special functions (like the one you told me) need to be treated. It's a bet that there won't be another one in the future, but it doesn't represent a big problem (I work well good of the one you pointed out)

[chaley]

Originally Posted by un_pogaz

It's a pity but it's "normal" because you use the function as Calibre calls it itself. It's a point to underline in their doc and usage.

Je comprends la française. For conversations between us feel free to write in French it if you think you will be clearer. I can write in French if needed, but my grammar and vocabulary aren't good. Des faux amis me donne beaucoup de problèmes.

I was just writing a post to say that overnight I found convincing arguments for this ability.

• There are people with a library of stored functions and/or templates. This lets the person use their library, converting to python as and when needed.
• Some of the built-in functions are complicated, for example list_re_group(). No need to bother to rewrite these.
• Some built-in functions would be hard to rewrite, for example template().

I looked at your code and it is quite elegant. Some questions:

• Why use object.__setattr__() in FormatterFuncsCaller __init__ instead of

  self.formatter = formatter

• What is the point of the recursive get_attribute() in FormatterFuncsCaller (line 892)? As far as I can see an instance of FormatterFuncsCaller doesn't have any attributes other than .formatter.
  EDIT: Never mind. It must be there to return .formatter and .call, at least
• What about not requiring the trailing '_' if there is no collision with a python keyword? FormatterFuncsCaller.__get_attribute__() could check the name with and without the underscore.
• Do we want the 'hidden' attributes .formatter and .funcs in PythonTemplateContext.attrs_set? I think these aren't something the user should use or see. No need for them to be in the output of __str__().
• This error exception should be simpler:
  

  EXCEPTION: Error in function evaluate on line 4 : AttributeError - 'FormatterFuncsCaller' object has no attribute 'a_python_template'

  Perhaps something like this:
  

  EXCEPTION: Error in function evaluate on line 4 : AttributeError - no function 'a_python_template' exists

• I am a little concerned about recursion in processing templates, but that can happen today. Oh well.

There are a few typos in comments, and a few cases where line spacing doesn't follow Kovid's rules. I can clean those up once we are satisfied with the implementation.

[un_pogaz]

Originally Posted by chaley

Why use object.__setattr__() in FormatterFuncsCaller __init__ instead of

self.formatter = formatter

In a first implementation, I removed/invalidated __setattr__ from FormatterFuncsCaller.

Originally Posted by chaley

What is the point of the recursive get_attribute() in FormatterFuncsCaller (line 892)? As far as I can see an instance of FormatterFuncsCaller doesn't have any attributes other than .formatter.

__dir__, __dict__ and other internal special function.
As I write this reply I am beginning to see your point and I have an idea, I will see.

Originally Posted by chaley

What about not requiring the trailing '_' if there is no collision with a python keyword? FormatterFuncsCaller.__get_attribute__() could check the name with and without the underscore.

Now that you mention it, Yep. The ability to call with '_' at the end will remain for all functions as a backup. Hum, I'll have to think about a solution to make the call priority right, even if putting '_' at the end of its template/function is unusual.

Originally Posted by chaley

Do we want the 'hidden' attributes .formatter and .funcs in PythonTemplateContext.attrs_set? I think these aren't something the user should use or see. No need for them to be in the output of __str__().

Hidden .formatter, Yes, there is no need to know that it is there. Just don't forget to write it in the manual as an indication (basically, we don't need it, but know current formatter is very "contextual").
.funcs less sure. I plan to write a doc about it once the feature is stable.

Originally Posted by chaley

This error exception should be simpler:
EXCEPTION: Error in function evaluate on line 4 : AttributeError - 'FormatterFuncsCaller' object has no attribute 'a_python_template'

Perhaps something like this:
EXCEPTION: Error in function evaluate on line 4 : AttributeError - no function 'a_python_template' exists

Noted

Originally Posted by chaley

I am a little concerned about recursion in processing templates, but that can happen today. Oh well.

Oh, yes, sure .funcs is a good way to blow up the stack
But like in any other program. If you're in python mode, it's either a notion you know, or you learn it very quickly . The main goal is the buildin and user function, therefore, little risk.
And then, a reccursive call, it is rarely done by "accidents".

EDIT: And a found a way to clean the list of native attribute. Now, all is inside __get_attribute__.
A also fix globals(), arguments() (which returns dict) and set_globals(), character().
Push on my github

[capink]

Originally Posted by chaley

I added the ability to use custom python context classes.

I am able to pass my custom python context. Problem is, I need to be able to pass it to the template dialog as well (like we do with global_vars), otherwise, any helper functions or attributes will fail in the template dialog.

Unfortunately, I didn't have the time to test this before the weekend.