python templates (was possible template functions)

[capink]

I am getting the following error while trying to show the template tester:

calibre, version 6.6.1
ERROR: Unhandled exception: <b>AttributeError</b>:'BuiltinAdd' object has no attribute 'is_python'

calibre 6.6.1*  embedded-python: True
Linux-5.15.0-48-generic-x86_64-with-glibc2.35 Linux ('64bit', 'ELF')
('Linux', '5.15.0-48-generic', '#54-Ubuntu SMP Fri Aug 26 13:26:29 UTC 2022')
Python 3.10.1
Interface language: None
Successfully initialized third party plugins: Category Tags (0, 2, 7) && Editor Chains (0, 5, 4) && Favourites Menu (1, 2, 0) && Find Duplicates (1, 9, 7) && Goodreads (1, 7, 0) && Import List (1, 8, 4) && Last Modified (0, 8, 4) && Manage Series (1, 4, 0) && Reading List (1, 14, 0) && View Manager (1, 9, 0)
Traceback (most recent call last):
  File "/home/______/calibre-6.6.1/src/calibre/gui2/actions/show_template_tester.py", line 47, in show_template_editor
    t = TemplateDialog(self.gui, self.previous_text,
  File "/home/______/calibre-6.6.1/src/calibre/gui2/dialogs/template_dialog.py", line 365, in __init__
    self.function_type_string(f, longform=False)), f)
  File "/home/______/calibre-6.6.1/src/calibre/gui2/dialogs/template_dialog.py", line 710, in function_type_string
    if self.all_functions[name].is_python:
AttributeError: 'BuiltinAdd' object has no attribute 'is_python'

I removed the template_dialog_ui.py so that the new file is compiled. Is there something else I should be doing?

[chaley]

Originally Posted by capink

I am getting the following error while trying to show the template tester:

This is my fault. I put the files in the wrong subfolders. What should happen:

• template_functions.py ==> gui2.preferences. Remove it from gui2.dialogs
• template_dialog.* ==> gui2.dialogs. Remove them from gui2.preferences
• The two files in utils are correct.

A corrected archive is attached.

[capink]

OK. It is working now. So far, so good

[un_pogaz]

Oh, arguments, nice, tanks.
Work good. Well, I'll see what I can do but it looks incredible.

Frankly, you make black magic with all that thing of template language.

[un_pogaz]

And after a night's sleep, I just realized the inconsistency of using the arguments for python:.

*inhale, sigh*

In fact, I think I even misunderstood the purpose of python template (it's an alternative form to Stored, not a 3rd option totally inedite).

If you think that deleting them would improve the code, go ahead.


Also, I was going to say that syntax highlighting is possible, since "Action Chains" does it in its module editor... but it uses something different (TextEdit from calibre.gui2.tweak_book.editor.text), so no, crap.

At least, maybe highlight python: to identify what we are in.
... Maybe an idea, I'm looking at it.

EDIT: And, it works.

Spoiler:

Well, really small bug, the highlighting of the key words of the modes is made at the beginning of each new line whereas normally it should be exclusively at the beginning of the full text. Detail, really. I won't have to think about it here, I'll wait until it's pushed to your github or to the calibre github to take the time to make a clean commit (if possible).

[chaley]

Originally Posted by un_pogaz

And after a night's sleep, I just realized the inconsistency of using the arguments for python:.

*inhale, sigh*

In fact, I think I even misunderstood the purpose of python template (it's an alternative form to Stored, not a 3rd option totally inedite).

If you think that deleting them would improve the code, go ahead.

I will leave it. Reasoning:

1. We should have a way to save and reuse python templates. It could be done with files but that seems a bit crude.
2. "Stored templates" already exists, so why not reuse it?
3. Stored GPM templates are callable with arguments.
4. Therefore stored python templates should also be callable.

In addition there is a bona fide reason to have arguments in stored python templates: template searches. A person could write a template that does some sort of search, but with variants. Instead of duplicating the code, store the template and call it with arguments that tell it which variant to use.

All this opens the question "What is the difference between a python template and a python formatter function? Good question. Formatter functions can "see" into the formatter and easily use existing formatter builtin functions, but templates cannot. If I had thought of python templates years ago perhaps I wouldn't have built the user-defined function mechanism. But we have it now and it is used, so it can't go away or be changed.

Also, I was going to say that syntax highlighting is possible, since "Action Chains" does it in its module editor... but it uses something different (TextEdit from calibre.gui2.tweak_book.editor.text), so no, crap.

At least, maybe highlight python: to identify what we are in.
... Maybe an idea, I'm looking at it.

EDIT: And, it works.

Spoiler:

Well, really small bug, the highlighting of the key words of the modes is made at the beginning of each new line whereas normally it should be exclusively at the beginning of the full text. Detail, really. I won't have to think about it here, I'll wait until it's pushed to your github or to the calibre github to take the time to make a clean commit (if possible).

I had already taken the code to do python syntax highlighting from gui2.widgets.PythonHighlighter and integrated it into the highlighter used by the template dialog. It works quite well. The difficulty was dynamically switching between highlighters.

I didn't solve the problem of highlighting the mode indicator.

EDIT: Attached is template_dialog.py with python syntax highlighting.

[chaley]

The implementation is in my github repo. I did a pull request and am waiting for feedback from Kovid.

[chaley]

Moderator Notice
Moved to the development forum

[un_pogaz]

Originally Posted by chaley

[*]Stored GPM templates are callable with arguments.

What? You can we pass arguments to the GPM?
*look and test*

program: arguments(arg0=0, arg1=1, arg2=2);
return strcat(arg0,' | ',arg1,' | ',arg2);

program: _test('gfgfgh', 456)

>> gfgfgh | 456 | 2

O_o HOLY SH**!!!!
But it's killer!! It changes so much all... Oh god.

*look the manual* Ugh, I missed it. Well, I need to re-read this page again.

Originally Posted by chaley

The implementation is in my github repo. I did a pull request and am waiting for feedback from Kovid.

Nice, a open pull request for improve the highlight and some fix.

[chaley]

@un_pogaz: thanks for the fixes.

I pushed changes to my repo to solve the "python:|program:" not on line one problem.

[chaley]

@kovid has accepted the changes. However, he made a good suggestion that I will implement, which changes the signature of the evaluate function. He suggests:

python:
def evaluate(book, ctx):

where 'ctx' has attributes (not dictionary keys) for all the other 'things' that can be passed, for example ctx.db. This is as extensible as **kwargs but is simpler to use and to understand.

The context object is an instance of class defined in calibre.utils.formatter, which I have currently defined as:

class PythonTemplateContext(object):

    def __init__(self, **kwargs):
        # Set attributes we already know must exist.
        self.db = None
        self.arguments = None
        self.globals = None

        # Create/set attributes from the named parameters. Doing it this way we
        # aren't required to change the signature of __init__ if we add
        # attributes in the future. However, if a user depends upon the existence
        # of some attribute and the context creator doesn't supply it then the
        # user will get an AttributeError exception.
        for k,v in kwargs.items():
            setattr(self, k, v)

I will document it in the comments that the template editor can paste into the template, and in the 'real' documentation.

EDIT: changes are in my repo and submitted to Kovid.

[chaley]

Here is the example python template from earlier in the thread, updated to use the new signature.

python:
def evaluate(book, context):
	if book.series is None:
		return ''
	ans = set()
	db = context.db
	for id_ in db.search_getting_ids(f'series:"={book.series}"', ''):
		ans.update([v.strip() for v in db.new_api.field_for('author_sort', id_).split('&')])
	return ', '.join(v.replace(',', ';') for v in sorted(ans))

[un_pogaz]

Yep. Sounds much better.

However, not sure if passing the named argument as an attribute is the best. I'd see more of a ctx.kwargs attribute to being able to process it as you wants.
Beyond the minor risk of collision in the future, using attributes means using getattr(ctx, 'name', default) which seems less elegant than ctx.kwargs as a dict, which can be used in a more dynamic way.
Unless it is usual in python to pass kwargs arguments as attributes for a class.

[capink]

Originally Posted by chaley

@kovid has accepted the changes. However, he made a good suggestion that I will implement, which changes the signature of the evaluate function. He suggests:

python:
def evaluate(book, ctx):

where 'ctx' has attributes (not dictionary keys) for all the other 'things' that can be passed, for example ctx.db. This is as extensible as **kwargs but is simpler to use and to understand.

The context object is an instance of class defined in calibre.utils.formatter, which I have currently defined as:

class PythonTemplateContext(object):

    def __init__(self, **kwargs):
        # Set attributes we already know must exist.
        self.db = None
        self.arguments = None
        self.globals = None

        # Create/set attributes from the named parameters. Doing it this way we
        # aren't required to change the signature of __init__ if we add
        # attributes in the future. However, if a user depends upon the existence
        # of some attribute and the context creator doesn't supply it then the
        # user will get an AttributeError exception.
        for k,v in kwargs.items():
            setattr(self, k, v)

I will document it in the comments that the template editor can paste into the template, and in the 'real' documentation.

EDIT: changes are in my repo and submitted to Kovid.

Can be useful for also defining convenience methods in the future, that simplifies repetitive tasks, and can be called using the ctx.convenience_method()

[chaley]

Originally Posted by un_pogaz

However, not sure if passing the named argument as an attribute is the best. I'd see more of a ctx.kwargs attribute to being able to process it as you wants.
Beyond the minor risk of collision in the future, using attributes means using getattr(ctx, 'name', default) which seems less elegant than ctx.kwargs as a dict, which can be used in a more dynamic way.
Unless it is usual in python to pass kwargs arguments as attributes for a class.

The idea is to make it easy for the person writing a template. It is definitely easier to write and understand ctx.something than ctx.kwargs['something ']. Also, it isn't obvious what **kwargs means even to seasoned programmers.

I defined PythonTemplateContext the way I did because I am worried that someone might define a context outside of base calibre (where I can't see it). I don't know why someone would do that, but it is possible. That is why PythonTemplateContext.__init__() uses **kwargs. If and when we add items to the context we pass them to PythonTemplateContext.__init__() as named parameters that are converted to attributes by __init__. This way adding new attributes to the context object won't break the plugin code because __init__'s signature doesn't change. Of course the new attributes won't be there, but that might be OK for the plugin.

I put in ctx.attributes() so someone could 'inspect' what attributes are in the context object.