Running PTM/PythonTemplateContexts in Job, not GUI

[DaltonST]

I saved this PTM template in Preferences > Template Functions:

Spoiler:

daltonst1
python:
def evaluate(book,context):
text = context.globals['val']
text = text + "xxxxxxxxxxxx"
return text

The globals dict was of course empty since it gets filled at runtime, so an error message about the key, 'val', missing was spurious. I still was able to save & apply the PTM template.

I then ran the above PTM template in a plugin's Job using this code:

 

        funcobject = user_template_functions['daltonst1']
        func_name = funcobject.name
        formatter = TemplateFormatter()
        formatter.funcs.update(formatter_functions()._functions_from_library)
        ptc = PythonTemplateContext()
        ptc.funcs = formatter_functions()._functions_from_library
        if func_name in ptc.funcs:
            func = ptc.funcs[func_name]
        elif func_name in formatter.funcs:
            func = formatter.funcs[func_name]
        else:
            func = funcobject 
        ptc.func = func
        ptc.func_name = func_name
        ptc.name = func_name
        ptc.globals = {}
        ptc.globals['val'] = "ABCDEFGHIJK9999999999MNOPQRSTUVWXYZ"
        ptc.arguments = ""
        ptc.db = guidb
        mi = guidb.new_api.get_metadata(current_book)
        ptc.book = mi
        ptc.formatter = formatter
        formatter.python_context_object = ptc
        formatter.func = func
        formatter.func_name = func_name
        formatter.name = func_name
        formatter.db = guidb
        formatter.mi = mi
        formatter.book = mi
        formatter.kwargs = None
        ffc = FormatterFuncsCaller(formatter)
        call = ffc.__getattribute__(func_name)
        output_text = call(formatter.python_context_object)

The debug log for the above Job shows:

Spoiler:

daltonst1 (<StoredObjectType.StoredPythonTemplate: 3>, <calibre.utils.formatter_functions.FormatterUserFu nction object at 0x0000022E5FCF07C0>)

ERROR in call(formatter.python_context_object): Error in function daltonst1 :: Error in function evaluate on line 3 : KeyError - 'val'

I went back into where I saved this PTM template in Preferences > Template Functions to work on the same PTM template, but the GUI raised the error shown below, and would not let me back into Preferences > Template Functions to work on the PTM template to fix the error (Catch-22):

Spoiler:

calibre, version 6.10.0
ERROR: Unhandled exception: <b>AttributeError</b>:'list' object has no attribute 'object_type'

calibre 6.10 embedded-python: True
Windows-10-10.0.22621-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22621')
Python 3.10.1
Windows: ('10', '10.0.22621', 'SP0', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Audit Log (1, 0, 18) && Author Book Count (2, 2, 2) && Author Book Count Hierarchy (1, 2, 3) && CalibreSpy (1, 0, 90) && Consolidate All Library Metadata (2, 0, 43) && Drop Search Results (1, 0, 13) && English Noun Frequency (1, 0, 15) && Entities Manager (1, 0, 18) && Extract People Other Metadata (1, 0, 0) && Extract RIS Citations (1, 0, 5) && Favourites Menu (1, 3, 0) && Job Spy (1, 0, 197) && Library Codes (1, 0, 53) && Library Splitter (1, 0, 6) && Media File Importer (1, 0, 26) && MultiColumnSearch (1, 0, 95) && QuarantineAndScrub (3, 6, 119) && Save Composite Custom Columns (1, 1, 1) && View Manager (1, 10, 2) && Zotero Metadata Importer (1, 0, 77)
Traceback (most recent call last):
File "calibre\gui2\preferences\main.py", line 308, in show_plugin
File "calibre\customize\__init__.py", line 675, in create_widget
File "calibre\gui2\preferences\__init__.py", line 267, in __init__
File "calibre\gui2\preferences\template_functions_ui.py ", line 31, in setupUi
File "calibre\gui2\dialogs\template_dialog.py", line 1090, in __init__
File "calibre\gui2\dialogs\template_dialog.py", line 453, in __init__
File "calibre\gui2\dialogs\template_dialog.py", line 858, in function_type_string
AttributeError: 'list' object has no attribute 'object_type'

Since I could not use Preferences > Template Functions any more, at all, I went into metadata.db, table Preferences, key 'user_template_functions', and carved out the entry for my PTM template from the JSON:

Spoiler:

[
"daltonst1",
"",
0,
"python:\ndef evaluate(book,context):\n\ttext = context.globals['val']\n\ttext = text + \"xxxxxxxxxxxx\"\n\treturn text"
],

I then ran Calibre, and could finally enter into Preferences > Template Functions since the PTM template in question no longer existed.

----------------------------------------------------------------------------------------------

Question: Can the PTM template cache get poisoned when the PTM template is run "in batch" via a Job? If so, is there a way to clear it so the template_dialog for Preferences > Template Functions can always be successfully executed?

Question: How do I create a PTM template using a dict with keys that only exist at runtime?

Question: Is this syntax supported: text = context.globals['val'] ? If not, what is the proper way to accomplish passing a single string value to a PTM template at runtime?


Thank you.

Happy/Merry Christmas.


DaltonST

[chaley]

Originally Posted by DaltonST

Question: Can the PTM template cache get poisoned when the PTM template is run "in batch" via a Job? If so, is there a way to clear it so the template_dialog for Preferences > Template Functions can always be successfully executed?

Is it possible? Probably. The formatter is supposed to be thread safe, but the management of the formatter functions isn't. How to fix it? Restart calibre should do it, unless the poison has made it into the database. In your example it didn't seem like the database was poisoned as the template data you "carved out" seemed syntactically correct.

The error indicates that the name of a stored template changed from a string to a list. I don't see any way that this can happen in base calibre. It is hard to follow the effects of your code as it modifies internal data structures. At a minimum I suspect possible threading problems.

On this topic, I wonder why you are reaching so far into the guts of the formatter to run your template? It is very hard to predict what will happen, given that the state save/restore mechanism is being bypassed and base calibre data structures are being modified. Why not use safe_format(), which is how templates are supposed to be called? Or unsafe_format() for that matter?

If you want to call a stored template then call safe_format() with something like

program: stored_template_name()

passing any arguments you want, or passing arguments in globals.

Question: How do I create a PTM template using a dict with keys that only exist at runtime?

    context.globals.get('dict_name', {}).get('key_name', None)

One techique I use is to check if the key is in globals. If it isn't I do what is required to initialize globals with default values.

Question: Is this syntax supported: text = context.globals['val'] ? If not, what is the proper way to accomplish passing a single string value to a PTM template at runtime?

Yes, it is supported. However, the key 'val' must be in the dict. If it might not be then use (as above)

    text = context.globals.get('val', whatever_default_you_want)

That said, I wonder if globals is the right place to pass the values. If what is being passed is truly an argument then the template should look in arguments. This lets you write something like

program:
    some_python_template(some_argument)

The python template would check the length of arguments and do what is appropriate if the arguments don't exist or are the wrong type.

Happy/Merry Christmas.


DaltonST

Thank you, and I hope the same for you.

[chaley]

I don't know if this is of any use to you, but here is how the Action Chains plugin invokes templates, with hopes that @capink doesn't mind me copying his code here. I picked this because it uses all of the template types, attributes in the python context, plugin-defined template functions, and user-provided templates and functions.

The main invocation function:

from calibre.ebooks.metadata.book.formatter import SafeFormat

[...]

class ACTemplateContext(PythonTemplateContext):
    pass

def get_template_output(template, kwargs, TEMPLATE_ERROR, book, global_vars={},
                        template_functions=None, context_attrs={}):
    if not template_functions:
        template_functions = get_template_functions[0]
    try:
        # calibre >= 6.7.0. support for python template context object
        context = ACTemplateContext()
        context.set_values(**context_attrs)
        output = SafeFormat().safe_format(template, kwargs, TEMPLATE_ERROR, book,
                                          global_vars=global_vars,
                                          template_functions=template_functions,
                                          python_context_object=context)
    except:
        output = SafeFormat().safe_format(template, kwargs, TEMPLATE_ERROR, book,
                                          global_vars=global_vars,
                                          template_functions=template_functions)
    return output

The template functions are built by this function. Much of the complexity comes from the fact that Action Chains adds many new Formatter Functions and that the plugin users can define their own functions.

def get_template_functions(plugin_action):
    all_functions = OrderedDict()

    builtin_functions = OrderedDict()
    for cls in BUILTIN_FUNCTIONS:
        builtin_functions[cls.name] = cls

    imported_functions = get_imported_functions(plugin_action)

    user_functions = get_user_functions(plugin_action)
    
    action_functions = get_action_functions(plugin_action)
    
    calibre_funcs = formatter_functions().get_functions()

    # Note: imported functions can either be class or instantiated object
    for function_name, function_obj in imported_functions.items():
        # dont override builtin functions
        if function_name in builtin_functions.keys():
            continue
        try:
            if isinstance(function_obj, (TemplateFunction, FormatterFunction)):
                function = function_obj
            elif issubclass(function_obj, TemplateFunction):
                function = function_obj(plugin_action)
            elif issubclass(function_obj, FormatterFunction):
                function = function_obj()
            all_functions[function_name] = function
        except TypeError as e:
            # TypeError: issubclass() arg 1 must be a class
            import traceback
            if DEBUG:
                prints('Action Chains: Error intializing imported function: Un-reconized object: {}\n{}'.format(function_obj, traceback.format_exc()))            
        except Exception as e:
            import traceback
            if DEBUG:
                prints('Action Chains: Error intializing imported function: {}\n{}'.format(function_name, traceback.format_exc()))

    for function_name, cls in user_functions.items():
        try:
            function = cls(plugin_action)
            all_functions[function_name] = function
        except Exception as e:
            import traceback
            if DEBUG:
                prints('Action Chains: Error intializing user function: {}\n{}'.format(function_name, traceback.format_exc()))

    for function_name, function in action_functions.items():
        try:
            all_functions[function_name] = function
        except Exception as e:
            import traceback
            if DEBUG:
                prints('Action Chains: Error intializing action function: {}\n{}'.format(function_name, traceback.format_exc()))

    for name, func in calibre_funcs.items():
        all_functions[name] = func

    # we keep this last in case we want to override some calibre formatter functions
    for function_name, cls in builtin_functions.items():
        function = cls(plugin_action)
        all_functions[function_name] = function

    return all_functions, builtin_functions, user_functions, action_functions, imported_functions

[capink]

Originally Posted by chaley

I don't know if this is of any use to you, but here is how the Action Chains plugin invokes templates, with hopes that @capink doesn't mind me copying his code here.

Dear chaley,

The code is as much yours as it is mine. You are welcome to do with it as you please. I am grateful and indebted to you not just for your work on templates and other areas of calibre, but also for your kind support whenever we ask for it (and sometimes when we don't even ask).

Happy holidays and merry Christmas to you.