10-07-2022, 08:36 AM | #1 |
Grand Sorcerer
Posts: 11,765
Karma: 7029857
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
python templates (was possible template functions)
EDIT: This thread started as a proposition for some new template functions but has morphed into a discussion of python templates. Most of what is said in this first post is no longer being considered.
------ Several times over the last years I have wanted to have 'real' python-like dictionaries in the template language. By that I mean a variable that holds "key:value" pairs, where both key and value are arbitrary strings. I am thinking about adding this support. Before I spend the time, I am asking "Would anyone else use this?" The problem: how to do it that makes sense in the context of the template language. What I am considering: a set of "dict_...()" functions as follows. In all cases the parameters are strings. The function explanation code uses python syntax.
Code:
program: # Get the already-computed dict if it exists d = globals(series_authors); if d then dict_from_string('series_authors', d) else # Compute the dict the first time through and save it as a global series = book_values('series', 'series:True', ':::', 0); for s in series separator ':::': authors = book_values('authors', 'series:"""=' & a & '"""', '&', 0); for a in authors separator '&': dict_set('series_authors', s, list_union(dict_get('series_authors'), a, '&')) rof rof; set_globals(series_authors=dict_to_string('series_authors) fi; # At this point we have a dict keyed by series containing a list of authors of that series. # Check if the current book has a series, and if so does it have more than one author. # Return the series name if it does, '' if it doesn't. authors = dict_get('series_authors', $series) if list_count(authors, '&') ># 1 then return $series else return '' fi You might ask "How does this differ from using select() on identfier-like strings? Some of the differences:
Last edited by chaley; 10-08-2022 at 08:56 AM. |
10-07-2022, 10:16 AM | #2 |
Wizard
Posts: 1,106
Karma: 1954138
Join Date: Aug 2015
Device: Kindle
|
The dict_get() and dict_set() looks like the best way to go about this. And yes they would be useful.
However, this brings up something I've thinking about for a while now; would it possible to add another mode called python mode to the template language. I don't know how easy to implement this would be. The idea is that it passes mi object were we can process it directly. It would be more handy than defining a template function in python — for every use once scenario — and calling it from other modes. Of-course, if the implementation involves more than just a simple pass-through, it would not be worth the effort. There are other template things that are only tangentially related to your proposal, so I will raise them in the template thread instead. |
Advert | |
|
10-07-2022, 11:58 AM | #3 | |
Grand Sorcerer
Posts: 11,765
Karma: 7029857
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
Code:
pydef x(a, b, ...) python code goes here it returns some string fedyp #Yes, this is strange but consistent A pydef() would generate a python class that like this, where "locals" is the formatter local variable dict. Code:
class x(object): def do_it(mi, locals_dict, a, b, ...) python code goes here the code must return a string or None A call would look like every other function call, with mi and the locals dict automagically added to the argument list. The actual call to x by the template processor would be something like Code:
python_functions['x'].do_it(mi, locals, a, b, c) The next biggest problem is the db isn't always accessible to the python code. It is available for any mi created by cache.py.get_metadata(). In some cases the mi is built by hand or from a json dict, such as the templates in save_to_disk. These templates will fail if the python code attempts to use the db (cache.py.someting()) because mi._proxy_metadata will be None. Template functions such as book_values() have the same problem so we might be able to ignore it. I think the above would more useful if it integrated with the proposed dict_() functions. The python code could set or get values from the internal support dictionary passed as a parameter. Any dict_() call template code would operate on the same dict, permitting the two to interact. |
|
10-07-2022, 01:37 PM | #4 | ||||
Wizard
Posts: 1,106
Karma: 1954138
Join Date: Aug 2015
Device: Kindle
|
Quote:
Quote:
Quote:
Quote:
Also, feedback from others would help crystallize this whole thing even more. |
||||
10-08-2022, 04:47 AM | #5 | |
Grand Sorcerer
Posts: 11,765
Karma: 7029857
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
Code:
evaluate(mi, db) I would create a class instance for it. The class instances would be cached, so you could store 'stuff' as instance variables and they would persist during the cache lifetime. Templates that are cached: icon and color rules, emblems, composites, and template searches. The caches are cleared whenever something changes the database, or in the template search case on every new search. This method avoids creating a new function syntax, which makes things easier both to implement and to document. It loses being able to intermix chaley-language templates and python-language templates, but that is probably a small loss. I would need to make the template editor handle these, probably by removing all syntactic highlighting in "python:" mode. Finally, this method removes the need for dict_... methods. If you know what a dict is and how to use them then you probably can write in python, or at least understand examples well enough to change them. |
|
Advert | |
|
10-08-2022, 05:00 AM | #6 | ||
Chalut o/
Posts: 411
Karma: 145324
Join Date: Dec 2017
Device: Kobo
|
Quote:
But for the class... I have a hard time seeing how we go from that: Code:
class x(object): def do_it(mi, locals_dict, a, b, ...) python code goes here the code must return a string or None Code:
python_functions['x'].do_it(mi, locals, a, b, c) Quote:
Code:
python: def main(mi): ... your python code ... ... Thinking about it, I'm not sure that python: is really useful if you add the pydef. We just have to define: Code:
program: pydef __main__(mi) # start your python code ... your python code ... fedyp # end your python code main() # run your python code Last edited by un_pogaz; 10-08-2022 at 05:07 AM. |
||
10-08-2022, 05:58 AM | #7 |
Chalut o/
Posts: 411
Karma: 145324
Join Date: Dec 2017
Device: Kobo
|
Wait wait wait.
2 seconds. Pause. Tell me if I'm wrong, but maybe I have a better idea. Currently, the 'Template functions' and the 'Stored template' are saved a the same place in the DB table "preferences>user_template_functions". In order to be able to sort out what is what, Calibre uses the number of arguments of these: 0 is a 'Stored template' ; 1+ is 'Template functions'. That's why you can't define 'Template functions' with 0 arguments. I have the impression that all this thing of python: is a way to circumvent this limitation. What if we treat the cause rather than the symptoms? What I suggest is that we revise the way we save this in the DB. Let's save separately 'Template functions' and 'Stored template' in their own preferences entry. Thus we can define functions with 0 arguments. We just need to do a version check of the DB and a data migration. So, I agree that this poses a problem of the assenting compatibility (you can't load a +6.7 DB with 5.0 Calibre without a nice mess in your template), but I think, perhaps, that it will be much easier to implement this than python: with the different syntax highlighting changes it announces. But it would be necessary to see with Kovid. Last edited by un_pogaz; 10-08-2022 at 06:06 AM. |
10-08-2022, 06:59 AM | #8 |
creator of calibre
Posts: 43,958
Karma: 22669822
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
I dont particularly mind having new versions create things that dont work in old versions, but I frown on the other way. That is things from old versions should preferably not stop working when going to a new version as much as possible.
|
10-08-2022, 07:32 AM | #9 | ||||||
Wizard
Posts: 1,106
Karma: 1954138
Join Date: Aug 2015
Device: Kindle
|
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Last edited by capink; 10-08-2022 at 08:06 AM. |
||||||
10-08-2022, 07:53 AM | #10 | |||||
Grand Sorcerer
Posts: 11,765
Karma: 7029857
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
This is getting far into implementation details, but doing so is useful for me because I must think about it so thanks.
Quote:
Code:
evaluate(self, formatter, kwargs, mi, locals, your parameters) In the case of a "pydef", if a template calls Code:
some_pydef_function(x) Code:
stored_class_instance.do_it(standard_argument_profile, x) Quote:
Whether the class instance is created or comes from the cache, the template processor calls Code:
stored_class_instance.evaluate(standard_argument_profile) NB: you can define other "helper" methods that live in the class, as in this example of a python template function. Code:
def evaluate(self, formatter, kwargs, mi, locals, val): return val.encode().hex() + self.afunc() def afunc(self): return 'aaa' Quote:
The requirement that template functions have either -1 (unspecified) or >= 1 arguments is there to help someone avoid writing functions that can't be called in single function mode. I have considered removing that requirement. For example, this template function with no arguments works. Code:
def evaluate(self, formatter, kwargs, mi, locals): return 'aaa' Quote:
Quote:
|
|||||
10-08-2022, 08:27 AM | #11 | ||
Grand Sorcerer
Posts: 11,765
Karma: 7029857
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
Quote:
Here is what I am thinking. You would create a python template that looks something like this: Code:
python: def evaluate(mi, db): # python code as needed, for example, this nonsensical program if len(mi.authors) > 5: return 'Too many authors!' # db is an instance of db.legacy(). # You can get an instance of db.cache using the attribute db.new_api # You can get an instance of db.view using the attribute db.data. ids = db.data.search_getting_ids('series:true') if len(ids) > 100: return 'Too many books in series!'; return 'All OK :)' Hmmm ... I could get rid of "python:", instead using "def evaluate(mi, db):" as the key. I am not sure that would is better. It might have some future compatibilty advantages if we want to use a different argument profile. The formatter could directly determine which argument profile the python template is using. On the other hand, using "python:" could permit us to define multiple evaluator methods. I don't see a use for this, but ???. I lean toward "python:". |
||
10-08-2022, 08:34 AM | #12 | |
Grand Sorcerer
Posts: 11,765
Karma: 7029857
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
Although directly using 'def' instead of putting the evaluation method in a class is possible (I think), doing so takes away useful stuff such as class instance variables (in effect globals) and helper methods. |
|
10-08-2022, 09:52 AM | #13 | |||
Chalut o/
Posts: 411
Karma: 145324
Join Date: Dec 2017
Device: Kobo
|
Quote:
Quote:
Quote:
But as there is a workaround easy to implement, detail. Thanks But I'm really curious to understand why it should be avoided to create a 0 arg function by default, what would be the risks and problems? It seems to me absurdly arbitrary , hence my theory 0 'Stored template' / +1 'function', I'm trying to find a way to make sense of it. Last edited by un_pogaz; 10-08-2022 at 09:56 AM. |
|||
10-08-2022, 10:22 AM | #14 | |
Grand Sorcerer
Posts: 11,765
Karma: 7029857
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
Skip forward some 5 years when General Program Mode has been enhanced enough where it was much more widely used. At this point zero-argument functions were quite acceptable. I never got around to removing the restriction because all the developer had to do was set arg_count to -1. I haven't decided whether to remove it or make it a warning. I am leaning toward a warning with a "Don't show again" checkbox. Assuming I get around to doing it. |
|
10-08-2022, 10:42 AM | #15 |
Grand Sorcerer
Posts: 11,765
Karma: 7029857
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Proof of concept
I did a proof of concept implementation using the code from the template functions implementation. It works.
Given the following template: Code:
python: def evaluate(self, mi, db): print(mi.authors) s = self.get_default_if_none(mi, 'series', '**no series**') return s + ':::' + str(self.get_default_if_none(mi, '#myint', 999)) def get_default_if_none(self, mi, field, default): v = mi.get(field, None) return default if v is None else v This isn't close to done. For example it doesn't include caching, error handling, or pulling the db out of the mi instance. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Python functions in database and calibre 5 | Terisa de morgan | Calibre | 7 | 09-27-2020 02:52 AM |
A little help with template functions in a composite column, please! | mopkin | Library Management | 2 | 11-05-2019 11:07 PM |
Using built-in template functions in a custom template function | ilovejedd | Library Management | 4 | 01-28-2018 12:20 PM |
Rules, templates, and functions for column coloring and composed icons | readin | Library Management | 7 | 08-11-2016 04:41 PM |
template: if one of the tag is something... maybe contains or in_list functions | fxp33 | Calibre | 4 | 07-19-2014 05:18 AM |