Template language changes

[ownedbycats]

That will be useful. thank you

[chaley]

25 April 2022 (In calibre source)

• New template function
  

  list_join(with_separator, list1, separator1 [, list2, separator2]*)

  Returns a list made by joining the items in the source lists (list1 etc) using with_separator between the items in the result list. Items in each source list[123...] are separated by the associated separator[123...]. A list can contain zero values. It can be a field like publisher that is single-valued, effectively a one-item list. Duplicates are removed using a case-insensitive comparison. Items are returned in the order they appear in the source lists. If items on lists differ only in letter case then the last is used. All separators can be more than one character.
  
  Example:
  

  program:
    list_join(':@:', $authors, '&', $tags, ',')

  You can use list_join on the results of previous calls to list_join as follows:
  

  program:
    a = list_join(':@:', $authors, '&', $tags, ',');
    b = list_join(':@:', a, ':@:', $#genre, ',', $#people, '&', 'some value', ',')

  You can use expressions to generate a list. For example, assume you want items for authors and #genre, but with the genre changed to the word "Genre: " followed by the first letter of the genre, i.e. the genre "Fiction" becomes "Genre: F". The following will do that:
  

  program:
    list_join(':@:', $authors, '&', list_re($#genre, ',', '^(.).*$', 'Genre: \1'),  ',')

[ownedbycats]

What would be the difference between this and list_union? I believe the latter doesn't have the additional expressions, at least.

[chaley]

Originally Posted by ownedbycats

What would be the difference between this and list_union? I believe the latter doesn't have the additional expressions, at least.

Several major differences:

• list_join takes more than two lists as input.
• Each list can have a different separator.
• The resulting list can have a different separator.
• It is easier to compose lists.

[ownedbycats]

I have a couple of column-update templates that use list_union so I'll give it a try.

[chaley]

The template language can now loop over an integer range using a new range() function.

The (condensed) grammar for loops over lists and integers is:

    for_expr        ::= for_list | for_range
    for_list        ::= 'for' identifier 'in' list_expr
                        [ 'separator' separator_expr ] ':' expression_list 'rof'
    for_range       ::= 'for' identifier 'in' range_expr ':' expression_list 'rof'
    range_expr      ::= 'range' '(' [ start_expr ',' ] stop_expr
                        [ ',' step_expr [ ',' limit_expr ] ] ')'

The new range() function generates a sequence of integer values using the Python 3 builtin range function. The limit_expr parameter is added to prevent (near-)infinite loops. If step_expr is positive then the loop counts up. If it is negative the loop counts down. It cannot be zero. When positive, generation stops when

current_value + step_expr >= stop_expr

The list is empty (no iteration) if start_expr >= stop_expr. If the number if integers in the sequence exceeds limit_expr (default 1000) an error is raised.

In the context of a for loop the list isn't actually generated, improving performance and memory usage. In any other context the range() function generates and returns the list.

Example: this loop uppercases every second letter in the title:

program:
	res = '';
	for i in range(strlen($title)):
	    c = substr($title, i, i+1);
	    res = strcat(res, if mod(i, 2) == 0 then uppercase(c) else c fi)
	rof

Here is the same example using all the range() parameters:

program:
	res = '';
	for i in range(0, strlen($title), 1, 100):
	    c = substr($title, i, i+1);
	    res = strcat(res, if mod(i, 2) == 0 then uppercase(c) else c fi)
	rof

This example counts the number of unique alphanumeric characters in the title:

program:
	t = $title;
	res = '';
	for i in range(strlen(t)):
		c = lowercase(substr(t, i, i+1));
		if '\w' in c then
			res = list_join(',', res, ',', c, ',')
		fi
	rof;
	list_count(res, ',')

The documentation for range() is:

range(start, stop, step, limit) -- returns a list of numbers generated by looping over the range specified by the parameters start, stop, and step, with a maximum length of limit. The first value produced is start. Subsequent values next_v = current_v + step. The loop continues while next_v < stop assuming step is positive, otherwise while next_v > stop. An empty list is produced if start fails the test: start >= stop if step is positive. The limit sets the maximum length of the list and has a default of 1000. The parameters start, step, and limit are optional. Calling range() with one argument specifies stop. Two arguments specify start and stop. Three arguments specify start, stop, and step. Four arguments specify start, stop, step and limit.

Examples:

range(5) -> '0, 1, 2, 3, 4'
range(0, 5) -> '0, 1, 2, 3, 4'
range(-1, 5) -> '-1, 0, 1, 2, 3, 4'
range(1, 5) -> '1, 2, 3, 4'
range(5, 0, -1) -> '5, 4, 3, 2, 1'
range(1, 5, 2) -> '1, 3'
range(1, 5, 2, 5) -> '1, 3'
range(1, 5, 2, 1) -> error(limit exceeded)

[chaley]

26 May 2022 (In calibre version 5.43)

• New template function
  

  urls_from_identifiers(identifiers, sort_results)

  Given a comma-separated list of identifiers, where an identifier is a colon-separated pair of values (id_name:id_value), returns a comma-separated list of HTML URLs generated from the identifiers. The list not sorted if sort_results is 0 (character or number), otherwise it is sorted alphabetically by the identifier name. The URLs are generated in the same way as the built-in identifiers column when shown in Book details.

[chaley]

04/July/2022 (in calibre 6.0)

• New template function book_count():
  

  book_count(query, use_vl)

  returns the count of books found by searching for query. If use_vl is 0 (zero) then virtual libraries are ignored. This function and its companion ``book_values()`` are particularly useful in template searches, supporting searches that combine information from many books such as looking for series with only one book. It cannot be used in composite columns unless the tweak allow_template_database_functions_in_composites is set to True. It can be used only in the GUI.
• New template function book_values():
  

  book_values(column, query, sep, use_vl)

  returns a list of the unique values contained in the column column (a lookup name), separated by sep, in the books found by searching for query. If use_vl is 0 (zero) then virtual libraries are ignored. This function and its companion book_count() are particularly useful in template searches, supporting searches that combine information from many books such as looking for series with only one book. It cannot be used in composite columns unless the tweak allow_template_database_functions_in_composites is set to True. It can be used only in the GUI.

Example: this template search uses these functions to find all series with only one book:

1. Define a stored template (Preferences->Advanced->Template functions) named series_only_one_book (the name is arbitrary). The template is:
   

   program:
   	vals = globals(vals='');
   	if !vals then
   		all_series = book_values('series', 'series:true', ',', 0);
   		for series in all_series:
   			if book_count('series:="' & series & '"', 0) == 1 then
   				vals = list_join(',', vals, ',', series, ',')
   			fi
   		rof;
   		set_globals(vals)
   	fi;
   	str_in_list(vals, ',', $series, 1, '')

   The first time the template runs (the first book checked) it stores the results of the database lookups in a global template variable named vals. These results are used to check subsequent books without redoing the lookups.
2. Use the stored template in a template search:
   

   template:"program: series_only_one_book()#@#:n:1"

   Using a stored template instead of putting the template into the search eliminates problems caused by the requirement to escape quotes in search expressions.

[chaley]

I changed the way the template functions use the database to improve reliability when simultaneously using multiple databases, to decouple the functions from the gui, and to permit more functions to be used for columns to be displayed in the content server.

These changes are in calibre source now. Would those of you who run from source pull the latest and let me know if any behavior changed for the worse?

[ownedbycats]

Anything in specific you want testing? Using only one library, everything seems to be working fine at first glance.

[chaley]

Originally Posted by ownedbycats

Anything in specific you want testing? Using only one library, everything seems to be working fine at first glance.

Thanks. I want to make sure that the functions that use the database still work:

• annotation_count()
• book_count()
• book_values()
• check_yes_no()
• connected_device_name()
• connected_device_uuid()
• current_virtual_library_name()
• is_marked()
• virtual_libraries()

If they are still OK (and I am 99% sure they are) then I can worry about other "features" later if and when problems arise.

BTW: I tried them all before I submitted the source.

[ownedbycats]

annotation_count(), connected_device_name() (and uuid), current_virtual_library_name(), is_marked(), and virtual_libraries() all are working as expected.

[chaley]

Originally Posted by ownedbycats

annotation_count(), connected_device_name() (and uuid), current_virtual_library_name(), is_marked(), and virtual_libraries() all are working as expected.

[chaley]

There are two hidden (plugin) tweaks that affect the template processor:

• The plugin tweak enable_template_debug_printing controls whether the sources of user-defined python template functions is printed to the log.
  
  In the plugin tweaks section, set
  

  enable_template_debug_printing=True

  The default is False.
• For those who always run calibre in debug mode (calibre-debug -g), I added a tweak to stop printing exception tracebacks. This removes the many (!) lines you see when typing in the template tester, telling you that the program isn't valid. You will still see the exception reason as the output of the formatter. If the exception occurred while evaluating a composite column you will see a single log line naming the column.
  
  In the plugin tweaks section, set
  

  show_stack_traces_in_formatter = False

  The default is True

[un_pogaz]

Originally Posted by chaley

For those who always run calibre in debug mode (calibre-debug -g), I added a tweak to stop printing exception tracebacks. This removes the many (!) lines you see when typing in the template tester, telling you that the program isn't valid. You will still see the exception reason as the output of the formatter. If the exception occurred while evaluating a composite column you will see a single log line naming the column.

In the plugin tweaks section, set

show_stack_traces_in_formatter = False

Oh, I'm interested, I had used the template dialog and it took me a while to stop panicking about all the exceptions I had introduced in my code.
But how do I use it? I add "show_stack_traces_in_formatter = False" in any tweark configuration or I edit directly "tweaks.json" ?