MobileRead Forums - View Single Post

chaley · 02-23-2021, 06:33 AM

12 Feb 2025 - (in calibre 7.26)

Book details search URLs for text, enumeration, series, and composite custom columns.
Spoiler:
This isn't a template language change but here is a good place to mention it. You can now define a "search template" for a custom column. It is used to generate a search URL for the column, replacing the standard calibre 'search'. You enter the template in the "Search template:" box in the column definition dialog (Preferences / Add your own columns). The documentation is in the tool tip. Worth mentioning here: there are three new metadata variables available to these search templates:
- item_value: the value of the item clicked in Book details.
- item_value_quoted: the value of the item URL-encoded with spaces replaced by plus signs.
- item_value_no_plus: the value of the item URL-encoded with spaces replaced by '%20'.
New template functions:
- make_url(path, [query_name, query_value]+)
  Spoiler:
  -- this function is the easiest way to construct a query URL. It uses a path, the web site and page you want to query, and query_name, query_value pairs from which the query is built. In general, the query_value must be URL-encoded. With this function it is always encoded and spaces are always replaced with '+' signs.
  
  At least one query_name, query_value pair must be provided.
  
  Example: constructing a Wikipedia search URL for the author Niccolò Machiavelli:
  Code:
  
  make_url('https://en.wikipedia.org/w/index.php', 'search', 'Niccolò Machiavelli')
  returns
  Code:
  
  https://en.wikipedia.org/w/index.php?search=Niccol%C3%B2+Machiavelli
  If you are writing a custom column book details URL template then use $item_name or field('item_name') to obtain the value of the field that was clicked on. Example: if Niccolò Machiavelli was clicked then you can construct the URL using:
  Code:
  
  make_url('https://en.wikipedia.org/w/index.php', 'search', $item_name)
  See also the functions make_url_extended, query_string and encode_for_url.
- make_url_extended(...) -- this function is similar to make_url but gives you more control over the URL components.
  Spoiler:
  The components of a URL are
  
  scheme:://authority/path?query string.
  
  See Uniform Resource Locater on Wikipedia for more detail.
  
  The function has two variants:
  Code:
  
  make_url_extended(scheme, authority, path, [query_name, query_value]+)
  and
  Code:
  
  make_url_extended(scheme, authority, path, query_string)
  This function returns a URL constructed from the scheme, authority, path, and either the query_string or a query string constructed from the query argument pairs. The authority can be empty, which is the case for calibre scheme URLs. You must supply either a query_string or at least one query_name, query_value pair. If you supply query_string and it is empty then the resulting URL will not have a query string section.
  
  Example 1: constructing a Wikipedia search URL for the author Niccolò Machiavelli:
  Code:
  
  make_url_extended('https', 'en.wikipedia.org', '/w/index.php', 'search', 'Niccolò Machiavelli')
  returns
  Code:
  
  https://en.wikipedia.org/w/index.php?search=Niccol%C3%B2+Machiavelli
  See the query_string() function for an example using make_url_extended() with a query_string.
  
  If you are writing a custom column book details URL template then use $item_name or field('item_name') to obtain the value of the field that was clicked on. Example: if Niccolò Machiavelli was clicked on then you can construct the URL using :
  Code:
  
  make_url_extended('https', 'en.wikipedia.org', '/w/index.php', 'search', $item_name')
  See also the functions make_url, query_string and encode_for_url.
- query_string([query_name, query_value, how_to_encode]+)-- returns a URL query string constructed from the (query_name, query_value, how_to_encode) triads.
  Spoiler:
  A query string is a series of items where each item looks like query_name=query_value where query_value is URL-encoded as instructed. The query items are separated by '&' (ampersand) characters.
  
  If how_to_encode is 0 then query_value is encoded and spaces are replaced with '+' (plus) signs. If how_to_encode is 1 then query_value is encoded with spaces replaced by %20. If how_to_encode is 2 then query_value is returned unchanged; no encoding is done and spaces are not replaced. If you want query_value not to be encoded but spaces to be replaced then use the re function, as in re($series, ' ', '%20')
  
  You use this function if you need specific control over how the parts of the query string are constructed. You could then use the resultingquery string in make_url_extended, as in
  Code:
  
  make_url_extended( 'https', 'your_host', 'your_path', query_string('encoded', 'Hendrik Bäßler', 0, 'unencoded', 'Hendrik Bäßler', 2))
  giving you the the URL that is invalid because of the space after "Hendrik"
  Code:
  
  https://your_host/your_path?encoded=Hendrik+B%C3%A4%C3%9Fler&unencoded=Hendrik Bäßler
  You must have at least one query_name, query_value, how_to_encode triad, but can have as many as you wish.
  
  The returned value is a URL query string with all the specified items, for example: name1=val1[&nameN=valN]*. Note that the '?' path / query string separator is not included in the returned result.
  
  If you are writing a custom column book details URL template then use $item_name or field('item_name') to obtain the unencoded value of the field that was clicked. You also have item_value_quoted where the value is already encoded with plus signs replacing spaces, and item_value_no_plus where the value is already encoded with %20 replacing spaces.
  
  See also the functions make_url, make_url_extended and encode_for_url.
- encode_for_url(value, use_plus) -- returns the value encoded for use in a URL as specified by use_plus.
  Spoiler:
  The value is first URL-encoded. Next, if use_plus is 0 then spaces are replaced by '+' (plus) signs. If it is 1 then spaces are replaced by %20.
  
  If you do not want the value to be encoding but to have spaces replaced then use the re function, as in re($series, ' ', '%20')
  
  See also the functions make_url, make_url_extended and query_string.

28 Oct 2024 (in calibre 7.21)

Improvement to template function has_notes. It now has two variants.
1. has_note(field_name, field_value) -- if field_value is not '' (the empty string) , return '1' if the value field_value in the field field_name has an attached note, otherwise ''.
  Spoiler:
  Example:
  Code:
  
  has_note('tags', 'Fiction')
  returns '1' if the tag "fiction" has a note, otherwise ''.
2. (NEW) has_note(field_name, '') -- if the second parameter is '' then return a list of values in field_name that have a note.
  Spoiler:
  If no item in the field has a note, return ''. Example:
  Code:
  
  has_note('authors', '')
  returns a list of authors that have notes, or '' if no author has a note.
  
  For example, the second variant is useful for showing column icons if any value in the field has a note rather than a specific value.
  
  You can also test if all the values have a note by comparing the list length of this function's return value against the list length of the values in field_name. Example:
  Code:
  
  list_count(has_note('authors', ''), '&') ==# list_count_field('authors')

23 Oct 2024 (in calibre preview 7.20.100)

New template binary operator inlist_field
Spoiler:
inlist_field does the same as inlist except that the right-hand expression must evaluate to a field (column) name. This operator is much faster than inlist because it doesn't convert the field contents to a string. In addition, it works on fields that use a separator other than comma, for example authors. Note that the operator requires the field name not the value, so using $tags will not work. Example:
Code:
```
'foo$' inlist_field 'tags'
```

18 Oct 2024 (in calibre preview 7.20.100)

New template function list_count_field()
Spoiler:
Code:
```
list_count_field(lookup_name)
```
returns the count of items in the field with the lookup name lookup_name. The field must be multi-valued such as authors or tags, otherwise the function raises an error. This function is much faster than list_count() because it operates directly on calibre data without converting it to a string first. Example: list_count_field('tags')

3 March 2024: (in calibre 7.7)

Change to get_notes() template function

Spoiler:
I changed the get_notes() function to return HTML with all image URLs fully expanded to use src="data". This lets the result be used in contexts other than the current calibre library, for example book jackets and book details from a different library. They also now work in tooltips, showing the images, but that is less important because a composite column containing notes probably shouldn't be showing on the book list

Earlier:

Spoiler:

13 Nov 2023: (in calibre 7.0)

New template function get_link(field_name, field_value)
Spoiler:
get_link(field_name, field_value) -- fetch the link for field field_name with value field_value. If there is no attached link, return the empty string.

Examples:
The following returns the link attached to the tag Fiction:
Code:
```
get_link('tags', 'Fiction')
```
This template makes a list of the links for all the tags associated with a book in the form value:link, ...:
Code:
```
program:
 ans = '';
 for t in $tags:
     l = get_link('tags', t);
     if l then
         ans = list_join(', ', ans, ',', t & ':' & get_link('tags', t), ',')
     fi
 rof;
 ans
```

26 Oct 2023 (in calibre beta 6.99)

New template function get_note()
Spoiler:
Code:
```
get_note(field_name, field_value, plain_text)
```
Fetch the note for field 'field_name' with value 'field_value'. If 'plain_text' is empty, return the note's HTML. If 'plain_text' is non-empty, return the note's plain text. If the note doesn't exist, return '' in both cases.
Example: get_note('tags', 'Fiction', '') returns the HTML of the note attached to the tag 'Fiction'.
New template function has_note()
Spoiler:
Code:
```
has_note(field_name, field_value)
```
Return '1' if the value 'field_value' in the field 'field_name' has an attached note, otherwise return ''.
Example: has_note('tags', 'Fiction') returns '1' if the tag 'fiction' has an attached note, otherwise return ''.

17 Sep 2023 (in calibre 6.27)

New template function format_date_field()
Spoiler:
Code:
```
format_date_field(field_name, format_string)
```
format the value in the field field_name, which must be the lookup name of date field, either standard or custom. See format_date() for the formatting codes. This function is much faster than format_date and should be used when you are formatting the value in a field (column). It can't be used for computed dates or dates in string variables. Examples::
Code:
```
   format_date_field('pubdate', 'yyyy.MM.dd')
   format_date_field('#date_read', 'MMM dd, yyyy')
```

10 May 2023 (in calibre 6.18)

Changed template function identifier_in_list()
Spoiler:
Code:
```
identifier_in_list(val, id_name [, found_val, not_found_val])
```
treat val as a list of identifiers separated by commas. An identifier has the format id_name:value. The id_name parameter is the id_name text to search for, either id_name or id_name:regexp. The first case matches if there is any identifier matching that id_name. The second case matches if id_name matches an identifier and the regexp matches the identifier's value. If found_val and not_found_val are provided then if there is a match then return found_val, otherwise return not_found_val. If found_val and not_found_val are not provided then if there is a match then return the indentfier:value pair, otherwise the empty string ('').

27 Apr 2023 (in calibre 6.18)

Changed template function has_extra_files()
Spoiler:
Code:
```
has_extra_files([pattern])
```
returns the count of extra files, otherwise '' (the empty string). If the optional parameter 'pattern', a regular expression, is supplied then the list is filtered to files that match pattern before the files are counted. The pattern match is case insensitive. This function can be used only in the GUI.

What changed: the 'pattern' parameter was added and the function returns a count instead of 'Yes'.
Changed template function extra_file_names()
Spoiler:
Code:
```
extra_file_names(sep [, pattern])
```
returns a sep-separated list of extra files in the book's '{}/' folder. If the optional parameter 'pattern', a regular expression, is supplied then the list is filtered to files that match pattern. The pattern match is case insensitive. This function can be used only in the GUI.

What changed: the 'pattern' parameter was added.

24 Apr 2023 (in calibre 6.17)

New template function has_extra_data()
Spoiler:
Code:
```
has_extra_data()
```
Returns 'Yes' if there are any extra files for the book (files in the folder data/ in the book's folder), otherwise '' (the empty string). See also the functions extra_file_names(), extra_file_size() and extra_file_modtime() This function can be used only in the GUI.

Note: this function will change in calibre 6.18
New template function extra_file_names()
Spoiler:
Code:
```
extra_file_names(sep)
```
Returns a sep-separated list of extra files in the book's data/ folder. See also the functions has_extra_files(), extra_file_size() and extra_file_modtime(). This function can be used only in the GUI.
Note: this function will change in calibre 6.18
New template function extra_file_size()
Spoiler:
Code:
```
extra_file_size(file_name)
```
Returns the size in bytes of the extra file file_name in the book's data/ folder if it exists, otherwise -1. See also the functions has_extra_files(), extra_file_names() and extra_file_modtime(). This function can be used only in the GUI.
New template function extra_file_modtime()
Spoiler:
Code:
```
extra_file_modtime(file_name, format_spec)
```
Returns the modification time of the extra file file_name in the book's data/ folder if it exists, otherwise -1. The modtime is formatted according to format_spec (see format_date() for details). If format_spec is the empty string, returns the modtime as the floating point number of seconds since the epoch. See also the functions has_extra_files(), extra_file_names() and extra_file_size(). The epoch is OS dependent. This function can be used only in the GUI.

08 Apr 2023 (in calibre 6.15)

Changed function series_sort().
This function now uses the book's language when doing article processing.

06 Jan 2023 (in calibre 6.12):

New function switch_if()
Code:
```
switch_if([test_expression, value_expression,]+ else_expression)
```
Spoiler:
For each "test_expression, value_expression" pair, checks if test_expression is True (non-empty) and if so returns the result of value_expression. If no test_expression is True then the result of else_expression is returned. You can have as many "test_expression, value_expression" pairs as you want.

15 Nov 2022 (in calibre 6.9.0)

The command line utility calibredb list can now use a template as a field (column).

Spoiler:

Use the field "template" to add the output of a template to each book output by calibredb list. The template itself can be on the command line or in a file.

The documentation, with the new template-related options in bold:

Spoiler:

Code:

calibredb list --help
Usage: calibredb.exe list [options]

List the books available in the calibre database.

Whenever you pass arguments to calibredb.exe that have spaces in them, enclose the arguments in quotation marks. For example: "C:\some path with spaces"

Options:
-f FIELDS, --fields=FIELDS
The fields to display when listing books in the
database. Should be a comma separated list of fields.
Available fields: author_sort, authors, comments,
cover, formats, identifiers, isbn, languages,
last_modified, pubdate, publisher, rating, series,
series_index, size, tags, template, timestamp, title,
uuid
Default: title,authors. The special field "all" can be
used to select all fields. In addition to the builtin
fields above, custom fields are also available as
*field_name, for example, for a custom field #rating,
use the name: *rating

--sort-by=SORT_BY The field by which to sort the results.
Available fields: author_sort, authors, comments,
cover, formats, identifiers, isbn, languages,
last_modified, pubdate, publisher, rating, series,
series_index, size, tags, template, timestamp, title,
uuid
Default: id

--ascending Sort results in ascending order

-s SEARCH, --search=SEARCH
Filter the results by the search query. For the format
of the search query, please see the search related
documentation in the User Manual. Default is to do no
filtering.

-w LINE_WIDTH, --line-width=LINE_WIDTH
The maximum width of a single line in the output.
Defaults to detecting screen size.

--separator=SEPARATOR
The string used to separate fields. Default is a
space.

--prefix=PREFIX The prefix for all file paths. Default is the absolute
path to the library folder.

--limit=LIMIT The maximum number of results to display. Default: all

--for-machine Generate output in JSON format, which is more suitable
for machine parsing. Causes the line width and
separator options to be ignored.

--template=TEMPLATE The template to run if "template" is in the field
list. Default: None

-t TEMPLATE_FILE, --template_file=TEMPLATE_FILE
Path to a file containing the template to run if
"template" is in the field list. Default: None

--template_heading=TEMPLATE_HEADING
Heading for the template column. Default: template.
This option is ignored if the option --for-machine is
set

GLOBAL OPTIONS:
--library-path=LIBRARY_PATH, --with-library=LIBRARY_PATH
Path to the calibre library. Default is to use the
path stored in the settings. You can also connect to a
calibre Content server to perform actions on remote
libraries. To do so use a URL of the form:
http://hostname:port/#library_id for example,
http://localhost:8080/#mylibrary. library_id is the
library id of the library you want to connect to on
the Content server. You can use the special library_id
value of - to get a list of library ids available on
the server. For details on how to setup access via a
Content server, see https://manual.calibre-
ebook.com/generated/en/calibredb.html.

-h, --help show this help message and exit

--version show program's version number and exit

--username=USERNAME
Username for connecting to a calibre Content server

--password=PASSWORD
Password for connecting to a calibre Content server.
To read the password from standard input, use the
special value: <stdin>. To read the password from a
file, use: <f:C:/path/to/file> (i.e. <f: followed by
the full path to the file and a trailing >). The angle
brackets in the above are required, remember to escape
them or use quotes for your shell.

--timeout=TIMEOUT The timeout, in seconds, when connecting to a calibre
library over the network. The default is two minutes.

Created by Kovid Goyal <kovid@kovidgoyal.net>

Example: produce a list of books with a series matching "great" and output the books with the authors of all the books in the series:
The command:

Spoiler:

The template (in a file). The template can be written in any of the template language modes. This one is in python mode:

Spoiler:

The output:

Spoiler:

29 Oct 2022 (in calibre 6.8.0)

Python templates can call stored templates and formatter functions.

Spoiler:

Breakpoints for Python templates in the Template tester dialog.

Spoiler:

12 Oct 2022 (in calibre 6.7.0)

Python Template Mode (PTM) lets you write templates using native python and the calibre API.
Spoiler:
The database API will be of most use. PTM templates are much faster than the other template modes and can do more complicated operations, but you must know how to write code in python using the calibre API.

A PTM template looks like this:
Code:
```
python:
def evaluate(book, context):
    # book is a calibre metadata object
    # context is an instance of calibre.utils.formatter.PythonTemplateContext,
    # which (currently) contains the following attributes:
    # db: a calibre legacy database object
    # globals: the template global variable dictionary
    # arguments: is a list of arguments if the template is called by a GPM template, otherwise None

    # your Python code goes here
    return 'a string'
```
You can add the above text to your template using the context menu, usually accessed with a right click. The comments are not significant and can be removed. You must use python indenting.

The context object supports str(context) that returns a string of the context's contents, and context.attributes that returns a list of the attribute names in the context.

Here is an example of a PTM template that produces a list of all the authors for a series. The list is stored in a Column built from other columns, behaves like tags. It shows in Book details and has on separate lines checked in Preferences->Look & feel->Book details. That option requires the list to be comma-separated. To satisfy that requirement the template converts commas in author names to semicolons then builds a comma-separated list of authors. The authors are then sorted, which is why the template uses author_sort.
Code:
```
python:
def evaluate(book, context):
    if book.series is None:
        return ''
    db = context.db.new_api
    ans = set()
    # Get the list of books in the series
    ids = db.search(f'series:"={book.series}"', '')
    if ids:
        # Get all the author_sort values for the books in the series
        author_sorts = (v for v in db.all_field_for('author_sort', ids).values())
        # Add the names to the result set, removing duplicates
        for aus in author_sorts:
            ans.update(v.strip() for v in aus.split('&'))
    # Make a sorted comma-separated string from the result set
    return ', '.join(v.replace(',', ';') for v in sorted(ans))
```
The output in Book details looks like this:

16 Sept 2022 (in calibre 6.5.0)

New function to_hex()
Code:
```
to_hex(val)
```
Returns the string val encoded in hex. This is useful when constructing calibre URLs.

14 Sept 2022 (in calibre 6.5.0)

New function strcmpcase()
Code:
```
strcmpcase(x, y, lt, eq, gt)
```
Spoiler:
does a case-sensitive lexical comparison of x and y. Returns lt if x < y, eq if x == y, otherwise gt.

Note: This is NOT the default behavior used by calibre, for example, in the lexical comparison operators (==, >, <, etc.). This function could cause unexpected results, preferably use ``strcmp()`` whenever possible.

Thanks to @un_pogaz for this function.

23 July 2022 (in calibre 6.2.1):

Performance improvement of the switch() function

Spoiler:
The switch function now does "shortcutting". It evaluates the comparison expressions in turn. If the comparison succeeds then the result expression for that comparison is evaluated and returned. No other result expressions are evaluated. This provides a performance improvement of at least 50%. It can be much higher if the switch() has many compare/result pairs and they are organized in decreasing probability order.

Behavior change: because most of the result expressions and some of the compare expressions are not evaluated, expressions with side effects that formerly were evaluated might not be. You can no longer expect side effects to work. For example, you shouldn't use expressions that contain assignments in a switch().
Add a tooltip in the column icon advanced rule editor explaining that an advanced rule can return a compound icon by separating the individual icons (file names) with a ':' (colon).

04 July 2022 (in calibre 6.0)

New template function book_count():
Code:
```
book_count(query, use_vl)
```
Spoiler:
returns the count of books found by searching for query. If use_vl is 0 (zero) then virtual libraries are ignored. This function and its companion ``book_values()`` are particularly useful in template searches, supporting searches that combine information from many books such as looking for series with only one book. It cannot be used in composite columns unless the tweak allow_template_database_functions_in_composites is set to True. It can be used only in the GUI.
New template function book_values():
Code:
```
book_values(column, query, sep, use_vl)
```
Spoiler:
returns a list of the unique values contained in the column column (a lookup name), separated by sep, in the books found by searching for query. If use_vl is 0 (zero) then virtual libraries are ignored. This function and its companion book_count() are particularly useful in template searches, supporting searches that combine information from many books such as looking for series with only one book. It cannot be used in composite columns unless the tweak allow_template_database_functions_in_composites is set to True. It can be used only in the GUI.
Example: this template search uses these functions to find all series with only one book:
Spoiler:
1. Define a stored template (Preferences->Advanced->Template functions) named series_only_one_book (the name is arbitrary). The template is:
  Code:
  
  program: vals = globals(vals=''); if !vals then all_series = book_values('series', 'series:true', ',', 0); for series in all_series: if book_count('series:="' & series & '"', 0) == 1 then vals = list_join(',', vals, ',', series, ',') fi rof; set_globals(vals) fi; str_in_list(vals, ',', $series, 1, '')
  The first time the template runs (the first book checked) it stores the results of the database lookups in a global template variable named vals. These results are used to check subsequent books without redoing the lookups.
2. Use the stored template in a template search:
  Code:
  
  template:"program: series_only_one_book()#@#:n:1"
  Using a stored template instead of putting the template into the search eliminates problems caused by the requirement to escape quotes in search expressions.

26 May 2022 (In calibre version 5.43)

New template function
Code:
```
urls_from_identifiers(identifiers, sort_results)
```
Spoiler:
Given a comma-separated list of identifiers, where an identifier is a colon-separated pair of values (id_name:id_value), returns a comma-separated list of HTML URLs generated from the identifiers. The list not sorted if sort_results is 0 (character or number), otherwise it is sorted alphabetically by the identifier name. The URLs are generated in the same way as the built-in identifiers column when shown in Book details.

14 May 2022 (In calibre version 5.43)

The template language can now loop over an integer range using a new range() function.
Spoiler:
The (condensed) grammar for loops over lists and integers is:
Spoiler:
Code:

for_expr ::= for_list | for_range for_list ::= 'for' identifier 'in' list_expr [ 'separator' separator_expr ] ':' expression_list 'rof' for_range ::= 'for' identifier 'in' range_expr ':' expression_list 'rof' range_expr ::= 'range' '(' [ start_expr ',' ] stop_expr [ ',' step_expr [ ',' limit_expr ] ] ')'
The range() function generates a sequence of integer values using the Python 3 builtin range function. The limit_expr parameter is added to prevent (near-)infinite loops. If step_expr is positive then the loop counts up. If it is negative the loop counts down. It cannot be zero. When positive, generation stops when
Code:
```
current_value + step_expr >= stop_expr
```
The list is empty (no iteration) if start_expr >= stop_expr. If the number if integers in the sequence exceeds limit_expr (default 1000) an error is raised.

In the context of a for loop the list isn't actually generated, improving performance and memory usage. In any other context the range() function generates and returns the list.

Template examples using range loops:
Spoiler:
This loop uppercases every second letter in the title:

Code:

program: res = ''; for i in range(strlen($title)): c = substr($title, i, i+1); res = strcat(res, if mod(i, 2) == 0 then uppercase(c) else c fi) rof

Here is the same example using all the range() parameters:

Code:

program: res = ''; for i in range(0, strlen($title), 1, 100): c = substr($title, i, i+1); res = strcat(res, if mod(i, 2) == 0 then uppercase(c) else c fi) rof

This example counts the number of unique alphanumeric characters in the title:

Code:

program: t = $title; res = ''; for i in range(strlen(t)): c = lowercase(substr(t, i, i+1)); if '\w' in c then res = list_join(',', res, ',', c, ',') fi rof; list_count(res, ',')
The documentation for range() is:
Spoiler:
range(start, stop, step, limit) -- returns a list of numbers generated by looping over the range specified by the parameters start, stop, and step, with a maximum length of limit. The first value produced is start. Subsequent values next_v = current_v + step. The loop continues while next_v < stop assuming step is positive, otherwise while next_v > stop. An empty list is produced if start fails the test: start >= stop if step is positive. The limit sets the maximum length of the list and has a default of 1000. The parameters start, step, and limit are optional. Calling range() with one argument specifies stop. Two arguments specify start and stop. Three arguments specify start, stop, and step. Four arguments specify start, stop, step and limit.

Examples:

Code:

range(5) -> '0, 1, 2, 3, 4' range(0, 5) -> '0, 1, 2, 3, 4' range(-1, 5) -> '-1, 0, 1, 2, 3, 4' range(1, 5) -> '1, 2, 3, 4' range(5, 0, -1) -> '5, 4, 3, 2, 1' range(1, 5, 2) -> '1, 3' range(1, 5, 2, 5) -> '1, 3' range(1, 5, 2, 1) -> error(limit exceeded)

25 April 2022 (In calibre V5.42)

New template function
Code:
```
list_join(with_separator, list1, separator1 [, list2, separator2]*)
```
Spoiler:
Returns a list made by joining the items in the source lists (list1 etc) using with_separator between the items in the result list. Items in each source list[123...] are separated by the associated separator[123...]. A list can contain zero values. It can be a field like publisher that is single-valued, effectively a one-item list. Duplicates are removed using a case-insensitive comparison. Items are returned in the order they appear in the source lists. If items on lists differ only in letter case then the last is used. All separators can be more than one character.

Example:
Code:
```
program:
  list_join(':@:', $authors, '&', $tags, ',')
```
You can use list_join on the results of previous calls to list_join as follows:
Code:
```
program:
  a = list_join(':@:', $authors, '&', $tags, ',');
  b = list_join(':@:', a, ':@:', $#genre, ',', $#people, '&', 'some value', ',')
```
You can use expressions to generate a list. For example, assume you want items for authors and #genre, but with the genre changed to the word "Genre: " followed by the first letter of the genre, i.e. the genre "Fiction" becomes "Genre: F". The following will do that:
Code:
```
program:
  list_join(':@:', $authors, '&', list_re($#genre, ',', '^(.).*$', 'Genre: \1'),  ',')
```

27 March 2022 (In calibre version 5.40)

The ability to define local functions.
Spoiler:
For example, the following example computes an approximate duration in years, months, days from a number of days. It uses the defined local function to_plural() to format the numbers for output.
Code:
```
program:
	days = 2112;
	years = floor(days/360);
	months = floor(mod(days, 360)/30);
	days = days - ((years*360) + (months * 30));

	def to_plural(v, str):
		if v == 0 then return '' fi;
		return v & ' ' & (if v == 1 then str else str & 's' fi) & ' '
	fed;

	to_plural(years, 'year') & to_plural(months, 'month') & to_plural(days, 'day')
```
Notes:
- The grammar for defining a function is
  Code:
  
  function ::= 'def' function_name '(' argument_expr [',' argument_expr]* ')' ':' expression_list 'fed' argument_expr ::= identifier | identifier '=' expression
- Functions must be defined before they are used.
- The outer scope local variables are not visible inside a function.
- The parameters in the definition can have default values.
- You can call a function with fewer arguments than defined parameters. Parameters matched to 'missing' arguments are given their default value or the empty string if there isn't a default value.
A 'concatenate strings' operator '&'.
Spoiler:
The expression
Code:
```
'aaa' & 'bbb' & 'ccc'
```
is equivalent to
Code:
```
strcat('aaa', 'bbb', 'ccc')
```

11 November 2021 (In calibre version 5.32)

Performance improvement: fix failure to inline the functions first_non_empty() and test().
Performance improvement: inline the strcat() function.

10 November 2021 (In calibre version 5.32)

Add the function current_virtual_library_name().
Spoiler:
Return the name of the current virtual library if there is one, otherwise the empty string. Library name case is preserved. Example:
Code:
```
program: current_virtual_library_name()
```
This function can be used only in the GUI.

31 October 2021 (In calibre 5.31.1)

Bug fix: the binary arithmetic operators (+ - * /) failed if either value was undefined.

Spoiler:
Example: $$#myint doesn't have a value for the book. The fix: make them work like the comparison operators: assume undefined is zero.

3 June 2021 (In calibre version 5.21)

Add the function date_arithmetic(date, calc_spec, fmt) to simplify computing new dates from existing ones.

Spoiler:

21 April 2021 (In calibre version 5.17)

Add the function "character('character_name')" that returns the special character named by character_name.
Spoiler:
For example, character('newline') returns a newline character ('\n'). The function can be used with strcat() to create multiline and tab-aligned output. The supported character names are 'newline', 'return', 'tab', and 'backslash'. The argument to character() can be an expression that returns the character name.
Show special characters in the template editor & debugger as escape sequences, e.g., a newline is displayed as \n.
Performance improvements in the template language parser.

18 April 2021 (In calibre version 5.16.1)

Fix a regression in calibre version 5.15 that broke stored templates. They always raise an exception if you aren't using the template debugger.
Add a tweak "Tab stop width in the template editor (ID: template_editor_tab_stop_width)" to control the width of tab stops in the template editor. The default is now 4 average-size characters.

14 Apr 2021 (in calibre version 5.15)

Add the infix operator 'pattern inlist list'. It returns True ('1') if the regular expression pattern matches any item in the comma-separated list, otherwise False ('').

12 Apr 2021 (in calibre version 5.15)

Add 'return expression' to the template language. The expression doesn't need to be in parentheses.
Improve syntax and runtime error messages
Fix not using metadata from the selected book when breakpointing.
Improved help text in the Stored Template dialog.
Allow hiding the help text in the Stored Template dialog.

06 Apr 2021 (in calibre version 5.15)

Add 'break' and 'continue' to the template language.
Add a "Test" button to the Stored Templates preferences dialog.

Spoiler:
Clicking this button opens a subsidiary template tester that can 'see' the templates being created/edited in the preferences dialog, where you can write a template that test the stored templates. It shows the values using the books selected before starting the preferences dialog.
Make TAB and SHIFT-TAB indent and unindent selected lines in the template tester.
Add template file load/save buttons and to the template tester context menu. The buttons and the context menu items do the same thing.
Add the ability to toggle word wrapping to the template tester context menu.
Make the top button line reflow to two lines if the dialog box isn't wode enough

29 Mar 2021 (in calibre version 5.15)

Add line-based breakpoints to the template tester

Spoiler:
The template tester now supports setting 'breakpoints' on lines. If a breakpoint is set then template evaluation pauses and a dialog is opened showing you the result of the expression, all the local variables, and a dropdown box that you can use to lookup metadata.
You can see this and the new highlighting in this screen capture.
Attachment 186204
Add syntax highlighting to identifiers and field references
Add template file load/save to the context menu

26 Mar 2021 (in calibre version 5.14)

Add new date formats 'to_number' and 'from_number'
Spoiler:
You can now convert dates to/from a floating point number representing the number of seconds since some platform-dependent start point. This number is sometimes called a 'timestamp'. Number-format dates are easier to compare and to modify using arithmetic. Details:
- to_number : convert the date & time into a floating point number (a `timestamp`)
- from_number : convert a floating point number (a `timestamp`) into an 'iso' formatted date. If you want a different date format then add the desired formatting string after 'from_number' and a colon (':'). Example:
  from_number:MMM dd yyyy

24 Mar 2021 (in calibre version 5.14)

Addition of field references
Spoiler:
You can now use $lookup_key instead of field('lookup_key') and $$lookup_key instead of raw_field('lookup_key'). Examples:
Code:
```
  * $authors ==> field('authors')
  * $#genre ==> field('#genre')
  * $$pubdate ==> raw_field('pubdate')
  * $$#my_int ==> raw_field('#my_int')
```

21 Mar 2021 (in calibre version 5.14)

Template tester - allow changing the font used in the template edit box

Spoiler:
You can now change both the font and the font size. Both are remembered.

20 Mar 2021 (in calibre version 5.14)

Template tester - show template results for multiple books

Spoiler:
If You select multiple books before launching the template tester then it will show you the template evaluation result for each book. The order in the results list is the order the books were selected.

Save the new dialog box size if it is changed.

11 Mar 2021 (in calibre version 5.14)

Performance improvement of the virtual_libraries() template function
Spoiler:
The performance of the virtual_libraries function has been improved substantially. The change is most important for composite columns that call virtual_libraries; the template is evaluated for every book that is displayed in the book list/cover browser. For example, on my Windows machine using a test database with 3900 books, 15 virtual libraries, and one composite column that calls virtual_libraries(), the performance change i
Code:
```
First call to virtual_libraries()
             old = 368,381 μs
             new = 382,352 μs

Typical time for each call thereafter until data changes:
             old = 165,338 μs
             new =       1 μs
```
Result: scrolling 20 books goes from 3.3 seconds to 20 microseconds.

On a smaller library of 200 books with 5 virtual libraries the numbers are:
Code:
```
First call to virtual_libraries()
             old = 7,852 μs
             new = 7,846 μs

Typical time for each call thereafter, until data changes:
             old = 195 μs
             new =  <1 μs
```
The performance improvement is directly related to the number of virtual libraries.

7 Mar 2021 (in calibre version 5.13):

Fix regression introduced sometime in calibre V5: expression lists as parameters
Spoiler:
In calibre versions before V5.something one could use expression lists as parameters. Example:
Code:
```
program:
    list_union(
			a = 'xxx'; list_union('a', field('tags'), ','), 
			field('#genre'),
			','
		)
```
This 'feature' was certainly rarely used but it was possible. Fixed in the interest of compatibility.
Allow expression lists in parenthesized expressions.
Spoiler:
As a consequence of the above, you can now put expression lists in parenthesized expressions. Example:
Code:
```
program:
	b = (
				a = field('#mytextmult');
				list_union(a, field('tags'), ',')
			);
	if a then
		list_union(b, field('#genre'), ',')
	else
		b
	fi
```

28 Feb 2021 (in calibre version 5.13):

Addition of common logical and arithmetic operators
Spoiler:
The template language now supports the common logical operators ('&&, '||', and '!), and arithmetic operators (unary '+', unary '', and binary '+', '-', '*', and '/').

The operator precedence in the template language is now:
- Function calls, constants, parenthesized expressions, statement expressions, assignment expressions. Remember that in the template language, 'if', 'for', and assignment return a value.
- Unary plus (+) and minus (-). These operators evaluate right to left. These and the other arithmetic operators return integers if the expression doesn't produce a fractional part.
- Multiply (*) and divide (/). These operators are associative and evaluate left to right. Use parentheses if you want to change the order of evaluation.
- Add (+) and subtract (-). These operators are associative and evaluate left to right.
- Numeric and string comparisons (these already existed). These operators return '1' if the comparison is True, otherwise ''. Comparisons are not associative: a < b < c produces a syntax error. Comparisons return '1' if True and '' if False.
- Unary logical not (!). This operator returns '1' if the expression is False (evaluates to the empty string), otherwise ''.
- Logical and (&&). This operator returns '1' if both the left-hand and right-hand expressions are True, the empty string '' if either is False, is associative, evaluates left to right, and does short-circuiting.
  Spoiler:
  Regarding short-circuiting: for example this program produces the answer '4'. Because of short-circuiting the right-hand expression, the assignment, is evaluated because the left-hand expression is True. The assignment is done.
  
  Code:
  
  program: a = 5; 'a' && (a = 4); a
  
  This program produces '5' because the the left-hand expression is False so because of short-circuiting the right-hand expression is not evaluated. The assignment is not done.
  
  Code:
  
  program: a = 5; '' && (a = 4); a
- Logical or (||). This operator returns '1' if either the left-hand or right-hand expression is True, '' if both are False, is associative, evaluates left to right, and does short-circuiting. It does an inclusive or, returning '1' if both the left- and right-hand expressions are True.

25 Feb 2021 (in calibre version 5.12):

Function raw_field: new optional default value
Spoiler:
raw_field(field_name_expression[, default_value_expression]) -- if the value of the field is undefined (is 'None') return the default_value_expression if it is provided, otherwise return 'None'. Examples:
Code:
```
v = raw_field('#my_int', -1000)
```
returns the value of #my_int or -1000 if #my_int is undefined.
Code:
```
v = raw_field('#my_int')
```
returns the value of #my_int or None if #my_int is undefined.

Note that multiple-value fields like 'tags' are never undefined, but are instead empty. The is_empty() function can be used in this case to provide a non-empty default.

23 Feb 2021:

New template function list_count_matching(v, pattern, sep)

Spoiler:
list_count_matching(list, pattern, separator) -- interprets 'list' as a list of items separated by 'separator', returning the number of items in the list that match the regular expression 'pattern'. Aliases: list_count_matching(), count_matching()

New infix compare operator 'in'.

Spoiler:

22 Feb 2021

Add the possibility of specifying the separator in 'for' statements used to break the string into list items.
Spoiler:
Syntax:
Code:
```
for <<id>> in <<value_expression>> separator <<expression>>:
```
The keyword 'separator' and its associated expression are optional.

Example:
Code:
```
for a in some_authors separator '&':
```
New function list_remove_duplicates()

Spoiler:
list_remove_duplicates(list, separator) -- return a list made by removing duplicate items in the source list. If items differ only in case, the last of them is returned. The items in source list are separated by separator, as are the items in the returned list.

20 Feb 2021

Fix subitems() sometimes returning empty list items.

19 Feb 2021

Fix 'for' not stripping leading and trailing blanks from the value assigned to the loop variable.

18 Feb 2021

New function set_globals()

Spoiler:
set_globals(id[=expression] [, id[=expression]]*) -- Sets the value of "global variables" that can be passed into the formatter. If 'id' is not defined then set the global to the value of expression if supplied, otherwise ''. See the globals() function for more information.

17 Feb 2021:

New function is_marked()

Spoiler:
is_marked() -- check whether the book is `marked` in calibre. If it is then return the value of the mark: either `true` (lower case) or the comma-separated list of named marks. Returns '' (the empty string) if the book is not marked. This function works only in the GUI.

17 Jan 2021 (in calibre version 5.11):

The functions add() and multiply() can now take more than 2 arguments.
New function field_exists()

Spoiler:
field_exists(field_name) -- checks if a field (column) named field_name exists, returning '1' if so and '' if not.

3 Jan 2021:

New function list_split()
Spoiler:
list_split(list_val, sep, id_prefix) -- splits the list_val into separate values using 'sep', then assigns the values to variables named 'id_prefix_N' where N is the position of the value in the list. The first item has position 0 (zero). The function returns the last element in the list.

Example: list_split('one, two, foo', ',', 'var') is equivalent to
var_0 = 'one'; var_1 = 'two'; var_3 = 'foo'.

Example: break apart and reformat a date expressed as a format and a list of values:
Code:
```
program:
# date_string would normally be field() to fetch the value from the book
# A 'date string' is a list with the first the desired format and the rest the values
	date_string = 'YMD,1984,12, 01';
	list_split(date_string, ',', 'li');
	if li_0 == 'YMD' then
		strcat(li_1, '-', li_2, '-', li_3)
	elif li_0 == 'YM'  then
		strcat(li_1, '-', li_2)
	elif li_0 == 'Y'  then
		li_1
	else
		'Invalid Format'
	fi
```

22 Dec 2020

New function connected_device_uuid(storage_location)

Spoiler:
connected_device_uuid(storage_location) -- if a device is connected then return the device uuid (unique id), otherwise return the empty string. Each storage location on a device has a different uuid. The location names are 'main', 'carda' and 'cardb'. This function works only in the GUI.

19 Dec 2020

Added a 'for' template language statement. See also 22 Feb 2021 above.
Spoiler:
Code:
```
    for <<id>> in <<expression>>:
        <<expression_list>>
    rof
```
The expression must evaluate to either a metadata field lookup key, for example 'tags or '#genre', or a comma-separated list of values. If the result is a valid lookup name then the field's value is fetched, otherwise the list is broken into its individual values. Each resulting value in the list is assigned to the variable 'id'' then the 'expression_list' is evaluated.

Example: This template removes the first hierarchical name for each value in Genre ('#genre'), constructing a list with the new names.
Code:
```
        program:
            new_tags = '';
            for i in '#genre':
                j = re(i, '^.*?\.(.*)$', '\1');
                new_tags = list_union(new_tags, j, ',')
            rof;
            new_tags
```
If the original Genre is 'History.Military, Science Fiction.Alternate History, ReadMe' then the template returns 'Military, Alternate History, ReadMe'.

You could use this template in calibre's 'Edit metadata in bulk -> Search & replace with 'Search for' set to 'template' to strip off the first level of the hierarchy and assign the resulting value to Genre.

Note: the last line in the template, 'new_tags', isn't necessary in this case because 'for' returns the value of the last 'expression' in the 'expression list'.

For completeness, note that the entire template can be replaced by the subitems() function, as in
Code:
```
program: subitems(field('#genre'), 1, 0)
```
and
Code:
```
{#genre:subitems(1, 0)}
```