Template language changes

[chaley]

Originally Posted by un_pogaz

Oh, I'm interested, I had used the template dialog and it took me a while to stop panicking about all the exceptions I had introduced in my code.
But how do I use it? I add "show_stack_traces_in_formatter = False" in any tweark configuration or I edit directly "tweaks.json" ?

Use the "Plugin tweaks" section. It is for tweaks that calibre doesn't know about, usually for plugins but also for "secret tweaks" like these. See the attached screen captures.

[chaley]

23 July 2022 (in calibre 6.2.1):

• Performance improvement of the switch() function
  The switch function now does "shortcutting". It evaluates the comparison expressions in turn. If the comparison succeeds then the result expression for that comparison is evaluated and returned. No other result expressions are evaluated. This provides a performance improvement of at least 50%. It can be much higher if the switch() has many compare/result pairs and they are organized in decreasing probability order.
  
  Behavior change: because most of the result expressions and some of the compare expressions are not evaluated, expressions with side effects that formerly were evaluated might not be. You can no longer expect side effects to work. For example, you shouldn't use expressions that contain assignments in a switch().
  
• Add a tooltip in the column icon advanced rule editor explaining that an advanced rule can return a compound icon by separating the individual icons (file names) with a ':' (colon).

[chaley]

14 Sept 2022 (in calibre 6.5.0)

• New function strcmpcase()
  

  strcmpcase(x, y, lt, eq, gt)

  does a case-sensitive lexical comparison of x and y. Returns lt if x < y, eq if x == y, otherwise gt.
  
  Note: This is NOT the default behavior used by calibre, for example, in the lexical comparison operators (==, >, <, etc.). This function could cause unexpected results, preferably use ``strcmp()`` whenever possible.
  
  Thanks to @un_pogaz for this function.

[chaley]

16 Sept 2022 (in calibre 6.5.0)

• New function to_hex()
  to_hex(val) -- returns the string val encoded in hex. This is useful when constructing calibre URLs.

[chaley]

12 Oct 2022 (in calibre 6.7.0)

• Python Template Mode (PTM) lets you write templates using native python and the calibre API. The database API will be of most use. PTM templates are much faster than the other template modes and can do more complicated operations, but you must know how to write code in python using the calibre API.
  
  A PTM template looks like this:
  

  python:
  def evaluate(book, context):
      # book is a calibre metadata object
      # context is an instance of calibre.utils.formatter.PythonTemplateContext,
      # which (currently) contains the following attributes:
      # db: a calibre legacy database object
      # globals: the template global variable dictionary
      # arguments: is a list of arguments if the template is called by a GPM template, otherwise None
  
      # your Python code goes here
      return 'a string'

  You can add the above text to your template using the context menu, usually accessed with a right click. The comments are not significant and can be removed. You must use python indenting.
  
  The context object supports str(context) that returns a string of the context's contents, and context.attributes that returns a list of the attribute names in the context.
  
  Here is an example of a PTM template that produces a list of all the authors for a series. The list is stored in a Column built from other columns, behaves like tags. It shows in Book details and has on separate lines checked in Preferences->Look & feel->Book details. That option requires the list to be comma-separated. To satisfy that requirement the template converts commas in author names to semicolons then builds a comma-separated list of authors. The authors are then sorted, which is why the template uses author_sort.
  

  python:
  def evaluate(book, context):
      if book.series is None:
          return ''
      db = context.db.new_api
      ans = set()
      # Get the list of books in the series
      ids = db.search(f'series:"={book.series}"', '')
      if ids:
          # Get all the author_sort values for the books in the series
          author_sorts = (v for v in db.all_field_for('author_sort', ids).values())
          # Add the names to the result set, removing duplicates
          for aus in author_sorts:
              ans.update(v.strip() for v in aus.split('&'))
      # Make a sorted comma-separated string from the result set
      return ', '.join(v.replace(',', ';') for v in sorted(ans))

  The output in Book details looks like this:
  

[chaley]

29 Oct 2022 (in calibre 6.8.0)

• Python templates can call stored templates and formatter functions.
  You can call a Python template (PTM), a template language template (GPM), or built-in or user template functions. Syntax:
  

  context.funcs.name(arguments)

  For example, this python template uses the built-in template() function to format the title and authors. It then checks if the book is in a series, and if so appends the series name and the number of books in that series:
  

  python:
  def evaluate(book, context):
  	# Use the built-in template function to format the title and authors
  	x = context.funcs.template('{Title} - {authors}')
  	# Count the books in the series. First get the series name
  	series = book.get('series')
  	if series:
  		# The book is in a series. Get the count of books in that series
  		db = context.db.new_api
  		series_id = db.get_item_id('series', series)
  		book_count = len(db.books_for_field('series', series_id))
  		# Append the number of books to the title - author string
  		x = x + f' --- In the series "{series}, which has {book_count} book{"s" if book_count > 1 else ""}'
  	else:
  		x = x + " --- This book isn't in a series"
      return x

  This screen capture shows the results:
  
  
  If a function or template name name is also a Python keyword then append an underscore, as in 'template_()'
• Breakpoints for Python templates in the Template tester dialog.
  

[chaley]

15 Nov 2022 (in calibre 6.9.0)

• The command line utility calibredb list can now use a template as a field (column).
  
  Use the field "template" to add the output of a template to each book output by calibredb list. The template itself can be on the command line or in a file.
  
  The documentation, with the new template-related options in bold:
  
  Spoiler:

  calibredb list --help
  Usage: calibredb.exe list [options]
  
  List the books available in the calibre database.
  
  Whenever you pass arguments to calibredb.exe that have spaces in them, enclose the arguments in quotation marks. For example: "C:\some path with spaces"
  
  Options:
    -f FIELDS, --fields=FIELDS
                          The fields to display when listing books in the
                          database. Should be a comma separated list of fields.
                          Available fields: author_sort, authors, comments,
                          cover, formats, identifiers, isbn, languages,
                          last_modified, pubdate, publisher, rating, series,
                          series_index, size, tags, template, timestamp, title,
                          uuid
                          Default: title,authors. The special field "all" can be
                          used to select all fields. In addition to the builtin
                          fields above, custom fields are also available as
                          *field_name, for example, for a custom field #rating,
                          use the name: *rating
  
    --sort-by=SORT_BY     The field by which to sort the results.
                          Available fields: author_sort, authors, comments,
                          cover, formats, identifiers, isbn, languages,
                          last_modified, pubdate, publisher, rating, series,
                          series_index, size, tags, template, timestamp, title,
                          uuid
                          Default: id
  
    --ascending           Sort results in ascending order
  
    -s SEARCH, --search=SEARCH
                          Filter the results by the search query. For the format
                          of the search query, please see the search related
                          documentation in the User Manual. Default is to do no
                          filtering.
  
    -w LINE_WIDTH, --line-width=LINE_WIDTH
                          The maximum width of a single line in the output.
                          Defaults to detecting screen size.
  
    --separator=SEPARATOR
                          The string used to separate fields. Default is a
                          space.
  
    --prefix=PREFIX       The prefix for all file paths. Default is the absolute
                          path to the library folder.
  
    --limit=LIMIT         The maximum number of results to display. Default: all
  
    --for-machine         Generate output in JSON format, which is more suitable
                          for machine parsing. Causes the line width and
                          separator options to be ignored.
  
    --template=TEMPLATE   The template to run if "template" is in the field
                          list. Default: None
  
    -t TEMPLATE_FILE, --template_file=TEMPLATE_FILE
                          Path to a file containing the template to run if
                          "template" is in the field list. Default: None
  
    --template_heading=TEMPLATE_HEADING
                          Heading for the template column. Default: template.
                          This option is ignored if the option --for-machine is
                          set
  
    GLOBAL OPTIONS:
      --library-path=LIBRARY_PATH, --with-library=LIBRARY_PATH
                          Path to the calibre library. Default is to use the
                          path stored in the settings. You can also connect to a
                          calibre Content server to perform actions on remote
                          libraries. To do so use a URL of the form:
                          http://hostname:port/#library_id for example,
                          http://localhost:8080/#mylibrary. library_id is the
                          library id of the library you want to connect to on
                          the Content server. You can use the special library_id
                          value of - to get a list of library ids available on
                          the server. For details on how to setup access via a
                          Content server, see https://manual.calibre-
                          ebook.com/generated/en/calibredb.html.
  
      -h, --help          show this help message and exit
  
      --version           show program's version number and exit
  
      --username=USERNAME
                          Username for connecting to a calibre Content server
  
      --password=PASSWORD
                          Password for connecting to a calibre Content server.
                          To read the password from standard input, use the
                          special value: <stdin>. To read the password from a
                          file, use: <f:C:/path/to/file> (i.e. <f: followed by
                          the full path to the file and a trailing >). The angle
                          brackets in the above are required, remember to escape
                          them or use quotes for your shell.
  
      --timeout=TIMEOUT   The timeout, in seconds, when connecting to a calibre
                          library over the network. The default is two minutes.
  
  
  Created by Kovid Goyal <kovid@kovidgoyal.net>

  
  Example: produce a list of books with a series matching "great" and output the books with the authors of all the books in the series:
  The command:
  
  Spoiler:

  calibredb list -f title,series,template -t templates/authors_for_series.txt \
                 -w 110 -s series:great --template_heading="Series Authors"

  
  The template (in a file). The template can be written in any of the template language modes. This one is in python mode:
  
  Spoiler:

  python:
  def evaluate(book, context):
      if book.series is None:
          return ''
      db = context.db.new_api
      ans = set()
      # Get the list of books in the series
      series = book.series
      if series in context.globals:
          return context.globals[series]
      ids = db.search(f'series:"={series}"', '')
      if ids:
          # Get all the author_sort values for the books in the series
          author_sorts = (v for v in db.all_field_for('author_sort', ids).values())
          # Add the names to the result set, removing duplicates
          for aus in author_sorts:
              ans.update(v.strip() for v in aus.split('&'))
      # Make a sorted comma-separated string from the result set
      res = ' & '.join(v for v in sorted(ans))
      context.globals[series] = res
      return res

  
  The output:
  
  Spoiler:

  id   title                      series       Series Authors
  1297 My: Books                  Great Series B, A & C, B & Flintstone, Wilma & Machiavelli, Niccolò &
                                               Nietzsche, Friedrich & Papper, A. Persone B. C. & Public, John
  1300 Foo & Bar                  Great Series B, A & C, B & Flintstone, Wilma & Machiavelli, Niccolò &
                                               Nietzsche, Friedrich & Papper, A. Persone B. C. & Public, John
  1313 Ünknown2 (foo)             Great Series B, A & C, B & Flintstone, Wilma & Machiavelli, Niccolò &
                                               Nietzsche, Friedrich & Papper, A. Persone B. C. & Public, John
  1338 Udknown3                   Great Series B, A & C, B & Flintstone, Wilma & Machiavelli, Niccolò &
                                               Nietzsche, Friedrich & Papper, A. Persone B. C. & Public, John
  1339 Put the Lime in the        Great Series B, A & C, B & Flintstone, Wilma & Machiavelli, Niccolò &
       Coconut                                 Nietzsche, Friedrich & Papper, A. Persone B. C. & Public, John
  1354 The Birth of Tragedy       Great Series B, A & C, B & Flintstone, Wilma & Machiavelli, Niccolò &
                                               Nietzsche, Friedrich & Papper, A. Persone B. C. & Public, John

[chaley]

06 Jan 2023 (in calibre 6.12):

• New function switch_if()
  

  switch_if([test_expression, value_expression,]+ else_expression)

  For each "test_expression, value_expression" pair, checks if test_expression is True (non-empty) and if so returns the result of value_expression. If no test_expression is True then the result of else_expression is returned. You can have as many "test_expression, value_expression" pairs as you want.

[chaley]

08 Apr 2023 (in calibre 6.15)

• Changed function series_sort().
  This function now uses the book's language when doing article processing.

[chaley]

24 Apr 2023 (in calibre 6.17)

New template functions:

• has_extra_data()
  

  has_extra_data()

  Returns 'Yes' if there are any extra files for the book (files in the folder data/ in the book's folder), otherwise '' (the empty string). See also the functions extra_file_names(), extra_file_size() and extra_file_modtime() This function can be used only in the GUI.
  
  Note: this function will change in calibre 6.18
• extra_file_names()
  

  extra_file_names(sep)

  Returns a sep-separated list of extra files in the book's data/ folder. See also the functions has_extra_files(), extra_file_size() and extra_file_modtime(). This function can be used only in the GUI.
  
  Note: this function will change in calibre 6.18
• extra_file_size()
  

  extra_file_size(file_name)

  Returns the size in bytes of the extra file file_name in the book's data/ folder if it exists, otherwise -1. See also the functions has_extra_files(), extra_file_names() and extra_file_modtime(). This function can be used only in the GUI.
• extra_file_modtime()
  

  extra_file_modtime(file_name, format_spec)

  Returns the modification time of the extra file file_name in the book's data/ folder if it exists, otherwise -1. The modtime is formatted according to format_spec (see format_date() for details). If format_spec is the empty string, returns the modtime as the floating point number of seconds since the epoch. See also the functions has_extra_files(), extra_file_names() and extra_file_size(). The epoch is OS dependent. This function can be used only in the GUI.

[BetterRed]

I want to show an icon if the data folder contains a file named todo.???? (might be ToDo.txt, TODO.EML, todo.stky, etc).

Something like extra_file_exists("todo.*") might be useful.

BR

[chaley]

Originally Posted by BetterRed

I want to show an icon if the data folder contains a file named todo.???? (might be ToDo.txt, TODO.EML, todo.stky, etc).

Something like extra_file_exists("todo.*") might be useful.

BR

program:
	files = extra_file_names(':');
	in_list(files, ':', 'todo\..*', 'Yes', '')

EDIT: where 'Yes' would be the icon name

[ownedbycats]

Would it make sense to include an explicit extra_file_count()? I think its currently possible with existing list functions (forgot exactly).

[chaley]

Originally Posted by ownedbycats

Would it make sense to include an explicit extra_file_count()? I think its currently possible with existing list functions (forgot exactly).

I am not interested in adding a plethora of specific functions that are trivially implementable with existing functions.

program:
	list_count(extra_file_names(':'), ':')

or this Python template:

python:
def evaluate(book, context):
	from calibre.gui2.ui import get_gui
	extra_files = get_gui().current_db.new_api.list_extra_files(book.id, use_cache=True)
	return str(len(extra_files)) if extra_files else ''

Edit: The Python template will be faster, which might be important if this is used for something like column icons.

[chaley]

Originally Posted by ownedbycats

Would it make sense to include an explicit extra_file_count()? I think its currently possible with existing list functions (forgot exactly).

Changing has_extra_files() to return a count instead of "Yes" is a good idea. I will change it for the next release.

To get "Yes" with the changed version you would do

program:
	if has_extra_files() then 'Yes' fi