Template language changes

[chaley]

06 Apr 2021 (in calibre source)

• Add 'break' and 'continue' to the template language.
• Add a "Test" button to the Stored Templates preferences dialog.
  
  Spoiler:

  Clicking this button opens a subsidiary template tester that can 'see' the templates being created/edited in the preferences dialog, where you can write a template that test the stored templates. It shows the values using the books selected before starting the preferences dialog.
  
• Make TAB and SHIFT-TAB indent and unindent selected lines in the template tester.
• Add template file load/save buttons and to the template tester context menu. The buttons and the context menu items do the same thing.
• Add the ability to toggle word wrapping to the template tester context menu.
• Make the top button line reflow to two lines if the dialog box isn't wode enough

29 Mar 2021 (in calibre source)

• Add line-based breakpoints to the template tester
  
  Spoiler:

  The template tester now supports setting 'breakpoints' on lines. If a breakpoint is set then template evaluation pauses and a dialog is opened showing you the result of the expression, all the local variables, and a dropdown box that you can use to lookup metadata.
  You can see this and the new highlighting in this screen capture.
  
  
• Add syntax highlighting to identifiers and field references
• Add template file load/save to the context menu

26 Mar 2021 (in calibre version 5.14)

• Add new date formats 'to_number' and 'from_number'
  
  Spoiler:

  You can now convert dates to/from a floating point number representing the number of seconds since some platform-dependent start point. This number is sometimes called a 'timestamp'. Number-format dates are easier to compare and to modify using arithmetic. Details:

  • to_number : convert the date & time into a floating point number (a `timestamp`)
  • from_number : convert a floating point number (a `timestamp`) into an 'iso' formatted date. If you want a different date format then add the desired formatting string after 'from_number' and a colon (':'). Example:
    from_number:MMM dd yyyy

24 Mar 2021 (in calibre version 5.14)

• Addition of field references
  
  Spoiler:

  
  You can now use $lookup_key instead of field('lookup_key') and $$lookup_key instead of raw_field('lookup_key'). Examples:
  

    * $authors ==> field('authors')
    * $#genre ==> field('#genre')
    * $$pubdate ==> raw_field('pubdate')
    * $$#my_int ==> raw_field('#my_int')

21 Mar 2021 (in calibre version 5.14)

• Template tester - allow changing the font used in the template edit box
  
  Spoiler:

  You can now change both the font and the font size. Both are remembered.
  

20 Mar 2021 (in calibre version 5.14)

• Template tester - show template results for multiple books
  
  Spoiler:

  If You select multiple books before launching the template tester then it will show you the template evaluation result for each book. The order in the results list is the order the books were selected.
  
  Save the new dialog box size if it is changed.
  

11 Mar 2021 (in calibre version 5.14)

• Performance improvement of the virtual_libraries() template function
  
  Spoiler:

  The performance of the virtual_libraries function has been improved substantially. The change is most important for composite columns that call virtual_libraries; the template is evaluated for every book that is displayed in the book list/cover browser. For example, on my Windows machine using a test database with 3900 books, 15 virtual libraries, and one composite column that calls virtual_libraries(), the performance change i
  

  First call to virtual_libraries()
               old = 368,381 μs
               new = 382,352 μs
  
  Typical time for each call thereafter until data changes:
               old = 165,338 μs
               new =       1 μs

  Result: scrolling 20 books goes from 3.3 seconds to 20 microseconds.
  
  On a smaller library of 200 books with 5 virtual libraries the numbers are:
  

  First call to virtual_libraries()
               old = 7,852 μs
               new = 7,846 μs
  
  Typical time for each call thereafter, until data changes:
               old = 195 μs
               new =  <1 μs

  The performance improvement is directly related to the number of virtual libraries.
  

7 Mar 2021 (in calibre version 5.13):

• Fix regression introduced sometime in calibre V5: expression lists as parameters
  
  Spoiler:

  
  In calibre versions before V5.something one could use expression lists as parameters. Example:
  

  program:
      list_union(
  			a = 'xxx'; list_union('a', field('tags'), ','), 
  			field('#genre'),
  			','
  		)

  This 'feature' was certainly rarely used but it was possible. Fixed in the interest of compatibility.
  
• Allow expression lists in parenthesized expressions.
  
  Spoiler:

  As a consequence of the above, you can now put expression lists in parenthesized expressions. Example:
  

  program:
  	b = (
  				a = field('#mytextmult');
  				list_union(a, field('tags'), ',')
  			);
  	if a then
  		list_union(b, field('#genre'), ',')
  	else
  		b
  	fi

Earlier:

Spoiler:

28 Feb 2021 (in calibre version 5.13):

• Addition of common logical and arithmetic operators
  
  Spoiler:

  
  The template language now supports the common logical operators ('&&, '||', and '!), and arithmetic operators (unary '+', unary '', and binary '+', '-', '*', and '/').
  
  The operator precedence in the template language is now:

  • Function calls, constants, parenthesized expressions, statement expressions, assignment expressions. Remember that in the template language, 'if', 'for', and assignment return a value.
  • Unary plus (+) and minus (-). These operators evaluate right to left. These and the other arithmetic operators return integers if the expression doesn't produce a fractional part.
  • Multiply (*) and divide (/). These operators are associative and evaluate left to right. Use parentheses if you want to change the order of evaluation.
  • Add (+) and subtract (-). These operators are associative and evaluate left to right.
  • Numeric and string comparisons (these already existed). These operators return '1' if the comparison is True, otherwise ''. Comparisons are not associative: a < b < c produces a syntax error. Comparisons return '1' if True and '' if False.
  • Unary logical not (!). This operator returns '1' if the expression is False (evaluates to the empty string), otherwise ''.
  • Logical and (&&). This operator returns '1' if both the left-hand and right-hand expressions are True, the empty string '' if either is False, is associative, evaluates left to right, and does short-circuiting.
    
    Spoiler:

    Regarding short-circuiting: for example this program produces the answer '4'. Because of short-circuiting the right-hand expression, the assignment, is evaluated because the left-hand expression is True. The assignment is done.
    

    program:
    	a = 5;
    	'a' && (a = 4);
    	a

    This program produces '5' because the the left-hand expression is False so because of short-circuiting the right-hand expression is not evaluated. The assignment is not done.
    

    program:
    	a = 5;
    	'' && (a = 4);
    	a

  • Logical or (||). This operator returns '1' if either the left-hand or right-hand expression is True, '' if both are False, is associative, evaluates left to right, and does short-circuiting. It does an inclusive or, returning '1' if both the left- and right-hand expressions are True.

25 Feb 2021 (in calibre version 5.12):

• Function raw_field: new optional default value
  
  Spoiler:

  
  raw_field(field_name_expression[, default_value_expression]) -- if the value of the field is undefined (is 'None') return the default_value_expression if it is provided, otherwise return 'None'. Examples:
  

  v = raw_field('#my_int', -1000)

  returns the value of #my_int or -1000 if #my_int is undefined.
  

  v = raw_field('#my_int')

  returns the value of #my_int or None if #my_int is undefined.
  
  Note that multiple-value fields like 'tags' are never undefined, but are instead empty. The is_empty() function can be used in this case to provide a non-empty default.
  

23 Feb 2021:

• New template function list_count_matching(v, pattern, sep)
  
  Spoiler:

  list_count_matching(list, pattern, separator) -- interprets 'list' as a list of items separated by 'separator', returning the number of items in the list that match the regular expression 'pattern'. Aliases: list_count_matching(), count_matching()
  
• New infix compare operator 'in'.
  
  Spoiler:

  Usage:
  

  pattern_expression in value_expression

  Example:
  

  program:
      if '(dragon|lizard)' in field('series') then
          'yes'
      else
          'no'
      fi

  The 'in' operator is equivalent to
  

  contains(value_expression, pattern_expression, '1', '')

  The above 'if', written with 'contains', would be
  

  program:
      if contains(field('series'), '(dragon|lizard)' , '1', '') then
          'yes'
      else
          'no'
      fi

22 Feb 2021

• Add the possibility of specifying the separator in 'for' statements used to break the string into list items.
  
  Spoiler:

  Syntax:

  for <<id>> in <<value_expression>> separator <<expression>>:

  The keyword 'separator' and its associated expression are optional.
  
  Example:
  

  for a in some_authors separator '&':

• New function list_remove_duplicates()
  
  Spoiler:

  list_remove_duplicates(list, separator) -- return a list made by removing duplicate items in the source list. If items differ only in case, the last of them is returned. The items in source list are separated by separator, as are the items in the returned list.
  

20 Feb 2021

• Fix subitems() sometimes returning empty list items.

19 Feb 2021

• Fix 'for' not stripping leading and trailing blanks from the value assigned to the loop variable.

18 Feb 2021

• New function set_globals()
  
  Spoiler:

  set_globals(id[=expression] [, id[=expression]]*) -- Sets the value of "global variables" that can be passed into the formatter. If 'id' is not defined then set the global to the value of expression if supplied, otherwise ''. See the globals() function for more information.

17 Feb 2021:

• New function is_marked()
  
  Spoiler:

  is_marked() -- check whether the book is `marked` in calibre. If it is then return the value of the mark: either `true` (lower case) or the comma-separated list of named marks. Returns '' (the empty string) if the book is not marked. This function works only in the GUI.
  

17 Jan 2021 (in calibre version 5.11):

• The functions add() and multiply() can now take more than 2 arguments.
• New function field_exists()
  
  Spoiler:

  field_exists(field_name) -- checks if a field (column) named field_name exists, returning '1' if so and '' if not.

3 Jan 2021:

• New function list_split()
  
  Spoiler:

  list_split(list_val, sep, id_prefix) -- splits the list_val into separate values using 'sep', then assigns the values to variables named 'id_prefix_N' where N is the position of the value in the list. The first item has position 0 (zero). The function returns the last element in the list.
  
  Example: list_split('one, two, foo', ',', 'var') is equivalent to
  var_0 = 'one'; var_1 = 'two'; var_3 = 'foo'.
  
  Example: break apart and reformat a date expressed as a format and a list of values:
  

  program:
  # date_string would normally be field() to fetch the value from the book
  # A 'date string' is a list with the first the desired format and the rest the values
  	date_string = 'YMD,1984,12, 01';
  	list_split(date_string, ',', 'li');
  	if li_0 == 'YMD' then
  		strcat(li_1, '-', li_2, '-', li_3)
  	elif li_0 == 'YM'  then
  		strcat(li_1, '-', li_2)
  	elif li_0 == 'Y'  then
  		li_1
  	else
  		'Invalid Format'
  	fi

22 Dec 2020

• New function connected_device_uuid(storage_location)
  
  Spoiler:

  connected_device_uuid(storage_location) -- if a device is connected then return the device uuid (unique id), otherwise return the empty string. Each storage location on a device has a different uuid. The location names are 'main', 'carda' and 'cardb'. This function works only in the GUI.

19 Dec 2020

• Added a 'for' template language statement. See also 22 Feb 2021 above.
  
  Spoiler:

  
  

      for <<id>> in <<expression>>:
          <<expression_list>>
      rof

  The expression must evaluate to either a metadata field lookup key, for example 'tags or '#genre', or a comma-separated list of values. If the result is a valid lookup name then the field's value is fetched, otherwise the list is broken into its individual values. Each resulting value in the list is assigned to the variable 'id'' then the 'expression_list' is evaluated.
  
  Example: This template removes the first hierarchical name for each value in Genre ('#genre'), constructing a list with the new names.
  

          program:
              new_tags = '';
              for i in '#genre':
                  j = re(i, '^.*?\.(.*)$', '\1');
                  new_tags = list_union(new_tags, j, ',')
              rof;
              new_tags

  If the original Genre is 'History.Military, Science Fiction.Alternate History, ReadMe' then the template returns 'Military, Alternate History, ReadMe'.
  
  You could use this template in calibre's 'Edit metadata in bulk -> Search & replace with 'Search for' set to 'template' to strip off the first level of the hierarchy and assign the resulting value to Genre.
  
  Note: the last line in the template, 'new_tags', isn't necessary in this case because 'for' returns the value of the last 'expression' in the 'expression list'.
  
  For completeness, note that the entire template can be replaced by the subitems() function, as in
  

  program: subitems(field('#genre'), 1, 0)

  and
  

  {#genre:subitems(1, 0)}

[chaley]

Reserved

[ownedbycats]

Perhaps this thread should be stickied?

[chaley]

Looks like "someone" did it.

I am not yet convinced that this thread should take up sticky space but we will see.

[theducks]

Originally Posted by chaley

Looks like "someone" did it. .

Peter T did it
I think <=6 should be the Max # of stickies per forum.

[capink]

I suggest removing the "Search is showing all books with some highlighted! How do I fix this?" thread from the stickies. I think/hope the issue causing this (in the Find Duplicates plugin) is fixed now. Also there is a similar thread in the main forum.

[ownedbycats]

Originally Posted by capink

I suggest removing the "Search is showing all books with some highlighted! How do I fix this?" thread from the stickies. I think/hope the issue causing this (in the Find Duplicates plugin) is fixed now. Also there is a similar thread in the main forum.

And nobody pays attention to either of them, considering the amount of threads about it...

[BetterRed]

Originally Posted by theducks

Peter T did it
I think <=6 should be the Max # of stickies per forum.

Agreed

Originally Posted by capink

I suggest removing the "Search is showing all books with some highlighted! How do I fix this?" thread from the stickies. I think/hope the issue causing this (in the Find Duplicates plugin) is fixed now. Also there is a similar thread in the main forum.

Done

BR

[chaley]

28 Feb 2021 (in calibre source): Addition of common logical and arithmetic operators

The template language now supports the common logical operators ('&&, '||', and '!), and arithmetic operators (unary '+', unary '-', and binary '+', '-', '*', and '/').

The operator precedence in the template language is now:

• Function calls, constants, parenthesized expressions, statement expressions, assignment expressions. Remember that in the template language, 'if', 'for', and assignment return a value.
• Unary plus (+) and minus (-). These operators evaluate right to left. These and the other arithmetic operators return integers if the expression doesn't produce a fractional part.
• Multiply (*) and divide (/). These operators are associative and evaluate left to right. Use parentheses if you want to change the order of evaluation.
• Add (+) and subtract (-). These operators are associative and evaluate left to right.
• Numeric and string comparisons (these already existed). These operators return '1' if the comparison is True, otherwise ''. Comparisons are not associative: a < b < c produces a syntax error. Comparisons return '1' if True and '' if False.
• Unary logical not (!). This operator returns '1' if the expression is False (evaluates to the empty string), otherwise ''.
• Logical and (&&). This operator returns '1' if both the left-hand and right-hand expressions are True, the empty string '' if either is False, is associative, evaluates left to right, and does short-circuiting.
  
  Spoiler:

  Regarding short-circuiting: for example this program produces the answer '4'. Because of short-circuiting the right-hand expression, the assignment, is evaluated because the left-hand expression is True. The assignment is done.
  

  program:
  	a = 5;
  	'a' && (a = 4);
  	a

  This program produces '5' because the the left-hand expression is False so because of short-circuiting the right-hand expression is not evaluated. The assignment is not done.
  

  program:
  	a = 5;
  	'' && (a = 4);
  	a

• Logical or (||). This operator returns '1' if either the left-hand or right-hand expression is True, '' if both are False, is associative, evaluates left to right, and does short-circuiting. It does an inclusive or, returning '1' if both the left- and right-hand expressions are True.

[BetterRed]

Originally Posted by chaley

28 Feb 2021 (in calibre source): Addition of common logical and arithmetic operators

The template language now supports the common logical operators ('&&, '||', and '!), and arithmetic operators (unary '+', unary '', and binary '+', '-', '*', and '/').

Delete this if you wish.

BR

[PeterT]

Originally Posted by BetterRed

Delete this if you wish.



BR

I think it's meant to be unary "-"

[chaley]

Originally Posted by BetterRed

BR

Originally Posted by PeterT

I think it's meant to be unary "-"

Yes, it was. Typo fixed. Thanks.

However, an empty unary operator does raise interesting questions of syntax and semantics.

[PeterT]

Lol!!

[BetterRed]

Originally Posted by PeterT

I think it's meant to be unary "-"

I assumed "" was a message for me in particular

BR

Hint - look at the title under my user name

[PeterT]

Oops. I'd missed that