[GUI Plugin] Action Chains

[capink]

Here is a new beta containing the latest changes. I will wait for few days before releasing this version.

• Update: Chain Variables: Add option to loop over all books per var in the table.
• Update: Chain Variables: Right clicking value cell opens template editor.
• Update: Add new template function book_vars to access book specific variables set by custom actions (using chain_loop.set_book_vars). They are no longer accessible from the globals template function, to provide consistent behavior that enables set_globals to work the same everywhere.
• Fix: Formulas: Bug with mean value when fed an empty list or a list with no numerical values. It now handles it by returning 0.

The change in the bullet point no. 3 only affect people who write custom actions, and set book specific variables using chain_loop.set_book_vars. I suspect zero people are using this. But just in case, these variables are now accessed in template using the new function like this:

program:
    book_vars('my_var_name')

The mean() function now returns '0' if fed an empty list. Also there are other functions beside mean() that have the same problems with empty lists, for some of them it does not make sense to return '0' e.g min() and max(). So, they will keep returning the same error for now.

[capink]

There was a problem with the beta I posted in the previous post. I uploaded a new one now.

[chaley]

Thank you for this. I think that these additions increase flexibility, allowing someone to solve a problem the way they think about it.

Originally Posted by capink

• Update: Chain Variables: Add option to loop over all books per var in the table.

Question: is the order of evaluation guaranteed to be the order of the vars in the table? In other words, can the template for second variable be sure that the first variable has been defined?

Suggestion: move the "Iterate" checkbox to the second column. It took me a while to find it. I didn't think of scrolling.

Originally Posted by capink

• Update: Chain Variables: Right clicking value cell opens template editor.

Is it reasonable to open the template editor on double-click if the value starts with "program:"?

Originally Posted by capink

• Update: Add new template function book_vars to access book specific variables set by custom actions (using chain_loop.set_book_vars). They are no longer accessible from the globals template function, to provide consistent behavior that enables set_globals to work the same everywhere.

Is it reasonable to add a template function set_book_vars(), perhaps only in chain variable templates?

A note: book_vars() (and a possible set_book_vars()) uses a different syntax than globals() and set_globals(). book_vars() requires you to pass an expression that is usually a constant string. The globals() and set_globals() functions require that the arguments be variable names; they cannot be expressions. I think this difference is confusing.

I don't see a way to unify the syntax of book_vars() and globals() other than by changing the formatter to return the variable name as a string if that variable isn't defined. If I did that then book_vars(aaa) would be identical to book_vars('aaa'). On the other hand the following template would return the book title instead of raising an undefined variable error:

program:
    field(title)

The following template would also work, producing the result 'hello there' as long as the variables 'hello' and 'there' are undefined.

program:
    strcat(hello, ' ', there)

Is this a good thing or a bad thing? I am leaning toward 'good' but I am concerned about 'mysterious behavior'. For example, the following template returns 'foo there' because 'hello' has been assigned the value 'foo'.

program:
    hello = 'foo';
    strcat(hello, ' ', there)

Final and unrelated question: I am having trouble understanding "Selection options" in Selection Modifier. What if I don't want any of the 3 choices? For example, if I do a search then I want to select the books returned by that search. I don't want the results modified at all, but there isn't a radio button "Don't change selection".

Perhaps I am overthinking it, but it seems to me that the three "groups" in the dialog are executed in order. VLs are cleared if that box is checked, a search is done (or not), resulting in selected books, and then the options are applied.

[capink]

Originally Posted by chaley

Thank you for this. I think that these additions increase flexibility, allowing someone to solve a problem the way they think about it. Question: is the order of evaluation guaranteed to be the order of the vars in the table? In other words, can the template for second variable be sure that the first variable has been defined?

Currently no. I thought about it and then got lazy and was skeptical whether anyone would need it. So I divided the vars into two groups, first group that does not require iteration is run first. The second group which requires iteration over all books is run after that. Each group is ordered. I will modify in the next release.

Originally Posted by chaley

Suggestion: move the "Iterate" checkbox to the second column. It took me a while to find it. I didn't think of scrolling.

OK. I will also modify the aesthetics of the table in the final release. Giving it more width and controlling the width of columns a little better. For now, you can move the order for each action.

Originally Posted by chaley

Is it reasonable to open the template editor on double-click if the value starts with "program:"?

Sure.

Originally Posted by chaley

Is it reasonable to add a template function set_book_vars(), perhaps only in chain variable templates?

Sure, no problem with that. I don't see the reason to make it only available in chain variables.

Originally Posted by chaley

A note: book_vars() (and a possible set_book_vars()) uses a different syntax than globals() and set_globals(). book_vars() requires you to pass an expression that is usually a constant string. The globals() and set_globals() functions require that the arguments be variable names; they cannot be expressions. I think this difference is confusing.

Yes. I will highlight this when I release the version officially.

Originally Posted by chaley

Is this a good thing or a bad thing? I am leaning toward 'good' but I am concerned about 'mysterious behavior'. For example, the following template returns 'foo there' because 'hello' has been assigned the value 'foo'.

Not sure about this. I would say let's wait and see if there are any complaints. I don't like the familiar template language to change (unless absolutely necessary), or to have two ways to do the same thing. That might end up be more confusing for some people.

Originally Posted by chaley

Final and unrelated question: I am having trouble understanding "Selection options" in Selection Modifier. What if I don't want any of the 3 choices? For example, if I do a search then I want to select the books returned by that search. I don't want the results modified at all, but there isn't a radio button "Don't change selection".

Perhaps I am overthinking it, but it seems to me that the three "groups" in the dialog are executed in order. VLs are cleared if that box is checked, a search is done (or not), resulting in selected books, and then the options are applied.

Yes the steps are carried out in the same order.

I have no problem with adding the option to not change selection. But I don't think it will used much except except maybe to display a search at the end of the chain without selecting any books.

If you combine a search with this new option, the current library view will change to reflect the new search, but no book will be selected. You have to combine the search with "choose all books in current view" or any other option to have books be selected.

Edit: I will do it because it might have some uses.

[chaley]

Originally Posted by capink

Not sure about this. I would say let's wait and see if there are any complaints. I don't like the familiar template language to change (unless absolutely necessary), or to have two ways to do the same thing. That might end up be more confusing for some people.

I agree, especially because book_vars() is a proper function while globals() is a language construct. globals() assigns values to the variables named as arguments. book_vars() doesn't do the assignment but instead returns the value for the user to use as needed.

Yes the steps are carried out in the same order.

I have no problem with adding the option to not change selection. But I don't think it will used much except except maybe to display a search at the end of the chain without selecting any books.

If you combine a search with this new option, the current library view will change to reflect the new search, but no book will be selected. You have to combine the search with "choose all books in current view" or any other option to have books be selected.

Edit: I will do it because it might have some uses.

Another argument: my sample average rating chain selects the entire library to do the Single Field Edit action that assigns the result to all the books. I do that with "Clear Search" followed by "Select all books in current library view." After the Single Field Edit finishes I want to clear the selection. There isn't a way to do that.

I think that adding these two options cover all the possibilities:

• Clear selection: no books are selected.
• Select first book: the first book (line) in the view is selected. This option permits the chain to leave the library looking "normal".

Having either of the above is sufficient. Having both might be "gilding the lily".

Note that clearing the selection sort-of does select the first book. The "title" field is lightly highlighted. If you run Quickview docked then when the selection is cleared Quickview shows the information for the Author of the first book. Hitting the down-arrow button does an actual selection of the second book. Maybe this implies that the second option above is the best choice.

[capink]

I will add both.

[capink]

New beta version here with the following changes:

• Update: Selection Modifier: Add two new options: "Clear Selections", "Select first book in current view"
• Update: Add new template function set_book_vars.
• Update: Chain Variables: Guarantee order of processing of variables is same as order in table.
• Update: Chain Variables: Double clicking value cell will open template dialog if value start with "program:"

[capink]

The Action chains defines some extra template functions that can be used only within the plugin, to supplement and offer new functionality to calibre's template language. There are two classes of these template functions:

• Functions that are part of the formulas action. You learn more about these functions here. Also note that as of version 1.3.0 these formula functions can be used anywhere in the plugin where you have access to template dialog, not just in the formulas action. Most of these functions are used for mathematical calculations, like sum, mean, min, max ... etc.
  
  Two important functions included in the formulas action (and can also used elsewhere in the plugin); are the from_selection() and from_search() functions. They return a list of values fetched from specified columns. You can see examples of how they work here.
  
  
• Functions defined by the plugin outside the formulas action. These are the following functions:
  1. selection_count()
     This function returns the number of selected books in library view. Can be used anywhere in the plugin, including the conditions evaluator to set conditions of running actions/chains based on number of selected books.
     
  2. containing_folder()
     This functions returns the folder path for the current book.
     
  3. cover_path()
     This functions returns the cover path for the current book.
     
  4. sanitize_path()
     Given a filename, this function will return the same filename after removing illegal characters.
     
  5. set_book_vars()
     This functions allows you to set book specific variables:
     
     

     set_book_vars('my_name', 'my_value')

     This would assign 'my_value' to variable named 'my_name' for current_book.
     
     Note: This function stores variables in a volatile memory that only lasts for the lifetime of the chain. For storing variables persistently, check the set_persistent_vars() function below.
     
  6. book_vars()
     
     Access book specific variables (like those created by set_book_vars)
     

     book_vars('my_name')

     The above would return the value assigned to variable 'my_name' for current_book.
     
     Note: This function only accesses variables stored in a volatile memory that only lasts for the lifetime of the chain. For storing and accessing variables persistently, check the persistent_vars() and set_persistent_vars() functions below.
     
  7. category_items()
     
     This is how to use the new category_items() template function:
     
     

     program:
         category_items('#genre')

     This would return all the #genre tags. You can replace #genre with any other category like tags, authors .... etc
     
     It can also take second optional arg to limit the result to set of book_ids
     
     

     program:
         book_ids = from_selection('id');
         category_items('#genre',book_ids)

     from_selection() can be replaced with from_search() to achieve the same result.
     
     The resulting items are separated by one of the following characters(s):
     1. If the category contains names like the author field, the items are separated with ' & '.
     2. If the category is a field that accepts multiple items like tag, the items are separated with ', '
     3. If the category is a field that only accepts one value like series, the items are separated with ' ::: '. This is necessary because fields like series can contain commas AND/OR ampersands.
  8. last_modified()
     Return the time when database was last modified.
     
     
  9. book_field()
     
     Calls calibre's field() for a given book_id
     
     

     program:
         book_id = 123;
         book_field(book_id, 'title')

  10. book_raw_field()
      
      Calls calibre's raw_field() for a given book_id
      
      

      program:
          book_id = 123;
          book_raw_field(book_id, 'title')

[capink]

@chaley: I am having a little problem with the for loop, relating to a new template function I've added to the plugin. The function is called category_items(). It works as follows

program:
    category_items('#genre')

This would return all the #genre tags.

It can also take second optional arg to limit the result to set of book_ids

program:
    book_ids = from_selection('id');
    category_items('#genre',book_ids)

from_selection() can be replaced with from_search() to achieve the same result. I've updated both to accept 'id' as column name and return a list of book_ids.

The problem arise when I run the new function on a field containing names like 'authors'. I have to return a list separated with an ampersand, because author names can have commas in them. The documentation says the for loop will only accept a comma separated list of values. Can this modified to allow the user to somehow choose a different separator, maybe by setting a variable before running the for loop.

Another thing: I started making another function top_category_items() that is the same to the above function, but only extracts the top hierarchy level. I then starting having doubt, because I thought maybe combining the category_items() with subitem() would be a better way to achieve this and also offers more possibilities. The problem here is with that approach, I have to remove duplicates myself. It can be done using template but a lot of hassle. I ran recently into this problem in this template. The solution would be to add a uniq() template that removes duplicates from a list. I can easily add this, but I think it might be better to be included in the template builtin functions. Would you be willing to add something like this? I not, I will add to the plugin.

[chaley]

Originally Posted by capink

@chaley: I am having a little problem with the for loop, relating to a new template function I've added to the plugin. The function is called category_items(). It works as follows

program:
    category_items('#genre')

This would return all the #genre tags.

It can also take second optional arg to limit the result to set of book_ids

program:
    book_ids = from_selection('id');
    category_items('#genre',book_ids)

from_selection() can be replaced with from_search() to achieve the same result. I've updated both to accept 'id' as column name and return a list of book_ids.

Good idea. I have used the calibre built-in version of this function many times. Are you using it to do the implementation? It is calibre.db.cache.get_categories. It supports a list of book ids, does caching, and has several other optimizations. It returns a dict

['lookup_name': set_of_values, ...]

The tag browser uses this function to build the category list.

The problem arise when I run the new function on a field containing names like 'authors'. I have to return a list separated with an ampersand, because author names can have commas in them. The documentation says the for loop will only accept a comma separated list of values. Can this modified to allow the user to somehow choose a different separator, maybe by setting a variable before running the for loop.

I can do this. The big question is how? What I would prefer to do is extend the language for the for statement, making its grammar

'for' varname 'in' expression ['separator' expression] ':'

Specifying the separator is optional. OK with you?

Another thing: I started making another function top_category_items() that is the same to the above function, but only extracts the top hierarchy level. I then starting having doubt, because I thought maybe combining the category_items() with subitem() would be a better way to achieve this and also offers more possibilities. The problem here is with that approach, I have to remove duplicates myself. It can be done using template but a lot of hassle. I ran recently into this problem in this template. The solution would be to add a uniq() template that removes duplicates from a list. I can easily add this, but I think it might be better to be included in the template builtin functions. Would you be willing to add something like this? I not, I will add to the plugin.

You can already do exactly this with list_union, as in

list_union('', list_to_remove_duplicates, sep)

However, I realize that this use of list_union is rather esoteric, so I will add the function

list_remove_duplicates(list, sep)

If you think that list_uniq(), or perhaps list_unique(), is a better name than I can use that.

[capink]

Originally Posted by chaley

Good idea. I have used the calibre built-in version of this function many times. Are you using it to do the implementation? It is calibre.db.cache.get_categories. It supports a list of book ids, does caching, and has several other optimizations. It returns a dict

Yes. I am using this for the function. I wouldn't have done it without the get_categories().

Originally Posted by chaley

I can do this. The big question is how? What I would prefer to do is extend the language for the for statement, making its grammar

'for' varname 'in' expression ['separator' expression] ':'

I leave this to you. What you suggests would work nicely for me.

Originally Posted by chaley

Specifying the separator is optional. OK with you?

Definitely. It is better not to force people to specify the separator every time unnecessarily.

Originally Posted by chaley

However, I realize that this use of list_union is rather esoteric, so I will add the function

list_remove_duplicates(list, sep)

If you think that list_uniq(), or perhaps list_unique(), is a better name than I can use that.

Yours seems more descriptive and better.

[chaley]

Should list_remove_duplicates be case sensitive or case insensitive? Almost everything else in calibre is case insensitive so that is where I lean.

Note that in the case of case-insensitive matching, the last one in the list wins. Example: the input list 'A, A, a, a, b, B' produces the answer 'a, B'

[capink]

I think it should follow the pattern of being case insensitive.

[capink]

Version 1.2.3

• Update: Chain Variables: Add option to loop over all books per var in the table.
• Update: Chain Variables: Right clicking value cell opens template editor.
• Update: Add new template function book_vars to access book specific variables set by custom actions (using chain_loop.set_book_vars). They are no longer accessible from the globals template function, to provide consistent behaviour that enables set_globals to work the same everywhere.
• Fix: Formulas: Bug with mean value when fed an empty list or a list with no numerical values. It now handles it by returning 0.

Version 1.2.4

• Update: Selection Modifier: Add two new options: "Clear Selections", "Select first book in current view"
• Update: Add new template function set_book_vars.
• Update: Chain Variables: Guarantee order of processing of variables is same as order in table.
• Update: Chain Variables: Double clicking value cell will open template dialog if value start with "program:"

Version 1.3.0

• Update: Add new template function: category_items().
• Update: Chain Variables: Double clicking the value cell always opens the formula dialog.
• Fix: from_selection() and from_search() formulas always return a constant list of 1,2,3,4,5 when in any template dialog, not just in the formula dialog.

This is how to use the new category_items() template function:

program:
    category_items('#genre')

This would return all the #genre tags. You can replace #genre with any other category like tags, authors .... etc

It can also take second optional arg to limit the result to set of book_ids

program:
    book_ids = from_selection('id');
    category_items('#genre',book_ids)

from_selection() can be replaced with from_search() to achieve the same result.

The resulting items are separated by one of the following characters(s):

1. If the category contains names like the author field, the items are separated with ' & '.
2. If the category is a field that accepts multiple items like tag, the items are separated with ', '
3. If the category is a field that only accepts one value like series, the items are separated with ' ::: '. This is necessary because fields like series can contain commas AND/OR ampersands.

Also note that the previously intoduced book_vars() template function is now used to access book specific variables, instead of the old way of accessing them using the globals function. The both work differently; book_vars() requires you to pass an expression that is usually a constant string. The globals() function require that the arguments be variable names. This can be illustrated by the following examples:

program:
    globals(my_var)

program:
    book_vars('my_var')

[chaley]

@capink: specifying a separator in a for statement and the function list_remove_duplicates(l, sep) are now available in master source.