[chaley]

I have made some changes in how templates are evaluated to improve performance.

1) Improvements in the base functions that fetch the metadata used by templates. The performance improvement varies from 10 to 15%. For example, on my 20,000-book test library with three custom columns (int, text/tags-like, and composite), and sorting on the composite column (worst case), startup time went from 21 seconds to 19 seconds.

2) Compilation of general program mode templates. Calibre now tries to convert general program mode templates to python to improve their performance. The performance improvement depends on the complexity of the template. For simple templates (one function), it doesn't help much. For more complex templates, it can help a lot. For example, if the composite column in the 20,000-book library being sorted contains the following template:

program: 
#	fs = formats_sizes();
	fs = "";
	v = select(fs, 'PRC');
	v = add(v, select(fs, 'EPUB'));
	v = add(v, select(fs, 'MOBI'));
	v = add(v, select(fs, 'TXT'));
	v = add(v, select(fs, "\'LIT"));
	v = format_number(v, '{0:5.0f}');
	human_readable(v)

then the performance improvement is 30%. On my machine the startup times are:

Calibre 0.8.42:                   21.1 seconds
New version w/o compilation:      19.1 seconds
New version w/compilation:        14.6 seconds
Either, startup sorting on title:  7.0 seconds

It is possible that the generated code is incorrect (bugs). If this should happen to you, then you can turn off compilation by setting the tweak "Compile General Program Mode templates to Python" to False. Before you do that, please file a bug report. Provide the template and the generated code, which is displayed if you run calibre in debug mode.

Performance of single-function- and template-program-mode templates are improved by the changes described in 1) above. They are not compiled, so do not benefit from the changes described in 2).

Converting complicated template-program-mode templates to GPM will improve performance. I added a new template function to assist with this: finish_formatting. This function formats the value and adds a prefix and/or a suffix in the same way that the {||} syntax does in non-GPM templates. For example, the template

{series:re(([^\s])[^\s]+(\s|$),\1)}{series_index:0>2s| - | - }{title}

can be written using GPM as

program: 
    strcat(
        re(field('series'), '([^\s])[^\s]+(\s|$)', '\1'), 
        finish_formatting(field('series_index'), '0>2s', ' - ', ' - '),
        field('title')
    )