series_index manipulation

[kcz]

Hello,

I want to use/create a function that returns the series_index as follows:

If the series_index is an integer (or better said the numbers after the decimal point are 00) return it as 0>2s, if it is a float then return it as 0>5.2f

e.g.:

INPUT      OUTPUT

2.00       02
2.50       02.50
3.01       03.01
04.00      04

How can I do that?

[chaley]

You don't give a context for where you want to use this value. I am assuming you want it in some save- or send- template.

One way using general program mode:

program: re(format_number(field('series_index'), '{0:05.2f}'), '\.00', '')

You would add any other fields to re's return value as needed, possibly using the template or finish_formatting functions.

Another way using template program mode:

{:'template("program: re(format_number(field('series_index'), '[[0:05.2f]]'), '\.00', '')")'}

Using "template" is required here because you cannot use { and } in template program mode and these characters are required by format_number. The template function converts the [[ and ]] to { and }.

Yet another way: write your own user defined template function to do what you want.

[kcz]

Thank you very much for your reply.

Originally Posted by chaley

You don't give a context for where you want to use this value. I am assuming you want it in some save- or send- template.

Yes, I am sorry. I want to use it in a metadata plugboard, so the easiest would be to use a custom column (built from other columns).

Can I use a program right in the custom column template, or do I have to create a user defined template function? I am new to calibre and python, so please forgive my dumb questions.

[chaley]

Originally Posted by kcz

Yes, I am sorry. I want to use it in a metadata plugboard, so the easiest would be to use a custom column (built from other columns).

You can put a template directly into the metadata plugboard box. You can get to the advanced template editor by right-clicking in the box.

Can I use a program right in the custom column template, or do I have to create a user defined template function?

It depends on what you mean by "program". If you mean a general program mode template such as I provided above then yes, you can put them directly into the custom column template. The easiest way to do this is to create the column with a simple template, click on a cell in that column, then press F2 to open the template program editor.

If you mean a real python program then no. To use python you must create a user defined function, then call that function from a template just as you would any other template function.

[kcz]

Originally Posted by chaley

You can put a template directly into the metadata plugboard box. You can get to the advanced template editor by right-clicking in the box.

Yes, this works, but I rather use a custom column for that first and use the custom column in the plugboard.

Originally Posted by chaley

It depends on what you mean by "program". If you mean a general program mode template such as I provided above then yes, you can put them directly into the custom column template. The easiest way to do this is to create the column with a simple template, click on a cell in that column, then press F2 to open the template program editor.

For some reason using F2 in the custom column creation does not work. I'm on a Mac, so maybe that's the reason. Anyway, if I paste your second example into the field, it works perfectly. Thank you.

Originally Posted by chaley

If you mean a real python program then no. To use python you must create a user defined function, then call that function from a template just as you would any other template function.

Ok, this seems pretty straight forward. I guess it makes more sense to use a user defined template function when the code is more complex or the function is used more often.

Cheers.

[chaley]

Originally Posted by kcz

For some reason using F2 in the custom column creation does not work. I'm on a Mac, so maybe that's the reason. Anyway, if I paste your second example into the field, it works perfectly. Thank you.

To use F2 (edit field) you must have already created the composite column. Create the column with a simple template. While looking at the books spreadsheet, click in any cell in the new column then do what the mac expects to edit the column.

Ok, this seems pretty straight forward. I guess it makes more sense to use a user defined template function when the code is more complex or the function is used more often.

Defining a user-defined function is usually driven by complexity or performance. Sometimes the template language doesn't have the power to do what is needed or the solution is so cumbersome that it is absurd. Sometimes the performance of the template language solution is unacceptable, which can happen when several "test" conditions are being evaluated because the template interpreter does not shortcut. All arguments (conditions) are computed, then the conditions are evaluated.

[kcz]

Originally Posted by chaley

Defining a user-defined function is usually driven by complexity or performance. Sometimes the template language doesn't have the power to do what is needed or the solution is so cumbersome that it is absurd. Sometimes the performance of the template language solution is unacceptable, which can happen when several "test" conditions are being evaluated because the template interpreter does not shortcut. All arguments (conditions) are computed, then the conditions are evaluated.

I have created a user defined template function called format_index with 1 argument. Here is the code:

def evaluate(self, formatter, kwargs, mi, locals, val):
    return re.sub('\.00', '', format_number(field(val), '[[0:05.2f]]'), flags=re.I)

The template in the custom column looks like this:

{series_index:format_index()}

Unfortunately I get the following error:

TEMPLATE ERROR global name 're' is not defined

Any idea what the problem is?

[chaley]

Originally Posted by kcz

I have created a user defined template function called format_index with 1 argument. Here is the code:

def evaluate(self, formatter, kwargs, mi, locals, val):
    return re.sub('\.00', '', format_number(field(val), '[[0:05.2f]]'), flags=re.I)

The template in the custom column looks like this:

{series_index:format_index()}

Unfortunately I get the following error:

TEMPLATE ERROR global name 're' is not defined

Any idea what the problem is?

You are conflating the template language and python, and also not importing the necessary python modules. When writing user defined functions you must use python and only python.

One version of your function in python is

def evaluate(self, formatter, kwargs, mi, locals, val):
	import math
	# Check if the argument is empty. If so, return the empty string
	if val == '':
		return ''
	# Not empty. Must be a number. If it isn't, bad things will happen.
	v = float(val)
	# If it is an integer, return an 2-digit string zero padded
	if math.floor(v) == v:
		return '{:02.0f}'.format(v)
	# Not an integer. Return a 2.2 digit string.
	return '{:05.2f}'.format(v)

[kcz]

Originally Posted by chaley

You are conflating the template language and python, and also not importing the necessary python modules. When writing user defined functions you must use python and only python.

Hmm, this is interesting. I was using the code of other functions as a template.
e.g. look at the re function. Very strange.
On my installation the re function looks like this:

def evaluate(self, formatter, kwargs, mi, locals, val, pattern, replacement):
    return re.sub(pattern, replacement, val, flags=re.I)

This is all very confusing.

There are so many things that do not make any sense to me. Let's say I want to return either an integer or a float depending on the occurrences within a series. e.g. let's say there are 3 books in series x and the 2nd book has 1.50 as the index value, so a function should return a float in the format aa.bb for all the books in the series.

technically you would write a function which loops through all the books in a series and if one index is not an integer then return the current index as aa.bb (even if the current one is an integer like 2.00)

series x: book one - 1.00, book 2 - 1.50, book 3 - 3.00, so the function would return 01.00, 01.50, 03.00.
If the second book had an index of 2.00, then the function would return 01, 02, 03.
(the funcion is called once for every book in the series)

But I have no idea how to even reference the books in a series in python. Maybe this is not even possible. I'm used to code in C and thus would access the data via dynamic sql, ran the loop once and stored the result that one or more values are a float in shared memory. In that case I had a map which told me series x had a float or not.
whenever I call the function, it would check if an entry in the map existed and if not it'd use above routine to generate the entry in the map.

[chaley]

Originally Posted by kcz

Hmm, this is interesting. I was using the code of other functions as a template.
e.g. look at the re function. Very strange.
On my installation the re function looks like this:

def evaluate(self, formatter, kwargs, mi, locals, val, pattern, replacement):
    return re.sub(pattern, replacement, val, flags=re.I)

That is accidental. See below

This is all very confusing.

The first and most important thing is that the template language is not python. I defined it over time to meet our needs. The language is interpreted by a custom language interpreter. Calibre can also translate the language into python, but only as a sequence of calls to the python implementation of template functions in the interpreter.

It is an accident that the template function and the python module are both named "re". If you look at the template function, its parameter profile is totally different from the one in the python module.

There are so many things that do not make any sense to me. Let's say I want to return either an integer or a float depending on the occurrences within a series. e.g. let's say there are 3 books in series x and the 2nd book has 1.50 as the index value, so a function should return a float in the format aa.bb for all the books in the series.

Template function should (actually, must) always return a string. That is the only datatype that the interpreter understands. If you return something else then it might work, but only because python is very willing to coerce types when needed.

In your example, you must return strings '1', '1.50', and '2', or some such. You would use python expressions to accomplish this.

technically you would write a function which loops through all the books in a series and if one index is not an integer then return the current index as aa.bb (even if the current one is an integer like 2.00)

series x: book one - 1.00, book 2 - 1.50, book 3 - 3.00, so the function would return 01.00, 01.50, 03.00.
If the second book had an index of 2.00, then the function would return 01, 02, 03.
(the funcion is called once for every book in the series)

If I understand correctly, you want to choose the format of the series for a particular book based on the union of information from all books. You cannot do this in a template function. These functions can see only the book you are working on. They cannot see data for any other book.

But I have no idea how to even reference the books in a series in python. Maybe this is not even possible. I'm used to code in C and thus would access the data via dynamic sql, ran the loop once and stored the result that one or more values are a float in shared memory. In that case I had a map which told me series x had a float or not.
whenever I call the function, it would check if an entry in the map existed and if not it'd use above routine to generate the entry in the map.

You would do something similar in python, but using calibre's data access layer. The problem is that a template function does not have access to the db, and in fact cannot have access to it. The reason: template functions can be evaluated in contexts where the db must not be opened, such as worker processes or file name generation during send-to operations.

Doing what you want requires writing a plugin. These can see the database and can do completely arbitrary things such as writing values into custom columns. Alternatively you could change calibre itself to do what you want.

[kcz]

Thanks again for the detailed information. Your explanations help a lot.

Originally Posted by chaley

If I understand correctly, you want to choose the format of the series for a particular book based on the union of information from all books. You cannot do this in a template function. These functions can see only the book you are working on. They cannot see data for any other book.

Yes, pretty much. I want to choose the format of the series index based on information of the series indexes within that series.
The format should depend on the:
- max number of digits of the integer part
- max number of digits of the fractional part

e.g. one book in the series has the index 13.203 (13 and 203 are the biggest numbers for the respective floating number parts), in which case the function should return the index for every book in this series as xx.yyy
(it's just an example. it is rather unlikely that book 13 has 203 sub books )
If another series has only numbers with fractional parts of .00 and the biggest index number is 13.00, then it should be formatted as xx

You would do something similar in python, but using calibre's data access layer. The problem is that a template function does not have access to the db, and in fact cannot have access to it. The reason: template functions can be evaluated in contexts where the db must not be opened, such as worker processes or file name generation during send-to operations.

Makes sense. Is there a description of the calibre data access layer? I found info on the plugin api and some other info, but nothing on the db layer.

Doing what you want requires writing a plugin. These can see the database and can do completely arbitrary things such as writing values into custom columns. Alternatively you could change calibre itself to do what you want.

There is the next problem. I found information on the APIs for the different plugins, but the info does not tell me which plugin type I would have to use.
The manual also does not tell me how to use 'hooks' within calibre.

There are several ways how someone could accomplish this formatting function:

1) create a function to be used by the template language

This function would calculate the 2 info points (as described above) every time it is called. Rather ineffecient, since it would have to loop through all books in the series to find the data.
How do I even write a plugin that provides a function for the template language?

2) create a function plus a meta data table for the series

A new table with 3 columns (string series, int max_int, int max_fract). In this case calibre would have to update this table, every time a series index is changed/added/deleted. Are there any hooks in calibre or would I have to write this part as well?
In this case the function would just access this table and format the current index accordingly.

The latter is much more performant, but also more complex.

In any case, the documentation did not answer any of my questions and I am afraid I do not have the time to get familiar with the calibre code base and the plugin interfaces, which I would had to do to be actually able to implement my formatting function.

I hoped that calibre would already provide a functionality like this, since it seems logical to me that someone would like to have aligned indexes in a filename without having to hardcode the padding. e.g. who wants a filename like 'seriesname [007.000] bookname', if the biggest index is 7.00 and the series has only integers?

I just hope that I will find the time to implement something like this in the future.

[chaley]

Originally Posted by kcz

Makes sense. Is there a description of the calibre data access layer? I found info on the plugin api and some other info, but nothing on the db layer.

No. You must look at source and examples. Existing plugins might be the best place to look for examples, because they are smaller and it is easier to determine what they are trying to do.

There is the next problem. I found information on the APIs for the different plugins, but the info does not tell me which plugin type I would have to use.
The manual also does not tell me how to use 'hooks' within calibre.

The only type I can imagine using successfully would be a user interface plugin. You would run the plugin on demand to compute the necessary numbers.

There are several ways how someone could accomplish this formatting function:

1) create a function to be used by the template language

This function would calculate the 2 info points (as described above) every time it is called. Rather ineffecient, since it would have to loop through all books in the series to find the data.
How do I even write a plugin that provides a function for the template language?

You don't. There is no way that a template function can access the calibre DB layer, directly or indirectly. They also cannot access the GUI layer, meaning that they cannot use interface plugins.

2) create a function plus a meta data table for the series

A new table with 3 columns (string series, int max_int, int max_fract). In this case calibre would have to update this table, every time a series index is changed/added/deleted. Are there any hooks in calibre or would I have to write this part as well?

This also won't work because the template function would need access to this table, which is in the database, which is not visible.

You could do it by creating a text custom column and having the UI plugin store the format string into that for each book. That would permit the template language to see the data.

In any case, the documentation did not answer any of my questions and I am afraid I do not have the time to get familiar with the calibre code base and the plugin interfaces, which I would had to do to be actually able to implement my formatting function.

It is true that calibre does not have a "developer's guide to calibre internals". I can't see that ever happening for two reasons: 1) no one wants to attempt to write it, and 2) maintaining it would be crushing because calibre changes so fast.

I hoped that calibre would already provide a functionality like this, since it seems logical to me that someone would like to have aligned indexes in a filename without having to hardcode the padding. e.g. who wants a filename like 'seriesname [007.000] bookname', if the biggest index is 7.00 and the series has only integers?

Actually, you are the first I know of to care about this particular variant. Perhaps some of the people who care don't tell us, or perhaps they use devices like Sony & Kobo or apps like CC that don't require the series in the file name. The issue of db access in template functions has come up before, but IIRC in the context of wanting things like "Series [Book 1 of N]" or "Avg rating: 3.5".

If I were you, I would just solve the problem by hand. Create a custom text column to hold the format string for a book and manually set its content using the mark-1 eyeball to determine the format you want.

Hmmm... the above leads me down another path. You could use the command line interface to dump the series information to a CSV table (calibredb), write a script to process that table and compute the resulting format strings, then use the command line interface to set the value of the custom column for the books in a series (calibredb search to get the ids for the books in a series then calibredb set_custom for each id).

[kcz]

Originally Posted by chaley

Hmmm... the above leads me down another path. You could use the command line interface to dump the series information to a CSV table (calibredb), write a script to process that table and compute the resulting format strings, then use the command line interface to set the value of the custom column for the books in a series (calibredb search to get the ids for the books in a series then calibredb set_custom for each id).

Wow, this is truly an excellent idea. I totally forgot about the command line interface.
In this case I can write a script in perl, bash or whatnot and run the script every time I modify a series.

Thank you so much. I guess I didn't see the forrest for the trees.

[chaley]

Originally Posted by kcz

Wow, this is truly an excellent idea. I totally forgot about the command line interface.
In this case I can write a script in perl, bash or whatnot and run the script every time I modify a series.

And some day if/when you have time, you could do it in python as a standalone calibre script (calibre-debug -e), using calibre's data layer. This would run *many* times faster because a) it would be one execution instead of one per book, b) it could check if anything had changed, and c) could update all the books in a series in one transaction. The program would do something like:
1) iterate through all the book ids in the db (there is a method for that), building a map of series -> desired number format and series -> existing number format.
2) iterate through the maps. If desired != existing:
2.1) use the db-level search method to get all ids for a series.
2.2) use the db-level update field multiple to set the custom column for all the books in the series to the "desired" value.

Thank you so much. I guess I didn't see the forrest for the trees.

You are welcome. It has been an interesting discussion.

[chaley]

I decided to give it a try because I need more experience with the new calibre db layer.

Here is a standalone python program built on calibre's db layer that does what I think you want. It fills in a custom text column with the proper format for a series, computed by looking at all the books in the series.

def init_cache(library_path):
	from calibre.db.backend import DB
	from calibre.db.cache import Cache
	backend = DB(library_path)
	cache = Cache(backend)
	cache.init()
	return cache

import math
from collections import defaultdict
	
cache = init_cache(library_path = 'd:/cbh_data/calibre.git/library.test_small')
series_info = defaultdict(dict)

for id_ in cache.all_book_ids():
	series = cache.field_for('series', id_)
	if series:
		sidx = cache.field_for('series_index', id_)
		str_sidx = str(sidx)
		components = str_sidx.split('.')
		if components[1] == '0':
			fract_part = 0
		else:
			fract_part = len(components[1])
		val_part = len(components[0])
		series_info[series]['val_part'] = max(series_info[series].get('val_part', 0), val_part)
		series_info[series]['fract_part'] = max(series_info[series].get('fract_part', 0), fract_part)
		
for series,sinfo in series_info.iteritems():
	fract_part = sinfo['fract_part']
	format_str = '{0:0%d.%df}'%(sinfo['val_part'] + fract_part + (1 if fract_part > 0 else 0), fract_part)
	sids = cache.search('series:"=' + series + '"')
	dct = {book_id:format_str for book_id in sids}
	cache.set_field('#text', dct)