Plugboard, template, and custom composite column recipes

[eschwartz]

@chaley -- merge into the OP?

[chaley]

Originally Posted by eschwartz

@chaley -- merge into the OP?

Sure, if you want to.

[eschwartz]

I want to but I can't. Why don't you do it instead?

[PapaJohn]

Don't know how long this has been going on, but fetch annotations is no longer updating
my Read Status column or comments column. I have a Kindle DX and a Paper White (separate libraries for each) and Calibre is not updating from either one. I am using Calibre 2.49 on Mac OS 10.10.5. I hope someone can help me get this functioning again.

[JJanssen]

I apologise since this is probably an FAQ, or at least previously answered, but I'm getting information overload from both this thread and the python docs.

For my Kindle plugboard I have used a line straight off the Calibre docs for years: {series}{series_index:0>3s| – | – }{title}. But since that hardcodes three digits, it fails since I occasionally use decimal series IDs. (omnibuses, short story collections, or other things that are not *actually* part of the series proper, but do have a *place* in the order). These then omit leading zeroes and sort out of sequence.

So another one mentioned in the docs is 0>5.2f -- five digits, 2 after the decimal point, if I read that correctly. That would work, but at the cost of more zeroes on my screen than I really want (since fractional indices are rare). I could do 4.1, which is better (I have never needed more than 9 sub-indices, and I could reindex the ones where I've used hundredths).

But what I *really* would want is to have 3 digits with leading zeroes normally, and *add* the decimals (ideally to the same 2 digit precision as in calibre itself) _only when needed_. So 001, 002, 003, 003.01, 003.50, 004, etc.

Is this possible or a pipedream? Looking at the python docs, I think it would almost have to involve something more than a simple format-string, which I think would make it impossible to do in calibre?

[eschwartz]

You *could* do it using a format specifier stored in another custom column.
You'd need to run a calibre-debug script periodically to recalculate the format specifiers.

Reason: a mere template cannot know the metadata for other book records.
And you need to know if there are other books in the series with a decimal series_index.

...

chaley wrote a calibre-debug script for a case like this, I will see if I can track it down for you.

EDIT: Here it is: https://www.mobileread.com/forums/sho....php?p=2916772

[JJanssen]

Originally Posted by eschwartz

EDIT: Here it is: https://www.mobileread.com/forums/sho....php?p=2916772

That looks remarkably like what I want. Thank you very much!

I'll look at it in detail once I have a) a test library set up and b) backups and c) some time

[JJanssen]

Originally Posted by JJanssen

That looks remarkably like what I want. Thank you very much!

So, after a false start or two (I accidentally the spaces in the path with a \, also accidentally the destination column) I have a script that reliably puts something useful in the #format_str field of my database. It's not quite the one that I want, but that's an implementation detail that will be easily solved in a few lines of python. I wanted to have the workflow working before I go there.

But how do I use that column in my plugboard?

If I want to replace "{series}{series_index:0>4.1f| – | – }{title}", my obvious naive approach would be "{series}{series_index:{format_str}| – | – }{title}", , or with added #, but it doesn't like that. My perusal of this thread hasn't found it yet either. Since this is obviously a Solved Problem, can someone point me at the solution?

Edit: I was in the python mood anyway, so here's my extra few lines, based on the very nice script by chaley:

Spoiler:

def init_cache(library_path):
	from calibre.db.backend import DB
	from calibre.db.cache import Cache
	backend = DB(library_path)
	cache = Cache(backend)
	cache.init()
	return cache

import math
from collections import defaultdict
	
cache = init_cache(library_path = '/Users/jasper/Documents/Calibre Test Library/')
series_info = defaultdict(dict)
fract_ids = []

for id_ in cache.all_book_ids():
	series = cache.field_for('series', id_)
	if series:
		sidx = cache.field_for('series_index', id_)
		str_sidx = str(sidx)
		components = str_sidx.split('.')
		if components[1] == '0':
			fract_part = 0
		else:
			fract_part = len(components[1])
			fract_ids.append(id_)
		val_part = len(components[0])
		series_info[series]['val_part'] = max(series_info[series].get('val_part', 0), val_part)
		series_info[series]['fract_part'] = max(series_info[series].get('fract_part', 0), fract_part)

for series,sinfo in series_info.iteritems():
	fract_part = sinfo['fract_part']
#	if fract_part == 1:  # Once fractional parts come into play, always have them at least 2 long.
#		fract_part = 2  # Not sure yet if I actually want that.
	print "Series name:",series
	brief_format_str = '{0:0%df}'%(sinfo['val_part'])
	print "Format String for books without fraction:",brief_format_str
	if fract_part > 0:
		full_format_str = '{0:0%d.%df}'%(sinfo['val_part'] + fract_part + (1 if fract_part > 0 else 0), fract_part)
		print "Format string for books with fraction:",full_format_str
	sids = cache.search('series:"=' + series + '"')
	dct = {}
	for book_id in sids:
		if book_id in fract_ids:
			dct[book_id] = full_format_str
		else:
			dct[book_id] = brief_format_str
	cache.set_field('#format_str', dct)

[eschwartz]

Use general program mode, with the finish_formatting() function.

(GPM is better for these complex cases, although you "could" make do with TPM.)

[JJanssen]

Originally Posted by eschwartz

Use general program mode, with the finish_formatting() function.

So, for this it looks like I'd simply make a custom column with a template written as a GPM such that the column would simply contain the desired end product, right? In order to do this, it seems I need to input a multi-line program in the Template field of a custom column? Because my Calibre only seems to give me a single line edit field, and if I paste a multiline snippet in it still becomes a single line.

Edit: Ah, I think I've got it. Add more tabs, then paste.
Edit2: nope, that still chokes on anything more than 1 line.

Edit3: The most obvious way to go seemed to be:

program:
	f_s = field('#format_str')
	f_s.format(str(field('series_index'))
	return f_s

But that gives me a template error, quite possibly simply because it's multiple lines. I've also tried a bunch of variants including a one-liner:

program:
	str(field('#format_str').format(field('series_index')))

(the wrapping str() was a desperation move)

Which also doesn't work. The only moderately complex one I;ve gotten to give me anything was one from the docs:

program:
    strcat(
        re(field('series'), '([^\s])[^\s]+(\s|$)', '\1'),
        finish_formatting(field('series_index'), '0>2s', ' - ', ' - '),
        field('title')
    )

But that, being a single function call, is of course a plausible one-liner.

[eschwartz]

You can simply include the GPM template directly in the plugboard definition. Right-click and there will be an option to use the Template Editor.
(When saved, it appears as if it is one line, but internally linebreaks are still saved as '\n' -- also, linebreaks may help you read it but they don't affect the parsing.)

Again, you just need to feed finish_formatting() with the formatting specification in the custom column.

{series_index:0>5.2f| – | – }

becomes

finish_formatting(field('#series_index'), field('#format_str'), ' – ', ' – ')

[JJanssen]

Originally Posted by eschwartz

You can simply include the GPM template directly in the plugboard definition. Right-click and there will be an option to use the Template Editor.
(When saved, it appears as if it is one line, but internally linebreaks are still saved as '\n' -- also, linebreaks may help you read it but they don't affect the parsing.)

I just found out there was such a thing as a template editor right before I read your message. Apparently it's even possible to change the template on a custom column without restarting Calibre! Who knew. Anyway. That's jsut my struggle with some UI bits.

Again, you just need to feed finish_formatting() with the formatting specification in the custom column.

{series_index:0>5.2f| – | – }

becomes

finish_formatting(field('#series'), field('#format_str'), ' – ', ' – ')

Okay, so that'd become something like

program:
strcat(
	field('series'),
	finish_formatting(field('series_index'), field('#format_str'), ' – ', ' – '),
	field('title')
)

, right? And that *almost* works, except that the template really doesn't like actually doing that finish_formatting operation -- "TEMPLATE ERROR unknown format code 'f' for object of type unicode". That was when I moved on to the extra column because that's what the docs seemed to imply.

Hardcoding a format..... right. wait. What that function expects is apparently "05.1f", not "{0:05.1f}". So it might work if I simplified the generating script a little.

Yup.

That does the trick.

So just for documentation purposes, here's my working setup:

Spoiler:

Intention: Series indexes will get a fixed number of digits before the decimal point, filled out with 0s in front whenever necessary. Those books with Integer indexes will get just those as index, and those books in a series that have a fractional index will get those with a number of digits after the decimal point that is also the same per series.

For instance, a series with a couple of fractional books, and mostly just lots of them, might get numbered:

001
001.5
002
003
003.1
003.2
[...]
011
013.3
801
802
803
803.2

The script to generate the custom formatting column:

def init_cache(library_path = None):
	from calibre.db.backend import DB
	from calibre.db.cache import Cache
	from calibre.utils.config import prefs
	if library_path == None:
		library_path = prefs['library_path']
	backend = DB(library_path)
	cache = Cache(backend)
	cache.init()
	return cache

cache = init_cache()
#cache = init_cache('/Users/jasper/Documents/Calibre Main Library/')
#cache = init_cache(library_path = '/Users/jasper/Documents/Calibre Test Library/')
#
# The parameter can be empty, for the currently opened library, 
# or you can pass a specific Library's path if you wish.
#
#-------------------- Usage specific code starts here ------------------

import math
from collections import defaultdict


series_info = defaultdict(dict)
fract_ids = []

for id_ in cache.all_book_ids():
	series = cache.field_for('series', id_)
	if series:
		sidx = cache.field_for('series_index', id_)
		str_sidx = str(sidx)
		components = str_sidx.split('.')
		if components[1] == '0':
			fract_part = 0
		else:
			fract_part = len(components[1])
			fract_ids.append(id_)
		val_part = len(components[0])
		series_info[series]['val_part'] = max(series_info[series].get('val_part', 0), val_part)
		series_info[series]['fract_part'] = max(series_info[series].get('fract_part', 0), fract_part)
		
for series,sinfo in series_info.iteritems():
	fract_part = sinfo['fract_part']
#	if fract_part == 1:  # Once fractional parts come into play, always have them at least 2 long.
#		fract_part = 2  # Not sure yet if I actually want that.
	print "Series name:",series
	brief_format_str = '0%d.0f'%(sinfo['val_part'])
	print "Format String for books without fraction:",brief_format_str
	if fract_part > 0:
		full_format_str = '0%d.%df'%(sinfo['val_part'] + fract_part + (1 if fract_part > 0 else 0), fract_part)
		print "Format string for books with fraction:",full_format_str
	sids = cache.search('series:"=' + series + '"')
	dct = {}
	for book_id in sids:
		if book_id in fract_ids:
			dct[book_id] = full_format_str
		else:
			dct[book_id] = brief_format_str
	cache.set_field('#format_str', dct)

(I've left in just a couple of the debug prints because, well, I think they're useful)
Custom column #format_str stores the format str, custom column #series_index_formatted stores the formatted series index, templated as

program:
	finish_formatting(field('series_index'), field("#format_str"),'','')

, and the plugboard then becomes

{series}{#series_index_formatted:| - | - }{title}

(The added templated column may or may not impact the performance in my usecase -- I'm betting not -- but I think the extra elegance of the system as a whole probably is worth it anyway.)

So, wow, that was definitely one of those cases where I was looking for the issue in all the wrong places. Sorry for trying your patience! And definitely thanks for pointing me in the right direction.

[eschwartz]

Reducing the number of custom columns will definitely have *some* performance impact, although how much in the grand scheme of things ???

P.S. Change:

cache = init_cache(library_path = '/Users/jasper/Documents/Calibre Test Library/')

to

from calibre.utils.config import prefs
cache = init_cache(library_path = prefs['library_path'])

That way you use the default library path, instead of requiring users to modify the script.

[JJanssen]

Originally Posted by eschwartz

P.S. Change:
[...]
That way you use the default library path, instead of requiring users to modify the script.

I've changed it in the text up there, but I'm keeping my own hardcoded version on my disk

EDIT: Noticing your edits, you seemed to be slightly confused at first what GPM is.
It isn't python, it is a cut-down mini-language that compiles to python and is documented in the calibre manual. You can however define new functions in python.

Yeah, I'm not sure why, but I read the documentation as "well, here's the templating stuff, but if you want to break out into actual python, here's how, use program:". I read the GPM functions docs as being GPM=*specific* additional functions, rather than as being all you can use.

[eschwartz]

Sure.
It's a trick I saw used elsewhere, which makes for much more reusable code (and which I should've used back when someone was having a lot of trouble editing an attached python script ).

Also, noticing your edits, you seemed to be slightly confused at first what GPM is.
It isn't python, it is a cut-down mini-language that compiles to python and is documented in the calibre manual. You can however define new functions in python.