[Kobo] Collections: using a template

[chaley]

@davidfor: last we talked about this I was playing with using a template to optionally rewrite collection names. You felt that a better approach would be to use a template to generate the names, not rewrite the names. After a month or so of using my original system I came to agree with you.

I have an implementation that permits one to add "template" to the list of collection attributes. It can be the only attribute if you wish. The driver invokes the template when looping through the collection attributes. The template returns a list of collection names separated by '#@#' (the same separator used in template searches).

Here is the template I am now using. It supports both my wife's and my Libra 2s. All the hard work is done in the generate_values() function. For my Libra 2 the template generates two sets of collection names, one set for my personal tags all prefixed with "CBHTags: ", and another set of the author's name's first letter prefixed by 'Authors: '. For my wife's it generates one set of names from her personal tags.

program:
	globals(serial_number='N4181B1027108');

	def generate_values(previous, col_val, sep, prefix, letters):
		res = '';
		for f in col_val separator sep:
			f = re(f, '^\s*(.*)\s*$', '\1');
			if f then
				if letters then f = substr(f, 0, letters) fi;
				res = ':@:' & prefix & f
			fi
		rof;
		return previous & res
	fed;

	if serial_number == 'N4181B1027108' then
		res = generate_values('', $authors, '&', 'Authors: ', 1);
		res = generate_values(res, $#cbhtags, ',', 'CBHTags: ', '');
		return res
	fi;
	if serial_number == 'N4181B5019336' then
		return generate_values('', $#dthtags, ',', 'DTH_Tags: ', '')
	fi

Here is a screenshot of the configuration tab.


There are two reasons I chose to add "template" as a collections attribute instead of making it either attributes or template but not both:

• It doesn't interfere with metadata management because in get_collections it acts like a column. This is important when dealing with device_collections.
• It permits generating collections from columns in addition to the template.

[davidfor]

I've gone back and forth over exactly how to do this. My original thought was either a template or the current list of columns. But, allowing both is probably the most compatible. Which is the way you are doing it. But, I'm not a fan of adding what is effectively a special column name. I would normally add a checkbox to enable the feature and that would be enough.

Can I get a copy of your code? That should save me some time, at least for a quick test to make sure I have understood what you are doing.

[chaley]

Originally Posted by davidfor

I've gone back and forth over exactly how to do this. My original thought was either a template or the current list of columns. But, allowing both is probably the most compatible. Which is the way you are doing it. But, I'm not a fan of adding what is effectively a special column name. I would normally add a checkbox to enable the feature and that would be enough.

I too have been fretting about this.

One problem with that is the interplay between the contents of the columns box and metadata management. If you use a checkbox then can I also use the columns box? What if I don't, making the columns box empty, which in the current code would prevent get_collections() from being called. Is using the checkbox equivalent to having "something" in the columns box? It also raises the question "What does the template return?" Is it a list of columns or a list of items?

Finally, how does one solve the problem of two users wanting different metadata management settings? That could be done with a checkbox and yet another template that returns the management setting desired for the connected device. I have been tempted to play with this, but as I don't need it I haven't taken the time.

The "special name" is safe because it can't conflict with a custom column. It could be made safer by using something like "%template" instead of "template".

But you know all the above.

Can I get a copy of your code? That should save me some time, at least for a quick test to make sure I have understood what you are doing.

Attached is a zip containing a) complete source files for books.py, driver.py, and kobotouch_config.py; and b) a patch file for the same changes.

The source files incorporate all your changes that have been checked in by Kovid.

[davidfor]

Originally Posted by chaley

I too have been fretting about this.

One problem with that is the interplay between the contents of the columns box and metadata management. If you use a checkbox then can I also use the columns box?

The checkbox would disable the template field. As the column field would still be enabled, both could be used. I'm tempted to add a checkbox for both to make it clear what is being used at any point.

What if I don't, making the columns box empty, which in the current code would prevent get_collections() from being called.

That's the easiest thing to fix. And I think that needs to be revisited. At the least, the variable used needs to be renamed.

Is using the checkbox equivalent to having "something" in the columns box? It also raises the question "What does the template return?" Is it a list of columns or a list of items?

Yes, the check box says to use the template as well as the columns. The template result should be a comma separated list of collections names. That way, a template of "{#some_column}" is the same as just using "#some_column" as the collections column.

Finally, how does one solve the problem of two users wanting different metadata management settings? That could be done with a checkbox and yet another template that returns the management setting desired for the connected device. I have been tempted to play with this, but as I don't need it I haven't taken the time.

That is unchanged from the current scenario. It has no effect on what collections are generated, just when or if they are applied. To be able to do anything else, I thing you would need to be able to store a configuration for each device. I looked at that when I rewrote the configuration dialog a few years ago, but the single configuration is to embedded in the USBMS driver to make it easy to do.

The "special name" is safe because it can't conflict with a custom column. It could be made safer by using something like "%template" instead of "template".

But you know all the above.
Attached is a zip containing a) complete source files for books.py, driver.py, and kobotouch_config.py; and b) a patch file for the same changes.

The source files incorporate all your changes that have been checked in by Kovid.

Thanks. That reduces some thinking and gave me a good starting point.

[chaley]

Originally Posted by davidfor

Yes, the check box says to use the template as well as the columns. The template result should be a comma separated list of collections names. That way, a template of "{#some_column}" is the same as just using "#some_column" as the collections column.

It can't be a comma separated list because that would break authors, other "is_names" custom columns, or single-valued columns where the value contains a comma. That is why my example code uses ':@:'.

I have most of the implementation to permit overriding metadata management with a template. See the screen capture. If you wish I can forward it when the implementation is finished.

FWIW: I have added a new formatter function that I have wanted for some time, list_join(). It simplified dealing with generating the item list. Here are the details.

list_join(with_separator, list1, separator1 [, list2, separator2]*)

Returns a list made by joining the items in the source lists (list1 etc) using with_separator between the items in the result list. Items in each source list[123...] are separated by the associated separator[123...]. A list can contain zero values. It can be a field like publisher that is single-valued, effectively a one-item list. Duplicates are removed using a case-insensitive comparison. Items are returned in the order they appear in the source lists. If items on lists differ only in letter case then the last is used. All separators can be more than one character.

Example:

program:
  list_join(':@:', $authors, '&', $tags, ',')

You can use list_join on the results of previous calls to list_join as follows:

program:
  a = list_join(':@:', $authors, '&', $tags, ',');
  b = list_join(':@:', a, ':@:', $#genre, ',', $#people, '&', 'some value', ',')

You can use expressions to generate a list. For example, assume you want items for authors and #genre, but with the genre changed to the word "Genre: " followed by the first letter of the genre, i.e. the genre "Fiction" becomes "Genre: F". The following will do that:

program:
  list_join(':@:', $authors, '&', list_re($#genre, ',', '^(.).*$', 'Genre: \1'),  ',')

[davidfor]

Having said all that, which took all day as I was doing other things and writing the code. So...

Attached is my version of using templates for collections. It is inspired (I think that's the best word) by your code.

Things it does:

• Configuration:
  • Add the collection template field.
  • For both the collections column and template, add a checkbox to enable/disable. For the columns, it will default to enabled to match the current behaviour.
  • The two fields are independent and either or both can be used.
  • If the checkboxes are checked and the fields are empty, nothing will be done with it.
  
• The driver has changes for the above options and a couple of other small changes.
• The "get_collections" method is where action is.
  • The collections template format is added as a parameter.
  • Instead of just passing in the serial number, I am passing in a dictionary of what I think could be useful. What is being passed in is:
    • serial_number
    • firmware_version - a tuple like (4,32,19501)
    • display_firmware_version - will look like "4.32.19501"
    • dbversion
    I considered adding the device name, but, that is already available with "connected_device_name('main')".
  • I have stripped out a lot of stuff that wasn't being used. I did not understand what was happening here when I took over the driver maintenance, I hated the code and have not revisited it.

I have done some testing, but nowhere near enough. It seems to work with the different combinations of the options. And the collection template I have been using is:

program:
globals(serial_number='N7828A0018128');
collections = '';
if serial_number == 'N7828A0018128' then
    collections = connected_device_name('main')
fi;
if (connected_device_uuid("main") == '30c617bc-d1e4-4cc0-bc69-ef7ca6ab88c9') then
   collections = list_union(collections, 'connected_device_uuid', ',')
fi;
return collections

I'm not completely sure if the configuration is right. I am looking for the method that will be easy to explain. I think this is it, but, I know that no matter what I do, sooner or later, I'll be writing a post thinking, "How do I explain it simpler than that?" If someone has a better idea, I'm open to it. Though I might ask you to write the help

[chaley]

I tried your code and with 3 changes it works for me.

The first change is in the driver. I have no collections attributes. However, the check used an 'and' where I think you want an 'or' because it required me to define some.

The second is in books. Because of how python handles references, collections_attributes had %template% appended to it for each book. That caused the template to be executed more than once per book

The third is in books. For the reasons I mentioned above, I can't use comma as the template separator (my authors are LN, FN). I changed it to what I was using, ':@:'

The files are attached.

[chaley]

FYI: Here is a mostly equivalent template to the one in post #1, using the new list_join() function. It is much simpler. Not to say it is simple, but simpler.

program:
	globals(serial_number='N4181B5019336');

	if serial_number == 'N4181B1027108' then
		return list_join(
			':@:', 
			list_re($#cbhtags, ',', '^(.*)$', 'CBH Tags: \1'), ',', 
			list_re($authors, '&', '(^.).*', 'Authors: \1'), '&')
	fi;
	if serial_number == 'N4181B5019336' then
		return list_join(
			':@:', 
			list_re($#dthtags, ',', '^(.*)$', 'DTH Tags: \1'), ',',
			list_re($authors, '&', '(^.*)', 'Authors: \1'), '&')
	fi

On my Kobo I have collections "Author: X" where X is the first letter of the last name of the author. On my wife's device the author is fully spelled out.

The second 'if' block shows why comma can't be the separator. We both like to see authors in LN, FN format. The full author names contain commas and would be split into two collection items. Why not use the standard Kobo author list? Neither of us like that list because it isn't sorted by the first letter you see on the line. I have to scan the line for the last name to figure out where I am.

The pattern for an unmodified calibre column is:

program:
	return list_join(':@:', $column_name, ',')

To return 'tags' and 'publisher' it would be

program:
	return list_join(':@:', $tags, ',', $publisher, ',')

[chaley]

A thought: you could 'solve' the separator problem by providing a box where the user can choose whatever separator is appropriate. It would default to comma, which would cover (I think) the majority of uses. Something like the following:


The user (me) would change it if values contain commas, which can happen for authors-like columns and single-value columns like publisher, series, enums, and non-tags text.

On the other hand, having a choice means one must understand when to choose. Maybe it is better to have a rule, not a choice.

[davidfor]

Originally Posted by chaley

I tried your code and with 3 changes it works for me.

The first change is in the driver. I have no collections attributes. However, the check used an 'and' where I think you want an 'or' because it required me to define some.

Whoops. I had planned to revisit that line for the first condition but realised later that I didn't need to. And forgot to test everything. Easily fixed

The second is in books. Because of how python handles references, collections_attributes had %template% appended to it for each book. That caused the template to be executed more than once per book

That's horrible. I have hit that in other languages but not in Python. And there was another spot it was needed a couple of lines earlier.

The third is in books. For the reasons I mentioned above, I can't use comma as the template separator (my authors are LN, FN). I changed it to what I was using, ':@:'

I really, really don't line this. And it's why I haven't replied. I don't want to do it, but, everything else is worse. The option for the delimiter is sort of tempting, but, I think it will just complicate things. And I suppose we can always change it later if needed.

I have checked the code in and created a pull request. I think I included all your changes. And then I noticed Kovid had created a new release. Is the weekend coming early?