Calibre always freezes with tag browser shown

[chaley]

Thanks. I will look.

Originally Posted by Noughty

I just changed this one to, #length wasn't needed here, only word Short:

program:
list_sort(
   list_difference(
      list_union(field('#shelf'), field('tags'), ','),
      list_union(strcat('Short', ',',
        'delete'),
         strcat(field('#genre'), ',',
                field('#mode'), ',', 
                field('#character')),
         ','), 
      ','),
   0, ',')

You can improve this by eliminating the first strcat, because all the arguments are constants. Try:

program:
list_sort(
   list_difference(
      list_union(field('#shelf'), field('tags'), ','),
      list_union('Short, delete',
         strcat(field('#genre'), ',',
                field('#mode'), ',', 
                field('#character')),
         ','), 
      ','),
   0, ',')

[Noughty]

It worked. Thanks.

Now it takes 2 mins and a few seconds to write all the data.

I'm thinking of another cheat. Just rename shelves according to the category. If it belongs to genre just add word genre/g or something in the beginning so when viewing the list it would show it alphabetically so all genres would be there. I can do the same with tags. A little work while renaming and after that it should look ok. Not as nice as having separate columns but it would work without any freezes as before. I'm not sure I would like how it looks. Having tags like g-action, g-adventure etc.

I guess I will keep it as an option if freezing up makes it too much trouble working with metadata.

Would making the list for list_intersection shorter make it easier for calibre? I could remove a lot from category character since there are way too many detail tags.

[chaley]

Originally Posted by Noughty

Would making the list for list_intersection shorter make it easier for calibre? I could remove a lot from category character since there are way too many detail tags.

A bit, but I think not enough to justify the cost.

If you don't mind getting more complicated, we *can* justify the cost of using a python evaluation function. For example, for the template we have been playing with in the last 2 posts, a custom function could easily be 100 times faster to evaluate. That won't change the time needed to build the categories for the tag browser, but it would drastically reduce the amount of time needed to build the columns so the categories can be built.

Here is the last template rewritten as a template function. To put this into your library, go to preferences -> advanced -> template functions. You will see some boxes. Fill them in as follows:

Function: myListMerge (you can use any name here you want, as 
          long as it is alphabetic. You might consider using the column name, 
          such as myWhateverItIs.)

Arg count: 0 (ignore the warning that will appear later)

Documentation: whatever you want to put here

Program Code:
def evaluate(self, formatter, kwargs, mi, locals):
	tags = set(mi.get('tags', []))
	shelf = set(mi.get('#shelf', []))
	genre = set(mi.get('#genre', []))
	mode = set(mi.get('#mode', []))
	charac = set(mi.get('#character', []))

	st = tags | shelf
	gmc = set(['Short', 'delete']) | genre | mode | charac
	res = st - gmc
	return ', '.join(res)

Press 'create' (or modify if you are changing it later), then apply.

You would use this new function in your composite column template as follows, and assuming it is named myListMerge:

{:'myListMerge()'}

Given you current experience, I think you can see how this function works. It first gets all the fields from the book metadata information (mi). It then does a union of tags and shelf (the '|' operator), and then a union of genre, mode, charac, and the constant set containing 'Short' and 'delete'. Finally it does the difference (the '-' operator), then turns the result back into a comma-separated string (the 'join').

Do note that this function assumes that the case of all items is identical and uses a case-sensitive compare. For example, it thinks that foo and Foo are different. Fixing this is possible, but it would slow things down somewhat.

[Noughty]

I created it named as otherts but I don't get any result. I need to write:

{'otherts()'} in custom column made from other tags (behaves like tags)?

I get error:
EXCEPTION: Value: unknown field 'otherts()'

[chaley]

Originally Posted by Noughty

I need to write:

{'otherts()'} in custom column made from other tags (behaves like tags)?

I get error:
EXCEPTION: Value: unknown field 'otherts()'

Ahh, my mistake, which I fixed in my example above. You should all {:'otherts()'}. I left out the colon (the second character).

For completeness, the :' tells the formatter that what follows is in template program mode, which allows use of functions as arguments and zero-parameter functions. The program goes up to the '}. Writing "{:'foobar()'}" is exactly the same as writing "program:foobar()".

You might then ask "why use program:"? That construct allows multiple lines and multiple statements, and is (I think) easier to read. On the other hand, template program mode expressions can be combined together in a single template, such as "{title} {:'foobar()'|- | }"

[Noughty]

It gives wrong result

Nothing gets taken away. It just merges lists (tags, shelf) and deletes 'Short', 'delete', but not other lists

[chaley]

Did I spell the lookup names of the columns correctly?

Are any of #shelf, #mode, #genre, or #character composite columns? I assumed that they were all text, like tags.

Do all the words have the same case?

If you add 'Short' as a tag, it appears in the output of the custom function? If so, that is odd because it does not for me. Note that if you add 'short' it will appear, but 'Short' should not.

[Noughty]

Everything is spelled correctly as far as I can tell.
They are composite columns using code like this:

program:
	list_union(
		list_intersection(field('tags'), 'values listed, ','),
		list_intersection(field('#shelf'), 'values listed', ','), 
		',')

It deletes Short, delete but not custom columns. I guess because they are custom. All values are with first letter uppercase, others lowercase.

Maybe I could just paste all these values from these into python code like 'short, delete' are? The ones I want to remove.

What function I would use in python to write the values I want to keep, not add. Like extract only those values? I'll try to write it myself using your example. I just need a function so I could use it for genre, character,mode columns. It would be more efficient as far as I understand.

[chaley]

Originally Posted by Noughty

Everything is spelled correctly as far as I can tell.
They are composite columns using code like this:

program:
	list_union(
		list_intersection(field('tags'), 'values listed, ','),
		list_intersection(field('#shelf'), 'values listed', ','), 
		',')

It deletes Short, delete but not custom columns. I guess because they are custom. All values are with first letter uppercase, others lowercase.

Try this program. Replace the Program Code of your otherts function with the code above.

def evaluate(self, formatter, kwargs, mi, locals):
	# Tags is a multiple text field, so it is already split
	tags = set(mi.get('tags', []))
	# Shelf is a composite column, so we must split it
	shelf = set([l.strip() for l in mi.get('#shelf', "").split(',')])
	# Genre is a composite column, so we must split it
	genre = set([l.strip() for l in mi.get('#genre', "").split(',')])
	# Mode is a composite column, so we must split it
	mode = set([l.strip() for l in mi.get('#mode', "").split(',')])
	# Character is a composite column, so we must split it
	charac = set([l.strip() for l in mi.get('#character', "").split(',')])

	st = tags | shelf
	gmc = set(['Short', 'delete']) | genre | mode | charac
	res = st - gmc
	return ', '.join(res)

Maybe I could just paste all these values from these into python code like 'short, delete' are? The ones I want to remove.

If you know the list and it isn't too long, that isn't a bad solution at all.

What function I would use in python to write the values I want to keep, not add. Like extract only those values? I'll try to write it myself using your example. I just need a function so I could use it for genre, character,mode columns. It would be more efficient as far as I understand.

If I understand your question, it would be something like:

def evaluate(self, formatter, kwargs, mi, locals):
	# Tags is a multiple text field, so it is already split
	tags = set(mi.get('tags', []))
	# Shelf is a composite column, so we must split it
	shelf = set([l.strip() for l in mi.get('#shelf', "").split(',')])
	
	# items to remove from the result
	to_remove = set(['Short', 'delete', 'foo', 'bar', somethingElse',
		 'andSoOn', 'until I get tired', 'of adding things']) 

	# combine the two fields containing items we might want to keep.
	st = tags | shelf
	# now remove the items we don't want in the result
	res = st - to_remove
	return ', '.join(res)

[Noughty]

I'll try it but shelf isn't composite column. Tags and shelves are simple tag columns.
I edited it and it seems to work

It doesn't seem to sort list. Is it possible to use {:'otherts()'} like program? I could use sort function when.

[chaley]

Originally Posted by Noughty

I'll try it but shelf isn't composite column. Tags and shelves are simple tag columns.

I wasn't sure, which is why I added the comment. Use the original line of code for shelf instead of the new line.

	shelf = set(mi.get('#shelf', []))

[Noughty]

Thanks for the comments with them I managed to make edit easily. Now just to figure out how to sort the list

Made it
Just added sort for the result. I might start to get python too... Well, a little

[chaley]

Originally Posted by Noughty

Thanks for the comments with them I managed to make edit easily. Now just to figure out how to sort the list

That is easy (if you know how ). Replace the "return ..." line with the following two lines.

	from calibre.utils.icu import sort_key
	return ', '.join(sorted(res, key=sort_key))

This sorts the list in order respecting the current language rules, then converts it to a string.

[Noughty]

And I just wrote

return ', '.join(sorted(res))

Is it wrong? The result seems to be correct...

Where could I read more about functions in python?

Seems the speed increased by half a minute. Maybe I could make other columns like this too (genre, character). Since they are basically the same it might not be hard.

def evaluate(self, formatter, kwargs, mi, locals):
	# Tags is a multiple text field, so it is already split
	tags = set(mi.get('tags', []))
	# Shelf is a composite column, so we must split it
	shelf = set([l.strip() for l in mi.get('#shelf', "").split(',')])
	
	# items to remove from the result
	to_remove = set(['Short', 'delete', 'foo', 'bar', somethingElse',
		 'andSoOn', 'until I get tired', 'of adding things']) 

	# combine the two fields containing items we might want to keep.
	st = tags | shelf
	# now remove the items we don't want in the result
	res = st - to_remove
	return ', '.join(res)

I need another function. I need to leave the values I list, not remove them. For example I am doing the genre column, so I would list all genre values I have in my library, so in the returned list only they remain.

I need to extract specific values

[chaley]

Originally Posted by Noughty

And I just wrote

return ', '.join(sorted(res))

Is it wrong? The result seems to be correct...

That works fine as long as the words do not contain non-English letters. If they do, then the words will probably sort incorrectly.