Search by device?

[JimmXinu]

Originally Posted by chaley

I decided to take this on as a challenge and see if I could make templates work efficiently in searches. The answer is yes. it is available in calibre 5.1.

I am intrigued... but also mildly confused. Can you give a couple examples of search with a template?

[chaley]

Originally Posted by JimmXinu

I am intrigued... but also mildly confused. Can you give a couple examples of search with a template?

There are two examples in the search interface section of the manual, at the very end of the section. I suggest you use the "Advanced search" dialog to build a few so you can see the syntax.

The idea, inspired by compurandom's query, is that some custom columns exist only for searching. These can go away, replaced by a search that uses a template to compute the value. Also, searches can now also use non-metadata template functions such as formats_sizes(), booksize(), connected_device_name(), current_library_name(), etc.

I am also working on "stored templates", which are basically template subprograms. These will interact well with template searches, permitting one to build a complex template to compute a value then "call" that template, with parameters if desired, from a search.

[chaley]

Originally Posted by chaley

I am also working on "stored templates", which are basically template subprograms. These will interact well with template searches, permitting one to build a complex template to compute a value then "call" that template, with parameters if desired, from a search.

Stored templates are now in calibre source and in the 5.1.1 beta.

[compurandom]

Originally Posted by chaley

The idea, inspired by compurandom's query, is that some custom columns exist only for searching. These can go away, replaced by a search that uses a template to compute the value.

The part of the original objection to this was that the columns cache calculated values. I figure that some templates are no worse than pulling the cached value, but have you noticed a performance difference with others?

Also, I've noticed that I can't sort one of my columns built from other columns, which makes me doubt how much caching is going on.

[chaley]

Originally Posted by compurandom

The part of the original objection to this was that the columns cache calculated values. I figure that some templates are no worse than pulling the cached value, but have you noticed a performance difference with others?

I haven't noticed a performance difference because my library is too small. That said, I came to the conclusion that the composite column cache doesn't help enough to justify not adding the feature. When searching the cache is per book, so it helps only if a composite column is referenced directly or indirectly more than once in a query.

EDIT: What I said above is misleading. The cache is initialized for a book the first time that book is referenced, either in a search or by being displayed. It is cleared when data changes for a book. Thus repeated searches using non-template field references will use the cache.

Also, I've noticed that I can't sort one of my columns built from other columns, which makes me doubt how much caching is going on.

This usually happens when the search/sort type set in the column definition doesn't match the what the template produced.

[compurandom]

Originally Posted by chaley

This usually happens when the search/sort type set in the column definition doesn't match the what the template produced.

Ah ha! That's been irritating me ever since I changed the column template! Thanks for the hint, fixing that is a huge relief!

[compurandom]

Ok, I just can't seem to get this to work in 5.2.

Originally Posted by chaley

Also, searches can now also use non-metadata template functions such as formats_sizes(), booksize(), connected_device_name(), current_library_name(), etc.

I was hoping to find out what the value of connected_device_name() is for my devices, so I go into template tester and enter

{:connected_device_name()}

and it gives me empty for no device, but when I connect a device, I get

EXCEPTION:  connected_device_name: invalid storage location ""

You listed other functions, so I figured I'd try them. so I tried booksize() and it gives

EXCEPTION:  evaluate() takes 5 positional arguments but 7 were given

as do the others.

What am I missing?

[compurandom]

Oh, and thanks very very very much for the template in search... I just saved a search that finds books with the wrong number of tags, something I've wanted for a very long time.

[chaley]

Originally Posted by compurandom

Ok, I just can't seem to get this to work in 5.2.

Before we get into the specific questions, it is worth talking a bit about the three template program modes: Single Function Mode (SFM), Template Program Mode (TPM), and General Program Mode (GPM).

SFM was intended to be 'simple', so it hides a lot of programming language bits. For example, the value of the column is always passed as the first argument to a function included in the template. It also doesn't support the difference between variables and strings; all values are strings.

Example, the following SFM template returns either the series name of the string "no series"

{series:ifempty(no series)}

The equivalent template in GPM is

program: ifempty(field('series'), 'no series')

Note that the first argument to ifempty is the value of the field 'series' and the second argument is the string 'no series'.

TPM is very close to GPM, but with oddities related to processing of {} characters and strings.

Personally, I use GPM unless the template is trivial, in which case I might use TPM. I never use SFM. In fact, if I could redo the template language development I would not build SFM.

Originally Posted by compurandom

I was hoping to find out what the value of connected_device_name() is for my devices, so I go into template tester and enter

{:connected_device_name()}

and it gives me empty for no device, but when I connect a device, I get

EXCEPTION:  connected_device_name: invalid storage location ""

There are two problems here. The first is that you are using SFM so there is an 'invisible' argument being passed to connected_device_name(), in this case the empty string because no column was specified. The second is that connected_device_name has 1 parameter, the storage location for the device. (NB: devices can have a different name per storage location.) In your SFM template the empty string is being passed as the storage location, and the function is telling you that location isn't valid.

Because of the invisible argument, connected_device_name must be used in TPM or GPM templates.

TPM:

{:'connected_device_name('main')'}

GPM:

program: connected_device_name('main')

Originally Posted by compurandom

You listed other functions, so I figured I'd try them. so I tried booksize() and it gives

EXCEPTION:  evaluate() takes 5 positional arguments but 7 were given

as do the others.

Here as well the invisible arguments are fouling things up. The booksize() function has no parameters. As above, you can use the function only in TPM and GPM.

TPM:

{:'booksize()'}

GPM:

program: booksize()

Originally Posted by compurandom

What am I missing?

In summary, two things:

• Functions without parameters cannot be used in SFM.
• The template tester has documentation for all the built-in functions. That is where you can see the function's parameters.

Originally Posted by compurandom

Oh, and thanks very very very much for the template in search... I just saved a search that finds books with the wrong number of tags, something I've wanted for a very long time.

You are welcome. I imagined that given their complexity, template searches would be saved and I ensured that worked. I also think that stored templates will be used (called) from template searches, making the search easier to read and debug, and facilitating searches with small variations.

[compurandom]

Ok, that was extremely helpful. Looking at https://manual.calibre-ebook.com/template_lang.html I coudln't figure it out, because that whole page is written for SFM and I've had a hard time finding any documentation of the other two other than a few lines of examples that I coudln't make sense of.

I feel like the above page should have most of your above message either inserted at the top or right after the "advanced formatting" heading.

[compurandom]

And I do think SFM is useful -- it's significantly shorter in the context of just pulling tags and doing minor changes to format them, like in plugboard. But now that the other two are more clear to me, I'll probably use them for everything else.

[chaley]

Originally Posted by compurandom

And I do think SFM is useful -- it's significantly shorter in the context of just pulling tags and doing minor changes to format them, like in plugboard. But now that the other two are more clear to me, I'll probably use them for everything else.

If I had it to do again I would get rid of SFM, making TPM the default. This would remove the need for the :' and '} stuff, and require using quotes for constants and '$' for the field value.

Some benefits: it would get rid of the confusion around nested functions. It would normalize specifying arguments to functions. It would remove the restriction on zero-parameter functions. It would make escaping commas and quotes more rational. It would make much more of the template language available by default.

The cost: adding '$' almost everywhere and adding quotes in some places.

I have looked at whether I could automatically convert SFM templates to TPM templates. I think I could, but it is certain that in some cases the conversion would fail. Then what? Confusing millions of users is seldom a good idea so (for the moment at least) I am not going there.

Another example: we just went through this calculation with the soon-to-be-released feature combining small tag browser groups together when using grouping by first letter. My initial take was "Oh no, we can't set a default that changes calibre's behavior." My second take was "If we set a default that might change behavior then a) people will know about the feature (discoverability), and b) it is easy to revert in Preferences." Kovid asked about and I ended up choosing the second, setting the threshold to 5, hoping that we won't see too many knives.

[compurandom]

Originally Posted by chaley

Another example: we just went through this calculation with the soon-to-be-released feature combining small tag browser groups together when using grouping by first letter.

That sounds like a feature I'm going to both love and hate.
I wish I could set the subcategorization scheme differently on, say, tags vs. saved searches.

Finding a threshold number that is a good compromise between all of them is hard.

[chaley]

Originally Posted by compurandom

I wish I could set the subcategorization scheme differently on, say, tags vs. saved searches.

Finding a threshold number that is a good compromise between all of them is hard.

In Preferences / Look & feel / Tag browser you can choose not to categorize some items. If that isn't sufficient, what are you suggesting? Are you looking to "partition" some and use "first letter" on others?

[compurandom]

Originally Posted by chaley

In Preferences / Look & feel / Tag browser you can choose not to categorize some items. If that isn't sufficient, what are you suggesting? Are you looking to "partition" some and use "first letter" on others?

Yes, maybe? I don't remember seeing the "dont parittion" setting before, maybe that helps. It's one of those things where I can't say, but I know it when I see it. More playing needed in time. And it sounds like there's a new setting where I'll play with it more.

Actually, some categories I'd like to partition into roughly 30 partitions, and others I'd like to partition by first letter (which gives about 30 partitions anyway). And one or two are already heirarchal and small and don't need to be partitioned at all.