Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 10-02-2020, 01:34 PM   #31
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 4,344
Karma: 641208
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Voyage
Quote:
Originally Posted by chaley View Post
I decided to take this on as a challenge and see if I could make templates work efficiently in searches. The answer is yes. it is available in calibre 5.1.
I am intrigued... but also mildly confused. Can you give a couple examples of search with a template?
JimmXinu is offline   Reply With Quote
Old 10-02-2020, 01:47 PM   #32
chaley
Grumpy old git
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 9,369
Karma: 3007492
Join Date: Jan 2010
Location: UK
Device: Reader
Quote:
Originally Posted by JimmXinu View Post
I am intrigued... but also mildly confused. Can you give a couple examples of search with a template?
There are two examples in the search interface section of the manual, at the very end of the section. I suggest you use the "Advanced search" dialog to build a few so you can see the syntax.

The idea, inspired by compurandom's query, is that some custom columns exist only for searching. These can go away, replaced by a search that uses a template to compute the value. Also, searches can now also use non-metadata template functions such as formats_sizes(), booksize(), connected_device_name(), current_library_name(), etc.

I am also working on "stored templates", which are basically template subprograms. These will interact well with template searches, permitting one to build a complex template to compute a value then "call" that template, with parameters if desired, from a search.

Last edited by chaley; 10-03-2020 at 03:23 AM.
chaley is offline   Reply With Quote
Old 10-03-2020, 03:24 AM   #33
chaley
Grumpy old git
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 9,369
Karma: 3007492
Join Date: Jan 2010
Location: UK
Device: Reader
Quote:
Originally Posted by chaley View Post
I am also working on "stored templates", which are basically template subprograms. These will interact well with template searches, permitting one to build a complex template to compute a value then "call" that template, with parameters if desired, from a search.
Stored templates are now in calibre source and in the 5.1.1 beta.
chaley is offline   Reply With Quote
Old 10-03-2020, 09:31 AM   #34
compurandom
Fanatic
compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.
 
Posts: 527
Karma: 294266
Join Date: Jun 2015
Device: kobo aura h2o, kobo forma
Quote:
Originally Posted by chaley View Post
The idea, inspired by compurandom's query, is that some custom columns exist only for searching. These can go away, replaced by a search that uses a template to compute the value.
The part of the original objection to this was that the columns cache calculated values. I figure that some templates are no worse than pulling the cached value, but have you noticed a performance difference with others?

Also, I've noticed that I can't sort one of my columns built from other columns, which makes me doubt how much caching is going on.
compurandom is offline   Reply With Quote
Old 10-03-2020, 10:52 AM   #35
chaley
Grumpy old git
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 9,369
Karma: 3007492
Join Date: Jan 2010
Location: UK
Device: Reader
Quote:
Originally Posted by compurandom View Post
The part of the original objection to this was that the columns cache calculated values. I figure that some templates are no worse than pulling the cached value, but have you noticed a performance difference with others?
I haven't noticed a performance difference because my library is too small. That said, I came to the conclusion that the composite column cache doesn't help enough to justify not adding the feature. When searching the cache is per book, so it helps only if a composite column is referenced directly or indirectly more than once in a query.

EDIT: What I said above is misleading. The cache is initialized for a book the first time that book is referenced, either in a search or by being displayed. It is cleared when data changes for a book. Thus repeated searches using non-template field references will use the cache.
Quote:
Also, I've noticed that I can't sort one of my columns built from other columns, which makes me doubt how much caching is going on.
This usually happens when the search/sort type set in the column definition doesn't match the what the template produced.

Last edited by chaley; 10-03-2020 at 11:18 AM.
chaley is offline   Reply With Quote
Old 10-03-2020, 11:43 AM   #36
compurandom
Fanatic
compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.
 
Posts: 527
Karma: 294266
Join Date: Jun 2015
Device: kobo aura h2o, kobo forma
Quote:
Originally Posted by chaley View Post
Quote:
Also, I've noticed that I can't sort one of my columns built from other columns
This usually happens when the search/sort type set in the column definition doesn't match the what the template produced.
Ah ha! That's been irritating me ever since I changed the column template! Thanks for the hint, fixing that is a huge relief!
compurandom is offline   Reply With Quote
Old 10-15-2020, 09:15 AM   #37
compurandom
Fanatic
compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.
 
Posts: 527
Karma: 294266
Join Date: Jun 2015
Device: kobo aura h2o, kobo forma
Ok, I just can't seem to get this to work in 5.2.

Quote:
Originally Posted by chaley View Post
Also, searches can now also use non-metadata template functions such as formats_sizes(), booksize(), connected_device_name(), current_library_name(), etc.
I was hoping to find out what the value of connected_device_name() is for my devices, so I go into template tester and enter

Code:
{:connected_device_name()}
and it gives me empty for no device, but when I connect a device, I get

Code:
EXCEPTION:  connected_device_name: invalid storage location ""
You listed other functions, so I figured I'd try them. so I tried booksize() and it gives
Code:
EXCEPTION:  evaluate() takes 5 positional arguments but 7 were given
as do the others.

What am I missing?
compurandom is offline   Reply With Quote
Old 10-15-2020, 10:20 AM   #38
compurandom
Fanatic
compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.
 
Posts: 527
Karma: 294266
Join Date: Jun 2015
Device: kobo aura h2o, kobo forma
Oh, and thanks very very very much for the template in search... I just saved a search that finds books with the wrong number of tags, something I've wanted for a very long time.
compurandom is offline   Reply With Quote
Old 10-15-2020, 12:00 PM   #39
chaley
Grumpy old git
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 9,369
Karma: 3007492
Join Date: Jan 2010
Location: UK
Device: Reader
Quote:
Originally Posted by compurandom View Post
Ok, I just can't seem to get this to work in 5.2.
Before we get into the specific questions, it is worth talking a bit about the three template program modes: Single Function Mode (SFM), Template Program Mode (TPM), and General Program Mode (GPM).

SFM was intended to be 'simple', so it hides a lot of programming language bits. For example, the value of the column is always passed as the first argument to a function included in the template. It also doesn't support the difference between variables and strings; all values are strings.

Example, the following SFM template returns either the series name of the string "no series"
Code:
{series:ifempty(no series)}
The equivalent template in GPM is
Code:
program: ifempty(field('series'), 'no series')
Note that the first argument to ifempty is the value of the field 'series' and the second argument is the string 'no series'.

TPM is very close to GPM, but with oddities related to processing of {} characters and strings.

Personally, I use GPM unless the template is trivial, in which case I might use TPM. I never use SFM. In fact, if I could redo the template language development I would not build SFM.
Quote:
Originally Posted by compurandom View Post
I was hoping to find out what the value of connected_device_name() is for my devices, so I go into template tester and enter

Code:
{:connected_device_name()}
and it gives me empty for no device, but when I connect a device, I get

Code:
EXCEPTION:  connected_device_name: invalid storage location ""
There are two problems here. The first is that you are using SFM so there is an 'invisible' argument being passed to connected_device_name(), in this case the empty string because no column was specified. The second is that connected_device_name has 1 parameter, the storage location for the device. (NB: devices can have a different name per storage location.) In your SFM template the empty string is being passed as the storage location, and the function is telling you that location isn't valid.

Because of the invisible argument, connected_device_name must be used in TPM or GPM templates.

TPM:
Code:
{:'connected_device_name('main')'}
GPM:
Code:
program: connected_device_name('main')
Quote:
Originally Posted by compurandom View Post
You listed other functions, so I figured I'd try them. so I tried booksize() and it gives
Code:
EXCEPTION:  evaluate() takes 5 positional arguments but 7 were given
as do the others.
Here as well the invisible arguments are fouling things up. The booksize() function has no parameters. As above, you can use the function only in TPM and GPM.

TPM:
Code:
{:'booksize()'}
GPM:
Code:
program: booksize()
Quote:
Originally Posted by compurandom View Post
What am I missing?
In summary, two things:
  • Functions without parameters cannot be used in SFM.
  • The template tester has documentation for all the built-in functions. That is where you can see the function's parameters.

Quote:
Originally Posted by compurandom View Post
Oh, and thanks very very very much for the template in search... I just saved a search that finds books with the wrong number of tags, something I've wanted for a very long time.
You are welcome. I imagined that given their complexity, template searches would be saved and I ensured that worked. I also think that stored templates will be used (called) from template searches, making the search easier to read and debug, and facilitating searches with small variations.
chaley is offline   Reply With Quote
Old 10-15-2020, 12:22 PM   #40
compurandom
Fanatic
compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.
 
Posts: 527
Karma: 294266
Join Date: Jun 2015
Device: kobo aura h2o, kobo forma
Ok, that was extremely helpful. Looking at https://manual.calibre-ebook.com/template_lang.html I coudln't figure it out, because that whole page is written for SFM and I've had a hard time finding any documentation of the other two other than a few lines of examples that I coudln't make sense of.

I feel like the above page should have most of your above message either inserted at the top or right after the "advanced formatting" heading.
compurandom is offline   Reply With Quote
Old 10-15-2020, 12:35 PM   #41
compurandom
Fanatic
compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.
 
Posts: 527
Karma: 294266
Join Date: Jun 2015
Device: kobo aura h2o, kobo forma
And I do think SFM is useful -- it's significantly shorter in the context of just pulling tags and doing minor changes to format them, like in plugboard. But now that the other two are more clear to me, I'll probably use them for everything else.
compurandom is offline   Reply With Quote
Old 10-15-2020, 02:57 PM   #42
chaley
Grumpy old git
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 9,369
Karma: 3007492
Join Date: Jan 2010
Location: UK
Device: Reader
Quote:
Originally Posted by compurandom View Post
And I do think SFM is useful -- it's significantly shorter in the context of just pulling tags and doing minor changes to format them, like in plugboard. But now that the other two are more clear to me, I'll probably use them for everything else.
If I had it to do again I would get rid of SFM, making TPM the default. This would remove the need for the :' and '} stuff, and require using quotes for constants and '$' for the field value.

Some benefits: it would get rid of the confusion around nested functions. It would normalize specifying arguments to functions. It would remove the restriction on zero-parameter functions. It would make escaping commas and quotes more rational. It would make much more of the template language available by default.

The cost: adding '$' almost everywhere and adding quotes in some places.

I have looked at whether I could automatically convert SFM templates to TPM templates. I think I could, but it is certain that in some cases the conversion would fail. Then what? Confusing millions of users is seldom a good idea so (for the moment at least) I am not going there.

Another example: we just went through this calculation with the soon-to-be-released feature combining small tag browser groups together when using grouping by first letter. My initial take was "Oh no, we can't set a default that changes calibre's behavior." My second take was "If we set a default that might change behavior then a) people will know about the feature (discoverability), and b) it is easy to revert in Preferences." Kovid asked about and I ended up choosing the second, setting the threshold to 5, hoping that we won't see too many knives.
chaley is offline   Reply With Quote
Old 10-15-2020, 03:16 PM   #43
compurandom
Fanatic
compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.
 
Posts: 527
Karma: 294266
Join Date: Jun 2015
Device: kobo aura h2o, kobo forma
Quote:
Originally Posted by chaley View Post
Another example: we just went through this calculation with the soon-to-be-released feature combining small tag browser groups together when using grouping by first letter.
That sounds like a feature I'm going to both love and hate.
I wish I could set the subcategorization scheme differently on, say, tags vs. saved searches.

Finding a threshold number that is a good compromise between all of them is hard.
compurandom is offline   Reply With Quote
Old 10-15-2020, 03:32 PM   #44
chaley
Grumpy old git
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 9,369
Karma: 3007492
Join Date: Jan 2010
Location: UK
Device: Reader
Quote:
Originally Posted by compurandom View Post
I wish I could set the subcategorization scheme differently on, say, tags vs. saved searches.

Finding a threshold number that is a good compromise between all of them is hard.
In Preferences / Look & feel / Tag browser you can choose not to categorize some items. If that isn't sufficient, what are you suggesting? Are you looking to "partition" some and use "first letter" on others?
chaley is offline   Reply With Quote
Old 10-15-2020, 03:46 PM   #45
compurandom
Fanatic
compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.compurandom ought to be getting tired of karma fortunes by now.
 
Posts: 527
Karma: 294266
Join Date: Jun 2015
Device: kobo aura h2o, kobo forma
Quote:
Originally Posted by chaley View Post
In Preferences / Look & feel / Tag browser you can choose not to categorize some items. If that isn't sufficient, what are you suggesting? Are you looking to "partition" some and use "first letter" on others?

Yes, maybe? I don't remember seeing the "dont parittion" setting before, maybe that helps. It's one of those things where I can't say, but I know it when I see it. More playing needed in time. And it sounds like there's a new setting where I'll play with it more.

Actually, some categories I'd like to partition into roughly 30 partitions, and others I'd like to partition by first letter (which gives about 30 partitions anyway). And one or two are already heirarchal and small and don't need to be partitioned at all.
compurandom is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Multiple same book on device: which search expression? un_pogaz Library Management 3 12-29-2018 11:50 AM
Aura Can't search bookstore on my device MV89 Kobo Reader 9 11-29-2013 05:21 PM
Search for Books on Device random43743 Calibre 3 04-23-2011 10:44 AM
search for books not on device alansplace Calibre 2 01-05-2011 01:07 PM
Search restriction bug when device is plugged in Manichean Calibre 6 09-11-2010 04:52 PM


All times are GMT -4. The time now is 01:08 AM.


MobileRead.com is a privately owned, operated and funded community.