Life of Pi & Calibre's search engine

[PatNY]

I notice that the default search in Calibre is to search anywhere in all the books' metadata instead of restricting the search to the main fields -- such as the "title" and "author" fields.

This results in dozens of books coming up when I went to search for "Life of Pi" in my library. I put in "life pi" in the search field and got the dozens of books. (Try it; you will get dozens of books too even if you don't have "Life of Pi" in your library.)

So I went to advanced search and put in "life pi" in the title field and nothing came up. I finally figured out that the advanced search will only give results on multiple words if they are in the exact sequence they appear in the title. In other words, it won't simply treat "life pi" as two separate keywords and give results. You can successfully search on "life of" or "of pi" or "life of pi" but "life pi" will give you zero results.

So it seems that the default search is overly broad. And the advanced search is too restrictive.

Is there any way to restrict the default search to the title and author fields?

Is it also possible to make the advanced search a little less restrictive in terms of the word sequencing for a title?

[mbovenka]

Originally Posted by PatNY

Is there any way to restrict the default search to the title and author fields?

Is it also possible to make the advanced search a little less restrictive in terms of the word sequencing for a title?

You can restrict the default search in Preferences -> Searching -> Limit the searched metadata.

If you use search prefixes in the standard search bar (like title:life pi) it will use substring search on both, so it will also match something like 'blah Pioneer blah blah the life of blah...'

[chaley]

Originally Posted by PatNY

Is there any way to restrict the default search to the title and author fields?

See Preferences -> Searching -> Limit the searched metadata. You enter the names of the columns you want calibre to search when a naked (non-prefixed) term is used.

Edit: duplicates mbovenka's information. Sorry...

[theducks]

Originally Posted by PatNY

I notice that the default search in Calibre is to search anywhere in all the books' metadata instead of restricting the search to the main fields -- such as the "title" and "author" fields.

This results in dozens of books coming up when I went to search for "Life of Pi" in my library. I put in "life pi" in the search field and got the dozens of books. (Try it; you will get dozens of books too even if you don't have "Life of Pi" in your library.)

So I went to advanced search and put in "life pi" in the title field and nothing came up. I finally figured out that the advanced search will only give results on multiple words if they are in the exact sequence they appear in the title. In other words, it won't simply treat "life pi" as two separate keywords and give results. You can successfully search on "life of" or "of pi" or "life of pi" but "life pi" will give you zero results.

So it seems that the default search is overly broad. And the advanced search is too restrictive.

Is there any way to restrict the default search to the title and author fields?

Is it also possible to make the advanced search a little less restrictive in terms of the word sequencing for a title?

Preferences: Searching
Control non prefixed columns ...

[PatNY]

Duh, I should have looked more carefully in preferences.

I actually did look in preferences first (because I know the Calibre developers don't overlook much!) but I went straight to "Tweaks" to look for an adjustment.

So thanks to chaley, ducks and mobovenka for the quick reply.

I've been using Calibre for over a year and have often been vexed by the overly broad results with default searches. I can't believe I didn't inquire about this before.

So this is a change many others might want to do as it makes searching the library a lot easier.

[DoctorOhh]

I'm glad the restriction on what you search has helped you, but your search technique may need refining beyond what restricting fields will do for you.

Originally Posted by PatNY

This results in dozens of books coming up when I went to search for "Life of Pi" in my library. I put in "life pi" in the search field and got the dozens of books. (Try it; you will get dozens of books too even if you don't have "Life of Pi" in your library.)

If you type Life of Pi in the search area you are searching for any instance Life or of or Pi. If you search for "Life of Pi" you are searching for any instance of this exact string Life of Pi. The first search may return hundreds of books. The second search most likely a few.

Originally Posted by PatNY

So I went to advanced search and put in "life pi" in the title field and nothing came up.

An advanced Life Pi search in the title field produces this result in the search field, title:"life pi". Note it is looking for the exact string life pi within all of the titles and as you note no title has the exact string life pi in the title.

Originally Posted by PatNY

I finally figured out that the advanced search will only give results on multiple words if they are in the exact sequence they appear in the title. In other words, it won't simply treat "life pi" as two separate keywords and give results. You can successfully search on "life of" or "of pi" or "life of pi" but "life pi" will give you zero results.

All of these results are as expected, are not bugs and will not change in future releases.

Understanding the difference in the search techniques I've outlined above will go a lot further in helping you search your library than any limit you put on fields to search.

That said to speed up searches I do remove comments from my list of fields to search.

[Pinecone]

How would you do an AND search (which is what he seems to need).

life pi = life OR pi
"life pi" = exact string "life pi"
what gives us? = life AND pi

[itimpi]

Use the 'Life and pi'

As you mentioned if no operator keyword is present the the 'or' operator is assumed. Brackets can be used if needed to provide grouping.

[PatNY]

Originally Posted by dwanthny

If you type Life of Pi in the search area you are searching for any instance Life or of or Pi.

But Walt, that's not what I am seeing. If I search on life pi or life of pi it will only result in books that have ALL of those letter strings (not just whole words) in any of the fields (with no restrictions set) or in just the author and title fields if it's restricted the way I now have Calibre set.

In other words, per the very first search I made as referenced in my original post, every resulting book had to have BOTH life and pi somewhere in the book's metadata. If could be the word "life" in the title, for example, and then the presence of the word "conspiracy" in the book's summary. If there was no pi string anywhere in the metadata, then the presence of the word life in the title alone would not be enough to cause the book to show up in the results.

This is very good, because if it was really an "OR" situation, that would have caused even more irrelevant books to show up. As it was, a lot of books showed up because "pi" is a common string in so many ordinary words (pizza, conspiracy, masterpiece etc).

BTW, in my actual search in calibre, I was using no quotes around the search terms. I just used them in my original post here to try to make the post easier to understand. I also didn't think it was a bug. But I did think the default search method (without any tweaks) was non-intuitive and non-standard compared to the common search engines you find on websites. I imagine it could confuse many who use Calibre for the first time and have hundreds of books in their library, particularly books with summaries filled in.

[PatNY]

Originally Posted by itimpi

Use the 'Life and pi'

As you mentioned if no operator keyword is present the the 'or' operator is assumed. Brackets can be used if needed to provide grouping.

Actually, the "and" operator is assumed. It might seem like it is an "or" situation since so many books will come up in searches on default settings, but it's only because it is catching the search strings in the book summaries (or other metatdata fields) as well.

[kiwidude]

@PatNY - personally I 100% agree that the default should be to not search the comments fields. It is one of the first things I change when I install calibre for other people. However it is a battle I briefly tried to fight and lost a long time ago. The concession I did help win was in getting a preferences option available (thanks to chaley for the work) so at least users could do it without modifying source code.

I'm guessing the other devs haven't changed their mind on this issue by now, but you never know

And yes, not sure what special juice itimpi was drinking today when he posted, but searches are most definitely an and not an or when not specified.

[kovidgoyal]

What you want is

title:life title:pi

not "life pi". Incidentally, if you search "life pi" in google (with the quotes) it does not match the string "life of pi" either. Instead it matches a bunch of pages that happen to have life<punctuation>pi in them (look at the part of the search results in bold).

Interpreting quotes as denoting an exact string match is standard practice in pretty much every search engine I have come across. The only difference is google seems to interpret a space in a quoted string as being "any punctuation" which calibre does not.

[DoctorOhh]

Originally Posted by PatNY

But Walt, that's not what I am seeing. If I search on life pi or life of pi it will only result in books that have ALL of those letter strings (not just whole words) in any of the fields (with no restrictions set) or in just the author and title fields if it's restricted the way I now have Calibre set.

You are correct it is looking for any book entry that has Life and of and pi any place in the fields searched. But my bigger point is accurate if you enclose Life of Pi in quotes like "Life of Pi" it will look for the exact string.

Originally Posted by PatNY

BTW, in my actual search in calibre, I was using no quotes around the search terms.

Again that is my point, you have to learn to use quotes. The advanced search does use quotes and finds things exactly as anyone would expect. Searching for "life pi" does not return anything because the string does not exist in your library database. Same search rules go for searching via Google.

Originally Posted by PatNY

But I did think the default search method (without any tweaks) was non-intuitive and non-standard compared to the common search engines you find on websites.

I think the default search is as I would expect it to be in other search engines. The point of confusion, as kiwidude pointed out, is that searches automatically search the comments area too so there is quite a bit more data being searched then the obvious Title, Author, Series you would expect.

[PatNY]

Originally Posted by dwanthny

You are correct it is looking for any book entry that has Life and of and pi any place in the fields searched. But my bigger point is accurate if you enclose Life of Pi in quotes like "Life of Pi" it will look for the exact string.

I used simply life pi for expediency. In this case I knew the whole title before I searched, but don't you think it is non-intuitive to have to put quotes around every entered phrase in a default basic search? And if one has hundreds of books in their library and has only a vague memory of some keywords in the title, then having to know the exact order of words and put every search in quotes would be timely and perhaps too error-prone. Searches are meant to be quick and efficient.

The advanced search does use quotes and finds things exactly as anyone would expect.

You're saying whether you put quotes in or not, the advanced search will assume them, right? Because as I explained in my previous post, I didn't actually use quotes in my life pi advanced search. I only put them in the initial post in this thread to differentiate the words from the rest of the sentence.

I'm not aware of any advanced search function I've used before that assumes quotes around the search terms when they are not actually typed in.

Searching for "life pi" does return anything because the string does not exist in your library database. Same search rules go for searching via Google.

I went to google and put in "life pi" and the entire first page of results I got are related to either the book, the movie that is now being made of the book, or the book's cover image. So Google does something very different.

You get the same results if you do a Google search on just life pi. Everything on the first page is related to the book.

Sure, there may be lots of results not related to the book, but since Google ranks results by relevancy, there is not a problem. The point is you can put life pi in, with or without quotes, and always come up with the book first.

I think the default search is as I would expect it to be in other search engines. The point of confusion, as kiwidude pointed out, is that searches automatically search the comments area too so there is quite a bit more data being searched then the obvious Title, Author, Series you would expect.

But the fact that the default search looks at all the metadata is exactly what I was referring to when I said it was "non-intuitive" and "non-standard." It results in overly-broad results.

I agree with kiwidude that the book summary metadata should not be included in the default basic search. In all the time I've been using Calibre, I've never once made a search based on something that I was looking for in the books summary section. It is 100% either title or author.

Perhaps the option to search ALL the book's metadata including the book summary is something that should be reserved for the advanced search.

[PatNY]

Originally Posted by kovidgoyal

What you want is

title:life title:pi

not "life pi". Incidentally, if you search "life pi" in google (with the quotes) it does not match the string "life of pi" either. Instead it matches a bunch of pages that happen to have life<punctuation>pi in them (look at the part of the search results in bold).

Interpreting quotes as denoting an exact string match is standard practice in pretty much every search engine I have come across. The only difference is google seems to interpret a space in a quoted string as being "any punctuation" which calibre does not.

Kovid, as I explained to Walt, I didn't actually use quotes in the search I did in Calibre. I only used them in the first post in this thread to differentiate the search terms from the rest of the sentence. I realize now that this may have resulted in some confusion as to what I used, so I later resorted to bolding the terms.

I did the advanced search without quotes, and noticed that it will assume quotes even if you don't use them, and therefore I got ZERO results when searching for life pi. That is what also vexed me, in addition to the overly broad results with the basic search. I'm not aware of other advanced search engines that will automatically assume quotes when just keywords are entered in without them.

From my perspective, all is fine thanks to kiwidude and chaley who put in tweaks in Preferences. But I have to imagine anyone with hundreds of books in their library and who is relatively new to Calibre might be similarly perplexed by its search engine behavior.