Strangeness when referencing VLs in other VLs

[ownedbycats]

I have a virtual_libraries() column. However, I'm having a bit of a strange interaction with it and I'm not sure if it's a glitch or some sort of strange user error.

I have two VLs here:

Fiction: tags:"=.Fiction"

Cleanup: search:"=Cleanup"

I wanted to hide all the "Cleanup" books in the Fiction VL. So I did this:

tags:"=.Fiction" AND NOT #vls:"=Cleanup"

This search worked as expected when I tested it in All Books. But when I put it into the Fiction VL itself, it results in every VL disappearing from books that have a tag starting with Fiction.

Here's one book before and after modifying the VL:





The weird part is that all the VLs are still returning their correct books, and if I use template tester to run virtual_libraries() directly on those books, it still shows the correct ones:







I did some further testing too. Two VLs here:

Physical Books: formats:paperbook

Loans: formats:overdrive


No overlapping books between the two of them. But when I changed VL to formats:paperbook AND NOT #vls:"=Loans" I got the same results: VLs disappearing from books with a paperbook format, virtual_libraries() giving expected results.

Any clue what might be happening here?

[chaley]

One possible problem: the search term for virtual libraries is "vl", not "vls". Try not using your column but using the underlying search term.

Do you really mean identity search? Don't you want contains?

EDIT: I haven't been able to make it fail in the same way.

[chaley]

Yes I can repeat it. VL search expressions shouldn't contain references to columns that use virtual_libraries(). The virtual_libraries() caching mechanism doesn't support recursion. This isn't something I am interesting in fixing.

[ownedbycats]

My column is #vls -- I built the search from the tag browser just to make sure I didn't make a typo.

tags:"=.Fiction" AND NOT vl:Cleanup resulted in the VLs staying in book details.

[ownedbycats]

Originally Posted by chaley

Yes I can repeat it. VL search expressions shouldn't contain references to columns that use virtual_libraries(). The virtual_libraries() caching mechanism doesn't support recursion. This isn't something I am interesting in fixing.

In that case it might make more sense to have my VL reference the Cleanup search instead of the VL.

[chaley]

To add some detail: when virtual_libraries() is used in a VL search expression the computation required is "To compute the books in this virtual library I must know what books are in this virtual library." This is a recursive question and is not computable. Instead of having calibre roll over dead I made the choice to answer "The books are in no VL."

Usually the "No books" answer doesn't hurt anything. Virtual libraries are really just searches. There are cases where a search must evaluate all composite columns (unprefixed search terms with no search field limits). If a composite column uses virtual_libraries() we have a recursive search. However, if the composite column value isn't used in the search then no harm is done and the search returns the correct value.

However, your case shows that the "No books" answer isn't optimal when a VL really is misconfigured. To fix this I submitted a change that raises an error instead of returning "No books". The error will usually be displayed by the composite if it had to recurse. The change doesn't (seem to) break the "usually" case described above.

Why "usually be displayed"? If the composite column is evaluated before it is used in the VL expression then it works. The cache prevents the recursion. The order of composite evaluation is complicated. Sometimes the evaluation order works and sometimes it doesn't.

[ownedbycats]

That makes sense.

Would the same issues occur if a VL was referenced in a saved search used by a different VL?

[chaley]

Originally Posted by ownedbycats

That makes sense.

Would the same issues occur if a VL was referenced in a saved search used by a different VL?

It depends on the type of reference. if the VL search references a saved search that uses virtual_libraries() then yes, probably. The bottom line: the safest approach is to never use virtual_libraries() in a VL search, directly or indirectly.