Here are the comments on the virtual_libraries_for_books() calibre database method that you must be using directly or indirectly:
Code:
# We get here if resolving the books in a VL triggers another VL
# cache calculation. This can be 'real' recursion, for example a
# VL expression using a template that calls virtual_libraries(),
# or a search using a location of 'all' that causes evaluation
# of a composite that uses virtual_libraries(). The first case
# is an error and the exception message should appear somewhere.
# However, the error can seem nondeterministic. It might not be
# raised if the use is via a composite and that composite is
# evaluated before it is used in the search. The second case is
# also an error but if the composite isn't used in a VL then the
# eventual answer will be correct because get_metadata() will
# clear the caches.
In your case this is (probably) saying that the evaluation of more than one composite column is referencing the column containing virtual_libraries(). This could cause virtual_libraries() to be called recursively while evaluating the search. For example, searching a VL expression "x" could trigger recursion because the unprefixed value "x" could check the virtual_libraries column when it doesn't really need to. Similar things can happen if a composite references another composite that references the virtual_libraries() column.
A 'fix' might be to call virtual_libraries() in the column evaluation instead of referencing the composite column containing the call to virtual_libraries(). This would remove order of evaluation concerns. But TBH I can't say without having access to all the composites and VLs you have set up. I'm sure that the complexity of the your setup will make things very hard to understand.
Another fix might be to never use naked searches.
The "real" fix requires computing a graph of accesses then evaluating the nodes in graph order. This isn't going to happen.