Sporadic virtual library recursion

[ownedbycats]

On my virtual_libraries() column, I get spordaic "recursion detected when processing..." errors, which affect one of my reading-status VLs (though it seems to pick them at random).

The error disappears upon making an edit to a book (or even opening the MDE and saving without any changes).

I'm trying to figure out where the recursion may be coming from.

All three VLs use a template search checking a stored template, readstatus().

Currently Reading:

template:"program:readstatus()#@#:t:~(currentlyreading|toberead)"

Unread:

template:"program:readstatus()#@#:t:=unread"

Read:

template:"program:readstatus()#@#:t:=read"

readstatus() #percentread is an integer and #readinglist is a series-like.

program:

	if ($$#percentread >=#1 && $$#percentread <=#99) || ($#readinglist == 'Reference')
	then 'currentlyreading' 

	elif $#readinglist=='To Be Read' && $$#percentread ==#0
	then 'toberead' 

	elif $$#percentread >=#100
	then 'read' 

	elif $$#percentread == 'None'
	then 'undefined' 

	elif $$#percentread >=#0
	then 'unread' 

	fi

My thought was that the regex search for Currently Reading was messing up things, but this would make more sense if it was the only one getting the errors. Any other ideas?

[chaley]

Here are the comments on the virtual_libraries_for_books() calibre database method that you must be using directly or indirectly:

                # We get here if resolving the books in a VL triggers another VL
                # cache calculation. This can be 'real' recursion, for example a
                # VL expression using a template that calls virtual_libraries(),
                # or a search using a location of 'all' that causes evaluation
                # of a composite that uses virtual_libraries(). The first case
                # is an error and the exception message should appear somewhere.
                # However, the error can seem nondeterministic. It might not be
                # raised if the use is via a composite and that composite is
                # evaluated before it is used in the search. The second case is
                # also an error but if the composite isn't used in a VL then the
                # eventual answer will be correct because get_metadata() will
                # clear the caches.

In your case this is (probably) saying that the evaluation of more than one composite column is referencing the column containing virtual_libraries(). This could cause virtual_libraries() to be called recursively while evaluating the search. For example, searching a VL expression "x" could trigger recursion because the unprefixed value "x" could check the virtual_libraries column when it doesn't really need to. Similar things can happen if a composite references another composite that references the virtual_libraries() column.

A 'fix' might be to call virtual_libraries() in the column evaluation instead of referencing the composite column containing the call to virtual_libraries(). This would remove order of evaluation concerns. But TBH I can't say without having access to all the composites and VLs you have set up. I'm sure that the complexity of the your setup will make things very hard to understand.

Another fix might be to never use naked searches.

The "real" fix requires computing a graph of accesses then evaluating the nodes in graph order. This isn't going to happen.

[ownedbycats]

I looked through my composite columns for any reference to virtual_libraries().

I found one in the composite I use to generate part of an SFM save template, but I don't think it's referenced anywhere else (except the save template obviously). I changed it to use a non-VL check instead. (Actually, that reminds me of something. I'll ask in the template questions thread.)

I also ran through my Cleanup saved searches as I use them for my 'Cleanup' VL. Nothing checking virtual libaries there.

There's a few virtual_libraries() references in my column icons. I don't think there's any relation there especially since I don't think it's possible to reference the icons elsewhere.

Another thought: I do have a custom column for displaying the read status in the tag browser and book details. I wonder if making the VL reference that instead would change anything.

program:
	if readstatus()=='currentlyreading' then 'Currently Reading'
	elif readstatus()=='toberead' then 'To Be Read'
	elif readstatus()=='read' then 'Read'
	elif readstatus()=='unread' then 'Unread'
	elif readstatus()=='undefined' then ''	
	fi

(Could probably make this a switch_if, will try later.)

[chaley]

Originally Posted by ownedbycats

I looked through my composite columns for any reference to virtual_libraries().

I found one in the composite I use to generate part of my save-to-device save template, but I don't think it's referenced anywhere else (except the save template obviously). I changed it to use a non-VL check instead.

I also ran through my Cleanup saved searches as I use them for my 'Cleanup' VL. Nothing checking virtual libaries there.

There's a few virtual_libraries() references in my column icons. I don't think there's any relation there especially since I don't think it's possible to reference the icons elsewhere.

Another thought: I do have a custom column for displaying the read status in the tag browser and book details. I wonder if making the VL reference that instead would change anything.

program:
	if readstatus()=='currentlyreading' then 'Currently Reading'
	elif readstatus()=='toberead' then 'To Be Read'
	elif readstatus()=='read' then 'Read'
	elif readstatus()=='unread' then 'Unread'
	elif readstatus()=='Undefined' then ''	
	fi

(Could probably make this a switch_if, will try later.)

Do you have a column showing the virtual libraries for a book? If so you should look for references to that column, including naked searches.

The source for the readstatus() stored template doesn't seem to reference composites. How could it be changed to use virtual_libraries()?

FWIW: In the program above, you should call readstatus() once, save the result to a variable, then check that variable. Calling it on every if branch will be much slower.

[ownedbycats]

Yes, I have a #vls column. I looked for references to that too but didn't see any.

No naked searches, I don't think. I use limited searches anyways and only on non-composites. All my VL searches use either exact or regex searches:

- Several have an exact search on non-composite #booktype.
- Two have format:whatever (for physical and loaned books)
- Two other (purchases and series) have fieldname:true. Series also has a regex to exclude several entries from #booktype.
- Cleanup is based on a nested saved search (it references a bunch of smaller saved searches). I looked through the searches and found no references to composite columns or virtual_libraries() and no naked searches.

readstatus() only calls two non-composite columns. The VL searches then call readstatus().

[chaley]

Originally Posted by ownedbycats

Yes, I have a #vls column. I looked for references to that too but didn't see any.

No naked searches, I don't think. I use limited searches anyways and only on non-composites. All my VL searches use either exact or regex searches:

- Several have an exact search on non-composite #booktype.
- Two have format:whatever (for physical and loaned books)
- Two other (purchases and series) have fieldname:true. Series also has a regex to exclude several entries from #booktype.
- Cleanup is based on a nested saved search (it references a bunch of smaller saved searches). I looked through the searches and found no references to composite columns or virtual_libraries() and no naked searches.

readstatus() only calls two non-composite columns. The VL searches then call readstatus().

Well, I can't help more with the info I have.

To help, run in debug mode. If you have disabled exception tracing in the formatter with the hidden tweak, reenable it. When it happens again note the exact error message and post the exception trace from the debug log.

[ownedbycats]

I'll try that -- it sometimes happens on startup. But since it's spordaic it might take a while

[chaley]

Originally Posted by ownedbycats

I'll try that -- it sometimes happens on startup. But since it's spordaic it might take a while

I can wait.