[Comfy.n]

Originally Posted by chaley

I did some tests on my 77,500-book library.

With no data files at all, repeated runs took an average of 18.5 seconds. There was one outlier that took 30 seconds, which I think happened because my cloud auto-backup daemon kicked in.

With 40, 50, and 60 data files it took an average of 18.7 seconds, with no significant outliers.

I tried this python template that marks matched books instead of using search. It is slower, taking around 1.5 seconds longer than the search. I am sure that the difference is in the GUI refresh to show the marks, but I didn't do the work to verify it.

python:
def evaluate(book, context):
	from calibre.db.constants import DATA_FILE_PATTERN
    from time import monotonic
    start = monotonic()
	count = 0;
    db = context.db
    db.data.set_marked_ids(set())
    db = db.new_api
    found = set()
	for id_ in db.all_book_ids():
		files = tuple(f.relpath.partition('/')[-1] for f in
			db.list_extra_files(id_, use_cache=True, pattern=DATA_FILE_PATTERN))
		if len(files) > 0:
			#print(f'{id_}: {files}')
			count += 1
            found.add(id_)
    context.db.data.set_marked_ids(found)
	return (f'template search time: {monotonic() - start:.2f}, {count}')

All these tests are on Windows 10 Pro, 16GB Ram, an Intel Core i7-10710U CPU, a 512GB SSD, MS Defender A/V, and current calibre source running in debug mode. I have the following plugins installed:
Attachment 206769

Bottom line: I can't reproduce a hang. The process can take several seconds. I think the process will slow down significantly if the computer is memory-challenged or if there are other processes competing for the CPU. In particular, an anti-virus that monitors file accesses could greatly affect performance because multiple accesses are required to check each book.

I copied the python code above and added the prefix template:""" as well as the ending """.
I wanted to compare timings to the GPM mode, but I get an error while trying this search. I mean, the search bar gets red. :\

Spoiler:

template:"""python:
def evaluate(book, context):
from calibre.db.constants import DATA_FILE_PATTERN
from time import monotonic
start = monotonic()
count = 0;
db = context.db
db.data.set_marked_ids(set())
db = db.new_api
found = set()
for id_ in db.all_book_ids():
files = tuple(f.relpath.partition('/')[-1] for f in
db.list_extra_files(id_, use_cache=True, pattern=DATA_FILE_PATTERN))
if len(files) > 0:
#print(f'{id_}: {files}')
count += 1
found.add(id_)
context.db.data.set_marked_ids(found)
return (f'template search time: {monotonic() - start:.2f}, {count}')"""