View Single Post
Old 03-07-2024, 07:50 AM   #26
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,488
Karma: 8065348
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
I did some tests on my 77,500-book library.

With no data files at all, repeated runs took an average of 18.5 seconds. There was one outlier that took 30 seconds, which I think happened because my cloud auto-backup daemon kicked in.

With 40, 50, and 60 data files it took an average of 18.7 seconds, with no significant outliers.

I tried this python template that marks matched books instead of using search. It is slower, taking around 1.5 seconds longer than the search. I am sure that the difference is in the GUI refresh to show the marks, but I didn't do the work to verify it.

Code:
python:
def evaluate(book, context):
	from calibre.db.constants import DATA_FILE_PATTERN
    from time import monotonic
    start = monotonic()
	count = 0;
    db = context.db
    db.data.set_marked_ids(set())
    db = db.new_api
    found = set()
	for id_ in db.all_book_ids():
		files = tuple(f.relpath.partition('/')[-1] for f in
			db.list_extra_files(id_, use_cache=True, pattern=DATA_FILE_PATTERN))
		if len(files) > 0:
			#print(f'{id_}: {files}')
			count += 1
            found.add(id_)
    context.db.data.set_marked_ids(found)
	return (f'template search time: {monotonic() - start:.2f}, {count}')
All these tests are on Windows 10 Pro, 16GB Ram, an Intel Core i7-10710U CPU, a 512GB SSD, MS Defender A/V, and current calibre source running in debug mode. I have the following plugins installed:
Click image for larger version

Name:	Clipboard01.jpg
Views:	540
Size:	30.3 KB
ID:	206769

Bottom line: I can't reproduce a hang. The process can take several seconds. I think the process will slow down significantly if the computer is memory-challenged or if there are other processes competing for the CPU. In particular, an anti-virus that monitors file accesses could greatly affect performance because multiple accesses are required to check each book.
chaley is offline   Reply With Quote