If you can run from source, please do ...

[kovidgoyal]

IIRC but dont quote me on this libprs500 was renamed to calibre around 2008 version 0.6

[Comfy.n]

https://web.archive.org/web/20091128....com/whats-new

a while ago this got me curious, then I found this early 0.6 changelog

[ownedbycats]

Surprised how little the site design has changed. (And it's not even that dated-looking.)

[Quoth]

Decent designs never look dated, but fads and fashion is quickly dated.

[JSWolf]

Originally Posted by Quoth

Decent designs never look dated, but fads and fashion is quickly dated.

I just wear what I like and find comfortable. I'm not the latest fasion unless it's something I want.

I did run LibPRS500 way back when because I had a PRS500.

[ownedbycats]

Can anyone else replicate this bug?

This saved search indefinitely freezes Calibre when running from source (eventually I kill process), but not on stable:

template:"""program: has_extra_files()#@#:b:yes"""

[DNSB]

Originally Posted by ownedbycats

Can anyone else replicate this bug?

This saved search indefinitely freezes Calibre when running from source (eventually I kill process), but not on stable:

template:"""program: has_extra_files()#@#:b:yes"""

It doesn't hang on my computer but since no extra files, it also returns no results.

[Comfy.n]

Originally Posted by ownedbycats

Can anyone else replicate this bug?

This saved search indefinitely freezes Calibre when running from source (eventually I kill process), but not on stable:

template:"""program: has_extra_files()#@#:b:yes"""

Yes I see a hiccup, I too have this saved search. Although in my case this was not an indefinite freeze, it took like 10 seconds to show the only 6 titles in my library that contain data files. Then I ran it on stable Calibre and it took more or less the same time to return the results.

edit: takes more like 15 to 17 seconds.

On the other hand, I have a "has author notes" python search kindly provided by C.haley, that runs very fast, returning 4600 titles almost instantly:

Spoiler:

template:"""python:
def evaluate(book, context):
db = context.db.new_api
authors_with_notes = context.globals.get('authors_with_notes')
if authors_with_notes is None:
print('authors notes is none')
authors_with_notes = db.get_all_items_that_have_notes('authors')
context.globals['authors_with_notes'] = authors_with_notes
context.globals['item_name_map'] = db.get_item_name_map('authors')
item_name_map = context.globals['item_name_map']
for aut in book.authors:
if item_name_map.get(aut) in authors_with_notes:
return '1'
return ''#@#:b:True"""

[ownedbycats]

I have about 60 books with data files (mostly alternative covers, or backups of the covers for fanfics). It takes about 5-7 seconds to load the search on stable Calibre, less if I'm in a VL.

Further tests I noticed on source the indefinite freeze is less likely to occur if I'm in a VL.

[chaley]

It doesn't freeze for me, using current source and in debug mode.

The only library I have with extra files is small, 100 books. The search is near instantaneous. On a large library with 77,000 books but no extra files the search takes a few seconds.

The function will take a while to run. It must look in every book folder to see if there is a non-empty data subfolder. The answer is cached so the future runs will be faster. The cache is cleared whenever an operation changes the contents of one of the folders.

Of note: has_extra_files() is not affected by the changes I made to get_metadata(). It doesn't use it.

Questions:

• What happens if you are running in debug mode?
• How long did you wait? Does it eventually finish if you wait a long time?
• Is it using the processor?
• Is it using the disk/ssd?

[chaley]

I did some tests on my 77,500-book library.

With no data files at all, repeated runs took an average of 18.5 seconds. There was one outlier that took 30 seconds, which I think happened because my cloud auto-backup daemon kicked in.

With 40, 50, and 60 data files it took an average of 18.7 seconds, with no significant outliers.

I tried this python template that marks matched books instead of using search. It is slower, taking around 1.5 seconds longer than the search. I am sure that the difference is in the GUI refresh to show the marks, but I didn't do the work to verify it.

python:
def evaluate(book, context):
	from calibre.db.constants import DATA_FILE_PATTERN
    from time import monotonic
    start = monotonic()
	count = 0;
    db = context.db
    db.data.set_marked_ids(set())
    db = db.new_api
    found = set()
	for id_ in db.all_book_ids():
		files = tuple(f.relpath.partition('/')[-1] for f in
			db.list_extra_files(id_, use_cache=True, pattern=DATA_FILE_PATTERN))
		if len(files) > 0:
			#print(f'{id_}: {files}')
			count += 1
            found.add(id_)
    context.db.data.set_marked_ids(found)
	return (f'template search time: {monotonic() - start:.2f}, {count}')

All these tests are on Windows 10 Pro, 16GB Ram, an Intel Core i7-10710U CPU, a 512GB SSD, MS Defender A/V, and current calibre source running in debug mode. I have the following plugins installed:


Bottom line: I can't reproduce a hang. The process can take several seconds. I think the process will slow down significantly if the computer is memory-challenged or if there are other processes competing for the CPU. In particular, an anti-virus that monitors file accesses could greatly affect performance because multiple accesses are required to check each book.

[Comfy.n]

Originally Posted by chaley

I did some tests on my 77,500-book library.

With no data files at all, repeated runs took an average of 18.5 seconds. There was one outlier that took 30 seconds, which I think happened because my cloud auto-backup daemon kicked in.

With 40, 50, and 60 data files it took an average of 18.7 seconds, with no significant outliers.

I tried this python template that marks matched books instead of using search. It is slower, taking around 1.5 seconds longer than the search. I am sure that the difference is in the GUI refresh to show the marks, but I didn't do the work to verify it.

python:
def evaluate(book, context):
	from calibre.db.constants import DATA_FILE_PATTERN
    from time import monotonic
    start = monotonic()
	count = 0;
    db = context.db
    db.data.set_marked_ids(set())
    db = db.new_api
    found = set()
	for id_ in db.all_book_ids():
		files = tuple(f.relpath.partition('/')[-1] for f in
			db.list_extra_files(id_, use_cache=True, pattern=DATA_FILE_PATTERN))
		if len(files) > 0:
			#print(f'{id_}: {files}')
			count += 1
            found.add(id_)
    context.db.data.set_marked_ids(found)
	return (f'template search time: {monotonic() - start:.2f}, {count}')

All these tests are on Windows 10 Pro, 16GB Ram, an Intel Core i7-10710U CPU, a 512GB SSD, MS Defender A/V, and current calibre source running in debug mode. I have the following plugins installed:
Attachment 206769

Bottom line: I can't reproduce a hang. The process can take several seconds. I think the process will slow down significantly if the computer is memory-challenged or if there are other processes competing for the CPU. In particular, an anti-virus that monitors file accesses could greatly affect performance because multiple accesses are required to check each book.

I copied the python code above and added the prefix template:""" as well as the ending """.
I wanted to compare timings to the GPM mode, but I get an error while trying this search. I mean, the search bar gets red. :\

Spoiler:

template:"""python:
def evaluate(book, context):
from calibre.db.constants import DATA_FILE_PATTERN
from time import monotonic
start = monotonic()
count = 0;
db = context.db
db.data.set_marked_ids(set())
db = db.new_api
found = set()
for id_ in db.all_book_ids():
files = tuple(f.relpath.partition('/')[-1] for f in
db.list_extra_files(id_, use_cache=True, pattern=DATA_FILE_PATTERN))
if len(files) > 0:
#print(f'{id_}: {files}')
count += 1
found.add(id_)
context.db.data.set_marked_ids(found)
return (f'template search time: {monotonic() - start:.2f}, {count}')"""

[chaley]

Originally Posted by Comfy.n

I copied the python code above and added the prefix template:""" as well as the ending """. I wanted to compare timings to the GPM mode, but I get an error while trying this search.

That template can't be used as a template search. It returns the wrong things. To run it, use the template tester.

[ownedbycats]

My steps:

1. Restarted in debug mode.
2. Switched to 'all books.'
3. Opened the task manager and put the window on top.
4. Clicked my 'has extra files' saved search.

Observations:

1. Disk usage on my library disk went to 100%.
2. No noticable increase of CPU usage.
3. The search completed after about 10-15 seconds.

The debug log didn't show anything really, just all the usual startup messages.

I repeated the steps except not in debug mode. Same results. I did reboot overnight, so maybe something else was interfering and causing the freeze.

[DNSB]

Originally Posted by ownedbycats

I repeated the steps except not in debug mode. Same results. I did reboot overnight, so maybe something else was interfering and causing the freeze.

Something I really hated when I worked in IT was telling someone to reboot their computer. Very often it "fixed" the problem so was pretty much the first thing we had the user try but no help in figuring out what the root issue was.