apsw.ThreadingViolationError and cache.write_lock

[capink]

I have a plugin (Last Modified Plugin) that updates date columns in response to to database events. One problem I am facing is that when mulitple database events are happening in succession (e.g. adding multiple books), I sometimes get a variety of apsw.ThreadingViolationError, as illustrated below:

Spoiler:

Traceback (most recent call last):
File "threading.py", line 932, in _bootstrap_inner
File "/home/user/calibre/src/calibre/db/backup.py", line 54, in run
self.do_one()
File "/home/user/calibre/src/calibre/db/backup.py", line 125, in do_one
self.db.clear_dirtied(book_id, sequence)
File "/home/user/calibre/src/calibre/db/cache.py", line 77, in call_func_with_lock
return func(*args, **kwargs)
File "/home/user/calibre/src/calibre/db/cache.py", line 1305, in clear_dirtied
self.backend.mark_book_as_clean(book_id)
File "/home/user/calibre/src/calibre/db/backend.py", line 1808, in mark_book_as_clean
self.execute('DELETE FROM metadata_dirtied WHERE book=?', (book_id,))
File "/home/user/calibre/src/calibre/db/backend.py", line 937, in execute
return self.conn.cursor().execute(sql, bindings)
apsw.ThreadingViolationError: You are trying to use the same object concurrently in two threads or re-entrantly within the same thread which is not allowed.


Exception in thread Thread-6:
Traceback (most recent call last):
File "threading.py", line 932, in _bootstrap_inner
File "calibre/db/backup.py", line 54, in run
File "calibre/db/backup.py", line 125, in do_one
File "calibre/db/cache.py", line 77, in call_func_with_lock
File "calibre/db/cache.py", line 1305, in clear_dirtied
File "calibre/db/backend.py", line 1808, in mark_book_as_clean
File "calibre/db/backend.py", line 937, in execute
apsw.ThreadingViolationError: You are trying to use the same object concurrently


Traceback (most recent call last):
File "calibre/db/backup.py", line 63, in do_one
File "calibre/db/cache.py", line 77, in call_func_with_lock
File "calibre/db/cache.py", line 1196, in check_dirtied_annotations
File "calibre/db/backend.py", line 1986, in dirty_books_with_dirtied_annotations
apsw.ThreadingViolationError: You are trying to use the same object concurrently in two threads or re-entrantly within the same thread which is not allowed.
Traceback (most recent call last):
File "calibre/db/cache.py", line 1290, in get_metadata_for_dump
File "calibre/db/cache.py", line 2457, in all_annotations_for_book
File "calibre/db/backend.py", line 1867, in all_annotations_for_book
File "calibre/db/backend.py", line 937, in execute

I tried multiple solutions for this (e.g. using QTimer.singleShot, FunctionDispatcher and a combination of both), but the only solution that seems to be working for me is by using a cache.write_lock as illustrated in the code below:

    def update_date_column(self, cache, col, timestamp, book_ids):
        if DEBUG:
            prints('Last Modified Plugin: Update column ({}) for book_ids: {}'.format(col, book_ids))
        id_map = {book_id: timestamp for book_id in book_ids}
        f = cache.fields[col]
        with cache.write_lock:
            f.writer.set_books(id_map, cache.backend)

Since this is the first time I am using this, I just want to know if this is the right way to go about things? Are there any pitfalls or gotchas I should be aware of?

[kovidgoyal]

Just use

cache.set_field()

[capink]

OK. I went for set_books() to avoid triggering events for date fields updated by the plugin. But I can solve this by caching timestamps for updated date fields and dropping any events triggered for the said columns shortly after. Thanks for your reply.