Performance in Bulk Metadata Changes

[pfooti]

Preface this with: sorry if this is a known issue, I've searched the forums a bit and couldn't find much about it.

I've been playing around with Calibre to handle a fairly large library. For this, I am very thankful. However, one thing I've noted is that changing meta-data within a file can take a very long time. For example, selecting ten books and saying "bulk edit" -> "swap author and title" (a common need of mine, due to poor tagging in the past) can pause the system for upwards of a minute or two.

What's more, because this conversion is running in the GUI response thread (instead of spawning a worker thread), the entire program freezes and is unresponsive during that time (I get a beach ball in OSX). Python isn't my preferred language, but I did check out the source and will poke at it at least a bit to see what's going on.

[user_none]

I believe the issue is the metadata database has to be locked to write the changes. This prevents anything else from working on it and locks up until the processes is complete.

[pfooti]

Originally Posted by user_none

I believe the issue is the metadata database has to be locked to write the changes. This prevents anything else from working on it and locks up until the processes is complete.

It's more than that, although it might not be an easily solvable problem in the current architecture. The callback for the Bulk Metadata Dialog box executes all the changes - if you look in metadata_bulk.py you'll see:

def sync(self):
for id in self.ids:

blablabla

If this were in C++ or Java, I'd know what to do - you create a new worker thread with everything in it. The issue is that making those changes takes a while, and is running in the GUI event thread. Simple work can happen in a GUI execution callback, but generally speaking anything that takes a while usually needs to be thrown into its own thread.

Dunno how to create a worker thread in python, but now's a good time to find out I imagine. Been meaning to add Python to my language repertoire. Of course if what you say is true, pushing the metadata update to another thread would introduce some locking issues - hooray for race conditions.

I have this theory about what's going on in general to cause this problem. The end of that method sets "self.changed = True", which looks to be happening inside the foreach loop, rather than at the end. If that's the case, any event handlers looking for changes to the database will get updated for each update, probably leading to the display's datamodel getting refreshed (re-sorted, re-rendered in the backing store) each iteration instead of once at the end. But it's hard to tell, because for some reason Python uses whitespace to define execution blocks, rather than curly braces or parentheses, making it relatively annoying to debug.

[kovidgoyal]

I could put the bulk metadata operations into a separate thread, but that wouldn't really solve anything. You shouldn't be doing anything else in calibre while a bulk metadata update is running as your data is in an inconsistent state. Moving to a separate thread will just mean you can display a progress bar while the user waits instead of a busy cursor.

Setting self.changed inside the loop doesn't matter. self.changed is read by the GUI only after the sync method terminates (remember it's happening in the GUI thread).