Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 01-02-2010, 05:54 PM   #1
pfooti
Junior Member
pfooti began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Jan 2010
Device: Kindle 2
Performance in Bulk Metadata Changes

Preface this with: sorry if this is a known issue, I've searched the forums a bit and couldn't find much about it.

I've been playing around with Calibre to handle a fairly large library. For this, I am very thankful. However, one thing I've noted is that changing meta-data within a file can take a very long time. For example, selecting ten books and saying "bulk edit" -> "swap author and title" (a common need of mine, due to poor tagging in the past) can pause the system for upwards of a minute or two.

What's more, because this conversion is running in the GUI response thread (instead of spawning a worker thread), the entire program freezes and is unresponsive during that time (I get a beach ball in OSX). Python isn't my preferred language, but I did check out the source and will poke at it at least a bit to see what's going on.

Last edited by pfooti; 01-02-2010 at 05:56 PM.
pfooti is offline   Reply With Quote
Old 01-02-2010, 06:10 PM   #2
user_none
Sigil & calibre developer
user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.
 
user_none's Avatar
 
Posts: 2,433
Karma: 950001
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
I believe the issue is the metadata database has to be locked to write the changes. This prevents anything else from working on it and locks up until the processes is complete.
user_none is offline   Reply With Quote
 
Enthusiast
Old 01-02-2010, 07:02 PM   #3
pfooti
Junior Member
pfooti began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Jan 2010
Device: Kindle 2
Quote:
Originally Posted by user_none View Post
I believe the issue is the metadata database has to be locked to write the changes. This prevents anything else from working on it and locks up until the processes is complete.
It's more than that, although it might not be an easily solvable problem in the current architecture. The callback for the Bulk Metadata Dialog box executes all the changes - if you look in metadata_bulk.py you'll see:

def sync(self):
for id in self.ids:

blablabla

If this were in C++ or Java, I'd know what to do - you create a new worker thread with everything in it. The issue is that making those changes takes a while, and is running in the GUI event thread. Simple work can happen in a GUI execution callback, but generally speaking anything that takes a while usually needs to be thrown into its own thread.

Dunno how to create a worker thread in python, but now's a good time to find out I imagine. Been meaning to add Python to my language repertoire. Of course if what you say is true, pushing the metadata update to another thread would introduce some locking issues - hooray for race conditions.

I have this theory about what's going on in general to cause this problem. The end of that method sets "self.changed = True", which looks to be happening inside the foreach loop, rather than at the end. If that's the case, any event handlers looking for changes to the database will get updated for each update, probably leading to the display's datamodel getting refreshed (re-sorted, re-rendered in the backing store) each iteration instead of once at the end. But it's hard to tell, because for some reason Python uses whitespace to define execution blocks, rather than curly braces or parentheses, making it relatively annoying to debug.
pfooti is offline   Reply With Quote
Old 01-02-2010, 07:59 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,419
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I could put the bulk metadata operations into a separate thread, but that wouldn't really solve anything. You shouldn't be doing anything else in calibre while a bulk metadata update is running as your data is in an inconsistent state. Moving to a separate thread will just mean you can display a progress bar while the user waits instead of a busy cursor.

Setting self.changed inside the loop doesn't matter. self.changed is read by the GUI only after the sync method terminates (remember it's happening in the GUI thread).
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
bulk metadata - 2 issues with series cybmole Calibre 7 09-27-2010 07:18 AM
Editing Metadata in Bulk ballast Calibre 5 08-15-2010 03:14 PM
Updating Metadata in Bulk Turt99 Calibre 5 06-07-2010 03:19 PM
metadata in bulk Lorraine Froggy Calibre 1 11-14-2009 09:42 PM
Bulk Metadata Download iain_benson Calibre 1 09-29-2009 11:42 AM


All times are GMT -4. The time now is 03:23 PM.


MobileRead.com is a privately owned, operated and funded community.