02-19-2009, 04:31 PM | #1 |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
Management issues with large databases
Notice there was a problem with caliber and HUGE dbs. Uploadaed a library of about 20 000 volumes and when i want to delete a few of them it goes into a non responsive mode. Tried just deleting one and it worked after 3 min, but for deleting something like 6000 volumes... well its been going at it for more than 40 min and shows as non responsive. Btw is caliber using any form of sql querries to manage the books? If its using a file to store the information might be the problem, and that might explain the issues with a large number of files.
anyway just wanted to point it out. Last edited by Student1; 02-19-2009 at 04:36 PM. |
02-19-2009, 04:36 PM | #2 |
creator of calibre
Posts: 44,323
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
It stores metadata in an sqlite database. Deleting should be pretty quick, though I haven't really tested it with large databases.
Ah looking at the code, I see why it is slow (basically the GUI updates after each delete), should be fixed in the next release. |
Advert | |
|
02-19-2009, 04:52 PM | #3 |
Wizard
Posts: 4,553
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
|
02-19-2009, 04:55 PM | #4 |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
great thanks! That going to help alot!!!
|
02-19-2009, 04:58 PM | #5 |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
|
Advert | |
|
02-19-2009, 04:58 PM | #6 |
creator of calibre
Posts: 44,323
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
|
02-19-2009, 05:00 PM | #7 |
creator of calibre
Posts: 44,323
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Updating the progress dialog is not an expensive operation, so removing that wont yield significant speedups
|
02-19-2009, 05:11 PM | #8 |
Wizard
Posts: 4,553
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
I think that there are not probably not enough calls in the main add loop to allow other windows processes to run! I find that while the add is running, things like starting new copies in Explorer seem to hang until either the progress dialog updates (when there is a burst of activity and any hung copies now start) or the total add process finishes.
I suspect this may well be a Windows specific issue with event handling, and it may also depend on whether you have a multi-core machine which changes the timing issues relating to event handling. However I could be wrong - Windows multi-tasking can sometimes behave in very strange ways. |
02-19-2009, 05:16 PM | #9 |
Groupie
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
|
Ok, was worth a try. Btw you really update this piece of software extremely fast! Great work!!
|
02-19-2009, 05:24 PM | #10 | |
Grand Sorcerer
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
|
Quote:
Some languages have some type of Yield command to get around this. But, the "right" way to do it is to run the process on another thread. Although I think Kovid does spawn another process and communicate with IP, perhaps he is putting the UI thread in a wait state rather than using some type of call back. BUt, I'm only guessing based on issues I have seen with windows apps. BOb Last edited by pilotbob; 02-19-2009 at 05:43 PM. |
|
02-19-2009, 05:24 PM | #11 |
creator of calibre
Posts: 44,323
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
This is the way recursive adding is structured:
1) there is a main GUI thread in which all operations that use the GUI toolkit must happen. This thread loops delivering events to the various GUI widgets 2) A progress dialog is opened in the main thread and a new thread is spawned to do the adding of books 3) The new thread first recursively finds all ebooks and creates a list of them 4) For each ebook, metadata is read. unfortunately because metadata reading sometimes requires GUI operations, it has to happen in the main GUI thread, this is the Reading metadata phase and is when the UI should be the least responsive 5) Once all metadata is read, the metadata + actual files are added to the library in the worker thread (technically, because sqlite in python is not thread safe actualt database modifications happen in yet another "database access" thread). This phase is the adding books to database phase If all of windows is slowing down during any of these phases, it just means that Windows' kernel scheduler really sucks. Note that you can avoid all this mess with threads by just using the command line Code:
calibredb add --help |
02-19-2009, 05:46 PM | #12 |
Grand Sorcerer
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
|
|
02-19-2009, 06:13 PM | #13 |
Wizard
Posts: 4,553
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
The strange thing is that it is not all of Windows that is slowing down. Apps that are already running are fine. It is just when I try to copy/paste files between Explorer Windows while Calibre is adding books the Explorer windows involved in the copy/paste seem to hang. All other windows (and even some with long-running copy operations) continue to run fine. Other Explorer windows are also fine as long as I do not attempt a copy/paste type operation.
I am quite prepared to believe that it is an idiosyncrasy of Windows Explorer - it was just strange that specific operations in one app (calibre) can have the effect they do on another one (Explorer). These are not things one should see in a proper multi-tasking OS. I am guessing that there is some mutext or something like that that the apps are fighting over. |
02-19-2009, 06:17 PM | #14 | |
Grand Sorcerer
Posts: 19,832
Karma: 11844413
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
|
Quote:
Are you running a virus software? If so, exclude your calibre folder from on-access scan. Also, I turn my recycle bin off... it makes a big difference when moving/deleting files in explorer. But, yes you can't recover files. BOb |
|
02-19-2009, 06:25 PM | #15 |
Wizard
Posts: 4,553
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
It is something more obscure than that.
I have tried running procmon in parallel when these symptoms occur. It definitely looks as though something is blocking Windows Explorer. I normally have Recycle bin disabled, and I have tested with Virus checking on/off and it makes no difference. The calibre folders have already been excluded explicitly in the Virus checker as well. I can live with the issue easily enough, but I thought it was worth bringing up in case it sparked a thought in someone's mind. I may find the time to run a much more detailed analysis of all what is running under the covers and track all calls to the system but the amount of diagnostic information one has to wade through to get to the bottom of such issues is a none-trivial task. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Multiple Databases | danchisum | Calibre | 1 | 07-29-2010 09:35 PM |
Charging Issues and Screen Issues | srj321 | Sony Reader | 2 | 07-11-2010 11:52 PM |
Multiple databases using the same book files | JanuaryF | Calibre | 5 | 03-28-2010 11:40 AM |
Two calibre databases | jjansen | Calibre | 2 | 02-11-2010 12:52 PM |
Library Management | jerryleejr | Calibre | 5 | 07-11-2008 12:34 PM |