Management issues with large databases

Student1 · 02-19-2009, 04:31 PM

Notice there was a problem with caliber and HUGE dbs. Uploadaed a library of about 20 000 volumes and when i want to delete a few of them it goes into a non responsive mode. Tried just deleting one and it worked after 3 min, but for deleting something like 6000 volumes... well its been going at it for more than 40 min and shows as non responsive. Btw is caliber using any form of sql querries to manage the books? If its using a file to store the information might be the problem, and that might explain the issues with a large number of files.

anyway just wanted to point it out.

kovidgoyal · 02-19-2009, 04:36 PM

It stores metadata in an sqlite database. Deleting should be pretty quick, though I haven't really tested it with large databases.

Ah looking at the code, I see why it is slow (basically the GUI updates after each delete), should be fixed in the next release.

itimpi · 02-19-2009, 04:52 PM

Quote:

Originally Posted by kovidgoyal

Ah looking at the code, I see why it is slow (basically the GUI updates after each delete), should be fixed in the next release.

Could that also be true when adding multiple files?

Student1 · 02-19-2009, 04:55 PM

great thanks! That going to help alot!!!

Student1 · 02-19-2009, 04:58 PM

Quote:

Originally Posted by itimpi

Could that also be true when adding multiple files?

Thats a good point! I think it does at a certain level because i see added files as the percentage goes up! Removing the auto update after each file might help a lot in the speed department!

kovidgoyal · 02-19-2009, 04:58 PM

Quote:

Originally Posted by itimpi

Could that also be true when adding multiple files?

No adding multiple files happens in the database layer and only after the add is complete is the GUI informed. (apart from the progress dialog)

kovidgoyal · 02-19-2009, 05:00 PM

Quote:

Originally Posted by Student1

Thats a good point! I think it does at a certain level because i see added files as the percentage goes up! Removing the auto update after each file might help a lot in the speed department!

Updating the progress dialog is not an expensive operation, so removing that wont yield significant speedups

itimpi · 02-19-2009, 05:11 PM

I think that there are not probably not enough calls in the main add loop to allow other windows processes to run! I find that while the add is running, things like starting new copies in Explorer seem to hang until either the progress dialog updates (when there is a burst of activity and any hung copies now start) or the total add process finishes.

I suspect this may well be a Windows specific issue with event handling, and it may also depend on whether you have a multi-core machine which changes the timing issues relating to event handling. However I could be wrong - Windows multi-tasking can sometimes behave in very strange ways.

Student1 · 02-19-2009, 05:16 PM

Ok, was worth a try. Btw you really update this piece of software extremely fast! Great work!!

pilotbob · 02-19-2009, 05:24 PM

Quote:

Originally Posted by itimpi

I think that there are not probably not enough calls in the main add loop to allow other windows processes to run!

This is usually due to long running process on the UI thread. The UI will start reporting the app as non-responding. Also, if you put another window on top of it, then move it off, the window will show the echo of the other window.

Some languages have some type of Yield command to get around this. But, the "right" way to do it is to run the process on another thread. Although I think Kovid does spawn another process and communicate with IP, perhaps he is putting the UI thread in a wait state rather than using some type of call back.

BUt, I'm only guessing based on issues I have seen with windows apps.

BOb

kovidgoyal · 02-19-2009, 05:24 PM

This is the way recursive adding is structured:

1) there is a main GUI thread in which all operations that use the GUI toolkit must happen. This thread loops delivering events to the various GUI widgets
2) A progress dialog is opened in the main thread and a new thread is spawned to do the adding of books
3) The new thread first recursively finds all ebooks and creates a list of them
4) For each ebook, metadata is read. unfortunately because metadata reading sometimes requires GUI operations, it has to happen in the main GUI thread, this is the Reading metadata phase and is when the UI should be the least responsive
5) Once all metadata is read, the metadata + actual files are added to the library in the worker thread (technically, because sqlite in python is not thread safe actualt database modifications happen in yet another "database access" thread). This phase is the adding books to database phase

If all of windows is slowing down during any of these phases, it just means that Windows' kernel scheduler really sucks.

Note that you can avoid all this mess with threads by just using the command line

Code:

calibredb add --help

Since there far less context switching going on it should be significantly faster than adding via the GUI

pilotbob · 02-19-2009, 05:46 PM

Quote:

Originally Posted by kovidgoyal

If all of windows is slowing down during any of these phases, it just means that Windows' kernel scheduler really sucks.

Compared to OX x and Linux for the most part... it does suck. But, I think the Windows shell has alot of perf issues too.

BOb

itimpi · 02-19-2009, 06:13 PM

The strange thing is that it is not all of Windows that is slowing down. Apps that are already running are fine. It is just when I try to copy/paste files between Explorer Windows while Calibre is adding books the Explorer windows involved in the copy/paste seem to hang. All other windows (and even some with long-running copy operations) continue to run fine. Other Explorer windows are also fine as long as I do not attempt a copy/paste type operation.

I am quite prepared to believe that it is an idiosyncrasy of Windows Explorer - it was just strange that specific operations in one app (calibre) can have the effect they do on another one (Explorer). These are not things one should see in a proper multi-tasking OS. I am guessing that there is some mutext or something like that that the apps are fighting over.

pilotbob · 02-19-2009, 06:17 PM

Quote:

Originally Posted by itimpi

The strange thing is that it is not all of Windows that is slowing down. Apps that are already running are fine. It is just when I try to copy/paste files between Explorer Windows while Calibre is adding books the Explorer windows involved in the copy/paste seem to hang.

This probably has to do with your file system / disk speed.

Are you running a virus software? If so, exclude your calibre folder from on-access scan.

Also, I turn my recycle bin off... it makes a big difference when moving/deleting files in explorer. But, yes you can't recover files.

BOb

itimpi · 02-19-2009, 06:25 PM

It is something more obscure than that.

I have tried running procmon in parallel when these symptoms occur. It definitely looks as though something is blocking Windows Explorer.

I normally have Recycle bin disabled, and I have tested with Virus checking on/off and it makes no difference. The calibre folders have already been excluded explicitly in the Virus checker as well.

I can live with the issue easily enough, but I thought it was worth bringing up in case it sparked a thought in someone's mind. I may find the time to run a much more detailed analysis of all what is running under the covers and track all calls to the system but the amount of diagnostic information one has to wade through to get to the bottom of such issues is a none-trivial task.

02-19-2009, 04:31 PM	#1
Student1 Groupie Posts: 159 Karma: 170 Join Date: Feb 2009 Device: PRS-505	Management issues with large databases Notice there was a problem with caliber and HUGE dbs. Uploadaed a library of about 20 000 volumes and when i want to delete a few of them it goes into a non responsive mode. Tried just deleting one and it worked after 3 min, but for deleting something like 6000 volumes... well its been going at it for more than 40 min and shows as non responsive. Btw is caliber using any form of sql querries to manage the books? If its using a file to store the information might be the problem, and that might explain the issues with a large number of files. anyway just wanted to point it out. Last edited by Student1; 02-19-2009 at 04:36 PM.

02-19-2009, 05:24 PM	#11
kovidgoyal creator of calibre Posts: 43,852 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	This is the way recursive adding is structured: 1) there is a main GUI thread in which all operations that use the GUI toolkit must happen. This thread loops delivering events to the various GUI widgets 2) A progress dialog is opened in the main thread and a new thread is spawned to do the adding of books 3) The new thread first recursively finds all ebooks and creates a list of them 4) For each ebook, metadata is read. unfortunately because metadata reading sometimes requires GUI operations, it has to happen in the main GUI thread, this is the Reading metadata phase and is when the UI should be the least responsive 5) Once all metadata is read, the metadata + actual files are added to the library in the worker thread (technically, because sqlite in python is not thread safe actualt database modifications happen in yet another "database access" thread). This phase is the adding books to database phase If all of windows is slowing down during any of these phases, it just means that Windows' kernel scheduler really sucks. Note that you can avoid all this mess with threads by just using the command line Code: calibredb add --help Since there far less context switching going on it should be significantly faster than adding via the GUI

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Multiple Databases	danchisum	Calibre	1	07-29-2010 09:35 PM
Charging Issues and Screen Issues	srj321	Sony Reader	2	07-11-2010 11:52 PM
Multiple databases using the same book files	JanuaryF	Calibre	5	03-28-2010 11:40 AM
Two calibre databases	jjansen	Calibre	2	02-11-2010 12:52 PM
Library Management	jerryleejr	Calibre	5	07-11-2008 12:34 PM

02-19-2009, 04:36 PM	#2
kovidgoyal creator of calibre Posts: 43,852 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It stores metadata in an sqlite database. Deleting should be pretty quick, though I haven't really tested it with large databases. Ah looking at the code, I see why it is slow (basically the GUI updates after each delete), should be fixed in the next release.

02-19-2009, 04:55 PM	#4
Student1 Groupie Posts: 159 Karma: 170 Join Date: Feb 2009 Device: PRS-505	great thanks! That going to help alot!!!

02-19-2009, 05:11 PM	#8
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	I think that there are not probably not enough calls in the main add loop to allow other windows processes to run! I find that while the add is running, things like starting new copies in Explorer seem to hang until either the progress dialog updates (when there is a burst of activity and any hung copies now start) or the total add process finishes. I suspect this may well be a Windows specific issue with event handling, and it may also depend on whether you have a multi-core machine which changes the timing issues relating to event handling. However I could be wrong - Windows multi-tasking can sometimes behave in very strange ways.

02-19-2009, 05:16 PM	#9
Student1 Groupie Posts: 159 Karma: 170 Join Date: Feb 2009 Device: PRS-505	Ok, was worth a try. Btw you really update this piece of software extremely fast! Great work!!

02-19-2009, 06:13 PM	#13
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	The strange thing is that it is not all of Windows that is slowing down. Apps that are already running are fine. It is just when I try to copy/paste files between Explorer Windows while Calibre is adding books the Explorer windows involved in the copy/paste seem to hang. All other windows (and even some with long-running copy operations) continue to run fine. Other Explorer windows are also fine as long as I do not attempt a copy/paste type operation. I am quite prepared to believe that it is an idiosyncrasy of Windows Explorer - it was just strange that specific operations in one app (calibre) can have the effect they do on another one (Explorer). These are not things one should see in a proper multi-tasking OS. I am guessing that there is some mutext or something like that that the apps are fighting over.

02-19-2009, 06:25 PM	#15
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	It is something more obscure than that. I have tried running procmon in parallel when these symptoms occur. It definitely looks as though something is blocking Windows Explorer. I normally have Recycle bin disabled, and I have tested with Virus checking on/off and it makes no difference. The calibre folders have already been excluded explicitly in the Virus checker as well. I can live with the issue easily enough, but I thought it was worth bringing up in case it sparked a thought in someone's mind. I may find the time to run a much more detailed analysis of all what is running under the covers and track all calls to the system but the amount of diagnostic information one has to wade through to get to the bottom of such issues is a none-trivial task.

Advert

Advert