Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 02-19-2009, 04:31 PM   #1
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
Management issues with large databases

Notice there was a problem with caliber and HUGE dbs. Uploadaed a library of about 20 000 volumes and when i want to delete a few of them it goes into a non responsive mode. Tried just deleting one and it worked after 3 min, but for deleting something like 6000 volumes... well its been going at it for more than 40 min and shows as non responsive. Btw is caliber using any form of sql querries to manage the books? If its using a file to store the information might be the problem, and that might explain the issues with a large number of files.


anyway just wanted to point it out.

Last edited by Student1; 02-19-2009 at 04:36 PM.
Student1 is offline   Reply With Quote
Old 02-19-2009, 04:36 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,248
Karma: 4961457
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
It stores metadata in an sqlite database. Deleting should be pretty quick, though I haven't really tested it with large databases.

Ah looking at the code, I see why it is slow (basically the GUI updates after each delete), should be fixed in the next release.
kovidgoyal is online now   Reply With Quote
Old 02-19-2009, 04:52 PM   #3
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,010
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by kovidgoyal View Post
Ah looking at the code, I see why it is slow (basically the GUI updates after each delete), should be fixed in the next release.
Could that also be true when adding multiple files?
itimpi is offline   Reply With Quote
Old 02-19-2009, 04:55 PM   #4
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
great thanks! That going to help alot!!!
Student1 is offline   Reply With Quote
Old 02-19-2009, 04:58 PM   #5
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
Quote:
Originally Posted by itimpi View Post
Could that also be true when adding multiple files?
Thats a good point! I think it does at a certain level because i see added files as the percentage goes up! Removing the auto update after each file might help a lot in the speed department!
Student1 is offline   Reply With Quote
Old 02-19-2009, 04:58 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,248
Karma: 4961457
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by itimpi View Post
Could that also be true when adding multiple files?
No adding multiple files happens in the database layer and only after the add is complete is the GUI informed. (apart from the progress dialog)
kovidgoyal is online now   Reply With Quote
Old 02-19-2009, 05:00 PM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,248
Karma: 4961457
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by Student1 View Post
Thats a good point! I think it does at a certain level because i see added files as the percentage goes up! Removing the auto update after each file might help a lot in the speed department!
Updating the progress dialog is not an expensive operation, so removing that wont yield significant speedups
kovidgoyal is online now   Reply With Quote
Old 02-19-2009, 05:11 PM   #8
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,010
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
I think that there are not probably not enough calls in the main add loop to allow other windows processes to run! I find that while the add is running, things like starting new copies in Explorer seem to hang until either the progress dialog updates (when there is a burst of activity and any hung copies now start) or the total add process finishes.

I suspect this may well be a Windows specific issue with event handling, and it may also depend on whether you have a multi-core machine which changes the timing issues relating to event handling. However I could be wrong - Windows multi-tasking can sometimes behave in very strange ways.
itimpi is offline   Reply With Quote
Old 02-19-2009, 05:16 PM   #9
Student1
Groupie
Student1 doesn't litterStudent1 doesn't litter
 
Posts: 159
Karma: 170
Join Date: Feb 2009
Device: PRS-505
Ok, was worth a try. Btw you really update this piece of software extremely fast! Great work!!
Student1 is offline   Reply With Quote
Old 02-19-2009, 05:24 PM   #10
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,468
Karma: 11248282
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by itimpi View Post
I think that there are not probably not enough calls in the main add loop to allow other windows processes to run!
This is usually due to long running process on the UI thread. The UI will start reporting the app as non-responding. Also, if you put another window on top of it, then move it off, the window will show the echo of the other window.

Some languages have some type of Yield command to get around this. But, the "right" way to do it is to run the process on another thread. Although I think Kovid does spawn another process and communicate with IP, perhaps he is putting the UI thread in a wait state rather than using some type of call back.

BUt, I'm only guessing based on issues I have seen with windows apps.

BOb

Last edited by pilotbob; 02-19-2009 at 05:43 PM.
pilotbob is offline   Reply With Quote
Old 02-19-2009, 05:24 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,248
Karma: 4961457
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
This is the way recursive adding is structured:

1) there is a main GUI thread in which all operations that use the GUI toolkit must happen. This thread loops delivering events to the various GUI widgets
2) A progress dialog is opened in the main thread and a new thread is spawned to do the adding of books
3) The new thread first recursively finds all ebooks and creates a list of them
4) For each ebook, metadata is read. unfortunately because metadata reading sometimes requires GUI operations, it has to happen in the main GUI thread, this is the Reading metadata phase and is when the UI should be the least responsive
5) Once all metadata is read, the metadata + actual files are added to the library in the worker thread (technically, because sqlite in python is not thread safe actualt database modifications happen in yet another "database access" thread). This phase is the adding books to database phase

If all of windows is slowing down during any of these phases, it just means that Windows' kernel scheduler really sucks.

Note that you can avoid all this mess with threads by just using the command line

Code:
calibredb add --help
Since there far less context switching going on it should be significantly faster than adding via the GUI
kovidgoyal is online now   Reply With Quote
Old 02-19-2009, 05:46 PM   #12
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,468
Karma: 11248282
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by kovidgoyal View Post
If all of windows is slowing down during any of these phases, it just means that Windows' kernel scheduler really sucks.
Compared to OX x and Linux for the most part... it does suck. But, I think the Windows shell has alot of perf issues too.

BOb
pilotbob is offline   Reply With Quote
Old 02-19-2009, 06:13 PM   #13
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,010
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
The strange thing is that it is not all of Windows that is slowing down. Apps that are already running are fine. It is just when I try to copy/paste files between Explorer Windows while Calibre is adding books the Explorer windows involved in the copy/paste seem to hang. All other windows (and even some with long-running copy operations) continue to run fine. Other Explorer windows are also fine as long as I do not attempt a copy/paste type operation.

I am quite prepared to believe that it is an idiosyncrasy of Windows Explorer - it was just strange that specific operations in one app (calibre) can have the effect they do on another one (Explorer). These are not things one should see in a proper multi-tasking OS. I am guessing that there is some mutext or something like that that the apps are fighting over.
itimpi is offline   Reply With Quote
Old 02-19-2009, 06:17 PM   #14
pilotbob
Grand Sorcerer
pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.pilotbob ought to be getting tired of karma fortunes by now.
 
pilotbob's Avatar
 
Posts: 19,468
Karma: 11248282
Join Date: Jan 2007
Location: Tampa, FL USA
Device: Kindle Touch
Quote:
Originally Posted by itimpi View Post
The strange thing is that it is not all of Windows that is slowing down. Apps that are already running are fine. It is just when I try to copy/paste files between Explorer Windows while Calibre is adding books the Explorer windows involved in the copy/paste seem to hang.
This probably has to do with your file system / disk speed.

Are you running a virus software? If so, exclude your calibre folder from on-access scan.

Also, I turn my recycle bin off... it makes a big difference when moving/deleting files in explorer. But, yes you can't recover files.

BOb
pilotbob is offline   Reply With Quote
Old 02-19-2009, 06:25 PM   #15
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,010
Karma: 777817
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
It is something more obscure than that.

I have tried running procmon in parallel when these symptoms occur. It definitely looks as though something is blocking Windows Explorer.

I normally have Recycle bin disabled, and I have tested with Virus checking on/off and it makes no difference. The calibre folders have already been excluded explicitly in the Virus checker as well.

I can live with the issue easily enough, but I thought it was worth bringing up in case it sparked a thought in someone's mind. I may find the time to run a much more detailed analysis of all what is running under the covers and track all calls to the system but the amount of diagnostic information one has to wade through to get to the bottom of such issues is a none-trivial task.
itimpi is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Multiple Databases danchisum Calibre 1 07-29-2010 09:35 PM
Charging Issues and Screen Issues srj321 Sony Reader 2 07-11-2010 11:52 PM
Multiple databases using the same book files JanuaryF Calibre 5 03-28-2010 11:40 AM
Two calibre databases jjansen Calibre 2 02-11-2010 12:52 PM
Library Management jerryleejr Calibre 5 07-11-2008 12:34 PM


All times are GMT -4. The time now is 06:56 AM.


MobileRead.com is a privately owned, operated and funded community.