Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 10-30-2011, 09:14 AM   #1
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
Performance issue with large library

I have a calibre library with just under 60,000 items in it. When adding or editing entries now it is very slow to update the database file and then refresh the screen. I am assuming that the issue is I/O in searching/updating/editing the metatdata.db file which is right at 83 megs in size.

So my question is "is there any way to move the metatdata.db file to my ssd drive while leaving the calibre library in its current location on a larger sata drive."

I am running calibre on 64bit ubuntu 11.10 with the os running from a 100 gig pcie ssd drive. All bulk storage is pushed out to standard sata drives. If the issue is I/O performance then moving the database search and updating to the smaller but much much faster ssd should speed stuff up.

If that isn't the issue and someone has a suggestion then I am all ears also

thanks
audeojude is offline   Reply With Quote
Old 10-30-2011, 09:22 AM   #2
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by audeojude View Post
So my question is "is there any way to move the metatdata.db file to my ssd drive while leaving the calibre library in its current location on a larger sata drive."
I'm sure you can, using the calibre.bat file found in the resources directory and if I recall correctly it will help some.

Quote:
Originally Posted by audeojude View Post
If that isn't the issue and someone has a suggestion then I am all ears also
Also while doing any editing make sure to hide the tag browser. Hopefully others might have more advice on the subject.

Last edited by DoctorOhh; 10-30-2011 at 09:25 AM.
DoctorOhh is offline   Reply With Quote
Advert
Old 10-30-2011, 10:38 AM   #3
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
I have hid the tag browser and don't see a significant improvement and in my installation under linux I don't seem to be able to locate the calibre.bat file either in my home directory or in /opt/calibre where calibre is installed.
audeojude is offline   Reply With Quote
Old 10-30-2011, 10:39 AM   #4
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
So when I edit metadata what gets changed. Just entries in the database or also in the book directory's on the file system?
audeojude is offline   Reply With Quote
Old 10-30-2011, 10:49 AM   #5
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,691
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by audeojude View Post
So when I edit metadata what gets changed. Just entries in the database or also in the book directory's on the file system?
Could be both
any changes to Author or Title affect the file/folder names.
all the rest is saved into the DB and is used for conversion, send to device,save (export) to disk AND generating the exportable "Catalog".
theducks is offline   Reply With Quote
Advert
Old 10-30-2011, 10:50 AM   #6
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
I have found a few posts in the past referencing this issue and that with the calibre.bat file you can change the path of the database. However the files I can find on my linux installation don't seem to match that. What would I need to change on linux to point to a different location for the database?
audeojude is offline   Reply With Quote
Old 10-30-2011, 11:03 AM   #7
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
The attached file is the Calibre Portable.bat file that was developed for Windows. It is copiously commented, and should provide all the information required to create a linux shell script equivalent.
Attached Files
File Type: zip Calibre Portable.zip (2.2 KB, 140 views)
itimpi is offline   Reply With Quote
Old 10-30-2011, 11:03 AM   #8
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
Based on what I have read about other peoples performance issues I am not being hit to bad. If I'm adding 100 titles or editing tags on them it might take 30 to 60 seconds for calibre to be come responsive again after the files have copied.

I think that my hardware is doing good things for me already. However based on the performance gain related in some other posts the I could probably drop that lag by almost 70 to 80 % by moving the database to my ssd drive. The ssd drive has a throughput of about 400 megs a second while the sata drive is running about 50 megs a second. With an older intel quad core processor and 8 gigs of ram, I don't think there is anything on that side I can speed up. I think it is all I/0 issues.

Searching for books in the library now takes a long time also maybe 10 seconds before the search returns results and 20 seconds before the interface becomes responsive again.
audeojude is offline   Reply With Quote
Old 10-30-2011, 11:05 AM   #9
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by audeojude View Post
I have found a few posts in the past referencing this issue and that with the calibre.bat file you can change the path of the database. However the files I can find on my linux installation don't seem to match that. What would I need to change on linux to point to a different location for the database?
Here is the section of the manual that references the environment variables.

This section might help too.

Last edited by DoctorOhh; 10-30-2011 at 11:37 AM.
DoctorOhh is offline   Reply With Quote
Old 10-30-2011, 11:09 AM   #10
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
I'm not sure I know how to create a script to make this work. I was hoping someone could point me to the file that has this in it and I could just edit a line to make it work.
audeojude is offline   Reply With Quote
Old 10-30-2011, 11:12 AM   #11
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by audeojude View Post
I'm not sure I know how to create a script to make this work. I was hoping someone could point me to the file that has this in it and I could just edit a line to make it work.
Look at post 7 above.
DoctorOhh is offline   Reply With Quote
Old 10-30-2011, 11:35 AM   #12
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
ah.. ok i was posting to the one above that and missed your post.. cool. thanks
audeojude is offline   Reply With Quote
Old 10-30-2011, 12:06 PM   #13
wallcraft
reader
wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.wallcraft ought to be getting tired of karma fortunes by now.
 
wallcraft's Avatar
 
Posts: 6,975
Karma: 5183568
Join Date: Mar 2006
Location: Mississippi, USA
Device: Kindle 3, Kobo Glo HD
Quote:
Originally Posted by audeojude View Post
I'm not sure I know how to create a script to make this work. I was hoping someone could point me to the file that has this in it and I could just edit a line to make it work.
The simplest approach is to edit the file calibre (on my system this is /opt/calibre/calibre), which is the shell script that starts calibre. Add a line like the following:
Code:
export CALIBRE_OVERRIDE_DATABASE_PATH="/home/alan/Calibre Library"
Where you would replace the full path with your new location. I'm not sure if "/metadata.db" should be added at the end, this would not be usual for a "path" but might be needed here.
wallcraft is offline   Reply With Quote
Old 10-30-2011, 12:18 PM   #14
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
Ok I have found a calibre configuration directory under .config in my user directory that has a file called global.py in it that has paths for the database and library.

however the path that was in there does not point to my existing database file though the library path does. I copied moved the database to the root of my home directory and pointed the path to it. However that did not work. after starting calibre it generated a new database in the old location which is the root of the library folder where the books are.
audeojude is offline   Reply With Quote
Old 10-30-2011, 12:30 PM   #15
audeojude
Connoisseur
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
When I try to use the above export path in the /opt/calibre/calibre file
calibre starts and says that the database is corrupted. It isn't as I can remove that line and it will run fine with the database in the root of the library.
here is a copy of that file. I have tried the path both as /home/scott and /home/scott/

#!/bin/sh
path=`readlink -f $0`
base=`dirname $path`
lib=$base/lib
export LD_LIBRARY_PATH=$lib:$LD_LIBRARY_PATH
export CALIBRE_OVERRIDE_DATABASE_PATH="/home/scott"
export MAGICK_HOME=$base
export MAGICK_CONFIGURE_PATH=$lib/ImageMagick-6.6.7/config
export MAGICK_CODER_MODULE_PATH=$lib/ImageMagick-6.6.7/modules-Q16/coders
export MAGICK_CODER_FILTER_PATH=$lib/ImageMagick-6.6.7/modules-Q16/filters
$base/bin/calibre "$@"
audeojude is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Best way to get a large Calibre library into the PE library? Filark enTourage Archive 0 04-20-2011 10:18 PM
Networked db/large library a no-go? concern Calibre 3 02-03-2010 09:11 PM
DX performance on large (>100MB) pdf files? Blackguard Amazon Kindle 6 06-16-2009 06:57 PM
Library size and performance MrFinch Calibre 4 01-01-2009 04:11 AM
Performance with large Libraries? itimpi Calibre 3 12-14-2008 02:46 PM


All times are GMT -4. The time now is 02:08 AM.


MobileRead.com is a privately owned, operated and funded community.