Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 09-01-2012, 02:48 PM   #1
jm2
Junior Member
jm2 began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Sep 2012
Device: iPhone, PRS-350
Store metadata for multiple libraries using Override Database Path EV

I have multiple calibre libraries stored on a network share. To improve performance on Windows 7, I want to use the CALIBRE_OVERRIDE_DATABASE_PATH environment variable to store the metadata.db files for each remote library in a local dir.

In 2009, itimpi showed how to use this setting for a single library. I'm wondering if there's a way to modify itimpi's batch file to provide different metadata.db paths for each library. I'm not sure how the script works, so I don't know where to start. Also, I should note that I use the portable version of calibre - would this work with the portable version?
jm2 is offline   Reply With Quote
Old 09-01-2012, 05:47 PM   #2
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
One way of doing this might be by using symbolic links. I just did some experimenting on my computer. Seems to work fine. I might even use this for real...

I use Linux, but as I understand it symbolic links are available in Windows as well.

http://en.wikipedia.org/wiki/Symbolic_link

I mount / on a small but fast SSD and /home on very big slow mechanical disc.

I have my main Calibre Library in a folder on the slow disc:
/home/adoby/Calibre/main

The metadata is, as usual, stored in the file:
/home/adoby/Calibre/main/metadata.db

I have prepared a folder on the SSD, with the correct rights:
/opt/SpeedUp/adoby/Calibre/main

My goal is to have the metadata.db there, to make use of the much faster SSD, but to keep the books on the big slow mechanical disc.

I do it like this:

(I recommend that you make a good backup before you try this...)

cp /home/adoby/Calibre/main/metadata.db /opt/SpeedUp/adoby/Calibre/main
mv /home/adoby/Calibre/main/metadata.db /home/adoby/Calibre/main/metadata.db.bak
ln -s /opt/SpeedUp/adoby/Calibre/main/metadata.db /home/adoby/Calibre/main/metadata.db

Done!

The first line copies metadata.db to the SSD.
The second line moves the original metadata.db away, by adding .bak to the name.
The third line creates the symboic link.

So now I have metadata.db on the SSD, but as far as Calibre is concerned, it is still at the usual place. But since it really is a symbolic link, the faster metadata.db on SSD will be used.

I did some testing and everything seems to work fine. I am not sure about how much faster things are, but the GUI seems a little snappier. Could be my imagination...

I assume that this is rather unsafe. I am not sure what happens if you rename or move a library. Or if you use portable Calibre. Perhaps you will have good use for that backup...

If you have more Calibre Libraries, just repeat the process, and move each metadata.db to SSD and do a symlink back to the original place.

Last edited by Adoby; 09-01-2012 at 05:54 PM.
Adoby is offline   Reply With Quote
Old 09-01-2012, 06:12 PM   #3
jm2
Junior Member
jm2 began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Sep 2012
Device: iPhone, PRS-350
Thanks for the detailed instructions! It turns out that only one of my libraries runs very slowly over the network. It's huge, with a metadata.db file that's over 12MB. I may try your method for this library, but I'm a little hesitant because I'm unfamiliar with symlinks. The only files I should need to backup are the metadata.db files, right?
jm2 is offline   Reply With Quote
Old 09-01-2012, 06:19 PM   #4
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Yes, most likely, but no guaranties. You really should have a full backup of all files. You never know.

Try to create a small "fake" library first, to make sure everything works, before you try with files you care about...

I assume that usually the whole metadata.db will automatically be cached in RAM. So the gains may not be very big when you just browse and read. When you update metadata, the speedup may be greater.

Plenty of memory in the computer probably is the best and safest way to speed up Calibre.

Last edited by Adoby; 09-02-2012 at 02:39 AM.
Adoby is offline   Reply With Quote
Old 09-01-2012, 11:31 PM   #5
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@jm2

Couple of useful shell extensions for doing this sort of thing

Link Shell Extension can do it via drag drop. LSE even implements Symlinks on XP's, they are not 'reachable' via Windows utilities, Shinagl uses a 3rd party library. Don't be put off by LSE's apparent complexity, most of it is hidden by default, to get the esoteric features you have to enable them via preferences.

FileTools can do it via navigating with a standard Select Folder dialogue

I use both, they are free to use and free from drama.

There's another gadget called Symlinker, which is a stand alone program that's sort of OK for one off tasks, like yours. I don't use it, but it seems to work, although I find it very confusing - IMO its UI has things in the wrong order - and it seems to have had hardlinks and junctions grafted into it in a half hearted way. It only works on Vista & W7.

The attached screen shots show LSE in action,

I moved a metadata.db from a Calibre Library on drive D: to a folder on Drive C: I then did a right click drag/drop of the metadata.db from the Drive C: folder back to the Calibre Library folder on drive D: and got the context menu you see in the first screen shot.

After selecting Create Symbolic Link Here I get what you see in the second screen shot. Note the icon for metadata.db in Calibre Quick has a green arrow icon overlay similar to a shortcut, which is my visual cue that its a symlink, and that its size is zero and the Link Target is the metadata.db on Drive C:

Calibre had no problem using a symlink metadata.db on Windows 7

I use the xplorer2 File Mangler - couldn't live without it nor LSE.

BR
Attached Thumbnails
Click image for larger version

Name:	Screenshot 1.jpg
Views:	400
Size:	134.6 KB
ID:	91676   Click image for larger version

Name:	Capture2.JPG
Views:	376
Size:	138.8 KB
ID:	91678  

Last edited by BetterRed; 09-01-2012 at 11:34 PM. Reason: error
BetterRed is offline   Reply With Quote
Old 09-02-2012, 02:14 AM   #6
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Good to know. The Nautilus file manager in Gnome also let you create the symlink using the GUI. So this can easily be done in Linux, as well, without actually using the command line.

I assume that usually the whole metadata.db will automatically be cached in RAM. So the gains may not be very big when you just browse and read. When you update metadata, the speedup may be greater.

Plenty of memory in the computer probably is the best and safest way to speed up Calibre.

Last edited by Adoby; 09-02-2012 at 02:40 AM.
Adoby is offline   Reply With Quote
Old 09-02-2012, 02:26 AM   #7
jm2
Junior Member
jm2 began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Sep 2012
Device: iPhone, PRS-350
Yes, thank you very much for that post.

Just to make sure I understand - if I use symlinker, for each remote library, I should:
  1. Backup the metadata.db file.
  2. Move the file from the remote location to a local dir.
  3. Choose the remote calibre library dir as "the place where I want my link" in symlinker.
  4. Choose the local copy of the metadata.db file as "the real folder I want to link."
It turns out the answer to my original question is no. The CALIBRE_OVERRIDE_DATABASE_PATH environment variable cannot be used with more than one library:

Quote:
Using the environment variable CALIBRE_OVERRIDE_DATABASE_PATH disables multiple-library support in calibre. Avoid setting this variable in calibre-portable.bat unless you really need it.
So I'm glad the symlink method works!

Alternatively, would using mklink in an elevated command prompt as follows also work? I've listed the remote dir (where the link will go) first and the metadata file path second:

Quote:
mklink "X:\Books\Calibre Library 1" "C:\Calibre Portable\Metadata\Calibre Library 1\Metadata.db"

Last edited by jm2; 09-02-2012 at 02:34 AM.
jm2 is offline   Reply With Quote
Old 09-02-2012, 02:45 AM   #8
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Yes. Move the metadata.db and create a symlink where it originally was, to the new location. Repeat for each Calibre Library.

I believe you have to specify both path and filename in both link and target. But I haven't tested it.

Code:
mklink "X:\Books\Calibre Library 1\Metadata.db" "C:\Calibre Portable\Metadata\Calibre Library 1\Metadata.db"

Last edited by Adoby; 09-02-2012 at 03:32 AM.
Adoby is offline   Reply With Quote
Old 09-02-2012, 03:44 AM   #9
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by Adoby View Post
Good to know. The Nautilus file manager in Gnome also let you create the symlink using the GUI. So this can easily be done in Linux, as well, without actually using the command line.
If M$ did something like that then Manuelo Barrowboy, Chairman of the EUSSR Presidium would get Joaquín Almunia to frog march M$ off to the Brussels Lubyanka and conduct yet another 15 year long witch hunt at taxpayers expense the only beneficiaries of which are... the legal profession.

And then Spain can have Dictador Almunia, just like Italy has Dittatore Mario Monti.

<br>

PS If you're putting the metadata.db on a RAM drive you probably ought to have a UPS.

Last edited by BetterRed; 09-02-2012 at 03:46 AM.
BetterRed is offline   Reply With Quote
Old 09-02-2012, 04:21 AM   #10
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Quote:
Originally Posted by BetterRed View Post
PS If you're putting the metadata.db on a RAM drive you probably ought to have a UPS.
I've already tested it.

But quickly put it back on the normal disc.

As I already said above, I don't think that the speedup is all that great, if you have plenty of memory for the disc cache... I haven't done any benchmarks though...
Adoby is offline   Reply With Quote
Old 09-02-2012, 04:58 AM   #11
jm2
Junior Member
jm2 began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Sep 2012
Device: iPhone, PRS-350
Adoby, you may be right that the speedup gains are limited for just browsing remote libraries. Only one of my libraries browses very slowly, and it may get better after a while - I haven't tried it for long. But with a remote library and metadata file, I think adding many books at the same time takes much longer than it would with a local library and metadata file.
jm2 is offline   Reply With Quote
Old 09-02-2012, 10:12 AM   #12
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,689
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by BetterRed View Post
If M$ did something like that then Manuelo Barrowboy, Chairman of the EUSSR Presidium would get Joaquín Almunia to frog march M$ off to the Brussels Lubyanka and conduct yet another 15 year long witch hunt at taxpayers expense the only beneficiaries of which are... the legal profession.

And then Spain can have Dictador Almunia, just like Italy has Dittatore Mario Monti.

<br>

PS If you're putting the metadata.db on a RAM drive you probably ought to have a UPS.
Not Probably! MUST

If you use remote storage:
don't forget to include all the (powered) network devices between the PC and the storage.

You did have the NAS on the UPS?
theducks is offline   Reply With Quote
Old 09-02-2012, 08:01 PM   #13
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by theducks View Post
Not Probably! MUST
Aaah, you Merry Cans, you maybe masters of the universe, but you never did master the understatement

¿ If you don't have a UPS, then you can probably rebuild from the opf's = yes/no? If 'yes' then I stand by 'probably'.

<br>
BetterRed is offline   Reply With Quote
Old 09-03-2012, 02:29 AM   #14
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
If you keep the entire Calibre library in a RAM drive, then the risk for a corrupt database or damaged books should be minimal, even without a UPS.
Adoby is offline   Reply With Quote
Old 09-03-2012, 03:48 AM   #15
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by Adoby View Post
If you keep the entire Calibre library in a RAM drive, then the risk for a corrupt database or damaged books should be minimal, even without a UPS.
I assume you mean because they wont even be there after a power down/fail

RAM Drives & Flash /Solid State Drives are not the same thing, the former loses everything when it loses power the latter two don't.

Or did Apple just patent RAM to mean something other than what it's been for 40+ years - watch out IBM, the Apple Patent Factory is going to sue you for having the temerity to invent RAM.

br
BetterRed is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How do I override default metadata search terms? DC_Sonoma Library Management 2 08-10-2012 09:40 AM
Adding books: getting metadata from path? lbolton Library Management 2 08-05-2012 09:50 PM
Can I have multiple libraries? mrgiedrnkr Calibre 59 08-14-2010 02:09 PM
Multiple libraries? netgirl Calibre 14 11-03-2009 10:13 AM
Rebuild Database? & Use Path instead of Filename? gai-jin Calibre 0 07-20-2009 02:58 PM


All times are GMT -4. The time now is 02:22 AM.


MobileRead.com is a privately owned, operated and funded community.