|
|
Thread Tools | Search this Thread |
09-01-2012, 02:48 PM | #1 |
Junior Member
Posts: 4
Karma: 10
Join Date: Sep 2012
Device: iPhone, PRS-350
|
Store metadata for multiple libraries using Override Database Path EV
I have multiple calibre libraries stored on a network share. To improve performance on Windows 7, I want to use the CALIBRE_OVERRIDE_DATABASE_PATH environment variable to store the metadata.db files for each remote library in a local dir.
In 2009, itimpi showed how to use this setting for a single library. I'm wondering if there's a way to modify itimpi's batch file to provide different metadata.db paths for each library. I'm not sure how the script works, so I don't know where to start. Also, I should note that I use the portable version of calibre - would this work with the portable version? |
09-01-2012, 05:47 PM | #2 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
One way of doing this might be by using symbolic links. I just did some experimenting on my computer. Seems to work fine. I might even use this for real...
I use Linux, but as I understand it symbolic links are available in Windows as well. http://en.wikipedia.org/wiki/Symbolic_link I mount / on a small but fast SSD and /home on very big slow mechanical disc. I have my main Calibre Library in a folder on the slow disc: /home/adoby/Calibre/main The metadata is, as usual, stored in the file: /home/adoby/Calibre/main/metadata.db I have prepared a folder on the SSD, with the correct rights: /opt/SpeedUp/adoby/Calibre/main My goal is to have the metadata.db there, to make use of the much faster SSD, but to keep the books on the big slow mechanical disc. I do it like this: (I recommend that you make a good backup before you try this...) cp /home/adoby/Calibre/main/metadata.db /opt/SpeedUp/adoby/Calibre/main mv /home/adoby/Calibre/main/metadata.db /home/adoby/Calibre/main/metadata.db.bak ln -s /opt/SpeedUp/adoby/Calibre/main/metadata.db /home/adoby/Calibre/main/metadata.db Done! The first line copies metadata.db to the SSD. The second line moves the original metadata.db away, by adding .bak to the name. The third line creates the symboic link. So now I have metadata.db on the SSD, but as far as Calibre is concerned, it is still at the usual place. But since it really is a symbolic link, the faster metadata.db on SSD will be used. I did some testing and everything seems to work fine. I am not sure about how much faster things are, but the GUI seems a little snappier. Could be my imagination... I assume that this is rather unsafe. I am not sure what happens if you rename or move a library. Or if you use portable Calibre. Perhaps you will have good use for that backup... If you have more Calibre Libraries, just repeat the process, and move each metadata.db to SSD and do a symlink back to the original place. Last edited by Adoby; 09-01-2012 at 05:54 PM. |
09-01-2012, 06:12 PM | #3 |
Junior Member
Posts: 4
Karma: 10
Join Date: Sep 2012
Device: iPhone, PRS-350
|
Thanks for the detailed instructions! It turns out that only one of my libraries runs very slowly over the network. It's huge, with a metadata.db file that's over 12MB. I may try your method for this library, but I'm a little hesitant because I'm unfamiliar with symlinks. The only files I should need to backup are the metadata.db files, right?
|
09-01-2012, 06:19 PM | #4 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Yes, most likely, but no guaranties. You really should have a full backup of all files. You never know.
Try to create a small "fake" library first, to make sure everything works, before you try with files you care about... I assume that usually the whole metadata.db will automatically be cached in RAM. So the gains may not be very big when you just browse and read. When you update metadata, the speedup may be greater. Plenty of memory in the computer probably is the best and safest way to speed up Calibre. Last edited by Adoby; 09-02-2012 at 02:39 AM. |
09-01-2012, 11:31 PM | #5 |
null operator (he/him)
Posts: 20,935
Karma: 27620688
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
@jm2
Couple of useful shell extensions for doing this sort of thing Link Shell Extension can do it via drag drop. LSE even implements Symlinks on XP's, they are not 'reachable' via Windows utilities, Shinagl uses a 3rd party library. Don't be put off by LSE's apparent complexity, most of it is hidden by default, to get the esoteric features you have to enable them via preferences. FileTools can do it via navigating with a standard Select Folder dialogue I use both, they are free to use and free from drama. There's another gadget called Symlinker, which is a stand alone program that's sort of OK for one off tasks, like yours. I don't use it, but it seems to work, although I find it very confusing - IMO its UI has things in the wrong order - and it seems to have had hardlinks and junctions grafted into it in a half hearted way. It only works on Vista & W7. The attached screen shots show LSE in action, I moved a metadata.db from a Calibre Library on drive D: to a folder on Drive C: I then did a right click drag/drop of the metadata.db from the Drive C: folder back to the Calibre Library folder on drive D: and got the context menu you see in the first screen shot. After selecting Create Symbolic Link Here I get what you see in the second screen shot. Note the icon for metadata.db in Calibre Quick has a green arrow icon overlay similar to a shortcut, which is my visual cue that its a symlink, and that its size is zero and the Link Target is the metadata.db on Drive C: Calibre had no problem using a symlink metadata.db on Windows 7 I use the xplorer2 File Mangler - couldn't live without it nor LSE. BR Last edited by BetterRed; 09-01-2012 at 11:34 PM. Reason: error |
09-02-2012, 02:14 AM | #6 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Good to know. The Nautilus file manager in Gnome also let you create the symlink using the GUI. So this can easily be done in Linux, as well, without actually using the command line.
I assume that usually the whole metadata.db will automatically be cached in RAM. So the gains may not be very big when you just browse and read. When you update metadata, the speedup may be greater. Plenty of memory in the computer probably is the best and safest way to speed up Calibre. Last edited by Adoby; 09-02-2012 at 02:40 AM. |
09-02-2012, 02:26 AM | #7 | ||
Junior Member
Posts: 4
Karma: 10
Join Date: Sep 2012
Device: iPhone, PRS-350
|
Yes, thank you very much for that post.
Just to make sure I understand - if I use symlinker, for each remote library, I should:
Quote:
Alternatively, would using mklink in an elevated command prompt as follows also work? I've listed the remote dir (where the link will go) first and the metadata file path second: Quote:
Last edited by jm2; 09-02-2012 at 02:34 AM. |
||
09-02-2012, 02:45 AM | #8 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Yes. Move the metadata.db and create a symlink where it originally was, to the new location. Repeat for each Calibre Library.
I believe you have to specify both path and filename in both link and target. But I haven't tested it. Code:
mklink "X:\Books\Calibre Library 1\Metadata.db" "C:\Calibre Portable\Metadata\Calibre Library 1\Metadata.db" Last edited by Adoby; 09-02-2012 at 03:32 AM. |
09-02-2012, 03:44 AM | #9 | |
null operator (he/him)
Posts: 20,935
Karma: 27620688
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Quote:
And then Spain can have Dictador Almunia, just like Italy has Dittatore Mario Monti. <br> PS If you're putting the metadata.db on a RAM drive you probably ought to have a UPS. Last edited by BetterRed; 09-02-2012 at 03:46 AM. |
|
09-02-2012, 04:21 AM | #10 | |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Quote:
But quickly put it back on the normal disc. As I already said above, I don't think that the speedup is all that great, if you have plenty of memory for the disc cache... I haven't done any benchmarks though... |
|
09-02-2012, 04:58 AM | #11 |
Junior Member
Posts: 4
Karma: 10
Join Date: Sep 2012
Device: iPhone, PRS-350
|
Adoby, you may be right that the speedup gains are limited for just browsing remote libraries. Only one of my libraries browses very slowly, and it may get better after a while - I haven't tried it for long. But with a remote library and metadata file, I think adding many books at the same time takes much longer than it would with a local library and metadata file.
|
09-02-2012, 10:12 AM | #12 | |
Well trained by Cats
Posts: 30,376
Karma: 58053698
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
Quote:
If you use remote storage: don't forget to include all the (powered) network devices between the PC and the storage. You did have the NAS on the UPS? |
|
09-02-2012, 08:01 PM | #13 |
null operator (he/him)
Posts: 20,935
Karma: 27620688
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
|
09-03-2012, 02:29 AM | #14 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
If you keep the entire Calibre library in a RAM drive, then the risk for a corrupt database or damaged books should be minimal, even without a UPS.
|
09-03-2012, 03:48 AM | #15 | |
null operator (he/him)
Posts: 20,935
Karma: 27620688
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Quote:
RAM Drives & Flash /Solid State Drives are not the same thing, the former loses everything when it loses power the latter two don't. Or did Apple just patent RAM to mean something other than what it's been for 40+ years - watch out IBM, the Apple Patent Factory is going to sue you for having the temerity to invent RAM. br |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
How do I override default metadata search terms? | DC_Sonoma | Library Management | 2 | 08-10-2012 09:40 AM |
Adding books: getting metadata from path? | lbolton | Library Management | 2 | 08-05-2012 09:50 PM |
Can I have multiple libraries? | mrgiedrnkr | Calibre | 59 | 08-14-2010 02:09 PM |
Multiple libraries? | netgirl | Calibre | 14 | 11-03-2009 10:13 AM |
Rebuild Database? & Use Path instead of Filename? | gai-jin | Calibre | 0 | 07-20-2009 02:58 PM |