How to relocate the full-text index, while keeping calibre library unmoved?

droopy · 06-23-2023, 05:52 PM

About two months ago, I discovered full-text search capability in Calibre. This is such a wonderful feature.

My Linux laptop has a 1 TB internal SSD. I have an external (USB plug in) 2 TB HDD. My calibre library resides on the 2TB HDD, and, as I just discovered, so does the full-text index/database.
Is there a way to relocate the full-text index/database to my SSD?

Seeing the search results for the titles is fast, even on the HDD, but it’s much slower to find example sentences where the keyword(s) in the books are found. So I think having the full-text index on the SSD will speed things up.

ownedbycats · 06-23-2023, 09:36 PM

Is there anything similar to a symlink/NTFS hardlink for Linux filesystems? That's what I would try.

BetterRed · 06-23-2023, 09:43 PM

You could try the CALIBRE_OVERRIDE_DATABASE_PATH Environment variable You'll have to move the metadata.db file too. Untested.

Or you could do it with a symlink; i.e. move the full-text-search.db file to your SSD, and create a symlink (named 'full-text-search.db') to it in your library folder. Tested on Windows - but it did not improve performance - still significantly slower than Windows and X1 with their search indices on a spinning rust drive.

BR

Comfy.n · 06-26-2023, 01:39 AM

Quote:

Originally Posted by BetterRed

X1 with their search indices on a spinning rust drive.

BR

https://www.voidtools.com/forum/view...hp?f=12&t=9787

did you try EV alpha? Been using it for a while now, it's amazing

Dsharma · 06-26-2023, 03:48 AM

@BetterRed has a great point about trying the CALIBRE_OVERRIDE_DATABASE_PATH environment variable or symlink method. Do keep in mind though, that it might not yield the speed improvement you're hoping for, as BR tested it on Windows with no noticeable difference.

If you're comfortable with it, give it a shot, and remember to backup everything before proceeding. It's always a good idea to have a fallback plan.

Also, I second @Comfy.n's suggestion. The EV alpha tool seems promising for performance improvement, if you haven't given that a shot yet.

BetterRed · 06-26-2023, 04:06 AM

Quote:

Originally Posted by Comfy.n

https://www.voidtools.com/forum/view...hp?f=12&t=9787

did you try EV alpha? Been using it for a while now, it's amazing

Can I assume you've used X1.

BR

Comfy.n · 06-26-2023, 04:20 AM

Quote:

Originally Posted by BetterRed

Can I assume you've used X1.

BR

I've seen you mention it before, but never gave it a try, actually.

WhereIsIt used to be my choice for catalogues, until I found this new Ev
I'll keep in mind to try X1!

BetterRed · 06-26-2023, 04:44 AM

Quote:

Originally Posted by Comfy.n

I've seen you mention it before, but never gave it a try, actually.

WhereIsIt used to be my choice for catalogues, until I found this new Ev
I'll keep in mind to try X1!

X1 targeted at corporations, $80/per annum/per seat.

Last time I looked the demo was rather limited in time, scope and size. A former client (major law firm) gifted me a lifetime subscription.

BR

kovidgoyal · 06-26-2023, 06:14 AM

That's a limitation of how sqlite does the sentence finding, it has to do a full scan of the text to find the sentences.

droopy · 07-31-2023, 06:55 PM

Quote:

Originally Posted by BetterRed

You could try the CALIBRE_OVERRIDE_DATABASE_PATH Environment variable You'll have to move the metadata.db file too. Untested.

Or you could do it with a symlink; i.e. move the full-text-search.db file to your SSD, and create a symlink (named 'full-text-search.db') to it in your library folder. Tested on Windows - but it did not improve performance - still significantly slower than Windows and X1 with their search indices on a spinning rust drive.

BR

Quote:

Originally Posted by Comfy.n

https://www.voidtools.com/forum/view...hp?f=12&t=9787

did you try EV alpha? Been using it for a while now, it's amazing

Quote:

Originally Posted by Dsharma

@BetterRed has a great point about trying the CALIBRE_OVERRIDE_DATABASE_PATH environment variable or symlink method. Do keep in mind though, that it might not yield the speed improvement you're hoping for, as BR tested it on Windows with no noticeable difference.

If you're comfortable with it, give it a shot, and remember to backup everything before proceeding. It's always a good idea to have a fallback plan.

Also, I second @Comfy.n's suggestion. The EV alpha tool seems promising for performance improvement, if you haven't given that a shot yet.

Dear all,
Thank you for your responses.

Dear BetterRed,
I'm on Linux. Do you think I'll face a similar fate as you if I try the symlink suggestion?
Can I ask why you haven't tried you CALIBRE_OVERRIDE_DATABASE_PATH variable suggestion?

Dear Comfy.n,
I'm on Linux. What's EV Alpha and will it help someone on Linux? I checked your link, and I see exe files. I've concluded it's for Windows.

Dear Dsharma,
Betterred tried the symlink method but not CALIBRE_OVERRIDE_DATABASE_PATH suggestion.

droopy · 07-31-2023, 06:56 PM

Quote:

Originally Posted by kovidgoyal

That's a limitation of how sqlite does the sentence finding, it has to do a full scan of the text to find the sentences.

dear Kovid,
thank you for your response.
Question 1: Were you talking to me? Or some other poster here?
Question 2: I've read and read what you wrote, but I'm not sure what the implication of your reply is. Could you please tell me what my takeaway should be?

BetterRed · 07-31-2023, 10:59 PM

@droopy - I only keep a calibre FTS database on my Test library, so my comments mainly relate to using symlinks to calibre library metadata databases.

Quote:

Originally Posted by droopy

I'm on Linux. Do you think I'll face a similar fate as you if I try the symlink suggestion?

Possibly, calibre loads the database into RAM at startup - that's why metadata searches, sorts etc are fast. On Windows/NTFS using a symlink to a database file on a Toshiba M1 NVMe SSD didn't improve performance over the SATA II WD Black HDD where my libraries are stored. If you have a faster SSD and slower HDD I suppose it might make a difference - suck it and see.

Quote:

Originally Posted by droopy

Can I ask why you haven't tried you CALIBRE_OVERRIDE_DATABASE_PATH variable suggestion?

I have multiple libraries (because I have disparate custom columns), however the env-var approach assumes a single library. Besides I see no reason why it would perform any better or worse than a symlink that references an SSD resident database.

BR

Comfy.n · 08-01-2023, 01:17 AM

Quote:

Originally Posted by droopy

Dear Comfy.n,
I'm on Linux. What's EV Alpha and will it help someone on Linux? I checked your link, and I see exe files. I've concluded it's for Windows.

Hi @droopy,

I mentioned Voidtools Everything (hence 'VE' instead of EV, how I like to call it...), because we were talking about search speed, in general. It's Windows-only, indeed.

FTS-wise, I don't see much difference in speed between my main Calibre install on a SSD, and a portable instance I have on an external HDD.

One way I know to obtain faster FTS results, on a large library, is to search a restricted subset of books. For instance, type on main calibre search box tags:fiction, then open the FTS dialogue and check "Restrict searched books" before running a query.

kovidgoyal · 08-01-2023, 01:19 AM

Quote:

Originally Posted by droopy

dear Kovid,
thank you for your response.
Question 1: Were you talking to me? Or some other poster here?
Question 2: I've read and read what you wrote, but I'm not sure what the implication of your reply is. Could you please tell me what my takeaway should be?

the implication is no you cannot speed up the sentence finding part of it significantly.

06-23-2023, 05:52 PM	#1
droopy Guru Posts: 833 Karma: 2912460 Join Date: Apr 2009 Device: Kobo Forma	How to relocate the full-text index, while keeping calibre library unmoved? About two months ago, I discovered full-text search capability in Calibre. This is such a wonderful feature. My Linux laptop has a 1 TB internal SSD. I have an external (USB plug in) 2 TB HDD. My calibre library resides on the 2TB HDD, and, as I just discovered, so does the full-text index/database. Is there a way to relocate the full-text index/database to my SSD? Seeing the search results for the titles is fast, even on the HDD, but it’s much slower to find example sentences where the keyword(s) in the books are found. So I think having the full-text index on the SSD will speed things up. Last edited by droopy; 07-31-2023 at 06:57 PM.

06-23-2023, 09:43 PM	#3
BetterRed null operator (he/him) Posts: 20,696 Karma: 26966376 Join Date: Mar 2012 Location: Sydney Australia Device: none	You could try the CALIBRE_OVERRIDE_DATABASE_PATH Environment variable You'll have to move the metadata.db file too. Untested. Or you could do it with a symlink; i.e. move the full-text-search.db file to your SSD, and create a symlink (named 'full-text-search.db') to it in your library folder. Tested on Windows - but it did not improve performance - still significantly slower than Windows and X1 with their search indices on a spinning rust drive. BR Last edited by BetterRed; 06-23-2023 at 10:19 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Option for Calibre 6 Full Text Search Index file location?	WHY	Calibre	20	09-13-2022 05:29 PM
Disable/Remove Full Text Index	amazing2061	Library Management	2	08-31-2022 01:48 PM
Full text search for partial library	Rob_W	Library Management	3	08-03-2022 05:01 AM
Relocate calibre config folder	Mervian	Calibre	3	09-03-2015 02:00 PM
Full text search in Ebook library	loguk	Library Management	1	08-13-2015 09:44 AM

06-23-2023, 09:36 PM	#2
ownedbycats Custom User Title Posts: 8,861 Karma: 62040409 Join Date: Oct 2018 Location: Canada Device: Kobo Libra H2O, formerly Aura HD	Is there anything similar to a symlink/NTFS hardlink for Linux filesystems? That's what I would try.

06-26-2023, 03:48 AM	#5
Dsharma Enthusiast Posts: 48 Karma: 3254262 Join Date: Apr 2023 Device: Kindle	@BetterRed has a great point about trying the CALIBRE_OVERRIDE_DATABASE_PATH environment variable or symlink method. Do keep in mind though, that it might not yield the speed improvement you're hoping for, as BR tested it on Windows with no noticeable difference. If you're comfortable with it, give it a shot, and remember to backup everything before proceeding. It's always a good idea to have a fallback plan. Also, I second @Comfy.n's suggestion. The EV alpha tool seems promising for performance improvement, if you haven't given that a shot yet.

06-26-2023, 06:14 AM	#9
kovidgoyal creator of calibre Posts: 44,030 Karma: 22669822 Join Date: Oct 2006 Location: Mumbai, India Device: Various	That's a limitation of how sqlite does the sentence finding, it has to do a full scan of the text to find the sentences.

Advert

Advert