indexing file size

[DrChiper]

Hi Kovid, thanks for the including of the full text search. Much appreciated.

I have a question tough about the index file size (full-text-search.db): it is about the same size as the complete library size itself, a ratio of 1:1 so to speak. Up to now I used the program docfetcher, which manages a ratio of 5:1.
Meaning my main library of about 19GiB has a full-text-search.db of ~19GiB, while docfetcher manages to squeeze their index into ~3,5GiB.

That's something to be aware of due to needed and available storage sizes (and according backups).

Is there any change that calibre's index file size is likely to become smaller due to possible future tweaking possibilities?

[kovidgoyal]

Nope the file size is the minimum needed to provide all the features the the FTS search has.

[DrChiper]

Ok. That is good to know on deciding to use FT.

[albell]

Originally Posted by DrChiper

I have a question tough about the index file size (full-text-search.db): it is about the same size as the complete library size itself, a ratio of 1:1 so to speak. Up to now I used the program docfetcher, which manages a ratio of 5:1.
Meaning my main library of about 19GiB has a full-text-search.db of ~19GiB,

It is definitely something that would be useful to know in the case of a very large library.

However, a perfect 1:1 ratio seems to me hard to believe: for example, I don't think that scanned or illustrated pdfs occupy the equivalent of their full size in the index, since it is the text layer only that is indexed.

Maybe when the database is generated it already has a minimum size (and therefore the ratio is not 1: 1, even if it seems to be in your case)?

Perhaps some users with heavier libraries who have already performed the indexing, can provide feedback on the size of their archive file.

[thiago.eec]

My library is not that big, but the ratio is almost 5:1 (5.04GB library and 1.18GB full-text-search.db).

Just to compare: using Power Search plugin, the index data is 1.24GB (generated by Elastic Search), virtually the same.

[DrChiper]

I did some checks about the contents of my library.
Types:

• 99,50% = epub
• 00,30% = pdf
• 00,13% = epub + pdf
• 00,06% = other

File sizes:

• 16.00% > 1M
• 84,08% < 1M
• 65,17% < 500K
• 23,78% < 250K
• 03,38% < 100K

IMHO nothing special.

[Sirtel]

The library I indexed is 7.6 GB and the size of the db file is 3.7 GB. Epubs only, with high-quality covers.

As my main library is close to 50 GB, I'm not going to index that.

[BetterRed]

FWIW - MS rule of thumb for their search index size is 10% of the size of files indexed, and the X1 indexes I have, which includes my calibre libraries, is a bit less than that.

But they only run on Windows.

BR

[kovidgoyal]

Remember that an index file is always gong to be much larger than the text file, because its not just text but contains information about which records contain every word, and at what offset and word count, this is what powers the NEAR operator. So essentially every word has some number of extra fields associated with it. And there will be page size overhead for efficient lookup. And of course the full text is also actually stored so that snippets can be shown.

And calibre actually indexes all text twice for the "find related words" functionality, which works by stemming all tokens.

[DrChiper]

Originally Posted by albell

Maybe when the database is generated it already has a minimum size (and therefore the ratio is not 1: 1, even if it seems to be in your case)?

Not quite sure what you mean by this. What I did prior to indexing was issuing the DB maintenance function for that database, which "vacuumed" it and did made inconsistency checks. So the DB to be indexed was sound and minimized.

That said, my gut feeling is that how bigger the DB to be indexed is, the bigger the resulting index DB will be:

@thiago.eec -> 5.04 GB, with index DB 1.18 GB
@Drchiper   -> 6.05 GB, with index DB 4.4 GB
@Sirtel     -> 7.60 GB, with index DB 3.7 GB

What is not yet is considered is the kind of involved ebook contents, which surely has impact. Mine hardly contain pictures, but do have many words (250+ pages) So: YMMV.

[Wiggo]

If anyone is interested

library 105 GB and index DB 21 GB.

[phossler]

Does FTI ignore 'low value' words, like 'the' and 'and' and 'to'?

[ownedbycats]

Originally Posted by phossler

Does FTI ignore 'low value' words, like 'the' and 'and' and 'to'?

I wouldn't think so, as otherwise "the exact match" would fail.

[Comfy.n]

Library folder: 333 GB
FTS db: 70 GB (located in default folder)
Hence, size of library files is 263GB, currently.

I'm using a pretty fast Kingston 2TB NVMe SSD, and I make annual backups to a 4TB external HDD. By the way, earlier this year I lost some data stored on a faulty Corsair NVMe SSD, so I'd tell everyone to stay away from those!