Full Text Search Indexes All Formats?

JimmXinu · 07-30-2022, 10:04 AM

I kicked off the full text search indexing and noticed that it reported ~twice as many books to index as I have in my library.

However, I do have both epub and azw3 for all of them. Which are textually identical.

Is there an option to only full text index one preferred format? Should there be?

In my case, I expect it would roughly halve the time and size of the search index.

ownedbycats · 07-30-2022, 10:18 AM

Calibre reports 6517 index files in my library of 6250 books.

formats:#>1 reports 264 books, which matches up.

What I find weird is that most of those secondary formats are PAPERBOOK, which is a 0-byte dummy file. Somehow I expected the FTS to only do formats that Calibre recognized and could open in its reader (well, it's a renamed text file, so it probably could...).

DrChiper · 07-30-2022, 11:13 AM

I have some ebooks in several languages and then some ebooks are in multiple formats too, among them PDF's. Now there are PDF's and PDF's with superimposed texts so they are searchable. No clue what this means for the indexing process but they seem to be all indexed somehow.

kovidgoyal · 07-30-2022, 11:36 AM

Yes it indexes all formats (well all formats calibre knows how to read). I prefer to be thorough rather than potentially miss something. Note that when searching cuplicate matches from multiple formats in the same book are coalesced.

JimmXinu · 07-30-2022, 12:49 PM

I agree that some people will benefit from indexing all formats and should probably be the default setting.

But for me it's an unneeded increase in DB size and indexing time.

Perhaps a tweak setting could be added at some point to limit indexing to a list of formats?

JimmXinu · 08-01-2022, 04:52 PM

FYI, I submitted a PR to implement this tweak.

A semi-related question: Indexing speed reverts to Slow all the time; is it supposed to?

Uncle Robin · 08-01-2022, 05:39 PM

Quote:

Originally Posted by JimmXinu

I agree that some people will benefit from indexing all formats and should probably be the default setting.

But for me it's an unneeded increase in DB size and indexing time.

Perhaps a tweak setting could be added at some point to limit indexing to a list of formats?

I support this idea. The Power Search plugin offers users the choice of which formats to index, and if adding this functionality to Calibre's internal FTS is feasible, it would be a very worthwhile enhancement

BetterRed · 08-01-2022, 07:16 PM

Quote:

Originally Posted by JimmXinu

A semi-related question: Indexing speed reverts to Slow all the time; is it supposed to?

The other content indexers I use also default to Slow; so, you could say its a de-facto standard.

BR

ownedbycats · 08-01-2022, 07:50 PM

Quote:

Originally Posted by JimmXinu

A semi-related question: Indexing speed reverts to Slow all the time; is it supposed to?

During my initial index (during the beta), it reverted to slow whenever I closed the indexer window.

kovidgoyal · 08-01-2022, 10:50 PM

Yes, it is meant to revert to slow by default. Fast is meant only for use during initial indexing, or when you add lots of books and want to index them quickly. Therefore when you close the indexer window, it reverts to slow. Fast makes the calibre UI (and your computer depending on specs) become very sluggish.

gbm · 08-01-2022, 11:20 PM

When I ran the indexer in fast I did not notice any slow downs, left the indexer window open until ti competed and contirlt to go to web pages and watch videos. Finished my 5500+ book formats in about 15 minutes.

Linux Mint 20.3 Cinnamon

bernie

Quote:

Originally Posted by kovidgoyal

Yes, it is meant to revert to slow by default. Fast is meant only for use during initial indexing, or when you add lots of books and want to index them quickly. Therefore when you close the indexer window, it reverts to slow. Fast makes the calibre UI (and your computer depending on specs) become very sluggish.

kovidgoyal · 08-02-2022, 01:18 AM

Well yes it depends on your computer's capabilities, how you have configured calibre, the type of files being indexed, etc.

ownedbycats · 08-02-2022, 07:23 AM

Quote:

Originally Posted by gbm

When I ran the indexer in fast I did not notice any slow downs, left the indexer window open until ti competed and contirlt to go to web pages and watch videos. Finished my 5500+ book formats in about 15 minutes.

Linux Mint 20.3 Cinnamon

bernie

In the beta, my initial index outright stalled out because it opened 100+ pdftohtml processes which pushed everything into swap memory.

Thankfully, this was later fixed.

ownedbycats · 12-09-2022, 04:30 PM

I was having a look at the index database to see if there was some way to take account of books that have no indexed text (dummy files, or non-OCR'd pdf scans).

The books_text table in the index database shows 5554 entries to my 6,462 book records. 842 boook entries have only dummy paperback/overdrive files, so it seems to roughly match up (taking into account a) unknown number of PDF scans and b) book records with multiple formats).

I don't want to run another full re-index, but I'm curious whether 5554 matches the number Calibre would report when doing that.

Alas, not sure what I can do. I have the ids of all the books indexed, so theoretically I could look for the missing numbers. But the gaps from deleted books...

07-30-2022, 10:04 AM	#1
JimmXinu Plugin Developer Posts: 6,973 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	Full Text Search Indexes All Formats? I kicked off the full text search indexing and noticed that it reported ~twice as many books to index as I have in my library. However, I do have both epub and azw3 for all of them. Which are textually identical. Is there an option to only full text index one preferred format? Should there be? In my case, I expect it would roughly halve the time and size of the search index.

07-30-2022, 10:18 AM	#2
ownedbycats Custom User Title Posts: 10,975 Karma: 75337983 Join Date: Oct 2018 Location: Canada Device: Kobo Libra H2O, formerly Aura HD	Calibre reports 6517 index files in my library of 6250 books. formats:#>1 reports 264 books, which matches up. What I find weird is that most of those secondary formats are PAPERBOOK, which is a 0-byte dummy file. Somehow I expected the FTS to only do formats that Calibre recognized and could open in its reader (well, it's a renamed text file, so it probably could...). Last edited by ownedbycats; 07-30-2022 at 10:21 AM.

12-09-2022, 04:30 PM	#14
ownedbycats Custom User Title Posts: 10,975 Karma: 75337983 Join Date: Oct 2018 Location: Canada Device: Kobo Libra H2O, formerly Aura HD	I was having a look at the index database to see if there was some way to take account of books that have no indexed text (dummy files, or non-OCR'd pdf scans). The books_text table in the index database shows 5554 entries to my 6,462 book records. 842 boook entries have only dummy paperback/overdrive files, so it seems to roughly match up (taking into account a) unknown number of PDF scans and b) book records with multiple formats). I don't want to run another full re-index, but I'm curious whether 5554 matches the number Calibre would report when doing that. Alas, not sure what I can do. I have the ids of all the books indexed, so theoretically I could look for the missing numbers. But the gaps from deleted books... Last edited by ownedbycats; 12-09-2022 at 04:33 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Full Text Search query	DrChiper	Calibre	2	07-26-2022 06:31 AM
Full text search?	excaliber	Library Management	3	08-07-2017 06:09 AM
Full Text Search?	silentguy	Calibre	4	02-22-2012 03:03 PM
Full Text Search Engine	Fat Abe	General Discussions	1	09-21-2010 05:30 PM
Google Book Search to search full-text books online	Bob Russell	Deals and Resources (No Self-Promotion or Affiliate Links)	1	08-19-2006 12:13 PM

07-30-2022, 11:13 AM	#3
DrChiper Bookish Posts: 1,017 Karma: 2003162 Join Date: Jun 2011 Device: PC, t1, t2, t3, Clara BW, Clara HD, Libra 2, Libra Color, Nxtpaper 11	I have some ebooks in several languages and then some ebooks are in multiple formats too, among them PDF's. Now there are PDF's and PDF's with superimposed texts so they are searchable. No clue what this means for the indexing process but they seem to be all indexed somehow.

07-30-2022, 11:36 AM	#4
kovidgoyal creator of calibre Posts: 45,359 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Yes it indexes all formats (well all formats calibre knows how to read). I prefer to be thorough rather than potentially miss something. Note that when searching cuplicate matches from multiple formats in the same book are coalesced.

07-30-2022, 12:49 PM	#5
JimmXinu Plugin Developer Posts: 6,973 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	I agree that some people will benefit from indexing all formats and should probably be the default setting. But for me it's an unneeded increase in DB size and indexing time. Perhaps a tweak setting could be added at some point to limit indexing to a list of formats?

08-01-2022, 04:52 PM	#6
JimmXinu Plugin Developer Posts: 6,973 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	FYI, I submitted a PR to implement this tweak. A semi-related question: Indexing speed reverts to Slow all the time; is it supposed to?

08-01-2022, 10:50 PM	#10
kovidgoyal creator of calibre Posts: 45,359 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Yes, it is meant to revert to slow by default. Fast is meant only for use during initial indexing, or when you add lots of books and want to index them quickly. Therefore when you close the indexer window, it reverts to slow. Fast makes the calibre UI (and your computer depending on specs) become very sluggish.

08-02-2022, 01:18 AM	#12
kovidgoyal creator of calibre Posts: 45,359 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Well yes it depends on your computer's capabilities, how you have configured calibre, the type of files being indexed, etc.