calibre 6 beta: Full text searching

[kovidgoyal]

Hi all,

calibre 6 is ready for Beta testing. The headline feature is Full text search, which means you can search the full text of all book files in your library from within calibre. Simply click the FT button at the left of the calibre search bar. calibre will first need to index the books in your library, a one time process, and then you can search for any word or combination of words in your entire library.

Once you turn on indexing for a library, any future books added to it will be automatically indexed in the background. Note that currently full text searching is not possible through the content server interface, that will likely be implemented later. Testing and feedback on the new Full text search is most welcome.

The other major changes are:

1) Moving to Qt 6 from Qt 5. This should have no user visible changes, except that there may be a few third party plugins that have not yet been ported to support Qt 6

2) Dropping support for 32-bit builds (because Qt 6 does not support them). In particular on Windows, the calibre installer will now automatically uninstall calibre 32bit and replace it with calibre 64bit.

3) Support for the ARM64 CPU architecture on Linux and Apple Silicon on macOS.

4) You can now specify different icon themes for light and dark mode

The betas are fully compatible with calibre 5 you can run both on the same library with no ill effects. calibre 5 will simply ignore the full text search data calibre 6 generates. The only caveat is that if you add books using calibre 5 they will not be automatically indexed, so you might have to periodically re-index the library if you regularly use both 5 and 6 on the same library.

You can download the betas from: https://download.calibre-ebook.com/betas/

For users running from source, use the qt6 branch with the beta builds, and master with regular calibre 5 builds.

Enjoy and thank you for helping to iron out the bugs of which I am sure there will be several since FTS is a fairly large feature, code wise.

[Sirtel]

Installed the beta and ran the indexer. Got this error immediately:

calibre, version 5.99.10
ERROR: Unhandled exception: <b>ZeroDivisionError</b>:float division by zero

calibre 5.99.10  embedded-python: True
Windows-10-10.0.19044 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19044')
Python 3.10.1
Windows: ('10', '10.0.19044', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: DeACSM (0, 0, 15) && DeDRM (7, 0, 0) && Amazon.com Multiple Countries (1, 0, 0) && Author Book Count (2, 2, 2) && Comments Cleaner (1, 6, 3) && Count Pages (1, 11, 2) && EpubCheck (0, 2, 4) && EpubMerge (2, 12, 0) && EpubSplit (3, 2, 0) && FanFicFare (4, 9, 0) && Favourites Menu (1, 1, 0) && Find Duplicates (1, 9, 4) && Generate Cover (2, 1, 0) && Job Spy (1, 0, 190) && Set KFX metadata (from KFX Output) (1, 58, 0) && KFX Output (1, 58, 0) && Kindle hi-res covers (0, 5, 0) && Kobo Books (1, 9, 0) && Kobo Utilities (2, 15, 1) && Modify ePub (1, 7, 0) && MultiColumnSearch (1, 0, 87) && Quality Check (1, 12, 0) && Search The Internet (1, 8, 1) && View Manager (1, 8, 0)
Traceback (most recent call last):
  File "calibre\gui2\fts\scan.py", line 141, in update
  File "calibre\gui2\fts\scan.py", line 63, in time_left
ZeroDivisionError: float division by zero

[kovidgoyal]

That's harmless and temporary, just ignore it.

[kovidgoyal]

Originally Posted by kovidgoyal

That's harmless and temporary, just ignore it.

https://github.com/kovidgoyal/calibr...2248f16e08f22c

[BetterRed]

Curious - did you roll your own or use an o/s text search engine such as Lucene or Sphinx?

And - a glitch in default main toolbar if no icons option is selected, viz:



I clicked it, even though it's invisible it still works

BR

[kovidgoyal]

Originally Posted by BetterRed

Curious - did you roll your own or use an o/s text search engine such as Lucene or Sphinx?

It uses sqlite's FTS5 engine, with a custom tokenizer I wrote to support more languages by leveraging the ICU and libstemmer libraries.

And - a glitch in default main toolbar if no icons option is selected, viz:

Attachment 194544

I clicked it, even though it's invisible it still works

BR

this will be a qt 6 regression, thanks, will look into it. EDIT: Actually this is present in calibre 5 as well. https://github.com/kovidgoyal/calibr...c06f82971ceb60

[Sirtel]

Originally Posted by kovidgoyal

https://github.com/kovidgoyal/calibr...2248f16e08f22c

Thanks.

The indexer has finished now. I noticed that the search results show only the first instance of the word/phrase for every book it was found in, not all results in a book. I guess this is intentional?

[kovidgoyal]

Originally Posted by Sirtel

Thanks.

The indexer has finished now. I noticed that the search results show only the first instance of the word/phrase for every book it was found in, not all results in a book. I guess this is intentional?

Yes, the idea is that you use this to find the book, quickly, searching through the books content in detail is best done inside the book in the calibre viewer or similar, where you can perform more sophisticated searches.

[Uncle Robin]

Originally Posted by kovidgoyal

Yes, the idea is that you use this to find the book, quickly, searching through the books content in detail is best done inside the book in the calibre viewer or similar, where you can perform more sophisticated searches.

This is what I was expecting, as it's the way the current PowerSearch plugin also works. Thanks for building this into Calibre itself

[Sirtel]

Originally Posted by Uncle Robin

This is what I was expecting, as it's the way the current PowerSearch plugin also works. Thanks for building this into Calibre itself

Yes, it's a nice feature. I'm also grateful.

[kovidgoyal]

Added a note to the documentation to clarify that it finds only one instance per book: https://github.com/kovidgoyal/calibr...4476bb0d2b8bba

[BetterRed]

Windows 10

If the main calibre window is hidden or minimised, and I click the "Select" button in the individual book panel. or the "Jump to this book in the library" in the results list the main calibre window is not bought to the front. My preference would be that it does come to the front.

But, "Mark" should leave the main window where it is. Mark multiple would be nice.

I've been wondering what you were up to. I tossed it a 3,300 book library, its indexing 6,900 format files - which formats does it index/not index?

As always very impressive

Originally Posted by kovidgoyal

EDIT: Actually this is present in calibre 5 as well.

I only discovered the no-icon toolbar option last week

BR

[kovidgoyal]

Originally Posted by BetterRed

Windows 10

If the main calibre window is hidden or minimised, and I click the "Select" button in the individual book panel. or the "Jump to this book in the library" in the results list the main calibre window is not bought to the front. My preference would be that it does come to the front.

I thought about this but am not sure its a good idea, I generally find clicking in one window and having another popup in front disruptive. Still I am not wedded to this, lets hear other opinions.

But, "Mark" should leave the main window where it is. Mark multiple would be nice.

There's a mark all button in the bottom of the dialog.

I've been wondering what you were up to. I tossed it a 3,300 book library, its indexing 6,900 format files - which formats does it index/not index?

Anything you can read with the calibre viewer or convert from in calibre, it will be able to index as well.

[jbacelar]

I was looking forward to you implementing this feature.
I'm going to try it immediately.
Thank you very, very much Kovid.

[BetterRed]

Originally Posted by kovidgoyal

I thought about this but am not sure its a good idea, I generally find clicking in one window and having another popup in front disruptive. Still I am not wedded to this, lets hear other opinions.

The tooltip says: "Scroll to this book in the calibre library and select it". Given there's a Mark option, what would be the point of Selecting the book if I'm not going to do something with it - e.g. edit, convert, tag, etc….

If I click on an EPUB file link in the book results panel the calibre viewer pops up in front - as I would expect. But… if I click on a DOCX file link, and I have DOCX unchecked in the "Use internal viewer for:" list, the calibre viewer also pops up, IMO it should pop the default handler for DOCX.

Originally Posted by kovidgoyal

There's a mark all button in the bottom of the dialog.

Yes I spotted that after I posted, maybe Mark All and Select All could be added to the results context menu.

BR