[GUI Plugin] Power (Full-text) Search

[mapozyan]

Calibre Power Search Plugin
===========================

Version 2.2.0, released on 27 March 2022. Created by Michael Apozyan
Platform: Linux, Windows, macOS
Calibre versions: 3.x, 4.x, 5.x, 6.x


Overview

This plugin adds full-text search functionality to your electronic library. Default shortcut key for Power Search dialog is Ctrl+Shift+S.
When you run Power Search, it will extract text from all your books on the fly and load all this text in a local full-text search engine. Plugin will never modify any books and records in Calibre library. As a search backend, we are using ElasticSearch. Once the search index is created and is up-to-date, Power Search will query search engine and display list of books where certain search keywords are found.
For large libraries, running search for the first time might take a while. Subsequent searches will normally take less than a second.

Usage

We tried to make Power Search easy to use. And yet it allows you running complex queries too.

• When you enter a query with multiple words, Power Serach will try to find all of the books that contain all of your query words. However, it won't be a phrase search, i.e. your query words might appear in different parts of the text.
  If you wish to search for a particular phrase, put it in double quotes.
  Example: "independence day"
  
• If you need to exclude documents containing specific word, prepend it with a minus sign.
  Example: good -bad
  
• To find books containing any of given words, use | operator (which means OR).
  Example: good | bad
  
• You can write complex expression by using round brackets.
  Example: (good | bad) pizza
  
• Power Search supports wildcard search with * (any number of characters) and ? (one character) symbols.
  Example: math*
  
• It's possible to write even more complex queries by using regular expressions put in forward slashes:
  Example: /14[1-2][0-9]/ renaissance

Requirements

• ElasticSearch >= 6.x
  
  In order to use Power Search Plugin, you need to have ElasticSearch service up and running.
  For local setup, just download ElasticSearch package from the official website
  https://www.elastic.co/downloads/elasticsearch and unzip it in any folder.
  
  If your system stays in trusted network (like your home network) and hence you are not much
  concerned about fulltext data security, you may simplify configuring ElasticSearch by
  disabling security module. Just go to config/elasticsearch.yml and add or modify following
  configuration value:
  
  xpack.security.enabled: false
  
  
• Pdftotext utility (optional, but highly recommended)
  
  Generally, Power Search Plugin is using Calibre's native converters to build full-text index.
  However, native PDF conversion is sometimes working too slowly. To speed up this process, you
  can install pdftotext utility and plugin will make use of it. Pdftotext is part of Xpdf and
  Poppler projects.
  
  Here are some hints:
  
  - Xpdf package can be found here (look for "Xpdf command line tools")
  https://www.xpdfreader.com/download.html
  
  - For Debian/Ubuntu, check for 'xpdf' or 'libpoppler' apt packages.
  

Configuring

• ElasticSearch engine network path
  If you installed ElasticSearch locally, it will usually be available at http://localhost:9200/ so you don't need to modify this setting. However, you can change it if you need different setup.
  
• ElasticSearch engine local path (optional)
  Example: c:\programs\elasticsearch-7.8.1
  If you plan to run ElasticSearch locally, Power Search might be set up to launch it in the background when needed. Just point here to the home directory where ElasticSearch is located.
  Note that in this case Power Search will manage to stop ElasticSearch when you close Calibre.
  
• Path to pdftotext tool (optional)
  If pdftotext tool can not be found in PATH, enter the full path to pdftotext executable file.
  
• Number of parallel processes for text extraction
  By default this number is equal to number of CPUs on your system minus one. You can change this value if neccessary.
  
• Index book formats
  You can enable/disable specific book file formats that should be indexed.
  
• Automatically index new books on search
  You can enable/disable an option of discovering new books and indexing them each time you run
  the search.

Version History:

Spoiler:

Version 2.2.0 - 27 March 2022
Add Calibre 6 support

Version 2.1.2 - 20 February 2022
Fixed crash on multi-display environments
Fixed limit of max 10k results

Version 2.1.1 - 24 October 2021
Minor fixes

Version 2.1.0 - 16 June 2021
Support for multiple libraries
Add dropdown menu to Search button
Bugfixes

Version 2.0.1 - 04 April 2021
Bugfix

Version 2.0.0 - 29 March 2021
Major feature update

Version 1.7.0 - 04 November 2020
Major feature update

Version 1.6.0 - 25 September 2020
Calibre 5 support

Version 1.5.0 - 08 September 2020
macOS support
Usability improvements

Version 1.4.0 - 24 August 2020
Usability improvements

Version 1.3.0 - 08 August 2020
Support for complex queries

Version 1.2.0 - 31 July 2020
Usability improvements

Version 1.1.1 - 27 July 2020
Bugfix

Version 1.1.0 - 26 July 2020
Add Windows support

Version 1.0.0 - 23 July 2020
Initial Linux release

Feedback

For bug reports and feature requests please contact mapozyan (at) yahoo.com
The project is hosted on GitHub at https://github.com/mapozyan/caps

Authors

• mapozyan
• thiago.ecc

Thanks to

• ownedbycats - for testing and ideas
• DNSB - for providing help to the community

[mapozyan]

--placeholder for test versions--

[ownedbycats]

I can't wait for the Windows version.

[mapozyan]

Version 1.1.0 released. Works on Windows!

[ownedbycats]

Possible bug in Windows, although I'm not sure if there's anything to be done: when doing a first search, all the ebook-convert console windows pop up in the foreground and steal focus, making the computer pretty much unusable until the search is either complete or cancelled.

Seeing as the initial indexing takes a while (I ended up cancelling when it was nearing an hour and a half, and the bar was only 1/5 full!) it's...a bit of a problem.

[mapozyan]

Hmm, I'll look into it. Thanks for feedback!

[kovidgoyal]

@mapozyan You need to set creationflags to CREATE_NO_WINDOW on windows when calling Popen()

[kovidgoyal]

Sorry typoe, you need DETACHED_PROCESS not CREATE_NO_WINDOW

[mapozyan]

Fixed issue in version 1.1.1. Please find in attachment.

Thanks for hint, @kovidgoyal!

Just a quick observation on how much time the initial index creation might take.
Processing my own library with ~3200 books and total size ~23GB took me around 5 hours.
My computer is a decent i7 laptop with 4 cores.

[ownedbycats]

I installed the new version and grabbed pdftotext. This time my computer is usable while the initial search runs in the background. pdftotext also speeds up the process quite a bit.

Another odd quirk: I have a six-core processor. The readme file indicated that the default number of parallel processes should be set to the number of CPUs, but in mine it was default set to 12. It's possible that was another reason it was running so slow before. It may be some oddity stemming from my particular CPU (AMD Ryzen 2600) though.

[mapozyan]

Some fancy technologies like Hyper-threading might double the real number of cores.
However, setting 'number of parallel processes' option to a higher value should not be disastrous. Your operating system is smart enough to deal with it.

[ownedbycats]

It's all indexed now!

Another bug: I put a phrase in "quotation marks" to test searching exact phrases and got an error about an unexpected character. Error log:

Spoiler:

calibre, version 4.21.0
ERROR: Unhandled exception: <b>RequestError</b>:RequestError(400, u'json_parse_exception', u"Unexpected character ('s' (code 115)): was expecting comma to separate Object entries\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 57]")

calibre 4.21 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.18362-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: DeDRM (6, 7, 0) && Open With (1, 5, 12) && Goodreads (1, 4, 0) && View Manager (1, 5, 6) && Manage Series (1, 2, 9) && Reading List (1, 6, 11) && NormComment (0, 0, 2) && Extract ISBN (1, 5, 0) && Job Spy (1, 0, 184) && MultiColumnSearch (1, 0, 82) && Library Codes (1, 0, 46) && Modify ePub (1, 4, 1) && Kindle hi-res covers (0, 4, 0) && Quality Check (1, 10, 1) && Kobo Utilities (2, 11, 4) && KoboTouchExtended (3, 2, 13) && Save To Format (0, 0, 6) && FanFicFare (3, 21, 2) && Embed Comic Metadata (1, 4, 0) && Find Duplicates (1, 7, 2) && Favourites Menu (1, 0, 5) && EpubCheck (0, 2, 3) && Fantastic Fiction (1, 3, 0) && Drop Search Results (1, 0, 10) && Calibre Power Search Plugin (1, 1, 1) && Barnes & Noble (1, 2, 15) && Count Pages (1, 10, 0) && FictionDB (1, 2, 0) && Generate Cover (1, 5, 25) && Kobo Books (1, 8, 0) && Import List (1, 4, 5)
Traceback (most recent call last):
  File "calibre_plugins.caps.main", line 208, in on_search
  File "calibre_plugins.caps.main", line 282, in do_search
  File "calibre_plugins.caps.elasticsearch.client.utils", line 152, in _wrapped
  File "calibre_plugins.caps.elasticsearch.client.__init__", line 1470, in search
  File "calibre_plugins.caps.elasticsearch.transport", line 392, in perform_request
RequestError: RequestError(400, u'json_parse_exception', u"Unexpected character ('s' (code 115)): was expecting comma to separate Object entries\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 57]")

[mapozyan]

Thanks for feedback @ownedbycats!

I did not try searching for exact phrase yet, need to figure out how to do it in elasticsearch.

Stay tuned for updates!

[ownedbycats]

Searching for a compound word that uses a hyphen (e.g. up-to-date) also returns results for the separate words.

[DNSB]

Originally Posted by ownedbycats

Another odd quirk: I have a six-core processor. The readme file indicated that the default number of parallel processes should be set to the number of CPUs, but in mine it was default set to 12. It's possible that was another reason it was running so slow before. It may be some oddity stemming from my particular CPU (AMD Ryzen 2600) though.

A Ryzen 2600 6 core is listed as 6 core 12 thread so the CPU supports hyperthreading. OTOH, I've found several times where exceeding the number of physical CPU cores slows processes down since hyperthreading does not duplicate the full CPU core.

And my test is still running after 8 hours (~11,500 books).

Edited to copy the edit comment to the body of the message:

Edit: Just finished. at 09:24. About 2 minutes after the message was written.