Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 07-23-2020, 05:38 AM   #1
mapozyan
Connoisseur
mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.
 
Posts: 77
Karma: 90088
Join Date: Jul 2020
Device: android
[GUI Plugin] Power (Full-text) Search

Calibre Power Search Plugin
===========================

Version 2.2.0, released on 27 March 2022. Created by Michael Apozyan
Platform: Linux, Windows, macOS
Calibre versions: 3.x, 4.x, 5.x, 6.x


Overview

This plugin adds full-text search functionality to your electronic library. Default shortcut key for Power Search dialog is Ctrl+Shift+S.
When you run Power Search, it will extract text from all your books on the fly and load all this text in a local full-text search engine. Plugin will never modify any books and records in Calibre library. As a search backend, we are using ElasticSearch. Once the search index is created and is up-to-date, Power Search will query search engine and display list of books where certain search keywords are found.
For large libraries, running search for the first time might take a while. Subsequent searches will normally take less than a second.

Usage

We tried to make Power Search easy to use. And yet it allows you running complex queries too.
  • When you enter a query with multiple words, Power Serach will try to find all of the books that contain all of your query words. However, it won't be a phrase search, i.e. your query words might appear in different parts of the text.
    If you wish to search for a particular phrase, put it in double quotes.
    Example: "independence day"
  • If you need to exclude documents containing specific word, prepend it with a minus sign.
    Example: good -bad
  • To find books containing any of given words, use | operator (which means OR).
    Example: good | bad
  • You can write complex expression by using round brackets.
    Example: (good | bad) pizza
  • Power Search supports wildcard search with * (any number of characters) and ? (one character) symbols.
    Example: math*
  • It's possible to write even more complex queries by using regular expressions put in forward slashes:
    Example: /14[1-2][0-9]/ renaissance

Requirements
  • ElasticSearch >= 6.x

    In order to use Power Search Plugin, you need to have ElasticSearch service up and running.
    For local setup, just download ElasticSearch package from the official website
    https://www.elastic.co/downloads/elasticsearch and unzip it in any folder.

    If your system stays in trusted network (like your home network) and hence you are not much
    concerned about fulltext data security, you may simplify configuring ElasticSearch by
    disabling security module. Just go to config/elasticsearch.yml and add or modify following
    configuration value:

    xpack.security.enabled: false

  • Pdftotext utility (optional, but highly recommended)

    Generally, Power Search Plugin is using Calibre's native converters to build full-text index.
    However, native PDF conversion is sometimes working too slowly. To speed up this process, you
    can install pdftotext utility and plugin will make use of it. Pdftotext is part of Xpdf and
    Poppler projects.

    Here are some hints:

    - Xpdf package can be found here (look for "Xpdf command line tools")
    https://www.xpdfreader.com/download.html

    - For Debian/Ubuntu, check for 'xpdf' or 'libpoppler' apt packages.

Configuring
  • ElasticSearch engine network path
    If you installed ElasticSearch locally, it will usually be available at http://localhost:9200/ so you don't need to modify this setting. However, you can change it if you need different setup.
  • ElasticSearch engine local path (optional)
    Example: c:\programs\elasticsearch-7.8.1
    If you plan to run ElasticSearch locally, Power Search might be set up to launch it in the background when needed. Just point here to the home directory where ElasticSearch is located.
    Note that in this case Power Search will manage to stop ElasticSearch when you close Calibre.
  • Path to pdftotext tool (optional)
    If pdftotext tool can not be found in PATH, enter the full path to pdftotext executable file.
  • Number of parallel processes for text extraction
    By default this number is equal to number of CPUs on your system minus one. You can change this value if neccessary.
  • Index book formats
    You can enable/disable specific book file formats that should be indexed.
  • Automatically index new books on search
    You can enable/disable an option of discovering new books and indexing them each time you run
    the search.


Version History:

Spoiler:


Version 2.2.0 - 27 March 2022
Add Calibre 6 support

Version 2.1.2 - 20 February 2022
Fixed crash on multi-display environments
Fixed limit of max 10k results

Version 2.1.1 - 24 October 2021
Minor fixes

Version 2.1.0 - 16 June 2021
Support for multiple libraries
Add dropdown menu to Search button
Bugfixes

Version 2.0.1 - 04 April 2021
Bugfix

Version 2.0.0 - 29 March 2021
Major feature update

Version 1.7.0 - 04 November 2020
Major feature update

Version 1.6.0 - 25 September 2020
Calibre 5 support

Version 1.5.0 - 08 September 2020
macOS support
Usability improvements

Version 1.4.0 - 24 August 2020
Usability improvements

Version 1.3.0 - 08 August 2020
Support for complex queries

Version 1.2.0 - 31 July 2020
Usability improvements

Version 1.1.1 - 27 July 2020
Bugfix

Version 1.1.0 - 26 July 2020
Add Windows support

Version 1.0.0 - 23 July 2020
Initial Linux release


Feedback

For bug reports and feature requests please contact mapozyan (at) yahoo.com
The project is hosted on GitHub at https://github.com/mapozyan/caps

Authors
  • mapozyan
  • thiago.ecc

Thanks to
  • ownedbycats - for testing and ideas
  • DNSB - for providing help to the community
Attached Thumbnails
Click image for larger version

Name:	screenshot.png
Views:	387
Size:	102.2 KB
ID:	195013  
Attached Files
File Type: zip Calibre Power Search Plugin.2.2.0.zip (1.69 MB, 29733 views)

Last edited by mapozyan; 07-11-2023 at 06:53 PM. Reason: Add Calibre 6 support
mapozyan is offline   Reply With Quote
Old 07-23-2020, 06:01 AM   #2
mapozyan
Connoisseur
mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.
 
Posts: 77
Karma: 90088
Join Date: Jul 2020
Device: android
--placeholder for test versions--

Last edited by mapozyan; 07-15-2022 at 12:24 PM. Reason: Add version 2.2.0
mapozyan is offline   Reply With Quote
Advert
Old 07-23-2020, 10:33 AM   #3
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,334
Karma: 57894427
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
I can't wait for the Windows version.
ownedbycats is online now   Reply With Quote
Old 07-26-2020, 03:16 PM   #4
mapozyan
Connoisseur
mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.
 
Posts: 77
Karma: 90088
Join Date: Jul 2020
Device: android
Version 1.1.0 released. Works on Windows!

Last edited by mapozyan; 07-31-2020 at 12:10 PM. Reason: Removed version 1.1.0
mapozyan is offline   Reply With Quote
Old 07-26-2020, 05:06 PM   #5
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,334
Karma: 57894427
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Possible bug in Windows, although I'm not sure if there's anything to be done: when doing a first search, all the ebook-convert console windows pop up in the foreground and steal focus, making the computer pretty much unusable until the search is either complete or cancelled.

Seeing as the initial indexing takes a while (I ended up cancelling when it was nearing an hour and a half, and the bar was only 1/5 full!) it's...a bit of a problem.

Last edited by ownedbycats; 07-26-2020 at 06:21 PM.
ownedbycats is online now   Reply With Quote
Advert
Old 07-27-2020, 02:49 AM   #6
mapozyan
Connoisseur
mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.
 
Posts: 77
Karma: 90088
Join Date: Jul 2020
Device: android
Hmm, I'll look into it. Thanks for feedback!
mapozyan is offline   Reply With Quote
Old 07-27-2020, 03:15 AM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@mapozyan You need to set creationflags to CREATE_NO_WINDOW on windows when calling Popen()
kovidgoyal is offline   Reply With Quote
Old 07-27-2020, 03:20 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Sorry typoe, you need DETACHED_PROCESS not CREATE_NO_WINDOW
kovidgoyal is offline   Reply With Quote
Old 07-27-2020, 05:30 AM   #9
mapozyan
Connoisseur
mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.
 
Posts: 77
Karma: 90088
Join Date: Jul 2020
Device: android
Fixed issue in version 1.1.1. Please find in attachment.

Thanks for hint, @kovidgoyal!

Just a quick observation on how much time the initial index creation might take.
Processing my own library with ~3200 books and total size ~23GB took me around 5 hours.
My computer is a decent i7 laptop with 4 cores.

Last edited by mapozyan; 07-31-2020 at 12:11 PM. Reason: Removed version 1.1.1
mapozyan is offline   Reply With Quote
Old 07-27-2020, 08:52 AM   #10
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,334
Karma: 57894427
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
I installed the new version and grabbed pdftotext. This time my computer is usable while the initial search runs in the background. pdftotext also speeds up the process quite a bit.

Another odd quirk: I have a six-core processor. The readme file indicated that the default number of parallel processes should be set to the number of CPUs, but in mine it was default set to 12. It's possible that was another reason it was running so slow before. It may be some oddity stemming from my particular CPU (AMD Ryzen 2600) though.

Last edited by ownedbycats; 07-27-2020 at 08:56 AM.
ownedbycats is online now   Reply With Quote
Old 07-27-2020, 09:15 AM   #11
mapozyan
Connoisseur
mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.
 
Posts: 77
Karma: 90088
Join Date: Jul 2020
Device: android
Some fancy technologies like Hyper-threading might double the real number of cores.
However, setting 'number of parallel processes' option to a higher value should not be disastrous. Your operating system is smart enough to deal with it.
mapozyan is offline   Reply With Quote
Old 07-27-2020, 10:03 AM   #12
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,334
Karma: 57894427
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
It's all indexed now!

Another bug: I put a phrase in "quotation marks" to test searching exact phrases and got an error about an unexpected character. Error log:

Spoiler:
Code:
calibre, version 4.21.0
ERROR: Unhandled exception: <b>RequestError</b>:RequestError(400, u'json_parse_exception', u"Unexpected character ('s' (code 115)): was expecting comma to separate Object entries\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 57]")

calibre 4.21 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.18362-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: DeDRM (6, 7, 0) && Open With (1, 5, 12) && Goodreads (1, 4, 0) && View Manager (1, 5, 6) && Manage Series (1, 2, 9) && Reading List (1, 6, 11) && NormComment (0, 0, 2) && Extract ISBN (1, 5, 0) && Job Spy (1, 0, 184) && MultiColumnSearch (1, 0, 82) && Library Codes (1, 0, 46) && Modify ePub (1, 4, 1) && Kindle hi-res covers (0, 4, 0) && Quality Check (1, 10, 1) && Kobo Utilities (2, 11, 4) && KoboTouchExtended (3, 2, 13) && Save To Format (0, 0, 6) && FanFicFare (3, 21, 2) && Embed Comic Metadata (1, 4, 0) && Find Duplicates (1, 7, 2) && Favourites Menu (1, 0, 5) && EpubCheck (0, 2, 3) && Fantastic Fiction (1, 3, 0) && Drop Search Results (1, 0, 10) && Calibre Power Search Plugin (1, 1, 1) && Barnes & Noble (1, 2, 15) && Count Pages (1, 10, 0) && FictionDB (1, 2, 0) && Generate Cover (1, 5, 25) && Kobo Books (1, 8, 0) && Import List (1, 4, 5)
Traceback (most recent call last):
  File "calibre_plugins.caps.main", line 208, in on_search
  File "calibre_plugins.caps.main", line 282, in do_search
  File "calibre_plugins.caps.elasticsearch.client.utils", line 152, in _wrapped
  File "calibre_plugins.caps.elasticsearch.client.__init__", line 1470, in search
  File "calibre_plugins.caps.elasticsearch.transport", line 392, in perform_request
RequestError: RequestError(400, u'json_parse_exception', u"Unexpected character ('s' (code 115)): was expecting comma to separate Object entries\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 57]")
ownedbycats is online now   Reply With Quote
Old 07-27-2020, 10:50 AM   #13
mapozyan
Connoisseur
mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.mapozyan composes epic poetry in binary.
 
Posts: 77
Karma: 90088
Join Date: Jul 2020
Device: android
Thanks for feedback @ownedbycats!

I did not try searching for exact phrase yet, need to figure out how to do it in elasticsearch.

Stay tuned for updates!
mapozyan is offline   Reply With Quote
Old 07-27-2020, 11:58 AM   #14
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,334
Karma: 57894427
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Searching for a compound word that uses a hyphen (e.g. up-to-date) also returns results for the separate words.

Last edited by ownedbycats; 07-27-2020 at 02:34 PM.
ownedbycats is online now   Reply With Quote
Old 07-27-2020, 12:22 PM   #15
DNSB
Bibliophagist
DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.
 
DNSB's Avatar
 
Posts: 34,236
Karma: 144198474
Join Date: Jul 2010
Location: Vancouver
Device: Kobo Sage, Forma, Clara HD, Lenovo M8 FHD, Paperwhite 4, Tolino epos
Quote:
Originally Posted by ownedbycats View Post
Another odd quirk: I have a six-core processor. The readme file indicated that the default number of parallel processes should be set to the number of CPUs, but in mine it was default set to 12. It's possible that was another reason it was running so slow before. It may be some oddity stemming from my particular CPU (AMD Ryzen 2600) though.
A Ryzen 2600 6 core is listed as 6 core 12 thread so the CPU supports hyperthreading. OTOH, I've found several times where exceeding the number of physical CPU cores slows processes down since hyperthreading does not duplicate the full CPU core.

And my test is still running after 8 hours (~11,500 books).

Edited to copy the edit comment to the body of the message:

Edit: Just finished. at 09:24. About 2 minutes after the message was written.

Last edited by DNSB; 07-27-2020 at 12:50 PM. Reason: Edit: Moved just finished message and time into the body of the message
DNSB is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Search the Internet kiwidude Plugins 408 03-16-2024 11:57 PM
[GUI Plugin] Clipboard Search kiwidude Plugins 24 03-16-2024 11:06 PM
[GUI Plugin] Recoll Full Text Search Satas Plugins 16 08-05-2016 03:54 AM
[GUI Plugin] Full Text Search (SOLR) peterpisljar Plugins 2 08-09-2015 08:16 AM
Make a simple Plugin for Full Text Search using Recoll Satas Development 9 07-20-2013 04:15 PM


All times are GMT -4. The time now is 05:16 AM.


MobileRead.com is a privately owned, operated and funded community.