Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 07-23-2020, 06:38 AM   #1
mapozyan
Enthusiast
mapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheese
 
Posts: 34
Karma: 1002
Join Date: Jul 2020
Device: android
[GUI Plugin] Power (Full-text) Search

Calibre Power Search Plugin
===========================

Version 1.7.0, released on 4 Nov 2020. Created by Michael Apozyan
Platform: Linux, Windows, macOS
Calibre versions: 3.x, 4.x, 5.x


Overview

This plugin adds full-text search functionality to your electronic library. Default shortcut key for Power Search dialog is Ctrl+Shift+S.
When you run Power Search, it will extract text from all your books on the fly and load all this text in a local full-text search engine. Plugin will never modify any books and records in Calibre library. As a search backend, we are using ElasticSearch. Once the search index is created and is up-to-date, Power Search will query search engine and display list of books where certain search keywords are found.
For large libraries, running search for the first time might take a while. Subsequent searches will normally take less than a second.

Usage

We tried to make Power Search easy to use. And yet it allows you running complex queries too.
  • When you enter a query with multiple words, Power Serach will try to find all of the books that contain all of your query words. However, it won't be a phrase search, i.e. your query words might appear in different parts of the text.
    If you wish to search for a particular phrase, put it in double quotes.
    Example: "independence day"
  • If you need to exclude documents containing specific word, prepend it with a minus sign.
    Example: good -bad
  • To find books containing any of given words, use | operator (which means OR).
    Example: good | bad
  • You can write complex expression by using round brackets.
    Example: (good | bad) pizza
  • Power Search supports wildcard search with * (any number of characters) and ? (one character) symbols.
    Example: math*
  • It's possible to write even more complex queries by using regular expressions put in forward slashes:
    Example: /14[1-2][0-9]/ renaissance

Requirements
  • ElasticSearch >= 6.x
    In order to use Power Search Plugin you need to have ElasticSearch service up and running.
    Its not a difficult procedure, just follow the instructions provided on the official website to install it:
    https://www.elastic.co/guide/en/elas...d-install.html
  • Pdftotext utility (optional, but highly recommended)
    Generally, Power Search Plugin is using Calibre's native converters to build full-text index.
    However, native PDF conversion is sometimes working too slow. To speed up this process, you can install pdftotext utility and plugin will make use of it. Pdftotext is part of Xpdf and Poppler projects. Specifically for Debian/Ubuntu check for 'xpdf' or 'libpoppler73' apt packages.

Configuring
  • ElasticSearch engine location
    If you installed ElasticSearch locally, it will usually be available at http://localhost:9200/ so you don't need to modify this setting. However, you can change it if you need different setup.
  • Path to pdftotext tool
    If pdftotext tool can not be found in PATH, enter the full path to pdftotext.
  • Number of parallel processes for text extraction
    By default this number is equal to number of CPUs on your system minus one. You can change this value if neccessary.
  • Index book formats
    You can enable/disable specific book file formats that should be indexed.
  • Automatically index new books on search
    You can enable/disable an option of discovering new books and indexing them each time you run
    the search.


Version History
  • 1.0.0: initial Linux release
  • 1.1.0: adds Windows support
  • 1.1.1: bug fix
  • 1.2.0: usability improvements
  • 1.3.0: support for complex queries
  • 1.4.0: usability improvements
  • 1.5.0: macOS support, usability improvements
  • 1.6.0: Calibre 5 support
  • 1.7.0: major feature update

Feedback

For bug reports and feature requests please contact mapozyan (at) yahoo.com
The project is hosted on GitHub at https://github.com/mapozyan/caps

Authors
  • mapozyan
  • thiago.ecc

Thanks to
  • ownedbycats - for testing and ideas
Attached Files
File Type: zip Calibre Power Search Plugin.1.7.0.zip (1.62 MB, 82 views)

Last edited by mapozyan; 11-04-2020 at 06:13 AM. Reason: Updated to 1.7.0
mapozyan is offline   Reply With Quote
Old 07-23-2020, 07:01 AM   #2
mapozyan
Enthusiast
mapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheese
 
Posts: 34
Karma: 1002
Join Date: Jul 2020
Device: android
Screenshot of plugin search dialog:
Attached Thumbnails
Click image for larger version

Name:	Screenshot from 2020-11-03 23-21-29.png
Views:	39
Size:	144.1 KB
ID:	183211  

Last edited by mapozyan; 11-04-2020 at 06:17 AM. Reason: Updated to 1.7.0
mapozyan is offline   Reply With Quote
Advert
Old 07-23-2020, 11:33 AM   #3
ownedbycats
Wizard
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
Posts: 1,135
Karma: 345192
Join Date: Oct 2018
Device: Kobo Aura HD
I can't wait for the Windows version.
ownedbycats is offline   Reply With Quote
Old 07-26-2020, 04:16 PM   #4
mapozyan
Enthusiast
mapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheese
 
Posts: 34
Karma: 1002
Join Date: Jul 2020
Device: android
Version 1.1.0 released. Works on Windows!

Last edited by mapozyan; 07-31-2020 at 01:10 PM. Reason: Removed version 1.1.0
mapozyan is offline   Reply With Quote
Old 07-26-2020, 06:06 PM   #5
ownedbycats
Wizard
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
Posts: 1,135
Karma: 345192
Join Date: Oct 2018
Device: Kobo Aura HD
Possible bug in Windows, although I'm not sure if there's anything to be done: when doing a first search, all the ebook-convert console windows pop up in the foreground and steal focus, making the computer pretty much unusable until the search is either complete or cancelled.

Seeing as the initial indexing takes a while (I ended up cancelling when it was nearing an hour and a half, and the bar was only 1/5 full!) it's...a bit of a problem.

Last edited by ownedbycats; 07-26-2020 at 07:21 PM.
ownedbycats is offline   Reply With Quote
Advert
Old 07-27-2020, 03:49 AM   #6
mapozyan
Enthusiast
mapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheese
 
Posts: 34
Karma: 1002
Join Date: Jul 2020
Device: android
Hmm, I'll look into it. Thanks for feedback!
mapozyan is offline   Reply With Quote
Old 07-27-2020, 04:15 AM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 38,208
Karma: 16793319
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@mapozyan You need to set creationflags to CREATE_NO_WINDOW on windows when calling Popen()
kovidgoyal is offline   Reply With Quote
Old 07-27-2020, 04:20 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 38,208
Karma: 16793319
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Sorry typoe, you need DETACHED_PROCESS not CREATE_NO_WINDOW
kovidgoyal is offline   Reply With Quote
Old 07-27-2020, 06:30 AM   #9
mapozyan
Enthusiast
mapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheese
 
Posts: 34
Karma: 1002
Join Date: Jul 2020
Device: android
Fixed issue in version 1.1.1. Please find in attachment.

Thanks for hint, @kovidgoyal!

Just a quick observation on how much time the initial index creation might take.
Processing my own library with ~3200 books and total size ~23GB took me around 5 hours.
My computer is a decent i7 laptop with 4 cores.

Last edited by mapozyan; 07-31-2020 at 01:11 PM. Reason: Removed version 1.1.1
mapozyan is offline   Reply With Quote
Old 07-27-2020, 09:52 AM   #10
ownedbycats
Wizard
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
Posts: 1,135
Karma: 345192
Join Date: Oct 2018
Device: Kobo Aura HD
I installed the new version and grabbed pdftotext. This time my computer is usable while the initial search runs in the background. pdftotext also speeds up the process quite a bit.

Another odd quirk: I have a six-core processor. The readme file indicated that the default number of parallel processes should be set to the number of CPUs, but in mine it was default set to 12. It's possible that was another reason it was running so slow before. It may be some oddity stemming from my particular CPU (AMD Ryzen 2600) though.

Last edited by ownedbycats; 07-27-2020 at 09:56 AM.
ownedbycats is offline   Reply With Quote
Old 07-27-2020, 10:15 AM   #11
mapozyan
Enthusiast
mapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheese
 
Posts: 34
Karma: 1002
Join Date: Jul 2020
Device: android
Some fancy technologies like Hyper-threading might double the real number of cores.
However, setting 'number of parallel processes' option to a higher value should not be disastrous. Your operating system is smart enough to deal with it.
mapozyan is offline   Reply With Quote
Old 07-27-2020, 11:03 AM   #12
ownedbycats
Wizard
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
Posts: 1,135
Karma: 345192
Join Date: Oct 2018
Device: Kobo Aura HD
It's all indexed now!

Another bug: I put a phrase in "quotation marks" to test searching exact phrases and got an error about an unexpected character. Error log:

Spoiler:
Code:
calibre, version 4.21.0
ERROR: Unhandled exception: <b>RequestError</b>:RequestError(400, u'json_parse_exception', u"Unexpected character ('s' (code 115)): was expecting comma to separate Object entries\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 57]")

calibre 4.21 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.18362-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: DeDRM (6, 7, 0) && Open With (1, 5, 12) && Goodreads (1, 4, 0) && View Manager (1, 5, 6) && Manage Series (1, 2, 9) && Reading List (1, 6, 11) && NormComment (0, 0, 2) && Extract ISBN (1, 5, 0) && Job Spy (1, 0, 184) && MultiColumnSearch (1, 0, 82) && Library Codes (1, 0, 46) && Modify ePub (1, 4, 1) && Kindle hi-res covers (0, 4, 0) && Quality Check (1, 10, 1) && Kobo Utilities (2, 11, 4) && KoboTouchExtended (3, 2, 13) && Save To Format (0, 0, 6) && FanFicFare (3, 21, 2) && Embed Comic Metadata (1, 4, 0) && Find Duplicates (1, 7, 2) && Favourites Menu (1, 0, 5) && EpubCheck (0, 2, 3) && Fantastic Fiction (1, 3, 0) && Drop Search Results (1, 0, 10) && Calibre Power Search Plugin (1, 1, 1) && Barnes & Noble (1, 2, 15) && Count Pages (1, 10, 0) && FictionDB (1, 2, 0) && Generate Cover (1, 5, 25) && Kobo Books (1, 8, 0) && Import List (1, 4, 5)
Traceback (most recent call last):
  File "calibre_plugins.caps.main", line 208, in on_search
  File "calibre_plugins.caps.main", line 282, in do_search
  File "calibre_plugins.caps.elasticsearch.client.utils", line 152, in _wrapped
  File "calibre_plugins.caps.elasticsearch.client.__init__", line 1470, in search
  File "calibre_plugins.caps.elasticsearch.transport", line 392, in perform_request
RequestError: RequestError(400, u'json_parse_exception', u"Unexpected character ('s' (code 115)): was expecting comma to separate Object entries\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 57]")
ownedbycats is offline   Reply With Quote
Old 07-27-2020, 11:50 AM   #13
mapozyan
Enthusiast
mapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheesemapozyan can extract oil from cheese
 
Posts: 34
Karma: 1002
Join Date: Jul 2020
Device: android
Thanks for feedback @ownedbycats!

I did not try searching for exact phrase yet, need to figure out how to do it in elasticsearch.

Stay tuned for updates!
mapozyan is offline   Reply With Quote
Old 07-27-2020, 12:58 PM   #14
ownedbycats
Wizard
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
Posts: 1,135
Karma: 345192
Join Date: Oct 2018
Device: Kobo Aura HD
Searching for a compound word that uses a hyphen (e.g. up-to-date) also returns results for the separate words.

Last edited by ownedbycats; 07-27-2020 at 03:34 PM.
ownedbycats is offline   Reply With Quote
Old 07-27-2020, 01:22 PM   #15
DNSB
Bibliophagist
DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.
 
DNSB's Avatar
 
Posts: 8,900
Karma: 41038543
Join Date: Jul 2010
Location: Vancouver
Device: Kobo Forma, Clara HD, Nexus 7 HD, iPad Pro, Tolino epos
Quote:
Originally Posted by ownedbycats View Post
Another odd quirk: I have a six-core processor. The readme file indicated that the default number of parallel processes should be set to the number of CPUs, but in mine it was default set to 12. It's possible that was another reason it was running so slow before. It may be some oddity stemming from my particular CPU (AMD Ryzen 2600) though.
A Ryzen 2600 6 core is listed as 6 core 12 thread so the CPU supports hyperthreading. OTOH, I've found several times where exceeding the number of physical CPU cores slows processes down since hyperthreading does not duplicate the full CPU core.

And my test is still running after 8 hours (~11,500 books).

Edited to copy the edit comment to the body of the message:

Edit: Just finished. at 09:24. About 2 minutes after the message was written.

Last edited by DNSB; 07-27-2020 at 01:50 PM. Reason: Edit: Moved just finished message and time into the body of the message
DNSB is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Search the Internet kiwidude Plugins 333 09-05-2020 06:04 PM
[GUI Plugin] Clipboard Search kiwidude Plugins 17 03-29-2020 04:07 AM
[GUI Plugin] Recoll Full Text Search Satas Plugins 16 08-05-2016 04:54 AM
[GUI Plugin] Full Text Search (SOLR) peterpisljar Plugins 2 08-09-2015 09:16 AM
Make a simple Plugin for Full Text Search using Recoll Satas Development 9 07-20-2013 05:15 PM


All times are GMT -4. The time now is 10:35 PM.


MobileRead.com is a privately owned, operated and funded community.