[GUI Plugin] Power (Full-text) Search

[mapozyan]

Originally Posted by retval

For statistical purposes I use an i5-4460, 3.20GHz. 16gb ram. At 2 hours the windows 10 manager indicated a usage of 30% CPU, 50% Ram and 40° of temperature on the ssd. It could continue to work normally on the pc.
After 8 hours it occupied 100% of the CPU, 94% of the ram and 43° of heat. It could no longer work, as it consumed all the resources. Of the 4 cores of the processor only 2 were used and 1.5 gb of ram for ElacticSearch. Is this behavior normal?

CPU usage might fluctuate based on what files (and file types) are currently being converted. 94% of RAM sounds too much, but 36K of books is also a lot!

Originally Posted by retval

The strange thing is that putting few books, about 20, and doing a search only in those selected books it searches the whole library, which is very slow. When cancelling it shows the results in the selected books. The logical thing is that the search on selected books does not use the whole library. I don't understand this.

Fair point, I will take a look, thanks.

Originally Posted by retval

One suggestion is to have PowerSearch display a result that indicates how many times the search term is found in each book. This could perhaps be displayed in an additional column. Then we could see the books in which the search finds better results, at least quantitatively..

This might be not so easy to implement, but thanks for suggestion, I will think about it.

[DNSB]

Just to correct an error in an earlier message, the last version of ElasticSearch that has an MSI installer is ElasticSearch 7.16.3 MSI

[DNSB]

For what it's worth, the only way I can get elasticsearch.exe to manually run on my computer requires forcing 'run as administrator'. Simply trying to run it from a normal user command line gives a mass of errors relating to permissions on various directories.

[partnoonein]

Does power search work with Calibre beta 5.99.8
(https://download.calibre-ebook.com/betas/)
I installed it in Calibre beta 5.99.8 and do not even see the power search icon (customizing toolbar option does not show the plugin icon)

[partnoonein]

Calibre beta 5.99.8

Furthermore I cannot configure it in Calibre beta 5.99.8 (see attached image).
Thank you

[mapozyan]

Hi,

Beta version is not supported yet, as it requires major update from Qt5 to Qt6.

[partnoonein]

Thank you so much for replying. May I ask you when you are planning to release the compatible version since it appears that Calibre 6 will soon be released. Thanks again.

Originally Posted by mapozyan

Hi,

Beta version is not supported yet, as it requires major update from Qt5 to Qt6.

[mapozyan]

I'll do my best to release it ASAP.

[partnoonein]

That would be very kind of you. Thank you in advance. You have made an excellent plugin and it works wonders with the collection. An invaluable tool for research.

Originally Posted by mapozyan

I'll do my best to release it ASAP.

[mapozyan]

Version 2.2.0 released! What's new in this release:
* Added support of upcoming Calibre 6 (technically its Calibre Beta 5.99.8 now)

New plugin stays compatible with older versions of Calibre.

I attached new version to the second post in this thread.

[retval]

Greetings, I have not had time to write, but I am resuming testing with PowerSearch.

If you remember the process "OpenJDK Platform binary" running as a service on windows 10 located in C:\Program Files\Elastic\Elasticsearch\7.16.1\jdk\bin\java.ex e" was not closing. I followed @mapozian's suggestion and changed the "OpenJDK Platform binary" service from Automatic to Manual. However, PowerSearch crashes and indicates that it cannot launch Elasticsearch. I had to leave it back to automatic to get it to work.
There are also 2 more processes running in parallel C:\Program Files\Elastic\Elasticsearch\7.16.1\bin\elactisearch.exe and C:\Program Files\Elastic\Elasticsearch\7.16.1\modules\x-pack-ml\platform\windows-x86_64\bin\controller.exe

It also has erratic behavior when searching for words in the recently added and indexed books, as it returns 0 results. There are about 80 books added today...could it be that it has not finished indexing them yet? I have indexed them several times. I understand that PowerSearch has to be instructed to index the new books and that it doesn't do it automatically or in the background.

I know the problem is mine for having too many files. When I search in groups of 10 files it takes about a minute to return the result. The first 30 seconds it freezes with the indication "not working", then it starts searching.

Is there something wrong and would it be good to reinstall powersearch or elactisearch? I like the concept of Powersearch so much that I am willing to keep trying to get it to work on my pc.

I understand that my suggestion to add the information of how many times the plug-in found a word in each text to know which book is better is difficult. However, many users use Calibre as a library manager rather than a book converter. In that it is great. We can manage books by subject, keywords, etc., which is very important in research but if, for example, I have a result of 50 books I can't discriminate which one will be best. PowerSearch can refine that search, since it is not the same a certain word in a footnote citation than several hundred times in a book or a paper. It would be a very significant leap to be able to do that.

Sorry for the length of this, but I am very excited about the potential of PowerSearch.
Thank you very much.

[mapozyan]

Hi retval and thanks for your feedback, I will try to take closer look ASAP.

[jonzim]

having 10s of thousands of books in my library, i find the built in calibre search ultra slow.

So I did something similar to this - built my own separate app to export the calibre metadata into elasticsearch so i could have faster searches and instantly find duplicates.... metadata import only takes seconds.

(it's actually a calibre importer app - i select a folder of books i want to import into calibre, and as i highlight a book from the filesystem to review and import, it finds whether or not that book is already in calibre)

but mine was a meta-data only solution - this goes beyond that, and is integrated into calibre itself

[retval]

Originally Posted by jonzim

having 10s of thousands of books in my library, i find the built in calibre search ultra slow.

Hi, in case it helps.
I have almost 37,000 files in Calibre and the built-in search takes between 4 to 6 seconds. Both Windows and My Documents (where the Calibre library is) are on a nvme SSD. I have 16gb ram and an I5. Those searches do not crash my pc.
When I had the library on a HDD it was slower, but never more than a minute. When it got slower a disk defrag helped. Maybe I should reinstall calibre to get it to read the library again.

[partnoonein]

I am using the beta version for Calibre 5.99.x that you just released. Thank you. Two questions.
1. Does the plugin need to reindex when migrating from 5.38.x to 5.99.x? I had a library in 5.38.x but the plugin seems to reindex everything.
2. Where are the elastic search indices stored?
Thanks again.