[GUI Plugin] Power (Full-text) Search

[Cicatriz]

It looks like the search request is not finding the index.
It creates an index with the ID: calibre-library-d6caee6b-5aa1-4295-8bb3-93d8b52dbc9a
However the search is only looking for an index of library: path: /library/_search, params: {index=library}

[mapozyan]

Ah this might be the case if you downgraded and then upgraded version of Power Search. Try to uninstall the plugin, delete the file %APPDATA\calibre\plugins\caps.json (or move it to somewhere) and install the plugin again. For Linux path to the config json file is ~/.config/calibre/plugins/caps.json.

[Cicatriz]

Thanks for the help. I couldn't get too far - I tried doing a bit of debugging, but I'm not super experienced with python and never touched calibre plugins before so gave up.

I went back to the elastic search 7 stream and manually installed. Everything is good now. I hope.

[cgrapski]

I don't know what I am doing wrong. When I try to run this - I get an error - that ElasticSearch timed out waiting to start.

[ownedbycats]

1. Make sure the right path is in the plugin config. If you recently updated it changes the version number.

2. If that doesn't work, open elasticsearch.exe ({InstallDir}\7.16.1\bin\elasticsearch.exe) and try again.

[retval]

Greetings to all.
Today I discovered this add-on and I am very excited about what I can do with it.
However, I cannot install Elasticsearch. On your website there is no msi executable for windows 10, there is only a zip with the elasticsearch-8.1.0-windows-x86_64 folder. I tried to use the exe's from the Bin subfolder with no result. Elacticsearch.bat file opens a cmd console but nothing else. It also does not connect to the server, although the win10 firewall asked me for permission and I gave it.
The instructions on the Elacticsearch web site are very confusing to me, as they refer to the use of code (I am a philosopher, not a computer guy).
Placing the path in the "options" section of PowerSearch does not work either.
I would appreciate any help, as this add-on would be very useful in my daily work.
I use Calibre 5.37. Win10 and my pc is an i5-4460, 3.20GHz. 16 gb ram.

[DNSB]

Originally Posted by retval

Greetings to all.
Today I discovered this add-on and I am very excited about what I can do with it.
However, I cannot install Elasticsearch. On your website there is no msi executable for windows 10, there is only a zip with the elasticsearch-8.1.0-windows-x86_64 folder. I tried to use the exe's from the Bin subfolder with no result. Elacticsearch.bat file opens a cmd console but nothing else. It also does not connect to the server, although the win10 firewall asked me for permission and I gave it.
The instructions on the Elacticsearch web site are very confusing to me, as they refer to the use of code (I am a philosopher, not a computer guy).
Placing the path in the "options" section of PowerSearch does not work either.
I would appreciate any help, as this add-on would be very useful in my daily work.
I use Calibre 5.37. Win10 and my pc is an i5-4460, 3.20GHz. 16 gb ram.

If you want to install ElasticSearch 7.16.1 using an MSI installer, you can download it from:

https://artifacts.elastic.co/downloa...rch-7.16.1.msi

As far as I know, that was the last version that came with an MSI installer since part of the breaking news for v7.17 said no more MSI. See ElasticSearch Breaking Changes for the official announcement.

Edit: Actually use the 7.16.3 installer from ElasticSearch_7.16.3

[Uncle Robin]

Originally Posted by DNSB

If you want to install ElasticSearch 7.16.1 using an MSI installer, you can download it from:

https://artifacts.elastic.co/downloa...rch-7.16.1.msi

As far as I know, that was the last version that came with an MSI installer since part of the breaking news for v7.17 said no more MSI. See ElasticSearch Breaking Changes for the official announcement.

That's very sad news about the end of the MSI, especially after reading the page on how to update to 7.17 - an enormous palaver for something that I only use for this plugin. I guess I'll grab the most recent MSI and hope that that version lasts long enough.

[retval]

Thank you very much.
Now I was able to install ElacticSearch with the older MSI version. This is the path I put in the PowerSearch options: C:\ElasticElasticSearch.exe

I have some doubts about the operation:
1.- I started to do the first indexing of my only library, but Pdftotext starts to open files in cascade which saturates the screen, so I had to stop it. The path in options that I put is: C:\Program Files\Glyph & Cog\XpdfReader-win64\xpdf.exe
Is it ok?

2.- I have a very large library of more than 36,000 books in pdf, epub, mobi, cbz, cbr and doc. (221 gb) on a ssd for faster access. I saw in the thread that Power Search goes up to 10,000, but it was not clear to me if it is a limit on initial indexing or is it a limit on search results.

3.- Seeing what other users have taken I think my library would be indexed in several days, I have no problem with that. But is there any way to index the first time in parts so as not to saturate the computer?

4.- I left 1.5 gb of ram in ElacticSearch according to what I read before. In Windows 10 Task Manager I saw that Calibre, using Power Search, consumes 30% of the memory and has a high power consumption.
I use Calibre 5.37. Win10 and my pc is an Intel i5-4460 (4 cores), 3.20GHz. 16 gb ram. What other settings can be made to make the computer run smoothly while doing the first indexing?

5.- Finally, I have my Calibre library duplicated on another computer.
Is there any way to get the initial indexing to the other pc or do I have to do it again on the other computer?
Thank you very much for your answers.

[thiago.eec]

Originally Posted by retval

1.- I started to do the first indexing of my only library, but Pdftotext starts to open files in cascade which saturates the screen, so I had to stop it. The path in options that I put is: C:\Program Files\Glyph & Cog\XpdfReader-win64\xpdf.exe
Is it ok?

Are you sure this is Pdftotext? Looks like it is another tool.

Originally Posted by retval

2.- I have a very large library of more than 36,000 books in pdf, epub, mobi, cbz, cbr and doc. (221 gb) on a ssd for faster access. I saw in the thread that Power Search goes up to 10,000, but it was not clear to me if it is a limit on initial indexing or is it a limit on search results.

The limit is for search results.

Originally Posted by retval

3.- Seeing what other users have taken I think my library would be indexed in several days, I have no problem with that. But is there any way to index the first time in parts so as not to saturate the computer?

Based on my numbers (30min to index 1,400 books), I'd say you only need 13/14 hours. Also, my system runs on an SSD, but my library is on a HDD. Of course, that's using full power (7 threads for a 8 virtual cores processor).

Originally Posted by retval

4.- I left 1.5 gb of ram in ElacticSearch according to what I read before. In Windows 10 Task Manager I saw that Calibre, using Power Search, consumes 30% of the memory and has a high power consumption.
I use Calibre 5.37. Win10 and my pc is an Intel i5-4460 (4 cores), 3.20GHz. 16 gb ram. What other settings can be made to make the computer run smoothly while doing the first indexing?

You can reduce the "Number of parallel processes for text extraction". It defaults to N-1, where N = CPU Count (virtual cores). Set it to smaller number, maybe N/2 or N/4.

Originally Posted by retval

5.- Finally, I have my Calibre library duplicated on another computer.
Is there any way to get the initial indexing to the other pc or do I have to do it again on the other computer?
Thank you very much for your answers.

Don't know.

[DNSB]

Originally Posted by retval

Thank you very much.
Now I was able to install ElacticSearch with the older MSI version. This is the path I put in the PowerSearch options: C:\ElasticElasticSearch.exe

I have some doubts about the operation:
1.- I started to do the first indexing of my only library, but Pdftotext starts to open files in cascade which saturates the screen, so I had to stop it. The path in options that I put is: C:\Program Files\Glyph & Cog\XpdfReader-win64\xpdf.exe
Is it ok?

Thank you very much for your answers.

You need to use pdftotext.exe not xpdf.exe which is a xpdfReader. My path to the executable is: C:\xpdf-tools-win-4.03\bin64\pdftotext.exe. You need to download the command lines tools package not the XpdfReader package.

Basically, you were opening every PDF file to view it so you would see your screen getting very, very busy.

[mapozyan]

Originally Posted by retval

5.- Finally, I have my Calibre library duplicated on another computer.
Is there any way to get the initial indexing to the other pc or do I have to do it again on the other computer? .

Good question. I will check the options. (From what I know ElasticSearch is having a feature called snapshots, so need to see how to use it properly)

[retval]

Greetings to all. Thank you for your replies.
I was able to install the programs correctly.
Elasticsearch was left with this path in PowerSearch
C:\Program Files\Elasticsearch\7.16.1\bin\elacticsearch.exe
Pdftotext was left here:
C:\Program Files\xpdf-tools-win-4.03\bin64\pdftotext.exe

As I indicated before I have more than 36000 books. That is why the first indexing took more than 12 hours (I don't know precisely because it finished in the early hours of the morning).
For statistical purposes I use an i5-4460, 3.20GHz. 16gb ram. At 2 hours the windows 10 manager indicated a usage of 30% CPU, 50% Ram and 40° of temperature on the ssd. It could continue to work normally on the pc.
After 8 hours it occupied 100% of the CPU, 94% of the ram and 43° of heat. It could no longer work, as it consumed all the resources. Of the 4 cores of the processor only 2 were used and 1.5 gb of ram for ElacticSearch. Is this behavior normal?

After the first indexing I did some searches and it reached 20% after an hour. I think it may be because of the large number of books. I tried it several times and the result was similar. When I cancelled the search it showed me the results I had already achieved.
The strange thing is that putting few books, about 20, and doing a search only in those selected books it searches the whole library, which is very slow. When cancelling it shows the results in the selected books. The logical thing is that the search on selected books does not use the whole library. I don't understand this.

Another strange thing is that there is always running an ElacticSearch process called "OpenJDK Platform binary" located in C:\Program Files\ElasticElasticsearch\7.16.1\jdk\bin\java.exe which takes up about 250mb of ram even though Calibre is closed. In fact it starts with windows 10.
How can that process be closed automatically so as not to waste resources?

One suggestion is to have PowerSearch display a result that indicates how many times the search term is found in each book. This could perhaps be displayed in an additional column. Then we could see the books in which the search finds better results, at least quantitatively..

Thanks again.

[DNSB]

Originally Posted by retval

Another strange thing is that there is always running an ElacticSearch process called "OpenJDK Platform binary" located in C:\Program Files\ElasticElasticsearch\7.16.1\jdk\bin\java.exe which takes up about 250mb of ram even though Calibre is closed. In fact it starts with windows 10.

How can that process be closed automatically so as not to waste resources?

It seems that you have installed ElasticSearch as a service under Windows. To free up that memory, you could stop the ElasticSearch service. That will mean having to make sure that you manually start ElasticSearch when you want to use it.

[mapozyan]

Originally Posted by DNSB

It seems that you have installed ElasticSearch as a service under Windows. To free up that memory, you could stop the ElasticSearch service.

IIRC when installing ElasticSearch you can also choose if you want to run it automatically in background. Later on, you can always change the startup type to "manual" if you wish. See screenshot below.

Originally Posted by DNSB

That will mean having to make sure that you manually start ElasticSearch when you want to use it.

No need to worry about it. PowerSearch will start ElasticSearch if needed and will also stop it when you close Calibre!