[GUI Plugin] Power (Full-text) Search

[DNSB]

One oddity. When I installed 1.2.0 and did a search in my main library, it added a couple of epubs which were there when the original install was done.

[mapozyan]

Originally Posted by DNSB

One oddity. When I installed 1.2.0 and did a search in my main library, it added a couple of epubs which were there when the original install was done.

This can happen when plugin detects that book modification time was changed. Not sure though why and when it might change, need to experiment a bit.

[DNSB]

Originally Posted by mapozyan

This can happen when plugin detects that book modification time was changed. Not sure though why and when it might change, need to experiment a bit.

I wouldn't have been surprised if it happened in my Intake library but my Main library doesn't get many changes. One of the book was an epub version of a dictionary which has never been changed.

[thiago.eec]

Originally Posted by mapozyan

Version 1.2.0 released.

Contains following usability improvements:

• Phrase search when using quotes (example query: "Mark Twain" poetry)
• Details section showing book conversion status

Adds support for DOC, AZW3, KFX file formats.

Nice work. Now we can easily spot when a file chokes on conversion (it will stay there on the list "forever").

[mapozyan]

Version 1.3.0 released.

Contains following usability improvements:

• Allows complex search queries (see topic start message for more info)
• Fixes bug with deleting book from search index

Many thanks to thiago.eec for his contribution in this release!

[ownedbycats]

I ran into an issue: I have some large image-based CBZ files that don't have any text (empty output file if I convert manually). It takes a few hours for Power Search to index them (note that even with one thread and calibre's job priority set to low, it still manages to eat up my system resources and make my computer unusable for the duration), and it tries to re-index them every time.

Any suggestions?

EDIT: If there was an option to exclude certain books (or even exclude certain formats entirely, I don't mind forgoing the ability to search my other CBZ files) that might help.

[mapozyan]

Hi ownedbycats,

Sure, it will be a great feature. I will try to add it in near future.

And thanks for your feedback, helpful as usual!

[JurriaanK]

I recently started de-duplicating my library with a newly discovered tool that compares 2 textfiles A, Bagainst each other and gives a percentage of text A occuring in text B.

The Debian package is called similarity-tester, the software was written in 1989 and lives here: https://dickgrune.com/Programs/similarity_tester/

Since I don't regularly use Calibre and only nibble at this forum very occasionally, I write this here, because your plugin seems to me to have nearly all aspects available to use this:

- convert books to text
- run external program
- do something with the result

and I've found no other mention of sim_text in combination with Calibre.

A couple of points I found when using it:

- it takes time to run. 3000 files on an Intel J1900 use about 60 seconds and there is no provision for a progress indicator. Can be added relatively simple, of course. There's three main loops: reading files, hashing files & comparing hashes.
- it runs on a single core. If you split the filelist and run permutations of the split sections on multiple cores, it runs faster - if you have enough memory.
- it takes memory to run also. 3000 files use about 2 GiB of memory.
- some patches to make it compile cleanly exists in Debian's bug tracker.
- the best way to run it is to feed it a list of files (-i parameter), then parse the output and if something is found, run the comparison for those single files in reverse (since if A occurs for 80% in B, maybe B is the 'extended edition' with a short story added, or something like that).

So, maybe someone can use this, I notice that detecting similar books is a regularly occuring question in Calibre, and this is a foolproof method.

[Uncle Robin]

"pdftotext" is listed as "optional", but attempting to configure Power Search without it brings up path errors? I can't be bothered with pdftotext as I have no PDFs in my Calibre library.

EDIT Never mind, pebkac as usual. I started it without opening the Options dialog, all seems VERY well. Thank you!

[mapozyan]

Version 1.4.0 released.

Now user can select supported file formats in options dialog.

Thanks to ownedbycats for feature suggestion.

[ownedbycats]

I installed it. Excluding CBZ files fixed my issue quite nicely. Thank you.

I did run into another issue though: for some reason it stopped searching the older books, only results are the ones added/updated recently. I reinstalled ElasticSearch while trying to figure out the CBZ issue and I think it broke the index. Is there a way to reset it?

EDIT: I removed the plugin and reinstalled it and that didn't work.

[mapozyan]

I'm not sure, it might happen if you are switching your library path. Otherwise it should be working well...

[ownedbycats]

Moving caps.json from the plugins folder seems to reset the Power Search settings. It's re-indexing all the files.

I accidentally deleted the ES data folder when I reinstalled ElasticSearch (I use Geek Uninstaller) but since the caps.json file listed the files as already-indexed, it didn't re-index them.

Another possible bug I noticed: When canceling partway through indexing (e.g. to adjust the number of threads), although the files are indexed (as can be confirmed by monitoring the elasticsearch programdata folder), the datetimes don't get added to caps.json so Power Search ends up re-indexing them unnecessarily.

[mapozyan]

Originally Posted by ownedbycats

Moving caps.json from the plugins folder seems to reset the Power Search settings. It's re-indexing all the files.

I accidentally deleted the ES data folder when I reinstalled ElasticSearch (I use Geek Uninstaller) but since the caps.json file listed the files as already-indexed, it didn't re-index them

This is a very good point! I maybe need to add Reindex feature for such cases.

Originally Posted by ownedbycats

Another possible bug I noticed: When canceling partway through indexing (e.g. to adjust the number of threads), although the files are indexed (as can be confirmed by monitoring the elasticsearch programdata folder), the datetimes don't get added to caps.json so Power Search ends up re-indexing them unnecessarily.

Also correct. Actually I thought that if user for some reason cancelled indexing process they would probably expect that the plugin will revert to its original state. But as I can see now I was wrong.

[Uncle Robin]

Do we need to uninstall 1.3.0 before installing 1.4.0? If so, how is that done, since Power Search doesn't show up in my list of installed plugins?