[GUI Plugin] Power (Full-text) Search

[ownedbycats]

Since a good part of my library is PDFs, using pdftotext sped things up considerably.

I did notice everything else lagging when I used 12 processes. I switched it to six and the lag disappeared.

[mapozyan]

Originally Posted by DNSB

And my test is still running after 8 hours (~11,500 books).

What is the progress after 8 hours? I'm curious if it's comparable to my numbers.

And yes, did you setup pdftotext properly?

[DNSB]

Originally Posted by mapozyan

What is the progress after 8 hours? I'm curious if it's comparable to my numbers.

And yes, did you setup pdftotext properly?

I did setup pdftotext properly. As I mentioned in the edit comment though perhaps I should have put it in the body of the message, the progress finished about 2 minutes after I wrote the message.

My main reason for checking this out was that I use ElasticSearch with Greylog which states that part of it's reason for existence is to work around the shortcomings of ElasticSearch.

[mapozyan]

Originally Posted by DNSB

My main reason for checking this out was that I use ElasticSearch with Greylog which states that part of it's reason for existence is to work around the shortcomings of ElasticSearch.

Elasticsearch is a super advanced search engine. I guess one will hardly notice any shortcomings until they run some web-scale projects on top of it.

That said, I hope it's more than enough for local library management.

[ownedbycats]

My library is about 4000 books and 20 GB, though the bulk of that is image-heavy PDF files (lots of video game strategy guides).

[thiago.eec]

Thanks for the plugin! Awesome idea.

I did not installed pdftotext, but I only have 18 PDFs on my library.
I'm testing it on a library with 1130 books (many with multiple formats: EPUB, AZW3/KFX) and about 3GB.

Info about the initial indexing: It took only 16 minutes to go from 0 to 99%. But now it is stuck at 99% for about 3h45min. My system is an i7 7700HQ (16GB of RAM). My processor has 4 cores (8 threads). Plugin has chosen 8 max parallel process. Now, the strange part: according to Task Manager (Windows), my CPU is only using 20% of its total capacity.

While writing this post, it finished, after 4h05min. Now it searches instantly! Nice!


------ My first impressions and questions ------


1) Question: When you have multiple formats for one book, does it lookup all the formats or just one?

2) Question: On caps.json, it only shows EPUB, MOBI, PDF and TXTs files. According to this, and other tests I have done, it does not index AZW3/KFX files. Is this correct?

3) Question: How the index works for new additions? Are the new files automatically indexed when I run ElasticSearch?

4) Suggestion: It would be really important to have more options for search. Right now, it searches word by word. So, I can't look for phrases or compound words (Ex: coffee table. It will search for books with "coffee" OR "table"). Also, accented characters are distinguished from non-accented.

5) Info: According to ElasticSearch Reference, to have more options for search, you would need to change your query from "match" to "query_string". This would allow operators, wild cards and regular expressions. P.S.: "match" queries can use operators too, but you would have to code that.

6) Info: The ZIP file attached to first post has another ZIP inside (with the full plugin).

[DNSB]

Originally Posted by thiago.eec

------ My first impressions and questions ------


1) Question: When you have multiple formats for one book, does it lookup all the formats or just one?

On my setup, it seems to have searched both epub and pdf format files. I don't keep other formats in my library.

Originally Posted by thiago.eec

3) Question: How the index works for new additions? Are the new files automatically indexed when I run ElasticSearch?

When I added a few files, the search seems to have imported them and searched them.

Originally Posted by thiago.eec

4) Suggestion: It would be really important to have more options for search. Right now, it searches word by word. So, I can't look for phrases or compound words (Ex: coffee table. It will search for books with "coffee" OR "table"). Also, accented characters are distinguished from non-accented.

5) Info: According to ElasticSearch Reference, to have more options for search, you would need to change your query from "match" to "query_string". This would allow operators, wild cards and regular expressions. P.S.: "match" queries can use operators too, but you would have to code that.

6) Info: The ZIP file attached to first post has another ZIP inside (with the full plugin).

A wild card or regex search capability would be great!

I used the 3rd zip file from message #9 in this thread. Note that my setup is on Windows x64.

I also restarted the full search after deleting the old setup and moving my computer related ebooks out of my calibre library. I also realized that I had not pointed to pdftotext properly and corrected that. Was a heck of a lot faster with those mostly oversized pdf files removed. 2 hours down to 5 minutes.

[mapozyan]

Thanks thiago.eec for feedback!

Originally Posted by thiago.eec

But now it is stuck at 99% for about 3h45min.

You might probably have encountered one of "bad" PDF files which is taking ages to process without pdftotext.

Originally Posted by thiago.eec

1) Question: When you have multiple formats for one book, does it lookup all the formats or just one?

It will lookup all formats.

Originally Posted by thiago.eec

2) Question: On caps.json, it only shows EPUB, MOBI, PDF and TXTs files. According to this, and other tests I have done, it does not index AZW3/KFX files. Is this correct?

Correct. Now plugin supports following formats: CHM, CBZ, FB2, PDB, DJVU, EPUB, MOBI, DOCX, PDF, TXT, RTF. I will try to add AZW3/KFX support in next release.

Originally Posted by thiago.eec

3) Question: How the index works for new additions? Are the new files automatically indexed when I run ElasticSearch?

Yes, the plugin will detect all files that were added to the library since the last search and will add them to index.

Originally Posted by thiago.eec

4) Suggestion: It would be really important to have more options for search. Right now, it searches word by word. So, I can't look for phrases or compound words (Ex: coffee table. It will search for books with "coffee" OR "table"). Also, accented characters are distinguished from non-accented.

Thanks, I am thinking on it. Will try to address in a next release.

Originally Posted by thiago.eec

6) Info: The ZIP file attached to first post has another ZIP inside (with the full plugin).

Haha indeed! Thanks for noticing!

[mapozyan]

Originally Posted by mapozyan

Yes, the plugin will detect all files that were added to the library since the last search and will add them to index.

Sorry, I was not specific enough.

The new files won't be automatically indexed instantly when you add new books. But they will be indexed once you run Power Search again.

[DNSB]

Originally Posted by mapozyan

Sorry, I was not specific enough.

The new files won't be automatically indexed instantly when you add new books. But they will be indexed once you run Power Search again.

Matches with what I saw. The search with the 3 added books went 33%, 66%, done.

[ownedbycats]

Originally Posted by mapozyan

Sorry, I was not specific enough.

The new files won't be automatically indexed instantly when you add new books. But they will be indexed once you run Power Search again.

While testing, I also had FanFicFare update some of my fanfics and then did a search for random words that appeared only in the newest chapters. It did re-index the ePubs that had changed.

[thiago.eec]

Originally Posted by mapozyan

You might probably have encountered one of "bad" PDF files which is taking ages to process without pdftotext.

The time consuming files are image PDFs and Fixed Layout books. But now I installed pdftotext and it is way faster!

Originally Posted by mapozyan

Correct. Now plugin supports following formats: CHM, CBZ, FB2, PDB, DJVU, EPUB, MOBI, DOCX, PDF, TXT, RTF. I will try to add AZW3/KFX support in next release.

I took a deeper look at the plugin. All files are converted to TXT, then passed to ElasticSearch. So, the allowed formats can be any of those handled by calibre. I changed the supported list to include DOC, AZW3, AZW4 and KFX and they all got indexed.

Originally Posted by mapozyan

Thanks, I am thinking on it. Will try to address in a next release.

I did a simple test here, and changing query from "match" to "match_phrase" did the job allowing phrases and compound words. Using "query_string" isn't that easy, though.

[mapozyan]

Originally Posted by thiago.eec

I took a deeper look at the plugin. All files are converted to TXT, then passed to ElasticSearch. So, the allowed formats can be any of those handled by calibre. I changed the supported list to include DOC, AZW3, AZW4 and KFX and they all got indexed.

Good to know, thanks!

I just decided to be conservative in first releases and support only those book formats that I could test well enough. I will extend this list in next release once I make sure it works well.

Originally Posted by thiago.eec

I did a simple test here, and changing query from "match" to "match_phrase" did the job allowing phrases and compound words. Using "query_string" isn't that easy, though.

My final goal is to implement search synax and semantic similar to Google search. After all, when doing search in Google we aren't using regular expressions, are we?

This however means that I would need to find a way of sorting results according to relevance. Don't know how easy is it to do in Calibre, but generally it seems to me a right way to go.

[mapozyan]

Version 1.2.0 released.

Contains following usability improvements:

• Phrase search when using quotes (example query: "Mark Twain" poetry)
• Details section showing book conversion status

Adds support for DOC, AZW3, KFX file formats.

[mapozyan]

Originally Posted by ownedbycats

Searching for a compound word that uses a hyphen (e.g. up-to-date) also returns results for the separate words.

I tried to search for "up-to-date" as a phrase. It still doesn't work correctly, so "up-to-date" will bring the same results as "up to date". Still, its much more useful than searching for individual words.