FTS - Exporting the search

[Comfy.n]

As we discussed incidentally in other thread (link to Kovid's reply), I suggested it would be nice to have the ability to export the FTS search. BTW, I don't mean exporting the title results related to some FTS query, but the actual search, including the results snippets.

I had never used Calibre CLI, but the recent addition of FTS to calibredb has been quite useful for that purpose, so I gave it a try.

Using the commands below, I was able to generate the attached files, which contain the results snippets:

calibredb fts_search --include-snippets --output-format=text NEAR("biphasic" "sleep",30) > fts-export.txt
calibredb fts_search --include-snippets --output-format=json NEAR("biphasic" "sleep",30) > fts-export.json

Files generated contain special characters that I had cleaned up by using Notepad++ and replacing them as following:

•  for >
  
•  for <

Although using the CLI requires some practice for average users, the exported search was useful to me, even having to do that cleaning/beautifying. And the fact that one needs to close any Calibre instance before using calibredb is a little bothersome...

So I present some questions:

- if there's a better way to have the FTS search neatly exported, perhaps through some advanced batch scripting?

- is there some way to call calibredb on a read-only fashion, so that Calibre GUI instance can be kept open?

- would it be possible, in the future, to obtain the exported search in the form of a catalogue, as BR suggested (but including the results snippets)? Or maybe some other form that could generate a file that can be "read like a book"... something like that.

[kovidgoyal]

use --match-start-marker "" --match-end-marker "" to remove the special characters.

run the calibre content server then calibredb can connect to it.

[Comfy.n]

Originally Posted by kovidgoyal

use --match-start-marker "" --match-end-marker "" to remove the special characters.

run the calibre content server then calibredb can connect to it.

Ah, that's cool!

I've used:

calibredb fts_search  --with-library http://localhost:8080/#calibre_library --match-start-marker "" --match-end-marker "" --include-snippets --output-format=text NEAR("biphasic" "sleep",30) > fts-export2.txt

Once I figured out how to reference the content server address, that went smoothly.



Still I miss the covers, which could be present in some way if the catalogue generation feature for the marked books in the GUI would allow including results snippets. That would be comfier

[Comfy.n]

I have assembled a Windows batch script that I use sometimes to export the FTS results snippets to a file.
It will generate txt, rtf, epub and pdf files. RTF is pretty good for my use cases.
It's a "beta" version, but seems to work in most cases. It doesn't like the NEAR FTS operator.

Here's what it does:

This batch script is designed to perform a full-text search on a Calibre library and convert the search results into different file formats like RTF, PDF, and EPUB. Let's go through the code step-by-step and explain what it does for the user:
1. The script sets the title and echoes a message about the program to the user.
2. It sets the character encoding to UTF-8 (code page 65001) to support special characters and emojis.
3. The user is prompted to enter a search query. The query can contain one or more words.
4. The user is asked whether they want to enclose the query in quotes. This option affects the search behavior in Calibre.
5. The script then displays the resulting query to the user.
6. The user is prompted to refine the search further, using additional criteria (e.g., tag:fiction). If provided, this criteria is stored in the "refine" variable.
7. The script prepares the query by removing any double quotes from it and sets the "fn" variable to the query.
8. It generates a timestamp ("dt") that will be used in the filenames of the output files.
9. The script checks if a folder named "FTS" exists in the user's profile directory. If not, it creates the folder.
10. The script informs the user that the FTS (Full-Text Search) process has started.
11. The script then uses the "calibredb" command-line tool to perform the full-text search on the Calibre library.
12. The search results are redirected to a temporary text file named "tmp.txt" in the "FTS" folder.
13. The script then begins the conversion process of the search results using the "ebook-convert" command-line tool from Calibre.
14. The "ebook-convert" tool is used to create an RTF, PDF, and EPUB file from the temporary text file. Various options for formatting and styling the output are provided.
15. The temporary files are renamed based on the query and timestamp, and they are moved to the "FTS" folder.
16. The script generates a log file named "log{timestamp}.txt" that records the execution status of the script.
17. The script displays a message to the user indicating that the script execution is completed and waits for user input (using the "pause" command).
Overall, this batch script enables users to search their Calibre library using a full-text query, and then convert the search results into different formats like RTF, PDF, and EPUB for further use. The converted files are saved in a folder named "FTS" in the user's profile directory. Additionally, a log file is created to record the execution status of the script. The script provides options for refining the search criteria and allows the user to choose whether to include the query in quotes for the search.

In the attached zip there are two versions, with and without user authentication to the Content Server. Hope this turns out to be useful to someone.