Search and Replace from a List

[moldy]

In Editor/Saved Searches I am trying to import a json file to search for multiple items and replace from a list. For example:

{
  "searches": [
    {
      "case_sensitive": false,
      "dot_all": false,
      "find": ["John", "Paul", "George", "Ringo"],
      "mode": "regex",
      "name": "B-S",
      "replace": ["Mick", "Kieth", "Ronnie", "Charlie"]
    }
  ],
  "version": 1
}

However when I run the search it gives the error:
calibre, version 7.6.0

ERROR: Unhandled exception: <b>TypeError</b>:unhashable type: 'list'

calibre 7.6  embedded-python: True
Windows-10-10.0.22631-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22631')
Python 3.11.5
Windows: ('10', '10.0.22631', 'SP0', 'Multiprocessor Free')
Interface language: None
EXE path: C:\Program Files\Calibre2\calibre-parallel.exe
Successfully initialized third party plugins: Count Pages (1, 13, 5)
Traceback (most recent call last):
  File "calibre\gui2\tweak_book\boss.py", line 1111, in run_saved_searches
  File "calibre\gui2\tweak_book\search.py", line 1411, in run_search
  File "calibre\gui2\tweak_book\search.py", line 1411, in <listcomp>
  File "calibre\gui2\tweak_book\search.py", line 1312, in get_search_regex
  File "calibre\ebooks\conversion\search_replace.py", line 14, in compile_regular_expression
TypeError: unhashable type: 'list'

I have done something similar in the past but I lost all my Calibre stuff in a PC crash and I can't remember how I did it.
Can someone point me in the right direction please?

[lomkiri]

Originally Posted by moldy

However when I run the search it gives the error:
TypeError: unhashable type: 'list'

You get this error because you pass a list where it expects a string.
I don't know how to do this in mode regex, but in mode function I'll do it this way :

      [...]
      "find": "(John)|(Paul)|(George)|(Ringo)",
      "mode": "function",
      [...]

and the function (edit: see next msg for a better code) :

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    repl = ["Mick", "Kieth", "Ronnie", "Charlie"]
    len_match = len(match)

    for i in range(1, len_match):
        if match[i]:
            return repl[i-1] if i <= len(repl) else match[0]

Explanation :
If we find "Paul", it will be in the 2nd capturing group (i.e. match[2], the other groups will be None), so we return the 2nd name of the list "repl", i.e. repl[1].

If "repl" is, by mistake, smaller than the number of the capturing groups, we avoid an exception not doing anything (returns match[0])

If you're sure of the exact correspondence of the lists in and out, you can just put
return repl[i-1]
it will be quicker if there are lots of occurrences

[lomkiri]

I found a better way for the function, with the property lastindex :

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    repl = ["Mick", "Kieth", "Ronnie", "Charlie"]
    idx = match.lastindex
    return repl[idx -1]
    
    # or, if you prefer safer : return repl[idx -1] if idx <= len(repl) else match[0]

Another way, with a dict:

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    equiv = {"John": "Mick", "Paul": "Kieth", "George": "Ronnie", "Ringo": "Charlie"}
    m = match[0]
    return equiv.get(m, m)    # if "m" is not in the dict, leave the text untouched

In both cases, it is possible to load the list or the dict in the function from a json file instead of defining it as a variable, maybe it was what you were thinking about? In that case, the function doesn't need to be modified at each use, only the regex and the json file.

[moldy]

Thanks for your help Lomkiri I'll give it a try and let you know how it goes.

[lomkiri]

Ok.
Ask if you need an example of how to load the list or the dict from a json file.

[theducks]

This is the Editor you are using (folk post other ways to Edit things here)
KISS
A Group of saved searches. 1 for search each word.
Then just run the group from the saved search pane.

[lomkiri]

I understood that moldy gave just an example, and that the list could be wide.
The solution I gave permits a list of any number of elements, in the case of 50 elements, for ex. it would be difficult to make the replacements one by one.

And the solution is relatively simple, the function has only 3 lines (one line is optional)

[moldy]

Lomkiri is correct. I actually have a dynamic list of over 200 items and their alternatives in a text file. I’m looking for a solution where I can copy and paste my lists into a find and replace function .
I don’t want the work done for me Theducks but I do appreciate being put on the right road by someone more knowledgeable and experienced.
I am away from home so haven’t been able to try Lokiri’s suggestions yet.

[moldy]

Originally Posted by lomkiri

Ok.
Ask if you need an example of how to load the list or the dict from a json file.

Thank you Lokiri - that would be very helpful if you did that and it will probably save me a lot of time tomorrow.

[lomkiri]

I actually have a dynamic list of over 200 items

Then I'm not sure it's a good idea to have 200 capturing groups in your regex, maybe it would be better to opt for the dict solution where no capturing group is necessary.

"find : "John|Paul|George|Ringo"
The function (with dict) doesn't change.

You could first create the dict in a json file :

{
  "John": "Mike",
  "Paul": "Keith",
  "George": "Ronnie",
  "Ringo": "Charlie"
}

and import it into the function.

Then, to create the regex, you could use python to extract the keys from this json file (with dict.keys()) and use the resulting list to create the string "John|Paul|George|Ringo", which you'll use to feed the "find" field

You could also create the json file using another python code, witch will read your text file, transform it in a dict, and write it with json.dump() (after an import json)

[theducks]

Maybe a fork of the Language cleaner plugin
That replaces $#+% words with others

[lomkiri]

Originally Posted by theducks

Maybe a fork of the Language cleaner plugin
That replaces $#+% words with others

I don't know this plugin, and I don't know witch solution is the easiest, but anyway, since I was writing my function while you were writing your message, here it is (it still is simple):

Originally Posted by moldy

Thank you Lokiri - that would be very helpful if you did that and it will probably save me a lot of time tomorrow.

Put your json file in the config-folder of calibre.
If you make a "replace all", it will load the file only once, during the first passage.

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    """ 
    Replace John|Paul|George|Ringo by Mick, Kieth, Ronnie or Charlie, 
    using a dict {"John": "Mick", ...}
    This dict is loaded from a json file, plaaced in the config folder of calibre.
    """
    from calibre.utils.config import JSONConfig
    m = match[0]

    # first passage
    # Load the dict from a json file and store it in the persistent dict "data"
    # (see the help of calibre for more details about the dict "data")
    if number == 1:
        fname = 'beastones.json'
        data['equiv'] = JSONConfig(fname)
        if not data['equiv']:
            print(f'Problem loading {fname}, no treatment will be done')
            
    # normal treatment
    return data['equiv'].get(m, m) 	# if the json file was not found, no changes are made

[lomkiri]

And here is the python code that extracts the keys from the json file, and create the regex to be put in the "find" field

(with your example, this function will print John|Paul|George|Ringo from the json file used by the regex-function. Modify the path of fname accordingly to your needs)

def get_regex():
    import json
    fname = '/data/temp/beastones.json'
    equiv = json.load(open(fname))
    if not equiv:
        print(f'Problem loading {fname}')
        return
    print( '|'.join(equiv.keys()))

I leave to you the function that creates the json file from your original text file ; if you know a little about python, with those examples or some searches in the net, it should be easy (use the method json.dump to create the json file from the python dict)

[moldy]

Originally Posted by theducks

Maybe a fork of the Language cleaner plugin
That replaces $#+% words with others

Thats an interesting idea theducks. I had a look and obviously I would have to write a replacement cleaner.py. Something to think about.

[moldy]

Originally Posted by lomkiri

And here is the python code that extracts the keys from the json file, and create the regex to be put in the "find" field

Hi lomkiri I'm working my way through your suggestions but am stuck here with the error:

ERROR: No replace function: You must create a Python function named replace in your code

Could you help me to move on please?