Saved Search/Regex Functions - Page 5

lomkiri · 03-24-2024, 03:26 PM

In the thread Search and Replace from a List, moldy asked how he could replace a list of names by the names of another list, i.e.
["John", "Paul", "George", "Ringo"] by
["Mick", "Keith", "Ronnie", "Charlie"].

A way to do this with generic regex and function is to use an external json file including a dict, so we have just to adapt the dict to our needs, and it works with any number of words in the list (even hundreds, which was the case for Moldy).

With our example, let put the file change_words.json in the config folder of calibre, this file containing :

Code:

{
  "John": "Mike",
  "Paul": "Keith",
  "George": "Ronnie",
  "Ringo": "Charlie"
}

Code:

find : <body[^>]*>\K(.+)</body>
"dot all" must be checked
This regex selects the whole html page, i.e. all text in the <body>
The regex inside the function will avoid everything inside <>, so the html tags won't be scanned.

The function, in its simpler way (without any counters) is:

Code:

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    # Replace words using a dict in a json file (without counters)
    from calibre.utils.config import JSONConfig
    import regex

    # Put the file 'change_words.json' in the config-folder of calibre
    # If you choose another name for the json, change it here:
    fname = 'change_words.json'

    # Load json only at first passage
    # data will retain its values throught all passages when "replace all"
    if number == 1:
        data['equiv'] = JSONConfig(fname)
        if not data['equiv']:
            print(f'Problem loading {fname}, no treatment will be done')

    # normal passage
    m = match.group()
    for key, val in data['equiv'].items():
        # Find key, excluding everything between <...>
        m = regex.sub(rf'\b{key}\b(?![^<>]*>)', val, m)
    return m

Note: Each replace is a whole page, so the number of changes will be the number of pages scanned, even if in the 1st page there is one hundred change and none in the other pages.

If we want a counter with the number of real changes, the code is the one below. Calibre will open a "debug window" at the end of the replace action with some counters. It is also possible to write a json file with the number of changes by word (enable by default). Set the variables "do_count" and "count_by_name" as you need.

Counters will be more accurate with "replace all".
If replacements are made one by one, the counters will be reset at each file.

Code:

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    # Replace words using a dict in a json file, with possibility of counters
    from calibre.utils.config import JSONConfig
    import regex

    ### Parameters
    # Put the file 'change_words.json' in the config-folder of calibre
    # If you choose another name for the json, change it here:
    fname = 'change_words.json'

    # If do_count is True, will write the total of changes.
    # It count_by_name is also True, the function will write the counters by name in the
    # file "change_words_counters.json" (in the config-folder of calibre)
    do_count = True    # put False if you don't want any counter
    count_by_name = True
    counters_fname = 'change_words_counters.json'
    ### End Parameters


    # === Last passage: if counters were asked in the heading of this function
    if match == None:
        if data['total'] == 0:
            print('No occurrence found.\n'
                  f"We had to change a list of {len(data['equiv'])} words (in {fname})")
            return

        if count_by_name:
            json = JSONConfig(counters_fname)
            json.clear()
            json.update(data['counters'])
            json.commit()

        print(f"We had to change a list of {len(data['equiv'])} words (in {fname})\n"
              f"In this list, {len(data['counters'])} words had at least one occurrence\n"
              f"=== The total of all changes is: {data['total']} ===\n\n"
              f"The file {counters_fname} has been written with the counters by word" if count_by_name else '')
        return

    # === First passage
    # Load the json file only at first passage
    # data will retain its values throught all passages when "replace all"
    if number == 1:
        data['equiv'] = JSONConfig(fname)
        if not data['equiv']:
            print(f'Problem loading {fname}, no treatment will be done')
        if do_count:
            replace.call_after_last_match = True    # Ask for last passage
            data['total'] = 0
            data['counters']= {}


    # === normal passage
    m = match.group()
    for key, val in data['equiv'].items():
        # Find key, excluding everything between <...>
        m, n = regex.subn(rf'\b{key}\b(?![^<>]*>)', val, m)
        if do_count and n:
            data['total'] += n
            data['counters'][key] = data['counters'].get(key, 0) + n
    return m

moldy · 03-26-2024, 06:30 AM

Thank you @lomkiri. The function, including the counters, works perfectly on my data file of 187 entries (and growing ever larger).

For my purposes I would have liked to have been able to add regex to the keys individually in the json file rather than globally in the line:

Code:

 m = regex.sub(rf'\b{key}\b(?![^<>]*>)', val, m)

This is in no way a criticism and the function is perfectly adequate for my needs as is.

Some of the entries in my data benefit from having the trailing '\b' removed while in others it causes errors. Some entries would be better with a little more regex added.

I can get round this by writing searches/replaces in a json with regex added and importing it into Saved Searches. Obviously this takes a little more time.

lomkiri · 03-26-2024, 09:59 AM

Quote:

Originally Posted by moldy

For my purposes I would have liked to have been able to add regex to the keys individually in the json file rather than globally

Modify your entries in the json in this way:
"old_name": ["new_name", "regex"]
If you don't provide a regex, the one from the function will be used. In this case, the entry shall be either (at your choice) :
"old_name": ["new_name"]
or "old_name": "new_name"
or "old_name": ["new_name", ""]
Example :

Code:

{
  "John": ["Mike", "\\b{}(?![^<>]*>)"],
  "Paul": "Keith",
  "George": ["Ronnie",  "{}\\b(?![^<>]*>)"],
  "Ringo": ["Charlie"]
}

Inside the regex, {} will be replaced in the function by old_name using str.format()
You must double all the antislashes in the json
You cannot use the regex to search curly brackets in the text, since format() will try to interpret them, if it's necessary, a work-around must be used.

The function is:

Code:

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    # Replace words using a dict in a json file, with possibility of counters
    # The regex for replacing each word may (or may not) be provided with the word

    ### Parameters
    # Put the file 'change_words.json' in the config-folder of calibre
    # If you choose another name for the json, change it here:
    fname = 'change_words.json'
    #
    # If do_count is True, will write the total of changes.
    do_count = True    # put False if you don't want any counter
    #
    # It count_by_name is also True, the function will write the counters by name in the
    # file "change_words_counters.json" (in the config-folder of calibre)
    count_by_name = True
    counters_fname = 'change_words_counters.json'
    #
    # The regex to be applied for the names that have no regex provided in the json:
    rgx_default = r'\b{}\b(?![^<>]*>)'  # Find the key, excluding everything between <...> :
    ### End Parameters

    from calibre.utils.config import JSONConfig
    import regex

    # === Last passage: if counters were asked in the heading of this function
    if match == None:
        if data['total'] == 0:
            print('No occurrence found.\n'
                  f"There is {len(data['equiv'])} entries in the file '{fname}'")
            return

        print(f"There is {len(data['equiv'])} entries in the file '{fname}'\n"
              f"In this dict, {len(data['counters'])} words had at least one occurrence\n"
              f"=== The total of all changes is: {data['total']} ===\n\n"
              f"The file {counters_fname} has been written with the counters by word" if count_by_name else '')

        if count_by_name and counters_fname:
            json = JSONConfig(counters_fname)
            json.clear()
            json.update(data['counters'])
            json.commit()
        return

    # === First passage
    # Load the json file only at first passage
    # The dict "data" retains its values throught all the passages when "replace all"
    if number == 1:
        data['equiv'] = JSONConfig(fname).copy()    # .copy() is here to avoid the json file to be formated or modified
        if not data['equiv']:
            print(f'Problem loading {fname}, no treatment will be done')
            return
        # Prepare the dict for the treatment :
        for key, val in data['equiv'].items():
            if isinstance(val, str):
                val = [val, rgx_default]                
            elif len(val) == 1:
                 val.append(rgx_default)                
            elif not val[1]:
                val[1] = rgx_default                
            data['equiv'][key] = val
        # Activate the counters:
        if do_count:
            replace.call_after_last_match = True    # Ask for last passage
            data['total'] = 0
            data['counters']= {}

    # === normal passage
    m = match.group()
    for key, val in data['equiv'].items():
        [new_name, rgx] = val
        m, n = regex.subn(rgx.format(key), new_name, m)
        if do_count and n:
            data['total'] += n
            data['counters'][key] = data['counters'].get(key, 0) + n
    return m

lomkiri · 03-26-2024, 08:09 PM

27/03 : I have slightly modified the function so the json file is not modified anymore (JSONConfig natively formats and orders the json source).
An entry {"John": ["Mike", ""], ... is now accepted (default regex applies)
Some cosmetic changes

28/03 : correction of a new bug that would have brought an exception in some cases.

moldy · 03-28-2024, 07:01 AM

Quote:

I have slightly modified the function so the json file is not modified anymore ....

Thanks lomkiri your latest installment of the function is a big improvement for me. I actually have two data files, one all lower case, and the other with a capitalised first letter. I have now merged these files into one.

The file now has 400 entries. Even with a file this size the function only takes a second or two to run.

My ultimate goal would be to have a function that would work as though it were possible to have a json something like this (similar to Saved Searches):

Code:

{"([P|p])aul([a-z]+)": "\1ol\2"}

I think that would find Paul paul Pauline pauline Paula paula and so on and invent several new names.

lomkiri · 03-28-2024, 08:05 AM

In that case, I don't understand the interest of my function anymore, it will just mimick the saved searches features with a json in a slightly different form, and then the proposal of Theducks applies. It's easy of course to modify the function, it's just adding a optional 3rd dim to the list for each element of the json, but you can do that as well with a json for saved searches and apply all searches at once.

If you don't want to mix those specific searches with your usual ones, you may either use a portable calibre for this specific task, or you may backup your different jsons of saved searches and "delete all saved searches + import the right json".

What do you think ? If you still want this feature, I'll adapt the function at night, but IMO the saved searches are exactly fitting you needs.

lomkiri · 03-28-2024, 02:16 PM

Mmmh, I wrote the former message in the train, and it was written too quickly :-).
No need to change anything in the function, just put an entry in you json :

Code:

"au": ["ol", "[Pp]\\K{}(?=[a-z]+\\b)"]

and it will do exactly what you've asked.

Explanation :
The regex will be stuffed in the function with "au", so we'll apply :
[Pp]\Kau(?=[a-z]+\b)
Find [Pp] then forget this (start from after [Pp]) (because of \K)
Find "au", but only if there is [a-z]+\b after it (because of the positive lookahead)
So only "au" is selected, and it will be replaced by "ol"

BUT: get the new version of the function above, I''ve corrected a little bug that would have brought an exception in some cases.

moldy · 03-29-2024, 06:37 AM

Quote:

In that case, I don't understand the interest of my function anymore, it will just mimick the saved searches features with a json in a slightly different form,

You're absolutely correct in what you say lomkiri. I had lost sight of my original goal which was to automate the process as much as possible. The small number of entries that need further regex can be put into normal saved searches.

So; using yesterday's (28 March) version of the function and my data file of 411 entries (so far) every substitution was made correctlly, the counters counted as expected and there were no exception errors.

I think your task on earth is now complete lomkiri. You are free to return to the mother ship.

03-26-2024, 06:30 AM	#62
moldy Enthusiast Posts: 38 Karma: 10 Join Date: Oct 2015 Device: Kindle	Thank you @lomkiri. The function, including the counters, works perfectly on my data file of 187 entries (and growing ever larger). For my purposes I would have liked to have been able to add regex to the keys individually in the json file rather than globally in the line: Code: m = regex.sub(rf'\b{key}\b(?![^<>]*>)', val, m) This is in no way a criticism and the function is perfectly adequate for my needs as is. Some of the entries in my data benefit from having the trailing '\b' removed while in others it causes errors. Some entries would be better with a little more regex added. I can get round this by writing searches/replaces in a json with regex added and importing it into Saved Searches. Obviously this takes a little more time.

03-26-2024, 08:09 PM	#64
lomkiri Zealot Posts: 136 Karma: 1000102 Join Date: Jul 2021 Device: N/A	27/03 : I have slightly modified the function so the json file is not modified anymore (JSONConfig natively formats and orders the json source). An entry {"John": ["Mike", ""], ... is now accepted (default regex applies) Some cosmetic changes 28/03 : correction of a new bug that would have brought an exception in some cases. Last edited by lomkiri; 03-28-2024 at 02:20 PM.

03-28-2024, 02:16 PM	#67
lomkiri Zealot Posts: 136 Karma: 1000102 Join Date: Jul 2021 Device: N/A	Mmmh, I wrote the former message in the train, and it was written too quickly :-). No need to change anything in the function, just put an entry in you json : Code: "au": ["ol", "[Pp]\\K{}(?=[a-z]+\\b)"] and it will do exactly what you've asked. Explanation : The regex will be stuffed in the function with "au", so we'll apply : [Pp]\Kau(?=[a-z]+\b) Find [Pp] then forget this (start from after [Pp]) (because of \K) Find "au", but only if there is [a-z]+\b after it (because of the positive lookahead) So only "au" is selected, and it will be replaced by "ol" BUT: get the new version of the function above, I''ve corrected a little bug that would have brought an exception in some cases. Last edited by lomkiri; 03-28-2024 at 07:14 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
About saved searches and regex	Carpatos	Editor	22	09-30-2020 10:56 PM
Regex-Functions - getting user input	CalibUser	Editor	8	09-09-2020 04:26 AM
Difference in Manual Search and Saved Search	phossler	Editor	4	10-04-2015 12:17 PM
Help - Learning to use Regex Functions	weberr	Editor	1	06-13-2015 01:59 AM
Limit on length of saved regex?	ElMiko	Sigil	0	06-30-2013 03:32 PM

03-28-2024, 08:05 AM	#66
lomkiri Zealot Posts: 136 Karma: 1000102 Join Date: Jul 2021 Device: N/A	In that case, I don't understand the interest of my function anymore, it will just mimick the saved searches features with a json in a slightly different form, and then the proposal of Theducks applies. It's easy of course to modify the function, it's just adding a optional 3rd dim to the list for each element of the json, but you can do that as well with a json for saved searches and apply all searches at once. If you don't want to mix those specific searches with your usual ones, you may either use a portable calibre for this specific task, or you may backup your different jsons of saved searches and "delete all saved searches + import the right json". What do you think ? If you still want this feature, I'll adapt the function at night, but IMO the saved searches are exactly fitting you needs.