Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Editor

Notices

Reply
 
Thread Tools Search this Thread
Old 03-11-2024, 12:10 PM   #1
moldy
Enthusiast
moldy began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Oct 2015
Device: Kindle
Search and Replace from a List

In Editor/Saved Searches I am trying to import a json file to search for multiple items and replace from a list. For example:

Code:
{
  "searches": [
    {
      "case_sensitive": false,
      "dot_all": false,
      "find": ["John", "Paul", "George", "Ringo"],
      "mode": "regex",
      "name": "B-S",
      "replace": ["Mick", "Kieth", "Ronnie", "Charlie"]
    }
  ],
  "version": 1
}
However when I run the search it gives the error:
calibre, version 7.6.0
Code:
ERROR: Unhandled exception: <b>TypeError</b>:unhashable type: 'list'

calibre 7.6  embedded-python: True
Windows-10-10.0.22631-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22631')
Python 3.11.5
Windows: ('10', '10.0.22631', 'SP0', 'Multiprocessor Free')
Interface language: None
EXE path: C:\Program Files\Calibre2\calibre-parallel.exe
Successfully initialized third party plugins: Count Pages (1, 13, 5)
Traceback (most recent call last):
  File "calibre\gui2\tweak_book\boss.py", line 1111, in run_saved_searches
  File "calibre\gui2\tweak_book\search.py", line 1411, in run_search
  File "calibre\gui2\tweak_book\search.py", line 1411, in <listcomp>
  File "calibre\gui2\tweak_book\search.py", line 1312, in get_search_regex
  File "calibre\ebooks\conversion\search_replace.py", line 14, in compile_regular_expression
TypeError: unhashable type: 'list'
I have done something similar in the past but I lost all my Calibre stuff in a PC crash and I can't remember how I did it.
Can someone point me in the right direction please?
moldy is offline   Reply With Quote
Old 03-11-2024, 02:38 PM   #2
lomkiri
Zealot
lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.
 
lomkiri's Avatar
 
Posts: 136
Karma: 1000102
Join Date: Jul 2021
Device: N/A
Quote:
Originally Posted by moldy View Post
However when I run the search it gives the error:
TypeError: unhashable type: 'list'
You get this error because you pass a list where it expects a string.
I don't know how to do this in mode regex, but in mode function I'll do it this way :

Code:
      [...]
      "find": "(John)|(Paul)|(George)|(Ringo)",
      "mode": "function",
      [...]
and the function (edit: see next msg for a better code) :
Code:
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    repl = ["Mick", "Kieth", "Ronnie", "Charlie"]
    len_match = len(match)

    for i in range(1, len_match):
        if match[i]:
            return repl[i-1] if i <= len(repl) else match[0]
Explanation :
If we find "Paul", it will be in the 2nd capturing group (i.e. match[2], the other groups will be None), so we return the 2nd name of the list "repl", i.e. repl[1].

If "repl" is, by mistake, smaller than the number of the capturing groups, we avoid an exception not doing anything (returns match[0])

If you're sure of the exact correspondence of the lists in and out, you can just put
return repl[i-1]
it will be quicker if there are lots of occurrences

Last edited by lomkiri; 03-12-2024 at 09:56 AM.
lomkiri is offline   Reply With Quote
Advert
Old 03-11-2024, 03:32 PM   #3
lomkiri
Zealot
lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.
 
lomkiri's Avatar
 
Posts: 136
Karma: 1000102
Join Date: Jul 2021
Device: N/A
I found a better way for the function, with the property lastindex :
Code:
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    repl = ["Mick", "Kieth", "Ronnie", "Charlie"]
    idx = match.lastindex
    return repl[idx -1]
    
    # or, if you prefer safer : return repl[idx -1] if idx <= len(repl) else match[0]
Another way, with a dict:
Code:
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    equiv = {"John": "Mick", "Paul": "Kieth", "George": "Ronnie", "Ringo": "Charlie"}
    m = match[0]
    return equiv.get(m, m)    # if "m" is not in the dict, leave the text untouched
In both cases, it is possible to load the list or the dict in the function from a json file instead of defining it as a variable, maybe it was what you were thinking about? In that case, the function doesn't need to be modified at each use, only the regex and the json file.

Last edited by lomkiri; 03-12-2024 at 10:25 AM.
lomkiri is offline   Reply With Quote
Old 03-12-2024, 09:41 AM   #4
moldy
Enthusiast
moldy began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Oct 2015
Device: Kindle
Thanks for your help Lomkiri I'll give it a try and let you know how it goes.
moldy is offline   Reply With Quote
Old 03-12-2024, 10:01 AM   #5
lomkiri
Zealot
lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.
 
lomkiri's Avatar
 
Posts: 136
Karma: 1000102
Join Date: Jul 2021
Device: N/A
Ok.
Ask if you need an example of how to load the list or the dict from a json file.

Last edited by lomkiri; 03-12-2024 at 10:26 AM.
lomkiri is offline   Reply With Quote
Advert
Old 03-12-2024, 11:20 AM   #6
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,996
Karma: 57259778
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
This is the Editor you are using (folk post other ways to Edit things here)
KISS
A Group of saved searches. 1 for search each word.
Then just run the group from the saved search pane.
theducks is offline   Reply With Quote
Old 03-12-2024, 12:34 PM   #7
lomkiri
Zealot
lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.
 
lomkiri's Avatar
 
Posts: 136
Karma: 1000102
Join Date: Jul 2021
Device: N/A
I understood that moldy gave just an example, and that the list could be wide.
The solution I gave permits a list of any number of elements, in the case of 50 elements, for ex. it would be difficult to make the replacements one by one.

And the solution is relatively simple, the function has only 3 lines (one line is optional)
lomkiri is offline   Reply With Quote
Old 03-12-2024, 01:10 PM   #8
moldy
Enthusiast
moldy began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Oct 2015
Device: Kindle
Lomkiri is correct. I actually have a dynamic list of over 200 items and their alternatives in a text file. I’m looking for a solution where I can copy and paste my lists into a find and replace function .
I don’t want the work done for me Theducks but I do appreciate being put on the right road by someone more knowledgeable and experienced.
I am away from home so haven’t been able to try Lokiri’s suggestions yet.
moldy is offline   Reply With Quote
Old 03-12-2024, 01:24 PM   #9
moldy
Enthusiast
moldy began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Oct 2015
Device: Kindle
Quote:
Originally Posted by lomkiri View Post
Ok.
Ask if you need an example of how to load the list or the dict from a json file.
Thank you Lokiri - that would be very helpful if you did that and it will probably save me a lot of time tomorrow.
moldy is offline   Reply With Quote
Old 03-12-2024, 01:33 PM   #10
lomkiri
Zealot
lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.
 
lomkiri's Avatar
 
Posts: 136
Karma: 1000102
Join Date: Jul 2021
Device: N/A
Quote:
I actually have a dynamic list of over 200 items
Then I'm not sure it's a good idea to have 200 capturing groups in your regex, maybe it would be better to opt for the dict solution where no capturing group is necessary.

"find : "John|Paul|George|Ringo"
The function (with dict) doesn't change.

You could first create the dict in a json file :
Code:
{
  "John": "Mike",
  "Paul": "Keith",
  "George": "Ronnie",
  "Ringo": "Charlie"
}
and import it into the function.

Then, to create the regex, you could use python to extract the keys from this json file (with dict.keys()) and use the resulting list to create the string "John|Paul|George|Ringo", which you'll use to feed the "find" field

You could also create the json file using another python code, witch will read your text file, transform it in a dict, and write it with json.dump() (after an import json)

Last edited by lomkiri; 03-12-2024 at 03:49 PM.
lomkiri is offline   Reply With Quote
Old 03-12-2024, 02:19 PM   #11
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,996
Karma: 57259778
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Maybe a fork of the Language cleaner plugin
That replaces $#+% words with others
theducks is offline   Reply With Quote
Old 03-12-2024, 02:28 PM   #12
lomkiri
Zealot
lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.
 
lomkiri's Avatar
 
Posts: 136
Karma: 1000102
Join Date: Jul 2021
Device: N/A
Quote:
Originally Posted by theducks View Post
Maybe a fork of the Language cleaner plugin
That replaces $#+% words with others
I don't know this plugin, and I don't know witch solution is the easiest, but anyway, since I was writing my function while you were writing your message, here it is (it still is simple):

Quote:
Originally Posted by moldy View Post
Thank you Lokiri - that would be very helpful if you did that and it will probably save me a lot of time tomorrow.
Put your json file in the config-folder of calibre.
If you make a "replace all", it will load the file only once, during the first passage.

Code:
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    """ 
    Replace John|Paul|George|Ringo by Mick, Kieth, Ronnie or Charlie, 
    using a dict {"John": "Mick", ...}
    This dict is loaded from a json file, plaaced in the config folder of calibre.
    """
    from calibre.utils.config import JSONConfig
    m = match[0]

    # first passage
    # Load the dict from a json file and store it in the persistent dict "data"
    # (see the help of calibre for more details about the dict "data")
    if number == 1:
        fname = 'beastones.json'
        data['equiv'] = JSONConfig(fname)
        if not data['equiv']:
            print(f'Problem loading {fname}, no treatment will be done')
            
    # normal treatment
    return data['equiv'].get(m, m) 	# if the json file was not found, no changes are made

Last edited by lomkiri; 03-13-2024 at 06:27 AM. Reason: comments added in the code
lomkiri is offline   Reply With Quote
Old 03-12-2024, 03:16 PM   #13
lomkiri
Zealot
lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.lomkiri ought to be getting tired of karma fortunes by now.
 
lomkiri's Avatar
 
Posts: 136
Karma: 1000102
Join Date: Jul 2021
Device: N/A
And here is the python code that extracts the keys from the json file, and create the regex to be put in the "find" field

(with your example, this function will print John|Paul|George|Ringo from the json file used by the regex-function. Modify the path of fname accordingly to your needs)

Code:
def get_regex():
    import json
    fname = '/data/temp/beastones.json'
    equiv = json.load(open(fname))
    if not equiv:
        print(f'Problem loading {fname}')
        return
    print( '|'.join(equiv.keys()))
I leave to you the function that creates the json file from your original text file ; if you know a little about python, with those examples or some searches in the net, it should be easy (use the method json.dump to create the json file from the python dict)

Last edited by lomkiri; 03-12-2024 at 03:48 PM.
lomkiri is offline   Reply With Quote
Old 03-13-2024, 08:35 AM   #14
moldy
Enthusiast
moldy began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Oct 2015
Device: Kindle
Quote:
Originally Posted by theducks View Post
Maybe a fork of the Language cleaner plugin
That replaces $#+% words with others
Thats an interesting idea theducks. I had a look and obviously I would have to write a replacement cleaner.py. Something to think about.
moldy is offline   Reply With Quote
Old 03-14-2024, 02:15 PM   #15
moldy
Enthusiast
moldy began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Oct 2015
Device: Kindle
Quote:
Originally Posted by lomkiri View Post
And here is the python code that extracts the keys from the json file, and create the regex to be put in the "find" field
Hi lomkiri I'm working my way through your suggestions but am stuck here with the error:


Code:
ERROR: No replace function: You must create a Python function named replace in your code
Could you help me to move on please?
moldy is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Search and Replace Ashjuk Sigil 10 02-25-2021 11:17 AM
Regex in search problems (NOT Search&Replace; the search bar) lairdb Calibre 3 03-15-2017 07:10 PM
save multiple search/replace, or search/replace multiple ebooks user743 Editor 12 04-12-2014 02:38 AM
Search and Replace Help Squidly21 Conversion 2 01-08-2014 12:19 AM
search and replace - drops blanks in replace ? cybmole Conversion 10 03-13-2011 03:07 AM


All times are GMT -4. The time now is 09:47 PM.


MobileRead.com is a privately owned, operated and funded community.