P.S. I don't think the function itself is customizable unless you run Calibre from source and modify the source code.
However, if you do the replacements in batches of 10 or less, you can preview the transliterated titles in the bulk s/r dialog to decide which titles need further "massaging" or modify the template to include additional regex replacement.
https://manual.calibre-ebook.com/gen..._ref.html#id78