adjusting a function

[roger64]

Hi

Some months ago, you gave us a nice function which allowed to split words "glued" together. After a mistake of mine, I had the opportunity to use this function on a lot of words on a French EPUB. I have of course installed a French dictionary. Please read on...
The results were amazingly good and quick.

Spoiler:

>([^<]+)<

import regex
from calibre import replace_entities, prepare_string_for_xml

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    def fix_word(m):
        word = m.group()
        if dictionaries.recognized(word):
            return word
        for i in xrange(1, len(word) - 1):
            a, b = word[:i], word[i:]
            if dictionaries.recognized(a) and dictionaries.recognized(b):
                return a + ' ' + b
        return word
    text = replace_entities(match.group(1))
    text = regex.sub(r'\b\w+\b', fix_word, text, flags=regex.VERSION1)
    text = prepare_string_for_xml(text)
    return '>' + text + '<'

Using the Calibre Editor spell checker before and after the use of this function, I could see that the number of words unknown to the dictionary went down from 1167 to 261. Taking into account the fact that probably 2/3 of the remaining ones were "noms propres" (proper nouns ?), I nevertheless realized that some few words had not been split (50 to 70 probably).

The cause was related with some kind of elided form. Here are some of them. One can easily discern the same pattern: a word followed by one letter and one curved apostrophe (in red here); these last two elements being characteristic of elided forms in French.

accompagnentn’auront
àl’origine
dansl’entrée
dem’expliquer
des’opposer
Etj’écrasai
ils’attendait
manueld’algèbre

What makes me hope that the function could be improved so as to take care of elided forms is that for all of them, the first suggestion of the dictionary of the Calibre editor is to split them correctly.

[kovidgoyal]

@roger64: Sorry, I'm a little swaamped at the moment, so I dont have time to look at this, hopefully someone else will be able to help you.

[roger64]

Thanks for your reply. I hope too.

[phossler]

@roger64 - what is a link to the post that had the function?

My Search didn't turn up anything

[roger64]

Here is the thread.
https://www.mobileread.com/forums/sho...d.php?t=251941

[roger64]

Hi

and here is a new version of this function taking into account the elided forms (at least for French language) thanks to Olivier, the author of -opensource- Grammalecte.

Spoiler:

import regex
from calibre import replace_entities, prepare_string_for_xml
 
def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    def fix_word(m):
        word = m.group()
        if dictionaries.recognized(word):
            return word
        for i in xrange(1, len(word) - 1):
            a, b = word[:i], word[i:]
            if dictionaries.recognized(a) and dictionaries.recognized(b):
                return a + ' ' + b
        m = regex.match(r"(\w+)((?:[dlnmts]|qu(?:oi|el)qu|puisqu|lorsqu|jusqu|qu)[’'`]\w+)", word)
        if m:
            return m.group(1) + " " + m.group(2)
        return word
    text = replace_entities(match.group(1))
    text = regex.sub(r"\b\w(?:[\w’'`-]*\w|\w+)\b", fix_word, text, flags=regex.VERSION1)
    text = prepare_string_for_xml(text)
    return '>' + text + '<'

or here: http://pastebin.com/quGQQzcN

[phossler]

Thanks.

Small issue

If there's <style> in the html file, the [Replace All] will process that text and generate errors

For example, I had a jacket.xhtml file in an epub, and the RE split some things that generated errors (not typos or mis-spellings)

Spoiler:

<style type="text/css">
.cbj_banner {
background: #eee;
col or: black;
border: thin solid black;
margin: 1 em;
padding: 1 em;
}
table.cbj_header td.cbj_title {
font-size: 1.5 em;
font-style: italic;
text-align: c enter;
}
table.cbj_header td.cbj_series {
text-align: c enter;
}
table.cbj_header td.cbj_author {
text-align: c enter;
}
table.cbj_header td.cbj_pubdata {
text-align: c enter;
}
table.cbj_header {
width: 100%;
}
table.cbj_header td.cbj_label {
text-align: right;
width: 33%;
}
table.cbj_header td.cbj_content {
text-align: left;
width: 67%;
}
hr.metadata_divider {
width: 90%;
margin-left: 5%;
border-top: solid white 0;
border-right: solid white 0;
border-bottom: solid black 1px;
border-left: solid white 0;
}
hr {
border-top: 0 solid white;
border-right: 0 solid white;
border-bottom: 2px solid black;
border-left: 0 solid white;
margin-left: 10%;
width: 80%;
}
.cbj_footer {
font-size: 0.8 em;
margin-top: 8px;
text-align: c enter;
}
</style>

Any way to make the function a little smarter?

I can always regenerate the jacket.xhtml file, but any other files that have <style> in would probably be changed also