Regular expression for removing blanks between letters

[Ghitulescu]

Some OCR softwares interpret/convert a spaced word as a suite of characters separated by blank spaces: like S w i t z e r l a n d. In some cases, these can be solved by hand (for instance only important concepts are widened), however, when entire paragraphs across the whole book are widened an automatized method would be very helpful.
I tried [a-z;A-Z].[a-z;A-Z]. and similar but these only identify the places, do not replace the correct letters.

I could not find any relevant thread, which I hope does not suggest there is no solution to this

Thank you for any hint or solution

[JSWolf]

The problem will be the correct space between two words. You would end up combining words by removing the space. I don't know how you would know a space between letters is in the word or between two words. You'll just have to fix this by hand.

[Ghitulescu]

Well, essentially there are very very very few one-letter words words like "a"
This is why I used a regex with two spaces.

The only practical solution is to use pairs of letters separated and preceded or followed by another space.
I hoped for a nice solution.

[JSWolf]

Originally Posted by Ghitulescu

Well, essentially there are very very very few one-letter words words like "a"
This is why I used a regex with two spaces.

The only practical solution is to use pairs of letters separated and preceded or followed by another space.
I hoped for a nice solution.

That won't work either without error. By hand is your only reliable solution. But what you can do is use regex to try to find the problem words. I would go with letter space letter space letter space to find 3 letters with spaces. Then you can fix the words by hand. You could also try letter space letter space and see how that goes.

[Tex2002ans]

Originally Posted by Ghitulescu

Some OCR softwares interpret/convert a spaced word as a suite of characters separated by blank spaces: like S w i t z e r l a n d. In some cases, these can be solved by hand (for instance only important concepts are widened), however, when entire paragraphs across the whole book are widened an automatized method would be very helpful.

I would tackle this in multiple passes.

But as JSWolf has stated, you have to be extremely careful of combining letters/words that shouldn't be. Quite often, books will have things like "Person B" + "Project X" + "time y".

Example Sentence

Let's take this as an example:

<p>A decent example of S w i t z e r l a n d that I found within a G e r m a n example.</p>

Step 1

You replace the space between with a temporary character, like '+' or '¬'.

BUT, you want to handle single-letter words NOT "A", "a", or "I":

Search: \b([B-HJ-Zb-z]) ([B-HJ-Zb-z])\b
Replace: \1+\2

After you run this, you'll get:

Spoiler:

<p>A decent example of S+w+i+t+z+e+r+l a n+d that I found within a G+e+r+m a n example.</p>

Step 2

Then you want to match the "A", "a", or "I" between two already connected letters:

Search: (\+\w) ([aAI]) (\w)\b
Replace: \1+\2+\3

Spoiler:

<p>A decent example of S+w+i+t+z+e+r+l+a+n+d that I found within a G+e+r+m+a+n example.</p>

Those 2 Regexes should get you 95%+ of the way there.

From there, you have to manually check/correct. (Apostrophes, accents, emphasized words that start with 'a', or other odd cases.)

Step 3

Once you've completed everything, you replace the temporary '+' with a blank. That will merge the words together:

Search: \+
Replace: ***LEAVE THIS COMPLETELY BLANK***

<p>A decent example of Switzerland that I found within a German example.</p>

Step 3 (Alternate)

Or, if you wanted to keep the emphasis, you can do something like this:

First replace "1 letter + plus sign + 1 letter" with a span:

Search: (\w)\+(\w)
Replace: <span class="emph">\1\2</span>

Spoiler:

<p>A decent example of <span class="emph">Sw</span>+<span class="emph">it</span>+<span class="emph">ze</span>+<span class="emph">rl</span>+<span class="emph">an</span>+d that I found within a <span class="emph">Ge</span>+<span class="emph">rm</span>+<span class="emph">an</span> example.</p>

Then tackle the dangling single letters at the end (the "+d" in Switzerland):

Search: <span class="emph">(\w+)</span>\+(\w)
Replace: <span class="emph">\1\2</span>

Spoiler:

<p>A decent example of <span class="emph">Sw</span>+<span class="emph">it</span>+<span class="emph">ze</span>+<span class="emph">rl</span>+<span class="emph">and</span> that I found within a <span class="emph">Ge</span>+<span class="emph">rm</span>+<span class="emph">an</span> example.</p>

Then keep merging the "emph spans followed by a plus sign" by running this until there's 0 replacements left:

Search: <span class="emph">(\w+)</span>\+<span class="emph">
Replace: <span class="emph">\1

<p>A decent example of <span class="emph">Switzerland</span> that I found within a <span class="emph">German</span> example.</p>

Then, I highly recommend running DiapDealer's "TagMechanic" (Sigil) or "Diap's Editing Toolbag" (Calibre) to flip those <span>s into <em>.

I wrote step-by-step instructions last year in "How do I change italic <i> shortcut to use <em> instead?".

This will ultimately get you the final outcome you want:

<p>A decent example of <em>Switzerland</em> that I found within a <em>German</em> example.</p>

[Ghitulescu]

Originally Posted by Tex2002ans

Search: \b([B-HJ-Zb-z]) ([B-HJ-Zb-z])\b
Replace: \1+\2

Thank you,
I have extended the query to fit 4-letter words (the nice ones ) and the work was substantially reduced, although still exists.

[Tex2002ans]

Originally Posted by Ghitulescu

I have extended the query to fit 4-letter words (the nice ones ) and the work was substantially reduced, although still exists.

Can you show a before/after example of what you mean?

Originally Posted by Ghitulescu

Thank you,

And for future info, the "gap between letters" is called letterspacing. It's sometimes used as emphasis instead of bold/italics, and can be replicated using CSS:



HTML:

<p>As much mud in the streets as if the waters had but newly retired from the face of the earth, and it would not be wonderful to meet a <em>Megalosaurus</em>, forty feet long or so, waddling like an elephantine lizard up <em>Holborn Hill</em>.</p>

CSS:

em {
	font-style: normal;
	font-weight: bold;
	letter-spacing: .2em;
}

For more detailed info, also see:

• MDN article on "letter-spacing"
• 2020 thread, "<i>, <em> or <span> for italics ?"
• 2017 thread, "Extended <head> chapter: NOT necessary?"

Especially all the posts in those two threads, we went into extremely detailed discussions about differences between italics/emphasis, bold/strong, plus different methods of application.

[Ghitulescu]

\b([A-Za-z]) ([A-Za-z]) ([A-Za-z]) ([A-Za-z])\b

as most words are 4+ letters long. Also, being foreign language, the elimination of I (first person) would have been counterproductive (lots of foreign glyphs are OCRed as I, for instance ïìîı, because they are longer than i, also l is considered as I in sans-serif fonts).

I could live with a handful of 3-letter long "escapees"

I know it was called letterspacing, but the use of this term would have forced me to rewrite the sentence once again I tried to use simple words
The OCR insert however spaces.

[JSWolf]

delete post

[JSWolf]

Originally Posted by Ghitulescu

\b([A-Za-z]) ([A-Za-z]) ([A-Za-z]) ([A-Za-z])\b

as most words are 4+ letters long. Also, being foreign language, the elimination of I (first person) would have been counterproductive (lots of foreign glyphs are OCRed as I, for instance ïìîı, because they are longer than i, also l is considered as I in sans-serif fonts).

I could live with a handful of 3-letter long "escapees"

I know it was called letterspacing, but the use of this term would have forced me to rewrite the sentence once again I tried to use simple words
The OCR insert however spaces.

Again do it by hand. What if you have something like "o n a bus"? You would end up with "ona bus"

You cannot regex this away. You have to do it by hand because you will combine letters/words you do not want to.

Use the regex for searching. But do the fixing by hand.

[Ghitulescu]

Lucky me: where blanks are (whitespaces), the OCR inserts a double-space.

Yes, handwork is needed.

[Tex2002ans]

Originally Posted by Ghitulescu

as most words are 4+ letters long.

And if you give more real-life examples, then the regex can be made more robust.

I tested both steps on the examples I gave, and it works perfectly fine on any 2+ single letters (not "a", "A", or "I") next to each other.

Originally Posted by Ghitulescu

Lucky me: where blanks are (whitespaces), the OCR inserts a double-space.

Good to hear. If those two-spaces only occur in the letterspaced words... then your life is easier, no fancy regex needed!

And can I ask:

Which OCR are you using?

Can you share an example page or something from this specific book?

I'd be interested in taking a look.

Originally Posted by Ghitulescu

Also, being foreign language, the elimination of I (first person) would have been counterproductive (lots of foreign glyphs are OCRed as I, for instance ïìîı, because they are longer than i, also l is considered as I in sans-serif fonts).

Which language? I was assuming English and no accents.

Yes, of course, different languages are going to have their own little single-letter-word quirks...

Like in Spanish, you'd want to avoid 'y' (since that = "and").

But then you would just swap out the [aAI] regex with a [yY] (or equivalent).

Accents, similar situation. You'll just have to make much uglier and harder-to-understand regex.

Originally Posted by Ghitulescu

I know it was called letterspacing, but the use of this term would have forced me to rewrite the sentence once again I tried to use simple words

Yep, it's just helpful when someone searches for a solution in the future for "how do I fix a gap between letters?".

Originally Posted by JSWolf

Again do it by hand. What if you have something like "o n a bus"? You would end up with "ona bus"

You cannot regex this away. You have to do it by hand because you will combine letters/words you do not want to.

Use the regex for searching. But do the fixing by hand.

You can use Regex to do the vast bulk of the corrections, then manually fix the edge cases.

Better/faster to do:

• 95% correct with a 2-step regex.
• 5% manually find/correct/fix.

than:

• 100% manually fix.

And as usual, I've been pondering on how to get Spellcheck Lists to help you solve this issue more efficiently.

Instead of using a '+' or '¬', it might be better to use a period:

<p>A decent example of S.w.i.t.z.e.r.l.a.n.d that I found within a G.e.r.m.a.n example.</p>

This allows you to spot all of them easily in Sigil's or Calibre's Spellcheck Lists:



All merged words right there in a simple list.

Although the period will bring a few other minor issues (like "a.m." or "p.m."), but the amount of time you'll save is massive.

[JSWolf]

The spellcheck lists is a very good idea.

[Jellby]

Originally Posted by Tex2002ans

Although the period will bring a few other minor issues (like "a.m." or "p.m."), but the amount of time you'll save is massive.

Maybe convert all periods to ¬ before dealing with the spaces (and back to periods after)? [That's what you use protecting groups for in chemistry ]

[Tex2002ans]

Originally Posted by Jellby

Maybe convert all periods to ¬ before dealing with the spaces (and back to periods after)?

Yes yes, or any sort of similarly "rare" character would work.

Although I'm thinking a mass replace of . may cause the book to explode if you forget to flip back!

Because . is probably commonly be used outside of text (within class names, filenames, etc.):

<link href="../Styles/Style0001¬css" type="text/css" rel="stylesheet"/>
<a href="http://www¬mobileread¬com">
<img src="image¬jpg" />

Originally Posted by Jellby

[That's what you use protecting groups for in chemistry ]

I knew ¬ is "not" in Logic.

Or "a gun":

· ¬<(o.o<)

Didn't know the chemistry usage though. Looks like I have something new to read about.