Edit Book: How to remove formattings, here in text?

[chaot]

I converted a PDF-file to ePub. Similar to here.


formatted text (Line Heights)

How I can remove these (or best any) formattings in Edit Book, here different Line Hights?
This is quite irritating!

I run Beautify files & Fix HTML, Check Book OK.

The yellow markings are (same sort of) EN DASHs, obviously they have nothing to do with the matter.

[Phssthpok]

This can be caused by invisible characters that can't be displayed in the editor. I've seen this happen with strange quote characters that just don't show up in the editor (some bizarre Unicode symbols that not all fonts can represent). One way to see if that's the problem is to copy the text into an editor that doesn't understand Unicode (e.g. Textpad on Windows), and any non-8-bit characters will show up as "?" or some such. You can also use a regex-function to convert characters above U+00FF to entities:

def replace(match, number, file_name, metadata, dictionaries, data, functions, *args, **kwargs):
    result = ''
    for c in match.group():
        if ord(c) > 255:
           result += '&#%d;' % ord(c)
        else:
           result += c
    return result

Select (mark) the text that looks funny, and do a replace-all on ".*" in the marked text using this function. Then look for any strange "&#nnn;" entities.

Note that the editor is showing you the raw HTML, so it's not an HTML formatting problem -- it's something that's giving the editor indigestion.

[chaot]

Select (mark) the text that looks funny, and do a replace-all on ".*" in the marked text using this function. Then look for any strange "&#nnn;" entities.

Please provide another explanation on how to do this.

[Phssthpok]

Originally Posted by chaot

Please provide another explanation on how to do this.

Hmm. I'm not sure I can say it any more clearly, but here goes in baby steps:
1) Select the mis-spaced text and press Ctrl-Shift-M (or Search->Mark Selected Text if you don't like using the keyboard). The search range dropdown (bottom line of the search panel) should change to say "Marked text" when you do this.
2) In the "Mode" dropdown to the left of the search range, select "regex function".
3) Press the "create/edit" button next to the "function" dropdown.
4) Paste in the code I gave you.
5) Put "." (without the quotes) in the "Find" box at the top.
6) Press the button marked "Replace all".
7) Change the mode from "regex function" to Regex.
8) Enter "&#\d+;" in the Find box (without the quotes).
9) Press "Find" and look at what it's selected. Does it correspond to a visible character in the preview panel, or what you remember of the original text? If not, delete it and see what happens. Otherwise, replace it with whatever character it should be (whatever it was before you started). You can of course look up the numeric entity code to find out what character it is (just ask Google).
10) Repeat (9) until you get to the end of the problematic text.

[chaot]

I’ve got it! Thank you very much!

And as honestly I would like to mention that former guidance for someone who had never had to deal with these functionalities was something insufficient. This is, of course, not to charge you but the circumstances. Here worth mentioning is my still insufficient knowledge.

I start to know what’s what.

Perhaps something long-winded, but I am interested and you seem also (others do not have to read).


A S. (viewing the typed #numbers is impossible) Textsites configured! The strange character middle left does not appear as code at all.

A → Anhang
S.  →page number wxyz
S.  →page number wxyz+1

As far as the report, evaluation follows (possibly).

Do you want a text part - for your own tests? I will try to manage that.

EDIT: The whole miracles are not yet reported. I have to stop now.

[chaot]

Correction of errors and clarifications!

(Et character)#8211; is ''–'' Unicode 'EN DASH' U+2013 (written as HTML Entity (decimal)(not to be confused with EM DASH)
There are lots (dozens) of encodings/notations for something simple??? as an EN DASH.

Private word!

Spoiler:

Allow me a private word: At the moment I live in a world (Spain, Jaen province) where it is all about one thing: OLIVES! ... their oil content, harvest yields and achievable prices. Thousands of poor labor migrants (Africa & East Europe) beat the olives from the trees and gather them together. Most likely, none of them will ever know anything more about an 'EN DASH' except the pure knowledge of the existence of such a sign, possibly not even that. - - - No wonder no one can understand the world anymore, and if only as a specialist idiot.

S.
 Unicode U+F644 | 0xF644 | 
 Unicode U+F64B | 0xF64B | 
 Unicode U+F647 | 0xF647 | 
 Unicode U+F64B | 0xF64B | 

S.
 Unicode U+F644 | 0xF644 | 
 Unicode U+F64B | 0xF64B | 
 Unicode U+F647 | 0xF647 | 
 Unicode U+F64C | 0xF64C | 

A
 Unicode U+F76E | 0xF76E | 
 Unicode U+F768 | 0xF768 | 
 Unicode U+F761 | 0xF761 | 
 Unicode U+F76E | 0xF76E | 
 Unicode U+F767 | 0xF767 | 

The line height of some characters is more than twice the pure character. This can be tested: See mouse-click line. This is partly and apparently due to the notation in boxes. It is also noteworthy that the mouse-click line at  is bold and still longer (poor English, I know).


different line heights (via C&P from gedit text editor to here)

Why in the above (#1) shown paragraph (and in many others) the line heights changes (suddenly, without a recognizable reason), yet I got no explanation. However, I can tell that since yesterday evening all line heights reduced to 'normal'. For why!? Don't ask me.

If I find out, I report.

[BetterRed]

Originally Posted by chaot

Thousands of poor labor migrants (Africa & East Europe) beat the olives from the trees and gather them ...

Spoiler:

We have machines to do that.

Brit, Nordic & Canuk backpackers pick some of our strawberries, mangoes etc. For free if you give them a feed, a hut, and some yoga classes or similar. Most of them wouldn't know an en dash from a dash for the bus, even though some have a Masters in Knot Very Much, Windsurfing, etc

[Terisa de morgan]

Originally Posted by BetterRed

Spoiler:

We have machines to do that.

Brit, Nordic & Canuk backpackers pick some of our strawberries, mangoes etc. For free if you give them a feed, a hut, and some yoga classes or similar. Most of them wouldn't know an en dash from a dash for the bus, even though some have a Masters in Knot Very Much, Windsurfing, etc

Spoiler:

Machines can only knock olives from tree (and not really machines, but mechanic poles you have to keep up). But you have to get them from the floor anyway. Just the other day one of my coworkers told us at lunch because he had been the weekend "vareando" as his father has some fields with olives

[Terisa de morgan]

Originally Posted by chaot

Correction of errors and clarifications!

(Et character)#8211; is ''–'' Unicode 'EN DASH' U+2013 (written as HTML Entity (decimal)(not to be confused with EM DASH)
There are lots (dozens) of encodings/notations for something simple??? as an EN DASH.

I think you're missing a bit point here. Everybody is an specialist in their woril. I'm totally sure you don't know all the varieties of olives, or oranges, or the type of screws than can be made. You assume that, because you know the work olive, orange or screw you understand everybody.
ndash / mdash /whatever... are not a concept themselves. The concept is white spaces, that everybody understand. Yes, an oil maker won't difference between mdash / ndash or whatever but I challenge you to know all the kind of oils or olives, and being able to understand the references and types of oil...

Sorry, software is not different of anything else, the main problem is that a lot of people thinks that software should be like, I don't know, something you don't have to learn and has to be clear and crystalline for everybody to understand. If you find a matter like you pretend software to be, please, tell me.

[chaot]

Spoiler:

@BetterRed, @Terisa de morgan: I got nothing to do with olives, but I got 2 open eyes. That's why I feel justified and able to report.

There are quite different ways to get the stuff from the trees, manuale and more or less machine. And the fully automatic! This, however, may take years (olives don't fall as easy as apples from the tree) and is less interesting for the olive tree owner.

To get them from the floor, it's the same, see above. Only with a significant difference. The fully automatic version (leaving the olives rot) no longer interests anyone.

There are a kind of sweepers (about 1.50 to 2 m wide), quite a dusty matter, I am standing next to it, which gather the whole snot (ie olives, earth, dog shit, etc.). "Will get washed in the factory," said the Canuk. (Canuks are something like, in German, Kanaken?)

Can you imagine where or for what finally that oil is used?

Anyway, the olives for the better oil is always collected with manual laid out nets or with on tractors fixed frames, similar to a 180-degree turned around umbrella. It is not quite irrelevant that this, our good or 'virgin' olive oil, the life-prolonging, often contains a certain proportion of meat (olives are very gladly affected by worms, diameter about 1 mm, which got a good live for at least weeks inside the olive).

Enjoy your meal.

Concerning the issue 'how I would prefer software to be', I think about.

[Phssthpok]

Fascinating though the discussion about olive-picking may be, which I know nothing about, here is a bit more info on a topic which I do know something about...

Originally Posted by chaot

 Unicode U+F644 | 0xF644 | 
 Unicode U+F64B | 0xF64B | 
 Unicode U+F647 | 0xF647 | 
 Unicode U+F64B | 0xF64B | 

S.
 Unicode U+F644 | 0xF644 | 
 Unicode U+F64B | 0xF64B | 
 Unicode U+F647 | 0xF647 | 
 Unicode U+F64C | 0xF64C | 

A
 Unicode U+F76E | 0xF76E | 
 Unicode U+F768 | 0xF768 | 
 Unicode U+F761 | 0xF761 | 
 Unicode U+F76E | 0xF76E | 
 Unicode U+F767 | 0xF767 | 

The characters you see here are in the "private use" area of the Unicode character set, in other words a set of characters which font designers can use for whatever suits them. In this case, the font you're using seems to render them as small capitals, so having got the entities like &​#63342; (which appears to render as a small-cap N in the font you're using) do a global search & replace:
Find: &​#63342;
Replace: <span class="small-cap">n</span>

Then hit "replace all".

Now repeat this process for all the other letters which appear like this until all the problematic entities have been replaced by the corresponding letters wrapped in <span class="small-cap">...</span>. Then a couple more steps:

(1) merge adjacent small-cap spans:
Find: <span class="small-cap">(.*?)</span><span class="small-cap">
Replace: <span class="small-cap">\1

Do "replace all" several times until the number of replacements shown in the resulting message box is 0.

(2) add a rule for "small-cap" in your CSS:
.small-cap {
font-variant: small-caps;
}

[chaot]

@Phssthpok: What you think!? I am a hard working man ... ... the book is already nearly finish. I found out what stand the problematic entities for (numbers and some characters) and did S&R. Unluckily, so I lost the small caps characters: e.g. &​#63342; appears as normal Unicode N (U+004E).

Then later, I thought, I will correct this, so to speak, self-produced error manually. This is, of course, unprofessional (I hate unprofessionalism). Additionally make possibly overseeing to transform some of the former small caps.

Luckily, before I touch new books for editing I make original copies. So I am able to process your suggestions now:

I merged all Files Browser entries (3.9 MB), then did the instructions in #4 (I didn't have to create the function again, it's stored under Phssthpok). You are linked with ''problematic Unicode''
Replace all: 3.900.000 ocurrences! So I hadn't to do point 8) and 9) of the instructions: the book is full of that stuff.

New problematik: C&P your example &​#63342; doesn't work (Error: Not found). I had to find such a character in the book and C&P that one to the Find-box. Here it is:  - apparently the same!

Now I thought for @Terisa de morgan (#9): can not be ... ''clear and crystalline for everybody to understand''

Find: <span class="small-cap">(.*?)</span><span class="small-cap">
Replace: <span class="small-cap">\1 ... must be

Find: <span class="small-cap">(.*?)</span>
Replace: <span class="small-cap">\1</pan>

Your help was tremendously valuable. I wouldn't have managed that in years. Especially not so professional.

[Phssthpok]

Originally Posted by chaot

I merged all Files Browser entries (3.9 MB), then did the instructions in #4

Why merge them? You can do S&R across all text files, or selected files, or... See the drop-down box next to the Mode drop-down.

Originally Posted by chaot

Find: <span class="small-cap">(.*?)</span><span class="small-cap">
Replace: <span class="small-cap">\1 ... must be

Find: <span class="small-cap">(.*?)</span>
Replace: <span class="small-cap">\1</pan>

Er, no. Starting with this:

<span class="small-cap">x</span><span class="small-cap">y</span><span class="small-cap">z</span>

My S&R will turn this into

<span class="small-cap">xyz</span>

after 2 repetitions, while yours will turn it into

<span class="small-cap">x</pan><span class="small-cap">y</pan><span class="small-cap">z</pan>

which is invalid XHTML. You should stick to what I told you.

[chaot]

In order to give the code the deserved honor, I have followed 'all' your instructions and can report: You are 100% right.

I have been busy with these books the last time. These are SCANS from two sources, so the scan errors are also different - and almost innumerable.
S&R is only a small help, it is almost always necessary a semi-manual procedure. This is an strenuous activity, if one wants to work quickly and without flaws. (Find-Replace-Find-Find-Replace-Find, etc.) And in the end proof-reading is necessary. I like to deal with texts, so that is a pleasure for me, which would mean a nonsensical and exhausting affair to others. But my temper does not allow to do a job twice.

So I can not start again. Fortunately, the SMALL CAPS are limited to definable, known places: selected headings. Now I filter these out and change to SMALL CAPS.

At the end still a small drop of bitterness (known a few days ago): Small caps does not function in my ereader. Now I write: ANHANG.

A few images and codes of the procedure!


Original


Small Caps conversion


merging

First repetition has 104 occurences, 2nd, 3rd and 4th repetition has 52 occurences (it's clear for why, without explaining here). I test only with ''Anhang''.

  <p class="calibre1">A<span class="small-cap">n</span><span class="small-cap">h</span><span class="small-cap">a</span><span class="small-cap">n</span><span class="small-cap">g</span>   </p>

  <p class="calibre1">A<span class="small-cap">nh</span><span class="small-cap">an</span><span class="small-cap">g</span>   </p>

  <p class="calibre1">A<span class="small-cap">nhan</span><span class="small-cap">g</span>   </p>

  <p class="calibre1">A<span class="small-cap">nhang</span>   </p>

---

I am looking forward to the next time. Problems are always enough.

[chaot]

Originally Posted by chaot

Why [...][generally] the line heights changes (suddenly, without a recognizable reason), yet I got no explanation. However, I can tell that since yesterday evening all line heights reduced to 'normal'.

If I find out, I report.

Yes, so it was, but only because I deleted a particular bad character with S&R.

In #12 I said ... the book is already nearly finish. That we should call a typical misestimation!

Report (as announced): Since some time I know what justified these increased line hights. Responsible are some bad characters, here my collection (so far):                  • ■    ° (isn't º)

Some are optical easyly detectable, others not. These characters influence the line height always the whole paragraph, from <p> to </p>. As I got the 'raw material' partly unedited, some paragraphs are multiple pages.


Prinz-Eugen-Brucken Smart comment tags also limites the bad characters influence.
(Most likely (unproofed) the influence spans from >... to ...<).

So you can probably imagine that the issue made me some headaches in the first time. But even today I find, more and more rarely, new ones.

I have to assume the 'pre-editor' introduced them absolutely intentional to confuse and/or to annoy everybody, who comes after.