END OF GUARDED AREA character

[phossler]

Something weird

I had 'box' char in an ePub, which usually means a missing or a undisplayable character. The editor's char display says it's END OF GUARDED AREA.

Google says that it's Unicode 97 (START is 96)

Where did the 97's come from (I don't think it's the editor) and how can I get rid of them?

[kovidgoyal]

Do a search replace for \u0097

[phossler]

Thanks - did it the hard way by (finally) copying the invisible character and pasting it into Find. I'll use the right way the next time

Any idea where they came from and how they got into the ePub?

I thought they might be from the html I loaded, but still no idea what they might have been used for

Is that something that Smarten Puncuation should fix?

[Tex2002ans]

Originally Posted by phossler

I thought they might be from the html I loaded, but still no idea what they might have been used for

Looks like "End of Guarded Area" (U+0097) is an ancient control character... and according to this explanation:

http://blog.codekills.net/2011/01/22...ry-bad---151-/

it looks like someone wrongly thought it was an em dash (because some renderers display it that way).

So do a search for that character like Kovid said, and then replace it with the actual EM DASH —.

[Brett Merkey]

copying the invisible character and pasting it into Find

The Replace should have been an em-dash to keep the intention of the original text.

[phossler]

Ah rats — I replaced it with space — I though it looked funny

Never though about an em-dash

I can get an unmodified copy from my backups and do it the right way

Thanks everyone

[Tex2002ans]

Originally Posted by phossler

Ah rats — I replaced it with space — I though it looked funny

Never though about an em-dash

Em Dashes are sometimes used this way in Fiction to interrupt (and then continue) dialogue:

See one article on the topic, "When a Comma Isn't Enough":

The recommendation in The Chicago Manual of Style is quite clear.

In the seventeenth edition, 6.87, we find:

There’s even more explanation:

And an example:

It looks like your book decided to put em dashes within the quotation marks:

“It precisely—” he glanced down at the watch strapped English-style to the underside of his free wrist “—one and one-half minutes [...].”

Originally Posted by phossler

I can get an unmodified copy from my backups and do it the right way

What was the format of the source document?

[phossler]

I don't remember the original format, most likely HTML since this was a PD book.

I might have 'misplaced' the em-dashs during an earlier edit session since they didn't show

Agree: Now that I knew what I was dealing with, when I went to an archive b/u copy, and re-edited it, I made sure to put the em-dash outside the quote marks of the dialog that they were breaking

I'm still curious to know what the character is actually used for

[Brett Merkey]

I made sure to put the em-dash outside the quote marks of the dialog that they were breaking

Not to extend your em-dash agony—

From this page:
https://mybookcave.com/authorpost/pu...otation-marks/

Finally, em dashes only go inside quotation marks for dialogue, to show that the speaker is abruptly cut off.

Em-dashes are such fun—at least in English. They do not follow the rules of other punctuation.

[Tex2002ans]

Originally Posted by Brett Merkey

Not to extend your em-dash agony—

[...]

Em-dashes are such fun—at least in English. They do not follow the rules of other punctuation.

Yes, I agree. They should go on the inside. (I didn't like that specific Em Dash form in the Fiction Editor's article, even though I agree with her recommendations in 99.9% of her other articles.)

And you should always err on matching the original source when you can... it's not a good idea to make a broad style change like that for no reason.

I also reference the Wikipedia page on Em Dashes quite often:

https://en.wikipedia.org/wiki/Dash#Em_dash

if you want to know about different usages.

Originally Posted by phossler

I'm still curious to know what the character is actually used for

Dang, I had a whole giant technical post written about it... but then I deleted it because I misread an earlier post.

I'll try to piece together some of the research I found on it.

It seems like "Start of Protected Area" and "End of Protected Area" were from the ol' terminal days:

https://en.wikipedia.org/wiki/C0_and_C1_control_codes

which says were used in Block-Oriented Terminals:

https://en.wikipedia.org/wiki/Block-...ented_terminal

It was used while filling out forms and things like that, so that you couldn't go outside of the bounds (imagine limiting to a 16 character last name, trying to type a 17th character wouldn't let you + make a beep).

It was also used so that certain blocks of text wouldn't go scrolling off the screen as the terminal moved down.

A lot of those old control codes were included in Unicode for legacy reasons, but outside a small handful, the rest are barely used nowadays.

[phossler]

Originally Posted by Brett Merkey

Not to extend your em-dash agony—

From this page:
https://mybookcave.com/authorpost/pu...otation-marks/

Good reference -- thanks

It would seem that the only em dash rule is that there is no em dash rule

However, if you have dialogue interrupted with a dialogue beat, you can show that with the em dashes outside the quotation in pairs. Read more about dialogue beats in our previous posts 8 Tips for Punctuating Dialogue and em dashes in Correct Em-Dash Formation for Ebooks.

Right: "You think"—he swallowed hard—"she did that?"
Wrong: "You think—" he swallowed hard "—she did that?

It would seem that in THIS case ...

  <p>“In precisely” — he glanced down at the watch strapped English-style to the underside of his free wrist — “one and one-half minutes ........ ”</p>

the em dashes should be outside the quotes (I think)

But I believe that the original had them inside. Since I edit just to please myself and make a book easier for me to read on the Kindle, I'll go with them outside for things like this

[phossler]

@Tex2002ans -- thanks for the info. I'm going to assume that the EOPA was intended to be an em dash and not some weird old control character

[BetterRed]

@phossler - I run Reports->Characters before I start editing an epub to check if it came with any spurious characters, and again when I finish editing to check I've not introduced any of my own.

An epub reader on a 3270, I'd like to see that

BR

[phossler]

@BR -- good tip. Would have saved a lot of head scratching.

That EOPA char was 0-width and invisible in the edit window. At least it showed as an empty square in Preview, or I never would have know I had a problem

[Tex2002ans]

Originally Posted by phossler

@BR -- good tip. Would have saved a lot of head scratching.

That EOPA char was 0-width and invisible in the edit window. At least it showed as an empty square in Preview, or I never would have know I had a problem

In Sigil, if you doubleclick on the character in the report, it should insert it into Find.

In Calibre, it doesn't put it in Find, but it does jump you to the next spot the character occurs.

Side Note: And I also take a quick look at that Report on every book for any strange characters. Usually things like soft hyphens stand out (if the thousands of red squigglies didn't give it away).

One of the latest books I worked on accidentally had the Cyrillic letter С instead of the Latin capital C.

Originally Posted by phossler

@Tex2002ans -- thanks for the info. I'm going to assume that the EOPA was intended to be an em dash and not some weird old control character

I can't find the link now, but a few of the sites discussed the technical encoding issues between:

- Windows-1252
- ISO-8859-1
- Unicode

While they're mostly the same... the obscure control code points just so happen to be where many differences lie. So when you make (wrong) assumptions about encoding:

EN DASH (original) -> "Start of Protected Area" (Unicode)
EM DASH (original) -> "End of Protected Area" (Unicode)

Programs botch encoding along the way!

https://stackoverflow.com/questions/...h-151-and-8212
https://stackoverflow.com/questions/...nicode-in-java
https://unix.stackexchange.com/quest...cter-in-a-file
https://stackoverflow.com/questions/...rea-characters

Doesn't help that many browsers/renderers also decide to be helpful and assume you were a dunce... and display those characters instead of keeping them invisible (look at the "Browser" column):

https://www.fileformat.info/info/uni...ement/list.htm

So it can easily still LOOK like an EM DASH (U+2014), even though under the surface it's the END OF GUARDED AREA (U+0097).