Cleaning bad characters - Page 2

kacir · 01-08-2008, 08:10 AM

Quote:

Originally Posted by alexxxm

wow, I'm really sorry having submitted this...
... now if only there was a way to let this thread disappear from the forum...

Do not be sorry.
In my experience, all those "fancy" "high ascii value" characters often display on my Reader PRS500 as %something. Curly quotes might be more eye pleasing, but when they do not display properly, I will gladly replace fancy, non-standard MSSoftware generated quotes, em-dashes, en-dashes, ... (ellipsis) characters and other stuf with their "vanilla ASCII" aproximation.

You might want to have a look at the software called demoroniser
http://www.fourmilab.ch/webtools/demoroniser/

JSWolf · 01-08-2008, 08:36 AM

Quote:

Originally Posted by astra_lestat

Regarding hypen and em-dashes.
I have encountered the same problem.

However, lately, I have noticed in some books there are no em-dashes, instead there are hypens with spaces before and after the hypen.
If I use Harry_T's example:

Their eyes met, and something—something unspoken but cogent—passed between them.

it is perfectly fine if you edit it like that:

Their eyes met, and something - something unspoken but cogent - passed between them.

Just don't replace em-dashes with hypen without spaces on either side of the hypen.
I am reading a hardback edition of a book right now and that's what I have in the book. No em-dashes at all.

If I use Book Designer to load and save the file, I do get the proper characters.

astra · 01-08-2008, 08:49 AM

Quote:

Originally Posted by JSWolf

If I use Book Designer to load and save the file, I do get the proper characters.

Do you mean if you use Book Cleaner that you edited for BD, right?

I use it now too

JSWolf · 01-08-2008, 08:52 AM

Book Designer does output proper curly quotes and em dashes. But you are right, the Book Cleaner files are needed in order to keep em dashes.

HarryT · 01-08-2008, 09:47 PM

Quote:

Originally Posted by kacir

Do not be sorry.
In my experience, all those "fancy" "high ascii value" characters often display on my Reader PRS500 as %something. Curly quotes might be more eye pleasing, but when they do not display properly, I will gladly replace fancy, non-standard MSSoftware generated quotes, em-dashes, en-dashes, ... (ellipsis) characters and other stuf with their "vanilla ASCII" aproximation.

Do you mean that this happens if you display plain text files? Yes, it very likely will do. However, if you convert your text files into "proper" Sony LRF files using the appropriate tools, they will then display correctly, and can significantly enhance the reading experience.

HarryT · 01-08-2008, 09:50 PM

Quote:

Originally Posted by astra_lestat

Regarding hypen and em-dashes.
I have encountered the same problem.

However, lately, I have noticed in some books there are no em-dashes, instead there are hypens with spaces before and after the hypen.
If I use Harry_T's example:

Their eyes met, and something—something unspoken but cogent—passed between them.

it is perfectly fine if you edit it like that:

Their eyes met, and something - something unspoken but cogent - passed between them.

Just don't replace em-dashes with hypen without spaces on either side of the hypen.
I am reading a hardback edition of a book right now and that's what I have in the book. No em-dashes at all.

My personal preference, when dealing with plain ASCII, is to indicate an em-dash with "--" with no spaces around it. This is the PG "standard" and works well, because it can very easily then be converted into a proper em-dash with a search and replace in an editor.

tompe · 01-08-2008, 11:03 PM

Quote:

Originally Posted by HarryT

My personal preference, when dealing with plain ASCII, is to indicate an em-dash with "--" with no spaces around it. This is the PG "standard" and works well, because it can very easily then be converted into a proper em-dash with a search and replace in an editor.

So how do they write what in LaTeX is written as --? That is used for ranges, e.g. 5--10. Or does this distinction only exist in TeX/LaTeX? It is an important distinction since when you have digit the -- should be placed higher than for an emdash.

Alexander Turcic · 01-09-2008, 03:09 AM

Quote:

Originally Posted by alexxxm

wow, I'm really sorry having submitted this...

Please don't be sorry! We are grateful for every snippet of code posted here.

HarryT · 01-09-2008, 06:33 AM

Quote:

Originally Posted by tompe

So how do they write what in LaTeX is written as --? That is used for ranges, e.g. 5--10. Or does this distinction only exist in TeX/LaTeX? It is an important distinction since when you have digit the -- should be placed higher than for an emdash.

I'm afraid I know nothing about LaTex. Personally I've always indicated a range as, for example "5 - 10". I don't know if there is any generally-accepted standard for that (outside LaTex, I mean).

Xenophon · 01-09-2008, 12:32 PM

LaTex supports emdash via ---, endash via --, and hyphen via -. In Math mode it also supports the minus-sign (not the same as the previous three) and a variety of dash-like entities of various widths and heights above the base-line. It turns out that all of these are distinct characters (yikes!), have differing uses typographically, and really ought to be supported as distinct things.

The thing tompe was asking about for ranges is an endash. Which, strangely enough, is exactly the standard typography for a range of numbers. Not to be confused with the emdash being used for setting of parenthetical clauses. Usually written in plain ascii as
"blah blah--parenthetical clause--blah blah"
but typeset with an emdash in place of the hyphen-hyphen (or sometimes space-hyphen-hyphen-space) sequence. And note that the typeset version has no spaces!

LaTex's usage makes sense, in that emdash is wider than endash is wider than hyphen. But it's at variance with the PG 'standard' which evolved out of trying to make plain ASCII look sort-of-kind-of like typesetting. By comparison, LaTex is trying to actually indicate what the typesetting should be (thus the distinctions between various kinds of hyphen-like entities).

More than anyone wanted to know, I'm sure.

Xenophon

DaleDe · 01-09-2008, 03:33 PM

Quote:

Originally Posted by astra_lestat

Regarding hypen and em-dashes.
I have encountered the same problem.

However, lately, I have noticed in some books there are no em-dashes, instead there are hypens with spaces before and after the hypen.
If I use Harry_T's example:

Their eyes met, and something—something unspoken but cogent—passed between them.

it is perfectly fine if you edit it like that:

Their eyes met, and something - something unspoken but cogent - passed between them.

Just don't replace em-dashes with hypen without spaces on either side of the hypen.
I am reading a hardback edition of a book right now and that's what I have in the book. No em-dashes at all.

Most gutenberg texts use a -- when an emdash is needed. There is little confusion when this is used except when the line happens to get split between the hyphens.

Dale

kacir · 01-09-2008, 04:07 PM

Quote:

Originally Posted by HarryT

Do you mean that this happens if you display plain text files? Yes, it very likely will do. However, if you convert your text files into "proper" Sony LRF files using the appropriate tools, they will then display correctly, and can significantly enhance the reading experience.

Up until now I did not bother with LRF files, because I strongly dislike proprietary formats and because I felt like rtf files were "the path of the least resistance". I have also developed some scripts, macros and tricks for rtf files while I was formatting books for my previous reader - Cassiopeia A-20

I can quickly write and debug macro for Vim and macro for OpenOffice.org . I even used to create rtf files and even "pocket word files" (in reality a slightly modified rtf file) "by hand" in an plain text editor (Vim) using a few scripts and templates.

I will definitely try playing with LRF files. The problem is all the tools that I have tried for creating LRF files have default settings that I dislike. Like large margins, serif font, and ... shudder ... full justification. I really prefer jagged right edge of text to uneven spaces between words, especially then there are too few words on a line - like when you view text on a 6" screen.

HarryT · 01-10-2008, 02:49 AM

Most people who create books here use Book Designer. It lets you set things up just how you like them, and once you get things set, it remembers your settings.

The other benefit of BD is that you can create multiple output formats from the same source.

01-09-2008, 12:32 PM	#25
Xenophon curmudgeon Posts: 1,487 Karma: 5748190 Join Date: Jun 2006 Location: Redwood City, CA USA Device: Kobo Aura HD, (ex)nook, (ex)PRS-700, (ex)PRS-500	LaTex supports emdash via ---, endash via --, and hyphen via -. In Math mode it also supports the minus-sign (not the same as the previous three) and a variety of dash-like entities of various widths and heights above the base-line. It turns out that all of these are distinct characters (yikes!), have differing uses typographically, and really ought to be supported as distinct things. The thing tompe was asking about for ranges is an endash. Which, strangely enough, is exactly the standard typography for a range of numbers. Not to be confused with the emdash being used for setting of parenthetical clauses. Usually written in plain ascii as "blah blah--parenthetical clause--blah blah" but typeset with an emdash in place of the hyphen-hyphen (or sometimes space-hyphen-hyphen-space) sequence. And note that the typeset version has no spaces! LaTex's usage makes sense, in that emdash is wider than endash is wider than hyphen. But it's at variance with the PG 'standard' which evolved out of trying to make plain ASCII look sort-of-kind-of like typesetting. By comparison, LaTex is trying to actually indicate what the typesetting should be (thus the distinctions between various kinds of hyphen-like entities). More than anyone wanted to know, I'm sure. Xenophon Last edited by Xenophon; 01-09-2008 at 12:33 PM. Reason: fixing grammar and clarifying example

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Book titles show up with Bad characters	Imatechie2006	Calibre	0	01-20-2010 06:18 PM
Expert help required : Cleaning bad pdf scans	Student1	PDF	12	03-03-2009 05:57 AM
Cleaning screen	brontus	Sony Reader	7	02-15-2009 05:38 PM
Cleaning the reader	pilotbob	Sony Reader	19	11-27-2007 05:41 PM

01-08-2008, 08:52 AM	#19
JSWolf Resident Curmudgeon Posts: 79,792 Karma: 146391129 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	Book Designer does output proper curly quotes and em dashes. But you are right, the Book Cleaner files are needed in order to keep em dashes.

01-10-2008, 02:49 AM	#28
HarryT eBook Enthusiast Posts: 85,547 Karma: 93383099 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	Most people who create books here use Book Designer. It lets you set things up just how you like them, and once you get things set, it remembers your settings. The other benefit of BD is that you can create multiple output formats from the same source.

Advert

Advert