Fixing Up Typography

[ahi]

Greetings,

I find reading eBooks with sloppy or otherwise subpar typography to be unpleasant. However I have no crazed notions that volunteer editors/converters owe me any specific level of typographic sophistication, and I accept that I will likely have to take matters into my own hand.

My question is, am I better off converting existing eBooks into some editable format (like Project Gutenberg's EPUBs), do my fixing, then convert them back; or is it better to just work straight from the plaintext and make my own eBooks from scratch?

Obviously I am looking to minimize my work and maximize the typographic quality. So please answer my question with that in mind. I should note though that I already tried calibre, and I don't think its conversion feature goes far enough for what I want.

I have this dream about creating a clever Python parser/converter that takes Project Gutenberg text files and creates typographically gorgeous 4.9" by 6.9" PDF files from them via XeLaTeX (to take advantage of a few high quality OTF fonts I have). However any such project will doubtless require a lot of work before it gets past the point of every new file thrown at it resulting in flawed output (i.e.: until the source code is expanded to account for said new files own little idiosyncrasies).

So in the meantime though, I would be grateful for any tips, tricks, and suggestions that are more grounded and achievable.

Sincerely,

AHI

[JSWolf]

ePub will allow you to specify the fonts you want to use. You can try that and see. Have a look at one of the stickies in the ePub section on how to do it.

[ahi]

Originally Posted by JSWolf

ePub will allow you to specify the fonts you want to use. You can try that and see. Have a look at one of the stickies in the ePub section on how to do it.

Is it also reasonably straightforward to play around with things like pagination, indents, footnotes, emphasis (via at least italics and bold), et al? The fonts are a plus, but ultimately it's what's done (or not done) with them that tends to bug me in books out there.

I want my eBooks to be (at least nearly) as professionally typeset as my dead tree books, and I'm just hoping there a less painful way than simply doing it all for myself manually or semi-manually (by writing a program for it).

[JSWolf]

There is no reader out there that will do it in a professional way other then doing how you want in a screen sized PDF. The software is the issue. The hardware can do it fine.

[pepak]

Originally Posted by ahi

My question is, am I better off converting existing eBooks into some editable format (like Project Gutenberg's EPUBs), do my fixing, then convert them back; or is it better to just work straight from the plaintext and make my own eBooks from scratch?

It really depends on the book. I have often found that, ironically, taking the paper book and scanning/ocring/fixing it myself is less work than taking a "finished" e-book and fixing that. In other cases, starting with plaintext (or stripping all tags from HTML) and fixing that can give a quite satisfactory result.

In any case, I very much doubt you will find a way to do the fixing fully automatically (by running a script) - at least I haven't, yet, and I have been trying to do it since I bought the reader last march. The closest I got is a semi-automatic process of checking each book manually and devising proper regular expressions to fix its quirks.

[rogue_ronin]

The problem is the markup, or lack thereof. The lack of information is hard to undo.

Many, if not most, of the source files that I find are plain text that someone may have hurriedly converted to a basic HTML, with little to no effort to provide accurate markup, or with their personal quirks embedded (I just looked at two files that had a blank paragraph following every paragraph -- why?! And I only realized that after de-crufting the files.) Often it's just text wrapped in <PRE> tags, unjoined paragraphs, or encrufted by the dreaded Word. There has been no effort to mark chapters, italics, bold, titles, subtitles, images, maps, tables, etc. There's rarely, if ever, any meta-information.

If it comes from Gutenberg, it will likely have (but may not) hard-returns to mark the chapters and titles. It may (or may not) have simple, textual markup tags like _ for underlining, * for bold, and ~ for italic -- although it varies from book to book. Gutenberg text files have hard line-wrapping as well. (Their HTML versions are far superior, especially since the Distributed Proofreaders days. I don't include them in this critique.) At least the Gutenberg stuff can be parsed -- check out Gutenmark. It's GPL, so you could probably start with its source code.

There are some remarkably clever simple textual markup systems. Check out Markdown, for one. The problem is that no one uses them. And you're left trying to redo something that was improperly done.

Certainly, virtually anything you find that didn't come from a professional source originally has lost its markup in the conversion. (And I'm of the mind that professional is an attitude, and a commitment, not just getting paid.) You sound like you want to do pro work. Most people just want to read, seem to have no sense of style, and don't think about all the information that they're losing when they make a crappy text file. Somewhere, (Yahoo books group?) I remember, I was feeling compelled to post to people about stripping markup from HTML files and redistributing them -- these folks actually think that they're doing a good thing! I believe they started reading books on their old TI calculators, and never moved on. (It's easy to strip markup, hard to add it. Destruction/Creation, Entropy/Life -- which side are you on?)

I don't know if it is possible to accurately program a utility for all the stupid ways people cruft up their files, but I'm hopeful. Maybe an AI could do it. (Come on, Skynet!) I'm looking for something myself that will just simplify the (X)HTML to basic <P align>, <H# align>, <BR>, <B>, <I>, <U>, <IMG>, <A>, <A HREF>, <CODE> and <BLOCKQUOTE> (is anything else necessary for most books?) -- adding styles later is much easier. (BTW, FB2 is an excellent schema for books, check it out. It adds cool stuff, allowing for poetry, etc.)

Oh, there's a nice utility called txt2html that looks promising for reasonably formatted text files. Millions of options.

I hope you can make such a utility. I'd gladly use it, especially if it were scriptable. Or heck, just use some combination of smart utilities like HTMLtidy, txt2html and hstrip (mentioned in link.) Add your own genius. Most of this stuff starts out in Perl, which you seem to understand. (I don't.)

Hope I've helped, and not just added to the noise,

m a r

[pepak]

Originally Posted by rogue_ronin

I'm looking for something myself that will just simplify the (X)HTML to basic <P align>, <H# align>, <BR>, <B>, <I>, <U>, <IMG>, <A>, <A HREF>, <CODE> and <BLOCKQUOTE> (is anything else necessary for most books?)

Generally, I keep to the tags you described, except that:
- I don't use attributes if I can avoid them
- I use <strong> instead of <b> and <em> instead of <i>
- I use <whatever id="..."> instead of <a name="...">
- <span> and <div> can be useful for styling
- I make sure to add class="..." to everything that can be reasonably identified, e.g. <blockquote class="poem"> or <em class="thought"> or <a class="note" href="#note-1">1</a>

[rogue_ronin]

Hmmm,

I tend to think in old-school ways. And I want to use the minimum I can get away with, without losing presentation. But I'm not an expert, just a hack; I dig the logical-expression mode, and I can grow. 'Course we're (or at least, I'm) also talking about adapting/stripping found files, not starting from scratch. They may have these tags already, and only need to be simplified. But I guess if you can find them and simplify them, you can translate them too.

-- attributes: the formatting is all I'm after -- is there a better way, in a single tag, to indicate alignment? Without CSS?
-- <strong> an <em> are fine with me, too. (But my REB1100 likes <b> and <i>... rbmake fixes that.) I still find <i> and <b> more human readable. (Which is a major part of my goals in reformatting.)
-- <whatever id="..."> is XML, right? Why is this better? I assume one can link to the created/named tag?
-- I've never understood <span> and <div> more than generally as a sort of box to drop things into. Chapters, paragraphs, etc are already containing almost everything. Care to elucidate (with an example maybe?) Most of the time I see them in Microsoft cruft, and it's a lot more readable to convert them to <i> or <b> -- or delete them as they don't do much more than repeat font info, or language, or some other useless overload. Occasionally, they suggest alignment.
-- I actually do that too, with the final product (ie: class="chapter", class="subtitle"), but aren't these attributes? I do it so that if someone wants to convert/strip my HTML, it's easy to find. But I don't drop it on every paragraph or blockquote. I can see why I might do it on a blockquote, though.

BTW, I just tried HTMLtidy and hstrip in sequence on a crufty file, and it did pretty well. HTMLtidy reordered the attributes to alphabetical for me, along with a lot of other stuff, so that it will be easier to replace similar paragraph tags. And hstrip did a great job of ditching CSS and javascript. The question now is "Did it eat any formatting?" That'll take longer to decide.

m a r

[pepak]

Originally Posted by rogue_ronin

-- attributes: the formatting is all I'm after -- is there a better way, in a single tag, to indicate alignment? Without CSS?

Nope. It's just that I am bent on using CSS whenever possible, because it can be easily centralized into one file.

-- <whatever id="..."> is XML, right? Why is this better? I assume one can link to the created/named tag?

Yes, you can link to any tag. It allows me to minimize the number of tags, e.g.:

<h2 id="chapter-5">Chapter 5</h2>
...
<a href="#chapter-5">jump to chapter 5</a>

vs.

<h2><a name="chapter-5">Chapter 5</a></h2>
...
<a href="#chapter-5">jump to chapter 5</a>

-- I've never understood <span> and <div> more than generally as a sort of box to drop things into.

Exactly. I use them to style things where I want the style centralized and can't use a better tag (e.g. because it's not allowed), or where I need the style to span multiple tags:

.story { font-style: italic; }
.story em {font-style: normal; }
<div class="story">
<p>Lorem ipsum <em>blabla</em> etc. etc.</p>
<p>More text with <span class="underline">underlined</span> text.</p>
</div>

Also to mark sections:

<div class="chapter">
<h2 id="chapter-2">Chapter 2</h2>
<p>...</p>
<p>...</p>
</div>

-- I actually do that too, with the final product (ie: class="chapter", class="subtitle"), but aren't these attributes? I do it so that if someone wants to convert/strip my HTML, it's easy to find. But I don't drop it on every paragraph or blockquote. I can see why I might do it on a blockquote, though.

It depends. I do mark e.g. telepathy/thoughts-to-self with <em class="thought"> or emphasised foreign (or intentionally misspelled) words with <em class="foreign">, but I do leave "normal emphasis" classless.

[Jellby]

Originally Posted by ahi

My question is, am I better off converting existing eBooks into some editable format (like Project Gutenberg's EPUBs), do my fixing, then convert them back; or is it better to just work straight from the plaintext and make my own eBooks from scratch?

I usually work from the plain text version or, at most, from HTML.

I have this dream about creating a clever Python parser/converter that takes Project Gutenberg text files and creates typographically gorgeous 4.9" by 6.9" PDF files from them via XeLaTeX (to take advantage of a few high quality OTF fonts I have).

I once had a similar dream, but it's too hard. Source files can use different conventions, and not everything is automatically parseable, how would the parser know whether "an'" has an apostrophe or a closing quote?

[HarryT]

I gave up on "curly quotes" a long time ago, for that very reason. If the source has them, fine. If it doesn't, it's a virtually impossible task to programatically get them right; you certainly can't assume that they always occur in pairs!

[ahi]

Thanks for all the helpful comments! It is good to know I am not the only one bemoaning carelessly typeset eBooks.

It seems to me that GutenMark (as per this page: http://www.sandroid.org/GutenMark/features.html) does more or less what my dream Python script ought to do. Admittedly I am not thrilled with the resulting PDFs, particularly not from an eBook reader device viewing standpoint. However, it seems to me that either (1) I ought to be able to adjust the LaTeX files to have the PDFs better match my taste, or (2) the sort of script I was thinking about is [as evidenced by GutenMark] reasonably possible to do and the GutenMark source might yield many a tips and tricks toward that.

Let me also add, that smart quotes are not as difficult as they may first seem. It is not something that can be blindly done by regexes; but a parser that treats the content as a stream and keeps in mind its current states (i.e.: am I currently in a quoted portion of the text?) ought to be able to fairly reliably restore them. Assuming of course that the original text did not take too many liberties. Most lack of matching front and back quotes is due to the convention of using only a back quote (and not a front one) on non-first paragraphs of a multi-paragraph quote by a single speaker.

I'll share any great discoveries, epiphanies, and gorgeously typeset books this endeavour may lead to.

Thanks again!

AHI

[HarryT]

Originally Posted by ahi

Most lack of matching front and back quotes is due to the convention of using only a back quote (and not a front one) on non-first paragraphs of a multi-paragraph quote by a single speaker.

Yes, it was specifically that that I had in mind. That "breaks" all the "change to smart qotes" scripts that I've personally tried.

[JSWolf]

Originally Posted by HarryT

Yes, it was specifically that that I had in mind. That "breaks" all the "change to smart quotes" scripts that I've personally tried.

There are cases where I've done the manual search/replace to fix quote and apostrophes. That works fine. Takes not too long overall to do it by search/replace.

[pepak]

Originally Posted by Jellby

how would the parser know whether "an'" has an apostrophe or a closing quote?

Actually, quotes, are quite doable using regular expressions. This regexp won't work 100%, but it will work most of the time:

Search = ([>_])’(.*?[^a-z_])’([<_])
Replace = $1opening_quote$2closing_quote$3

(note: underscore in the search string represents a space)