What "Cleaning Up" Do Project Gutenberg Texts Need [closed]

[vivaldirules]

Originally Posted by Robert Marquard

Distributed Proofreaders

Now there's a group that could use some help. It certainly benefits us here if the texts at Gutenberg are accurate. Think I'll sign up to volunteer. Thanks for mentioning it!

[kacir]

Originally Posted by vivaldirules[I

[COLOR="Navy"]
> 1. get rid of that ugly legalese at the top of the file.
> 2. make the title-page and front-matter look nice.
> 3. hotlink the table of contents. make one if necessary.
> 4. make all the headers big, bold, and distinctive, and
> 5. start chapters on a new page, maybe even a recto.
> 6. get rid of the empty lines between paragraphs, and
> 7. use book-style indents on each paragraph instead.
> 8. use full justification. or at least half-ragged.
> 9. use a reasonable line-width. full-screen is too wide.
> 10. white-space is free in an e-book, so use it liberally.
> 11. make block-quotes distinctive, for remix purposes.
> 12. links are great, but spare us the ugly blue underlines.
> 13. is an unlucky number.
> 14. don't put pagenumbers inside the text/paragraphs.
> 15. turn pg-ascii underscored text into _real_ italics.
> 16. pictures (even doodad thingees) enliven the text.
> 17. navigation aids among chapters are quite useful.
> 18. footnotes should have links going _both_ ways.
> 19. if it works better that way, turn a table on its side.
> 20. resize tables and images so they fit on one screen.
> 21. give your readers the luxury of generous leading!
> 22. block-quotes should be indented on the left and right.
> 23. create running heads and/or footers on each page.
> 24. (leaving some space for you...)
> 25. (leaving some space for you...)
> 26. show where we are in the book (page 39 of 208).
> 27. make the framework of the document _obvious_.
> 28. what the heck, just for the fun of it, make an index!
> 29. make the typesize big enough to be read easily!
> 30. get rid of that ugly legalese at the bottom of the file.

At point 1.
I do agree that that legalese is ugly.
I do not think, however that we should remove it entirely.
I suggest moving it to the end of the file and placing a line like this "for licence see bottom of file" instead.

At point 4.
I personally do not like fancy headers. When I format book for myself I remove all fancy formating from headers.

at point 8.
This is again a matter of personal preference. Fully justified text does *look* better. When it comes to hours and hours of continuous reading many people find those "ragged" left justified paragraphs easier to read.

point 23.
Here again I have to politely disagree with you

point 24.
set page margins as small as possible. We have paid a lot of money to have the screen as big as possible. So why do we want to waste 20% of the screen real estate by margins. Wide margins make sense in a printed books. Reader, however, has "built in margin" around the screen. This is the issue that bothers me the most when I see e-book from the connect store

point 25.
Use sans serif font. Just like with fully justified text the serif font looks better in a printed books. In a low resolution display (and anything below 300dpi *is* low) the sans-serif font is much more readable.

point 30.
see point 1.

If you want to use Microsoft Word for formating the documents, beware. Microsoft products use symbols (like left and right quotes) that are not according to standards. This is most notable when you use standards compliant browser (like Firefox) to view html page that was generated using MS Word. Or, if you have text with fancy curly quotes and you upload it an an rtf file to the reader. Plain, simple, basic quote " does not look as nice as typographical one, but it will display correctly on any reading device.

[Nate the great]

@kacir
We remove the license stuff because we do not wish to agree to the license.

[JSWolf]

The problem (in my opinion) with ZML is that it's non-standard. There are NO converters that will read it and convert it into something that can then be used to generate a book that will be readable on a portable book reader. Book Designer won't read it, libprs500 won't convert it. So non of the devices we have... cell phones, PDA, eink readers will actually have anything to do with ZML. So why not just do the cleanup of the PG text and output standard HTML that is 100% adequate for the task at hand and thus can either then be read as is or converted easily to be read on a portable reading device? What is is about ZML that make it that much better then HTML? And what is there in a PG text that HTML cannot handle?

bowerbird, you do make some valid points about the consistency (standardization) and some of the problems with PG text. And then you go wanting to use ZML which is not a standard at all that nobody can use to convert to something they can actually read. Can you see why we have a problem here? You are trying to fix something that may possibly be broken and add a layer of inconsistency to that so it comes out even more broken then before you started.

[JSWolf]

Originally Posted by Nate the great

@kacir
We remove the license stuff because we do not wish to agree to the license.

I thought we removes the license stuff because it is easier to remove then to format it so it looks decent.

What I do is keep the PG stuff at the beginning of the book as it's only one page. I do remover the multi-page license at the end. I have (in the past) formatted it so it looks decent and is more readable. But nobody (IMHO) is really going to bother to read it.

[tompe]

How do you represent the semantical structure with HTML?

I am suprised that no "exchange format" has been used for conversion and that format put up in the book section here. Now I find books formatted for Sony that looked interesting but I wanted Mobipocket format but they were not available,

[jbenny]

Originally Posted by bowerbird

jbenny said:
> Two existing tools that do most of what you want: Gutenmark and HTML Book Fixer.

um, no. not even close, really. honorable efforts, but my intention is much wider.

-bowerbird

Boy, I can feel your disdain all the way across the internet. The two tools I mentioned may not be perfect, but they already exist and are very useful. When your vaporware program reaches perfection and is actually available, let me know and I'll give it a try.

[DaleDe]

Originally Posted by JSWolf

I thought we removes the license stuff because it is easier to remove then to format it so it looks decent.

What I do is keep the PG stuff at the beginning of the book as it's only one page. I do remover the multi-page license at the end. I have (in the past) formatted it so it looks decent and is more readable. But nobody (IMHO) is really going to bother to read it.

Having it at the beginning is problematic for many ebook readers that can only jump back to the beginning of a document. the eb1150 requires a tag name="toc" to allow it to find the toc and many people leave it out so then it has the same problem. The TOC needs to be near the beginning to that it can be reached easily. It should not be 10 pages in! Therefore I vote for moving it or removing it.

Dale

[DaleDe]

Originally Posted by tompe

How do you represent the semantical structure with HTML?

I am suprised that no "exchange format" has been used for conversion and that format put up in the book section here. Now I find books formatted for Sony that looked interesting but I wanted Mobipocket format but they were not available,

The exchange format should be xhtml and its logical container epub. This is what we should promote and focus on. BD's format html0 is horrible html and coming up with another one is silly IMHO since we have one already. We should be active in ensuring the xhtml does what we need and epub does what we need.

Mobipocket is committed to importing epub even if they don't support it natively.

Dale

[JSWolf]

Originally Posted by kacir

At point 1.
I do agree that that legalese is ugly.
I do not think, however that we should remove it entirely.
I suggest moving it to the end of the file and placing a line like this "for licence see bottom of file" instead.

If we move to the end, we can put in a ToC entry to it in case anyone does want to read it.

At point 4.
I personally do not like fancy headers. When I format book for myself I remove all fancy formating from headers.

My way of doing it.. main text in serif font and headers in san-serif and larger with bold (I think). Nothing fancy is needed. Just an easy way to let us know this is a chapter title.

at point 8.
This is again a matter of personal preference. Fully justified text does *look* better. When it comes to hours and hours of continuous reading many people find those "ragged" left justified paragraphs easier to read.

Personally I prefer justified with small margins or no margins (on the Sony Readers)

point 23.
Here again I have to politely disagree with you

I do have headers and /or footers for the books I create in LRF. But there i no need for these in the actual text. It just makes it like a converted PDF where we then have to strip it out. In my case Book Designer or html2lrf creates the headers/footers without them being in the text.

point 24.
set page margins as small as possible. We have paid a lot of money to have the screen as big as possible. So why do we want to waste 20% of the screen real estate by margins. Wide margins make sense in a printed books. Reader, however, has "built in margin" around the screen. This is the issue that bothers me the most when I see e-book from the connect store

I have been taking to setting the margins to 0 as I like to have as much on screen as I can fit. And it takes less pages and less battery to read a given book that way. Wide margins annoy me. On a paper book, there are reasons for the margins and they are fine. But on a reading device, it makes n sence to have wide margins.

point 25.
Use sans serif font. Just like with fully justified text the serif font looks better in a printed books. In a low resolution display (and anything below 300dpi *is* low) the sans-serif font is much more readable.

This is personal preference. I like the serif better.

If you want to use Microsoft Word for formating the documents, beware. Microsoft products use symbols (like left and right quotes) that are not according to standards. This is most notable when you use standards compliant browser (like Firefox) to view html page that was generated using MS Word. Or, if you have text with fancy curly quotes and you upload it an an rtf file to the reader. Plain, simple, basic quote " does not look as nice as typographical one, but it will display correctly on any reading device.

The MS Word generated curly quotes do work in Book Designer for generating LRF and Mobi format books.

[bowerbird]

jbenny said:
> Boy, I can feel your disdain all the way across the internet.

you know, i _really_ dislike it when people try to put words in my mouth,
or try to tell other people what _i_ think, or how _i_ feel about something.

"honorable" is _not_ a term of "disdain", not in my neck of the woods.

and i've had lots of friendly discussion with ron burkey, who made gutenmark,
so i really resent that you'd try to soil the pleasant nature of our relationship...

ron would tell you himself that my scope is entirely different than his was...
he would also tell you -- as he told a lot of people -- that as gutenmark grew
better and better, his ambitions grew even faster, so it "fell short" of them...

and since he'd gotten tired of maintaining it -- an important consideration,
wouldn't you say? -- he was very supportive of me and my similar efforts.

***

as for the other tool you cited, it's aimed at .html files, so its focus is limited.
and even more importantly -- to me -- it's windows-only. so, as a mac person,
it _doesn't_exist_ for people like me. my tools are cross-platform, thank you,
mac and windows, and i'll even make linux versions if i get some beta-testers.

***

plus, your characterization of my work as "vaporware" is an entirely vapid insult.
if you really want to "give it a try", then visit the websites i pointed to up above.
because when you call something "vaporware", and someone else can simply
\visit a website and see it in action, it makes you look kind of... well, _stupid_.

anyway, where's a moderator when some _real_ disrespect is being manifested?
(and no, that's not a call for a moderator. it's better to let rudeness reveal itself.)

-bowerbird

[Nate the great]

Since bowerbird is going off in his own little world, I decided to create my own autoformatter for PG ebooks. My planned output format is very basic html. It's the closest to a universal format right now. I think I will be able to implement most of the suggestions in the first post of this thread.

Originally Posted by DaleDe

The exchange format should be xhtml and its logical container epub. This is what we should promote and focus on. BD's format html0 is horrible html and coming up with another one is silly IMHO since we have one already. We should be active in ensuring the xhtml does what we need and epub does what we need.

Mobipocket is committed to importing epub even if they don't support it natively.

Dale

One problem with xhtml as an exchange format is lack of backward compatibility. (I also don't know what the tags are, so if you could point me at a tutorial I would appreciate it.) On another point, does any device use epub yet?

[kovidgoyal]

Since I have a little experience writing converters, I'd just like to say that if somebody does write a new improved gutenberg to html converter to use a well defined semantic scheme by CSS classes. This would make the HTML much more suited to conversion into a ebook format like epub or LRF.

Some important things to have in the generated HTML would be

1. A meta tag identifying the type of file (i.e. identifying it as the output of that automatic converter). This is necessary for parsing the semantic information.

2. CSS classes for things like page breaks, chapter titles, chapter subtitles, inline vs. block vs full page images.

3. Use of semantic HTML tags like <em>, <strong> instead of <bold> and <i>

etc.

[tompe]

And it would be nice if new tools are platform independent.

[jbenny]

Originally Posted by Nate the great

Since bowerbird is going off in his own little world, I decided to create my own autoformatter for PG ebooks. My planned output format is very basic html. It's the closest to a universal format right now. I think I will be able to implement most of the suggestions in the first post of this thread.

One problem with xhtml as an exchange format is lack of backward compatibility. (I also don't know what the tags are, so if you could point me at a tutorial I would appreciate it.) On another point, does any device use epub yet?

Not a tutorial, but a very nicely organized reference: http://xhtml.com/en/xhtml/reference/

I agree with DaleDe that the use of XHTML makes the most sense. For those who don't know how it differs from HTML, the reference I mentioned above will tell you exactly what is supported. XHTML is not some strange new beast. It is simply a more formalized version of the HTML that we all know. Some of the bad habits of HTML have been eliminated, some tags deprecated and a consistent structure is required. There is XHTML 1.0 Transitional, XHTML 1.0 Frameset, XHTML 1.0 Strict and XHTML 1.1. The latest 1.1 version is essentially 1.0 Strict and continues the process of clearing out some of the crud from HTML. There is a good reason that the IDPF folks specified XHTML 1.1 for use in epub.

Oh, and there is no reason that XHTML can't be backwards compatible, since it is really just HTML that has been cleaned up a bit. Even on such non-closed HTML tags like <br>, in XHTML you can use either <br/> or <br />. The second version should work correctly in older browsers.

The only major issue with XHTML is the use of <?xml version="1.0" encoding="utf-8"?> at the start of each document. This is the proper declaration to use, but Internet Explorer in particular has a problem with this and goes into "quirks" mode. If you need IE compatibility, you can leave this line out. For other purposes (especially epub), you should have it.