What "Cleaning Up" Do Project Gutenberg Texts Need [closed]

[GregS]

bowerbird No worries, I am not about to burst in and tell people what to do. I will watch for a while, make a few small suggestions, and if other things work out, maybe present them with some tools to make life easier.

The thing is what they are doing is almost exactly what I had in mind. There is no telling them what they should be doing, though I might have something to say on IDs and xml-namespace use.

I use Gutenberg a lot, but I usually go there to find something specific, I totally missed the tei side of things. Ie I usually get to the text without dallying elsewhere on the site (I have been using it for well over four years).

You have been of genuine help, and I thank you. I am still confused over Z.M.L if you like under a promise of strict confidentiality maybe you could send something to my email address and I could discuss it privately with you. I am intrigued, I don't doubt the efficiencies you claim. The reference I previously followed was a error (server problems probably). I am however no fan of python, I don't see myself ever using it again (too many special syntaxes).

[bowerbird]

greg said:
> You have been of genuine help, and I thank you.

that's what dialog is all about.

it can only happen if both sides hold up their half,
so you deserve as much credit as you're giving me.

another note, if you wanna mush pgtei and epub together, is
that some of the personalities in those various camps do not
get along particularly well with each other, just so you know.
the only thing they seem to agree on is they both hate me... :+)

there are other people bridging the gap, so it's not a gulf
which is impassible, but feel your way carefully at first...

-bowerbird

p.s. i'm using perl (*.pl). that's just for web stuff though.
my fav language is basic, specifically the realbasic compiler.

[GregS]

Whoops I misread .pl as .py, sorry for the confusion (ps. if I find Python horrible, you can imagine how I feel about Pearl).

I very much see epub as a handy standard for eink displays etc.,. much better than the pig's breakfast of different formats, and at least predictable enough for translation to other things.

Ebub can't possibly do what TEI can, but it also does not have the overheads, everything is a trade-off.

We do need a standard, it does not have to be incredibly good, but it will do for ebook reading - epub fits the bill, it is more important at this time to get the HW vendors all on the same track in order to pull publishers and public together.

Anyhow, it leaves plenty of room for other solutions. At the moment I see epub as an easy root for traditional publishers to move into the digital market, without s steep learning curve and able to consolidate sales in a single format. I see TEI as having a key role in text repositories of all kinds (text in the sense of literature), however this does not stop other forms being adapted there are things badly displayed by HTML, that may well benefit from Z.M.L. if it is easy to code in, compact and efficient that could really be a killer in many areas being badly served by HTML. Just a suggestion.

[HarryT]

Originally Posted by bowerbird

kovidgoyal, i have substantial replies to your previous posts,
which i would like to post, but i don't want to _monopolize_
the conversation here. i'd like to give other people a chance.
when two people overtake a thread, it can get boring fast...

Go ahead and post; even though it's Kovid who asked the questions, I'm sure that the answers will be of interest to others. Anyone who doesn't wish to read this thread can simply ignore it.

If your response is a personal reply to Kovid only, and of no relevance to anyone else, using PMs is probably best.

[Moderator]

[bowerbird]

greg said:
> ps. if I find Python horrible, you can imagine how I feel about Pearl).

to me, they're all just tools. i had to go to a scripting language because
realbasic only creates offline apps, and i wanted to do web equivalents...

i use only the most basic of programming concepts -- mostly for/next
and split routines -- so any language is capable of doing what i want...


> We do need a standard, it does not have to be incredibly good,
> but it will do for ebook reading - epub fits the bill, it is
> more important at this time to get the HW vendors all on
> the same track in order to pull publishers and public together.

i think those are all shibboleths. but there's no need to discuss that.

***

harry said:
> Go ahead and post

i did, eventually, after giving other people some breathing room... :+)

-bowerbird

[GregS]

bowerbird "to me, they're all just tools. i had to go to a scripting language because
realbasic only creates offline apps, and i wanted to do web equivalents..."

I understand this position. However, the potential for interpretive scripting to give users control over their digital lives is something that is an off-topic in this thread, but from other points of view is worth considering. The particular nature of each language in that context at least, is important. The fact is if you put the effort in all things appear equal, but elegance, consistency, ease of learning and editing transparency become critical if that potential is ever to be realised. Have a look at LUA and REBOL, what I consider new generation languages (Python is distinctly old generation, and as for PEARL - never mind).

I suggest this as another possiblity for your own project.

[bowerbird]

greg-

thanks for the suggestion. :+)

but i'm too old a dog to be learning new languages.

youngsters familiar with a language can take my pseudocode
and churn out versions of my programs in a matter of weeks,
if not less. so i have no need to take on that burden...

-bowerbird

[HarryT]

Bowerbird,

I think that many of us would like to see a practical example of what your tools can do to a PG book.

Could you please, using your tools, produce an HTML version of:

http://www.gutenberg.org/dirs/etext01/wrnpc12.txt

That's the plain text PG version of "War and Peace", and contains lots of interesting "structural elements" such as footnotes. I'd like to what your tools make of it, if you'd be so kind.

Thank you,

[kacir]

Originally Posted by bob_ninja

bowerbird,
I use Java which has regular expressions capability. I plan to simply build some search/replace regexp patterns and allow a user to enter his/her own to customize according to individual preferences. Shouldn't be too bad. Actually most of the work is for interfaces, GUI and CLI.

I personally use Vim text editor for performing power use of Regular Expressions.
Vim - www.Vim.org - has Regular Expressions that are more powerful and better documented that Regular Expressions in Perl.

Vim has many advantages over the use of a programing language with RE support. The most notable advantage is that playing with REs in Vim (or its graphical version Gvim) is interactive - you can try replacement like this:
<esc>:%substitute/string_to_replace/replacement_string/g<enter>
Then you inspect the results, use Undo if needed and try again.
If it works like intended you simply create a vim macro with that substitute command.

Please send a personal message to me if you are interested in using Vim.
It does not matter what operating system you use, there is version of Vim for it. Even if you use such obscure and/or specialized OS as Minix, QNX, NetBSD, Amiga OS (yes, the operating system for Commodore Amiga), or even Vista


There is also a very nice formatter for text called par.
see following link http://vim.wikia.com/wiki/Awesome_text_formatter for description.

[bowerbird]

harry, my voice is filtered here, so i've answered you here: http://z-m-l.com/mr/harry_wants_an_example.zml

[HarryT]

Originally Posted by bowerbird

harry, my voice is filtered here, so i've answered you here: http://z-m-l.com/mr/harry_wants_an_example.zml

Thank you for that reply. You ask "why War and Peace"? It's not because of its length, but because it makes rather complex use of footnotes; it's by far the most technically complex book that I've done a "hand" conversion of. It's because I've processed it "by hand", and am very familiar with the difficulties it poses that I'd be extremely interested to see what can be made of it by automated processing tools.

[bowerbird]

i can handle the footnotes automatically. i'll convert it if you'll do quality-control on it.

[HarryT]

I'd be happy to offer comments on it.

[bowerbird]

"offering comments" isn't quite the same as doing a quality-control pass. which will you do?

[HarryT]

I'll do whatever you'd like me to do, within reason. Tell me what you'd like! As I say, I'm very interested to see how far automated processing can go with a "real world" example such as this.