What "Cleaning Up" Do Project Gutenberg Texts Need [closed]

[kovidgoyal]

Originally Posted by bowerbird

but i don't want to _monopolize_
the conversation here.

Are you sure? ;-)

[bowerbird]

kovidgoyal said:
> 2) I care because I am trying to drill into your thick head
> that light weight markup is not the best solution for ebooks.

first off, let me say that i really enjoy language like "drill into your thick head".

that kind of cartoonish imagery is a tip-off -- to me anyway -- that there is
a good sense of humor that's operative in this discussion. which is important.

although some people might tend to interpret stuff like that _seriously_,
that's a mistake. way back, i'm glad i learned the 3 rules of cyberspace:
> 1. don't offend people.
> 2. don't _be_ offended.
> 3. of the first two rules, the second is much more imporant.

i _could_ become offended instead, but what would that net me?
just some high blood pressure, and a pissy attitude towards life.
who needs that? i would _much_ rather stay cool as a cucumber.
especially since, when you get emotional, you say stupid stuff,
because you're not thinking straight, and you lose the argument.

and the _best_ part of all of this is that, even if kovidgoyal _was_
trying to dish out an insult that he wanted me to take personally,
i've beaten him at his own little game by deftly sidestepping it...

there's a lesson in there for all of you...

-bowerbird

[bowerbird]

i held up one response i'd written to kovidgoyal, to let others
have a chance in this conversation, without realizing that i'd
included a few responses to other people in that same post,
so here they are, separated out now...

***

nate said:
> I hope you aren't really comparing yourself to Mobipocket.

well, i'm not _french_, so i don't have the cool _accent_...

plus, in case you didn't notice, there _is_ a small difference
between _6_ figures (which is my minimum asking price)
and _7_ figures (which mobipocket was actually sold for)...

but if there's something you think mobi programmers did
that another programmer cannot do, well, then what is it?

(you can forget the d.r.m., because i'm religiously opposed.)

besides, the absence of a mobi mac version means i'm unimpressed.

***

chuck said:
> If the semantics of the book are automatically added into
> the PG text as XML tag pairs then three benefits will result:
> 1) An XML schema checker can validate that the semantics are valid.
> 2) An XSLT style sheet can easily, and on the fly, convert the book
> to ASCII, PostScript, HTML, Etc.
> 3) New style sheets can leverage existing annotated books to
> support new formats.

these are pretty much the standard arguments for x.m.l. markup...

and yeah, the main problem is step #3, in that there aren't any
"existing annotated books", and no easy way to create them...


> Given the existing support for parsing and processing XML

if you are aware of tools that actually provide "existing support",
you might want to go over to the distributed proofreaders forums
and let them know how they can use them, because they want 'em.


> it would be straightforward (although perhaps not easy), to
> create a copy editing tool which sucked in a book, added its
> best guess at what the semantics were (and there is great work
> to leverage from the ZML work here) and then generate
> an annotated result.

and this is the main reason why i'm _not_ turning my source code loose.
i don't want people using my hard-won routines to create x.m.l. markup.


> One might hope that all copy editors/proof readers can agree
> that something "Is a heading" without having to agree on how
> headings should be presented, or treated in the book presentation.

there's little disagreement on what the structures are in a given book.
human readers have figured that out fine for a few hundred years now,
thanks to the expertise of our whip-smart typographers along the way.

the difficulty is in programming this "intelligence" into a conversion tool.

the first tactic my antagonists over on the p.g. listserves tried to use was
that this required "artificial intelligence" that was too complex to program.

what i told them, and what i'll tell you as well, is that it's not all that hard...
you just have to work at it, and work at it some more, and then even more.

but it _can_ be done. i did it. and, you know, i'm not even mobipocket...

of course, you _could_ always just wait until i've mounted my mirror...

because then you'll be able to take advantage of the z.m.l. labeling of
every structure in every book, and use it to apply your heavy-markup.

oh, but then your heavy-markup won't be able to do anything more
than my light-markup, it'll just be more complicated to maintain and
more complicated for developers to add value. but, you know, you'll
have the library in the heavy-markup state you prefer, which is nice,
for you, i guess... maybe it'll help you you sleep better at night... :+)

but chuck, thank you, sincerely, for staying constructive in your post.

-bowerbird

[kovidgoyal]

Originally Posted by bowerbird

kovidgoyal said:
> 2) I care because I am trying to drill into your thick head
> that light weight markup is not the best solution for ebooks.

first off, let me say that i really enjoy language like "drill into your thick head".

that kind of cartoonish imagery is a tip-off -- to me anyway -- that there is
a good sense of humor that's operative in this discussion. which is important.

although some people might tend to interpret stuff like that _seriously_,
that's a mistake. way back, i'm glad i learned the 3 rules of cyberspace:
> 1. don't offend people.
> 2. don't _be_ offended.
> 3. of the first two rules, the second is much more imporant.

i _could_ become offended instead, but what would that net me?
just some high blood pressure, and a pissy attitude towards life.
who needs that? i would _much_ rather stay cool as a cucumber.
especially since, when you get emotional, you say stupid stuff,
because you're not thinking straight, and you lose the argument.

and the _best_ part of all of this is that, even if kovidgoyal _was_
trying to dish out an insult that he wanted me to take personally,
i've beaten him at his own little game by deftly sidestepping it...

there's a lesson in there for all of you...

-bowerbird

Sigh are you ever going to actually answer any of my points or just keep producing more meaningless verbiage.

[Robert Marquard]

Originally Posted by kovidgoyal

Sigh are you ever going to actually answer any of my points or just keep producing more meaningless verbiage.

No. He has done that on the gutvol-d mailing list for years.

[bowerbird]

i've said it's coming, so just be patient, ok?

as was remarked on the thread split from here,
some people "stopped reading" because of the
"direction" in which this thread has progressed.

that's what happens when a thread devolves...

is it an effect you _want_ to make happen?
(like robert here?)

let the conversation breathe a little bit, and
let others have their chance to make posts.

-bowerbird

[RWood]

I did a little research on the web and found some posts of bowerbird from 3 to 4 years ago where he said that he was almost there and said to just be patient. The same words he uses today.

He has submitted a book of his sayings to Gutenberg and it seems that many of the phrases and constructions he posts here are nearly the same as what he put in the book.

In short this one trick pony keeps doing the same thing over and over again.

[bowerbird]

rwood said:
> He has submitted a book of his sayings to Gutenberg

um, no.

those were collected by one of my "fans" there.

it does give me a good chuckle to go read 'em
every so often. why not give people the u.r.l.?

i strung along the p.g. listserve for a long time
because people there were willing to risk betting
their credibility arguing against my light-markup
as long as it was "theoretical". so i was happy to
let them raise the stakes in our little poker game.

because i knew that once i started laying out the
actual _evidence_ sitting there on my hard-drive
-- "showing my hand", as it were -- i would _win_
the entire pot. and indeed, now my "critics" there
have nothing left to bet, because they've lost _all_
credibility they had, such that now when i deliver
regular updates on how my work is progressing,
there's no longer a peep of an argument from 'em.

so i suggest that you not lose _your_ credibility too.

i've already laid out plenty of evidence of my work:
> http://z-m-l.com/go/pudding_sampler.html
> https://www.mobileread.com/forums/sho...3&postcount=83

if you have criticism of it -- constructive or not --
i will love to hear it. but don't make the mistake
of thinking you can beat my hand until you check
the cards that i've already laid out on the table...

-bowerbird

[bowerbird]

but hey, i love it when people have so much _passion_ about
what i'm saying that they start "researching" me on the web...

-bowerbird

[bowerbird]

kovidgoyal said:
> Sigh this is a discussion about the merits of light weight markup,

no, it was a discussion about how to make p.g. e-texts beautiful.
but it got re-routed into something else, so i went with the flow...

but a discussion of light-markup in general is too far off the mark.
i'll respond to your points, because you've been so very impatient,
and insistent, but it's time to draw some lines to bound the topic...


> It's about trying to figure out whether spending time and effort
> on creating apps that support light weight markup is worth it.

it is? that seems kind of silly to me. no, _very_ silly.

you think it's _not_ worth time, so you _won't_ do it.

and i think it _is_ worth time, so i _will_ do it. so there.

everyone's happy. end of discussion. everyone's happy.

and down the line, i'll have a library in light-markup format,
and the world-at-large will then decide if that is worthwhile.

i suggest that you prepare a heavy-markup library to
compete with mine, because i'd hate to win by default.


> 1) Features not supported by light weight markup

are you trying to tell me what my system can and cannot do?
because it always makes me laugh when someone does that.


> 2) I care because I am trying to drill into your thick head
> that light weight markup is not the best solution for ebooks.

and i'm trying to share with people that i've found that it _is_.

and that's all i want to do, to _tell_ them. to share information.

because i'm way past the "discussion" stage on this little topic.
if you wanted to take part in that, you should have been on the
project gutenberg listserves for the last 4 years. because _now_
i'm at the "proof is in the pudding, and here's my pudding" stage.

and i don't particularly care if the message penetrates through
"your thick head" or not. it won't be decided here... or by us...
it'll be decided by the real people who actually use my library and
either (1) like it and continue to use it, or (2) don't like it and stop.
so your general opinion on the value of light-markup means nothing.
as does mine. this issue will be decided by real users in the real world.


> 1) If your tools are not open source you're not giving them to people you're
> giving people the ability to use them. A subtle, but important distinction.

that's exactly right. i'm giving them the ability to use the compiled apps,
and i'm not giving them the source-code. and that's exactly what i intend.

if you want the source-code to programs that do what mine do, write it...

i won't give you fish. i will teach you how to fish. but i won't give you fish.
and i couldn't care less if that bothers you or not. might even hope it does.


> 2) Again the point of this discussion is
> to weigh the merits of light weight markup
> as a format for ebooks, not to decide
> whether you've spent your time wisely or not.

no, the thread was created to talk about the various ways that people
bring typographical beauty to the ugly e-texts from project gutenberg.

i shared a list that i had made, and invited other people to add to it...

if you want to start a discussion about the merits of light markup,
go and start _that_ thread. but, like i said, i'm past that talk stage...
i'm creating pudding, and giving people samples so they can taste it.

but, please, if you have any questions about what z.m.l. can handle
-- any structure that is typically found in books, even only rarely --
then do feel free to ask me about it, and i'll tell you how i'd work it...

a hypothetical discussion of the general merits, though? no thanks.
i'm sure you know -- as a coder -- that after chewing on something
well enough to explain it in the detail required by a compiler, there is
something terribly unsatisfying about vague and general handwaving.

if you want to show me books from the project gutenberg library
you think i can't digitize, fine, bring 'em on. (they exist. about 1%.
i'm leaving those to the heavy-markup crowd.) but if you want to
throw out a claim that there are _many_ that i can't do, who cares?
i'm gonna prove you wrong with the pudding of the 99.2% i can do.


> 3) My concern was writing converters to zml not from zml.
> If you want to push zml as an ebook format, considering that
> there are currently no ebooks in zml you'd better worry about
> writing converters to zml not from zml.

maybe you didn't hear me say i will convert the p.g. library myself.
there will be approximately 15,000 books in z.m.l. format soon...

i've also created post-o.c.r. clean-up programs geared toward z.m.l.,
which people can use to turn google's scan-sets into nice zml-books.

and once authors realize how easy it is to make a kick-ass e-book
with z.m.l., the number of _new_ books in the format will explode.

so, while i'm certainly touched by your "concern" about writing
converters to z.m.l., i'd suggest to you that it's misplaced, and
perhaps you could find a more appropriate cause to care about.

-bowerbird

[kovidgoyal]

@bowerbird
IOW you cannot respond to my concerns. Good bye and good luck.

[bowerbird]

kovidgoyal said:
> CSS float, boxes with custom borders, boxes with
> background colors for emphasis. Drop caps. I could go on.

you know, i didn't even respond to this initially.
but i get the feeling that you think these points
are doing some kind of damage to my argument.
(otherwise, why _have_ you been so insistent that
i haven't responded to your points. i don't get it.)

the reason i didn't respond is because i hate
to rain on your parade with the big n-slash-a,
but "not applicable" is the only honest answer.

the big tip-off is that you went to the c.s.s. pile.

every one of your points here is _presentational_.

float? custom borders/colors? even _drop_caps_?

presentational. and _unimportant_ presentational.
nothing more than doodads, and almost trivially so.

i'm concerned with the _structural_ aspects of books.
so that's what my system puts into the file-format...

the _structures_ of a book are things like headers,
and whether a blob of text is a table, or a poem,
or a block-quote, or an epigram, or a dedication,
things like that. what it _is_. not what it looks like.

(some people call these "semantic" entities, but
that's a slight misuse of the term, in my opinion.
a chapter heading doesn't _mean_ anything --
which is how the word "semantic" is defined --
it simply _is_. so i use the term "structural"...
but where i _do_ agree with those other people
is that _presentational_ aspects are arbitrary,
and therefore do not need to be hard-coded.
i do not buy into their emphatic religion that
the "semantic" and presentational _must_ be
separated to the point of complete exclusion,
but agree any specific presentation is arbitrary.)

your issues here are completely presentational...

so, no, there's no way to code them in a .zml file.

further, that's because presentational options are
under the control of the _reader_, not the _author_.

that is, the author doesn't get to declare drop-caps.
or the coloring of boxes, or the corners on boxes, or
any presentational stuff like that. sorry about that.
(ok, not really. because, truth be told, my authors
won't even want to be bothered with stuff like that,
or they wouldn't start using z.m.l. in the first place.)

if the reader wants drop-caps, and the viewer-app
gives that choice to the reader, then it's the reader
who'll specify that choice, and have it displayed so.

and yes, even though drop-caps are a _doodad_,
i probably _will_ make 'em optional for the reader.
(they are not in any of my viewer-apps up to now,
and they are not high on the priority list of to-do's,
but they ain't on the bottom either, mostly because
they're gonna be really simple to write the code for.)

but as for custom-boxes and custom-backgrounds,
those _are_ indeed on the bottom of my priority list.

personally, i'm quite delighted by the sunken look of
quoted passages on many forum boards, like this one, and
i guess it would be very easy to program, but i'll still
hold off on it, because it seems to be complete fluff,
and i do not want to give the impression that i have
stooped to the low level of coding the complete fluff.)

anyway, so yeah, if you have anything _structural_
that you think my z.m.l. cannot do, please say so...

but presentational doo-dads, i have no time for...
(but sure, i'll put them somewhere on the to-do list
if an honest-to-goodness z.m.l. user requests them.)

-bowerbird

[bowerbird]

kovidgoyal said:
> IOW you cannot respond to my concerns.

once again, you're just a little bit too impatient...


> Good bye and good luck.

that might be the best for everyone concerned... :+)

and i'm sure you'll take away the impression that
i "failed to respond" to your points. so be it...

the fact of the matter is i can't _find_ any points
that you've made that merit much response. sorry.

so if anyone else out there can see such points,
do please draw my attention to them. thanks...

because i'm quite confident i can answer them all.
after all, i was subjected to _years_ of questioning
over on the project gutenberg listserves, until now,
these days, i'm the only one left standing over there.

as i put it in a recent recounting of the history there,
in the early days, i was surrounded by a pack of dogs
that would bark and bark and bark at every post i made.

but lately, like in that sherlock holmes story where the
fact that the dog _didn't_ bark at the murderer became
the tip that led sherlock to solve the crime, the dogs
over on the p.g. listserve don't bark at me any more...

-bowerbird

[kovidgoyal]

To summarize your response, zml cannot support setting presentational aspects. And you dont intend it too, ever. And since you aren't open sourcing it, there's no chance it will pick up features like that in the future.

HTML+CSS can support both structural and presentational aspects. They give the author more control and more freedom. As such, IMO they are a much better match for a *general* ebook format.

Look at it this way:

zml is forcing restrictions on authors. Indeed your whole attitude is that authors dont know whats good for them and you're going to tell them that.

HTML+CSS encourages authors to represent things semantically, but if they really want to add presentational aspects, it allows them to do so.

To me the second approach is simply superior. If you want to encourage authors/digitizers to use only structural markup, a better approach would have been to write an authoring tool that supports only structural elements, via the GUI and allows authors to "edit the source" for advanced features. Something like LyX does for LaTeX.

[DaleDe]

Well to get back to the original theme I just finished reading a gutenberg book that was actually in fairly good shape. But even so it had some annoying problems still in it after I have gone through and beautified it once.

These included: punctuation without spaces. two sentences run together with a period and no spaces after the period. Spelling checkers are a great tool to find problems in scanned books but some of them won't find this since they have been taught (programmed) to ignore words of this kind since they might be filenames.

The second problem was paragraph splits where they didn't belong. The sentence was not over and the new paragraph started with a small letter. It should not have been a paragraph split.

Hopefully a program could detect this sort of thing.

Dale