A simple tagged text format with an HTML converter

[RobertDDL]

I like plain text, because it is the rock solid bottom on which we can safely stand, whichever device, software or OS we use or will use in the future. And with fiction, not much more than plain text is usually really needed. Not much more... There already exist a number of tagged text formats, but for the use on my website I've come up with my own, with the intention of keeping it as basic and simple as possible both for the reader and the writer/editor. I have also written a tool that converts this format to HTML, from where it can be easily further converted to ePub etc., for instance with Calibre.

If you are interested you find the details, a sample (converted to HTML), and the conversion tool (for the Windows command line) here:

www.dunyazad-library.net/plaintext.htm

Thank you for any comments, suggestions, questions etc.!

[Turtle91]

I'm pretty sure this has been tried before. You may wish to look over at Project Gutenberg and see if that would help you out.

They started as a text-only storage of public domain books from around the world. It was required to submit books in text - as that was the format to last through the generations, as you argue. However, it quickly became apparent that they needed to allow some mark-up to identify some formatting; Italics, Bold, block indents, headers, to name a few. so they came up with their "tags" *Bold*, _Italic_ ===headers===, etc.

Then the argument said...why don't we allow HTML? A clean HTML document is exactly the same thing but it could be read on many different computers: <b>Bold</b>. <i>Italic</i>, <h1>Headers</h1>

Then the eBook industry said...well, lets make a standard format that can be read by anyone and we will base it on very clean, very simple HTML....and ePub was born.

I honestly don't think creating a New/different markup style will help the industry - or readers for that matter - at all. I do wish you luck however. I have been known to write programs that do all sorts of things that no one else seems to appreciate as much as I do!

Cheers!

[RobertDDL]

Originally Posted by Turtle91

I honestly don't think creating a New/different markup style will help the industry -

Oh, I have no such ambitions at all!

Originally Posted by Turtle91

- or readers for that matter

They have to put up with it, when they download plain text files from my site

I think it does have some advantages for them, but, of course, this depends upon what you compare it with, and what your (or their) preferences are.

Originally Posted by Turtle91

I have been known to write programs that do all sorts of things that no one else seems to appreciate as much as I do!

Happened to me ever since I've started programming, in the 70s...

Originally Posted by Turtle91

Cheers!

Thanks!

[RobertDDL]

I have just uploaded version 0.21 -- officially it's still beta, but it should be complete and stable enough for practical use now.

This is a minimalist design, so close to plain text that I do not even call it a markup language. Still, I have been writing software for 40 years, have edited and typeset texts for 20 years, and have been a publisher for more than 10 years now, so, some thought and experience has gone into this.

Anyway, feel free to use, comment, criticize, improve, or ignore!

[skreutzer]

What about doing the very same thing for Markdown, maybe with help of Markua if needed?

[jandrew]

Originally Posted by skreutzer

What about doing the very same thing for Markdown, maybe with help of Markua if needed?

I'm not sure I see the value of this over just using pandoc which can do markdown->epub2/3 (among many other conversions). And pandoc is fairly widely used and available for Win/Mac/Linux.

[Hitch]

Originally Posted by jandrew

I'm not sure I see the value of this over just using pandoc which can do markdown->epub2/3 (among many other conversions). And pandoc is fairly widely used and available for Win/Mac/Linux.

I could be mis-remembering, but I'm 99% sure that user_none started work on a Markdown editor, with which to make ePUBs, about...3 years ago? I know he'd been seeking input from folks. As I've heard naught about it, I'd hazard a guess that he found that the interest was basically zero.

Using Markdown, et al, is all well and good for the wonks of the world...but it needs to be something that the creator himself or herself wants to use, because the sad truth is, most folks won't use it. If you're OCD, then, great, Markdown and its ilk are for you, but most people aren't, and won't. People like to do what's easy--and what's easy is using Word, Scrivener, Wordperfect, OO (because it's free), LO (ditto), etc. People do whatever is easiest and/or free. They do.

Using Markdown (which I use all day long) is more work, upfront, than using a word-processor. I can't just hit a button and italicize something, or make a header--I have to tag it. It's annoying as crap when I am trying to work all bloody day long, already typing my fingers off to clients. I have no idea how many keystrokes a day are added to my workload because I have to type ".h3" whenever I want to create a section head, or _italic_ or *bold* blah-blah, but it's a bunch. Worse, the "flavors" of Markdown I use--one at Desk and the other at Teamworks--are not the SAME, so I have to remember that a section in one is 3 hashmarks, but that's the list function in the other. Giant PITA.

I cannot fathom, given what I see all day in terms of pure human laziness (in terms of people who never bothered to learn how to even use the BASICS of their word-processor, much less anything else like lists, styles, and the like) that people will suddenly adopt Markdown (or its ilk), causing them MORE work. There would have to be a massively great reason to promote adoption--and what is that? And if it's not adopted widely, how is it supported?

Just sayin'.

Hitch

[skreutzer]

A Markdown editor? Is that a plain text editor with buttons to trigger the Markdown syntax for the current selection? I too don't get the Markdown hype, but maybe that's just because I'm in favor of XML because of its rich tooling ;-) As you said, people will use what's easy, so I did OpenOffice/LibreOffice first, while its usability concept is ill-suited for writing long texts in my opinion, especially it's too easy to produce a badly formatted document.

Regarding Markdown flavors, one would go after the one that aims for standardization while retaining compatibility to as much other flavors as possible, or, if such thing doesn't exist, the most widely used one. If RobertDDL is interested, he could write converters from the other flavors to the standard flavor. Anyway, I only brought Markdown up because I think using Markdown as input for conversion would be way better to define yet another new plain text markup format, which naturally will look pretty much just like Markdown.

Using Markdown would be more future-proof because documentation will be around for some time (unfortunately, it's not self-expressive) and hopefully somebody will write a converter to convert Markdown texts to whatever format comes along in the future or to a bunch of formats, of which some probably will survive longer. I mean, plain text is plain text, ASCII or UTF-8, and as long as there are universal computers around, one will be able to extract at least the plain text and just have to replace the formatting instructions. Pandoc would already consume such files as input, and if more flexibility is needed, I could integrate such a converter even in my toolchain, if freely licensed and based upon free/open technologies.

[RobertDDL]

Originally Posted by skreutzer

If RobertDDL is interested, he could write converters from the other flavors to the standard flavor.

No, sorry, I'm not. I have designed my own, whatever you may call it, enhanced plain text format, as a deliberately simpler alternative to existing markup languages. It covers the very few essential formatting options that are really needed with 99% of fiction, when you want to stay with plain text -- it requires zero learning for the reader, maybe 5 minutes for the author/editor, and will hardly run into any compatibility troubles.

For me, the main disadvantage of plain text, when used for long texts, is the lack of a working TOC. (Bookmarks aren't the problem, as they are usually provided by the reading software or device.) I have written the conversion tool to HTML mainly for the purpose that HTML can then be converted to ePub, which results in an operational TOC. (It wouldn't be much of an effort for a reading software to generate a TOC from the HTML heading tags, this shouldn't take more than a fraction of a second, but I haven't seen it so far and I'm not able to write it myself.)

So, I am aware that other markup formats are more powerful and offer many more features, but my point is, that in most cases they are not needed (non-fiction would be a different matter). Where they are, Markdown or other markup languages are of course better suited, but they aren't what I'm interested in.

[skreutzer]

So what are you actually interested in? Just to make your own format because you can? That's fine, I don't mind ;-)

Originally Posted by RobertDDL

No, sorry, I'm not.

Of course you don't have to even bother with interconverting the Markdown flavors, my suggestion was directed to Hitch in case incompatibilities between Markdown flavors are considered a real issue here. But what about switching your parser from your custom syntax to Markdown compatible, but reduced instruction set?

An inline TOC can't be the job of a reading software (or shouldn't), because it would have to make decisions regarding layout, and sometimes the result would be inconsistent with the rest of the publication. Therefore it is the job of the author or generator software, and it would be almost trivial if HTML wouldn't have been designed that poorly in the first place. At least EPUB3 does a much better job in this regard. Please note that reading systems usually implement the TOC of an EPUB in an operational way.

[jandrew]

Originally Posted by RobertDDL

So, I am aware that other markup formats are more powerful and offer many more features, but my point is, that in most cases they are not needed (non-fiction would be a different matter). Where they are, Markdown or other markup languages are of course better suited, but they aren't what I'm interested in.

So why not choose to just use a subset of markdown syntax that does what you want? They would still be valid markdown texts and people on any platform could convert them to other formats. Your choice of defining your own special markup and providing a windows only executable to produce html only limits the accessibility of your texts.

[RobertDDL]

Originally Posted by jandrew

So why not choose to just use a subset of markdown syntax that does what you want? They would still be valid markdown texts and people on any platform could convert them to other formats. Your choice of defining your own special markup and providing a windows only executable to produce html only limits the accessibility of your texts.

I have made this choice many years ago -- admittedly I could have used a subset of an existing format, but, well, I didn't, and I had some reasons for not doing it. (For instance, my format allows to easily grep for headings up to a specified level, which Markdown doesn't.) The texts are fully accessible to anyone who can read plain text. The HTML-converter (of which the source code is included in the download file) is written in Euphoria and can be very easily ported to Linux or Mac.

But, really, anyone who may see some advantage in my format for a specific purpose is welcome to use it, everybody else doesn't have to!

[Hitch]

Maybe I'm daft--lord knows, many around here would certainly and inarguably assert that as QED--but isn't this the same freaking discussion that everybody had around DocBook? Around XSLT? That it was all gonna be future-proof, that it was simple, that it provided cleaner formating, blah-blah-blah? And what happened? Where'd THAT go?

I'll tell you where: it went the same way as the dinosaur. There wasn't a damn thing wrong with it, and frankly, a lot RIGHT with it. It was infinitely more usable, flexible and powerful than what we're all stuck doing now.

And yet: it's DEAD. Deader'n Julius Caesar. Why? Utter lack of adoption.

Maybe I'm conflating two topics, but I don't think so. Some of you want cleaner code. Great. But the only way that dog's ever going to hunt is if it's adopted WIDELY. Otherwise, we are all just spinning our wheels. If you're only making books for yourself, super-duper. Then, hell, you can do whatever you want.

But for the rest of us, all this "let's do markdown" or some variant thereof is just wheel-spinning folly. Because it won't happen. Authors and other literary creatures are not going to give up programs that do stuff FOR them, just because the output is cleaner.

Want proof? Look at all the utterly goofy and frivolous "writing programs" like Scrivener, Liquid Story Binder, and the like. They sell like hotcakes, for $40-$45/pop, compared to something nearly more powerful, and certainly more useful, YWriter (which is also FREE, mind you). Why? Besides marketing? Because they freaking LOOK COOL. They have FOOFY shit in them. "Desk Backgrounds" and "themes" and all that crap. They have "tools" that don't actually DO anything (the storyboarding "function" in LSBXE is my favorite completely useless function), but, bygod, they let someone FEEL like they are a "real writer."

The output from all of these isn't even as damn clean as plain RTF from a Word file, but you can't tell them that. Every single thing that you can do in Scrivener, that sells it, you can do in Word, too. But hey, nobody sold it as a "writer's program" with, OH WOW, integrated MUSIC!!, so they'd rather pay $40 for a "writer's program."

So, sorry, as much as I'd love--LOVE--to see something like DocBook, Markdown, yadda, come into play, from my perspective--which is absolutely admittedly selfish and centered around what I receive in, from my clientele--I just don't see that being a big hit.

And this post isn't aimed at or really written to, RobertDDL. He's made his choice, and he's doing what he wants. But for all the other sorta endless variants on "let's do a markdown editor," or what-have-you...it's great for Geeksters, but it's going to require someone who doesn't mind investing thousands of hours in a HOBBY that will never do anything but let them feel good about what they've done. It'll be like Sigil was, when Valloric had it and then decided to dump it--dependent on the kindness of strangers. Look around on this forum, going back 3-4 years. Look at ALL the new threads about "new tools" for making ebooks, new programs.

That's all I'm saying. Don't mean to piss on anyone's Oat Toasties, or rain on parades, either. (And, n.b.: I'm beta-testing TWO, count 'em, TWO new "ebook-making tools" right now. I'll give them honest tries, but it's going to take a LOT to convince me to give up using good old NoteTabPro and Sigil, in favor of something or somethings else. Because when push comes to shove, the clean-up issues in files are the same, all the time--and most of it can't be automated, not without sacrificing quality. The human eye is still needed, and that's where ALL the time comes in. I just don't see the bulk of users deciding to make my life easier any time soon.)

My $.02, FWIW.

Hitch

[RobertDDL]

Hitch, I mostly agree with your rant (except I'd say LibreOffice instead of MS Word), but it is a bit off the mark here, when we are talking about plain text.

Plain text files are always readable, and can be created and edited with any editor, regardless of their particular design. When you make a plain text file, you have to make some design decisions: do I use length-limited lines or not? Do I ignore italics, or do I use some symbol to indicate them? Do I ignore scene breaks, or do I show them, and if so, how? Do I treat headings like any other lines of text, or do I distinguish them in some way? If so, how do I differentiate between heading levels? And so on...

I'm not talking about markup languages, but you have to decide these questions -- even by ignoring them, you decide them one way or the other. No matter how you decide them, the text will be readable, and neither you nor the reader will be bound to any particular software or system, but some of these decisions may have some advantages or disadvantages for the reader (though, of course, they may have different preferences).

I think that Markdown has made a very poor design decision regarding headings. I say that I'm doing it better, allowing headings of all levels to be easily grepped, and allowing the file to be easily converted to HTML and from there to ePub. What more can you want from a plain text file? But this is not a markup language, it's a style guide. It doesn't need to be adopted widely, because it's not in competition with anything else, and there are no compatibility issues whatever -- it's just plain text, with the design issues involved in any plain text decided in some particular way.

So, really, I think that all these strong "we do not need another format, why don't you stay with what already exists" replies are not valid criticism. If you are happy with Markdown, use it. Otherwise, either ignore what I'm doing, or have a look at it and criticise it for its shortcomings and faults, but not for the mere fact that I'm not using Markdown or a subset of it, and that I stand up and say in mixed company what exactly I do instead.

[jandrew]

Originally Posted by RobertDDL

I think that Markdown has made a very poor design decision regarding headings. I say that I'm doing it better, allowing headings of all levels to be easily grepped

Not sure what you mean here, markdown always supported atx style headings like:

# level one heading

## level two heading (and so on)

or am I missing your point in this grepping matter?

So, really, I think that all these strong "we do not need another format, why don't you stay with what already exists" replies are not valid criticism. If you are happy with Markdown, use it. Otherwise, either ignore what I'm doing, or have a look at it and criticise it for its shortcomings and faults, but not for the mere fact that I'm not using Markdown or a subset of it, and that I stand up and say in mixed company what exactly I do instead.

If your system existed in isolation that would be fine, but shouldn’t readers of this thread who might be unfamiliar with text based writing systems be able to read comments about how your system compares to existing solutions?

There is nothing particularly wrong with your solution per se, but the fact is that your dthtm.exe/dunyazad system is basically a very limited, but syntactically differing, version of a pandoc/markdown system. You have not offered any advantages that I can see, over just using a minimal subset of markdown.

On the other hand, what pandoc/markdown offers (beyond additional markup extensions) is:

• it has been widely used and tested for years
• it is actively maintained/developed
• there are many text editors providing markdown support (like syntax highlighting and preview modes) for win/mac/linux/ios/android platforms
• multiple format conversion including: html, pdf, epub2/3 and more

So, even if people just want to stick with headings, pararaphs, and italics, if they go with pandoc/markdown they have the above advantages. If their writing later requires footnotes, tables, citations, lists, code blocks, images, or such, they can add to their knowledge of pandoc/markdown as needed.

Certainly I get that you aren’t trying to be everything for everybody, and your system works for you and that is fine, and you are, of course, already invested in your system. I am not trying to attack your system for not being something else. But others might appreciate some context before investing.

cheers, andrew