Sigil as front end for automated XML based processing workflows?

[skreutzer]

Well, I would at least still like to know the following:

Originally Posted by skreutzer

For the topic of my initial question, I would really like to hear how you clean the input from authors in terms of structure, and if/how the added structural information is used for a later automated processing.

You don't have to answer if you don't like, but with the experience and the working environment you're in, I would indeed consider some hints about it valuable.

[Toxaris]

Originally Posted by Hitch

Gang:

I suspect that my view of the possibility of this functioning, widely, is utterly jaded by my own experiences, and I am thus just mostly talking to myself, to hear myself talk. I don't like this at the best of times, so I'm just going to bug out of this discussion. I don't genuinely think I have anything to truly add, other than cynicism, so ...that's not helpful.

Hitch

I am going to bug out too. Apparently I am unable to explain the issues clearly enough. They are not limited to closed software that needs buying, but also to open source software.

Just look at writer software out there. There are various programs out there (both commercial and open source). They are aimed to supporting and helping writers and their creative process. Only a very small portion of writers are using it, altough most of them know it is out there (I think) and available. If writers are not even considering using software geared towards them, how would convince them to use this?

In real life I work in the integration field of software applications, I know quite well of the issues of different formats and mapping, especially with XML. I have professionally architected a framework to simplify the process to a great extent. I know the benefits of what you suggest. I just don't see it coming for this area without great, great issues.
Without Word support, you can forget about it. Since you are ethically prohibited to use it, there is no way you can support it.

[Doitsu]

@skreutzer: Two other Open Source projects, recently announced ePub support:

AbiWord 3.0
Calligra Author 2.6

But, as is all too common in the Linux world, both projects have not released any Windows binaries thus excluding about 90% of all potential users.

It'd be great, if you could create Windows builds for these projects. You could also either join the development team of these projects to further your ideas or create a customized version that encourages using styles instead of direct formatting.

However, before you proceed any further, it couldn't hurt to ask authors:

- what tool they actually use for writing books (Word, Jutoh, AWP, Scrivener etc),
- why they don't use Sigil,
- what authoring features they'd like to see implemented etc.

(You can create a poll in the Writer's Corner forum.)

[Bodhi]

Well as I understand that release note about Abiword 3.0 it shall be available for Windows.

In case of Calligra I agree, because their releases are quite rarely for Windows.

[skreutzer]

Originally Posted by Toxaris

They are not limited to closed software that needs buying, but also to open source software.

“Open Source” software might need buying, too. Commercial use is a freedom of free software.

Originally Posted by Toxaris

There are various programs out there (both commercial and open source).

Commercial and “Open Source” aren't opposites. Closed, proprietary, restrictive, unethical and free software are.

Originally Posted by Toxaris

how would convince them to use this?

By the benefits?

Originally Posted by Toxaris

just don't see it coming for this area without great, great issues.

Well, one might start simple and improve from it.

Originally Posted by Toxaris

Without Word support, you can forget about it. Since you are ethically prohibited to use it, there is no way you can support it.

Why? Word XML output and Word itself are two separate things, XML itself is an open format and can be read, no matter with which software it was generated. The XML output of Word may be unfortunate, so it might be technically unusable. Further, there's no need to change the mind of all Word users - if Word is incapable of outputting usable XML, then writing in Word excludes its users from the benefits of automated processing. I'm looking for a software to apply semantic markup for myself, which could also be used by other users as front end for automated processing workflows. Such a tool could also be used to apply semantic markup to a text written in Word and exported to plain text.

Originally Posted by Doitsu

AbiWord 3.0

In AbiWord 2.8.2, it seems there's no direct way to edit a style template. However, since direct formatting can be disabled and XHTML output is present, it could be used as front end for automated processing workflows. Looks interesting to me, I will look into it. Thanks for the hint!

Originally Posted by Doitsu

But, as is all too common in the Linux world, both projects have not released any Windows binaries thus excluding about 90% of all potential users.

It'd be great, if you could create Windows builds for these projects. You could also either join the development team of these projects to further your ideas or create a customized version that encourages using styles instead of direct formatting.

Obviously, by supporting a proprietary operating system, you'll never get people to change to the free alternatives, since it is the goal of free software projects like AbiWord and Calligra to provide free alternatives in the first place (otherwise you could just be fine with a proprietary “solution”). It is a freedom of free software to build and distribute binaries for whatever system preferred (as long as it comes with the source code), even if it doesn't make sense at all. I won't even think about providing Windows binaries, I'm working on making things more free, not the opposite.

Originally Posted by Doitsu

- what tool they actually use for writing books (Word, Jutoh, AWP, Scrivener etc),
- why they don't use Sigil,
- what authoring features they'd like to see implemented etc.

What I'm looking for is some kind of “special purpose”, which would be used by a formatting guy if the writer doesn't want to do the semantic markup himself, or by the writer after he has completed the entire text. It should be possible to apply style templates on the flow (and direct formatting needs to be prevented), but I'm convinced that writing and formatting are two separate steps.

[DiapDealer]

Mixing ideology with functionality is almost always a recipe for disaster--whether in open or proprietary projects. I, myself, am not really interested in coding projects that can't (relatively) easily be compiled and/or run on the three major platforms. That's why I've always appreciated Sigil so much.

[Doitsu]

Originally Posted by skreutzer

Obviously, by supporting a proprietary operating system, you'll never get people to change to the free alternatives, since it is the goal of free software projects like AbiWord and Calligra to provide free alternatives in the first place (otherwise you could just be fine with a proprietary “solution”).

OTOH, by not providing Windows binaries programmers are also effectively preventing 90% of potential free software users from switching to free software, because if end users cannot test free software solutions under Windows, they're not very likely to replace their commercial software products with them.

Originally Posted by skreutzer

It is a freedom of free software to build and distribute binaries for whatever system preferred (as long as it comes with the source code), even if it doesn't make sense at all.

While it does make sense to distribute software targeted at programmers and advanced users in source form, it doesn't make any sense at all to provide GUI-based software targeted at average end users in source form only. In doing so, software developers are basically discriminating exactly against those users who stand to benefit the most by switching to free software solutions, which also in a way flies in the face of most free software concepts, because, AFAIK, most definitions also include non-discrimination clauses.

Originally Posted by DiapDealer

Mixing ideology with functionality is almost always a recipe for disaster--whether in open or proprietary projects. I, myself, am not really interested in coding projects that can't (relatively) easily be compiled and/or run on the three major platforms. That's why I've always appreciated Sigil so much.

+1

[DiapDealer]

Just to be clear... I'm talking about standalone projects. Obviously addons/plugins and the like are exempt. I'm thinking of Toxaris here (who I wouldn't want to offend with my previous post). Clearly his Word addon can't be cross-platform; and that doesn't change its value (to me and others).

My point is: write code that will compile on as many platforms as possible. Test on as many platforms as possible and provide as many binaries as possible (along with the source) for those who have no interest in compiling. Let the user worry about whether or not it all fits their personal ideological preferences. That's my definition of "open." Not excluding anyone, for any reason.

[eschwartz]

Originally Posted by skreutzer

Obviously, by supporting a proprietary operating system, you'll never get people to change to the free alternatives, since it is the goal of free software projects like AbiWord and Calligra to provide free alternatives in the first place (otherwise you could just be fine with a proprietary “solution”). It is a freedom of free software to build and distribute binaries for whatever system preferred (as long as it comes with the source code), even if it doesn't make sense at all. I won't even think about providing Windows binaries, I'm working on making things more free, not the opposite.

But you can always provide ReactOS binaries.

[Hitch]

Originally Posted by skreutzer

Well, I would at least still like to know the following:



You don't have to answer if you don't like, but with the experience and the working environment you're in, I would indeed consider some hints about it valuable.

I don't seem to be able to communicate very effectively with you. I don't know if that's my failure, or your obdurancy. I'm truly not trying to be argumentative, but I don't much like wasting my time, either. You've pretty cavalierly blown off my five years of experience in dealing with the very marketplace you say you're aiming at, and in another post blow off the 90% of users on Windows software. I'm just not sure, at this point, how my input can be useful.

And when you say, "For the topic of my initial question, I would really like to hear how you clean the input from authors in terms of structure, and if/how the added structural information is used for a later automated processing," I'm not really sure what you're asking me. Are you asking me, generally, what do we do, or proprietarily, what do we do?

Generally, we do what everyone else does, I presume, that's competent. The first step is an "either or."

1. Depending upon the Word file, either I take a first pass at in IN Word, sometimes using Tox's ePUBTools, or we export it directly to HTML;
2. Then we use proprietary clips/programs, to clean the HTML further.
3. If it's Barb, who loves her some PERL, she uses a series of PERL clips that she's compiled, essentially, in the "accrued" sense of the word, to clean the HTML; if it's pretty much anyone else, we use a series of clips in NTPro to clean the HTML.
4. Basically, either searches for "the usual;" we search for garbage spans, eyeball them, remove them if indeed garbage, or replace them if they are the typical "span class = italic" with inline italic/em tags.
5. We search for broken dialogues and broken paragraphs, and all the usual crap that everbody searches for. Section breaks, page breaks, multiple uses of the enter key to create vertical whitespace, all the normal stuff. Places where the user used "Normal 18pt Bold" for a header, instead of a header class.
6. Then, once the base file is cleaned up, we do the eyeball portions.
7. The eyeball portions are looking for those styles that the user may or may not have created, and implementing them in the XHTML.
8. For example, we get a LOT of books that have a style tag of "MSONormal," but in the actual manuscript, the user hit the "tab" key 3 times to create an indented text message. That won't export into HTML, about...75% of the time.
9. We find those and manually "fix" them to have the correct CSS. If the user created a half-baked style for it, we do a simple F&R with a class of CSS.
10. Once the file is basically cleaned and ready to go, we import it into Sigil, and finish it there. Divide the chapters at the chapter markers (those are all put in with our clips/PERL during pre-processing); eyeball the NCX; eyeball anything else; and then finish up the file.
11. Once that phase is done, and the client has approved the ePUB, we tweak it some more (proprietary) and prep it to be fed to KindleGen to make a MOBI.
12. That's it. Nothing more or less. Just boring, repetitive, tedious human work.

It's not rocket science, it's just repetitive tedium. The problem is, as I see it, no two authors make the SAME mistake in the SAME way each time. Sometimes, errata is "MSONormal center bold" for a header; sometimes it's "MSONormal 18pt Bold" for that same header. Sometimes, we get files that are simply inexplicable, as to how styling that's in there doesn't show up in Styles, and doesn't show up in exported HTML (very common with any Pages-->Word "conversions" output).

It would all be swell and good if authors had 5 styles to pick from, used 'em, and that was that. You could automate the process and Bob's yer uncle. But that's not what they are accustomed to, and that's not what they want. As Tox points out better than I could in his post, there are SCADS of tools out there already, that would work better (from a conversion standpoint), that authors already don't want to use. {shrug}.

I just think that you are expecting right-brainers to somehow magically see the advantages of working in a left-brained environment, and my experience, for what it's worth, is that that ain't ever gonna happen. Not only are they utterly disinterested in what is going on behind the scenes, they don't WANT to know, don't CARE to know, and somehow, think it makes them less creative if they understand the "how." This is my experience. Feel free to ignore it. However, of all the phone calls I take, I cannot tell you how many tell me either, that 'I'm not good with computers," or, even worse, "I'm really very tecchie but I need help with this," the latter of which means that when the time comes for that person to download a file from a browser interface, the s**t will hit the fan. Nor do they know where their downloads folder is, or how to drag-and-drop. That's what that last sentence means. I'm not disparaging them, but your idea just seems to utterly ignore the reality of a writer's inner world. That's how I interpret what you've said thus far. Sort of, "well, I'm making this tool for myself (which is fine; that doesn't faze me), and for writers, and if they don't want to learn it, the hell with them."

And if that's the entire gist--that you'll make it for yourself, and if anyone else wants to use it, they can--then great. But if you're asking everyone here for input and assistance and feedback, for a tool to be used widely, that would, purportedly, make OUR jobs easier, then you need to ALSO be open to the fact that maybe some of us might have a little more experience in the real-world environment in which you expect this to function. Just an idea.

Mixing ideology with functionality is almost always a recipe for disaster--whether in open or proprietary projects. I, myself, am not really interested in coding projects that can't (relatively) easily be compiled and/or run on the three major platforms. That's why I've always appreciated Sigil so much.

+100.

Hitch

[Toxaris]

Originally Posted by DiapDealer

Just to be clear... I'm talking about standalone projects. Obviously addons/plugins and the like are exempt. I'm thinking of Toxaris here (who I wouldn't want to offend with my previous post). Clearly his Word addon can't be cross-platform; and that doesn't change its value (to me and others).

No sweat, I didn't take it personal at all. I am just happy more people than just myself find it useful and it actually gives me a happy feeling that it is used internationally.

[Freeshadow]

Tox: how would you convince them to use this?
skreutzer: By the benefits?

That's exactly the point where you didn't listen to Hitch.
I've met enough people being just of the cut Hitch speaks of:
learn resistant homo habilis or lernresistentes Gewohnheitstier if you prefer.

The only authors potentially willing to learn about good tools might be scientific ones because they heavily need a stable toolset for their work. Their work turns to shitpaper w/o a good reference apparatus so a lot of them happily switched from text editors aimed at authors
(like NotaBene) to *TeX.

As for making people switch OSs because of software... Why?
Have you considered that they might use more than one program?
Where do you help if your tool isn't all systems available and so a potential switcher cannot try and use it while waiting for an alternative for a different piece of software s/he uses too?
If you truly want to promote free software you should do so.
If you want to pull people to Linux... Don't.
Linux isn't for people wanting a free windows.
Linux is for people wanting Linux - the other ones won't switch their thinking about how an OS should operate.

P.S.: If Linus has no problem with offering his stuff for multiple OSes nonfree ones included: http://subsurface.hohndel.org/de
Why should you? More Catholic than the Pope eh?

[st_albert]

Originally Posted by Freeshadow

P.S.: If Linus has no problem with offering his stuff for multiple OSes nonfree ones included: http://subsurface.hohndel.org/de
Why should you? More Catholic than the Pope eh?

Nah, Linus isn't the Pope. RMS is the Pope.

[st_albert]

Originally Posted by skreutzer

You don't have to answer if you don't like, but with the experience and the working environment you're in, I would indeed consider some hints about it valuable.

At the risk of beating a dead horse, I'll briefly share my experience.

I work for a small-press "traditional" publisher, publishing in both print and e-book (usually both for a given title, but sometimes only one or the other). Typically our authors supply manuscripts as Word documents, and they are unaware of the proper use of styles. They do, however, often have a definite idea of how they want their manuscript typeset, and as you would expect in a WYSIWYG enviornment, they achieve the desired results by hook or crook. For example, one passage is poetry, indented with tabs; another is a blockquote; and a third is supposed to be a letter, so it's in a script font. Paragraph indentation is achieved via tabs and/or spaces. Vertical spacing is done with empty paragraphs. There may or may not be page breaks at the end of chapters.

Once a manuscript is accepted, a manuscript passes through one or more editors, and back and forth to the author and publisher, until it is finally ready for prime time. Everyone uses Word and its native format.

Wouldn't it be wonderful if they all used a collaborative, web-based process?

Not. Going. To. Happen.

Once the manuscript has seen edits, it comes to me. My goal is to "clean it up" by replacing the direct formatting and hook-or-crook styling with a uniform standardized set of paragraph and character House styles, while preserving the intent of the author's formatting in so far as possible.

If the book is destined for print, I pull it into InDesign, and do the cleanup there. If there won't be a print version, or if we need a quick ebook as an ARC, I use LibreOffice, save as an .odt file and export that to epub via Writer2xhtml.

The actual cleanup issues were well described by Hitch, though we don't get some of the more egregious cases -- manuscripts like that would never be accepted in the first place. Some things can be handled automatically: removing whitespace at the beginning of a paragraph, replacing direct formatting (e.g. italics, bold) with the appropriate character style, imposing house styles in place of whatever styles the author did or didn't use (usually the "defalult" style).

Treatment of chapter headers, styling the first paragraph, and some other idiosyncratic details, however, require case-by-case intervention. Sometimes you just have to look at it to decide what the author was trying to do.

After epub export from either InDesign or LibreOffice, it goes to Sigil for chapter splitting, adding the cover, adding lots of metadata to the .opf file, and generation of TOC's (typically both an .ncx and an inline TOC.xhtml). A modified version is also prepared for use with kindlegen to make the Amazon Kindle version.

Going forward, we use the InDesign file (if there is one) as the master file to collect all of the inevitable corrections that only magically appear after the book is in print, and when necessary generate updated ebooks from it.

HTH
Albert

[Hitch]

I would say that, in all likelihood, 95% of what Albert and I (I = "we" at my company) do is identical. The process is the process. There's just no silver bullet. God knows, I wish there were. But as long as we have individual authors using "hook or crook" to get manuscripts to do what they think they want it to look like, we'll have different ad hoc styling (inline styling) problems to solve. I don't have the luxury that my buddy Albert has--no editors to help look for that type of garbage--so that's why we get the "more egregious" cases.

Here is an ACTUAL sample of what we deal with; I have a woman that's been emailing me for the last 6-12 months. She had a book in print, in Europe, as she is. She had a "friend" do the scanning of the print book, in France. The resulting scan had all the usual horrors--page numbers midst-page, section breaks, varying margins, columns set (yes), you-name-it. She couldn't even begin to work on it. The friend didn't use Abbyy; used a desktop scanner and some type of older scanning software.

I took pity on her (yes: I'm an idiot) and cleaned it up a bit, doing nothing more than tagging the italics and bold, removing the 5 gajillion styles, conforming the fonts, margins and other settings, removing all the section breaks. However, it had a ZILLION broken paragraphs, which of course she never saw. When I fixed the margins, she could of course now see them (doesn't know from a pilcrow). I'd sent it back to her for proofing, and for fixing the broken paras. Here is, I s**t thee not, text from an actual email from LAST NIGHT, after I painstakingly took screenshots, sent her links to a tutorial, etc., about cleaning up the broken paragraphs:

I've just looked and took the first few pages, the Prologue, and justified the margins. This corrects most of the broken paragraphs in one move. I'll do it chapter by chapter as I go.

As I've said: you can't make this stuff up. Not visible to the eye, so, it's fixed, right? This is a lovely person, but....It's just not possible to force the horse to drink the water. YOU may see benefits, but until an author sees a DIRECT benefit to him or herself, and the software makes his or her life EASIER...well.

Hitch