Sigil as front end for automated XML based processing workflows?

[skreutzer]

Originally Posted by DiapDealer

Mixing ideology with functionality is almost always a recipe for disaster--whether in open or proprietary projects. I, myself, am not really interested in coding projects that can't (relatively) easily be compiled and/or run on the three major platforms.

Proprietary and/or restrictive software can prevent you and everybody else in the entire world from being able to compile it any time without even a chance of doing something about it except a port to a free system.

Originally Posted by Doitsu

OTOH, by not providing Windows binaries programmers are also effectively preventing 90% of potential free software users from switching to free software, because if end users cannot test free software solutions under Windows, they're not very likely to replace their commercial software products with them.

Actually, there's plenty of free software ported to non-free, proprietary, restrictive operating systems, which is in wide use. But the idea of educating people to switch largely failed (except for formats, which is a different issue), because people don't look at the ethical, legal and technological reasons for free software, but only for convenience. As long as something works or provides certain features, users tend to ignore the serious consequences they'll face later because of this choice. Free software on a non-free system even encourages the use of the non-free system, since one can benefit from both the proprietary software and the results of the free software world. Please note that it is about freedom and digital rights, not about commercial vs. non-commercial. Free software can be commercial software and is in several cases.

Originally Posted by Doitsu

While it does make sense to distribute software targeted at programmers and advanced users in source form, it doesn't make any sense at all to provide GUI-based software targeted at average end users in source form only. In doing so, software developers are basically discriminating exactly against those users who stand to benefit the most by switching to free software solutions, which also in a way flies in the face of most free software concepts, because, AFAIK, most definitions also include non-discrimination clauses.

There are things called “automated build systems”, so there isn't any difficulty or “discrimination” at all. There is the option of obtaining pre-compiled binaries from somebody one trusts (while free software licenses require that “somebody” to provide the source code along or on request) in order to save time. While binary-only distribution is non-free and binary distribution is still risky, there's no problem with source code distribution. As you might have noticed that I don't object against binary distribution of free software (with source code along or on request) for free systems, but as said, it makes no sense to provide binaries for non-free systems, since it defies the whole point of why the free software was written for initially. The source-only distribution is something you inserted, if I'm remembering correclty. And as of non-discrimination clauses, I don't know why you mention them since it is possible in general to provide binaries for free and non-free systems, while you have to keep in mind that you have no right to be provided with such, and even if there aren't any binaries available, it isn't discriminating at all, because the ordinary user isn't prevented from using the software, he just has to wait a little longer while the code gets compiled. You can't make it any faster, the developer and everybody else has to compile the code first before using a binary resulting from the compilation.

Originally Posted by DiapDealer

Just to be clear... I'm talking about standalone projects. Obviously addons/plugins and the like are exempt. I'm thinking of Toxaris here (who I wouldn't want to offend with my previous post). Clearly his Word addon can't be cross-platform; and that doesn't change its value (to me and others).

My point is: write code that will compile on as many platforms as possible. Test on as many platforms as possible and provide as many binaries as possible (along with the source) for those who have no interest in compiling. Let the user worry about whether or not it all fits their personal ideological preferences. That's my definition of "open." Not excluding anyone, for any reason.

I agree completely, but you have to keep in mind that I and a lot of other people don't possess non-free platforms, so there's no way for me and others to provide binaries for such systems. The reason for this situation is that using such a proprietary platform would automatically lead into a discriminating dependency, so I and others could be excluded from using that platform at any time, and are initially excluded from changing that platform etc.

Originally Posted by eschwartz

But you can always provide ReactOS binaries.

Indeed, because it is always legally possible to provide binaries for any system, free or non-free. As said, it is a question of if it makes sense. First, if it is technically possible to provide binaries, and second, if ReactOS is considered to be a target platform which should be supported (but still, if the answer is “no”, other people are entitled to provide binaries for it).

Originally Posted by Hitch

I don't seem to be able to communicate very effectively with you. I don't know if that's my failure, or your obdurancy. I'm truly not trying to be argumentative, but I don't much like wasting my time, either. You've pretty cavalierly blown off my five years of experience in dealing with the very marketplace you say you're aiming at, and in another post blow off the 90% of users on Windows software. I'm just not sure, at this point, how my input can be useful.

I don't target a marketplace, I target a solution. What I've already got out of this thread is the impression that Sigil isn't the ideal candidate to use it as a tool for semantic markup.

As for the description of your workflow, it looks like you're doing semantic markup by replacing direct formatting with CSS classes, and after you've done all the polishing of the input, my main interest is for automating step 10, and also extend step 10 by creating automatically a PDF from the very same polished XHTML input. This isn't difficult, it needs just a little time to implement it. For the steps 1-9 (starting with any number between 1-9), I initially asked with this thread, if Sigil could be used to make these steps easier by applying semantic markup to an input file, and if a word processor or a writing software would provide semantic markup initially, some of the steps 1-9 could be eliminated in the first place, if an author uses that software (if he doesn't, he has to pay you, I suppose). You yourself describe the issue of "MSONormal center bold" and "MSONormal 18pt Bold", which I don't like and for which I'm looking for a solution. That's the fault of the word processor to allow such direct formatting, which can't be processed automatically and is therefore less usable as an output with semantic markup. If the exported output is incomplete, that's pretty bad for the software that created the output, especially if the information is present if proprietary formats are used.

Originally Posted by Hitch

It would all be swell and good if authors had 5 styles to pick from, used 'em, and that was that. You could automate the process and Bob's yer uncle. But that's not what they are accustomed to, and that's not what they want. As Tox points out better than I could in his post, there are SCADS of tools out there already, that would work better (from a conversion standpoint), that authors already don't want to use.

That's exactly what I'm looking for! Maybe it won't be used by most people (with the consequences they have to pay according to this decision), but it will at least for myself and the people I'm working with. But I can't find *SCADS* of tools that do semantic markup, Doitsu mentioned AbiWord, I mentioned LyX. Any other?

Originally Posted by Hitch

I just think that you are expecting right-brainers to somehow magically see the advantages of working in a left-brained environment, and my experience, for what it's worth, is that that ain't ever gonna happen. Not only are they utterly disinterested in what is going on behind the scenes, they don't WANT to know, don't CARE to know, and somehow, think it makes them less creative if they understand the "how." This is my experience. Feel free to ignore it. However, of all the phone calls I take, I cannot tell you how many tell me either, that 'I'm not good with computers," or, even worse, "I'm really very tecchie but I need help with this," the latter of which means that when the time comes for that person to download a file from a browser interface, the s**t will hit the fan. Nor do they know where their downloads folder is, or how to drag-and-drop. That's what that last sentence means. I'm not disparaging them, but your idea just seems to utterly ignore the reality of a writer's inner world. That's how I interpret what you've said thus far. Sort of, "well, I'm making this tool for myself (which is fine; that doesn't faze me), and for writers, and if they don't want to learn it, the hell with them."

Basically, your description is accurate ;-) I can save them (and probably you) time at the back end - for the front end, as we've already established, there is no other way to either educate them or do the manual work for them, for which indeed one could get paid for. However, hopefully there is at all a tool available which makes the education part easy by preventing direct formatting, because if that feature is still present, much too often writers will start to shoot themselves in the foot again, since writing and typesetting are completely separated steps, and software shouldn't intermix it - especially if there's already a guy who gets paid to do the typesetting, formatting and e-book preparation, so all what's needed is to convey the information about what is meant to mean what, and there's no need to do this in a visually obscure way.

Originally Posted by Hitch

And if that's the entire gist--that you'll make it for yourself, and if anyone else wants to use it, they can--then great. But if you're asking everyone here for input and assistance and feedback, for a tool to be used widely, that would, purportedly, make OUR jobs easier, then you need to ALSO be open to the fact that maybe some of us might have a little more experience in the real-world environment in which you expect this to function. Just an idea.

Well, sure I'm open for that kind of feedback and suggestions, but before your last post, there wasn't any description of your process and workflow except complaints about the input, which is, as said, a problem which can only be solved on a case-by-case basis (manual work or the use of a semantic markup writing software).

Originally Posted by Toxaris

No sweat, I didn't take it personal at all. I am just happy more people than just myself find it useful and it actually gives me a happy feeling that it is used internationally.

In fact, it is a very good thing to provide conversions from secret formats and proprietary software to open formats. Hopefully this doesn't lead to a situation where people start to build workflows around your add-on (and therefore around the proprietary software), which would contradict the problem you want to solve initially. But I guess, you have no control over that...

Originally Posted by Freeshadow

That's exactly the point where you didn't listen to Hitch.
I've met enough people being just of the cut Hitch speaks of:
learn resistant homo habilis or lernresistentes Gewohnheitstier if you prefer.

That might be true for some cases, for others it is a business question (not for me, but for writers, formatters, distributors...some portion of people).

Originally Posted by Freeshadow

The only authors potentially willing to learn about good tools might be scientific ones because they heavily need a stable toolset for their work. Their work turns to shitpaper w/o a good reference apparatus so a lot of them happily switched from text editors aimed at authors
(like NotaBene) to *TeX.

I don't want a “good tool with a lot to learn about” of the kind you have in mind, I need a basic text editor with the ability of semantic markup (style templates) and no direct formatting features.

Originally Posted by Freeshadow

As for making people switch OSs because of software... Why?

No, it's because of ethics, law and technological usability.

Originally Posted by Freeshadow

Have you considered that they might use more than one program?
Where do you help if your tool isn't all systems available and so a potential switcher cannot try and use it while waiting for an alternative for a different piece of software s/he uses too?

That's just because up to now, incredibly much is built and depends upon proprietary platforms. Imagine a world where you buy a piece of software, and you actually own it. Imagine that you are able and entitled to change every aspect of it, and if you lack the skills to do it yourself, there would be a lot of people and services who would be glad to do it for you in a free market. All software would be interoperable and compatible, with proprietary software, there's nothing than artificial technical and legal restrictions.

Originally Posted by Freeshadow

If you want to pull people to Linux... Don't.
Linux isn't for people wanting a free windows.
Linux is for people wanting Linux - the other ones won't switch their thinking about how an OS should operate.

I don't care about the Linux kernel in particular that much...there are other kernels and other free operating systems as well. But if you think the Linux kernel is bad or the Unix-like way is bad because of some obscure reason, it seems you don't realize how good all of it works at the moment, and you might consider how incredibly it would work if you would throw all the money to the developers of free operating systems instead paying proprietary developers for making their software artificially scarce. There's no way how any company in the world could compete with their limited amount of employees with millions and millions of developers world wide. That those millions and millions of free software developers don't have the full effect at the moment is a result of mere protectionism on the proprietary side, they're kept busy by re-engineering proprietary drivers and secret formats, they're prevented from developing useful solutions due to patent issues and artificial technical restrictions. Questions about how an OS should operate don't play a role in this, since free software could provide all kinds of OS types, given the time or given the money. Without money, over time free software will survive and advance over proprietary software, and with money this will just increase the speed of progress. Only changes in the law in favor of the proprietary developers or technical counter-measures such as “Secure Boot” (= you can't boot the software you want) might prevent that, and proprietary software developers are exactly as the media companies known for their strong lobby. The Free Software Foundations of the world struggle to educate and draw attention to this issue and lobby on behalf of all computer users, since the users are already completely dependend on proprietary systems. The solution is to encourage the use of free software, to build alternatives for proprietary software and to educate about this problem, so it isn't a question about the Linux kernel or a free Windows or how an OS should operate. It's a question about which digital rights a user deserves, and the proprietary world has already established an answer for you: none. I don't agree to that answer, to the contrary.

Originally Posted by Freeshadow

P.S.: If Linus has no problem with offering his stuff for multiple OSes nonfree ones included: http://subsurface.hohndel.org/de
Why should you? More Catholic than the Pope eh?

Linus Torvalds isn't a representative for the free software movement, he is a representative of the open source movement. He doesn't care about freedom at all, which can be seen in a lot of instances. The linux kernel developers violate their own license by taking so called “binary blobs” into it, which is binary distribution without source code. Seems you haven't ever heard of Richard Stallman (founder of the Free Software Foundation), who started it all as an organized effort? See here (especially important for e-books!), here and here (sorry, no playlist, 16 parts).

Originally Posted by st_albert

Not. Going. To. Happen.

Thanks for your description of your workflow! Since you work for a small press “traditional” publisher, it seems you are already in advantage. More cleaner manuscripts, long-term relationship to the authors (publishing more than one book I assume, since the first book won't make money for the publisher usually) etc. So I cited the “Not. Going. To. Happen.” since it indeed might be an issue that can never be solved. On the other hand, if you get manuscripts in Microsoft Word which already contain direct formatting, I wonder if it would be possible for the second book and following ones to introduce the “collaborative, web-based process” or a semantic writing software to your customers (and giving them a lower price or some other benefit for the time they saved you) since they would just click for a style like they now click for a direct formatting (however, styles still should be displayed as true or approximate WYSIWYG), and while I can't imagine to write texts initially into an online form (but could be used to paste text into it and apply formatting there instead of plain text with just some hints for the formatter), as no advanced Word features are used, a different software could be used to write into (maybe a modified version of LibreOffice). Is it that the users think that they can't access their texts in the future if it isn't saved in Word format? Is there a mental detachment that texts not saved in Word can't look good on a screen or won't be nicely printable on the printer at home? Such consideration would indeed be an unsolvable problem, and I guess that's the real case why we can't have semantic markup and automated processing workflows based on a clean input. If accurate, nobody than word processor developers could solve the issue, and since they're not going to do so, automated processing workflows and front ends to feed them will remain a benefit for professional users only, even if they're easy to set up and commonly available to everybody.

Originally Posted by st_albert

After epub export from either InDesign or LibreOffice, it goes to Sigil for chapter splitting, adding the cover, adding lots of metadata to the .opf file, and generation of TOC's (typically both an .ncx and an inline TOC.xhtml).

I guess this doesn't take you long time, since I'm already able to programmatically create EPUBs. Chapter splitting and generation of the TOC (both .ncx and inline XHTML TOC) could be done automatically, cover, metadata could be added automatically (from a database source or a configuration file).

For print, I assume you use the InDesign PDF export after you've imported (copied?) it from the Word source, right?. My workflow would look like doing the clean-up in LibreOffice, export to semantic XHTML, generate EPUB and PDF from the XHTML automatically. Maybe the gain in time and manual labour is too less to consider an automated processing workflow for it, since the InDesign file is already used to generate both, PDF and EPUB (at the expense of the need for InDesign).

Originally Posted by Hitch

Here is an ACTUAL sample of what we deal with; I have a woman that's been emailing me for the last 6-12 months. She had a book in print, in Europe, as she is. She had a "friend" do the scanning of the print book, in France. The resulting scan had all the usual horrors--page numbers midst-page, section breaks, varying margins, columns set (yes), you-name-it. She couldn't even begin to work on it. The friend didn't use Abbyy; used a desktop scanner and some type of older scanning software.

You might find it funny to find out that I for the very same reasons typewrite old texts by hand in order to digitalize them ;-)

Originally Posted by Hitch

I've just looked and took the first few pages, the Prologue, and justified the margins. This corrects most of the broken paragraphs in one move. I'll do it chapter by chapter as I go.

Yes...and now, who's fault is it? Yours? Obviously not. The ladie's? Probably not, how should she know? It's the fault of the word processor developers, since a writer shouldn't think at all about things of typesetting such as margins and stuff. If the text goes to e-book, there isn't any margin. If it does to print, depending on the paper dimensions, there might be no room for margins.

In regard of your last posts, they helped me a lot to more precisely understand the users point of view, since I just came to the idea that writers might be simply “used to Microsoft Word”, which seems to do what they want (which is a bad trick of the word processors), and since writers have out-sourced the e-book and print preparation to you guys, they never get to see the implications and consequences of their decisions (and only might wonder about the fee for something looking fairly simple). Not only is Sigil a tool for the back end (not writing in it directly, applying template styles for semantic markup unlikely in Sigil), also there's no way to create an alternative, because the user can't associate the benefits for using it (except you provide him a lower price or additional results or service, if clean input is provided). To develop a completely integrated automated processing workflow from the writing front end to the output in various formats as a single application, is a very complex task, so it's probably a better idea to work on specialized processing workflows. I might get in touch with the LibreOffice people, if there could be a mode introduced without direct formatting, but as with margins, this could be considered as contra-WYSIWYG and therefore not of relevance. In any case, there are still lots of uses for automated processing workflows, there might just be no common solution for writers of ordinary texts, since this opportunity is prevented by technological decisions made a long time ago and supported up to the present day. Thanks for your patience, hints and insights, it at least saved me from considering wasting a lot of time on Sigil development for features which are urgently needed in other pieces of software.

[Hitch]

Very quickly:

I think, to speak to solely one thing, you missed the funny in my telling of the lady's reply about the broken paragraphs; when she said "I've just looked and took the first few pages, the Prologue, and justified the margins. This corrects most of the broken paragraphs in one move. I'll do it chapter by chapter as I go," she wasn't actually setting margins, as the goal; she thought she was FIXING broken paragraphs, because the ragged line endings she could "see" with her naked eyes magically moved to the right margin (when she chose "justified"). I'd been trying to get her to fix the pilcrows (paragraph codes) appearing mid-sentence and mid-paragraph that is the inevitable output of AbbyyFineREader

THAT was the funny. Not the typographic aspects. That would have to be fixed (broken paras) whether in markup, markdown, Word, OO, Latex or Bob's Big ePUB-Baker. ;-)

Hitch

[Toxaris]

I will only respond to two parts of your dialogue, since I just don't agree with you. All noble initiatives aside, this is a lost cause in my opinion. To each its own I guess...

Originally Posted by skreutzer

That's exactly what I'm looking for! Maybe it won't be used by most people (with the consequences they have to pay according to this decision), but it will at least for myself and the people I'm working with. But I can't find *SCADS* of tools that do semantic markup, Doitsu mentioned AbiWord, I mentioned LyX. Any other?

There are more, like yWriter and Scrivener. They free to use, but not open source. Then again, they would not work for you anyway, since they have either been compiled for Mac and/or Windows. They can run on Linux (or at least yWriter can) via WINE.

Originally Posted by skreutzer

In fact, it is a very good thing to provide conversions from secret formats and proprietary software to open formats. Hopefully this doesn't lead to a situation where people start to build workflows around your add-on (and therefore around the proprietary software), which would contradict the problem you want to solve initially. But I guess, you have no control over that...

As I have said multiple times, the DOCX format is not secretative or proprietary. It is a well documented open format. It is the OOXML (Office Open XML) and been accepted as an open standard in 2006 by ECMA and in 2008 by ISO.
The old format was a closed proprietary format. The new one isn't and can be used by anyone if they want, regardless of the OS.
The reason I created this, is that I want another type of output not suited for a wordprocessor. Actually, a lot of information very relevant to a wordprocessor is thrown out the window since it has no use in an e-book. So, that is not a good output for a wordprocessor.
Without a doubt someone can come up with a document that my add-in chokes upon. I am always looking for documents that can actually that, since it helps me to improve the process. Still, the add-in was initially created to help my own process and for some others. That grew to support more, but in the end I still make the decisions. I can say that if I do not longer support the add-in, it will be open-source. Untill then, I keep it closed source for various reasons.

[Doitsu]

Originally Posted by skreutzer

But the idea of educating people to switch largely failed (except for formats, which is a different issue), because people don't look at the ethical, legal and technological reasons for free software, but only for convenience.

No, it largely failed, because most free software solutions still don't offer some features of commercial software products that end users depend on.

Sigil is one of the exceptions, because, IMHO, there's no commercial product with a similar feature set for advanced users.

Originally Posted by skreutzer

There are things called “automated build systems”, so there isn't any difficulty or “discrimination” at all.

Yet none have been set up for Calligra Suite and Abiword, which is usually an indicator that building these software products is not a trivial matter, even for programmers.

Programmers who claim to provide free software, which actually only 10% of end users can use, are effectively discriminating against 90% of all other users. Saying that the other 90% are free to build the software themselves is akin to saying that millionaires are free to sleep under bridges.

It would be more accurate to define such software as "free software for Linux/Unix users and experienced programmers only."

If you want to provide a useful software product that helps both authors and ebook producers, you'll have to get off your high horse and drop your very unproductive holier-than-thou attitude, because it'll get you nowhere.

[skreutzer]

Originally Posted by Hitch

I think, to speak to solely one thing, you missed the funny in my telling of the lady's reply about the broken paragraphs; when she said "I've just looked and took the first few pages, the Prologue, and justified the margins. This corrects most of the broken paragraphs in one move. I'll do it chapter by chapter as I go," she wasn't actually setting margins, as the goal; she thought she was FIXING broken paragraphs, because the ragged line endings she could "see" with her naked eyes magically moved to the right margin (when she chose "justified"). I'd been trying to get her to fix the pilcrows (paragraph codes) appearing mid-sentence and mid-paragraph that is the inevitable output of AbbyyFineREader

THAT was the funny. Not the typographic aspects. That would have to be fixed (broken paras) whether in markup, markdown, Word, OO, Latex or Bob's Big ePUB-Baker. ;-)

Oh, I think I understood it correctly: the adjustment of the visual appearance seemed to fix the mid-sentence and mid-paragraph problems, while indeed those problems remained in the document. My response might have been unprecise or misleading, because I also consider this as part of the issue of combining typesetting with writing. From the perspective of typesetting, the result after the adjustment looks visually OK, from the writing perspective, the text is corrupted by a lot of stuff that's not supposed to be there, which breaks the usability for any other target format than directly printing the visually adjusted Microsoft Word file via PDF. I just confused changing the margin with setting the paragraphs to "justified", but both would only be a visual adjustment. Two ways to solve it: visually show the pilcrows and fix them manually or prevent them from getting inserted into the text in the first place.

Originally Posted by Toxaris

There are more, like yWriter and Scrivener. They free to use, but not open source. Then again, they would not work for you anyway, since they have either been compiled for Mac and/or Windows.

You are correct. If they were freely licensed and wouldn't depend on proprietary programming libraries, binaries for free operating systems could be compiled. However, even if the output of yWriter and Scrivener could be usable for automated processing workflows, for myself and others I need a free writing program with semantic markup support, because otherwise it would require a proprietary application in order to be able to benefit from the freely licensed processing workflow tools, a dependency which I have to avoid.

Originally Posted by Toxaris

As I have said multiple times, the DOCX format is not secretative or proprietary. It is a well documented open format. It is the OOXML (Office Open XML) and been accepted as an open standard in 2006 by ECMA and in 2008 by ISO.

You should read this: http://web.archive.org/web/201311270...XML_objections - Microsoft is very known to specify formats, which are documented publicly, but contain "features" which make it hard to implement by artificial obstacles for the sole purpose of protectionism. Since there's also a better open specification to do the same thing (ODT), DOCX should be considered proprietary, because otherwise it would look more like ODT.

Originally Posted by Toxaris

The reason I created this, is that I want another type of output not suited for a wordprocessor. Actually, a lot of information very relevant to a wordprocessor is thrown out the window since it has no use in an e-book. So, that is not a good output for a wordprocessor.
Without a doubt someone can come up with a document that my add-in chokes upon. I am always looking for documents that can actually that, since it helps me to improve the process. Still, the add-in was initially created to help my own process and for some others. That grew to support more, but in the end I still make the decisions. I can say that if I do not longer support the add-in, it will be open-source. Untill then, I keep it closed source for various reasons.

I agree completely, but again, don't think that publishing the source code (which is far from actual free licensing) wouldn't change much, except that it could improve the security of your users and encourage custom adjustments (without collaboration on improvements).

Originally Posted by Doitsu

Yet none have been set up for Calligra Suite and Abiword, which is usually an indicator that building these software products is not a trivial matter, even for programmers.

Indeed! Still, if somebody at some time gets it done, he can provide a how-to or package binaries for users who trust him. For instance, I can't build Sigil on my system because I can't figure out the cause of a build problem. I just don't know the reason for it, if I would know, I probably could fix it myself. So all that's missing is more or less a hint how to solve it, and if I could solve it, I could help other people who have problems which I up to know had already solved myself. So it's more a question of a broad user base, popularity, for such things to become common for an application.

Originally Posted by Doitsu

Programmers who claim to provide free software, which actually only 10% of end users can use, are effectively discriminating against 90% of all other users. Saying that the other 90% are free to build the software themselves is akin to saying that millionaires are free to sleep under bridges.

No. The 90% chose to use systems for which it is impossible to develop free software, so I don't see a reason why free software developers should waste time for making proprietary operating systems look more attractive, which is the exact opposite of what free software developers try to achieve with their program in the first place.

Originally Posted by Doitsu

If you want to provide a useful software product that helps both authors and ebook producers, you'll have to get off your high horse and drop your very unproductive holier-than-thou attitude, because it'll get you nowhere.

I don't mind if you think of me like this, but I wonder how you can think that the concept of free software could be "unproductive" and get me "nowhere". In general, there can't be any doubt that such impression is false, and for this specific case, I've already proofed that I can implement the backend, and I'm at the moment working on generalizing it, so the missing front-end is the topic of this thread - even if I do nothing more on the topic, for myself the LibreOffice XHTML output is quite sufficient, but I'm looking for a solution which also other people could use, which I could give to them in order to write in that application initially or later copy the plain text into it in order to do the semantic markup with it.

[DiapDealer]

Originally Posted by skreutzer

but I'm looking for a solution which also other people could use

To put it quite frankly... if that were true, we wouldn't be having this conversation. You would be done and would have released what you have. But it's very clear that the ideological discussion surrounding the concept of "free software" is of much more concern to you than actually creating usable software that is free (for everyone) to use. Philosophy and application development (free or otherwise) just don't mix.

In short: less talkey more codey.

[roger64]

Hi

@skreutzer

I am a Linux user, and I use mostly free software to produce EPUB (but not PDF...).

From LibreOffice I use a converter from odt to EPUB which produces very usable xhtml files (and a very correct EPUB2). I use Sigil to add any other features I wish.

The converter is writer2xhtml. It works well with OpenOffice, but you need a patched version for LibreOffice 4 and beyond because of a minor Java problem.

This software has not been developped for nearly 20 months and, as hard as I can, I got no news from its developper, Henrik Just, which is also the author of writer2latex. He just seems to have disappeared into thin air.

I know nothing about semantics, but maybe you could have a look at his efficient code (LGPL licence) to see what could be salvaged for further use.

[skreutzer]

Thanks for your hint! Unfortunately, I think writer2epub and writer2latex aren't automatable, and for my purposes I would also like to be able to intervene intermediate processing results to do custom adjustments to it. Since I've already implemented XML to EPUB2/EPUB3/LaTeX/XSL-FO converters, those transformations aren't rocket science to me. However, do you know of some special features of writer2epub and/or writer2latex which you think I could overtake from those tools? Do you want to produce PDF via LaTeX while writer2latex isn't working yet? Do you need any bug fixing for those tools or do they just work fine? As for writer2epub, the EPUB output could also be used as input for an automated processing workflow, so in general it could make sense to improve and to support writer2epub. The only disadvantage of writer2latex is that LaTeX isn't XML, so I would consider it as a target output format, not as an intermediate format for automated processing workflows. Besides of that, I still appreciate such a tool - but I guess OpenOffice/LibreOffice PDF output is already fairly nice, isn't it?

I'm also confident that the code of those tools handle much more than my simple basic first version of html2epub, because those tools are quite around for some time and I just developed in less than a month in spare time, so probably there's lots of things which I could look for solutions in the code of those tools. However, I just implemented what I needed for my own purposes, those tools probably aim for most complete support of ODT features. I'm not sure if I should have the same goal right from the start, as there is much of other things which need work, too. But over time, more features may be added, maybe based upon writer2epub and writer2latex solutions.

Edit: Actually, I guess writer2latex is strongly influenced by the direct formatting issue, so writer2latex has to implement a LaTeX replacement for whatever may occur in ODT, and might still not be able to produce an identical result (but maybe is able). Semantic markup would simplify the case, so that OpenOffice/LibreOffice would be just the tool to write in and to apply style templates, and print typesetting by LaTeX would be an entirely separated step from it, resulting in a different visual appearance as the OpenOffice/LibreOffice WYSIWYG.

[roger64]

Please pay attention to this: I do not use writer2epub but writer2xhtml written by Henrik Just. writer2xhtml is an extension of OpenOffice/LibreOffice. Unhappily I do not know writer2latex.

I can give to writer2xhtml any odt file to convert and it manages to export to a clean EPUB2 that I can later perfect if need be.

Among the nice features I use often are its ability to append an external css stylesheet and embedded fonts, and its convenient use of style-mapping. The preference panel offers a broad choice of options, among them applying manual formatting (bad), or only style formatting (good). You can also fine tune the splitting of your document and many other things.

I have been using it for the last three years.

For PDF export, LibreOffice and OpenOffice are quite good but they export odt files.
What interest me is exporting EPUB to PDF, so that I can also benefit from the enhancements made using Sigil. Calibre can do it but I prefer using Prince which I find more precise for this task (Calibre is of course much more versatile and can be used for a gazillion other tasks).

[st_albert]

Originally Posted by roger64

Hi


I am a Linux user, and I use mostly free software to produce EPUB (but not PDF...).

From LibreOffice I use a converter from odt to EPUB which produces very usable xhtml files (and a very correct EPUB2). I use Sigil to add any other features I wish.

The converter is writer2xhtml. It works well with OpenOffice, but you need a patched version for LibreOffice 4 and beyond because of a minor Java problem.

Hi, Roger64: Do you know where that patch can be obtained? I've also been using writer2xhtml routinely, but since upgrading to LO ver 4+ I can't use it as an add-in. It does, however work very well as a stand-alone program to convert the .odt file to epub2 (or xhtml, for that matter, but I haven't needed to try that). @skreutzer take note.

Using it stand-alone is very convenient, as it can be configured/customized to use our house styles, etc. It seems much faster run from the command line than the add-in used to be.

Hmmm, now that I think about it, I wonder if it could be configured to convert the "direct formatting" to our house character styles? I'll have to look into that. Completely automating that process would be playing with fire, though. Some of the conversions need the input from the "Mark-1 eyeball."

Albert

[skreutzer]

@roger64: Oh, sorry, it is my fault! I just misread writer2xhtml as writer2epub, since I've read about the latter one before and writer2xhtml didn't came to my mind as a separate tool because OpenOffice/LibreOffice has already a XHTML export integrated in the GUI. I looked a little bit into the description of those tools, and fortunately they're written in Java (portable, widely installed), LGPLed and usable as command line tools, so they would be automatable. Regarding your description of the features, those tools could be the missing link between LibreOffice as front end for semantic markup and an automated processing workflow as backend, with the opportunity of highly customizable intervention at each processing step.

As you might probably know, EPUB isn't much more than zipped XHTMLs, so by unzipping an EPUB, you could use the files directly for a xhtml2latex or xhtml2fo tool I'm going to develop in the future. Still, the XHTML in the EPUB needs to be semantic (where Sigil may come back into the picuture...).

@st_albert: Full automatization could only be achieved if LibreOffice would enforce semantic markup and prevent direct formatting (probably as a special mode). All you would have to do in such a case would be to match your house styles to the styles found in the ODT. If possible, you could export your house styles to the writer, he would import them, so there wouldn't even be a need to match styles - still, LibreOffice seems to be not capable of such template exporting/importing, and it would only work for authors or content providers who you can convince to use the templates in the first place with LibreOffice. If not, you or a formatter may use LibreOffice to apply your house styles to the text written in another writing tool, word processor or text editor.

Now I'll go and try the standalone tools for myself, so that they probably can be integrated into a fully automated processing workflow. Without question, such an implementation would need quite some time, but if it gets solved once (and over time, I'm confident it will), it can be used by everyone since it is free software, and also extended and improved for all kinds of special purposes and customizations.

[skreutzer]

I just successfully built the writer2latex package from the sources (latest version of the public repository) on gNewSense 3.0 with OpenJDK 1.6 for OpenOffice 3.2.1 and did a first conversion from ODT to valid XHTML. The result looks slightly cleaner than the adjustments I did by hand in my demo video (I used the same input ODT file). So there isn't much writer2latex saved me, but I guess now I may benefit from all of the command line options, so that the conversion results might directly feed into an automated processing workflow, where I will now also try ODT to EPUB and LaTeX conversion. For the described operating system environment, I probably may provide some kind of support, also small fixes, if needed. However, it would be most interesting to find out on how dependent the writer2latex is on OpenOffice libraries and API - hopefully not much, so it could even be modified to take other input files than ODT, like XHTML or custom XML, too. For the output, the writer2latex package tries to represent the visual appearance of the input ODT file as closely as possible, which quite isn't the best concept for automated processing, since there may be several output formats where there's no sensible way to represent the OpenOffice WYSIWYG appearance even with approximately similarity. Additionally, in order to be flexible, it is required to be able to replace portions of the data with custom content or styles. It might be too early to estimate how useful the package is and if/how it could be changed, but it already works out of the box, and I might use it to build a fully automated workflow in order to have a first brief demonstration for real-world application.

In case such a solution could be interesting for some of you, I guess it would be better to fork this conversation about writer2latex specific issues and updates on my experiments with it.

Please note that I'll refer to the standalone tools of the writer2latex package instead of the OpenOffice extension, since manually clicking on things isn't a solution for processing in bulk, and the need of starting OpenOffice is already eliminated if an ODT file is provided.

Update: Obviously, the writer2latex package makes extensive usage of configuration files, which is ideal for automated processing. The user manual describes almost all the features which I would want for the task, so my own solution would have looked like quite similar (but probably without the goal to preserve the OpenOffice WYSIWYG as close as possible), and by integrating writer2latex development time from 2002-2012 might be saved. There's still the issue of dependency on OpenOffice API libraries, which might be OK or solved by replacing the dependencies with ordinary ODT XML reading. The documentation doesn't mention EPUB output, but the XHTML conversion allows the removal of direct formatting, style name matching and the insertion of custom stylesheet references. Even if EPUB conversion doesn't support such features, the XHTML output itself would be sufficient to be integrated to a good EPUB, or the EPUB converter might be extended to provide the same features. If for the input side the dependency on OpenOffice could be removed from the standalone tool, and also (for instance) XHTML or plain text added as input formats (or by generating an ODT from XHTML, plain text, RTF as first step of the processing workflow), the package would be an invaluable part of automated XML document processing. Missing output formats and customization features could be added, predesigned configuration files provided, and GUI tools developed in order to enable ordinary authors to edit the configuration files. The future efforts might be to limit the things the writer2latex package tries to do itself, and to combine it with specialized transformations which will take care of complex processing steps in a more readable and customizable way. I'll continue to investigate.

[roger64]

Originally Posted by st_albert

Hi, Roger64: Do you know where that patch can be obtained? I've also been using writer2xhtml routinely, but since upgrading to LO ver 4+ I can't use it as an add-in. It does, however work very well as a stand-alone program to convert the .odt file to epub2 (or xhtml, for that matter, but I haven't needed to try that). @skreutzer take note.

.../...

Albert

I had published it on the writer2latex forum but do not find the link anymore...

Anyway, I had published it to on MR too and so you will find the links and the explanations for the patched version here:
https://www.mobileread.com/forums/sho...&postcount=224

Hope it works for you.

[roger64]

Originally Posted by skreutzer

In case such a solution could be interesting for some of you, I guess it would be better to fork this conversation about writer2latex specific issues and updates on my experiments with it..

I am quite happy that you took on writer2latex, and I would be very interested to know about future developments. As you intend to follow on this path, it's probably a good idea to create your own thread about this project.

Maybe some MR expert could advise you about where to create such a specific topic on MR?

[skreutzer]

Seems like writer2latex is pretty well programmed in terms of architecture - unfortunately, it seems like the basic concept of the entire package is to perform ODT to XYZ conversion as an inseparable step, which is configurable, but heavily relying on OpenOffice. I started to try to detach the EPUB generator from the package to some extend in order to get a standalone XHTML to EPUB converter, which is difficult because the EPUB packer only takes XHTML as result of an ODT conversion. So without further modifications, writer2latex's EPUB generator can't be used to take any XHTML export from any writing software, except if a writing software would be able to export to ODT in the first place or XHTML to ODT is performed as the first step. The benefit of writer2latex's concept is that configuration files probably won't need to be interpreted and applied several times but only once, but probably the configuration is different for each conversion anyway. I don't know if it is worth the time to invest for a standalone XHTML to EPUB converter which would just resemble what my html2epub tool does already, so I may abandon this first try and change to LaTeX conversion and use ODT to XHTML of writer2latex as part of an automated processing workflow, which then could be the input for my html2epub tool.