Future Thoughts on Getting Decent epub 3 support

[DiapDealer]

Originally Posted by CliffB

To KevinH & DiapDealer:
I understand the issues and appreciate your insight. Hopefully there will be an alternative to WebKit before it is removed from QT5, or other unsurmountable problems come up.

There already is an alternative. But it would still take a lot of work to port the existing Book View to use it.

[Hitch]

Originally Posted by CliffB

To st_albert & Hitch:
Yes, we are a very different type of user, but there are a many of us out here. We are small independent publishers and authors. Personally I work in the Seattle and Portland areas but sometimes work with OpenStax CNX in Houston. We are quite tech savvy, most of us can hack HTML, CSS, JavaScript. However, we do not like to hack code while we are writing because it is a huge distraction and reduces productivity (hence the value of Book View).

I have 4 books out now for which I used Sigil as the final editor. Each one was written in LibreOffice or PanDocs Markdown and I used Google Docs for research and collaboration with other authors. My intention was to import them into Sigil as the last stage of the tool chain and then do the final edit. Of course they dragged on for another 6 months to a year and were substantially rewritten AFTER being imported to Sigil. I would say that 1/4 to 1/2 of the total writing was done in Sigil when all was finally said and done.

In your usage case, you are the final editor?? In this case the author has finished and passes the work to you for final editing. In our case we are the author AND the final editor.

Assuming that you were asking me (and Albert):

We're not a publisher, nor do we (generally) provide editing services in which we are making the changes, rather than implementing the changes requested by the authors. We are, for all intents and purposes, the modern-day equivalent of the printer--totally GIGO. (Although I admit, there are certain things that get, ahem...tweaked...if I see them. e.g., "Forward" is a pet peeve of mine. As that can often appear opposite our credit line on a copyright page, we change it if we see it.)

In other words: an author, an imprint, or (for white-label) a layout house sends us a file. This file will be Word/Pages/OO/LO, for the first two, or an INDD file, for the latter. We create a review set of eBooks (ePUB/MOBI). Those go out to the client. When the client finds the ubiquitous edits, changes, typos, etc., we receive a Proof Form (usu. in Word), and we implement the changes. The revised files are sent to the client. Lather, rinse, repeat until the book is "approved." With the clients and the small imprints, it's a somewhat "retail" experience; lots of hand-holding and such. With the White label clients, it's in, out, done. (My dream in life is to have all white label and imprint clients.)

Does that clarify?

Hitch

[CliffB]

Yes, Hitch I was asking, thanks for the insight.

How much time do you spend, and what tools do you use to import or convert content?

We find import and conversion to be very problematic and time consuming. We have used various tools such as PanDoc, Writer2Xhtml, Calibre, etc. Lately many of us have just given up and simply cut and paste the content (as text) into Sigil (in Book View) and then quickly apply formatting. The original content is open in one window in the originating application (LO, Word, Google Docs or whatever) and Sigil is in another window, both side by side. We cut and paste, apply formatting, compare and make small changes one chapter at a time. This may seem tedious but it actually is faster and more reliable than using a conversion program most of the time.

Using conversion tools such as Pandocs, Writer2Xhtml or Calibre sometimes works good with some content, but other times results in missing content or mangled content. Conversion with these tools is not always reliable. Some source documents were originated in Word, LO or Google Docs, opened in Word or LO and then imported back to LO or Word. There are levels upon levels of tag soup in most of these files.

Cutting and Pasting into Book View as text results in totally scrubbed and cleaned files. Re-applying any lost formating is pedestrian compared to sorting out the tag soup.

[Hitch]

Originally Posted by CliffB

Yes, Hitch I was asking, thanks for the insight.

How much time do you spend, and what tools do you use to import or convert content?

Well, the first part of the question is too difficult to answer in a simple number. In fiction, we have a named scale: Gilligan, Professor and Skipper. (FYI, all: these are internal names, and I'd really like it if you wouldn't repeat them.) As you've probably already inferred or discerned, a Gilligan is a 3hour book (standard fiction, <75K), a Professor is a 4-hour book and a Skipper is a whole lotta clean-up and fat that needs removing...6+ hours. Now, these terms were created back in '09 or so, when it was easier to create a book than it is now. To many different devices, differing base formats at Amazon, media-queries, etc. So, really, a Gilligan is probably not less than 5 hours, and so on. But we like the names, so we kept 'em.

We find import and conversion to be very problematic and time consuming. We have used various tools such as PanDoc, Writer2Xhtml, Calibre, etc. Lately many of us have just given up and simply cut and paste the content (as text) into Sigil (in Book View) and then quickly apply formatting. The original content is open in one window in the originating application (LO, Word, Google Docs or whatever) and Sigil is in another window, both side by side. We cut and paste, apply formatting, compare and make small changes one chapter at a time. This may seem tedious but it actually is faster and more reliable than using a conversion program most of the time.

There is, quite simply, no magic tool. For fiction, we export to HTML; and then depending on the book, it's cleaned either in NoteTab Pro with our proprietary clips, or Epsilon, ditto. Epsilon is our cleaner and coder for more complex books. It's just the way it worked out.

Nonetheless, it's export HTML (from whatever) or get an ePUB from INDD. Then either NTPro or Epsilon, for the basic cleaning. Now, obviously, this can be quick...or not. Some Word files, short fiction, only require a single pass. I (personally) use Toxaris' ePUB tools plugin to assist in pre-cleaning Word files. Despite my old beliefs, there are some things that are done more rapidly, or more easily, in Word, pre-export.

The bulk of the clean-up is in these tools. NTPro, Epsilon, and what Barb (our head bookmaker) calls "her girl PERL," a program she's written that does a series of repetitive tasks.

Once the HTML is cleaned and styles are assigned to the various elements, the HTML and CSS are imported into Sigil if we're doing fiction. And Epsilon if we are doing non-fic or heavily laid-out fiction. To stay on-topic, assuming it's fiction and it's pushed into Sigil.

At that point, the file splits are inserted; fonts are imported and subset; the file is previewed; tested against ePUBcheck, and then finalized.

Using conversion tools such as Pandocs, Writer2Xhtml or Calibre sometimes works good with some content, but other times results in missing content or mangled content. Conversion with these tools is not always reliable. Some source documents were originated in Word, LO or Google Docs, opened in Word or LO and then imported back to LO or Word. There are levels upon levels of tag soup in most of these files.

I've not found any insta-conversion program that works worth a crap. Some of the guys here on MR have stated that using the Calibre EDITOR (not the converter) works really well. This is using the Editor as an import tool--so, you'd export your HTML to your HTML editor, clean it up a bit, and then paste it into the Calibre editor, and pop out an ePUB. Of course, you could also use the Calibre editor as an HTML editor, too, to simplify it even further.

Cutting and Pasting into Book View as text results in totally scrubbed and cleaned files. Re-applying any lost formating is pedestrian compared to sorting out the tag soup.

You know...it's really not that hard to clean up the "tag soup," as you call it. A lot of what you've described, both here and previously, is adding to your woes; all the back-forth with your writers and all that. I think that the time you're "saving," by having the authors and publishers work in their own files--so you don't have to--is being eaten up on the flip-side when you go to convert.

If you have to work in a shared environment, where they'll be making edits and all that, why not something like Jutoh or Scrivener? Wouldn't that be easier? If you and your authors had Scrivener, you could simply ship the .scriv file back and forth, OR, each of you could work in the file, push a button, create an ePUB or MOBI for proofing, and bobs-yer-uncle.

I admit that I am a bit boggled by the process you have. If I may, why are you working in ePUB? As a reviewing medium?

Hitch

[eschwartz]

Originally Posted by Hitch

I've not found any insta-conversion program that works worth a crap. Some of the guys here on MR have stated that using the Calibre EDITOR (not the converter) works really well. This is using the Editor as an import tool--so, you'd export your HTML to your HTML editor, clean it up a bit, and then paste it into the Calibre editor, and pop out an ePUB. Of course, you could also use the Calibre editor as an HTML editor, too, to simplify it even further.

Well, importing an HTML file into calibre's Editor is not really different from importing an HTML file into Sigil. Mostly because the file is imported as-is in either case.

But calibre includes a DOCX Input plugin* for the conversion backend, which is also used to open a DOCX directly in the Editor.
Unlike conversion, importing a DOCX into the Editor does not go through stage two of the conversion pipeline -- the CSS flattening.

Conversion:
1) Input plugin, DOCX ==> OEB
2) CSS flattening and other monstrous nightmares, OEB ==> OEB
3) Output plugin, OEB ==> EPUB

Editor:
1) Input plugin, DOCX ==> OEB
2) OEB ==> EPUB

See calibre's manual: Introduction to conversion

...
...

Given that the Editor now understands DOCX properly, I can only regard converting DOCX ==> EPUB in order to get an EPUB for editing, as nothing short of medium insanity.

...
...

So, "Import an (HTML or) DOCX file as a new book" is not substantially different from using Toxaris' Word addin to export EPUB; obviously, the Word addin uses a different method for translating DOCX into an EPUB, but the same general goal is present in both. I have no idea which route you might consider to have preferable output -- maybe they're both good/bad in different ways.

There's a fun experiment for you.
Compare a series of books, exported with Toxaris' tools (just a straight export, none of the other fancy pre-processing options) vs. imported directly to the calibre Editor.
Then tell us all, in your expert, professional opinion, which one does a better job of preserving styles and producing as clean an EPUB as you can actually get from Word.


EDIT:
* -- DOCX Input plugin is one of the builtin plugins. Sorry technically most things in calibre are expressed as plugins, that's the reason why people can write new, external plugins for new formats, e.g. the KEPUB plugins.

[Hitch]

Originally Posted by eschwartz

There's a fun experiment for you.
Compare a series of books, exported with Toxaris' tools (just a straight export, none of the other fancy pre-processing options) vs. imported directly to the calibre Editor.
Then tell us all, in your expert, professional opinion, which one does a better job of preserving styles and producing as clean an EPUB as you can actually get from Word.

Hmmm...is there a way to Un-K someone around here?

You daft bugger. As if I had that kind of spare time.

I can, however, grab the next X files that come through, and do a quick-n-dirty comparison. I'll have to grab the Calibre plugin for importing Docx.

I happen to have a fully-styled Word file (an out-of-copyright Agatha Christie), that I created for a tutorial. I can run that comparison on that one, to get started.

I'll let youze guys know what I think. This will NOT be before Xmas, kiddles.

Hitch

[Toxaris]

One thing though, with my export tool you can preserve the stylenames without a problem. There is an option if you also want a stylesheet created with those styles. Otherwise the names will be retained, but you need to define the styles. This is deliberate, because it is quite easy to mangle styles in Word, giving strange results.

[HarryT]

Originally Posted by Hitch

I can, however, grab the next X files that come through, and do a quick-n-dirty comparison. I'll have to grab the Calibre plugin for importing Docx.

I'm pretty sure that Calibre can handle docx files natively, Hitch. The plug-in is for doc files, I believe.

[BetterRed]

Originally Posted by HarryT

I'm pretty sure that Calibre can handle docx files natively, Hitch. The plug-in is for doc files, I believe.

I can confirm that 100%, DOCX is built in.

Just start the editor (ebook-edit), then File->Import HTML or DOCX. I use it every day.

BR

[eschwartz]

Originally Posted by Hitch

Hmmm...is there a way to Un-K someone around here?

You daft bugger. As if I had that kind of spare time.

[st_albert]

Originally Posted by CliffB

To st_albert & Hitch:
In your usage case, you are the final editor?? In this case the author has finished and passes the work to you for final editing. In our case we are the author AND the final editor.

My situation is -- from the standpoint of workflow -- similar to Hitch's, though I do most of my freelance work for a single publisher. I do no actual editing, modulo some occasional copy edits. I get the "final" manuscript as a .doc(x) and either import it into LibreOffice or InDesign, depending on whether there will be a print edition or not. Whether it'd ID or LO, I then apply a set of standard paragraph and character styles ("House styles" intended to give a certain "look and feel" to the manuscript) and remove all "direct" or "ad-hoc" formatting. Then export as epub.

In the LO case, I use the stand-alone javascript "writer2latex.jar" supplied with writer2xhtml, along with heavily customized config.xml and a CSS file that matches the LO styles in name and definitions. It actually works better than ID's "export to digital editions" in terms of the epub being closer to finished product. Then into sigil to add additional metadata and the cover, and maybe some tweaks of the stylesheet.

Typically most of the time is spent cleaning up the .doc(x) styles before export.

BTW we mostly do fiction, with only a few non-fiction titles, so the layout is very simple. E.G. few tables, no equations, sometimes footnotes. Piece of cake.

Albert

PS I also do the revisions of the inevitable errors that only appear after publication

[DiapDealer]

While getting a feel for various users' workflows is invaluable (and in a few cases enlightening) to any rational consideration of Sigil's future (development-wise), I'd like to make a plea that it be limited to a more general workflow discussion--in relation to this thread. Detailed workflow discussions where Sigil only represents a small piece of the puzzle are probably best located in the Workshop forum.

As I said; It's been enlightening--and I'm not upset or anything, but I think we're rapidly heading toward "deserves its own thread" territory if we're not careful.

[Hitch]

Originally Posted by DiapDealer

While getting a feel for various users' workflows is invaluable (and in a few cases enlightening) to any rational consideration of Sigil's future (development-wise), I'd like to make a plea that it be limited to a more general workflow discussion--in relation to this thread. Detailed workflow discussions where Sigil only represents a small piece of the puzzle are probably best located in the Workshop forum.

As I said; It's been enlightening--and I'm not upset or anything, but I think we're rapidly heading toward "deserves its own thread" territory if we're not careful.

Dude, you're a greenie, right? I don't object if we're all moved into a new thread. I certainly didn't mean to thread-jack. :-)

Hitch

[Hitch]

Originally Posted by DiapDealer

While getting a feel for various users' workflows is invaluable (and in a few cases enlightening)

You're just making fun of my Gilligan names. That's all that comment is about.

Hitch

[DiapDealer]

Originally Posted by Hitch

You're just making fun of my Gilligan names. That's all that comment is about.

I admit it, you're not entirely wrong.

But the other revelation was that someone would actually use Sigil to compose books with. Just seems so counter-productive to me--but I don't want to judge.