Space character before Soft Hyphen missed in epub>docx conversion

[quinta@ebf.cz]

Hello.

I ran into a probably quite rare conversion problem: A Soft Hyphen, when inserted before a word (ie between a space character and the first character of a word), can cause the space character not to be converted (epub > docx).

Naturally, I am aware that such placement is very strange for Soft Hyphen. I don't know how such Soft Hyphens could get into epub. In any case, I have seen several documents exported in this way, and it is very difficult to fix them without access to the source epub file…



sample epub+docx attached
Calibre version: 6.26

[JSWolf]

The soft hyphens should be removed as they don't work well in ePub. Most software won't display them properly and searching will not work.

[quinta@ebf.cz]

OK, I'm also not a fan of using Soft Hyphens in epub. But that doesn't change the fact that Calibre's conversion behavior appears to be flawed. And that this error is very problematic, especially if you have no control over the epub and only have to work with the conversion output.

[Quoth]

Soft hyphens are for websites. Ebooks should leave it to the renderer to hyphenate.
There should never be a space on any side of a soft hyphen as the only reason for them is a hint as to where to break in a word. Unexpected never-to-be encountered formatting can break conversions.

Someone has used a crazy tool or formatting of the epub. Fix the epub by deleting all soft hyphens before conversion to docx using the Calibre editor, which should be easy.

Auto-hyphens also need to be off on WP source for creating an ebook and on for PDF, because the Wordprocessor has no knowledge of page width of an ebook, but does have the page size set for a PDF.

[quinta@ebf.cz]

Quoth: Thank you, I agree, but once again: Source epub file is not always available. In such case you have to work with conversion output. And this is quite difficult without space characters.

[DNSB]

Since AFAIR, a soft hyphen is an invisible format character indicating a possible hyphenation location, placing a soft hyphen between a space character and a word is pretty much a garbage in, garbage out situation.

Since you appear to have the epub to be able to convert it, why not simply remove the soft hyphens and reconvert? To me, this would be the more sensible approach and allows those who use soft hyphens properly to continue using them.

If you are only given the converted output, time to punt the garbage back to the originator and tell them to fix it.

[theducks]

The EPUB is the source or you could not convert (DRM infested).

Try using the built in 'Polish' tool (you may need to add it to a toolbar):remove soft hyphens.

If that does not get them all,
You will need to use the editor (and some REGEX)

[quinta@ebf.cz]

Originally Posted by DNSB

Since you appear to have the epub to be able to convert it, why not simply remove the soft hyphens and reconvert?

Attached epub is just demo file for testing.

Of course, problem can _easily_ be solved in epub.

But conversion should _never_ remove spaces, and source epub file is not _always_ available. Sorry, but suggestions to solve it by editing epub are not very useful/relevant.

[Quoth]

Originally Posted by quinta@ebf.cz

But conversion should _never_ remove spaces. Sorry, but suggestions to solve it by editing epub are not very relevant.

Did you miss where I explained that conversions break when the source has formatting that should not exist?

The source is broken. There should never ever be a space with a soft hyphen. A soft hypen should only ever be inside a word, and shouldn't be in an epub anyway.

No regex needed. Simply replace every soft hyphen with nothing. It's not like non-breaking spaces, which are needed in ebooks, like between a number and a street or a number and a type of unit etc.

[DNSB]

Originally Posted by quinta@ebf.cz

Attached epub is just demo file for testing.

Of course, problem can _easily_ be solved in epub.

But conversion should _never_ remove spaces, and source epub file is not _always_ available. Sorry, but suggestions to solve it by editing epub are not very useful/relevant.

So what you are wanting is to have a file with a structural error since a soft hyphen should never occur between a space and a word fixed. If you do not have the original epub to fix it yourself, get the person who generated the flawed ePub to fix their garbage output. Otherwise, GIGO since a soft hyphen should never occur outside of a word, the space in front of the soft hyphen is an obvious error.

Out of a perhaps morbid curiosity, was this a commercially available, public domain or freely distributable epub? If so, you might want to complain to the source.

[JSWolf]

The only place I know for sure that soft hyphens work and do not break searching is on a Kindle with KF8 format eBooks.

[JSWolf]

Originally Posted by quinta@ebf.cz

Attached epub is just demo file for testing.

Of course, problem can _easily_ be solved in epub.

But conversion should _never_ remove spaces, and source epub file is not _always_ available. Sorry, but suggestions to solve it by editing epub are not very useful/relevant.

The source ePub has to be available. Otherwise, where did the DocX come from?

I know two ways soft hyphens can get into an ePub. One is with calibre's polish and the other is with the Hyphenate This! plugin. But both do not put soft hyphens outside of words as in your example.

If you really do not have the source ePub, remove all of the soft hyphens and do a spell check.

[quinta@ebf.cz]

Ok. I know how to remove SH from epub. This is quite easy, no need of more explanations about that.

EDIT:
Key informations for such sort of task:
- SH can be searched/replaced by regular expression \xad (or \u00ad) (because SH is character U+0173, etc.)
- be awared: soft hyphen character itself is not visible in Calibre editor
- removing soft hyphens action is also part of Calibre "Polish ebook" tool

Originally Posted by Quoth

I explained that conversions break when the source has formatting that should not exist?

In other words, you are saying that dropping of space during the "space+SH" export is intentional. I don't think so.

BTW, interesting fact: Seems not every space from "space+SH" combos is dropped. Don't know what does it mean. Just interesting. : )

[quinta@ebf.cz]

Originally Posted by JSWolf

If you really do not have the source ePub, remove all of the soft hyphens and do a spell check.

Thanks, this is a suitable emergency procedure.
Even though it would be much better not to have to use it. So I actually hope the conversion will be fixed, sooner or later.

[Quoth]

Originally Posted by quinta@ebf.cz

In other words, you are saying that dropping of space during the "space+SH" export is intentional. I don't think so.

No, I'm not writing that. If the source for any kind of conversion is broken in an unexpected way, then a conversion may do something unexpected. Maybe it's a bug, but it's a bug that wouldn't show up in years of testing because you do not get a space beside a soft hyphen ever.

Your input file has a serious mistake. Fix broken input. All computer programs are famous for Garbage In gives Garbage Out.

I'm sure there are other stupid things that should never ever be in an epub that will break conversion and Amazon's conversions break more easily than Calibre. But Amazon produces perfect mobo, azw3 and KFX from epub uploads to KDP that are corrent. This is a broken epub. It's a format error I've never seen in ten years.