Feature Request: Make soft hyphen unicode character visible

[Morea]

Hi,

instead of generally using the Hyphenate this plugin I like to manually add soft hyphens to selected (long) words.

My problem: after doing so, I cannot see the added soft hyphen in the Calibre Editor. They are invisible.

Could please Calibre show a character for that soft hyphen in the hyphenated word, a greyed-out "-" or something like that?

This would make editing much easier in my opinion.

[kovidgoyal]

Just use a unicode symbol for them that isnt used in the text such as a special dash – or a centered dot · . Then you can use search and replace to easily switch them visible or invisible.

[Morea]

But that would be a little bit too handmade solution for my taste. It could need to do several times search and replace along different html files. And when I forget the last one, I have funny characters in my ebook.

Why can't Calibre use a visible character for shy on its own like Libreoffice Writer does?

[kovidgoyal]

You can search and replace all html files with a single click.

[Morea]

Yes I know. But what entity code do I have to search for?

And who gets the idea to search for invisible characters when he wants to find out the reason for a behaviour of his book file?

That's not very intuitive, in my opinion.

[kovidgoyal]

Soft hyphens are not a very intuitive thing in the first place. In fact, they are a horrible kludge, which I would encourage you not to inflict on your books. They break all sorts of things, like searching.

And if you want to search for a soft hyphen use regex mode, with the find string as

\xad

[Morea]

They break all sorts of things, like searching.

Why?

Perhaps you can give up such a feature in English, but not in German with its long and composited words... you will never find all German word combinations in a dictionary.

[kovidgoyal]

Because when you search for abcdef it wont match abcdef in the book because in the book it will contain soft hyphens inside abcdef. Now some software goes the extra mile to try to work around that, but it is a huge pain. And there is no reason german or any other, language cannot be hyphenated dynamically.

[Morea]

Nice to see that Calibre Viewer seems to goes that extra mile.
But my book reader does not.

So the solution for my bookreader is a bigger german dict file.

Thank you for your explanations!

[JSWolf]

Would you not be better finding a different program to read your eBook that supports hyphenation?

[Tex2002ans]

Originally Posted by Morea

Why?

Perhaps you can give up such a feature in English, but not in German with its long and composited words... you will never find all German word combinations in a dictionary.

Soft Hyphens are very bad.

I wrote about some of the reasons why in 2018: "Auto-Hyphenate" (many of those linked posts going back to 2013 + 2008!).

Across various devices/programs, they break:

• Search
• Spellchecking
• Dictionaries
  
  • Where you press+hold to get a definition.
• Highlighting
• Copy/Pasting
• Auto-Hyphenation
  
  • Especially in non-English languages.
    
  • Some languages have hyphenation rules where:
    
    • different/duplicate letters appear
      
    • accents drop off
      
    • hyphens needed at the end AND beginning of lines
      
• ...

Like JSWolf said, much better to use a device/app that has built-in hyphenation for your language.

Almost all modern devices/apps have this now... for example, Kobos/Kindles already have built-in German hyphenation.

If you are on a very old device, perhaps there is a case to apply HyphenateThis! to your personal copy... but definitely never use it in an ebook meant for sale.

Note: Hyphenation Dictionaries work by patterns.

They list combinations of letters where hyphens can occur, then apply that across the entire text.

It's not like they list hundreds of thousands of every word known to man:

- hyphenate
- hyphenated
- hyphenates
- hyphenation
- hyphenations
- hyphenating

Instead, these hyphenation dictionaries list hundreds of patterns/rules like:

- "If a word ends in -ing OR -tion, you can stick a hyphen there."
- "If a word begins with anti- or semi-, you can stick a hyphen there."

Every language is going to have different patterns/rules, and people have already created these dictionaries for many of the main languages... even smaller ones like Welsh.

So even if you came up with some super cool new English word like:

- superduperliciousness

the device will auto-hyphenate correctly:

- su-per-duper-li-cious-ness

or let me toss it in my:

- hyphenatinginator
- hy-phen-ating-i-na-tor

Spoiler:

The dictionaries got those 2 made-up words almost perfect.

Could've had one between "du-per" + "at-ing".

But 99.9999+% of all auto-hyphens will be correct. There are only a few hundred words in English where the patterns got it wrong, and those exceptions are known and accounted for.

Originally Posted by Morea

Yes I know. But what entity code do I have to search for?

And who gets the idea to search for invisible characters when he wants to find out the reason for a behaviour of his book file?

That's not very intuitive, in my opinion.

Usually you notice invisible characters when things start going very wrong/strange:

• Random question marks ? or � appearing in the middle of your EPUB
• Red squigglies under perfectly spelled words
• Random hyphens appearing/disappearing as you type.
• ...

To get a list of ALL characters used in the EPUB:

In Calibre's Editor:

Tools > Reports > Characters

and in Sigil:

Tools > Reports > Characters in HTML Files

This will list "invisible characters" like Soft Hyphens, Thin Spaces, Zero-Width Joiners, etc.

You can double-click on a character in the Report, and it will fill in the correct codes in the Find box for you.

If I spot soft hyphens in the ebook, I make sure to always remove them.

[Quoth]

I remove them. They are for web-pages, not needed on ebooks.

[JSWolf]

Originally Posted by Quoth

I remove them. They are for web-pages, not needed on ebooks.

I am going to disagree. The one place soft-hyphens work and work very well is on a Kindle when using KF8. I use the Hyphenate This! plugin and it the Kindle handles soft-hyphens rather well.

[Quoth]

Almost no commercial or PD texts have soft hyphens of about 6,500 ebooks I've checked. I removed them from one.

I do read on the Sage mostly, but I read for years on PW1, then DXG, then PW3 and still check ebooks on mobi and azw3 on the Kindle Keyboard gen3 (maybe 1st decent 800 x 600 kindle when they retro-added azw3). The DXG was a bad buy, but I'd passed on the PW1
Then I got an original H20. I really can't see the value of adding soft hyphens.
I don't see a problem on the Sony PRS350, T2, tiny Y-Ben 4.7″, or various Android apps.

[phossler]

Originally Posted by JSWolf

I am going to disagree. The one place soft-hyphens work and work very well is on a Kindle when using KF8. I use the Hyphenate This! plugin and it the Kindle handles soft-hyphens rather well.

Roger that.

Hyphenate This works well and has language-specific dictionaries that can be loaded

I use it's option to remove soft hyphens until I'm ready to side load the book and then I add them back. Only takes a minute