Hyphenation not according Dutch rules

[GeoffR]

Originally Posted by outlandos

So the problem won't be fixed when 3.16 arrives?

It should be fixed, they are talking about Koreader, Coolreader, or FBReader which are third-party readers that have nothing to do with the readers that come with the Kobo firmware. (Edit: The Kobo firmware has included Dutch hyphenation dictionaries since version 2.8.1b)

Have a look at the second screenshot in this post. The KEPUB hyphenation problem demonstrated in the In the second paragraph of the screenshot sounds like the problem you describe. It has been fixed in firmware 3.16.0.

[Markismus]

The pull request is merged, so koreader now has hyphenation support for Dutch.

You could try to enable hyphenation support for Dutch in Kobo Nickel by editing the metadata of your ebook with Calibre:
Rightclick on your ebook in the Calibre Library>edit Metadata>Languages>Dutch

The first book I checked had no language defined. So how would Kobo know it should Dutch hyphenation?

[outlandos]

Originally Posted by Markismus

The pull request is merged, so koreader now has hyphenation support for Dutch.

You could try to enable hyphenation support for Dutch in Kobo Nickel by editing the metadata of your ebook with Calibre:
Rightclick on your ebook in the Calibre Library>edit Metadata>Languages>Dutch

The first book I checked had no language defined. So how would Kobo know it should Dutch hyphenation?

This fixed my problem, obviously kobo is no google and thus could not detect the language of the book, manually setting it on Dutch did help. Thanks all!

[mbovenka]

Originally Posted by Ken Maltby

You are probably correct, I should have used Swedish or Norwegian.

Yup, Finnish is a Finno-Ugric language (related to Hungarian and Estonian).

[outlandos]

Originally Posted by davidfor

Firmware version 3.16.0 has fixes for hyphenation. The changes were for all languages. Hopefully it fixes the problems with Dutch.

As 3.16.0 was not released for the H2O, you have a choice of manually installing if from one of the other firmware packages, or waiting for the next release. Kobo don't preannounce any releases, so we don't now when it will be.

Originally Posted by GeoffR

It should be fixed, they are talking about Koreader, Coolreader, or FBReader which are third-party readers that have nothing to do with the readers that come with the Kobo firmware. (Edit: The Kobo firmware has included Dutch hyphenation dictionaries since version 2.8.1b)

Have a look at the second screenshot in this post. The KEPUB hyphenation problem demonstrated in the In the second paragraph of the screenshot sounds like the problem you describe. It has been fixed in firmware 3.16.0.

Unfortunately, 3.16.0 did not fix the Dutch hyphenation completely.

As soon as punctuation is involved, it goes wrong. To demonstrate this, I have made 2 screenshots, both from the same e-book (which is a kepub file, set on Dutch):

pic 1: http://static.tweakers.net/ext/f/KRl...PUTlx/full.jpg

pic 2: http://static.tweakers.net/ext/f/Acz...6PGRi/full.jpg

Also, for some reason interview gets cut off wrong too, eventough it is an official Dutch word (http://www.vandale.nl/opzoeken?patte....VYGRNfntlBc):

pic 3: http://static.tweakers.net/ext/f/Jn6...L3csk/full.jpg

Anyway, isn't 3.16.0 supposed to fix hyphenation when punctuation is involved?

[GeoffR]

Originally Posted by outlandos

Unfortunately, 3.16.0 did not fix the Dutch hyphenation completely.

As soon as punctuation is involved, it goes wrong. To demonstrate this, I have made 2 screenshots, both from the same e-book (which is a kepub file, set on Dutch):

pic 1: http://static.tweakers.net/ext/f/KRl...PUTlx/full.jpg

pic 2: http://static.tweakers.net/ext/f/Acz...6PGRi/full.jpg

I had a look in the Dutch hyphenation dictionary, and it appears that wer-p and kam-p are listed as valid hyphenations, so if they are wrong then the problem might be with the dictionary. If you have a better dictionary you can install it as a replacement.

[outlandos]

Originally Posted by GeoffR

I had a look in the Dutch hyphenation dictionary, and it appears that wer-p and kam-p are listed as valid hyphenations, so if they are wrong then the problem might be with the dictionary. If you have a better dictionary you can install it as a replacement.

They are very wrong.

But how do I change it? And should I let someone at kobo on this forum know that it's Dutch hyphenation dictionary is invalid?

[GeoffR]

Originally Posted by outlandos

They are very wrong.

But how do I change it? And should I let someone at kobo on this forum know that it's Dutch hyphenation dictionary is invalid?

I might be misreading the dictionary, I am not sure about the format, perhaps kam-p and wer-p are only valid if they are in the middle of a word and not at the end.

I've been reading kepub books for the last few days and haven't seen any problems like this in English though.

However there are still some problems with KEPUB hyphenation in 3.16.0, one being that it doesn't respect the LEFTHYPHENMIN and RIGHTHYPHENMIN settings like the EPUB reader does, so maybe there are other problems remaining too.

You can replace the Dutch hyphenation dictionary by creating a TGZ file namd KoboRoot.tgz containing the dictionary named ./usr/local/Kobo/hyphenDicts/hyph_nl.dic (e.g. on Linux: tar czf KoboRoot.tgz ./usr/local/Kobo/hyphenDicts/hyph_nl.dic) and placing KoboRoot.tgz in the .kobo directory of the device and safetly eject.

[GeoffR]

Experimenting a bit more, it seems that the KEPUB reader does in fact read the LEFTHYPHENMIN and RIGHTHYPHENMIN values from the hyphenation dictionary, but for some reason it is miscalculating their value. It seems to be reading LEFTHYPHENMIN 3 as if it were LEFTHYPHENMIN 2.

Also the Dutch hyphenation dictionary is missing the LEFTHYPHENMIN and RIGHTHYPHENMIN values altogether, which could be causing problems.

If you or someone else wanted to experiment you could try extracting the Dutch hyphenation dictionary from the 3.16.0 firmware and editing it (with a unix-aware text editor such as Notepad++ if using Windows) so that it starts like this (or whatever values you prefer, increased by one):

UTF-8
LEFTHYPHENMIN 3
RIGHTHYPHENMIN 3

and then installing it as described in my previous post. It would be interesting to see if that fixed (or at least improved) the problem you are experiencing.

[outlandos]

Originally Posted by GeoffR

Experimenting a bit more, it seems that the KEPUB reader does in fact read the LEFTHYPHENMIN and RIGHTHYPHENMIN values from the hyphenation dictionary, but for some reason it is miscalculating their value. It seems to be reading LEFTHYPHENMIN 3 as if it were LEFTHYPHENMIN 2.

Also the Dutch hyphenation dictionary is missing the LEFTHYPHENMIN and RIGHTHYPHENMIN values altogether, which could be causing problems.

If you or someone else wanted to experiment you could try extracting the Dutch hyphenation dictionary from the 3.16.0 firmware and editing it (with a unix-aware text editor such as Notepad++ if using Windows) so that it starts like this (or whatever values you prefer, increased by one):

UTF-8
LEFTHYPHENMIN 4
RIGHTHYPHENMIN 4

and then installing it as described in my previous post. It would be interesting to see if that fixed (or at least improved) the problem you are experiencing.

Just for giving the interested a heads-up:

GeoffR was so nice to send me an edited 'hyph_nl' with the above code modification. The result: 'onderwerp' (which means 'subject' in Dutch) still was spelled onderwer-p. The other two words ('trainingskamp' and 'interview') did end up in the middle of a line, most likely because at default the value =/ 4, so that doesn't tell us much.

So unfortunately, the modification did not solve the problem.

[GeoffR]

I found that the Dutch hyphenation dictionary included in the Kobo firmware is just a copy of the openoffice one available here.

There is a hyphenation program available here that can be used to test the hyphenation of words using the hyphenation dictionary.

Using the above program with the Dutch hyphenation dictionary (with no HYPHENMIN values, exactly as it comes with the Kobo firmware) gives the following hyphenations:

onderwerp on=der=werp
onderwerp, on=der=wer=p,

So the hyphenation onderwer-p, is valid according to that dictionary.

But adding LEFTHYPHENMIN 3 and RIGHTHYPHENMIN 3 to the dictionary gives the hyphenations:

onderwerp onder=werp
onderwerp, onder=werp,

so I hoped that the HYPHENMIN additions would fix the problem, but there seems to be more to it.


Edit: There are rules such as kam1p and wer1p in the Dutch hyphenation dictionary which seem to be responsible for these strange hyphenations, removing them fixes these paricular cases but I don't know Dutch so I don't know what other problems removing them would cause.

But it does seem that stripping the trailing punctuation from the words before hyphenating them would solve all these problems. The dictionaries have some rules to handle punctuation, but they only seem to be for apostrophes and such that are a part of the word, not other leading and trailing punctuation. It doesn't make sense to me that trailing punctuation should affect the result of the hyphenation algorithm.

[GeoffR]

Some more testing (I've made a Dutch ebook that contains nothing but "onderwerp" followed by various punctuation) and have found some interesting things:

1. The Dutch hyphenation dictionary supplied with the Kobo firmware is incorrectly encoded: It is marked as UTF-8 but it is actually encoded as ISO8859-1. However converting the file to UTF-8 didn't fix the onderwerp problem. It could be that the hyphenation dictionary needs to be rebuilt from scratch as a UTF-8 dictionary?

2. The bad hephenation of onderwerp occurs in the EPUB reader as well as the KEPUB reader. I've never seen this type of problem in the EPUB reader with the English hyphenation dictionary in over two years of reading epubs, so I suspect the real problem is the Dutch hyphenation dictionary.

Edit: I find can add rules such as 4p, to the dictionary to prevent the bad hyphenation in the EPUB reader, but it doesn't work for the KEPUB reader. So as well a problem with the hyphenation dictionary, there are still problems with the KEPUB hyphenation algorithm too.

[cramoisi]

@outlandos: what about trying this method I posted yesterday on the Firmware 3.16.0 thread ?

First, since the hyphen algo is supposed corrected in 3.16.0, (and if I were you), I should try to update the hyphen dictionary inside the kobo by adding the one of openoffice (link and method in the post linked). If it continues to not working, I should use the other method and not using the hyphen algo of kobo at all but use calibre + kobo_extra.css to hyphenate correctly the books.

Do you use calibre to send books into your gloHD ? If it’s the case, with KoboTouch plugin or KoboTouchExtended ? Hence, do you read epubs or kepubs ?

[JSWolf]

Why not just fix the Dutch hyphenation dictionary every time an incorrect hyphen is found?

[cramoisi]

Originally Posted by JSWolf

Why not just fix the Dutch hyphenation dictionary every time an incorrect hyphen is found?

Updating the whole dictionary with a correct one (at libreoffice or openoffice) seems easier to me The idea is to read books, not to kill the pleasure by debugging a system while reading...