[GUI Plugin] Hyphenate This!

[jveth]

Hi Saulus, thank you very much for this plugin. I am very glad I found this.
However, the plugin unfortunately does not support non-standard hyphenations, although the recent libreoffice dictionaries do.
This results in incorrect soft-hyphenation also for me, as others already reported in posts
#101 by mattheo
#141 by karakai
#207 by imaginer

I am a bit surprised, that only hungarians report such problems. since, according to the very nice documentation in tb87nemeth.pdf , still there are much more languages involved, including (although perhaps less frequent as in hungarian), also english and german.

Anyway, for this plugin, there does not seem any progress for non-standard hyphenation since #101 (may 2003).
In order to try to help you (and us hungarians as well ), I decided to dig into this a bit more.


1) While the pyhyphen-2.0.5 module works as expected, still the hyphenator-0.5.1 module cannot handle the non-standard cases.

Although I do not know python, still I succeded to construct the following simple test script:

from hyphen import Hyphenator as Hypy2				# pyhyphen-2.0.5
h2hu = Hypy2('hu_HU')
def hypins2hu(p): return '-'.join(h2hu.syllables(p))

from zzhyphenator import Hyphenator as Hyp0			# hyphenator-0.5.1 (Berendsen,2008)
h0hu = Hyp0('/usr/share/hyphen/hyph_hu_HU.dic')
hypins0hu = h0hu.inserted

def printithu(zz, txt):
    print '2.0.5: ' , hypins2hu(zz) , ' --- ' , '"' + zz + '", as reported by' , txt
    print '0.5.1: ' , hypins0hu(zz)
    return

printithu(u'valamennyit',	'mattheo #101')
printithu(u'valamennyi',	'mattheo #101')
printithu(u'rosszabb',		'mattheo #101')

printithu(u'poggyászomban',	'imaginer #207')

Without adding my own findings, only running the test for the reported cases:

2.0.5: va-la-meny-nyit --- "valamennyit", as reported by mattheo #101
0.5.1: va-la-meny-nyt
2.0.5: va-la-meny-nyi --- "valamennyi", as reported by mattheo #101
0.5.1: va-la-meny-ny
2.0.5: rosz-szabb --- "rosszabb", as reported by mattheo #101
0.5.1: rosz-zabb
2.0.5: pogy-gyá-szom-ban --- "poggyászomban", as reported by imaginer #207
0.5.1: pogy-yá-szom-ban

clearly show, that the soft-hyphened words in 0.5.1 obviously are corrupted. They differ from both the expected valid hyphenated and also the non-hyphenated version. Also, they are no more identical to the original, when removing soft hyphens that mostly happens when reflowing the text, no actual hyphen needed at this point.

2) Unfortunately, even with correct hyphenator, the non-standard break-points cannot be used for SHY. I hope this is obvious from the above examples.

Still insisting on my not knowing python , I changed hjob.py in the plugin as follows:

                for w in wlist:
                    if len(w) >= min_len and u'-' not in w:
                        ww = w
                        for hh in h.iterate(ww):
                            #print 'Hyphenator hint: ' , hh[0] , '-' , hh[1]<---># trace
                            if hh[0] + hh[1] == ww:
                                #w = hh[0] + '-' + w[len(hh[0]):]<------><------># *** TEST *** see all possibilities
                                w = hh[0] + u'\u00AD' + w[len(hh[0]):]
                    newt += w

this now seems to do a much better job .

3) My first thought was, of course, to use the actual pyhyphen module. However, when trying to include the standard installed pyhyphen module in my system, I was compelled to discover that, without sufficiently knowing python, still it is beyond my scopes.

So the next thing for me was to check, whith ignoring the non-standard cases, whether the old hyphenator works well with current dictionaries.
Conclusion: Using the 100.000 word large sample from the Hungarian gigaword corpus referred to by tb87nemeth.pdf, I found that the old module still seems OK at the moment. That is, all standard hyphenation points are identically observed by both 0.5.1 and 2.0.5.

[SauliusP.]

Originally Posted by jveth

I am a bit surprised, that only hungarians report such problems.

Hi there,

Not only Hungarians, probably, have this problem, others might just ignore them. I have encountered something similar myself in Lithuanian books, however, I simply to not care.

As for Hungarian, I do not know much about it, but I like Aurora's "Kurvak, gengszterek". Maybe I'll try to incorporte pyhyphen just as a tribute to them :-)

Give me few more weeks, I am quite busy with real life currently.

[098799]

Hi, I tried to get this plugin to work, but get the following error

calibre, version 2.5.0
BŁĄD: Nieznany wyjątek: <b>LookupError</b>:unknown encoding: .ab4i

calibre 2.5  isfrozen: False is64bit: True
Linux-3.16.0-28-generic-x86_64-with-Ubuntu-14.10-utopic Linux ('64bit', 'ELF')
('Linux', '3.16.0-28-generic', '#38-Ubuntu SMP Fri Dec 12 17:37:40 UTC 2014')
Python 2.7.8
Linux: ('Ubuntu', '14.10', 'utopic')
Successfully initialized third party plugins: Hyphenate This!
Traceback (most recent call last):
  File "calibre_plugins.hyphenatethis.hyphenatethisaction", line 118, in hyphenate
  File "calibre_plugins.hyphenatethis.hyphenatethisaction", line 92, in _select_books
  File "calibre_plugins.hyphenatethis.hyphenator.hyphenator", line 165, in __init__
  File "calibre_plugins.hyphenatethis.hyphenator.hyphenator", line 89, in __init__
LookupError: unknown encoding: .ab4i

Any idea?

[SauliusP.]

Hi,

If you could upload your book and used dictionary somewhere and share link via PM, that would be a big help.

[098799]

I made it work by downloading yet another dictionary from http://www.textcontrol.com/en_US/dow.../dictionaries/

Sorry to have bothered you! The plugin looks great. Hyphenation is probably the one thing I really wished I had in my Kindle.

[Ruskie_it]

Amen to that...

[EbokJunkie]

SauliusP.
May we hope you'll find time to convert plugin to command line utility?
That would be great!

[SauliusP.]

I've never considered that, nor did I investigate abilities of CLI Calibre. I definitely would not develop the plugin as an independent standalone application, as it heavily depends on Calibre's features.

And I have no use case for it personally. Sorry :|

[eschwartz]

Perhaps you could take a look at how Modify EPUB does it.

Note: It uses calibre-debug to access the plugin through calibre APIs.
Note2: It is the only command-line plugin, probably for reason -- it is essentially shtick. Do it if it makes you feel happy...

[Aellea]

I've just updated my Calibre to 2.20 (64bit) together with this plugin. I used Hyphenate This! successfuly in the past (older versions).

After this update it is no longer working. Confirmation dialog (format selection) opens but nothing happens afterwards. There are no obvious errors either.

system: WIN 7 (64bit)
format: AZW3
lang: Polish (using hyph_pl_PL.dic)
log: Hyphenator using self LR: 2 2

Why? Do I need an older/other Calibre version or something?

[JSWolf]

I just tested the Hyphenate This! plugin with Calibre 64-bit 2.20 under Windows 8.1 and it worked no problem. I did have the same problem you did when I mistakenly did not select the format I wanted to hyphenate. You have to click on the format you want even if it's the only format you have.

[SauliusP.]

Originally Posted by Aellea

Why? Do I need an older/other Calibre version or something?

No, as indicated by another comment, you just need to explicitly click on the format you want to hyphenate. I myself have this problem from the first version of Calibre 2. Calibre 1.x somehow preserved selected format in the dialogue box. It's a bit annoying, as "half-selected" format might be misleading: one clicks "OK" and nothing happens.

[nyrk]

Saulius P.:

I just want to add my THANKS for this great plugin that turns reading on my kindle into an uninterrupted process and thus into a lot more fun!

I certainly don't take such efforts for granted and I can only hope that you will continue to maintain/update this plugin, at least as long as devices don't offer hyphenation options in first place.

I will indeed express my feelings with Paypal (from within the plugin settings) !

It may not be a huge amount, but as always with good freeware, I can imagine that everyone who likes a free product may show his appreciation by contributing as much as he's willing to spare so that many small contributions add up to something more substantial!
Imo, good work should be appreciated by those who benefit from it - also but not solely through words.

Thank you - please keep up the good work!

[SauliusP.]

Originally Posted by nyrk

I certainly don't take such efforts for granted and I can only hope that you will continue to maintain/update this plugin, at least as long as devices don't offer hyphenation options in first place.

Thank you for your warm expression, that means a lot for me!

[dragonkore]

This is amazing, thank you. It really does make a big difference in terms of formatting an ebook to look more like a printed one.