Any good Perl scripters out there?

[kovidgoyal]

Since I'm on a roll here's one for HTML it might need a little adjustment

import re
from hyphenate import hyphenate_word as hyphenate

def process_text(match):
    src = match.group(1)
    return re.sub('\S+', lambda match : u'\u00ad'.join(hyphenate(match.group())), src)

src = open('file', 'rb').read()
result = re.sub(r'>([^><]+)<', process_text, src)

[Goshzilla]

okay I can defintely see the advantages of hyphenating from the html file because I have to deal with Gutenmark, a program that was written specifically under the assumption that gutenberg text is formatted in a specific way.

[DaleDe]

Originally Posted by Goshzilla

okay I can defintely see the advantages of hyphenating from the html file because I have to deal with Gutenmark, a program that was written specifically under the assumption that gutenberg text is formatted in a specific way.

The simplest way to add soft hyphens doesn't even require any programming. You could use a sed script to do a global substitution of each word with its soft hyphen equivalent. This would be a quick a dirty way to accomplish the goal particularly if you really only want to hyphenate longer words.

[Goshzilla]

I've noticed some bizar hyphenations being used in the program. For instance sometimes words like "y-our" get hyphenated, same goes for "s-mall" and "u-sual" in some instances plural forms "word-s" or "travel-s" when I use the hyphenation python script on individiual words like that it doesn't return erroneous results, but when I take an entire line, use

a=f.readline()
b=a.split()
(while loop that goes through the length of b)
a=re.sub(b[i],'^'.join(hyphenate(b[i])), a)


the line hyphenates words incorrectly. I think this has something to do with the way .sub works by finding a "pattern" in the string, so once a line has been altered, sub doesn't quit work as well because I'm looking at a new string.


update: It's the dang punctuation that messes it up, if I don't remove the punctuation marks before calling the hyphenation method, the word gets hyphenated incorrectly. I could either just alter the simple script I wrote, or alter the hyphenate code.

I could use a line like a=re.sub("[,;:.!]", '', a) to remove the punctuation marks, but when there is an apostrophe I would like to cut out the word preceeding the apostrophe mark, so something like "Washington's" becomes "Washington"

[wallcraft]

Originally Posted by NatCh

Do the em dashes in question have spaces on either side of them?

I.e. "word — word" vs. "word—word"

The reason I ask is that the absence of spaces may make the parser think it's supposed to be a single word — I think the first is the "technically" proper usage, anyway ....

Space are usually included (or they used to be) in Britain, but in the US the spaces are usually omitted. See the wikipedia article on Dash.

[NatCh]

Thanks for the primer, wallcraft — all this time I time I thought I was being non-standard, and I just learned I've been developing my own "house style!"

[kovidgoyal]

If you do modify hyphenate send me the modified code.

Originally Posted by Goshzilla

I've noticed some bizar hyphenations being used in the program. For instance sometimes words like "y-our" get hyphenated, same goes for "s-mall" and "u-sual" in some instances plural forms "word-s" or "travel-s" when I use the hyphenation python script on individiual words like that it doesn't return erroneous results, but when I take an entire line, use

a=f.readline()
b=a.split()
(while loop that goes through the length of b)
a=re.sub(b[i],'^'.join(hyphenate(b[i])), a)


the line hyphenates words incorrectly. I think this has something to do with the way .sub works by finding a "pattern" in the string, so once a line has been altered, sub doesn't quit work as well because I'm looking at a new string.


update: It's the dang punctuation that messes it up, if I don't remove the punctuation marks before calling the hyphenation method, the word gets hyphenated incorrectly. I could either just alter the simple script I wrote, or alter the hyphenate code.

I could use a line like a=re.sub("[,;:.!]", '', a) to remove the punctuation marks, but when there is an apostrophe I would like to cut out the word preceeding the apostrophe mark, so something like "Washington's" becomes "Washington"

[Goshzilla]

I never did modify the existing hyphenation algorithm. I only know basic coding, basic objects, I am still learning about writing my own search trees and hash tables, and I still haven't figured all the little nuances in Python just yet.

So here is what I wrote:
import re
from hyphenate import hyphenate_word as hyphenate
def process(y):
if (y=='\n'):
return y
b=re.sub("[,()*;:!?.]", '', y)
b=re.sub('"','',b)
b=re.sub('[\[\]{}<>]','',b)
k=b.split()
i=0
while(i<len(k)):
y=re.sub(k[i],'^'.join(hyphenate(k[i])),y)
i=i+1
return y


import re
from hyphenate import hyphenate_word as hyphenate
from process import process as process
f=open('gltrv10.txt', 'rb')
g=open('r23.txt', 'w')
a=f.readline()
while(a!=''):
g.write(process(a))
a=f.readline()

[kovidgoyal]

Cool glad you could fix your problem and keep learning python, it's a rewarding experience.

[Goshzilla]

I just thought I would revive this really old thread, in that I discovered that the Nook does support the soft hyphens. Only sort of, it breaks the word at the point where the soft hyphen is used, but it does not insert a "-" at the point where it breaks the word. Still, that's better than nothing.

Now I would like to be able to edit the contents of epub files to include hyphenations for every word found within the body of the text. I imagine this "BeautifulSoup" program will be what I need to accomplish such a task?

[kovidgoyal]

Yes beautifulsoup can be used for that.

[Goshzilla]

So I've been working on this all this morning and I have come across a stopping block.

I can extract text from anything within the <p> tags, but I have a problem trying to write a loop to hyphenate a sentence, the issue is that there are specially defined characters like "&rdquo;" which is the entry for a right sided double quotation mark. I can strip out the text from a paragraph, but it still leaves symbols like that intact.


update***

I got it done, finally. Though it isn't the most efficient code in the world, but it can now scan through an entire directory for html files, and modify them one by one, inserting hyphens. There is however one issue I'd like to get around. I am hitting a stopping block when I try to edit an html file that is encoded in something other than ascii.

[kovidgoyal]

Use the xml_to_unicode function in calibre and pass Beautifulsoup the unicode.

[Goshzilla]

Okay I've done as much as I could out of it, I've uploaded an epub of Hound of the Baskervilles to demonstrate the soft hyphen reflow through out the entire text.

update: I'm perfecting the code step by step. With the latest iteration, I can have the program crawl through a directory with subfolders each containing html files, and then modify them all subfolder by subfolder. Now I can do bulk editing quickly.