Plugin for tidying ePub files

[Doitsu]

@CalibUser: I've just tested the updated plugin with my Linux machine and appears to be working fine. (I only tested the line break fix.)

[gipsy]

The Fix for false line breaks doesn't work in greek language.

I use the following regex to fix the lines breaks.

Find: ([\p{Greek},'–’“”][</ib>]*)</p>\s+<p>([<ib>]*[\p{Greek},'–’“”])
Replace:\1 \2

I try to change the

	if allBreaks == 'Yes':
		CorrectText("Fixed false line breaks:", r'([a-z])</p>\s+<p[^>]*>([A-Z])', r'\1 \2')

with the in HTMLProcessor.py

	if allBreaks == 'Yes':
		CorrectText("Fixed false line breaks:", r'([\p{Greek}\,\'–’“”][</ib>]*)</p>\s+<p>([<ib>]*[\p{Greek},\'–’“”])', r'\1 \2')

but the lines doesn't combine
I don't know any python. Is my code ok?
Thanks

[CalibUser]

@Doitsu: Thanks for testing the plugin

@gipsy: In your code:

r'([\p{Greek}\,\'–’“”][</ib>]*)</p>\s+<p>([<ib>]*[\p{Greek},\'–’“”])'

you have escaped the comma with a slash - there is no need to do this as the r in front of the code (stands for raw) means that you do not need to escape characters; however, you do need to escape the single quote mark otherwise this would signify the end of the expression. I presume that {Greek} represents Greek characters?

[gipsy]

You are right. But again they don't compine
Yes is for greek characters.

  <p>ο Πυθέας ήπιε το υπόλοιπο</p>

  <p>γάλα από το κύπελλο, σκούπισε</p>

  <p>δυο σταγόνες στα χείλη του με την ανάστροφη του</p>

  <p>χεριού του και σηκώθηκε.</p>

With the regex it's change to

  <p>ο Πυθέας ήπιε το υπόλοιπο γάλα από το κύπελλο, σκούπισε δυο σταγόνες στα χείλη του με την ανάστροφη του χεριού του και σηκώθηκε.</p>

but it doesn't with the changes in py

[Doitsu]

Originally Posted by gipsy

The Fix for false line breaks doesn't work in greek language.

Sigil and Python use different Regex engines. Sigil uses PCRE and Python uses an older, less powerful version.
AFAIK, Python doesn't support the \p{Greek} syntax. I.e., Greek letters need to be explicitly expressed as Unicode ranges (0370–03FF).

[eschwartz]

Originally Posted by CalibUser

I believe the problem on Linux is the path specified in the plugin for the dictionary (I don't have Linux so I can't confirm this). I have updated the plugin in the first post in this thread so that when the plugin is run for the first time, it asks for the location and filename of the dictionary (see the epub in the first post for details) that is used for correcting hyphenated words that should not be hyphenated. Hopefully this will resolve the problem in Linux so that it will not run and run, nor require the Cancel button to be pressed to exit.

Yep:

       userProfile = (os.environ['USERPROFILE'])       #Get path to user profile

       try:
               f = open(userProfile+'\\AppData\\Local\sigil-ebook\\sigil\\user_dictionaries\\WordDictionary.txt', 'r', encoding='utf-8')

There was no way that would ever work on anything other than Windows. It is highly os-specific.
For that matter, it is highly environment-specific -- it would also break hard on a PortableApps.com install, for example.


Is there any way in Sigil/the plugin container to access the value of the Sigil configuration folder? This would be a far, far better way of handling it. (If there isn't a way, then it would be a generally useful thing to have...)
Asking the user to manually select the dictionary just to get around the issue of finding the configuration directory is overkill (and slightly onerous) -- although it could be useful if one has multiple dictionaries and wants to use a specific one, that is probably an edge case. EDIT: And of course the instructions already make it clear that that won't work.

[KevinH]

FWIW,
The next release of Sigil will include an interface to the hunspell spellchecker and will provide a list of paths to the hunspell dictionaries.

If I can figure out how best to bundle sigil's version of gumbo for use by plugins, and if DiapDealer and I can fix some bugs, we should have a release out in 2 or 3 weeks.

Kevin

[Doitsu]

Originally Posted by eschwartz

There was no way that would ever work on anything other than Windows. It is highly os-specific.
For that matter, it is highly environment-specific -- it would also break hard on a PortableApps.com install, for example.

This is certainly a valid point, but since CalibUser doesn't have a Mac or a Linux machine, I appreciate it that he at least made an effort to make his plugin OSX/Linux compatible, even though he doesn't have a way of testing it.

@CalibUser: Python has a boatload of built-in functions for cross-platform file handling that make it really easy to implement cross-platform file support.

Since the Sigil plugin root directory and the user_dictionary directory are sibling directories it's relatively easy to get the user_dictionary directory location.

For example, you could use the following code to get the dictionary folder:

import os, inspect
def run(bk):
    # get plugin directory path
    plugin_path = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    print(plugin_path)
    
    # get rid of the last two directories
    tmp_path = plugin_path.split(os.path.sep)[:7]
    print(tmp_path)
    
    # add the dictionary path
    tmp_path.extend(['user_dictionaries', 'WordDictionary.txt'])
    
    # convert list back to file path
    dictionary_path = os.path.sep.join(tmp_path)
    print(dictionary_path)

The above code will produce the following output:

Windows:

C:\Users\Doitsu\AppData\Local\sigil-ebook\sigil\plugins\test
['C:', 'Users', 'Doitsu', 'AppData', 'Local', 'sigil-ebook', 'sigil']
C:\Users\Doitsu\AppData\Local\sigil-ebook\sigil\user_dictionaries\WordDictionary.txt

Linux (DiapDealer's build):

/home/doitsu/.local/share/sigil-ebook/sigil/plugins/test
['', 'home', 'doitsu', '.local', 'share', 'sigil-ebook', 'sigil']
/home/doitsu/.local/share/sigil-ebook/sigil/user_dictionaries/WordDictionary.txt

I've attached a test plugin that you can play with.

[gipsy]

Originally Posted by Doitsu

Sigil and Python use different Regex engines. Sigil uses PCRE and Python uses an older, less powerful version.
AFAIK, Python doesn't support the \p{Greek} syntax. I.e., Greek letters need to be explicitly expressed as Unicode ranges (0370–03FF).

Thanks Doitsu.

I change it to

	if allBreaks == 'Yes':
		CorrectText("Fixed false line breaks:", r'(([\x{0370}-\x{03FF}\x{1F00}-\x{1FFF},\'–’“”][</ib>]*)</p>\s+<p>([<ib>]*[\x{0370}-\x{03FF}\x{1F00}-\x{1FFF},\'–’“”]))', r'\1 \2')

from a regex sample someone had priveded me here to get rid of hyphen and IT WORKED!

Originally Posted by eschwartz

For that matter, it is highly environment-specific -- it would also break hard on a PortableApps.com install, for example.

I run a portable installation. When you run the Sigil. It copies the sigil setting (hunspell, user dictionaries, plugins etc to appdata\local\sigil-ebook and when you exit from sigil they copied back to portable location.

[gipsy]

@CalibUser: Those are some fixes in greek language if you want to place them in your plygin. I try to find a solution and for some other things and i keep you posted

#Greek line break fix
	if allBreaks == 'Yes':
		CorrectText("Fixed false line breaks:", r'(([\x{0370}-\x{03FF}\x{1F00}-\x{1FFF},\'–’“”][</ib>]*)</p>\s+<p>([<ib>]*[\x{0370}-\x{03FF}\x{1F00}-\x{1FFF},\'–’“”]))', r'\1 \2')
	return(0)

	#Fixes Έ when PDFd as 'Ε or "Ε
	CorrectText("Changed 'Ε,\"Ε to Έ", r'(\'Ε|\"Ε)', r'Έ')

	#Fixes Ύ when PDFd as 'Υ or "Υ
	CorrectText("Changed 'Υ,\"Υ to Ύ", r'(\'Υ|\"Υ)', r'Ύ')

	#Fixes Ί when PDFd as 'Ι or "Ι
	CorrectText("Changed 'Ι,\"Ι to Ί", r'(\'Ι|\"Ι)', r'Ί')

	#Fixes Ό when PDFd as 'Ο or "Ο
	CorrectText("Changed 'Ο,\"Ο to Ό", r'(\'Ο|\"Ο)', r'Ό')

	#Fixes Ά when PDFd as 'Α or "Α
	CorrectText("Changed 'Α,\"Α to Ά", r'(\'Α|\"Α)', r'Ά')

	#Fixes Ή when PDFd as 'Η or "Η
	CorrectText("Changed 'Η,\"Η to Ή", r'(\'Η|\"Η)', r'Ή')

	#Fixes Ώ when PDFd as 'Ω or "Ω
	CorrectText("Changed 'Ω,\"Ω to Ώ", r'(\'Ω|\"Ω)', r'Ώ')

	#Fixes ύ when PDFd as ΰ
	CorrectText("Changed ΰ to ύ", r'ΰ', r'ύ')

EDIT: The fix line breaks doesn't work fine. I'm gonna check it again

[DiapDealer]

Originally Posted by Doitsu

@CalibUser: Since the Sigil plugin root directory and the user_dictionary directory are sibling directories it's relatively easy to get the user_dictionary directory location.

For example, you could use the following code to get the dictionary folder:

import os, inspect
def run(bk):
    # get plugin directory path
    plugin_path = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    print(plugin_path)
    
    # get rid of the last two directories
    tmp_path = plugin_path.split(os.path.sep)[:7]
    print(tmp_path)
    
    # add the dictionary path
    tmp_path.extend(['user_dictionaries', 'WordDictionary.txt'])
    
    # convert list back to file path
    dictionary_path = os.path.sep.join(tmp_path)
    print(dictionary_path)

The above code will produce the following output:

Windows:

C:\Users\Doitsu\AppData\Local\sigil-ebook\sigil\plugins\test
['C:', 'Users', 'Doitsu', 'AppData', 'Local', 'sigil-ebook', 'sigil']
C:\Users\Doitsu\AppData\Local\sigil-ebook\sigil\user_dictionaries\WordDictionary.txt

Linux (DiapDealer's build):

/home/doitsu/.local/share/sigil-ebook/sigil/plugins/test
['', 'home', 'doitsu', '.local', 'share', 'sigil-ebook', 'sigil']
/home/doitsu/.local/share/sigil-ebook/sigil/user_dictionaries/WordDictionary.txt

I've attached a test plugin that you can play with.

Thanks for mentioning the 'inspect' module method of finding the current script directory. Saves me the trouble. Though there may be shorter ways to get the current directory of the script that's being run, it's the only one that's guaranteed to work even when a script was invoked as a module.

I would, however suggest something other than the relatively fragile method of converting a path to a list of strings and then using the [:7] slice to strip off the last two directories. If the depth of that path ever increases, it won't point to the sigil preferences directory anymore. To be clear: it's the [:7] slice I find fragile, not the list of strings conversion and eventual re-joining.

I would suggest using [:-2] if you're going to split the path into a list of strings that later get rejoined. Or just use os.path.dirname twice without converting to a list of strings and rejoining.

It's all a bit fragile I guess (even mine), considering that the plugin directory could conceivably change in relation to the Sigil preferences directory.

import os, inspect

# get plugin directory path
plugin_path = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
print(plugin_path)
    
# get rid of the last two directories
tmp_path = os.path.dirname(os.path.dirname(plugin_path))
print(tmp_path)
    
# add the dictionary path
dictionary_path = os.path.join(tmp_path, 'user_dictionaries', 'WordDictionary.txt')
print(dictionary_path)

This will all be simplified when accessing hunspell/dictionaries is incorporated into the plugin launcher framework, but Doitsu's above suggestion could work in the meantime (cross-platform) and wouldn't break even when the new version is released.

You could also determine the path of the current plugin script in the run method of a plugin by using:

def run(bk):
    ppath = bk._w.plugin_dir

It's not really recommended to access those wrapper script properties/methods directly--as they could change at any time. Though in this particular instance ... I don't foresee the plugin_dir property of the wrapper script ever disappearing or having its name changed.

[gipsy]

#Greek line break fix
	if allBreaks == 'Yes':
		CorrectText("Fixed false line breaks:", r'([\u0370-\u03FF,\u1F00-\u1FFF,\'–’“”][</ib>]*)</p>\s+<p[^>]*>([<ib>]*[\u0370-\u03FF,\u1F00-\u1FFF,\'–’“”])', r'\1 \2')
	return(0)

This work with greek in line breaks

[eschwartz]

Originally Posted by Doitsu

This is certainly a valid point, but since CalibUser doesn't have a Mac or a Linux machine, I appreciate it that he at least made an effort to make his plugin OSX/Linux compatible, even though he doesn't have a way of testing it.

Oh certainly it is appreciated. I just thought I'd mention it as something to keep in mind in the future, in order to avoid cross-platform issues from the beginning.

Originally Posted by gipsy

I run a portable installation. When you run the Sigil. It copies the sigil setting (hunspell, user dictionaries, plugins etc to appdata\local\sigil-ebook and when you exit from sigil they copied back to portable location.

I guess I was wrong then. That should teach me to make assumptions.

(I guess it really depends on the app. I know they prefer if at all possible to not do that, it reduces the "portability" angle by potentially leaving unwanted cruft on the host computer.)



It doesn't look like there is any way to override the settings folder location in Sigil.
(And it uses the deprecated-since-5.4 DataLocation, rather than AppDataLocation on Windows and AppConfigLocation on unix -- did Qt have to split it? -- which explains why the config folder is in ~/.local/share/sigil-ebook -- I have always wondered at that non-standard location.)

[DiapDealer]

Originally Posted by eschwartz

(And it uses the deprecated-since-5.4 DataLocation, rather than AppDataLocation on Windows and AppConfigLocation on unix -- did Qt have to split it? -- which explains why the config folder is in ~/.local/share/sigil-ebook -- I have always wondered at that non-standard location.)

It's just inherited. Whether through negligence or changes in Qt over the years. *shrug*

And there's just no "real" pressing need to convert and potentially lose user-settings/plugins in an upgrade (or create a one-time script to copy stuff to the new location). Maybe someday it will change, but it's just not high on the list of priorities at the moment.

[eschwartz]

Originally Posted by DiapDealer

It's just inherited. Whether through negligence or changes in Qt over the years. *shrug*

And there's just no "real" pressing need to convert and potentially lose user-settings/plugins in an upgrade (or create a one-time script to copy stuff to the new location). Maybe someday it will change, but it's just not high on the list of priorities at the moment.

Sorry that was just me being random. I happened to notice that at the same time I noticed there was no way to override the settings dir. (And I was bemused to see Qt hasn't figured out cross-platorm config dirs yet.)

Whether either is *necessary*, I won't venture to say. I agree once it's been used you shouldn't break everyone's settings just to conform to more "proper" standards.