epubcheck plugin for Sigil

[KevinH]

I have added a new python3 module "certifi" to the Python 3.7 framework builds to work around the missing network certificate. This change has been pushed to master. The fix for this mac specific ssl issue will appear in the next release.

Thanks for your bug report.

[odamizu]

KevinH and Doitsu!

[odamizu]

Sadly, when I tested a clean install of the plugin on 0.9.16, I got the same error:

*** PYTHON ERROR ***
SSL version: OpenSSL 1.1.1c-dev xx XXX xxxx
An exception of type URLError occurred.
Arguments:
(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)'),)

Not a biggie for me as I have disabled the auto-update ("github": false) and am content to manually update the plugin when necessary, but I wanted to report this in case it's helpful.

Originally Posted by KevinH

I have added a new python3 module "certifi" to the Python 3.7 framework builds to work around the missing network certificate. This change has been pushed to master. The fix for this mac specific ssl issue will appear in the next release.

Thanks for your bug report.

[KevinH]

That problem really should not be happening anymore. I have compiled in a static copy of full openSSL library and added the certfi module to provide the certificates.

I will try on another mac machine I have to see what is going on here.

Sorry about that.

Kevin

Originally Posted by odamizu

Sadly, when I tested a clean install of the plugin on 0.9.16, I got the same error:

*** PYTHON ERROR ***
SSL version: OpenSSL 1.1.1c-dev xx XXX xxxx
An exception of type URLError occurred.
Arguments:
(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)'),)

Not a biggie for me as I have disabled the auto-update ("github": false) and am content to manually update the plugin when necessary, but I wanted to report this in case it's helpful.

[KevinH]

Okay, It seems the python 3.6 and later devs for macOs are complete brainless morons. Their "solution" given here:

https://github.com/python/python/blo...icates.command

is to have the user post process its python executable after installation to use symlinks to trick its ssl.c source code of its ssl module to load the certificates from certfi and then removing the true openssl certificares from where they were installed by openssl.

All of this only works if you always install openssl in the same place, and only install python in the same place - not relocatable and embedded like Sigil uses it, and of course means you then break any code signing of Sigil when you run it.

This is a complete joke for being an official solution. It simply will not work for us at all.

Not sure why Python just does not use its own internal openssl build as they do on other platforms.

It is a horrible hack and not a solution at all.

Not sure how and even if there is a fix for us that does not require the user to build and install openssl 1.1.1 on their own mac.

KevinH

[KevinH]

Yes, confirmed. Python's trick is to remove the official openssl /usr/local/ssl/cert.pem file and replace it with a hard symlink to the version installed with certifi potentially breaking all other apps that need or use openssl.

Just horrible.

[DiapDealer]

Ouch. That sounds like that limits a plugin from even checking a file on https servers for update availability in general. Not just this plugin. Another thing to consider: I moved Sigil's update checker routines to embedded python calls quite a while ago. You may want to check if that's going to be a problem (or IS a problem now) on Macs.

[KevinH]

Yes that is something I will definitely check. Qt does have routines to handle this and we even use them in places in the Sigil source. Perhaps for safety sake we could go back to using Qt's version of handling the update requests just for Sigil?

Originally Posted by DiapDealer

Ouch. That sounds like that limits a plugin from even checking a file on https servers for update availability in general. Not just this plugin. Another thing to consider: I moved Sigil's update checker routines to embedded python calls quite a while ago. You may want to check if that's going to be a problem (or IS a problem now) on Macs.

[KevinH]

Odamizu,

Are you okay using the command line (Terminal.app) and if so would you be willing to try the following for me.

This assumes you have installed version 0.9.16 in Sigil.app in /Applications. If not we will have to modify the commands:

1. First fire up Terminal.app and check to see if you have a /usr/local/ssl directory

ls /usr/local/ssl

2. If not, we will now create that directory as follows. It will prompt you for your own password.

cd /usr/local
sudo mkdir ssl

3. Now lets move inside the Sigil.app itself and do an ls to look for a certificate

cd /Applications/Sigil.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/certifi/

ls

4. Look for a file called cacert.pem. Copy it into /usr/local/ssl/ directory and rename it to cert.pem by doing the following:

sudo cp cacert.pem /usr/local/ssl/cert.pem

5. Now close out of Terminal.app and fire up Sigil, remember to reset the last date checked in epubcheck's preferences json file and then try using epubcheck again.

Does this fix the issue?

Thanks,

KevinH

Originally Posted by odamizu

Sadly, when I tested a clean install of the plugin on 0.9.16, I got the same error:

*** PYTHON ERROR ***
SSL version: OpenSSL 1.1.1c-dev xx XXX xxxx
An exception of type URLError occurred.
Arguments:
(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)'),)

Not a biggie for me as I have disabled the auto-update ("github": false) and am content to manually update the plugin when necessary, but I wanted to report this in case it's helpful.

[kovidgoyal]

Dont use cerifi. Instead include the certificates .pem file in the Sigil bundle and set the SSL_CERT_FIlE env var in Info.plist to point to it. Like this:

SSL_CERT_FILE='@executable_path/../Resources/resources/mozilla-ca-certs.pem'

[DiapDealer]

Originally Posted by KevinH

Perhaps for safety sake we could go back to using Qt's version of handling the update requests just for Sigil?

Fine by me. I only moved it to the embedded Python routines because I was compiling Qt without SSL support for Windows (way back when) to avoid the issue where another version of openssl being on the path was causing Sigil crashes. With QtWebEngine, there's really no viable way to exclude ssl support in Qt anyway.

That won't help plugin devs look up https urls with plugins on macs, though.

EDIT: Ooh... I hope Kovid's proposal works out for mac plugins in general, though! (not that we still can't move Sigil's update checker back to Qt).

[KevinH]

Thanks Kovid! We will have to fix this with a new release.

That said, as a simpler workaround right now based on Kovid's approach, just adding an the following environment variable should then work since we can use the certifi certificate that is in this release until we add our own and change Info.plist and add the mozilla cert.

export SSL_CERT_FILE="/Applications/Sigil.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/certifi/cacert.pem"

That should do effectively the same thing unless we mess up the environment somehow when we launch python plugins.

Also, another workaround could be to do this upfront in the plugin code using info the the python path to extract the correct path to the certificate no matter where the user install Sigil.app on a mac.

[DiapDealer]

Originally Posted by KevinH

That should do effectively the same thing unless we mess up the environment somehow when we launch python plugins.

I believe you copy the system environment to the newly-created Python process environment when launching a Sigil plugin on Mac (if I recall correctly) before inserting a few new environment variables. So your temporary workaround should work.

We could also insert this new one in the same place in PluginRunnner.cpp (for the next release), since the env var really only needs to exist for the python process and not Sigil in general.

[KevinH]

Yes, that way we could keep the certifi package just to get an up to date ceritificate that will always be stored in site-packages and then get the current app location and build the proper environment var to be included just before launching the plugin.

I would guess most people install Sigil.app into /Applications but I run it from all over the place and have about 5 versions of it on my machine, so I want to build up the path to the Python.framework inside Sigil.app dynamically at runtime and then export it.

[KevinH]

I am testing the version with PluginRunner setting the correct path to the site-packages/certifi/cacert.pem and exporting it as SSL_CERT_FILE. If that works, I can make a new release just for macOS to replace the current binary.