[SSL: CERTIFICATE_VERIFY_FAILED]

Korben Dallas · 04-09-2018, 06:55 AM

Since January (approximately) the functionality of downloading metadata ceased to work properly. Every attempt to download any metadata fails invariably—see following log:

Spoiler:

Calling the urls from the log via Firefox (59.0.2), Edge, and Internet Explorer (11.334.16299.0) results in valid pages, i.e. no error.

So now the question is:

Why doesn't Calibre find the appropriate certificates?

Or put otherwise:

Where should I put the required certificate files, and in which format?

or

How can I prevent Calibre from trying to check the validity of these certificates? (As a last resort…)

Btw: My System:

Windows 10 Pro 64bit (ver 1709 build 16299.334)
Calibre 3.21 [64bit]
no proxy settings found whatsoever

kovidgoyal · 04-09-2018, 07:40 AM

You've probably got some software on your computer that is intercepting SSL connections and inserting its own certificates in between. Either malware/trojan of some kind or security software (which is typically also malware, just one that you have to pay for).

gbm · 04-09-2018, 10:06 AM

Quote:

Originally Posted by Korben Dallas

Since January (approximately) the functionality of downloading metadata ceased to work properly. Every attempt to download any metadata fails invariably—see following log:

Spoiler:

Calling the urls from the log via Firefox (59.0.2), Edge, and Internet Explorer (11.334.16299.0) results in valid pages, i.e. no error.

So now the question is:

Why doesn't Calibre find the appropriate certificates?

Or put otherwise:

Where should I put the required certificate files, and in which format?

or

How can I prevent Calibre from trying to check the validity of these certificates? (As a last resort…)

Btw: My System:

Windows 10 Pro 64bit (ver 1709 build 16299.334)
Calibre 3.21 [64bit]
no proxy settings found whatsoever

Do you have one of those "free coupon" tool bars installed? Better yet what is you home page? If it is Ask dot com, can you change it, if it is and you can not change it you need to run malwarebites.

bernie

Korben Dallas · 04-10-2018, 05:24 AM

Feeding the urls from the log to wget I recognized a similar error. wget too is unable to check the validity of the certificate of www.google.com.

Most convientently wget has an option as --no-check-certificate to avoid certificate errors.

According to the docs the same should be possible in Python, either in the code or at runtime via an environment viariable named PYTHONHTTPSVERIFY.

@Kovid: How can I tell, if some software is intercepting SSL connection? To me it looks more of a problem with openSSL not finding the appropriate certificates.

@gbm: I don't have any "free coupon" tool bars installed.

kovidgoyal · 04-10-2018, 05:45 AM

There's no simple way. Basically you use a tool to manually inspect the certificate returned by opening a connection to the website, compare it to the certificate from a computer that does not exhibit this problem and see if the certificates are the same. If they are not the same them you are sufferring from an SSL MITM attack.

IITRC you can view certificates using the openssl command line tool can be used to view certificates, or if you are not scared to program, you can write a short script that does that in any programming language with openssl bindings.

kovidgoyal · 04-10-2018, 05:46 AM

And FYI OpenSSL loads certificates from the windows certificate store on windows.

Korben Dallas · 04-12-2018, 09:47 AM

Some further investigation took me to the fact that over the years a multitude of openSSL file versions have accumulated on my system. Every application that comes bundled with some version of openSSL stores the relevant dlls in their own location.

To clean up the mess I downloaded the most recent version of openSSL and read some of the docs, which is obviously always a good idea if something seems to go wrong

.

In the end I managed to eliminate both the errors with Calibre's metadata download facility and wget complaining about not being able to check certificate validity by pointing openSSL via its environment variables

OPENSSL_CONF
SSL_CERT_FILE

to

the configuration file that came with openSSL package (openssl.cfg)
a collection of certificates that is part of Firefox's NSS (cacert.pem, found here: https://curl.haxx.se/docs/caextract.html

The certificate collection is necessary because openSSL does not use the Windows certificate store.

kovidgoyal · 04-12-2018, 10:19 AM

openssl most definitely does use the windows certificate store. https://github.com/openssl/openssl/b...gines/e_capi.c

If it is not doing so on your system, that is because of problems specific to your system.

26rdf · 03-02-2020, 05:48 AM

Hello,
I don't quite understand (not even at all) the advice offered to work around the problem. For my part, I found that by simply launching Google chrome (and closing it), and then launching Caliber and searching for meta data, the problem disappeared.
Similarly, when the search (metadata) is performed from a Caliber installed on a virtual machine (VmWare), the problem does not exist.
I am completely incompetent to explain this phenomenon.

04-09-2018, 06:55 AM	#1
Korben Dallas Junior Member Posts: 6 Karma: 10 Join Date: Dec 2010 Device: Sony Reader PRS-350	[SSL: CERTIFICATE_VERIFY_FAILED] Since January (approximately) the functionality of downloading metadata ceased to work properly. Every attempt to download any metadata fails invariably—see following log: Spoiler: calibre, version 3.21.0 ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details". Running identify query with parameters: {u'authors': [u'Hans Dominik'], u'identifiers': {u'uri': u'http://gutenberg.spiegel.de/buch/atlantis-8709/1'}, u'title': u'Atlantis', u'timeout': 30} Using plugins: Google (1, 0, 0), Amazon.com (1, 2, 1), Overdrive (1, 0, 0) The log from individual plugins is below **************************** Amazon.com (1, 2, 1) ************************** Found 0 results Downloading from Amazon.com took 0.743000030518 User-agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0 Server: auto Making google query: https://www.google.com/search?q=Atla...Awww.amazon.de Plugin Amazon.com failed Traceback (most recent call last): File "site-packages\calibre\ebooks\metadata\sources\identify. py", line 48, in run File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 1346, in identify File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 1282, in search_search_engine File "site-packages\calibre\ebooks\metadata\sources\search_en gines.py", line 230, in google_search File "site-packages\calibre\ebooks\metadata\sources\search_en gines.py", line 65, in query File "site-packages\mechanize\_mechanize.py", line 239, in open_novisit File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open File "site-packages\mechanize\_opener.py", line 195, in open File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain File "site-packages\calibre\utils\browser.py", line 28, in https_open File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)> ********************************************** ************************** ************************** Google (1, 0, 0) ************************** Found 0 results Downloading from Google took 0.713000059128 Making query: https://books.google.com/books/feeds...max-results=20 Failed to make identify query: u'https://books.google.com/books/feeds/volumes?q=intitle%3AAtlantis%2Binauthor%3AHans+ina uthor%3ADominik&min-viewability=none&start-index=1&max-results=20' Traceback (most recent call last): File "site-packages\calibre\ebooks\metadata\sources\google.py ", line 362, in identify File "site-packages\mechanize\_mechanize.py", line 239, in open_novisit File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open File "site-packages\mechanize\_opener.py", line 195, in open File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain File "site-packages\calibre\utils\browser.py", line 28, in https_open File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)> ********************************************** ************************** ************************** Overdrive (1, 0, 0) ************************** Found 0 results Downloading from Overdrive took 1.93400001526 Initial query is Hans Dominik Cross reference query is Atlantis ********************************************** **************************** The identify phase took 2.01 seconds The longest time (1.934000) was taken by: Overdrive Merging results from different sources and finding earliest publication dates from the worldcat.org service We have 0 merged results, merging took: 0.00 seconds Calling the urls from the log via Firefox (59.0.2), Edge, and Internet Explorer (11.334.16299.0) results in valid pages, i.e. no error. So now the question is: Why doesn't Calibre find the appropriate certificates? Or put otherwise: Where should I put the required certificate files, and in which format? or How can I prevent Calibre from trying to check the validity of these certificates? (As a last resort…) Btw: My System: Windows 10 Pro 64bit (ver 1709 build 16299.334) Calibre 3.21 [64bit] no proxy settings found whatsoever

04-12-2018, 09:47 AM	#7
Korben Dallas Junior Member Posts: 6 Karma: 10 Join Date: Dec 2010 Device: Sony Reader PRS-350	Some further investigation took me to the fact that over the years a multitude of openSSL file versions have accumulated on my system. Every application that comes bundled with some version of openSSL stores the relevant dlls in their own location. To clean up the mess I downloaded the most recent version of openSSL and read some of the docs, which is obviously always a good idea if something seems to go wrong . In the end I managed to eliminate both the errors with Calibre's metadata download facility and wget complaining about not being able to check certificate validity by pointing openSSL via its environment variables OPENSSL_CONF SSL_CERT_FILE to the configuration file that came with openSSL package (openssl.cfg) a collection of certificates that is part of Firefox's NSS (cacert.pem, found here: https://curl.haxx.se/docs/caextract.html The certificate collection is necessary because openSSL does not use the Windows certificate store.

03-02-2020, 05:48 AM	#9
26rdf Junior Member Posts: 1 Karma: 10 Join Date: Mar 2018 Device: Kobo Aura	CERTIFICATE_VERIFY_FAILED Hello, I don't quite understand (not even at all) the advice offered to work around the problem. For my part, I found that by simply launching Google chrome (and closing it), and then launching Caliber and searching for meta data, the problem disappeared. Similarly, when the search (metadata) is performed from a Caliber installed on a virtual machine (VmWare), the problem does not exist. I am completely incompetent to explain this phenomenon.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
SSL: CERTIFICATE_VERIFY_FAILED Error	Bengb	Library Management	10	07-24-2017 11:56 AM
Aeon.co - Recipe download failed - CERTIFICATE_VERIFY_FAILED	emanu	Recipes	13	07-16-2016 12:29 AM
IRC SSL cert	jgoguen	Feedback	0	08-03-2015 12:38 PM
Does CC support SSL connection?	coleman	Calibre Companion	7	04-03-2014 11:31 PM

04-09-2018, 07:40 AM	#2
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You've probably got some software on your computer that is intercepting SSL connections and inserting its own certificates in between. Either malware/trojan of some kind or security software (which is typically also malware, just one that you have to pay for).

04-10-2018, 05:24 AM	#4
Korben Dallas Junior Member Posts: 6 Karma: 10 Join Date: Dec 2010 Device: Sony Reader PRS-350	Feeding the urls from the log to wget I recognized a similar error. wget too is unable to check the validity of the certificate of www.google.com. Most convientently wget has an option as --no-check-certificate to avoid certificate errors. According to the docs the same should be possible in Python, either in the code or at runtime via an environment viariable named PYTHONHTTPSVERIFY. @Kovid: How can I tell, if some software is intercepting SSL connection? To me it looks more of a problem with openSSL not finding the appropriate certificates. @gbm: I don't have any "free coupon" tool bars installed.

04-10-2018, 05:45 AM	#5
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	There's no simple way. Basically you use a tool to manually inspect the certificate returned by opening a connection to the website, compare it to the certificate from a computer that does not exhibit this problem and see if the certificates are the same. If they are not the same them you are sufferring from an SSL MITM attack. IITRC you can view certificates using the openssl command line tool can be used to view certificates, or if you are not scared to program, you can write a short script that does that in any programming language with openssl bindings.

04-10-2018, 05:46 AM	#6
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	And FYI OpenSSL loads certificates from the windows certificate store on windows.

04-12-2018, 10:19 AM	#8
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	openssl most definitely does use the windows certificate store. https://github.com/openssl/openssl/b...gines/e_capi.c If it is not doing so on your system, that is because of problems specific to your system.