Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 04-09-2018, 07:55 AM   #1
Korben Dallas
Junior Member
Korben Dallas began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Dec 2010
Device: Sony Reader PRS-350
[SSL: CERTIFICATE_VERIFY_FAILED]

Since January (approximately) the functionality of downloading metadata ceased to work properly. Every attempt to download any metadata fails invariably—see following log:

Spoiler:

calibre, version 3.21.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{u'authors': [u'Hans Dominik'], u'identifiers': {u'uri': u'http://gutenberg.spiegel.de/buch/atlantis-8709/1'}, u'title': u'Atlantis', u'timeout': 30}
Using plugins: Google (1, 0, 0), Amazon.com (1, 2, 1), Overdrive (1, 0, 0)
The log from individual plugins is below

****************************** Amazon.com (1, 2, 1) ******************************
Found 0 results
Downloading from Amazon.com took 0.743000030518
User-agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0
Server: auto
Making google query: https://www.google.com/search?q=Atla...Awww.amazon.de
Plugin Amazon.com failed
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\identify. py", line 48, in run
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 1346, in identify
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 1282, in search_search_engine
File "site-packages\calibre\ebooks\metadata\sources\search_en gines.py", line 230, in google_search
File "site-packages\calibre\ebooks\metadata\sources\search_en gines.py", line 65, in query
File "site-packages\mechanize\_mechanize.py", line 239, in open_novisit
File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open
File "site-packages\mechanize\_opener.py", line 195, in open
File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open
File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
File "site-packages\calibre\utils\browser.py", line 28, in https_open
File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

************************************************** ******************************

****************************** Google (1, 0, 0) ******************************
Found 0 results
Downloading from Google took 0.713000059128
Making query: https://books.google.com/books/feeds...max-results=20
Failed to make identify query: u'https://books.google.com/books/feeds/volumes?q=intitle%3AAtlantis%2Binauthor%3AHans+ina uthor%3ADominik&min-viewability=none&start-index=1&max-results=20'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\google.py ", line 362, in identify
File "site-packages\mechanize\_mechanize.py", line 239, in open_novisit
File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open
File "site-packages\mechanize\_opener.py", line 195, in open
File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open
File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
File "site-packages\calibre\utils\browser.py", line 28, in https_open
File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

************************************************** ******************************

****************************** Overdrive (1, 0, 0) ******************************
Found 0 results
Downloading from Overdrive took 1.93400001526
Initial query is Hans Dominik
Cross reference query is Atlantis

************************************************** ******************************
The identify phase took 2.01 seconds
The longest time (1.934000) was taken by: Overdrive
Merging results from different sources and finding earliest publication dates from the worldcat.org service
We have 0 merged results, merging took: 0.00 seconds



Calling the urls from the log via Firefox (59.0.2), Edge, and Internet Explorer (11.334.16299.0) results in valid pages, i.e. no error.

So now the question is:
  • Why doesn't Calibre find the appropriate certificates?
Or put otherwise:
  • Where should I put the required certificate files, and in which format?
or
  • How can I prevent Calibre from trying to check the validity of these certificates? (As a last resort…)

Btw: My System:
  • Windows 10 Pro 64bit (ver 1709 build 16299.334)
  • Calibre 3.21 [64bit]
  • no proxy settings found whatsoever
Korben Dallas is offline   Reply With Quote
Old 04-09-2018, 08:40 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 33,747
Karma: 10215946
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You've probably got some software on your computer that is intercepting SSL connections and inserting its own certificates in between. Either malware/trojan of some kind or security software (which is typically also malware, just one that you have to pay for).
kovidgoyal is offline   Reply With Quote
Advert
Old 04-09-2018, 11:06 AM   #3
gbm
Wizard
gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.gbm ought to be getting tired of karma fortunes by now.
 
Posts: 1,465
Karma: 5435106
Join Date: Jun 2010
Device: Windows 10 tablet,Hisence Sero 7 Pro RIP, Nook STR, jetbook lite
Quote:
Originally Posted by Korben Dallas View Post
Since January (approximately) the functionality of downloading metadata ceased to work properly. Every attempt to download any metadata fails invariably—see following log:

Spoiler:

calibre, version 3.21.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{u'authors': [u'Hans Dominik'], u'identifiers': {u'uri': u'http://gutenberg.spiegel.de/buch/atlantis-8709/1'}, u'title': u'Atlantis', u'timeout': 30}
Using plugins: Google (1, 0, 0), Amazon.com (1, 2, 1), Overdrive (1, 0, 0)
The log from individual plugins is below

****************************** Amazon.com (1, 2, 1) ******************************
Found 0 results
Downloading from Amazon.com took 0.743000030518
User-agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0
Server: auto
Making google query: https://www.google.com/search?q=Atla...Awww.amazon.de
Plugin Amazon.com failed
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\identify. py", line 48, in run
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 1346, in identify
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 1282, in search_search_engine
File "site-packages\calibre\ebooks\metadata\sources\search_en gines.py", line 230, in google_search
File "site-packages\calibre\ebooks\metadata\sources\search_en gines.py", line 65, in query
File "site-packages\mechanize\_mechanize.py", line 239, in open_novisit
File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open
File "site-packages\mechanize\_opener.py", line 195, in open
File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open
File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
File "site-packages\calibre\utils\browser.py", line 28, in https_open
File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

************************************************** ******************************

****************************** Google (1, 0, 0) ******************************
Found 0 results
Downloading from Google took 0.713000059128
Making query: https://books.google.com/books/feeds...max-results=20
Failed to make identify query: u'https://books.google.com/books/feeds/volumes?q=intitle%3AAtlantis%2Binauthor%3AHans+ina uthor%3ADominik&min-viewability=none&start-index=1&max-results=20'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\google.py ", line 362, in identify
File "site-packages\mechanize\_mechanize.py", line 239, in open_novisit
File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open
File "site-packages\mechanize\_opener.py", line 195, in open
File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open
File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
File "site-packages\calibre\utils\browser.py", line 28, in https_open
File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

************************************************** ******************************

****************************** Overdrive (1, 0, 0) ******************************
Found 0 results
Downloading from Overdrive took 1.93400001526
Initial query is Hans Dominik
Cross reference query is Atlantis

************************************************** ******************************
The identify phase took 2.01 seconds
The longest time (1.934000) was taken by: Overdrive
Merging results from different sources and finding earliest publication dates from the worldcat.org service
We have 0 merged results, merging took: 0.00 seconds



Calling the urls from the log via Firefox (59.0.2), Edge, and Internet Explorer (11.334.16299.0) results in valid pages, i.e. no error.

So now the question is:
  • Why doesn't Calibre find the appropriate certificates?
Or put otherwise:
  • Where should I put the required certificate files, and in which format?
or
  • How can I prevent Calibre from trying to check the validity of these certificates? (As a last resort…)

Btw: My System:
  • Windows 10 Pro 64bit (ver 1709 build 16299.334)
  • Calibre 3.21 [64bit]
  • no proxy settings found whatsoever
Do you have one of those "free coupon" tool bars installed? Better yet what is you home page? If it is Ask dot com, can you change it, if it is and you can not change it you need to run malwarebites.

bernie
gbm is offline   Reply With Quote
Old 04-10-2018, 06:24 AM   #4
Korben Dallas
Junior Member
Korben Dallas began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Dec 2010
Device: Sony Reader PRS-350
Feeding the urls from the log to wget I recognized a similar error. wget too is unable to check the validity of the certificate of www.google.com.

Most convientently wget has an option as --no-check-certificate to avoid certificate errors.

According to the docs the same should be possible in Python, either in the code or at runtime via an environment viariable named PYTHONHTTPSVERIFY.

@Kovid: How can I tell, if some software is intercepting SSL connection? To me it looks more of a problem with openSSL not finding the appropriate certificates.

@gbm: I don't have any "free coupon" tool bars installed.
Korben Dallas is offline   Reply With Quote
Old 04-10-2018, 06:45 AM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 33,747
Karma: 10215946
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
There's no simple way. Basically you use a tool to manually inspect the certificate returned by opening a connection to the website, compare it to the certificate from a computer that does not exhibit this problem and see if the certificates are the same. If they are not the same them you are sufferring from an SSL MITM attack.

IITRC you can view certificates using the openssl command line tool can be used to view certificates, or if you are not scared to program, you can write a short script that does that in any programming language with openssl bindings.
kovidgoyal is offline   Reply With Quote
Advert
Old 04-10-2018, 06:46 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 33,747
Karma: 10215946
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
And FYI OpenSSL loads certificates from the windows certificate store on windows.
kovidgoyal is offline   Reply With Quote
Old 04-12-2018, 10:47 AM   #7
Korben Dallas
Junior Member
Korben Dallas began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Dec 2010
Device: Sony Reader PRS-350
Some further investigation took me to the fact that over the years a multitude of openSSL file versions have accumulated on my system. Every application that comes bundled with some version of openSSL stores the relevant dlls in their own location.

To clean up the mess I downloaded the most recent version of openSSL and read some of the docs, which is obviously always a good idea if something seems to go wrong .

In the end I managed to eliminate both the errors with Calibre's metadata download facility and wget complaining about not being able to check certificate validity by pointing openSSL via its environment variables
  • OPENSSL_CONF
  • SSL_CERT_FILE
to
The certificate collection is necessary because openSSL does not use the Windows certificate store.
Korben Dallas is offline   Reply With Quote
Old 04-12-2018, 11:19 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 33,747
Karma: 10215946
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
openssl most definitely does use the windows certificate store. https://github.com/openssl/openssl/b...gines/e_capi.c


If it is not doing so on your system, that is because of problems specific to your system.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
SSL: CERTIFICATE_VERIFY_FAILED Error Bengb Library Management 10 07-24-2017 12:56 PM
Aeon.co - Recipe download failed - CERTIFICATE_VERIFY_FAILED emanu Recipes 13 07-16-2016 01:29 AM
IRC SSL cert jgoguen Feedback 0 08-03-2015 01:38 PM
Does CC support SSL connection? coleman Calibre Companion 7 04-04-2014 12:31 AM


All times are GMT -4. The time now is 03:46 PM.


MobileRead.com is a privately owned, operated and funded community.