New York Times Recipe failing to verify SSL Cert

[Ramblurr]

Using the latest calibre (2.19.0) on Fedora 21 (official release, not distro package), the NYTimes recipe is failing to fetch with an SSL error. The debug log is below.

Recent SSL + python related bugs in other apps were fixed by installed the python-service-identity package (or service_identity on pypi), but this doesn't seem to be related.

Verifying the domain manually works with openssl works fine:

$ openssl s_client -connect myaccount.nytimes.com:443
<snip>
 Verify return code: 0 (ok)

I noticed the recipe source code uses http://www.nytimes.com/auth/login as the login url which redirects to https://myaccount.nytimes.com/auth/login, but when i edited the recipe to use that url, the SSL error persists.

calibre, version 2.19.0 (linux2, isfrozen: True)
Conversion Error: Failed: Fetch news from New York Times

Fetch news from New York Times
Resolved conversion options
calibre version: 2.19.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x7f3fdbec1c50>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': 'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutput object at 0x7f3fdbec63d0>,
 'page_breaks_before': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Traceback (most recent call last):
  File "site.py", line 51, in main
  File "site-packages/calibre/utils/ipc/worker.py", line 193, in main
  File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1041, in run
  File "site-packages/calibre/customize/conversion.py", line 241, in __call__
  File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 116, in convert
  File "site-packages/calibre/web/feeds/news.py", line 887, in __init__
  File "<string>", line 391, in get_browser
  File "site-packages/mechanize/_mechanize.py", line 203, in open
  File "site-packages/mechanize/_mechanize.py", line 230, in _mech_open
  File "site-packages/mechanize/_opener.py", line 204, in open
  File "site-packages/mechanize/_urllib2_fork.py", line 457, in http_response
  File "site-packages/mechanize/_opener.py", line 221, in error
  File "site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
  File "site-packages/mechanize/_urllib2_fork.py", line 571, in http_error_302
  File "site-packages/mechanize/_mechanize.py", line 203, in open
  File "site-packages/mechanize/_mechanize.py", line 230, in _mech_open
  File "site-packages/mechanize/_opener.py", line 193, in open
  File "site-packages/mechanize/_urllib2_fork.py", line 344, in _open
  File "site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
  File "site-packages/mechanize/_urllib2_fork.py", line 1170, in https_open
  File "site-packages/mechanize/_urllib2_fork.py", line 1118, in do_open
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

[kovidgoyal]

You are missing the root certificate needed to verify that site. Install/update your distros ca-certificates package.

[Ramblurr]

Thanks for responding Kovid.

I reinstalled my distros ca-certificates (which uses mozilla's bundle btw). I definitely have the root certificate.

The issue is in fact that calibre, or a library, is looking in the wrong place for the cert bundle.

But first let me demonstrate I do in fact have the root CA's cert.

Using Chrome, which bundles its own certs, I see the chain as:

1. GeoTrust Global CA
    DE 28 F4 A4 FF E5 B9 2F A3 C5 03 D1 A3 49 A7 F9 96 2A 82 12

2. RapidSSL CA
    C0 39 A3 26 9E E4 B8 E8 2D 00 C5 3F A7 97 B5 A1 9E 83 6F 47

3. *.nytimes.com

    DB 76 F2 CF 5F A4 05 5E D2 95 63 6E 6A 8D 5F 6A 66 D9 54 56

all fingerprints SHA1

Checking to see whether the GeoTrust Global CA with above fingerprint, is in my system

$ awk -v cmd='openssl x509 -noout -fingerprint' ' /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-bundle.trust.crt  | grep DE:28:F4:A4:FF:E5:B9:2F:A3:C5:03:D1:A3:49:A7:F9:96:2A:82:12 -B1

SHA1 Fingerprint=DE:28:F4:A4:FF:E5:B9:2F:A3:C5:03:D1:A3:49:A7:F9:96:2A:82:12

So it is there. Verifying manually with openssl (see original post), confirms this too. I have the root cert installed.

Looking in the wrong place

I used strace to that the CA cert bundle isn't being accessed.

using

$ strace -e open,access ebook-convert nytimes.recipe foo.mobi --test --username XXX --password ZZZ &> strace.log

If you want the whole log I can post it, but the important bit is bolded below.

open("/opt/calibre/lib/python2.7/site-packages/calibre/urllibmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/opt/calibre/lib/python2.7/site-packages/calibre/urllib.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/opt/calibre/lib/python2.7/site-packages/calibre/urllib.pyo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
open("/etc/ssl/cert.pem", O_RDONLY)     = -1 ENOENT (No such file or directory)
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
open("/opt/calibre/lib/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 3
open("/opt/calibre/lib/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/opt/calibre/lib/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 3
Traceback (most recent call last):
  File "site.py", line 51, in main
  File "site-packages/calibre/ebooks/conversion/cli.py", line 360, in main
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1041, in run
  File "site-packages/calibre/customize/conversion.py", line 241, in __call__
  File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 116, in convert
  File "site-packages/calibre/web/feeds/news.py", line 887, in __init__
  File "<string>", line 391, in get_browser
  File "site-packages/mechanize/_mechanize.py", line 203, in open
  File "site-packages/mechanize/_mechanize.py", line 230, in _mech_open
  File "site-packages/mechanize/_opener.py", line 193, in open
  File "site-packages/mechanize/_urllib2_fork.py", line 344, in _open
  File "site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
  File "site-packages/mechanize/_urllib2_fork.py", line 1170, in https_open
  File "site-packages/mechanize/_urllib2_fork.py", line 1118, in do_open
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
open("/opt/calibre/lib/python2.7/site-packages/calibre/shutil.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/opt/calibre/lib/python2.7/site-packages/calibre/shutilmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/opt/calibre/lib/python2.7/site-packages/calibre/shutil.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/opt/calibre/lib/python2.7/site-packages/calibre/shutil.pyo", O_RDONLY) = -1 ENOENT (No such file or directory)
+++ exited with 1 +++

On my system the cert bundle is in /etc/pki/tls/cert.pem or /etc/ssl/certs/ca-bundle.crt, which are both symlinks to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Symlinking the bundle to /etc/ssl/cert.pem, and running calibre fixes the validation error.

However this isn't a good solution nor workaround. I assume the bug is in python, not in calibre proper. But could you add additional cert path search locations?

Here is a simple proof of concept: https://gist.github.com/Ramblurr/bf48299caaadeb17d392

[kovidgoyal]

Sigh, I love linux distros. Neither calibre nor python hardcode any paths to ssl certs. Those come from the openssl library. See the functions X509_get_default_cert_file_env and X509_get_default_cert_file

IIRC those in turn can be controlled via environment variables SSL_CERT_FILE and SSL_CERT_DIR

Presumably, your distro patches its build of openssl to not use the default path. However, calibre comes with its own bundled dependencies, including the openssl libs, which will therefore not have your distro specific patch. So they will not be able to find the files.

Whatever distro you are using should either set the environment variables, or the symlinks, or better yet stop patching upstream packages willy nilly.

Since I doubt your distro is likely to see reason, your remaining workarounds are to either set those env vars yourself, use symlinks, or delete the bundled openssl libs in the calibre package, which will then cause it to use the distro versions (assuming the C runtimes are ABI compatible).

[Ramblurr]

Yea, distro standards are generally a mess. Each distro has "their way" and if you deviate you're in for a world of hurt.

Fedora's "way" in this case is simply they expect you to rely on their system packages, bundling everything separate is anathame (in their eyes). Of course they are so slow at pushing updates to calibre that we want a separately bundled app anyways. I might start building calibre from the src rpm and just updating it myself.

Anyways if you're interested in a workaround:

1. Replacing the bundled libs with symlinks

On Fedora 21 the ABIs are compatible, for now. In /opt/calibre/lib symlinked as follows.

libcrypto.so.1.0.0 -> /usr/lib64/libcrypto.so
libssl.so.1.0.0 -> /usr/lib64/libssl.so.10

2. Setting env vars

Launching calibre manually with "env SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt calibre" works, or you can edit the .desktop files (/usr/local/share/applications/calibre-*desktop) to set the variable yourself.

Option one is easiest for me, and I'll be doing that until I take the time to roll my own RPM of calibre.

Kovid, any chance calibre could detect the location? According to the article below, the best solution is

1. Check if /etc/pki/tls/certs/ca-bundle.crt exists, if so use with SSL_CERT_FILE [Fedora, Redhat, Arch etc]
2. Check if /etc/ssl/certs exists, if so use with SSL_CERT_DIR [Debian, Ubuntu, etc]
3. Fall back on default settings

Here is a fascinating read on the origin and evolution of the /etc/ssl/ slash /etc/pki mess:

A note about SSL/TLS trusted certificate stores, and platforms (OpenSSL and GnuTLS) -
https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/

[kovidgoyal]

That should do the trick https://github.com/kovidgoyal/calibr...e7e57250bce8a5

I haven't tested it on a Fedora machine since I dont want to build a VM just for this, let me know if it doesn't work in the next release.

[Ramblurr]

This is fixed for me in v.2.20. Thanks Kovid.