SSL certificate verification failure in python on Mavericks

Question

I'm stuck on a persistent SSL verification issue.

SSL: CERTIFICATE_VERIFY_FAILED

I discovered the error while building a Django app that had users authenticate using Mozilla Persona.

(python3.4)> import requests
(python3.4)> requests.get('https://verifier.login.persona.org')

I get a SSL: CERTIFICATE_VERIFY_FAILED tracing back from requests to urllib3 to ssl:

...
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/ssl.py", line 805, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

...
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.4.1-py3.4.egg/requests/packages/urllib3/connectionpool.py", line 543, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

...
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.4.1-py3.4.egg/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

Difference between python3 and python2

Here's where it starts to get interesting: I don't get the same issue when using python2.7:

(python2.7)> import requests
(python2.7)> requests.get('https://verifier.login.persona.org')
<Response [200]>

My first thought was that the two versions of requests might be using different certs[1], so I was pretty surprised to find the two files were exactly the same:

(bash)$ diff `python3.4 -c "import requests; print(requests.certs.where())"` \
             `python2.7 -c "import requests; print requests.certs.where()"`
# no diff

Error recreated in openssl and solved using -CAFile

Interestingly, the issue is not limited to python3.4[2].

(bash)$ openssl s_client -connect github.com:443
...
Verify return code: 20 (unable to get local issuer certificate)

Edit A comment from Steffen informed me that this "debugging" method isn't actually informative, as s_client expects a -CApath in order to verify. However, the fact that I can specify the same certificate that the requests package is using and I don't get the same error is still interesting:

(bash)$ openssl s_client -connect github.com:443 \
        -CAfile `python3 -c 'import requests; print(requests.certs.where())'`
...
Verify return code: 0 (ok)

At this point, I'm completely out of my element. I don't know is if this is really an openssl issue, or something about OSX Mavericks[3]. Here's the version of openssl I'm using:

(bash)$ openssl version
OpenSSL 1.0.1f 6 Jan 2014

Mavericks KeyChain.app

For OS-specific solutions, I've tried clearing my login KeyChain[4], to no avail.

Issues with pip

There's one last bit of evidence that may or may not be relevant. python3.4 ships with pip intact. However, the pip3 command is useless to me. No matter what I try to install:

(bash)$ pip3 install [new-lib] # pip 1.5.6

I get:

Downloading/unpacking [new-lib]
    Cannot fetch index base URL https://pypi.python.org/simple/
    Could not find any downloads that satisfy the requirement [new-lib]
Cleaning up...
    No distributions at all found for [new-lib]
    Storing debug log for failure in ~/.pip/pip.log

Although this isn't (explicitly) an SSL error, it seems similar[5] and a successful workaround has been to install an older version of pip using easy_install in my virtualenvs[5]. I'm crossing my fingers that the two issues are related.

Recap:

Seeking possible solutions for SSL certificate failure error (without using verify = False in the requests calls).
I get the error in python3.4 but not python2.7 even though the cert.pem used in both cases is exactly the same.
Though I can recreate an SSL error using openssl s_client -connect I can avoid it by specifying -CAFile to the cert.pem used by the requests library.
My best guess is that this is an issue particular to Mavericks, but I have no idea how to proceed.
I'm hoping to find a solution that also allows me to use pip3 to install python3.4 packages as expected.

Thanks for your help!

[1]: python2.7 on my machine was installed using Enthought. But installing a system version of python2.7 and the requests library works too.

[2]: See openssl, python requests error: "certificate verify failed" for a similar problem using python 2.7

[3]: It seems Mavericks introduced a change in openssl? http://curl.haxx.se/mail/archive-2013-10/0036.html

[4]: Clearning KeyChain.app from here: https://superuser.com/a/721629/261875

[5]: SSL error with pip3: https://stackoverflow.com/a/22051466/2506078

This is definitely something very specific to your machine. Using requests I can connect to that URL without issue. I'm also on Mavericks but `openssl version` returns '0.9.8y' from Feb 2013 (which is prior to the link you posted about a change in OpenSSL). — Ian Stapleton Cordasco, Sep 14 '14 at 16:57
As for the first error with `s_client`: `s_client` has no default verification path so any verification will return an error unless you explicitly give a CApath, no matter which host you connect to. — Steffen Ullrich, Sep 14 '14 at 17:39
From where did you install Python 3.4? Please show the results of `pythonx.x -c 'import sys;print(sys.version)'` for both the Python 2 that works and the Python 3 that fails. — Ned Deily, Sep 14 '14 at 17:53
@Ned Python 3.4 was installed as is (.dmg) from python.org: python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:22:19), [GCC 4.2.1 (Apple Inc. build 5577)]; python 2.7 was installed from an Enthought distribution: python 2.7.6 | 32-bit (default, Apr 11 2014, 12:06:39) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]. However, my system python 2.7 works as well: python 2.7.6 (default, Nov 18 2013, 15:12:51) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] — pedmiston, Sep 14 '14 at 19:43

Ned Deily · Accepted Answer · 2014-09-15T04:10:13.520

2

From the additional information that you have supplied, it appears you have installed the 32-bit-only version of Python 3.4.1 from python.org. This version is intended primarily for use on OS X 10.5 systems; as such, it is linked with the version of OpenSSL supplied by Apple with 10.5. You can avoid this problem by using the 64-bit/32-bit 3.4.1 installer from python.org; this version is recommended for OS X 10.6+ and is linked with the newer version of Apple's OpenSSL. Otherwise, you could manually download distributions from PyPI using curl or a browser and have install pip install them from the downloaded file(s).

edited Sep 15 '14 at 04:10

answered Sep 15 '14 at 00:41

Ned Deily

83,389
16
128
151

The fact that this works on Python 2 without any extra dependencies, but doesn't on Python 3 makes me think that this has no relation to SNI. The hint for SNI is usually that it works on Python 3 but not Python 2. – Ian Stapleton Cordasco Sep 15 '14 at 01:11
1

It is easy to demonstrate that `pip` downloads work fine with the Python 3.4.1 from the python.org 64-bit/32-bit Python 3.4.1 installer but fail with the 32-bit-only installer and it is is easy to demonstrate that they are linked with different versions of the Apple-supplied OpenSSl libraries: `pythonx.y -c 'import ssl;print(ssl.OPENSSL_VERSION)`. The latter show a `0.9.7` version; SNI support was added in `0.9.8` which the former uses. – Ned Deily Sep 15 '14 at 02:56
None of pypi.python.org, verifier.login.persona.org or github.com depend on SNI. That means they return the same certificate if you use SNI or not, so this can't be the problem. – Steffen Ullrich Sep 15 '14 at 03:32
@sigmavirus24 Sorry, of course you are right that it could be some other difference between the two OS X SSL versions rather than just `SNI` support. Any suggestions as to what? – Ned Deily Sep 15 '14 at 03:32
They differ in supported ciphers etc, but from what I see that should not be a problem either. All these sites work without problem with SSLv3 client up to a TLS1.1 client (probably TLS1.2 too). And because the error clearly says, that the verification fails it cannot be a problem of shared ciphers. I think the difference is to have a fallback to OS X keyring or to have it not (see my response). – Steffen Ullrich Sep 15 '14 at 03:37
Thanks, differences in the keyring support could be it. I'll look into it further later. I've edited my answer to remove the references to SNI. In any case, the solution for the OP is the same: use the other python.org installer. – Ned Deily Sep 15 '14 at 04:14
@Ned was right on. printing `ssl.OPENSSL_VERSION` from python3 revealed that it was using 0.9.7, whereas the same command from python2 was 0.9.8. Installing the 64-bit up'ed the OpenSSL version to 0.9.8. Thanks for the help everyone! I hope this is useful for others. – pedmiston Sep 15 '14 at 14:41

Steffen Ullrich · Answer 2 · 2014-09-14T17:53:51.587

Just a guess: the OpenSSL as shipped with Mac OS X (which is still 0.9.8) has special hooks in it so that it falls back to OS X keyring if verification fails against the CAs given to OpenSSL itself. But, if you use your own OpenSSL it does not have this fallback.

This means, that if you use the built-in OpenSSL with python2 it will successfully verify the site if it finds a CA inside the OS X keyring, even if it is not in the cert store provided by requests itself. But if you have compiled python3 against your own OpenSSL it will only use the CAs which are provided by requests itself and not fall back to OS X keyring and thus will fail to verify if the CA is not in requests keyring.

For details about this "feature" of Mac OS X and the problems it introduces see https://hynek.me/articles/apple-openssl-verification-surprises/.

Unfortunately this does not explain why openssl successfully verifies against the default certificates of the requests library, unless there is yet another OpenSSL version involved, i.e. a version used by python3 without the keyring fallback and a recent version on the command line which has the fallback.

Thanks. Do you know any way for me to check which version of openssl each version of python is using? At this point getting further in the debugging process is essential, but I'm hoping for a solution that will tell me how to get python3.4 behaving as expected. — pedmiston, Sep 14 '14 at 20:02
@pedmiston To figure out which version of openssl python is using, do `python -c 'import ssl; print(ssl.OPENSSL_VERSION)'` — Ian Stapleton Cordasco, Sep 15 '14 at 01:10