"SSL: CERTIFICATE_VERIFY_FAILED" error from within a Python snap

The snap I maintain connects to Launchpad to create issues and upload log information.

Since yesterday (at least), the connection to Launchpad doesn’t work anymore. The program returns the following error:

SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed (_ssl.c:852)

I thought it might be a problem with old certificates in the snap (the version in the store is a few months old), so I rebuilt the snap locally to try, but I still get the same error.

This happens on different devices, with different network connections.

What can I do to solve the problem?

For more info:

If you only started seeing failures some time yesterday, I wonder if this could be related to the a signature on the LetsEncrypt root CA expiring?

https://twitter.com/letsencrypt/status/1443621997288767491

(api.launchpad.net uses a LetsEncrypt certificate).

OpenSSL in Ubuntu 18.04 should be fine, so it might be worth checking if a rebuild of your app works any better.

Depending on how much time you have, it might be worth seeing whether upgrading to core20 makes a difference too.

It’s also possible that some configuration changes are needed on the Launchpad side. It doesn’t seem to be serving up a full certificate chain:

$ openssl s_client -connect api.launchpad.net:443
CONNECTED(00000003)
...
---
Certificate chain
 0 s:CN = launchpad.net
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
...

That is, it’s serving up its own certificate and the intermediary, but not the root certificate. I wouldn’t have thought this would usually be a problem, since the root cert would need to be on the system already for it to be trusted. But perhaps it makes a difference here?

I tried rebuilding locally (using base18), installed the snap and tried again, but it failed with the same error.

I managed to find a way to reproduce the problem easily from within the snap using python3:

$ snap run --shell qabro
$ python3 # Run python interpreter from within the snap

Then:

>>> from launchpadlib.launchpad import Launchpad
>>> launchpad = Launchpad.login_anonymously('just testing', 'staging', version='devel')
>>> # For some reason, login_anonymously() works, but...
>>> launchpad = Launchpad.login_with('just testing', 'staging')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/snap/qabro/x1/lib/python3.6/site-packages/launchpadlib/launchpad.py", line 568, in login_with
    credential_save_failed, version)
  File "/snap/qabro/x1/lib/python3.6/site-packages/launchpadlib/launchpad.py", line 375, in _authorize_token_and_login
    credentials = authorization_engine(credentials, credential_store)
  File "/snap/qabro/x1/lib/python3.6/site-packages/launchpadlib/credentials.py", line 598, in __call__
    request_token_string = self.get_request_token(credentials)
  File "/snap/qabro/x1/lib/python3.6/site-packages/launchpadlib/credentials.py", line 615, in get_request_token
    token_format=Credentials.DICT_TOKEN_FORMAT)
  File "/snap/qabro/x1/lib/python3.6/site-packages/launchpadlib/credentials.py", line 192, in get_request_token
    response, content = _http_post(url, headers, params)
  File "/snap/qabro/x1/lib/python3.6/site-packages/launchpadlib/credentials.py", line 112, in _http_post
    url, method='POST', headers=headers, body=urlencode(params))
  File "/snap/qabro/x1/lib/python3.6/site-packages/httplib2/__init__.py", line 1709, in request
    conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
  File "/snap/qabro/x1/lib/python3.6/site-packages/httplib2/__init__.py", line 1424, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/snap/qabro/x1/lib/python3.6/site-packages/httplib2/__init__.py", line 1346, in _conn_request
    conn.connect()
  File "/snap/qabro/x1/lib/python3.6/site-packages/httplib2/__init__.py", line 1138, in connect
    self.sock = self._context.wrap_socket(sock, server_hostname=self.host)
  File "/snap/qabro/x1/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/snap/qabro/x1/usr/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/snap/qabro/x1/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/snap/qabro/x1/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
>>> # ... login_with() does not.

It might be helpful to try using the openssl s_client command from within your snap’s sandbox too: that should show whether openssl thinks the certificate is valid independent of Python.

It also seems that api.launchpad.net and api.staging.launchpad.net are returning different certificate chains:

Certificate chain
 0 s:CN = launchpad.net
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

vs.

Certificate chain
 0 s:CN = staging.launchpad.net
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

The staging server seems to still be using the chain containing the expired signature, so it might not be representative of whether production works.

Thanks for your feedback!

After discussing with a colleague who was having a similar issue, he suggested that I install/upgrade the python3-certifi package. It was not part of my snapcraft.yaml, so I added it in my main part:

diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml
index ef2945b..79bbd6f 100644
--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@@ -41,6 +41,7 @@ parts:
       - setuptools-rust
       - wheel
     stage-packages:
+      - python3-certifi
       - dbus-x11
       - dmidecode
       - pciutils

and… it’s working now!

Not exactly sure what’s going on here, though.

Upstream httplib2 doesn’t use the system’s certificate trust roots, but will use certifi’s if it’s installed. https://github.com/httplib2/httplib2/issues/203 exists upstream for the missing Let’s Encrypt root certificate.

1 Like