Extending system certificates

It would be desirable to allow adding system-wide certificates to an Ubuntu Core system, one example is the way some organisations use Docker with self-signed certs. To enable this we would need to make /etc/ssl/certs/ca-certificates.crt writeable or make it a symlink to something writable (e.g. /etc/writeable). We also need a directory where new certs can be added (probably /var/lib/snapd/certs). Certs could be added on snap installation and removed on snap removal, either by themselves (some kind of hook i.e. configure) or probably the better option, by snapd itself. After a new cert is added we need to call update-ca-certificates. We could protect access to this functionality either via an assertion or through a restricted interface.

What do you think?

1 Like

It sounds like docker needs a way to see the certificate, not the entire system, it’s a bit unclear also why that can’t be achieved with a snap change.

At least conceptually as we control other things each snap might have/should/could have a different view of what are the “system” certificates.

1 Like

I think it would be fine if a system administrator could add arbitrary certificate to the system. I agree with @pedronis about how this should be handled for docker though. Not sure how to achieve this technically.

I think docker is just an example. In general it does make sense to provide a way to extend the supported certificates.

2 Likes

If I remember correct @tvoss has a change for docker already which allows adding additional certificates.

1 Like

it’s not a good example, having a way to add global certificates to make single snaps happy doesn’t seem very secure, because that’s done on classic it’s probably not a good reason alone

2 Likes

Yup, docker already supports the notion of a certificate pool/dir. I’m exposing the respective config variable via a command-line option such that dockerd can be pointed to the correct directory.

+1 for not abusing the system-wide certificate store.

Exactly, Docker is just one example, there are others. We may have worked around it in the Docker case but would this approach work for all software where a global cert needs adding?

I think this is a core configuration thing. We should allow people to control the SSL of a device they own.

i think this needs three bits:

  • something like a system service that imports certificates from a pre-defined dir (which the service watches with inotify for changes) should be shipped in the core snap…
  • that directory only snapd should be able to write to …
  • an interface that allows snaps to trigger an import of a shipped cert into that dir is provided by snapd

This is just one bad example which shouldn’t be used to justify having system-wide certificates changed. If people want to host their custom Docker hub, they should do so via a Docker setting itself, instead of compromising the entire system trust chain with what will often be locally generated certificate managed with less care than system root CAs.

Someday we’ll most likely need that system-wide CA setting indeed, and by then it’d be great to have a first-class solution that encourages not giving away that fundamental safety net so easily.

I proposed https://github.com/docker/docker/pull/32376 upstream. Currently under review.

We landed https://github.com/snapcore/snapd/pull/3271 now into snapd which will change snapd to always take the CA certificates from the core snap.

@jdstrand commented on the PR:

This PR is a 6 of one half dozen of the other type of thing. This PR means that now locally installed certificates on classic won’t work and there is no mechanism for extending the core snap’s certificates. Debian and Ubuntu have ‘update-ca-certificates’ (man 8 update-ca-certificates) which can look at /usr/local/share/ca-certificates that might be used to help address this, but I don’t know what mechanisms are available in other distros. I suspect more people are affected by this PR not being applied than those that will be when it is applied, but I predict bug reports.

This is actually falling into the same category. Before we relied on the base system to provide the chain of certificates all snaps trust but now we delegate this to the core snap. @jdstrand is right that this might cause other bugs but actually it should fix snaps which are currently build with Ubuntu in mind on all other distributions. Mid term we need to look into a possible solution how we can marry these with the certificates from the host system and not marry them at all and require the user to install any additional certificate into the snap system. Any thoughts on this?

I’m failing to see how this is addressing the original point raised in this thread. It seems instead to address an unrelated issue which was not presented.

What’s the background?

Our corporate network security team does deep packet inspection of all https traffic. HTTPS traffic is decoded, inspected, and re-encoded at the perimeter. We add the gateway’s signing cert into our ca-certificates chain to allow traffic to flow. It looks like snaps bypasses the local CA chain.

Is there is a way to::
1 inject our cert into the snaps ca chain, or…
2. pass a parameter to snap that would ignore certificate errors.

Thanks!
Kurt

1 Like

I think this is an essential feature for enterprise intranet deployments in general. This goes way beyond the Docker use-case.

Snaps may be deployed in the enterprise onto hosts that are configured to trust a set of internal enterprise-managed CAs – these may be issuing certs for private domains that are essential to enterprise infrastructure – LDAP, email, web APIs, you name it. There are all sorts of reasons for this. HTTPS deep inspection happens at some places, sure, but it can be as simple as wanting to secure intranet hosts that only resolve with intranet DNS names.

If snaps can’t communicate with hosts because they’re only able to see the core snap’s CA roots, it’s a real problem for using snaps in a private enterprise deployment. It’s also a problem if each snap needs to have it’s own CA roots specially configured, because that then complicates management of these roots over time – they’ll eventually expire or need to get rotated, and it should be possible for enterprise IT to easily roll out new roots to all the hosts in a standard fashion per OS.

1 Like

@niemeyer
@jdstrand

We touch-based on this during the recent sprint.

The general field experience is as follows:

  • 1 application - N HTTP(s) destinations;
  • each HTTPS destination may be behind its own proxy (potentially with TLS termination) and may have its own TLS termination. Proxies generally implement SSLBump generally as they need to inspect incoming and outgoing traffic and impersonate target websites with dynamically-generated certificates for target subjAltNames;
  • TLS-certificates may be signed by an enterprise CA which requires either adding a snap-wide CA certificate or configuring an application to use per-destination CA or CA cert chain;
  • applications may not expose options to reconfigure per-destination CA certificates, therefore, even if a snap exposes an option to configure such cert, not every application will be able to use a cert provided in a file.

Subjectively, this could be addressed by providing a way to create overlays for per-snap CA certificate store. This is not about letting a snap to override read-only core snap certs or about modifying global system cert store - we need a way for an admin to configure additional CA certs for a given snap only, which would avoid system-wide modifications.

One good example is a Juju controller. It needs to handle the following HTTPS destinations:

  1. peer controller HTTPS traffic (self-signed CA + controller certs);
  2. charm store (public CA-signed cert but enterprise proxies use SSLBump);
  3. Juju artifact mirrors (streams.canonical.com, SSLBump on proxy servers);
  4. cloud provider endpoint (e.g. a TLS-terminated MAAS endpoint with enterprise-CA-signed certificates).
    https://people.canonical.com/~dima/juju-proxy-design.png

Each destination has its own http transport object and there are no overrides for CA certificates for individual endpoints. If Juju controller was placed into a confined snap there would be a need to provide per-snap CA certificates as default HTTP transport configuration relies on system trusted cert store.

This is just one example - there are others and modifying every application to match packaging would be wrong in my view - it’s a general problem.

1 Like

Just a ping (I was off when this was posted), I’m spending some thinking cycles on this, I’ll try to post something not too far from now.

People are talking about the ability to add certificates to individual snaps, but the issue I encounter is that I need to add a trusted certificate to the snapd base. I cannot install any snap packages from the store because of a corporate environment that intercepts SSL communications in order to inspect them. This causes SSL verification to fail.

We need the ability to configure the overall snapd environment to either consult the OS certificate store and/or be able to add a trusted CA certificate to snapd in order to allow it to communicate out.

In fact, when making snaps, I wouldn’t want these certificates to carry over. If I create a snap that I intend to publish, I don’t want it to have my corporate CA root cert sitting in there.

2 Likes