I’m failing to see how this is addressing the original point raised in this thread. It seems instead to address an unrelated issue which was not presented.
What’s the background?
Our corporate network security team does deep packet inspection of all https traffic. HTTPS traffic is decoded, inspected, and re-encoded at the perimeter. We add the gateway’s signing cert into our ca-certificates chain to allow traffic to flow. It looks like snaps bypasses the local CA chain.
Is there is a way to::
1 inject our cert into the snaps ca chain, or…
2. pass a parameter to snap that would ignore certificate errors.
Thanks!
Kurt
1 Like
I think this is an essential feature for enterprise intranet deployments in general. This goes way beyond the Docker use-case.
Snaps may be deployed in the enterprise onto hosts that are configured to trust a set of internal enterprise-managed CAs – these may be issuing certs for private domains that are essential to enterprise infrastructure – LDAP, email, web APIs, you name it. There are all sorts of reasons for this. HTTPS deep inspection happens at some places, sure, but it can be as simple as wanting to secure intranet hosts that only resolve with intranet DNS names.
If snaps can’t communicate with hosts because they’re only able to see the core snap’s CA roots, it’s a real problem for using snaps in a private enterprise deployment. It’s also a problem if each snap needs to have it’s own CA roots specially configured, because that then complicates management of these roots over time – they’ll eventually expire or need to get rotated, and it should be possible for enterprise IT to easily roll out new roots to all the hosts in a standard fashion per OS.
1 Like
We touch-based on this during the recent sprint.
The general field experience is as follows:
- 1 application - N HTTP(s) destinations;
- each HTTPS destination may be behind its own proxy (potentially with TLS termination) and may have its own TLS termination. Proxies generally implement SSLBump generally as they need to inspect incoming and outgoing traffic and impersonate target websites with dynamically-generated certificates for target subjAltNames;
- TLS-certificates may be signed by an enterprise CA which requires either adding a snap-wide CA certificate or configuring an application to use per-destination CA or CA cert chain;
- applications may not expose options to reconfigure per-destination CA certificates, therefore, even if a snap exposes an option to configure such cert, not every application will be able to use a cert provided in a file.
Subjectively, this could be addressed by providing a way to create overlays for per-snap CA certificate store. This is not about letting a snap to override read-only core snap certs or about modifying global system cert store - we need a way for an admin to configure additional CA certs for a given snap only, which would avoid system-wide modifications.
One good example is a Juju controller. It needs to handle the following HTTPS destinations:
- peer controller HTTPS traffic (self-signed CA + controller certs);
- charm store (public CA-signed cert but enterprise proxies use SSLBump);
- Juju artifact mirrors (streams.canonical.com, SSLBump on proxy servers);
- cloud provider endpoint (e.g. a TLS-terminated MAAS endpoint with enterprise-CA-signed certificates).
https://people.canonical.com/~dima/juju-proxy-design.png
Each destination has its own http transport object and there are no overrides for CA certificates for individual endpoints. If Juju controller was placed into a confined snap there would be a need to provide per-snap CA certificates as default HTTP transport configuration relies on system trusted cert store.
This is just one example - there are others and modifying every application to match packaging would be wrong in my view - it’s a general problem.
1 Like
Just a ping (I was off when this was posted), I’m spending some thinking cycles on this, I’ll try to post something not too far from now.
People are talking about the ability to add certificates to individual snaps, but the issue I encounter is that I need to add a trusted certificate to the snapd base. I cannot install any snap packages from the store because of a corporate environment that intercepts SSL communications in order to inspect them. This causes SSL verification to fail.
We need the ability to configure the overall snapd environment to either consult the OS certificate store and/or be able to add a trusted CA certificate to snapd in order to allow it to communicate out.
In fact, when making snaps, I wouldn’t want these certificates to carry over. If I create a snap that I intend to publish, I don’t want it to have my corporate CA root cert sitting in there.
2 Likes
Hi!
I had a similar problem when testing the example of creating a snap to the httpstat utility.
When testing the example with a URL where the certificate was signed by Go Daddy Secure Certificate Authority - G2.
On my workstation where I have gdig2.pem installed CURL didn’t complain. However when testing in the snap I always encountered the unsafe site error.
I have verified that in folder ‘/snap/core/current/etc/ssl/’ there is no CA for Go Daddy!
As this folder structure is read only I decided to do the following test:
sudo mount --bind /etc/ssl /snap/core/current/etc/ssl
After that the httpstat snap example worked!
I believe this is not a fancy solution but I have not found a better way to add a CA.
Thanks,
Marcelo Módolo
I know this is an evil hack, but isn’t that on which The Internet is built…
So if you need a stable system that has the SSL environment extended, here’s a systemd generator that dynamically bindmounts /etc/ssl (and any other paths you’d like, for good measure):
#!/bin/bash
OUTPUT_DIR="$1"
UNITFILES="$( ls /etc/systemd/system/snap-core{,18}-*.mount )"
BINDS="/etc/ssl /etc/environment" # add more paths as needed
for BIND in ${BINDS}; do
for UNITFILE in ${UNITFILES}; do
if [[ $UNITFILE =~ snap-([-a-z0-9]+)-([0-9]+).mount$ ]]; then
UNIT="${BASH_REMATCH[0]}"
SNAP="${BASH_REMATCH[1]}"
REVISION="${BASH_REMATCH[2]}"
else
echo "Could not parse $UNIT…" 2>&1
exit 1
fi
BINDUNIT="snap-${SNAP}-${REVISION}${BIND//\//-}.service"
BINDTARGET="/snap/${SNAP}/${REVISION}${BIND}"
WANTSPATH="${OUTPUT_DIR}/${UNIT}.wants"
# Couldn't get heredoc to work here…
echo "# Automatically generated by $( basename $0 )
[Unit]
Description=${BIND} bindmount for ${SNAP}-${REVISION}
After=${UNIT}
Requires=${UNIT}
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount --bind ${BIND} ${BINDTARGET}
ExecStop=/bin/umount ${BINDTARGET}" > ${OUTPUT_DIR}/${BINDUNIT}
mkdir -p "${WANTSPATH}"
ln -s "../${BINDUNIT}" "${WANTSPATH}"
done
done
I couldn’t get it to work well with mount units, because those don’t support files being the targets of a mount. I’ll leave it as an exercise to the reader to make this work for paths that don’t exist on the target (NB: you need to (d’uh, it’s a snap, you can’t…).touch the target if bindmounting a single file).
Just adding my feedback that this is becoming a requirement for us as well. We have customers that do deep packet inspection so we need a way to add new trusted signing certificates to the system.
Please explain how I could add this generator to ubuntu.
I read through that doc. Just not clear on the process mainly because I am a Linux noob. For example does the file have an extension? Which directory do i place it in because I dont see the same directory paths that are listed in the doc. I was really just trying to find a way to get a snap use the system certificates but I am not having any luck. I tried the command to bind mount the certificate from /etc/ssl/certs to /snap/etc/ssl/certs and it seems to work but the snap still doesnt seem to get it. Oh well, I will keep searching for an answer.
/run/systemd/system-generators/*
/etc/systemd/system-generators/*
/usr/local/lib/systemd/system-generators/*
/usr/lib/systemd/system-generators/*…
Generators are small executables that live in /usr/lib/systemd/system-generators/ and other directories listed above.
Just place the file in any of the above paths and make it executable.
Im using Ubuntu 18.04 and I dont see a system-generators directory in any of those locations. I do have /run/systemd/generator directory. I tried copying the file there and made it executable. After a reboot the file is gone. I tried creating a system-generator director under /run/systemd and copying the file there but after a reboot the whole directory is gone. I am missing something…
@64bitfury just create one of those dirs 
/run is temporary, that’s why your file created there is gone.
Created directory /etc/systemd/sys-generators and dropped the file in it. Then made the file executable. After reboot I do not see the contents of /etc/ssl/certs in /snap/core/current/etc/ssl/certs
You don’t need to reboot, systemctl daemon-reload is enough to trigger them.
Try running the generator yourself with a path to some custom directory and check its contents.
$ ./gen.sh $PWD/foo
$ tree foo
foo
├── snap-core18-1650-etc-environment.service
├── snap-core18-1650-etc-ssl.service
├── snap-core18-1650.mount.wants
│ ├── snap-core18-1650-etc-environment.service -> ../snap-core18-1650-etc-environment.service
│ └── snap-core18-1650-etc-ssl.service -> ../snap-core18-1650-etc-ssl.service
├── snap-core18-1668-etc-environment.service
├── snap-core18-1668-etc-ssl.service
├── snap-core18-1668.mount.wants
│ ├── snap-core18-1668-etc-environment.service -> ../snap-core18-1668-etc-environment.service
│ └── snap-core18-1668-etc-ssl.service -> ../snap-core18-1668-etc-ssl.service
├── snap-core-8886-etc-environment.service
├── snap-core-8886-etc-ssl.service
├── snap-core-8886.mount.wants
│ ├── snap-core-8886-etc-environment.service -> ../snap-core-8886-etc-environment.service
│ └── snap-core-8886-etc-ssl.service -> ../snap-core-8886-etc-ssl.service
├── snap-core-8890-etc-environment.service
├── snap-core-8890-etc-ssl.service
└── snap-core-8890.mount.wants
├── snap-core-8890-etc-environment.service -> ../snap-core-8890-etc-environment.service
└── snap-core-8890-etc-ssl.service -> ../snap-core-8890-etc-ssl.service
I can’t help you more, sorry, I’m not using this myself, it was just a proof of concept for a workaround.
Thank you for your assistance. Got your script working.
But how? Please share your knowledge.
Ok. systemctl daemon-reload is not enough for Ubuntu 18. Full restart helped.