LD_LIBRARY_PATH in snapped docker


When snap installing docker (or microk8s) and you “docker run” something the LD_LIBRARY_PATH is not set in the spawned container. In the following example MY_VAR is set but LD_LIBRARY_PATH not:

ubuntu@ip-172-31-52-95:~$ sudo docker run  -e "LD_LIBRARY_PATH=foo" -e "MY_VAR=is_set" jupyterhub/jupyterhub:0.8.1 env

The result is not being able to start some containers:

ubuntu@ip-172-31-52-95:~$ sudo docker run  jupyterhub/jupyterhub:0.8.1
/opt/conda/bin/python: error while loading shared libraries: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

This behavior is similar to what setuid is doing when ignoring LD_LIBRARY_PATH for security reasons. I am not sure how this fits into the bigger picture.

What is the right way to tackle this? Can this behavior be configured in the snap?

The issue was first reported here: https://github.com/docker/docker-snap/issues/30 and its effect on running kubeflow on microk8s is tracked here: https://github.com/kubeflow/kubeflow/issues/1367

Thank you

1 Like

Any updates on this?

This is using docker from Arch packages:

maciek@galeon:~ docker run --rm -e 'LD_LIBRARY_PATH=/mnt' alpine env
maciek@galeon:~ docker run  -e "LD_LIBRARY_PATH=foo" -e "MY_VAR=is_set" jupyterhub/jupyterhub:0.8.1 env

And this is using docker from snaps:

maciek@galeon:~ sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' alpine env
maciek@galeon:~ sudo docker run --rm  -e "LD_LIBRARY_PATH=foo" -e "MY_VAR=is_set" jupyterhub/jupyterhub:0.8.1 env

I’m using version 17.06.2-ce (179). Note this was run on Arch, but I don’t see how Ubuntu should be any different here.

Thank you for taking the time to test this on Arch. It was very helpful.

Here are some tests with instances from EC2:

Not affected distributions:

Affected distributions:

The getauxval manual page documents the AT_SECURE flag that is set when a setuid root binary is executed. Apparmor manages that flag and affects the environment of the running process. This means that certain environment variables get unset.

I don’t have any better answers but this explains part of the mechanism why this affects only some distributions and not others.

Also something interesting is that alpine (docker image) inside ubuntu (host) looks fine where as ubuntu inside ubuntu does this environment scrubbing:

> sudo microk8s.docker run --rm -e 'LD_LIBRARY_PATH=foo' alpine env
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
8e3ba11ec2a2: Pull complete 
Digest: sha256:7043076348bf5040220df6ad703798fd8593a0918d06d3ce30c6c93be117e430
Status: Downloaded newer image for alpine:latest
> sudo microk8s.docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu env
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
124c757242f8: Pull complete 
2ebc019eb4e2: Pull complete 
dac0825f7ffb: Pull complete 
82b0bb65d1bf: Pull complete 
ef3b655c7f88: Pull complete 
Digest: sha256:72f832c6184b55569be1cd9043e4a80055d55873417ea792d989441f207dd2c7
Status: Downloaded newer image for ubuntu:latest

As @zyga-snapd suggested there seems to be an unwanted confinement in the apparmor profile of the docker daemon.

Is there a way to request for unconfined execution file access in the apparmor profile in an daemons AppArmor profile. For example, is there a way to change the “/** pix” to “/** pux” in /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-docker?

The long story. Where this request came from?

First install microk8s and stop the docker daemon. We will be starting that daemon manually.

sudo snap install microk8s --edge --classic
# Give it a moment for the dust to settle
sudo systemctl stop snap.microk8s.daemon-docker.service

Let’s start docker without the confinement imposed by the snap.

sudo /snap/microk8s/current/usr/bin/dockerd

Open a second terminal and try this:

sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

You will see the LD_LIBRARY_PATH is set as expected. Stop the docker daemon and let’s try to start it from a shell we get from snap run.

snap run --shell microk8s.docker
# Inside this shell start dockerd
sudo /snap/microk8s/current/usr/bin/dockerd

Again open a second terminal and run:

sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

LD_LIBRARY_PATH is not set this time. The confinement we impose in the snap has some effect.

Lets go a bit deeper and try to start the daemon with the apparmor profile the snap is using.

Stop the docker daemon you have running. Exit the snap shell. Start a new daemon with the apparmor profile of daemon-docker (https://github.com/snapcore/snapd/wiki/snap-confine-Overview):

aa-exec -p snap.microk8s.daemon-docker /snap/microk8s/current/usr/bin/dockerd

Trying the following still fails:

sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

Go and edit /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-docker. Change the “/** pix” to “/** pux” in the profile. Save it and reload it with:

sudo /etc/init.d/apparmor reload

Start docker with the updated profile:

aa-exec -p snap.microk8s.daemon-docker /snap/microk8s/current/usr/bin/dockerd

And the following works:

sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

As described in https://manpages.debian.org/jessie/apparmor/apparmor.d.5.en.html we request unconfined execution file access for dockerd instead of inherit.

Can you try using SNAP_SAVED_LD_LIBRARY_PATH=… This will be used as LD_LIBRARY_PATH by the snap execution chain (snap run -> snap-confine -> snap-exec -> command)

Where should I use the “SNAP_SAVED_LD_LIBRARY_PATH”? Should I expose this variable in the docker daemon wrapper?

The variable gets picked up by the execution chain. Try setting it in the environment of the snap for that command:

       command: something

Now having said that, that’s just my guesstimate. You can just set it from command line without rebuilding the snap.

No luck. The LD_LIBRARY_PATH still gets scrubbed when new containers are spawned.

Can you see the value in your wrapper script? I wonder if it gets scrubbed twice?

The LD_LIBRARY_PATH seems scrubbed but the SNAP_SAVED_LD_LIBRARY_PATH not.

ubuntu@ip-172-31-9-63:~$ sudo snap run --shell microk8s.daemon-docker
root@ip-172-31-9-63:~# echo $LD_LIBRARY_PATH

root@ip-172-31-9-63:~# echo $SNAP_SAVED_LD_LIBRARY_PATH

snapenv filters out any such variables as are in the environment already,
unfortunately for this case.


Yeah, I was just reading snapenv.go's envMap

@mwhudson, @zyga-snapd, just to clarify. I do not need LD_LIBRARY_PATH in the environment where dockerd is. I need it when docker containers are spawned.

So we have the docker containers inside the environment setup by the snapped dockerd inside the environment created by the snap. We have no problem the snap scrubbing LD_LIBRARY_PATH, we care that the dockerd scrubs LD_LIBRARY_PATH when spawning its own containers.

Are you considering the case the AppArmor profile of the docker daemon being inherited by the docker containers? (the “ix” case in the profile)

Ok, what is happening here is that dockerd is performing an aa_change_onexec() to docker’s internal docker-default apparmor template. The classic template has this rule:


and dockerd runs under the profile ‘snap.microk8s.daemon-dockerd’. When it performs the aa_change_onexec(), the change to the docker-default is allowed (due to the above rule), but because dockerd is running under the ‘snap.microk8s.daemon-dockerd’ profile and not ‘unconfined’, then the kernel’s ‘unsafe_exec’ routine is triggered which causes LD_LIBRARY_PATH to be scrubbed.

When you changed the ‘/** pix,’ to ‘/** pux,’ you made it so that everything dockerd ran (it will even reexec itself), it ran unconfined such that by the time the aa_change_onexec() was performed, the process was unconfined and there was no environment scrubbing. Eg, consider the following denial (obtained through strategic changing of the profile to non-complain and additional audit rules):

audit(1535400520.484:17939): apparmor="DENIED" operation="exec" info="change_profile onexec"
 error=-13 profile="snap.microk8s.daemon-docker" name="/usr/bin/env" pid=15562
 comm="runc:[2:INIT]" target="docker-default"

To make this work, the above change_profile rule can be changed to:

change_profile unsafe /**,

which will prevent the environment scrubbing from occurring. Eg:

$ grep -E '(pix|change_profile)' /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-docker
  /** pix,
  change_profile unsafe /**,

$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-docker && microk8s.docker run --rm -i -t -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

I’ll create a PR against the default template and docker_support.go for this.

FYI, https://github.com/snapcore/snapd/pull/5739