LD_LIBRARY_PATH in snapped docker

snap
1

Hi,

When snap installing docker (or microk8s) and you “docker run” something the LD_LIBRARY_PATH is not set in the spawned container. In the following example MY_VAR is set but LD_LIBRARY_PATH not:

ubuntu@ip-172-31-52-95:~$ sudo docker run  -e "LD_LIBRARY_PATH=foo" -e "MY_VAR=is_set" jupyterhub/jupyterhub:0.8.1 env
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=3ecen9ffc0c7d
MY_VAR=is_set
DEBIAN_FRONTEND=noninteractive
LANG=C.UTF-8
HOME=/root

The result is not being able to start some containers:

ubuntu@ip-172-31-52-95:~$ sudo docker run  jupyterhub/jupyterhub:0.8.1
/opt/conda/bin/python: error while loading shared libraries: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

This behavior is similar to what setuid is doing when ignoring LD_LIBRARY_PATH for security reasons. I am not sure how this fits into the bigger picture.

What is the right way to tackle this? Can this behavior be configured in the snap?

The issue was first reported here: https://github.com/docker/docker-snap/issues/30 and its effect on running kubeflow on microk8s is tracked here: https://github.com/kubeflow/kubeflow/issues/1367

Thank you

1 Like
2

Any updates on this?

3

This is using docker from Arch packages:

maciek@galeon:~ docker run --rm -e 'LD_LIBRARY_PATH=/mnt' alpine env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=269a850f33f0
LD_LIBRARY_PATH=/mnt
HOME=/root
maciek@galeon:~ docker run  -e "LD_LIBRARY_PATH=foo" -e "MY_VAR=is_set" jupyterhub/jupyterhub:0.8.1 env
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=3bc2e65f10d9
LD_LIBRARY_PATH=foo
MY_VAR=is_set

And this is using docker from snaps:

maciek@galeon:~ sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' alpine env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=56bcceff2600
LD_LIBRARY_PATH=foo
HOME=/root
maciek@galeon:~ sudo docker run --rm  -e "LD_LIBRARY_PATH=foo" -e "MY_VAR=is_set" jupyterhub/jupyterhub:0.8.1 env
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=882b0196d6d0
LD_LIBRARY_PATH=foo
MY_VAR=is_set
DEBIAN_FRONTEND=noninteractive
LANG=C.UTF-8
HOME=/root

I’m using version 17.06.2-ce (179). Note this was run on Arch, but I don’t see how Ubuntu should be any different here.

4

Thank you for taking the time to test this on Arch. It was very helpful.

Here are some tests with instances from EC2:

Not affected distributions:

Affected distributions:

5

The getauxval manual page documents the AT_SECURE flag that is set when a setuid root binary is executed. Apparmor manages that flag and affects the environment of the running process. This means that certain environment variables get unset.

I don’t have any better answers but this explains part of the mechanism why this affects only some distributions and not others.

6

Also something interesting is that alpine (docker image) inside ubuntu (host) looks fine where as ubuntu inside ubuntu does this environment scrubbing:

> sudo microk8s.docker run --rm -e 'LD_LIBRARY_PATH=foo' alpine env
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
8e3ba11ec2a2: Pull complete 
Digest: sha256:7043076348bf5040220df6ad703798fd8593a0918d06d3ce30c6c93be117e430
Status: Downloaded newer image for alpine:latest
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=7774af3231d2
LD_LIBRARY_PATH=foo
HOME=/root
> sudo microk8s.docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu env
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
124c757242f8: Pull complete 
2ebc019eb4e2: Pull complete 
dac0825f7ffb: Pull complete 
82b0bb65d1bf: Pull complete 
ef3b655c7f88: Pull complete 
Digest: sha256:72f832c6184b55569be1cd9043e4a80055d55873417ea792d989441f207dd2c7
Status: Downloaded newer image for ubuntu:latest
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=3dbae5298178
HOME=/root
7

As @zyga-snapd suggested there seems to be an unwanted confinement in the apparmor profile of the docker daemon.

Is there a way to request for unconfined execution file access in the apparmor profile in an daemons AppArmor profile. For example, is there a way to change the “/** pix” to “/** pux” in /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-docker?

The long story. Where this request came from?

First install microk8s and stop the docker daemon. We will be starting that daemon manually.

sudo snap install microk8s --edge --classic
# Give it a moment for the dust to settle
sudo systemctl stop snap.microk8s.daemon-docker.service

Let’s start docker without the confinement imposed by the snap.

sudo /snap/microk8s/current/usr/bin/dockerd

Open a second terminal and try this:

sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

You will see the LD_LIBRARY_PATH is set as expected. Stop the docker daemon and let’s try to start it from a shell we get from snap run.

snap run --shell microk8s.docker
# Inside this shell start dockerd
sudo /snap/microk8s/current/usr/bin/dockerd

Again open a second terminal and run:

sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

LD_LIBRARY_PATH is not set this time. The confinement we impose in the snap has some effect.

Lets go a bit deeper and try to start the daemon with the apparmor profile the snap is using.

Stop the docker daemon you have running. Exit the snap shell. Start a new daemon with the apparmor profile of daemon-docker (https://github.com/snapcore/snapd/wiki/snap-confine-Overview):

aa-exec -p snap.microk8s.daemon-docker /snap/microk8s/current/usr/bin/dockerd

Trying the following still fails:

sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

Go and edit /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-docker. Change the “/** pix” to “/** pux” in the profile. Save it and reload it with:

sudo /etc/init.d/apparmor reload

Start docker with the updated profile:

aa-exec -p snap.microk8s.daemon-docker /snap/microk8s/current/usr/bin/dockerd

And the following works:

sudo docker run --rm -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env

As described in https://manpages.debian.org/jessie/apparmor/apparmor.d.5.en.html we request unconfined execution file access for dockerd instead of inherit.

8

Can you try using SNAP_SAVED_LD_LIBRARY_PATH=… This will be used as LD_LIBRARY_PATH by the snap execution chain (snap run -> snap-confine -> snap-exec -> command)

9

Where should I use the “SNAP_SAVED_LD_LIBRARY_PATH”? Should I expose this variable in the docker daemon wrapper?

10

The variable gets picked up by the execution chain. Try setting it in the environment of the snap for that command:

apps:
    foo:
       command: something
       environment:
           SNAP_SAVED_LD_LIBRARY_PATH: ...

Now having said that, that’s just my guesstimate. You can just set it from command line without rebuilding the snap.

11

No luck. The LD_LIBRARY_PATH still gets scrubbed when new containers are spawned.

12

Can you see the value in your wrapper script? I wonder if it gets scrubbed twice?

13

The LD_LIBRARY_PATH seems scrubbed but the SNAP_SAVED_LD_LIBRARY_PATH not.

ubuntu@ip-172-31-9-63:~$ sudo snap run --shell microk8s.daemon-docker
root@ip-172-31-9-63:~# echo $LD_LIBRARY_PATH

root@ip-172-31-9-63:~# echo $SNAP_SAVED_LD_LIBRARY_PATH
/snap/microk8s/x1/lib:/snap/microk8s/x1/usr/lib:/snap/microk8s/x1/lib/x86_64-linux-gnu:/snap/microk8s/x1/usr/lib/x86_64-linux-gnu
14

snapenv filters out any such variables as are in the environment already,
unfortunately for this case.

Cheers,
mwh

15

Yeah, I was just reading snapenv.go's envMap

16

@mwhudson, @zyga-snapd, just to clarify. I do not need LD_LIBRARY_PATH in the environment where dockerd is. I need it when docker containers are spawned.

So we have the docker containers inside the environment setup by the snapped dockerd inside the environment created by the snap. We have no problem the snap scrubbing LD_LIBRARY_PATH, we care that the dockerd scrubs LD_LIBRARY_PATH when spawning its own containers.

Are you considering the case the AppArmor profile of the docker daemon being inherited by the docker containers? (the “ix” case in the profile)

17

Ok, what is happening here is that dockerd is performing an aa_change_onexec() to docker’s internal docker-default apparmor template. The classic template has this rule:

  change_profile,

and dockerd runs under the profile ‘snap.microk8s.daemon-dockerd’. When it performs the aa_change_onexec(), the change to the docker-default is allowed (due to the above rule), but because dockerd is running under the ‘snap.microk8s.daemon-dockerd’ profile and not ‘unconfined’, then the kernel’s ‘unsafe_exec’ routine is triggered which causes LD_LIBRARY_PATH to be scrubbed.

When you changed the ‘/** pix,’ to ‘/** pux,’ you made it so that everything dockerd ran (it will even reexec itself), it ran unconfined such that by the time the aa_change_onexec() was performed, the process was unconfined and there was no environment scrubbing. Eg, consider the following denial (obtained through strategic changing of the profile to non-complain and additional audit rules):

audit(1535400520.484:17939): apparmor="DENIED" operation="exec" info="change_profile onexec"
 error=-13 profile="snap.microk8s.daemon-docker" name="/usr/bin/env" pid=15562
 comm="runc:[2:INIT]" target="docker-default"

To make this work, the above change_profile rule can be changed to:

change_profile unsafe /**,

which will prevent the environment scrubbing from occurring. Eg:

$ grep -E '(pix|change_profile)' /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-docker
  /** pix,
  #change_profile,
  change_profile unsafe /**,

$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-docker && microk8s.docker run --rm -i -t -e 'LD_LIBRARY_PATH=foo' ubuntu:14.04 env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=ff3507633ae5
TERM=xterm
LD_LIBRARY_PATH=foo
HOME=/root

I’ll create a PR against the default template and docker_support.go for this.

18

FYI, https://github.com/snapcore/snapd/pull/5739