Call for testing of the docker snap

Hi again,

Apologies for not getting back to this sooner, but a couple of things:

  • the AppArmor profile you modified is generated by snapd so it should not really ever be modified manually except during development perhaps
  • the correct place to get modifications made to the AppArmor profile generated by snapd is not in the docker snap itself but rather in the docker-support interface which is in the snapd project: https://github.com/snapcore/snapd/blob/master/interfaces/builtin/docker_support.go
  • after modifications made to the snapd interface have been merged to master, they will not usually be released right away to stable (though they will likely land in edge within a day or so) and instead will follow the snapd release cycle, but if there’s something critically broken or missing then we can try to get it released to the stable channel sooner
  • the necessary bits in docker-support to accommodate docker 18.09.03-ce are now in the stable channel, so we should push up an update soon

Additionally, you can submit MPs (merge proposals - launchpad parlance for pull requests) against the docker snap on launchpad by:

  1. Create a launchpad/ubuntu SSO account
  2. Clone the repository from git+ssh://$YOUR_LP_USER@git.launchpad.net/~docker/+git/snap
  3. Create a local git branch with the changes you want to see
  4. Push your branch to the launchpad with the remote url $YOUR_LP_USER@git.launchpad.net:~$YOUR_LP_USER/+git/docker-snap - note that you don’t need to do anything to create the repo in LP beforehand as launchpad will create the repo for you when you push it.
  5. Go to code.launchpad.net/~$YOUR_LP_USER/+git/docker-snap/ and click on the branch you want to propose a merge for
  6. Click “Propose for merging”
  7. Under “Target repository” choose Other: and fill in lp:~docker/+git/snap and the target branch as master (the snap builds from the master branch)
  8. Fill in the description of the change and click propose merge
  9. We (probably me but possibly someone else in the future) will review the MP and do our best to get it integrated

Lastly, as you may have noticed snapd 2.38 has landed in stable now so we should be pushing out a new update to docker version 18.09.03-ce on the beta channel soon

2 Likes

I tried to email the canonical team about this but their “Contact Cononical” button was broken (I opened an issue about it here https://github.com/canonical-web-and-design/snapcraft.io/issues/1846)

I’m not sure where the right place to raise this issue is, but I think this snap should be removed from snapcraft until it installs in a way that makes it work fully. This disclaimer on the snap:

This build requires all files that Docker uses, such as dockerfiles, to be in $HOME.

Means that this install doesn’t fully work. You can call that a limitation, but I think that too many users won’t the ramifications of such a limitation until it’s too late. For example, here is an issue caused in a completely different system which integrates with/relies upon Docker: https://github.com/hashicorp/nomad/issues/5360#issuecomment-466990744. It’s extremely difficult to discern that the snap-installation caused the issue. I am very fortunate that someone else has already figured it out.

I have been working on restoring a coworker’s machine to working order after she installed docker via snap without understanding why that was a bad idea. Uninstalling Docker via snap and reinstalling it the recommended way made Nomad start working again, but only to a limited degree because snap left the following apparmor profiles active on the computer despite the docker snap being removed (verified via snap list):
snap.docker.compose
snap.docker.docker
snap.docker.dockerd
snap.docker.help
snap.docker.hook.install
snap.docker.hook.post-refresh
snap.docker.machine

I’m not very familiar with apparmor but now I get to figure out how to remove the damage snap did and make everything right again so that my fresh install of Docker can work like a fresh install. All because my coworker thought she found an easier way to install Docker. While this is an issue which could be fixed, it seems like a better fix is to prevent people from needing to uninstall a broken (or limited?) Docker so they can working (or full featured?) version.

I don’t think this snap is doing anyone any favors and would like to see it removed, at least until it is fixed so that it installs in a way that doesn’t have limitations that break its integration with other projects that rely on Docker.

Hi @kurtwheeler

I understand your frustration with not being able to access files outside of $HOME when using docker, but doing so is a feature of snaps in general, protecting snaps from polluting and modifying the rest of the system. If you don’t wish to use the docker snap anymore, you can always uninstall it with:

$ sudo snap remove docker

While the apparmor profiles may still be active in the kernel (which may or may not be a snapd bug, not sure if we should remove the apparmor profiles from the kernel after removing a snap), you can always completely remove those apparmor profiles by rebooting your machine after removing the snap as the profiles on disk are removed immediately on snap removal.

Lastly, so we can understand your use case and potentially improve the snap in the future, can you provide specific examples of why docker needs to access files outside of a user’s $HOME?

Thanks,
Ian

Hi @ijohnson,

Thanks for getting back to me so quickly. I actually did remove the docker snap with that exact command, however it wasn’t obvious to me that a reboot would be necessary for it to take full effect. Perhaps it could issue a warning after removing a snap that in order for a non-snap install to work the system needs to be rebooted? Or perhaps those profiles should be removed immediately, I’m not entirely sure. Either way I’ve spent a good portion of today trying to figure out how to fix the AppArmor configuration for my coworker so she can resume using docker via the standard installation method, so I’m not sure the current uninstall workflow is ideal.

So I’m not actually sure why Nomad requires Docker to access files outside of a user’s $HOME… The only reason I knew that this was the issue causing Nomad to not working on her machine is because I found this issue: https://github.com/hashicorp/nomad/issues/5360.

If I had to guess I would say it’s because Nomad operates outside the $HOME directory itself and needs Docker to be able to access files it creates. Perhaps if I also installed Nomad using snap this wouldn’t have been an issue? I do see that there is a Nomad snap: https://snapcraft.io/nomad, but it’s been over a year since it was updated and I would be very surprised if it could work well with the same limitation as it needs a lot of permissions and control over the machine it is running on. (Which I think makes a lot of sense for an orchestration tool). I do know that it’s not possible to run Nomad inside of a Docker container for these reasons.

Sorry I can’t be much more help, but I don’t know much about this domain. I hadn’t even heard of AppArmor until it kept Docker from working for me.

Thanks again for your quick reply,

  • Kurt

This is by design since snap process are not killed as part or removal and we don’t want them to go unconfined.

Is not killing snap processes also by design? It’s not clear to me why that would be desirable to not kill all processes associated with a snap when it’s removed?

I also realize that properly killing all processes in a snap requires some work done as part of [WIP] Refresh App Awareness

It could be made to and if it did, then once we were sure that all the processes were killed, snapd could unload the profiles. I believe this was always the intent, but it doesn’t do it yet.

1 Like

A workaround for this bug on Ubuntu Core 16 is to change the overlay2 storage-driver back to aufs:

In short terms

$ sudo sed -i ‘s/overlay2/aufs/’ /var/snap/docker/current/config/daemon.json
$ sudo snap restart docker

done

@See https://git.launchpad.net/~docker/+git/snap/commit/?h=bugfix/change-aufs-overlay2

Detailed

With storage-driver overlay2

$ sudo docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete 
Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:109: jailing process inside rootfs caused \\\"permission denied\\\"\"": unknown.

$ sudo su
root$ journalctl --no-pager -e -k | grep apparmor | grep -v kmod | grep snap.docker.dockerd

Apr 25 21:50:47 localhost.localdomain kernel: audit: type=1400 audit(1556229047.116:15): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.docker.dockerd" pid=1374 comm="apparmor_parser"
Apr 25 21:50:52 localhost.localdomain kernel: audit: type=1400 audit(1556229052.260:46): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.docker.dockerd" pid=1547 comm="apparmor_parser"
Apr 25 21:50:57 localhost.localdomain kernel: audit: type=1400 audit(1556229057.708:88): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.docker.dockerd" pid=1771 comm="apparmor_parser"
Apr 25 21:51:01 localhost.localdomain kernel: audit: type=1400 audit(1556229061.208:97): apparmor="STATUS" operation="profile_load" profile="snap.docker.dockerd" name="docker-default" pid=1856 comm="apparmor_parser"
Apr 25 21:55:23 localhost.localdomain kernel: audit: type=1400 audit(1556229323.728:103): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/6cba9d1d59f62094649efe897713fade57986b030ed22a80210053af583c49fc/diff/" pid=2110 comm="runc:[2:INIT]" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Change the docker storage-driver overlay2 to aufs

$ sudo sed -i ‘s/overlay2/aufs/’ /var/snap/docker/current/config/daemon.json
$ sudo snap restart docker

With storage-driver aufs

$ sudo docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete 
Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/
1 Like

Hi, I would like to use the docker snap. However, I don’t want to put everything in my home folder. (Since that is on a NVME SSD)
So I thought I could just make a symlink in my home to point to somewhere else, but I guess the confinement is too smart for that? Because it does not seem to work.
If you could confirm this, then I shall need to move to normal docker again.

It should be smart enough to be able to use a bind-mount instead of a symlink (that usually works for other snaps, but docker might be special here) :wink:

@ogra is correct, you can use a bind-mount of your data from outside of the home folder to somewhere in your home folder

1 Like

I fear I don’t really understand. :slight_smile: But I also see that I might not have explained my situation well enough.

So I have the docker snap running. Then I want to run gitlab in a docker container. Normally I use volumes and specify something like this: /opt/gitlab/config:/etc/gitlab To mount the directory directory of the host in the container. That does not work with the snap, since it can only access things in ~/.

If I do /home/user/gitlab/config:/etc/gitlab all works as expected. However, if /home/user/gitlab is a symlink to say /opt/gitlab it does not work anymore.

I think you are referring to adding a bind-mount to the snap yaml, but I would rather not build my own. But then again, I just might not understand what you mean. :slight_smile:

For your example of /opt/gitlab/config, you would (as root) perform a bind mount that gives /home/user/gitlab/config a view into /opt/gitlab/config, that is to say run:

$ sudo mount --bind /opt/gitlab/config /home/user/gitlab/config

Note this will only last for your current session, to perform this at every boot, you can add an entry to your /etc/fstab like this:

/opt/gitlab/config /home/user/gitlab/config none defaults,bind 0 0

Note that the above was not tested so you may need to adjust it slightly, etc.

Ah, thanks! I only knew about bind mounts in the context of containers. :slight_smile:

@ijohnson why lxd snap has not this limitation? We can mount disk from everywhere with it.

Using home or media to access things does not mean this is more secure: do you know what files are in media or home? NO. Only ops know it.

You should not try to replace ops job.

we do know that these (/home and /media/$USER/) are typically the only places a user can access without privilege escalation in a default install, so snapd gained interfaces to allow acces to these dirs to be able to access user data.

I think i missed something.

Lxd snap, which has some privileges, can create unprivileged containers which can access anything on the host filesystem, if you mount it on the container.

What prevents docker snap to do the same?

LXD has a special interface specifically for its use called lxd-support. This is likely providing different rules than the docker snap is gaining.

1 Like

ok but why 2 container providers , packaged as snaps by canonical, doing globally the same thing, don’t use the same tips to create containers as simply as possible ?

why one is complicated and one is not, since canonical is buidling them 2?

Can’t get the docker snap to work. After reading this whole thread and trying various things, still no joy. I’m running on btrfs for my filesystem and have tried “overlay2” and “btrfs” storage drivers as well as the default.

Seem to get the furthest with the --edge channel for docker and the overlay2 driver - I can get a sensible response from docker info, but docker run hello-world fails on all drivers.

All throw differing errors.

⚡ snap info core
name:      core
summary:   snapd runtime environment
publisher: Canonical✓
contact:   snaps@canonical.com
license:   unset
description: |
  The core runtime environment for snapd
type:         core
snap-id:      99T7MUlRhtI3U0QFgl5mXXESAiSwt776
tracking:     stable
refresh-date: 9 days ago, at 12:21 BST
channels:
  stable:    16-2.40                      2019-08-12 (7396) 92MB -
  candidate: 16-2.40                      2019-07-17 (7396) 92MB -
  beta:      16-2.41~pre1                 2019-08-20 (7640) 93MB -
  edge:      16-2.41~pre1+git1439.bebc78f 2019-08-22 (7654) 93MB -
installed:   16-2.40                                 (7396) 92MB core

I’m happy to try and debug this further, but there seems to be so many variables I’m not sure where to start.