Call for testing of the docker snap


#21

Hi,

I followed the same steps I listed above to reproduce the problem:

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete 
Digest: sha256:2557e3c07ed1e38f26e389462d03ed943586f744621577a99efb77324b0fe535
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:109: jailing process inside rootfs caused \\\"permission denied\\\"\"": unknown.
ERRO[0004] error waiting for container: context canceled 

The Apparmor syslog messages:

$ journalctl --no-pager -e -k | grep apparmor | grep -v kmod | grep snap.docker.dockerd
Jan 15 23:25:22 localhost kernel: audit: type=1400 audit(1547591122.578:23266): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="snap.docker.dockerd" pid=24945 comm="apparmor_parser"
Jan 15 23:25:28 localhost kernel: audit: type=1400 audit(1547591128.486:23288): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="snap.docker.dockerd" pid=25250 comm="apparmor_parser"
Jan 15 23:25:34 localhost kernel: audit: type=1400 audit(1547591134.559:23317): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.docker.dockerd" pid=25511 comm="apparmor_parser"
Jan 15 23:25:43 localhost kernel: audit: type=1400 audit(1547591143.047:23361): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="snap.docker.dockerd" name="docker-default" pid=25939 comm="apparmor_parser"
Jan 15 23:25:43 localhost kernel: audit: type=1400 audit(1547591143.415:23364): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="snap.docker.dockerd" name="docker-default" pid=26037 comm="apparmor_parser"
Jan 15 23:25:43 localhost kernel: audit: type=1400 audit(1547591143.887:23367): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="snap.docker.dockerd" name="docker-default" pid=26124 comm="apparmor_parser"
Jan 15 23:33:29 localhost kernel: audit: type=1400 audit(1547591609.053:23377): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="snap.docker.dockerd" name="docker-default" pid=945 comm="apparmor_parser"
Jan 15 23:33:41 localhost kernel: audit: type=1400 audit(1547591621.721:23387): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="snap.docker.dockerd" name="docker-default" pid=1275 comm="apparmor_parser"
Jan 15 23:33:42 localhost kernel: audit: type=1400 audit(1547591622.329:23392): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/@/var/snap/docker/common/var-lib-docker/overlay2/cdf26482d3545a13f95e18e82f13385a824d6cb0cfd789b95f9db1525f7c5108/diff/" pid=1307 comm="runc:[2:INIT]" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

#22

Thanks for the output. The last denial is the culprit here it seems. It’s very odd that the path that docker attempts to access is name="/@/var/snap/docker/common/var-lib-docker/overlay2/cdf26482d3545a13f95e18e82f13385a824d6cb0cfd789b95f9db1525f7c5108/diff/
The filepath it should be using and the one it has access to is "/var/snap/docker/common/*", I’m not sure why that leading "/@" is there, but that’s why it got denied.

I’ll have to look into this more, but thanks for the info. I will post back if I’m able to figure out a fix for this or a workaround for you.


#23

I’m noticing the error="mkdir </...>: read-only file system" problem when I try to --volume mount when using the “docker-outside-of-docker” paradigm. In other words, I share two volumes with a container (a standard directory and the docker socket [/var/run/docker.sock]), then I share the standard directory as a volume into a subsequent container.

Interestingly, it HAS worked once or twice before, however it almost always fails. I know computers are deterministic, so I have no idea how this happens. I don’t think it is a race condition, but I imagine it is some form of statefulness.

There are others with a similar experience, the current recommendation from the internet is to uninstall the snap and install Docker natively. Perhaps you could do a side-by-side install and compare the difference?

I love the snap and I hate to move away from it.


#24

Snappy updated docker and I’m running into the same issue as @tk83:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:109: jailing process inside rootfs caused \\\"permission denied\\\"\"": unknown.

It has the same cause: AppArmor denies access to a path prefixed with /@.

I think the /@ prefix has something to do with btrfs because the btrfs subvolume is called @. I’m using overlay2 as storage driver and it does not happen with vfs. But overlay2 worked before the update.

Any ideas?


#25

Hi @ijohnson

I am a new comer to snap but an old timer to Docker. After installation I’ve spotted a few sore points which perhaps have slipped from the candidate/testing channel to the stable channel:

  1. The first one is about documentation, it should be also added to the readme that Docker in snap cannot autoconfigure user namespace when activated with the default (userns-remap=default). I guess this should work with the manual way but I haven’t tested yet. (see after for the error with this option)
  2. The second one is annoying because the dockerd started by the snap thingy has the following flag --debug. That option is fine for he candidate channel but not the stable one. Trying to setup debug to false in the daemon.json fails (the dockerd does not start because of conflicting directives). That’s a bummer and I do not want to run dockerd with debug in prod. Do you know where should I create a bug report for that?

Here is the error when using the default Docker user namespace configuration:

Error during "dockremap" user creation: Error adding user "dockremap": Failed to add user with error: fork/exec /usr/sbin/adduser: permission denied; output: ""

I can help if needed, but as I said I’m a completely new comer w.r.t. to snaps, not for the rest :wink:
Cheers,
Huygens


#26

For point 2., I’ve found the culprit line: https://git.launchpad.net/~docker/+git/snap/tree/bin/dockerd-wrapper#n63

I cannot provide a Pull Request or Merge Request or even create a branch or an issue. Could someone do it for me? @ijohnson?


#27

Hi @Huygens you can file a bug at https://bugs.launchpad.net/ubuntu/+source/docker

We should be updating the docker snap shortly, we need some policy updates to the docker-support interface before we can push an update to the docker snap. I can remove the --debug when we do that, we need to wait for snapd 2.38 to be released before we can do this however.


#28

I will create a bug report so it is not forgotten.

I did not see the link between those items, but that’s probably because of my lack of knowledge of snap. One thing though, it seems that snapd 2.38 was released 6 days ago. So did you mean that you need to wait before it is deployed/available on Ubuntu?

Anyway thanks for the feedback!


#29

while there has been a github release, the QA process is still happening and as such 2.38 hasn’t been released to the stable channel of the core snap yet:

$ snap info core 
name:      core
summary:   snapd runtime environment
publisher: Canonical✓
contact:   snaps@canonical.com
license:   unset
description: |
  The core runtime environment for snapd
type:         core
snap-id:      99T7MUlRhtI3U0QFgl5mXXESAiSwt776
tracking:     candidate
refresh-date: 2 days ago, at 13:52 CDT
channels:
  stable:    16-2.37.4               2019-03-12 (6531) 95MB -
  candidate: 16-2.38                 2019-03-22 (6673) 93MB -
  beta:      16-2.38                 2019-03-21 (6673) 93MB -
  edge:      16-2.38+git1204.b555daa 2019-03-26 (6708) 93MB -
installed:   16-2.38                            (6673) 93MB core

(when I say “released”, usually I mean "released to the stable channel :slight_smile:)


#30

I created the bug report and suggested that if debug is still wished to be default, then it should better be set inside the daemon.json file so that we as user can override it.

Thanks for the clarification by the way.


#31

As I mentioned previously I’m trying to set-up user namespace for Docker:

it should be also added to the readme that Docker in snap cannot autoconfigure user namespace when activated with the default (userns-remap=default). I guess this should work with the manual way but I haven’t tested yet. (see after for the error with this option)

My previous attempt was to use the Docker default setting for userns-remap. It should create a dockremap user and proper subuid and subgid. It failed with permission denied when attempting to create the user.

So on the host, I create a group (nsgroup) and a user (nsuser) and configured the daemon.json to include the configuration: "userns-remap": "nsuser:nsgroup".

I restarted the snap, but it fails to start for the following reason:

docker.dockerd: level=info msg="User namespaces: ID ranges will be mapped to subuid/subgid ranges of: nsuser:nsgroup"
docker.dockerd: Can't create ID mappings: %!v(MISSING): open /etc/subuid: permission denied

So this time, it does not try to create the user by cannot create the mapping, although I have already created them.

I have tried to add the subuid, subgid and passwd files to /var/snap/docker/current/etc/ by greping for nsuser and nsgroup so that only the relevant information are in those files. But after restarting the snap I still have the same error as above.

I got thinking and I think the problem is AppArmor, especially after this audit message:

kernel: audit: type=1400: apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/subuid" pid=4553 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Anybody having success with the Docker snap and User Namespaces?


#32

This is AppArmor, I’m going to open another bug as there is an easy fix.

You need to edit the following file /var/lib/snapd/apparmor/profiles/snap.docker.dockerd and add the subuid and subgid to the line below (search for passwd in the file). This should be harmless security-wise as AppArmor will only allow read only permission to dockerd on those file and they do not contain sensitive information.

/etc/{passwd,group,nsswitch.conf} r,  # very common

After the change it should look like:

/etc/{passwd,group,nsswitch.conf,subuid,subgid} r,  # very common

After having done that, I was not sure which service activated the file. Normally you would only need to restart apparmor.service, but it is possible that you need also to restart snapd.service. I did both without verifying. After starting docker, I got the result (snippet from sudo docker info command):

Security Options:
 apparmor
 seccomp
  Profile: default
 userns

And running sudo docker run hello-world did work as well.


#33

Sadly, it seems that after a reboot, the AppArmor profile is reset breaking dockerd.

So, I’m going to drop testing Docker with snap until the bugs I reported are resolved. :disappointed_relieved:

Thank you anyway for the support :+1: and I hope to be back when the ride will be smoother (note that if the project would be on GitLab or GitHub, I could consider contributing, but it is currently a sort of private launchpad project making it hard to contribute).


#34

Are you still looking for testers? I have a little experience with Docker and have not installed it on some of my PCs. Environments include Debian Buster (with root on ZFS) and Ubuntu 18.04 and 16.04.
I had tried to install the snap some time ago on the Debian Buster system and got the AppArmor errors/warnings (IIRC) and did not succeed. I could have another go at that if it would be helpful.

thanks,
hank


#35

Hi again,

Apologies for not getting back to this sooner, but a couple of things:

  • the AppArmor profile you modified is generated by snapd so it should not really ever be modified manually except during development perhaps
  • the correct place to get modifications made to the AppArmor profile generated by snapd is not in the docker snap itself but rather in the docker-support interface which is in the snapd project: https://github.com/snapcore/snapd/blob/master/interfaces/builtin/docker_support.go
  • after modifications made to the snapd interface have been merged to master, they will not usually be released right away to stable (though they will likely land in edge within a day or so) and instead will follow the snapd release cycle, but if there’s something critically broken or missing then we can try to get it released to the stable channel sooner
  • the necessary bits in docker-support to accommodate docker 18.09.03-ce are now in the stable channel, so we should push up an update soon

Additionally, you can submit MPs (merge proposals - launchpad parlance for pull requests) against the docker snap on launchpad by:

  1. Create a launchpad/ubuntu SSO account
  2. Clone the repository from git+ssh://$YOUR_LP_USER@git.launchpad.net/~docker/+git/snap
  3. Create a local git branch with the changes you want to see
  4. Push your branch to the launchpad with the remote url $YOUR_LP_USER@git.launchpad.net:~$YOUR_LP_USER/+git/docker-snap - note that you don’t need to do anything to create the repo in LP beforehand as launchpad will create the repo for you when you push it.
  5. Go to code.launchpad.net/~$YOUR_LP_USER/+git/docker-snap/ and click on the branch you want to propose a merge for
  6. Click “Propose for merging”
  7. Under “Target repository” choose Other: and fill in lp:~docker/+git/snap and the target branch as master (the snap builds from the master branch)
  8. Fill in the description of the change and click propose merge
  9. We (probably me but possibly someone else in the future) will review the MP and do our best to get it integrated

Lastly, as you may have noticed snapd 2.38 has landed in stable now so we should be pushing out a new update to docker version 18.09.03-ce on the beta channel soon


#36

I tried to email the canonical team about this but their “Contact Cononical” button was broken (I opened an issue about it here https://github.com/canonical-web-and-design/snapcraft.io/issues/1846)

I’m not sure where the right place to raise this issue is, but I think this snap should be removed from snapcraft until it installs in a way that makes it work fully. This disclaimer on the snap:

This build requires all files that Docker uses, such as dockerfiles, to be in $HOME.

Means that this install doesn’t fully work. You can call that a limitation, but I think that too many users won’t the ramifications of such a limitation until it’s too late. For example, here is an issue caused in a completely different system which integrates with/relies upon Docker: https://github.com/hashicorp/nomad/issues/5360#issuecomment-466990744. It’s extremely difficult to discern that the snap-installation caused the issue. I am very fortunate that someone else has already figured it out.

I have been working on restoring a coworker’s machine to working order after she installed docker via snap without understanding why that was a bad idea. Uninstalling Docker via snap and reinstalling it the recommended way made Nomad start working again, but only to a limited degree because snap left the following apparmor profiles active on the computer despite the docker snap being removed (verified via snap list):
snap.docker.compose
snap.docker.docker
snap.docker.dockerd
snap.docker.help
snap.docker.hook.install
snap.docker.hook.post-refresh
snap.docker.machine

I’m not very familiar with apparmor but now I get to figure out how to remove the damage snap did and make everything right again so that my fresh install of Docker can work like a fresh install. All because my coworker thought she found an easier way to install Docker. While this is an issue which could be fixed, it seems like a better fix is to prevent people from needing to uninstall a broken (or limited?) Docker so they can working (or full featured?) version.

I don’t think this snap is doing anyone any favors and would like to see it removed, at least until it is fixed so that it installs in a way that doesn’t have limitations that break its integration with other projects that rely on Docker.


#37

Hi @kurtwheeler

I understand your frustration with not being able to access files outside of $HOME when using docker, but doing so is a feature of snaps in general, protecting snaps from polluting and modifying the rest of the system. If you don’t wish to use the docker snap anymore, you can always uninstall it with:

$ sudo snap remove docker

While the apparmor profiles may still be active in the kernel (which may or may not be a snapd bug, not sure if we should remove the apparmor profiles from the kernel after removing a snap), you can always completely remove those apparmor profiles by rebooting your machine after removing the snap as the profiles on disk are removed immediately on snap removal.

Lastly, so we can understand your use case and potentially improve the snap in the future, can you provide specific examples of why docker needs to access files outside of a user’s $HOME?

Thanks,
Ian


#38

Hi @ijohnson,

Thanks for getting back to me so quickly. I actually did remove the docker snap with that exact command, however it wasn’t obvious to me that a reboot would be necessary for it to take full effect. Perhaps it could issue a warning after removing a snap that in order for a non-snap install to work the system needs to be rebooted? Or perhaps those profiles should be removed immediately, I’m not entirely sure. Either way I’ve spent a good portion of today trying to figure out how to fix the AppArmor configuration for my coworker so she can resume using docker via the standard installation method, so I’m not sure the current uninstall workflow is ideal.

So I’m not actually sure why Nomad requires Docker to access files outside of a user’s $HOME… The only reason I knew that this was the issue causing Nomad to not working on her machine is because I found this issue: https://github.com/hashicorp/nomad/issues/5360.

If I had to guess I would say it’s because Nomad operates outside the $HOME directory itself and needs Docker to be able to access files it creates. Perhaps if I also installed Nomad using snap this wouldn’t have been an issue? I do see that there is a Nomad snap: https://snapcraft.io/nomad, but it’s been over a year since it was updated and I would be very surprised if it could work well with the same limitation as it needs a lot of permissions and control over the machine it is running on. (Which I think makes a lot of sense for an orchestration tool). I do know that it’s not possible to run Nomad inside of a Docker container for these reasons.

Sorry I can’t be much more help, but I don’t know much about this domain. I hadn’t even heard of AppArmor until it kept Docker from working for me.

Thanks again for your quick reply,

  • Kurt

#39

This is by design since snap process are not killed as part or removal and we don’t want them to go unconfined.


#40

Is not killing snap processes also by design? It’s not clear to me why that would be desirable to not kill all processes associated with a snap when it’s removed?

I also realize that properly killing all processes in a snap requires some work done as part of [WIP] Refresh App Awareness