Thanks for the output. The last denial is the culprit here it seems. It’s very odd that the path that docker attempts to access is name="/@/var/snap/docker/common/var-lib-docker/overlay2/cdf26482d3545a13f95e18e82f13385a824d6cb0cfd789b95f9db1525f7c5108/diff/
The filepath it should be using and the one it has access to is "/var/snap/docker/common/*", I’m not sure why that leading "/@" is there, but that’s why it got denied.
I’ll have to look into this more, but thanks for the info. I will post back if I’m able to figure out a fix for this or a workaround for you.
I’m noticing the error="mkdir </...>: read-only file system" problem when I try to --volume mount when using the “docker-outside-of-docker” paradigm. In other words, I share two volumes with a container (a standard directory and the docker socket [/var/run/docker.sock]), then I share the standard directory as a volume into a subsequent container.
Interestingly, it HAS worked once or twice before, however it almost always fails. I know computers are deterministic, so I have no idea how this happens. I don’t think it is a race condition, but I imagine it is some form of statefulness.
There are others with a similar experience, the current recommendation from the internet is to uninstall the snap and install Docker natively. Perhaps you could do a side-by-side install and compare the difference?
Snappy updated docker and I’m running into the same issue as @tk83:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:109: jailing process inside rootfs caused \\\"permission denied\\\"\"": unknown.
It has the same cause: AppArmor denies access to a path prefixed with /@.
I think the /@ prefix has something to do with btrfs because the btrfs subvolume is called @. I’m using overlay2 as storage driver and it does not happen with vfs. But overlay2 worked before the update.
I am a new comer to snap but an old timer to Docker. After installation I’ve spotted a few sore points which perhaps have slipped from the candidate/testing channel to the stable channel:
The first one is about documentation, it should be also added to the readme that Docker in snap cannot autoconfigure user namespace when activated with the default (userns-remap=default). I guess this should work with the manual way but I haven’t tested yet. (see after for the error with this option)
The second one is annoying because the dockerd started by the snap thingy has the following flag --debug. That option is fine for he candidate channel but not the stable one. Trying to setup debug to false in the daemon.json fails (the dockerd does not start because of conflicting directives). That’s a bummer and I do not want to run dockerd with debug in prod. Do you know where should I create a bug report for that?
Here is the error when using the default Docker user namespace configuration:
Error during "dockremap" user creation: Error adding user "dockremap": Failed to add user with error: fork/exec /usr/sbin/adduser: permission denied; output: ""
I can help if needed, but as I said I’m a completely new comer w.r.t. to snaps, not for the rest
Cheers,
Huygens
We should be updating the docker snap shortly, we need some policy updates to the docker-support interface before we can push an update to the docker snap. I can remove the --debug when we do that, we need to wait for snapd 2.38 to be released before we can do this however.
I will create a bug report so it is not forgotten.
I did not see the link between those items, but that’s probably because of my lack of knowledge of snap. One thing though, it seems that snapd 2.38 was released 6 days ago. So did you mean that you need to wait before it is deployed/available on Ubuntu?
while there has been a github release, the QA process is still happening and as such 2.38 hasn’t been released to the stable channel of the core snap yet:
I created the bug report and suggested that if debug is still wished to be default, then it should better be set inside the daemon.json file so that we as user can override it.
it should be also added to the readme that Docker in snap cannot autoconfigure user namespace when activated with the default (userns-remap=default). I guess this should work with the manual way but I haven’t tested yet. (see after for the error with this option)
My previous attempt was to use the Docker default setting for userns-remap. It should create a dockremap user and proper subuid and subgid. It failed with permission denied when attempting to create the user.
So on the host, I create a group (nsgroup) and a user (nsuser) and configured the daemon.json to include the configuration: "userns-remap": "nsuser:nsgroup".
I restarted the snap, but it fails to start for the following reason:
docker.dockerd: level=info msg="User namespaces: ID ranges will be mapped to subuid/subgid ranges of: nsuser:nsgroup"
docker.dockerd: Can't create ID mappings: %!v(MISSING): open /etc/subuid: permission denied
So this time, it does not try to create the user by cannot create the mapping, although I have already created them.
I have tried to add the subuid, subgid and passwd files to /var/snap/docker/current/etc/ by greping for nsuser and nsgroup so that only the relevant information are in those files. But after restarting the snap I still have the same error as above.
I got thinking and I think the problem is AppArmor, especially after this audit message:
This is AppArmor, I’m going to open another bug as there is an easy fix.
You need to edit the following file /var/lib/snapd/apparmor/profiles/snap.docker.dockerd and add the subuid and subgid to the line below (search for passwd in the file). This should be harmless security-wise as AppArmor will only allow read only permission to dockerd on those file and they do not contain sensitive information.
/etc/{passwd,group,nsswitch.conf} r, # very common
After the change it should look like:
/etc/{passwd,group,nsswitch.conf,subuid,subgid} r, # very common
After having done that, I was not sure which service activated the file. Normally you would only need to restart apparmor.service, but it is possible that you need also to restart snapd.service. I did both without verifying. After starting docker, I got the result (snippet from sudo docker info command):
Sadly, it seems that after a reboot, the AppArmor profile is reset breaking dockerd.
So, I’m going to drop testing Docker with snap until the bugs I reported are resolved.
Thank you anyway for the support and I hope to be back when the ride will be smoother (note that if the project would be on GitLab or GitHub, I could consider contributing, but it is currently a sort of private launchpad project making it hard to contribute).
Are you still looking for testers? I have a little experience with Docker and have not installed it on some of my PCs. Environments include Debian Buster (with root on ZFS) and Ubuntu 18.04 and 16.04.
I had tried to install the snap some time ago on the Debian Buster system and got the AppArmor errors/warnings (IIRC) and did not succeed. I could have another go at that if it would be helpful.
after modifications made to the snapd interface have been merged to master, they will not usually be released right away to stable (though they will likely land in edge within a day or so) and instead will follow the snapd release cycle, but if there’s something critically broken or missing then we can try to get it released to the stable channel sooner
the necessary bits in docker-support to accommodate docker 18.09.03-ce are now in the stable channel, so we should push up an update soon
Additionally, you can submit MPs (merge proposals - launchpad parlance for pull requests) against the docker snap on launchpad by:
Create a launchpad/ubuntu SSO account
Clone the repository from git+ssh://$YOUR_LP_USER@git.launchpad.net/~docker/+git/snap
Create a local git branch with the changes you want to see
Push your branch to the launchpad with the remote url $YOUR_LP_USER@git.launchpad.net:~$YOUR_LP_USER/+git/docker-snap - note that you don’t need to do anything to create the repo in LP beforehand as launchpad will create the repo for you when you push it.
Under “Target repository” choose Other: and fill in lp:~docker/+git/snap and the target branch as master (the snap builds from the master branch)
Fill in the description of the change and click propose merge
We (probably me but possibly someone else in the future) will review the MP and do our best to get it integrated
Lastly, as you may have noticed snapd 2.38 has landed in stable now so we should be pushing out a new update to docker version 18.09.03-ce on the beta channel soon
I’m not sure where the right place to raise this issue is, but I think this snap should be removed from snapcraft until it installs in a way that makes it work fully. This disclaimer on the snap:
This build requires all files that Docker uses, such as dockerfiles, to be in $HOME.
Means that this install doesn’t fully work. You can call that a limitation, but I think that too many users won’t the ramifications of such a limitation until it’s too late. For example, here is an issue caused in a completely different system which integrates with/relies upon Docker: https://github.com/hashicorp/nomad/issues/5360#issuecomment-466990744. It’s extremely difficult to discern that the snap-installation caused the issue. I am very fortunate that someone else has already figured it out.
I have been working on restoring a coworker’s machine to working order after she installed docker via snap without understanding why that was a bad idea. Uninstalling Docker via snap and reinstalling it the recommended way made Nomad start working again, but only to a limited degree because snap left the following apparmor profiles active on the computer despite the docker snap being removed (verified via snap list):
snap.docker.compose
snap.docker.docker
snap.docker.dockerd
snap.docker.help
snap.docker.hook.install
snap.docker.hook.post-refresh
snap.docker.machine
I’m not very familiar with apparmor but now I get to figure out how to remove the damage snap did and make everything right again so that my fresh install of Docker can work like a fresh install. All because my coworker thought she found an easier way to install Docker. While this is an issue which could be fixed, it seems like a better fix is to prevent people from needing to uninstall a broken (or limited?) Docker so they can working (or full featured?) version.
I don’t think this snap is doing anyone any favors and would like to see it removed, at least until it is fixed so that it installs in a way that doesn’t have limitations that break its integration with other projects that rely on Docker.
I understand your frustration with not being able to access files outside of $HOME when using docker, but doing so is a feature of snaps in general, protecting snaps from polluting and modifying the rest of the system. If you don’t wish to use the docker snap anymore, you can always uninstall it with:
$ sudo snap remove docker
While the apparmor profiles may still be active in the kernel (which may or may not be a snapd bug, not sure if we should remove the apparmor profiles from the kernel after removing a snap), you can always completely remove those apparmor profiles by rebooting your machine after removing the snap as the profiles on disk are removed immediately on snap removal.
Lastly, so we can understand your use case and potentially improve the snap in the future, can you provide specific examples of why docker needs to access files outside of a user’s $HOME?
Thanks for getting back to me so quickly. I actually did remove the docker snap with that exact command, however it wasn’t obvious to me that a reboot would be necessary for it to take full effect. Perhaps it could issue a warning after removing a snap that in order for a non-snap install to work the system needs to be rebooted? Or perhaps those profiles should be removed immediately, I’m not entirely sure. Either way I’ve spent a good portion of today trying to figure out how to fix the AppArmor configuration for my coworker so she can resume using docker via the standard installation method, so I’m not sure the current uninstall workflow is ideal.
So I’m not actually sure why Nomad requires Docker to access files outside of a user’s $HOME… The only reason I knew that this was the issue causing Nomad to not working on her machine is because I found this issue: https://github.com/hashicorp/nomad/issues/5360.
If I had to guess I would say it’s because Nomad operates outside the $HOME directory itself and needs Docker to be able to access files it creates. Perhaps if I also installed Nomad using snap this wouldn’t have been an issue? I do see that there is a Nomad snap: https://snapcraft.io/nomad, but it’s been over a year since it was updated and I would be very surprised if it could work well with the same limitation as it needs a lot of permissions and control over the machine it is running on. (Which I think makes a lot of sense for an orchestration tool). I do know that it’s not possible to run Nomad inside of a Docker container for these reasons.
Sorry I can’t be much more help, but I don’t know much about this domain. I hadn’t even heard of AppArmor until it kept Docker from working for me.
Is not killing snap processes also by design? It’s not clear to me why that would be desirable to not kill all processes associated with a snap when it’s removed?
I also realize that properly killing all processes in a snap requires some work done as part of [WIP] Refresh App Awareness
Cheers,
Huygens
)
and I hope to be back when the ride will be smoother (note that if the project would be on GitLab or GitHub, I could consider contributing, but it is currently a sort of private launchpad project making it hard to contribute).