Docker fails with permission denied inside containers

ayr-ton · May 19, 2020, 6:36pm

Only when using the docker snap on the new Ubuntu 20.04 I’m receiving these permissions issues:

➜  ~ docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
d51af753c3d3: Pull complete
fc878cd0a91c: Pull complete
6154df8ff988: Pull complete
fee5db0ff82f: Pull complete
Digest: sha256:747d2dbbaaee995098c9792d99bd333c6783ce56150d1b11e333bbceed5c54d7
Status: Downloaded newer image for ubuntu:latest
bash: /root/.bashrc: Permission denied
root@bdd281a64702:/# exit
➜  ~ docker logs 9b5bfdbdca50
/bin/bash: /entrypoint: Permission denied

Then, when trying to use docker from apt it works as expected, but when I uninstall and try the snap version I’m getting this issue.

My user is inside the docker group and even with sudo I’m seeing the same error.

It’s happening with all containers.

purplerhino · May 19, 2020, 6:58pm

I had this happen last night, I gave up and installed docker-ce from the docker repos. I have been having a lot of problems with the docker snap lately and went searching to see if I was alone and found this.

ayr-ton · May 19, 2020, 7:07pm

The last update was 2020-05-08, so seems that some other Ubuntu update might have crashed docker snap.

lucyllewy · May 19, 2020, 7:09pm

FWIW I’m not seeing this issue with Docker from latest/beta. @tianon might be able to advise some troubleshooting ideas…

ayr-ton · May 19, 2020, 7:13pm

I’m invoking @tianon, Stormborn of the House Snapcraft, First of His Name, the Unburnt, King of the Andals and the First Men, Super User of the Great Grass Sea, Fixer of Bugs, and Father of Snaps.

purplerhino · May 19, 2020, 7:28pm

I do agree problems might be tied to recent package upgrades and not just 20.04, I just apt upgraded my 18.04 droplet and after it rebooted the docker snap was fubar there as well.

ayr-ton · May 19, 2020, 7:50pm

The daemon logs seems fine. Only being able to catch these erros inside container logs…
Still trying to get more logs.

benton_drew · May 19, 2020, 9:59pm

I am seeing these errors too on all container runs. I am running Ubuntu 20.04. Earlier today I was running fine and then applied the latest 20.04 update (the auto-updater ran so snap packages were updated too; additionally I ran sudo apt upgrade) and docker snap is not longer working. I have tried the stable and beta snap channels and am getting the same errors on both channels.

benton_drew · May 19, 2020, 11:03pm

There seems to be an issue with the docker snap and linux kernel package 5.4.0-31-generic. I rolled back all my updates and that did nothing. Then I booted into Ubuntu with kernel 5.4.0-29-generic and the docker snap is working again.

ayr-ton · May 20, 2020, 4:02am

Good catch, let me try to reproduce it…

ayr-ton · May 20, 2020, 12:58pm

New logs. I’m getting apparmor denials when running: journalctl --no-pager -e -k | grep apparmor | grep -v kmod | grep snap.docker.dockerd

https://paste.ubuntu.com/p/7v75M82bt4/

jdstrand · May 20, 2020, 1:44pm

Note, it looks like the very last log message in the paste was truncated. Here is a summary of the paste:

This is “just noise”. Docker is running a ‘ps’ and not able to get information on some snaps that are installed on the system (that docker almost certainly doesn’t care about).

This is perhaps interesting, but I don’t know why a kernel upgrade would cause this.

ijohnson · May 20, 2020, 2:42pm

FWIW I have seen this with the docker snap for many releases also in older releases such as disco, etc. It’s unclear what docker is trying to do here but it’s harmless afaict.

t.kanstantsin · May 21, 2020, 12:18pm

Had same issue after recent kernel update. But another versions

Bug were on 5.3.0-53.47 or 5.3.0.53.45
On 5.3.0-51-generic works fine

Alculete · May 21, 2020, 4:34pm

Same for me with the error bellow on docker logs
execlineb: fatal: unable to open /init for reading: Permission denied
snap 2.44.3+20.04
snapd 2.44.3+20.04
series 16
ubuntu 20.04
kernel 5.4.0-31-generic
OS: Ubuntu Server 20.04 LTS x86_64

PeterSR · May 21, 2020, 5:52pm

I am having the same issue and spend a long time tracing it down what seems to be AppArmor.

Different images I tried:

Ubuntu

$ docker run -it ubuntu /bin/bash
bash: /root/.bashrc: Permission denied
root@60bb6a5cca3f:/#

Journal:

May 21 19:44:37 Yoga-C940 audit[7988]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.bashrc" pid=7988 comm="bash" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 21 19:44:37 Yoga-C940 kernel: audit: type=1400 audit(1590083077.908:171): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.bashrc" pid=7988 comm="bash" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

nginx

$ docker run -it nginx  
2020/05/21 17:42:36 [emerg] 1#1: open() "/etc/nginx/nginx.conf" failed (13: Permission denied)
nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (13: Permission denied)

Journal:

May 21 19:43:20 Yoga-C940 audit[7860]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/nginx/nginx.conf" pid=7860 comm="nginx" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 21 19:43:20 Yoga-C940 kernel: audit: type=1400 audit(1590083000.646:170): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/nginx/nginx.conf" pid=7860 comm="nginx" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Apache / httpd

docker run -it httpd           
/bin/sh: 0: Can't open /usr/local/bin/httpd-foreground

Journal:

May 21 19:46:09 Yoga-C940 audit[8143]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/httpd-foreground" pid=8143 comm="httpd-foregroun" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 21 19:46:09 Yoga-C940 kernel: audit: type=1400 audit(1590083169.799:172): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/httpd-foreground" pid=8143 comm="httpd-foregroun" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Using --cap-add=SYS_PTRACE and --security-opt=apparmor:unconfined as mentioned here, doesn’t seem to fix anything.

Snap version:

snap    2.44.3+20.04
snapd   2.44.3+20.04
series  16
ubuntu  20.04
kernel  5.4.0-31-generic

Docker version:

Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.13.4
 Git commit:        1752eb3
 Built:             Sat Nov 16 01:05:26 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.13.4
  Git commit:       9552f2b
  Built:            Sat Nov 16 01:07:48 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Can confirm that booting up in kernel 5.4.0-29-generic does not cause the above issue.

jdstrand · May 22, 2020, 4:44pm

The apparmor denials seem to indicate that docker has not transitioned the container into the apparmor profile for the container.

lucyllewy · May 22, 2020, 5:11pm

This is a red herring. I’m also running 5.4.0-31-generic and cannot reproduce your error.

jdstrand · May 22, 2020, 5:22pm

This seems to be https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1879690

lucyllewy · May 22, 2020, 5:35pm

aha. that would explain why I’m not seeing the issue. I’m using the zfs storage driver for docker, not overlayfs.