Docker fails with permission denied inside containers

t.kanstantsin · May 21, 2020, 12:18pm

Had same issue after recent kernel update. But another versions

Bug were on 5.3.0-53.47 or 5.3.0.53.45
On 5.3.0-51-generic works fine

Alculete · May 21, 2020, 4:34pm

Same for me with the error bellow on docker logs
execlineb: fatal: unable to open /init for reading: Permission denied
snap 2.44.3+20.04
snapd 2.44.3+20.04
series 16
ubuntu 20.04
kernel 5.4.0-31-generic
OS: Ubuntu Server 20.04 LTS x86_64

PeterSR · May 21, 2020, 5:52pm

I am having the same issue and spend a long time tracing it down what seems to be AppArmor.

Different images I tried:

Ubuntu

$ docker run -it ubuntu /bin/bash
bash: /root/.bashrc: Permission denied
root@60bb6a5cca3f:/#

Journal:

May 21 19:44:37 Yoga-C940 audit[7988]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.bashrc" pid=7988 comm="bash" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 21 19:44:37 Yoga-C940 kernel: audit: type=1400 audit(1590083077.908:171): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.bashrc" pid=7988 comm="bash" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

nginx

$ docker run -it nginx  
2020/05/21 17:42:36 [emerg] 1#1: open() "/etc/nginx/nginx.conf" failed (13: Permission denied)
nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (13: Permission denied)

Journal:

May 21 19:43:20 Yoga-C940 audit[7860]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/nginx/nginx.conf" pid=7860 comm="nginx" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 21 19:43:20 Yoga-C940 kernel: audit: type=1400 audit(1590083000.646:170): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/etc/nginx/nginx.conf" pid=7860 comm="nginx" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Apache / httpd

docker run -it httpd           
/bin/sh: 0: Can't open /usr/local/bin/httpd-foreground

Journal:

May 21 19:46:09 Yoga-C940 audit[8143]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/httpd-foreground" pid=8143 comm="httpd-foregroun" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 21 19:46:09 Yoga-C940 kernel: audit: type=1400 audit(1590083169.799:172): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/httpd-foreground" pid=8143 comm="httpd-foregroun" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Using --cap-add=SYS_PTRACE and --security-opt=apparmor:unconfined as mentioned here, doesn’t seem to fix anything.

Snap version:

snap    2.44.3+20.04
snapd   2.44.3+20.04
series  16
ubuntu  20.04
kernel  5.4.0-31-generic

Docker version:

Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.13.4
 Git commit:        1752eb3
 Built:             Sat Nov 16 01:05:26 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.13.4
  Git commit:       9552f2b
  Built:            Sat Nov 16 01:07:48 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Can confirm that booting up in kernel 5.4.0-29-generic does not cause the above issue.

jdstrand · May 22, 2020, 4:44pm

The apparmor denials seem to indicate that docker has not transitioned the container into the apparmor profile for the container.

lucyllewy · May 22, 2020, 5:11pm

This is a red herring. I’m also running 5.4.0-31-generic and cannot reproduce your error.

jdstrand · May 22, 2020, 5:22pm

This seems to be https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1879690

lucyllewy · May 22, 2020, 5:35pm

aha. that would explain why I’m not seeing the issue. I’m using the zfs storage driver for docker, not overlayfs.

jdstrand · May 22, 2020, 6:00pm

For those seeing this bug, I suggest for now downgrading your kernel. There were Ubuntu updates to Ubuntu 19.10 and Ubuntu 20.04 LTS that caused this change and regressed docker and those changes will be reverted in the coming days.

If you are running other distros and/or kernels and seeing this exact issue (ie, the entrypoint issue in the first post in this topic), please comment in the bug (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1879690).

benton_drew · May 22, 2020, 10:58pm

Interesting I assumed a variable. I ran a test on my home machine which has Ubuntu 20.04 installed directly on the machine and didn’t run into the error. The error is occurring on my work machine where I am running in a VirtualBox VM with a Windows 10 host. In that case, the 5.4.0-31-generic kernel is causing the docker snap not to work but the 5.4.0-29-generic kernel is fine.

aleon1220 · May 28, 2020, 7:26am

My details Executed at 2020-05-27T12:00:00Z 18:00
Error

Building phpfpm
Step 1/28 : FROM ubuntu:18.04
 ---> c3c304cb4f22
Step 2/28 : RUN apt-get update -y && apt-get -y dist-upgrade &&     DEBIAN_FRONTEND=noninteractives apt-get -y --no-install-recommends install apt-utils libreadline-dev     php php-common php-mbstring php-xml php-mysql php-fpm php-curl php-gd     php-mbstring php-gettext php-token-stream php-zip php-pgsql     wkhtmltopdf xvfb unzip zip composer php-dev libmcrypt-dev php-pear php-redis wget
 ---> Running in cfb4180165be
W: Unable to read /etc/apt/apt.conf.d/01-vendor-ubuntu - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/01autoremove - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/01autoremove-kernels - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/70debconf - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/docker-autoremove-suggests - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/docker-clean - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/docker-gzip-indexes - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/docker-no-languages - open (13: Permission denied)
E: Error reading the CPU table
ERROR: Service 'phpfpm' failed to build: The command '/bin/sh -c apt-get update -y && apt-get -y dist-upgrade &&     DEBIAN_FRONTEND=noninteractives apt-get -y --no-install-recommends install apt-utils libreadline-dev     php php-common php-mbstring php-xml php-mysql php-fpm php-curl php-gd     php-mbstring php-gettext php-token-stream php-zip php-pgsql     wkhtmltopdf xvfb unzip zip composer php-dev libmcrypt-dev php-pear php-redis wget' returned a non-zero code: 100

My EC2 server’s details:

Linux ip-10-0-7-59 5.3.0-1019-aws #21~18.04.1-Ubuntu SMP Mon May 11 12:33:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 5.3.0-1019.21~18.04.1-aws 5.3.18

From logs Journal control

May 28 07:21:09 ip-10-0-7-59 kernel: audit: type=1400 audit(1590650469.929:116): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/entrypoint.sh" pid=12269 comm="entrypoint.sh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 28 07:22:10 ip-10-0-7-59 kernel: audit: type=1400 audit(1590650530.465:117): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/entrypoint.sh" pid=12419 comm="entrypoint.sh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 28 07:23:10 ip-10-0-7-59 kernel: audit: type=1400 audit(1590650590.987:118): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/entrypoint.sh" pid=12569 comm="entrypoint.sh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Snap version
snap 2.44.3
snapd 2.44.3
series 16
ubuntu 18.04
kernel 5.3.0-1019-aws

docker version

Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.13.4
 Git commit:        1752eb3
 Built:             Sat Nov 16 01:05:26 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.13.4
  Git commit:       9552f2b
  Built:            Sat Nov 16 01:07:48 2019
  OS/Arch:          linux/amd64
  Experimental:     false

jdstrand · May 28, 2020, 5:43pm

@aleon1220 - the fix for this issue is https://launchpad.net/ubuntu/+source/linux-aws/5.4.0-1012.12. This is currently in focal-proposed and is undergoing QA. Feel free to downgrade to a prior kernel or use the one in focal-proposed.

To others running Ubuntu 20.04 with the release kernel, the update is available in 5.4.0-33.37. For Ubuntu 19.10, the update is available in 5.3.0-55.49.

PeterSR · May 31, 2020, 10:16am

I am now running 5.4.0-33-generic and it seems to be working

pnunn · June 7, 2020, 11:06am

I’ve just upgraded my Ubuntu server to focal which is running 5.4.0-33-generic, but I’m still getting the permission errors when trying to install a bind mount for a service installation.

This is still a major issue making docker impossible to use properly.

Any further clues how to fix this?

turux · June 7, 2020, 2:23pm

Just wanted to report that I’m experiencing the same issue too.
I have tested it on a Raspberry Pi 4 (Ubuntu Server 20.04, 5.4.0-1011-raspi) and on a Hyper-V hosted VM (Ubuntu Server 20.04, 5.4.0-1012-azure) both running snapd 2.45.
On the other hand it works perfectly fine on a Debian VM (Debian 9, 4.9.0-12-amd64, snapd 2.42.1)

jdstrand · June 8, 2020, 3:52pm

@turux - linux-raspi doesn’t yet have the revert, but it is on its way: https://launchpad.net/ubuntu/+source/linux-raspi/5.4.0-1012.12

jdstrand · June 8, 2020, 3:56pm

@pnunn - 5.4.0-33.37-generic should have the fix, it is possible this is a different issue. What is the output of both of these commands: a) cat /proc/version_signature and b) sudo journalctl |grep audit (please only paste denials for the time period you experienced the problem).

pnunn · June 9, 2020, 5:01am

Thanks @jdstrand. I have just tried installing the service again with the same result (within portainer) I get

mkdir /var/lib/docker: read-only file system

I did

sudo journalctl | grep audit | grep “Jun 09” and got nothing.

cat /proc/version_signature gives Ubuntu 5.4.0-33.37-generic 5.4.34

Looking at journalctl however, I do get

Jun 09 04:53:23 dockerhost docker.dockerd[698]: time="2020-06-09T04:53:23.486182895Z" level=error msg="logs call failed" error="failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.nyrw2oe4ksxcd32s3wq67ipni" module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx

Jun 09 04:53:23 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:23.486197708Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.o9woiyvnkf5587oob2hg6etk3” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:23 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:23.486213795Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.mjxbv3z20a18j7pbjxmuee4dq” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:23 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:23.486332155Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.v90a5jh2b2oorp90hiu9r5ikb” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:23 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:23.486347096Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.d5uu09ckhox270xlyctosg3fc” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.311923694Z” level=error msg=“fatal task error” error=“mkdir /var/lib/docker: read-only file system” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx service.id=6urrlojqlgpaltmn0asas99oq task.id=oqgbcksrc0cbrfp7764qihdj5
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512780554Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.d5uu09ckhox270xlyctosg3fc” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512782805Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.v90a5jh2b2oorp90hiu9r5ikb” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512798903Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.o9woiyvnkf5587oob2hg6etk3” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512923783Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.nyrw2oe4ksxcd32s3wq67ipni” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512939349Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.oqgbcksrc0cbrfp7764qihdj5” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx

Something else interesting, is the code on the portainer page to install it on the command line also fails but with an error saying

time=“2020-06-09T04:47:38.851926104Z” level=error msg=“Error creating service portainer_agent: rpc error: code = InvalidArgument desc = Co
ntainerSpec: “–-mount” is not a valid repository/tag”

Which is odd.

jdstrand · June 9, 2020, 2:13pm

@pnunn - this looks like a different issue and you aren’t getting EPERM errors AFAICS. Would you mind creating a new topic with the above information (mentioning ‘no such container’ in the subject). This way the right people can hopefully help you. Thanks!

pnunn · June 10, 2020, 12:26am

Have done… hope this can be sorted as its starting to be a problem. I tried switching to the non snap install, but none of my containers were there when I restarted portainer on the new setup so I reverted back to the snapshot I took before trying.

pnunn · June 19, 2020, 11:04am

Only problem @jdstrand no one is giving my other thread any love.