Docker fails with permission denied inside containers

For those seeing this bug, I suggest for now downgrading your kernel. There were Ubuntu updates to Ubuntu 19.10 and Ubuntu 20.04 LTS that caused this change and regressed docker and those changes will be reverted in the coming days.

If you are running other distros and/or kernels and seeing this exact issue (ie, the entrypoint issue in the first post in this topic), please comment in the bug (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1879690).

1 Like

Interesting :slight_smile: I assumed a variable. I ran a test on my home machine which has Ubuntu 20.04 installed directly on the machine and didn’t run into the error. The error is occurring on my work machine where I am running in a VirtualBox VM with a Windows 10 host. In that case, the 5.4.0-31-generic kernel is causing the docker snap not to work but the 5.4.0-29-generic kernel is fine.

My details Executed at 2020-05-27T12:00:00Z 18:00 Error

Building phpfpm
Step 1/28 : FROM ubuntu:18.04
 ---> c3c304cb4f22
Step 2/28 : RUN apt-get update -y && apt-get -y dist-upgrade &&     DEBIAN_FRONTEND=noninteractives apt-get -y --no-install-recommends install apt-utils libreadline-dev     php php-common php-mbstring php-xml php-mysql php-fpm php-curl php-gd     php-mbstring php-gettext php-token-stream php-zip php-pgsql     wkhtmltopdf xvfb unzip zip composer php-dev libmcrypt-dev php-pear php-redis wget
 ---> Running in cfb4180165be
W: Unable to read /etc/apt/apt.conf.d/01-vendor-ubuntu - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/01autoremove - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/01autoremove-kernels - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/70debconf - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/docker-autoremove-suggests - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/docker-clean - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/docker-gzip-indexes - open (13: Permission denied)
W: Unable to read /etc/apt/apt.conf.d/docker-no-languages - open (13: Permission denied)
E: Error reading the CPU table
ERROR: Service 'phpfpm' failed to build: The command '/bin/sh -c apt-get update -y && apt-get -y dist-upgrade &&     DEBIAN_FRONTEND=noninteractives apt-get -y --no-install-recommends install apt-utils libreadline-dev     php php-common php-mbstring php-xml php-mysql php-fpm php-curl php-gd     php-mbstring php-gettext php-token-stream php-zip php-pgsql     wkhtmltopdf xvfb unzip zip composer php-dev libmcrypt-dev php-pear php-redis wget' returned a non-zero code: 100

My EC2 server’s details:

  • Linux ip-10-0-7-59 5.3.0-1019-aws #21~18.04.1-Ubuntu SMP Mon May 11 12:33:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

  • Ubuntu 5.3.0-1019.21~18.04.1-aws 5.3.18

From logs Journal control

May 28 07:21:09 ip-10-0-7-59 kernel: audit: type=1400 audit(1590650469.929:116): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/entrypoint.sh" pid=12269 comm="entrypoint.sh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 28 07:22:10 ip-10-0-7-59 kernel: audit: type=1400 audit(1590650530.465:117): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/entrypoint.sh" pid=12419 comm="entrypoint.sh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 28 07:23:10 ip-10-0-7-59 kernel: audit: type=1400 audit(1590650590.987:118): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/entrypoint.sh" pid=12569 comm="entrypoint.sh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Snap version snap 2.44.3 snapd 2.44.3 series 16 ubuntu 18.04 kernel 5.3.0-1019-aws

docker version

Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.13.4
 Git commit:        1752eb3
 Built:             Sat Nov 16 01:05:26 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.13.4
  Git commit:       9552f2b
  Built:            Sat Nov 16 01:07:48 2019
  OS/Arch:          linux/amd64
  Experimental:     false

@aleon1220 - the fix for this issue is https://launchpad.net/ubuntu/+source/linux-aws/5.4.0-1012.12. This is currently in focal-proposed and is undergoing QA. Feel free to downgrade to a prior kernel or use the one in focal-proposed.

To others running Ubuntu 20.04 with the release kernel, the update is available in 5.4.0-33.37. For Ubuntu 19.10, the update is available in 5.3.0-55.49.

1 Like

I am now running 5.4.0-33-generic and it seems to be working :+1:

I’ve just upgraded my Ubuntu server to focal which is running 5.4.0-33-generic, but I’m still getting the permission errors when trying to install a bind mount for a service installation.

This is still a major issue making docker impossible to use properly.

Any further clues how to fix this?

Just wanted to report that I’m experiencing the same issue too.
I have tested it on a Raspberry Pi 4 (Ubuntu Server 20.04, 5.4.0-1011-raspi) and on a Hyper-V hosted VM (Ubuntu Server 20.04, 5.4.0-1012-azure) both running snapd 2.45.
On the other hand it works perfectly fine on a Debian VM (Debian 9, 4.9.0-12-amd64, snapd 2.42.1)

@turux - linux-raspi doesn’t yet have the revert, but it is on its way: https://launchpad.net/ubuntu/+source/linux-raspi/5.4.0-1012.12

@pnunn - 5.4.0-33.37-generic should have the fix, it is possible this is a different issue. What is the output of both of these commands: a) cat /proc/version_signature and b) sudo journalctl |grep audit (please only paste denials for the time period you experienced the problem).

Thanks @jdstrand. I have just tried installing the service again with the same result (within portainer) I get

mkdir /var/lib/docker: read-only file system

I did

sudo journalctl | grep audit | grep “Jun 09” and got nothing.

cat /proc/version_signature gives Ubuntu 5.4.0-33.37-generic 5.4.34

Looking at journalctl however, I do get

Jun 09 04:53:23 dockerhost docker.dockerd[698]: time="2020-06-09T04:53:23.486182895Z" level=error msg="logs call failed" error="failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.nyrw2oe4ksxcd32s3wq67ipni" module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx

Jun 09 04:53:23 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:23.486197708Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.o9woiyvnkf5587oob2hg6etk3” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:23 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:23.486213795Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.mjxbv3z20a18j7pbjxmuee4dq” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:23 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:23.486332155Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.v90a5jh2b2oorp90hiu9r5ikb” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:23 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:23.486347096Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.d5uu09ckhox270xlyctosg3fc” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.311923694Z” level=error msg=“fatal task error” error=“mkdir /var/lib/docker: read-only file system” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx service.id=6urrlojqlgpaltmn0asas99oq task.id=oqgbcksrc0cbrfp7764qihdj5
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512780554Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.d5uu09ckhox270xlyctosg3fc” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512782805Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.v90a5jh2b2oorp90hiu9r5ikb” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512798903Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.o9woiyvnkf5587oob2hg6etk3” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512923783Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.nyrw2oe4ksxcd32s3wq67ipni” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx
Jun 09 04:53:26 dockerhost docker.dockerd[698]: time=“2020-06-09T04:53:26.512939349Z” level=error msg=“logs call failed” error=“failed getting container logs: No such container: portainer_agent.28gaupt04ahvpacxxi5sgg1qx.oqgbcksrc0cbrfp7764qihdj5” module=node/agent/taskmanager node.id=28gaupt04ahvpacxxi5sgg1qx

Something else interesting, is the code on the portainer page to install it on the command line also fails but with an error saying

time=“2020-06-09T04:47:38.851926104Z” level=error msg=“Error creating service portainer_agent: rpc error: code = InvalidArgument desc = Co
ntainerSpec: “–-mount” is not a valid repository/tag”

Which is odd.

@pnunn - this looks like a different issue and you aren’t getting EPERM errors AFAICS. Would you mind creating a new topic with the above information (mentioning ‘no such container’ in the subject). This way the right people can hopefully help you. Thanks!

Have done… hope this can be sorted as its starting to be a problem. :slight_smile: I tried switching to the non snap install, but none of my containers were there when I restarted portainer on the new setup so I reverted back to the snapshot I took before trying.

Only problem @jdstrand no one is giving my other thread any love.