Docker snap "failed to mount overlay" and where to report bugs?

Where do I report bugs for the Docker snap? This contact url is snappy-devel@lists.ubuntu.com. Are users expected to report bugs by sending emails to that mailing list or should the metadata be updated?

The Docker snap seems to be broken for a while. Steps to reproduce:

$ snap install docker
$ docker run hello-world
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
$ journalctl -u snap.docker.dockerd.service
Sep 15 15:00:04 howard-vm systemd[1]: Started Service for snap application docker.dockerd.
Sep 15 15:00:04 howard-vm docker.dockerd[223951]: time="2020-09-15T15:00:04.535569095+02:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
Sep 15 15:00:05 howard-vm docker.dockerd[223951]: failed to start daemon: error initializing graphdriver: driver not supported
Sep 15 15:00:05 howard-vm systemd[1]: snap.docker.dockerd.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 15:00:05 howard-vm systemd[1]: snap.docker.dockerd.service: Failed with result 'exit-code'.
Sep 15 15:00:05 howard-vm systemd[1]: snap.docker.dockerd.service: Scheduled restart job, restart counter is at 7.
Sep 15 15:00:05 howard-vm systemd[1]: Stopped Service for snap application docker.dockerd.
Sep 15 15:00:05 howard-vm systemd[1]: Started Service for snap application docker.dockerd.
Sep 15 15:00:05 howard-vm docker.dockerd[224045]: time="2020-09-15T15:00:05.996086911+02:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2

I have the same result on any 20.04 machine I tried the docker snap on in the past few months. Is this snap the recommended way to install Docker on Ubuntu? Are there any other installation instructions I should follow to get it working?

Did you add yourself to the docker group? I tried both on my existing 20.04 and on a fresh 20.04 VM and both cases a fresh docker install works for me, I had to use sudo docker run hello-world in the VM though which defaults to not having the ubuntu user in the docker group for whatever reason

Yes, it fails both with and without sudo because the daemon fails to start. I’m using ZFS on root on every machine, can that have any influence?

ZFS could certainly be an issue here, none of my 20.04 machines are using ZFS, and neither is the VM that I tried. Do you see any system journal denials when this happens?

Sep 16 16:06:49 howard-vm systemd[1]: Started Service for snap application docker.dockerd.
Sep 16 16:06:49 howard-vm kernel: aufs aufs_fill_super:918:mount[15942]: no arg
Sep 16 16:06:49 howard-vm kernel: overlayfs: missing 'lowerdir'
Sep 16 16:06:49 howard-vm docker.dockerd[15915]: time="2020-09-16T16:06:49.738317626+02:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
Sep 16 16:06:49 howard-vm kernel: overlayfs: filesystem on '/var/snap/docker/common/var-lib-docker/check-overlayfs-support487212484/upper' not supported as upperdir
Sep 16 16:06:50 howard-vm docker.dockerd[15915]: failed to start daemon: error initializing graphdriver: driver not supported
Sep 16 16:06:50 howard-vm systemd[1]: snap.docker.dockerd.service: Main process exited, code=exited, status=1/FAILURE
Sep 16 16:06:50 howard-vm systemd[1]: snap.docker.dockerd.service: Failed with result 'exit-code'.
Sep 16 16:06:50 howard-vm systemd[1]: snap.docker.dockerd.service: Scheduled restart job, restart counter is at 5.
Sep 16 16:06:50 howard-vm systemd[1]: Stopped Service for snap application docker.dockerd.
Sep 16 16:06:50 howard-vm systemd[1]: snap.docker.dockerd.service: Start request repeated too quickly.
Sep 16 16:06:50 howard-vm systemd[1]: snap.docker.dockerd.service: Failed with result 'exit-code'.
Sep 16 16:06:50 howard-vm systemd[1]: Failed to start Service for snap application docker.dockerd.

No sorry I mean Apparmor or seccomp denials, what does

journalctl --no-pager | grep DENIED

show?

Hm, I thought journalctl -f showed all logs, including AppArmor and seccomp denials?

Sep 16 16:03:04 howard-vm dbus-daemon[3196]: apparmor="DENIED" operation="dbus_method_call"  bus="session" path="/org/gtk/Notifications" interface="org.gtk.Notifications" member="RemoveNotification" mask="send" name="org.gtk.Notifications" pid=3802 label="snap.snap-store.ubuntu-software" peer_pid=3571 peer_label="unconfined"
Sep 16 16:04:48 howard-vm audit[14460]: AVC apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14460 comm="ps" requested_mask="read" denied_mask="read" peer="/usr/sbin/cupsd"
Sep 16 16:04:48 howard-vm audit[14460]: AVC apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14460 comm="ps" requested_mask="read" denied_mask="read" peer="/usr/sbin/cups-browsed"
Sep 16 16:04:48 howard-vm kernel: audit: type=1400 audit(1600265088.650:4834): apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14460 comm="ps" requested_mask="read" denied_mask="read" peer="/usr/sbin/cupsd"
Sep 16 16:04:48 howard-vm kernel: audit: type=1400 audit(1600265088.650:4835): apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14460 comm="ps" requested_mask="read" denied_mask="read" peer="/usr/sbin/cups-browsed"
Sep 16 16:04:48 howard-vm audit[14460]: AVC apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14460 comm="ps" requested_mask="read" denied_mask="read" peer="snap.snap-store.ubuntu-software"
Sep 16 16:04:48 howard-vm kernel: audit: type=1400 audit(1600265088.666:4836): apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14460 comm="ps" requested_mask="read" denied_mask="read" peer="snap.snap-store.ubuntu-software"
Sep 16 16:04:50 howard-vm audit[14577]: AVC apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14577 comm="ps" requested_mask="read" denied_mask="read" peer="/usr/sbin/cupsd"
Sep 16 16:04:50 howard-vm kernel: audit: type=1400 audit(1600265090.334:4838): apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14577 comm="ps" requested_mask="read" denied_mask="read" peer="/usr/sbin/cupsd"
Sep 16 16:04:50 howard-vm audit[14577]: AVC apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14577 comm="ps" requested_mask="read" denied_mask="read" peer="/usr/sbin/cups-browsed"
Sep 16 16:04:50 howard-vm kernel: audit: type=1400 audit(1600265090.338:4839): apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14577 comm="ps" requested_mask="read" denied_mask="read" peer="/usr/sbin/cups-browsed"
Sep 16 16:04:50 howard-vm audit[14577]: AVC apparmor="DENIED" operation="ptrace" profile="snap.docker.dockerd" pid=14577 comm="ps" requested_mask="read" denied_mask="read" peer="snap.snap-store.ubuntu-software"

hmm I don’t see anything specific there, but can you try changing the storage-driver for the docker daemon via $SNAP_DATA/config/daemon.json to something else like "zfs" specifically? The docker docs actually say if your rootfs is zfs then the storage-driver should be zfs, so probably the docker snap needs to be adjusted to use zfs instead of overlay2 if that’s what the rootfs is using when the snap is installed

cc @tianon

Hmm, I suppose that makes sense, although I’d extend the caution that the zfs graph driver in Docker was a contributed driver, and is neither well-maintained nor well-tested (it’s purely best-effort).

Well given the fact that it seems the docker snap is not working at all on zfs on Ubuntu now, I think a best effort fix here is better than nothing, though it would be good to hear from @galgalesh that using the zfs storage-driver actually helps before making the switch of course

@tianon can someone who has access to the docker snap please change the listing so we don’t point to a dead mailing list? Maybe point it here?

@tianon

I edited the config file in /var/snap/docker/current/config/daemon.json and changed the driver to zfs. This fixes my issue, the Docker daemon now starts successfully!

When I apt install docker.io, it automatically switches to the zfs storage driver. Any idea why the snap isn’t doing this?

Note: I created a MR to add the config file location in the description