With the recent autopkgtest machine shortage, I’ve been running Snapcraft’s autopkgtest suite locally using adt’s lxd runner. However, today the tests that actually install and run snaps started erroring with this:
test_plainbox.PlainboxTestCase.test_install_and_execution ... cannot create freezer cgroup hierarchy for snap plainbox-simple: Permission denied
ERROR
I know that string is coming from snap-confine, and I also know that a new core snap was released to stable today, where I know this ran successfully yesterday. I haven’t determined a root cause yet, but here are the host’s logs at the time of one failure:
Nov 15 12:32:02 Pandora kernel: [15454.900544] audit: type=1702 audit(1510777922.074:785): op=linkat ppid=24124 pid=24131 auid=4294967295 uid=166536 gid=166536 euid=166536 suid=166536 fsuid=166536 egid=166536 sgid=166536 fsgid=166536 tty=pts21 ses=4294967295 comm="python3" exe="/usr/bin/python3.5" res=0
Nov 15 12:32:02 Pandora kernel: [15454.900552] audit: type=1302 audit(1510777922.074:786): item=0 name="/usr/bin/dpkg" inode=12642577 dev=08:01 mode=0100755 ouid=165536 ogid=165536 rdev=00:00 nametype=NORMAL
Nov 15 12:32:10 Pandora kernel: [15462.936627] audit: type=1702 audit(1510777930.110:787): op=linkat ppid=24124 pid=24131 auid=4294967295 uid=166536 gid=166536 euid=166536 suid=166536 fsuid=166536 egid=166536 sgid=166536 fsgid=166536 tty=pts21 ses=4294967295 comm="python3" exe="/usr/bin/python3.5" res=0
Nov 15 12:32:10 Pandora kernel: [15462.936635] audit: type=1302 audit(1510777930.110:788): item=0 name="/usr/lib/x86_64-linux-gnu/libdb-5.3.so" inode=12651717 dev=08:01 mode=0100644 ouid=165536 ogid=165536 rdev=00:00 nametype=NORMAL
Nov 15 12:32:38 Pandora kernel: [15490.916142] audit: type=1400 audit(1510777958.090:789): apparmor="STATUS" operation="profile_replace" label="lxd-brave-muskrat_</var/lib/lxd>//&:lxd-brave-muskrat_<var-lib-lxd>://unconfined" name="snap.plainbox-simple.plainbox" pid=28401 comm="apparmor_parser"
Nov 15 12:32:38 Pandora kernel: [15491.386617] audit: type=1400 audit(1510777958.562:790): apparmor="DENIED" operation="capable" namespace="root//lxd-brave-muskrat_<var-lib-lxd>" profile="/snap/core/3440/usr/lib/snapd/snap-confine" pid=28464 comm="snap-confine" capability=2 capname="dac_read_search"
And here is the container log from the time of the failure:
Nov 15 20:32:37 brave-muskrat systemd[1]: Reloading.
Nov 15 20:32:37 brave-muskrat systemd[1]: Reloading.
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/snap-plainbox\x2dsimple-x1.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Mounting Mount unit for plainbox-simple...
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /user.slice: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /init.scope: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/var-lib-lxcfs.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/resolvconf.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-mqueue.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-zero.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/proc-meminfo.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-tty.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-fuse.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-urandom.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dbus.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-net-tun.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-null.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/atd.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/cloud-config.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/proc-sys-fs-binfmt_misc.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/system-getty.slice: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/snapd.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-lxd.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-udev-trigger.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/cloud-init-local.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/rc-local.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/sys-fs-fuse-connections.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-journal-flush.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-ptmx.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-logind.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/snap-core-3440.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/proc-diskstats.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/irqbalance.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/run-snapd-ns.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-random.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/proc-uptime.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/polkitd.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/cron.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/unattended-upgrades.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-.lxd\x2dmounts.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/mdadm.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/networking.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/ssh.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/ufw.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/run-snapd-ns-core.mnt.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/lvm2-monitor.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/snap.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/rsyslog.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/sys-kernel-debug.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/run-user-1000.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/proc-swaps.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/apport.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/lxd-containers.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/cloud-final.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/dev-full.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-user-sessions.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/-.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/systemd-udevd.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/proc-stat.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/cloud-init.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/console-getty.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/ondemand.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/accounts-daemon.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/proc-cpuinfo.mount: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Nov 15 20:32:37 brave-muskrat systemd[1]: Mounted Mount unit for plainbox-simple.
Nov 15 20:32:38 brave-muskrat snapd[10947]: 2017/11/15 20:32:38.587881 kernel_os.go:192: cannot get boot vars: open /boot/grub/grubenv: no such file or directory
Nov 15 20:32:38 brave-muskrat systemd[1]: Unmounted Mount unit for plainbox-simple.
Nov 15 20:32:39 brave-muskrat systemd[1]: Reloading.
Nov 15 20:32:39 brave-muskrat systemd[1]: Reloading.
I don’t see obvious issues. Any ideas?