Yocto: snaps won't start after compiling snapd with strict confinement

om26er · October 24, 2020, 5:47pm

We created a yocto image, with snapd. We also patched the kernel to include these apparmor patches AppArmor kernel patches for 5.x kernels (had to fix the last one, wouldn’t apply) – snapd now reports its running with strict confinment. However we are no longer able to start any snap.

$ snap version
snap    2.47.1
snapd   2.47.1
series  16
poky    3.1.3
kernel  5.4.59-v8

$ snap debug sandbox-features
apparmor:             kernel:caps kernel:dbus kernel:domain kernel:file kernel:mount kernel:namespaces kernel:network kernel:network_v8 kernel:policy kernel:ptrace kernel:query kernel:rlimit kernel:signal parser:unsafe policy:default support-level:full
confinement-options:  classic devmode strict
dbus:                 mediated-bus-access
kmod:                 mediated-modprobe
mount:                freezer-cgroup-v1 layouts mount-namespace per-snap-persistency per-snap-profiles per-snap-updates per-snap-user-profiles stale-base-invalidation
seccomp:              bpf-actlog bpf-argument-filtering kernel:allow kernel:errno kernel:kill_process kernel:kill_thread kernel:log kernel:trace kernel:trap kernel:user_notif
udev:                 device-cgroup-v1 device-filtering tagging

And finally

$ SNAPD_DEBUG=1 SNAP_CONFINE_DEBUG=1 snap run hello-world
2020/10/24 17:25:28.022598 tool_linux.go:68: DEBUG: re-exec not supported on distro "poky" yet
2020/10/24 17:25:28.093402 cmd_run.go:404: DEBUG: SELinux not enabled
DEBUG: umask reset, old umask was  022
DEBUG: security tag: snap.hello-world.hello-world
DEBUG: executable:   /usr/lib/snapd/snap-exec
DEBUG: confinement:  non-classic
DEBUG: base snap:    core
DEBUG: ruid: 0, euid: 0, suid: 0
DEBUG: rgid: 0, egid: 0, sgid: 0
DEBUG: apparmor label on snap-confine is: /usr/lib/snapd/snap-confine
DEBUG: apparmor mode is: enforce
DEBUG: creating lock directory /run/snapd/lock (if missing)
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: opening lock directory /run/snapd/lock
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: opening lock file: /run/snapd/lock/.lock
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: sanity timeout initialized and set for 30 seconds
DEBUG: acquiring exclusive lock (scope (global), uid 0)
DEBUG: sanity timeout reset and disabled
DEBUG: ensuring that snap mount directory is shared
DEBUG: unsharing snap namespace directory
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: releasing lock 5
DEBUG: opened snap-update-ns executable as file descriptor 5
DEBUG: opened snap-discard-ns executable as file descriptor 6
DEBUG: creating lock directory /run/snapd/lock (if missing)
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: opening lock directory /run/snapd/lock
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: opening lock file: /run/snapd/lock/hello-world.lock
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: set_effective_identity uid:0 (change: no), gid:0 (change: yes)
DEBUG: sanity timeout initialized and set for 30 seconds
DEBUG: acquiring exclusive lock (scope hello-world, uid 0)
DEBUG: sanity timeout reset and disabled
DEBUG: initializing mount namespace: hello-world
DEBUG: snappy_udev_init
DEBUG: forked support process 1214
DEBUG: unsharing the mount namespace (per-snap)
DEBUG: changing apparmor hat to mount-namespace-capture-helper
DEBUG: helper process waiting for command
DEBUG: sanity timeout initialized and set for 30 seconds
DEBUG: scratch directory for constructing namespace: /tmp/snap.rootfs_MHHMtV
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
DEBUG: performing operation: (disabled) use debug build to see details
cannot perform operation: mount --rbind /var/tmp /tmp/snap.rootfs_MHHMtV//var/tmp: Permission denied
DEBUG:

om26er · October 24, 2020, 6:00pm

NOTE: on Yocto /var/tmp is a symlink to /var/volatile/tmp, maybe that is causing the issue ?

om26er · October 24, 2020, 7:00pm

I got this thing working after deleting two symlinks (and creating real directories)

/var/tmp (pointed to /var/volatile/tmp)
/var/log (pointed to /var/volatile/log)

So now the question is: How would a real fix look like ?

zyga-snapd · October 26, 2020, 11:16am

Yeah. My advice is to:

Extend the apparmor profile to allow for /var/volatile/tmp as as an alternative to /var/tmp
Add a flag to /var/tmp that symbolic link (on the host) may be dereferenced to obtain the bind-mount source directory. Apparmor still constrains the actual set but this will make it clear what is going on.
Do the same thing with /var/volatile/log

I’m happy to review the patches.

om26er · October 26, 2020, 6:12pm

I haven’t worked much with apparmor, but would something like below work ? Logically that looks problematic because in the presence of both /var/tmp and /var/volatile/tmp (as real dirs) it would try to mount those directories, so which one will “win” ?

diff --git a/cmd/snap-confine/snap-confine.apparmor.in b/cmd/snap-confine/snap-confine.apparmor.in
index ab801e45fb..1925b97736 100644
--- a/cmd/snap-confine/snap-confine.apparmor.in
+++ b/cmd/snap-confine/snap-confine.apparmor.in
@@ -184,6 +184,10 @@
     mount options=(rw rbind) /var/tmp/ -> /tmp/snap.rootfs_*/var/tmp/,
     mount options=(rw rslave) -> /tmp/snap.rootfs_*/var/tmp/,
 
+    # support poky (yokto)
+    # https://forum.snapcraft.io/t/yocto-snaps-wont-start-after-compiling-snapd-with-strict-confinement/20728/4
+    mount options=(rw rbind) /var/volatile/tmp/ -> /tmp/snap.rootfs_*/var/tmp/,
+
     mount options=(rw rbind) /run/ -> /tmp/snap.rootfs_*/run/,
     mount options=(rw rslave) -> /tmp/snap.rootfs_*/run/,
 
@@ -199,6 +203,10 @@
     mount options=(rw rbind) /var/log/ -> /tmp/snap.rootfs_*/var/log/,
     mount options=(rw rslave) -> /tmp/snap.rootfs_*/var/log/,
 
+    # support poky (yokto)
+    # https://forum.snapcraft.io/t/yocto-snaps-wont-start-after-compiling-snapd-with-strict-confinement/20728/4
+    mount options=(rw rbind) /var/volatile/log/ -> /tmp/snap.rootfs_*/var/log/,
+
     mount options=(rw rbind) /usr/src/ -> /tmp/snap.rootfs_*/usr/src/,
     mount options=(rw rslave) -> /tmp/snap.rootfs_*/usr/src/,

om26er · October 26, 2020, 6:35pm

Where shall I do that, any pointers would help.

om26er · October 26, 2020, 9:17pm

fyi, that patch does fix the issue on poky.

zyga-snapd · October 27, 2020, 8:16am

Yeah, those look sensible.

zyga-snapd · October 27, 2020, 8:19am

This is in cmd/snap-confine/mount-support.c. There’s a structure that describes what kind of bind mounts to make. Have a look at https://github.com/snapcore/snapd/blob/master/cmd/snap-confine/mount-support.c#L661 for the /var/log case. This structure is defined earlier in the file in https://github.com/snapcore/snapd/blob/master/cmd/snap-confine/mount-support.c#L166 - you will have to add a flag to dereference the source. One more idea is to use the existing altpath flag but see how it is used now.