Snapcraft adt failures with the new core release

brauner 2017-11-16 15:13:14 UTC #28

I’m obviously blatantly ignorant about some of your requirements. So my advice obviously needs to be checked against yours.

zyga 2017-11-16 15:14:07 UTC #29

I’m not sure I understand. Are you referring to chowning of /sys/fs/cgroup/freezer/snap.example or /sys/fs/cgroup/freezer

brauner 2017-11-16 15:16:28 UTC #30

No I was likely to hasty. Does the C code that creates the cgroup in https://github.com/snapcore/snapd/blame/b981a864fa3d6193975e167018e8edc93c984b30/cmd/libsnap-confine-private/cgroup-freezer-support.c#L34 run as setuid()?

zyga 2017-11-16 15:17:40 UTC #31

Yes it does. Though it also runs with an apparmor profile (look for snap-conifne.apparmor.in)

brauner 2017-11-16 15:28:10 UTC #32

Right, his is the portion that runs under non-classic confinement in:

github.com

snapcore/snapd/blob/aca5f623b4c45a0950a4e8a38fbcbc2883238dac/cmd/snap-confine/snap-confine.c#L229


			// 777 permissions for /var/lib and we need to fixup
			// for systems that had their NS created with an
			// old version
			sc_maybe_fixup_permissions();
			sc_maybe_fixup_udev();


			// Associate each snap process with a dedicated snap freezer
			// control group. This simplifies testing if any processes
			// belonging to a given snap are still alive.
			// See the documentation of the function for details.
			sc_cgroup_freezer_join(snap_name, getpid());
			sc_unlock(snap_name, snap_lock_fd);


			// Reset path as we cannot rely on the path from the host OS to
			// make sense. The classic distribution may use any PATH that makes
			// sense but we cannot assume it makes sense for the core snap
			// layout. Note that the /usr/local directories are explicitly
			// left out as they are not part of the core snap.
			debug
			    ("resetting PATH to values in sync with core snap");
			setenv("PATH",

I assume. That’s likely your AppArmor profile getting in the way. Otherwise if that really runs as setuid() root this must work.

zyga 2017-11-16 15:33:13 UTC #33

I just created a xenial container and updated snapd inside. With squashfuse installed and the core snap installed I tried running a simple busybox snap:

ubuntu@xenial:/root$ /snap/bin/snapd-hacker-toolbelt.busybox 
cannot create freezer cgroup hierarchy for snap snapd-hacker-toolbelt: Permission denied

This error is from the mkdirat call, not from fchown.

As for apparmor:

ubuntu@xenial:/root$ dmesg | grep DENIED
[ 9775.971367] audit: type=1400 audit(1510844464.690:88): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-trusty_<var-lib-lxd>" profile="/sbin/dhclient" name="/dev/pts/4" pid=11638 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=165536 ouid=165536
[ 9775.971544] audit: type=1400 audit(1510844464.690:89): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-trusty_<var-lib-lxd>" profile="/sbin/dhclient" name="/dev/pts/4" pid=11638 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=165536 ouid=165536
[ 9871.850641] audit: type=1400 audit(1510844560.570:94): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-artful_</var/lib/lxd>" name="/sys/fs/cgroup/unified/" pid=13442 comm="systemd" fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
[ 9873.026539] audit: type=1400 audit(1510844561.745:95): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-artful_</var/lib/lxd>" name="/var/lib/lxcfs/" pid=13721 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[ 9878.090184] audit: type=1400 audit(1510844566.809:96): apparmor="DENIED" operation="file_lock" profile="lxd-artful_</var/lib/lxd>" pid=13962 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 9878.090193] audit: type=1400 audit(1510844566.809:97): apparmor="DENIED" operation="file_lock" profile="lxd-artful_</var/lib/lxd>" pid=13962 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 9878.090196] audit: type=1400 audit(1510844566.809:98): apparmor="DENIED" operation="file_lock" profile="lxd-artful_</var/lib/lxd>" pid=13962 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 9878.090198] audit: type=1400 audit(1510844566.809:99): apparmor="DENIED" operation="file_lock" profile="lxd-artful_</var/lib/lxd>" pid=13962 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[11221.481436] audit: type=1400 audit(1510845910.202:117): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-xenial_<var-lib-lxd>" profile="/snap/core/3440/usr/lib/snapd/snap-confine//snap_update_ns" name="/dev/null" pid=17805 comm="5" requested_mask="r" denied_mask="r" fsuid=165536 ouid=0

I see a number of failures here: The file_inherit of /dev/null is one interesting aspect. Should we adjust the profile for LXD / snapd somehow?

EDIT: Actually they are all interesting. What do you think?

brauner 2017-11-16 15:39:13 UTC #34

Yeah, I pointed this out above.

Before you are trying to mkdirat() you seem to be changing into an AppArmor profile “mount-namespace-capture-helper”. Maybe interesting to see what this does.

brauner 2017-11-16 15:44:04 UTC #35

ubuntu@xenial:/root$ dmesg | grep DENIED
[ 9775.971367] audit: type=1400 audit(1510844464.690:88): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-trusty_<var-lib-lxd>" profile="/sbin/dhclient" name="/dev/pts/4" pid=11638 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=165536 ouid=165536
[ 9775.971544] audit: type=1400 audit(1510844464.690:89): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-trusty_<var-lib-lxd>" profile="/sbin/dhclient" name="/dev/pts/4" pid=11638 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=165536 ouid=165536
[ 9871.850641] audit: type=1400 audit(1510844560.570:94): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-artful_</var/lib/lxd>" name="/sys/fs/cgroup/unified/" pid=13442 comm="systemd" fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
[ 9873.026539] audit: type=1400 audit(1510844561.745:95): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-artful_</var/lib/lxd>" name="/var/lib/lxcfs/" pid=13721 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[ 9878.090184] audit: type=1400 audit(1510844566.809:96): apparmor="DENIED" operation="file_lock" profile="lxd-artful_</var/lib/lxd>" pid=13962 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 9878.090193] audit: type=1400 audit(1510844566.809:97): apparmor="DENIED" operation="file_lock" profile="lxd-artful_</var/lib/lxd>" pid=13962 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 9878.090196] audit: type=1400 audit(1510844566.809:98): apparmor="DENIED" operation="file_lock" profile="lxd-artful_</var/lib/lxd>" pid=13962 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 9878.090198] audit: type=1400 audit(1510844566.809:99): apparmor="DENIED" operation="file_lock" profile="lxd-artful_</var/lib/lxd>" pid=13962 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[11221.481436] audit: type=1400 audit(1510845910.202:117): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-xenial_<var-lib-lxd>" profile="/snap/core/3440/usr/lib/snapd/snap-confine//snap_update_ns" name="/dev/null" pid=17805 comm="5" requested_mask="r" denied_mask="r" fsuid=165536 ouid=0
I see a number of failures here: The file_inherit of /dev/null is one interesting aspect. Should we adjust the profile for LXD / snapd somehow?

I’m not sure why you’d want to inherit an fd for /dev/null or /dev/pts/<idx> which must be the fd lxd currently uses from the host for its exec session. The AppArmor denies seem reasonable to me. (Apart from the cgroup2 deny but that’s probably apparmor not knowing about cgroup2.

zyga 2017-11-16 15:47:20 UTC #36

On second thought the helper that is doing the mount namespace capture (bind mount) is probably a red herring. I’ll test a tweak to see what happens.

What about the various mount denials?

zyga 2017-11-16 15:52:22 UTC #37

I turned of kernel rate limiting by running sysctl kernel.printk_ratelimit=0 and patched the profile to allow access to /dev/null but I don’t see any denials and the error is exactly the same as before.

brauner 2017-11-16 15:55:31 UTC #38

I turned of kernel rate limiting by running sysctl kernel.printk_ratelimit=0 and patched the profile to allow access to /dev/null but I don’t see any denials and the error is exactly the same as before.

That is orthogonal to what we are discussing here I think.

snap-confine itself seems to be running under an AppArmor profile as indicated by:

github.com

snapcore/snapd/blob/aca5f623b4c45a0950a4e8a38fbcbc2883238dac/cmd/snap-confine/snap-confine.c#L157


	if (!sc_is_hook_security_tag(security_tag)) {
		struct sc_error *err SC_CLEANUP(sc_cleanup_error) = NULL;
		snap_context = sc_cookie_get_from_snapd(snap_name, &err);
		if (err != NULL) {
			error("%s\n", sc_error_msg(err));
		}
	}


	struct sc_apparmor apparmor;
	sc_init_apparmor_support(&apparmor);
	if (!apparmor.is_confined && apparmor.mode != SC_AA_NOT_APPLICABLE
	    && getuid() != 0 && geteuid() == 0) {
		// Refuse to run when this process is running unconfined on a system
		// that supports AppArmor when the effective uid is root and the real
		// id is non-root.  This protects against, for example, unprivileged
		// users trying to leverage the snap-confine in the core snap to
		// escalate privileges.
		die("snap-confine has elevated permissions and is not confined"
		    " but should be. Refusing to continue to avoid"
		    " permission escalation attacks");
	}

Where is this profile?

brauner 2017-11-16 15:56:59 UTC #39

Oh ok, I see it just checks whether it runs confined it seems.

zyga 2017-11-16 16:00:39 UTC #40

The profile is in snap-confine.apparmor.in in the source tree or in /etc/apparmor.d/*snap-confine.real

NOTE: There will be two files because snapd re-executes itself so one will be for the packaged version and one will be for the snapd-from-core-snap.

brauner 2017-11-16 16:10:18 UTC #41

Your AppArmor profile seems to be allowing to read the freezer cgroup but not write to it, am I right?

github.com

snapcore/snapd/blob/096ede7efefb371294a43225c116b99500306f86/cmd/snap-confine/snap-confine.apparmor.in#L59


capability sys_admin,
capability dac_override,
/sys/fs/cgroup/devices/snap{,py}.*/ w,
/sys/fs/cgroup/devices/snap{,py}.*/tasks w,
/sys/fs/cgroup/devices/snap{,py}.*/devices.{allow,deny} w,


# cgroup: freezer
# Allow creating per-snap cgroup freezers and adding snap command (task)
# invocations to the freezer. This allows for reliably enumerating all
# running tasks for the snap.
/sys/fs/cgroup/freezer/ r,
/sys/fs/cgroup/freezer/snap.*/ w,
/sys/fs/cgroup/freezer/snap.*/tasks w,


# querying udev
/etc/udev/udev.conf r,
/sys/**/uevent r,
/lib/udev/snappy-app-dev ixr, # drop
/run/udev/** rw,
/{,usr/}bin/tr ixr,
/usr/lib/locale/** r,

zyga 2017-11-16 16:11:41 UTC #42

Hmm maybe I read my apparmor wrong but this should say we can open the freezer directory and write and create the snap.* sub-directory there. If this was failing it would fail outside of LXD as well.

mborzecki 2017-11-16 16:16:13 UTC #43

A minimal sample, extracted from cgroup-freezer-support.c, that fails for me: https://paste.ubuntu.com/25975101/

brauner 2017-11-16 16:19:02 UTC #44

Are you running it as root or setuid?

mborzecki 2017-11-16 16:20:27 UTC #45

I actually tried this:

build with gcc -DDO_WAIT..
move the binary to the container’s /, chown root:root, chmod u+s
run it, it will stop waiting for SIGCONT
go back to host, find the test binary’s pid in global namespace
while on the host trace-cmd -e syscalls -e cgroup -e fs -p function -P <global-pid>
back in the container kill -CONT <pid>, see cannot create freezer cgroup hierarchy: Permission denied
back to the host C-c, trace-cmd report … and get a segfault in trace-cmd

mborzecki 2017-11-16 16:25:53 UTC #46

I’m running with setuid

brauner 2017-11-16 16:33:16 UTC #47

Ah ffs, I know what’s going on. So obvious that I didn’t even consider it. You’re running setuid but not setgid, right? By that I mean snap-confine.