Snapd 2.32 breaks live-server installer

Today’s live-server installer (the one with serial 20180320) does not work because subiquity never starts. subiquity’s logs show that it is getting permission denied when reading from stdin, which is a bit surprising to say the least!

There are apparmor denials in syslog like this:

Mar 21 01:41:11 ubuntu-server kernel: [ 1919.759983] audit: type=1400 audit(1521596471.871:78): apparmor="DENIED" operation="file_inherit" profile="/usr/lib/snapd/snap-confine" name="/dev/tty1" pid=2566 comm="snap-confine" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0

which definitely seem related. Switching snap-confine’s apparmor over to complain mode lets subiquity start.

I’ve done some digging into what’s going on and it’s a bit strange. (Hi @jdstrand!)

The way subiquity starts (or is supposed to start) is that the subiquity snap defines a systemd service which runs a shell script that does some setup and then invokes “/sbin/agetty -n --noclear -l /snap/bin/subiquity tty1 $TERM”, “/snap/bin/subiquity” being another command defined by the snap.

It seems to be this invocation of a snapped command from within another shell script that is under classic confinement. If I put just “/sbin/agetty -n --noclear -l /snap/bin/subiquity tty1 $TERM” in a shell script and invoke it, subiquity starts as expected. If I copy a profile snapd wrote for a classically confined command and hack it to apply to the shell script, subiquity does not start. This seems wrong as I thought classic confinement was basically meant to let you do anything!

Having written all this out, I think I can work around it (by having my shell script invoke $SNAP/usr/bin/subiquity, not /snap/bin/subiquity) but this all smells wrong.

Oh and another way I can make the denial go away is to add “/dev/tty1 rw” to snap-confine’s apparmor profile.

1 Like

What is also strange is why this is breaking with 2.32 - iirc we did not change the snap-confine apparmor profile for 2.32.

What changed iiuc is that snapd now loads the snap-confine in a livecd environment (/etc/init.d/apparmor does not load any profiles but https://github.com/snapcore/snapd/commit/0f40f37c416587bd2e3b51db66601262869d2dc7 causes snapd to load snap-confine’s even for the host). Previously it ran unconfined.

Also I should say that I have a workaround for this now so it’s not urgent. But still curious.

Prior to 2.32 and subiquity aside, classic and strict mode snaps did not run on livecd for two reasons: snap-confine policy was not loaded and if it was, the permissions were not adjusted for overlayfs. In 2.32, snapd notices the fact that it is running on overlayfs and updates system-key, adds a snippet to /var/lib/snapd/apparmor/snap-confine/overlay-root for running on overlayfs and loads the apparmor profile for snap-confine. classic and strict mode snaps now are allowed to work.

Notice that the AppArmor denial you are seeing is not against the subiquity snap itself, but is against snap-confine with how you are invoking subiquity. By defining a daemon, you end up with ‘snap run subiquity.subiquity-service’ in the systemd service file, which prior to your recent upload called /snap/bin/subiquity, which is effectively ‘snap run subiquity’. The calls are then roughly: systemd execs ‘snap run subiquity.subiquity-service’ -> snap-confine -> command-subiquity-service.wrapper -> $SNAP/usr/bin/subiquity-service -> agetty -> snap run subiquity -> snap-confine -> command-subiquity.wrapper -> python3 subiquity. The tty1 denial is not surprising to me at all due to fd inheritance on tty1 with this call stack of the snap calling another snap command.

What is curious to me is how subiquity worked at all before 2.32. Was the use of ‘/sbin/agetty -n --noclear -l **/snap/bin/**subiquity tty1 $TERM’ new (eg, maybe you used to use $SNAP/usr/bin)? If not, I’m guessing that snapd downgraded itself to forced devmode where snap-confine was never invoked by ‘snap run’.

In general, a classic snap is meant to be able to invoke other snaps, but file inheritance is always going to be at play. I’d prefer to not add ‘/dev/tty1 rw,’ to snap-confine just for the subiquity snap and prefer your current workaround. If people feel strongly otherwise or other snaps need this access, we can add the rule.

2 Likes

Thanks for the explanation. I think my “workaround” is actually clearer tbh so I certainly don’t require a fix. I think this worked before because subiquity is only used in a live session so before 2.32 apparmor was never involved at all?

I don’t really understand why the unconfined -> “policy that prevents writes to tty1” transition is different to the “policy that allows writes to tty1” -> “policy that prevents writes to tty1” transition but I assume there is a reason :slight_smile:

So I think at the end of the day, this is just something that has to be a known gotcha: classic snaps can’t pass arbitrary file handles when invoking any snap, even another or even the same classic snap.

Commands from classic snaps do not run under the ‘unconfined’ security label, they run under their own snap-specific security labels (eg, ‘snap.subiquity.subiquity’) with policy that is ‘effectively unconfined’. As such, with this call stack the policy transitions are unconfined -> snap-confine -> snap.subiquity.subiquity-service -> snap-confine -> snap.subiquity.subiquity. Since we are going from a !unconfined to differently-!unconfined security label when passing the fd, file_inherit is triggered.

Yes I understand all that, what I don’t understand is why it’s designed that way, i.e. why file_inherit triggers in this situation and not unconfined -> confined.

The idea behind this is with confined -> unconfined the access is allowed because unconfined has no restrictions to begin with, so we allow passing the fd without a validation check. For unconfined -> confined we also allow it because we perform so-called ‘implicit delegation’. For confined -> confined (and the special case of confined -> unconfined -> confined) we’ll do a revalidation. It is implemented this way to try to maintain a level of security (confined -> confined) but also usability for profilers (unconfined -> confined, confined -> unconfined). While it is a fine intermediary step, on the AppArmor roadmap are various improvements to make things work better:

  1. policy delegation - where delegation is specified in policy
    a. rule delegation (eg, px -> firefox + { /foo rw, /bar r, } (syntax TBD)
    b. object delegation - where only an object (not rules) is delegated. Idea is, parent opens a file, passes the object to the child for the child to use. The child can use it but not open it on its own (rule delegation would also allow open)
  2. Application directed delegation
    a. rule delegation - where an application can direct apparmor to delegate some rules via an api
    b. object delegation - where an application can direct apparmor to delegate a specific object only (eg, a particular fd to use, but not open)

With this delegation work, fds that have been delegated don’t undergo revalidation and so no extra policy bits are needed. This will work across namespace boundaries too, which will take care of a lot of our disconnected path issues (ie, can remove ‘attach_disconnected’). ‘1’ should allow us to let delegation happen for your use case, thus avoiding revalidation and the denial.

The above is just an overview of the roadmapped items. If you’re interested in more details, I suggest participating on the apparmor mailing list or discussing on irc in #apparmor on OFTC.

1 Like

Hi, thanks for getting back to this :slight_smile:

I’m not entirely sure I understand the roadmap items (does it matter that in my case, the more-confined application, snap-confine, is not actually going to touch the file descriptor that it’s passed in, just pass it on to a less confined application) and it getting DENIED if it did try to touch it?) but I’m happy to hear that there is a plan :slight_smile:

Are there any news on this? We think we were just bitten by something similar:

sep 27 17:08:50 mymachine audit[27944]: AVC apparmor="DENIED" operation="file_inherit" profile="/snap/core/7713/usr/lib/snapd/snap-confine" pid=27944 comm="snap-confine" family="unix" sock_type="stream" protocol=0 requested_
sep 27 17:08:50 mymachine audit[27944]: AVC apparmor="DENIED" operation="file_inherit" profile="/snap/core/7713/usr/lib/snapd/snap-confine" pid=27944 comm="snap-confine" family="unix" sock_type="stream" protocol=0 requested_
sep 27 17:08:50 mymachine audit[28113]: AVC apparmor="DENIED" operation="file_inherit" profile="/snap/core/7713/usr/lib/snapd/snap-confine" pid=28113 comm="snap-confine" family="unix" sock_type="stream" protocol=0 requested_
sep 27 17:08:50 mymachine audit[28113]: AVC apparmor="DENIED" operation="file_inherit" profile="/snap/core/7713/usr/lib/snapd/snap-confine" pid=28113 comm="snap-confine" family="unix" sock_type="stream" protocol=0 requested_
sep 27 17:08:50 mymachine audit[28113]: AVC apparmor="DENIED" operation="file_inherit" profile="/snap/core/7713/usr/lib/snapd/snap-confine" pid=28113 comm="snap-confine" family="unix" sock_type="stream" protocol=0 requested_
sep 27 17:08:50 mymachine kernel: audit: type=1400 audit(1569596930.815:854): apparmor="DENIED" operation="file_inherit" profile="/snap/core/7713/usr/lib/snapd/snap-confine" pid=28113 comm="snap-confine" family="unix" sock_t
sep 27 17:08:50 mymachine kernel: audit: type=1400 audit(1569596930.815:855): apparmor="DENIED" operation="file_inherit" profile="/snap/core/7713/usr/lib/snapd/snap-confine" pid=28113 comm="snap-confine" family="unix" sock_t
sep 27 17:08:50 mymachine kernel: audit: type=1400 audit(1569596930.815:856): apparmor="DENIED" operation="file_inherit" profile="/snap/core/7713/usr/lib/snapd/snap-confine" pid=28113 comm="snap-confine" family="unix" sock_t

when trying to call npm from a node app.

Is there something that we could add to snap-confine’s apparmor profile to workaround this @jdstrand?