Upgrade snapd from 2.59.5 to 2.60.2 gives apparmor DENIED on app launch for every app

Foorack · August 23, 2023, 8:08am

Upgrading from 2.59.5 to 2.60.x breaks all snap apps from launching:

$ code cannot create user data directory: /home/COMPANY/myownuser/snap/code/137: Permission denied

We’ve been able to replicate this on 30+ machines, on both Ubuntu 22 and 20.

The issue is also very reproducable:

snap revert snapd      # downgrades back to 19457
snap refresh snapd     # upgrades to 19993 again
spotify                # broken

The issue can be fixed by copying the apparmor config 19993 to the apparmor.d location, but this is not a permanent fix, as any time afterwards any snap is refreshed the user has to run “systemctl restart apparmor” (without needing to copy the file again).

$ snap debug confinement
strict

$ snap debug sandbox-features
apparmor:             kernel:caps kernel:dbus kernel:domain kernel:file kernel:ipc kernel:mount kernel:namespaces kernel:network kernel:network_v8 kernel:policy kernel:ptrace kernel:query kernel:rlimit kernel:signal parser:cap-audit-read parser:cap-bpf parser:include-if-exists parser:mqueue parser:qipcrtr-socket parser:snapd-internal parser:unsafe parser:userns parser:xdp policy:default support-level:full
confinement-options:  classic devmode strict
dbus:                 mediated-bus-access
kmod:                 mediated-modprobe
mount:                layouts mount-namespace per-snap-persistency per-snap-profiles per-snap-updates per-snap-user-profiles stale-base-invalidation
seccomp:              bpf-actlog bpf-argument-filtering kernel:allow kernel:errno kernel:kill_process kernel:kill_thread kernel:log kernel:trace kernel:trap kernel:user_notif
udev:                 tagging

$ dmesg
[156564.432059] audit: type=1400 audit(1692777552.951:4302): apparmor="DENIED" operation="open" class="file" profile="/snap/snapd/19993/usr/lib/snapd/snap-confine" name="/home/COMPANY/myownuser/" pid=207427 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=1710908145 ouid=1710908145
[156823.696946] audit: type=1400 audit(1692777812.212:4303): apparmor="DENIED" operation="open" class="file" profile="/snap/snapd/19993/usr/lib/snapd/snap-confine" name="/home/COMPANY/myownuser/" pid=211830 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=1710908145 ouid=1710908145
[156837.093575] audit: type=1400 audit(1692777825.608:4304): apparmor="DENIED" operation="open" class="file" profile="/snap/snapd/19993/usr/lib/snapd/snap-confine" name="/home/COMPANY/myownuser/" pid=211935 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=1710908145 ouid=171090814

Any idea what could be causing this?

Foorack · August 23, 2023, 8:11am

Running the reload command in this other thread works to fix the issue as well:

$ code         # BROKEN
cannot create user data directory: /home/COMPANY/myownuser/snap/code/137: Permission denied
$ sudo apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine.real
$ code         # WORKS

But this is suddenly starting to happen on a large amount of machines, and it would be inconvenient if this command has to be run every time snapd has refreshed.

Foorack · August 23, 2023, 8:41am

$ cat /etc/apparmor.d/tunables/home.d/company
@{HOMEDIRS}+=/home/EU/ /home/COMPANY/

$ export SNAP_CONFINE_DEBUG=1
$ code
DEBUG: -- snap startup {"stage":"snap-confine enter", "time":"1692780051.983582"}
DEBUG: umask reset, old umask was  022
DEBUG: security tag: snap.code.code
DEBUG: executable:   /snap/snapd/19993/usr/lib/snapd/snap-exec
DEBUG: confinement:  classic
DEBUG: base snap:    core20
DEBUG: ruid: 1710908145, euid: 0, suid: 0
DEBUG: rgid: 1872200513, egid: 1872200513, sgid: 1872200513
DEBUG: apparmor label on snap-confine is: /snap/snapd/19993/usr/lib/snapd/snap-confine
DEBUG: apparmor mode is: enforce
DEBUG: -- snap startup {"stage":"snap-confine mount namespace start", "time":"1692780051.984761"}
DEBUG: preparing classic execution environment
DEBUG: -- snap startup {"stage":"snap-confine mount namespace finish", "time":"1692780051.984795"}
DEBUG: set_effective_identity uid:1710908145 (change: yes), gid:1872200513 (change: yes)
DEBUG: creating user data directory: /home/COMPANY/myownuser/snap/code/137
cannot create user data directory: /home/COMPANY/myownuser/snap/code/137: Permission denied

lmf-mx · August 23, 2023, 11:08pm

Seeing this as well on multiple machines. Also using a path set in home.d in exactly the same way, /home/COMPANY/. This is the default for SSSD users.

milachew · August 24, 2023, 8:38am

Tryed liveISO Ubuntu 23.04 and manual upgrading to snapd 2.60.2.

Firefox, VSCode works without problem.

Upd: also works on liveISO 22.04

Foorack · August 24, 2023, 8:44am

@milachew Are you using a local account? Can you try move your home folder to /home/COMPANY/ubuntu, update /etc/passwd, and add the home.d path for apparmor?

You don’t need a fully AD-joined machine to replicate, you just need to move it to a sub folder in /home.

$ echo "@{HOMEDIRS}+=/home/COMPANY/" > /etc/apparmor.d/tunables/home.d/company

ogra · August 24, 2023, 10:11am

Seems the latest snapd ships its own apparmor implementation to be more independent from the host packages and it smells like that implementation does not source the home.d overrides… as a quick interim workaround you can simply roll back snapd via:

sudo snap revert snapd

alexmurray · August 24, 2023, 1:27pm

~~For anyone affected by this could you please try running the following:~~

echo 'include if exists "/etc/apparmor.d/tunables/home.d"' | sudo tee -a /var/lib/snapd/apparmor/snap-tuning
sudo systemctl restart snapd.apparmor.service # or just reboot

~~and let me know if that helps for now?~~

Actually I see now this issue is in the apparmor profile for snap-confine, which unfortunately does not include the snap-tuning file - so the above suggestion will likely not work. This will need to be fixed in snapd itself I think.

mvo · August 24, 2023, 6:32pm

Sorry for the trouble with the update. We are working on a fix in

https://github.com/snapcore/snapd/pull/13118

and plan to make a beta snap with this available asap.

mvo · August 25, 2023, 6:11pm

We release potential fix for testing into a store branch. If you are affected refreshing to

$ sudo snap refresh --channel=candidate/hotfix-lp2032668 snapd

and seeing if that fixes the issue would be greatly appreciated.

fastidious · August 25, 2023, 6:24pm

As someone new to snap, after I try this, and it works, what do I need to do to “go back to normal” once what’s broken is fixed?

ogra · August 25, 2023, 6:48pm

Just a:

$ sudo snap refresh --stable snapd

Will get you back

marc.richman · August 25, 2023, 7:21pm

the hotfix seems to work fine for me, How did this bug make it past testing!?!

mvo · August 26, 2023, 7:23am

Thanks for testing and confirming that it works.

Sadly the bug escaped testing because while we have an integration test for the feature this test was not using the vendored apparmor feature from the snapd and only this combination triggers the bug. This is of course fixed as well and as part of the fix both variants are integration tested now.

mvo · August 28, 2023, 7:19am

I promoted the fix now to the “stable/hotfix-lp2032668” branch so you could do:

$ sudo snap refresh --channel=stable/hotfix-lp2032668 snapd

and you will automatically keep tracking stable once the fix has landed in stable and stable/hotfix-lp2032668 get closed.

We will work on fast-tracking the fix into the stable branch for everyone once QA checks have passed.

fastidious · August 28, 2023, 11:48am

Thank you. Will you post a follow up here once the fix makes its way to stable?

marc.richman · August 28, 2023, 5:01pm

I have an ubuntu 18.04 system that is not working with the latest/stable/hotfix-lp2032668 when I try to reload the config I get this interesting error

apparmor_parser -r -d /var/lib/snapd/apparmor/profiles/snap-confine.snapd.20092

AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap-confine.snapd.20092 in /var/lib/snapd/apparmor/profiles/snap-confine.snapd.20092 at line 3: Could not open ‘if’

fastidious · August 31, 2023, 3:32pm

This hasn’t made it to stable yet, correct?

mvo · September 1, 2023, 6:40am

It is in candidate right now, we plan to promote it to stable on Monday.

mvo · September 1, 2023, 6:41am

Thanks for this error report, do you see this when you run apparmor_parser manually or is this something that snapd will have somewhere in it’s error logs? The former is expected, the apparmor_parser on 18.04 is older than the apparmor_parser that the snapd snap ships so only the version in /snap/snapd/current/usr/lib/snapd/appamor_parser will support all the new syntax.