I’ll bet this is your issue. I think you need to allow root to access the home folders as root (i.e. not squashed as this line indicates your export currently is configured) or snaps will fail IIRC.
Try ensuring that your home folders are world readable/executable then, so that root can still access them when squashed to nfsnobody. You might be able to use ACLs to limit the exposure a bit more than world r-x by adding an entry for nfsnobody. We’re veering away from my expertise tho, so I’ll defer to others who might understand NFS permissions better than I.
So, everything looks correct: the rules
network inet,
network inet6,
are present in the preprocessed profile. With these rules, AppArmor should not complain, so I wonder if it’s possible that for some reason snap-confine is still using the old profile. Can you please check:
- That indeed you are still getting the same error as before
- Run
sudo apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine.real
- Try running firefox again, and verify whether the error is gone.
- If not, try adding a
-Koption to the line in step 2 and try again - If it still does not work, please paste the output of
sudo apparmor_parser -d /etc/apparmor.d/usr.lib.snapd.snap-confine.real
Ok so, if I reboot to start fresh. Then just run apparmor_parser line, which produce not output. I still get
cannot open path of the current working directory: Permission denied
But, if I restart snapd service, then run again apparmor_parser line then I get
cannot create user data directory: /home/p00000012366/snap/firefox/1300: Permission denied
Which seems better. Tried at this point to rm -Rf ~/snap but still the same error.
That’s interesting! It would look like snapd generates the correct apparmor profile snippets when it starts, but then
- It doesn’t load them
- It (or something else) deletes them later
Are you able to figure out what happens to the /var/lib/snapd/apparmor/snap-confine/nfs-support file? From what you wrote, it would seem that something is deleting it, and we need to find out if there are regular circumstances for this.
Also, about the profile not being loaded when snapd restarts: can you trigger a snapd restart, then run firefox, and show the journal logs? I want to see especially if the operation="profile_load" lines for the snap-confine profile are visible.
Looking at …, there is:
# Those are discussed on https://forum.snapcraft.io/t/snapd-vs-upstream-kernel-vs-apparmor
# and https://forum.snapcraft.io/t/snaps-and-nfs-home/
#include "/var/lib/snapd/apparmor/snap-confine"
hopefully the parser does not treat #include as part of the comment. Then the manpage states:
#include mechanism
AppArmor provides an easy abstraction mechanism to group common access requirements; this
abstraction is an extremely flexible way to grant site-specific rights and makes writing new
AppArmor profiles very simple by assembling the needed building blocks for any given program.
The use of '#include' is modelled directly after cpp(1); its use will replace the '#include'
statement with the specified file's contents. The leading '#' is optional, and the
'#include' keyword can be followed by an option conditional 'if exists' that specifies
profile compilation should continue if the specified file or directory is not found.
so it’s supposed to be a file? The <abstratction/..> entries seem to be files.
Although when I deliberately add some badly formatted text into /var/lib/snapd/apparmor/snap-confine/cap-bpf it does complain, clearly the file is read and #include works.
Maybe adding some bogus text to /var/lib/snapd/apparmor/snap-confine/nfs-support and then trying to reload the profile would prove that it’s really being used?
What if the profile is changed to network,? This should grant access to all network.
I guess one could also edit the profile and use (complain,attach_disconnected) for both s-c and mount namespace helper, which should only log the denials.
Strangly, the file /var/lib/snapd/apparmor/snap-confine/nfs-support is removed after every reboot.
$ file /var/lib/snapd/apparmor/snap-confine/nfs-support
/var/lib/snapd/apparmor/snap-confine/nfs-support: cannot open `/var/lib/snapd/apparmor/snap-confine/nfs-support' (No such file or directory)
$ journalctl -b -u snapd
mai 10 08:52:59 jammy systemd[1]: Starting Snap Daemon...
mai 10 08:53:00 jammy snapd[676]: AppArmor status: apparmor is enabled and all features are available
mai 10 08:53:00 jammy snapd[676]: overlord.go:263: Acquiring state lock file
mai 10 08:53:00 jammy snapd[676]: overlord.go:268: Acquired state lock file
mai 10 08:53:00 jammy snapd[676]: daemon.go:247: started snapd/2.55.3+22.04ubuntu1 (series 16; classic) ubuntu/22.04 (amd64) linux/5.15.0-27-generic.
mai 10 08:53:01 jammy snapd[676]: daemon.go:340: adjusting startup timeout by 1m30s (pessimistic estimate of 30s plus 5s per snap)
mai 10 08:53:10 jammy systemd[1]: Started Snap Daemon.
mai 10 08:53:11 jammy snapd[676]: storehelpers.go:721: cannot refresh: snap has no updates available: "bare", "cmake", "core", "core20", "dust", "firefox", "gnome-3-38-2004", "gtk-common-themes", "pdftk", "snap-store", "snapd", "snapd->
mai 10 08:53:11 jammy snapd[676]: autorefresh.go:539: auto-refresh : tous les snaps sont mis à jour
$ systemctl restart snapd
$ file /var/lib/snapd/apparmor/snap-confine/nfs-support
/var/lib/snapd/apparmor/snap-confine/nfs-support: ASCII text
$ journalctl -b -u snapd
mai 10 08:52:59 jammy systemd[1]: Starting Snap Daemon...
mai 10 08:53:00 jammy snapd[676]: AppArmor status: apparmor is enabled and all features are available
mai 10 08:53:00 jammy snapd[676]: overlord.go:263: Acquiring state lock file
mai 10 08:53:00 jammy snapd[676]: overlord.go:268: Acquired state lock file
mai 10 08:53:00 jammy snapd[676]: daemon.go:247: started snapd/2.55.3+22.04ubuntu1 (series 16; classic) ubuntu/22.04 (amd64) linux/5.15.0-27-generic.
mai 10 08:53:01 jammy snapd[676]: daemon.go:340: adjusting startup timeout by 1m30s (pessimistic estimate of 30s plus 5s per snap)
mai 10 08:53:10 jammy systemd[1]: Started Snap Daemon.
mai 10 08:53:11 jammy snapd[676]: storehelpers.go:721: cannot refresh: snap has no updates available: "bare", "cmake", "core", "core20", "dust", "firefox", "gnome-3-38-2004", "gtk-common-themes", "pdftk", "snap-store", "snapd", "snapd->
mai 10 08:53:11 jammy snapd[676]: autorefresh.go:539: auto-refresh : tous les snaps sont mis à jour
mai 10 08:54:54 jammy snapd[676]: main.go:155: Exiting on terminated signal.
mai 10 08:54:54 jammy snapd[676]: overlord.go:504: Released state lock file
mai 10 08:54:54 jammy systemd[1]: Stopping Snap Daemon...
mai 10 08:54:54 jammy systemd[1]: snapd.service: Deactivated successfully.
mai 10 08:54:54 jammy systemd[1]: Stopped Snap Daemon.
mai 10 08:54:54 jammy systemd[1]: snapd.service: Consumed 9.273s CPU time.
mai 10 08:54:54 jammy systemd[1]: Starting Snap Daemon...
mai 10 08:54:54 jammy snapd[7343]: AppArmor status: apparmor is enabled and all features are available
mai 10 08:54:54 jammy snapd[7343]: overlord.go:263: Acquiring state lock file
mai 10 08:54:54 jammy snapd[7343]: overlord.go:268: Acquired state lock file
mai 10 08:54:54 jammy snapd[7343]: daemon.go:247: started snapd/2.55.3+22.04ubuntu1 (series 16; classic) ubuntu/22.04 (amd64) linux/5.15.0-27-generic.
mai 10 08:54:54 jammy snapd[7343]: daemon.go:340: adjusting startup timeout by 1m30s (pessimistic estimate of 30s plus 5s per snap)
mai 10 08:54:54 jammy snapd[7343]: backend.go:133: snapd enabled NFS support, additional implicit network permissions granted
mai 10 08:55:03 jammy systemd[1]: Started Snap Daemon.
Nothing interesting here. I can’t tell why the file is not generated at the first snapd start at boot time. Is there a way to produce a debug from the snapd.service ?
@mardy could this be a race? I’m thinking that when snapd stars during boot, the nfs entries under /home aren’t mounted yet, as it goes through autofs, hence there’s no need for the snippet so it gets removed?
That’s right, as we use autofs to mount homes. But strangly, when I restart snapd, no home is mounted under /home still, so what does detect snapd which requires nfs-support file ?
snapd uses both the list of active mounts and /etc/fstab: https://github.com/snapcore/snapd/blob/master/osutil/nfs_linux.go#L31
@eoli3n, does yout /etc/fstab contain an entry with nfs4 type, or do you have just autofs there?
I have no NFS mount in the fstab. Only autofs, which mounts homes and some others. Just after reboot I only get that line
$ mount | grep nfs
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
Maybe snapd could test if autofs is used to mount homes as nfs
$ grep nfs /etc/auto.master.d/home
* -fstype=nfs,vers=4,rw,soft,rsize=8192,wsize=8192 nfsserver:/home/&
Or maybe something like
$ automount --dumpmaps | grep '/home' | grep nfs
* | -fstype=nfs,vers=4,rw,soft,rsize=8192,wsize=8192 nfsserver:/home/&
Hi @eoli3n, could you please try this snapd binary on your machine?
Just please be extra-careful with it, because I didn’t test it at all, so be ready to restore the original file:
sudo mv /usr/lib/snapd/snapd{,.old}
# assuming you saved the downloaded file in /tmp:
sudo cp /tmp/snapd /usr/lib/snapd/snapd
sudo systemctl restart snapd
and actually, rebooting the machine would be the best test. Please remember to restore the original snapd binary in any case, just to be safe. If you confirm that this test works, then we can hopefully provide a fix in the next version.
Please disregard my previous message; I did not know how autofs worked, and the fix I’ve proposed is not going to help in any way.
While we work on a long-term solution, a workaround you could apply is creating a systemd unit file that would be executed before the snapd unit and that would cause the mount of a NFS-backed home directory (it does not matter which one); because, in that case, when snapd starts it would detect the NFS mount and support it properly.
Ok, so what I did is to create a snapd service config override which force snapd to start later as
$ mkdir /lib/systemd/system/snapd.service.d
$ cat << EOF > /lib/systemd/system/snapd.service.d/test.conf
[Unit]
After=snapd.socket multi-user.target
EOF
$ systemctl daemon-reload
$ reboot
After reboot
$ file /var/lib/snapd/apparmor/snap-confine/nfs-support
/var/lib/snapd/apparmor/snap-confine/nfs-support: ASCII text
That workaround is ok, because at startup I automount some exports but snapd started before autofs.
Now, how to fix that one ?
$ firefox
cannot create user data directory: /home/p00000012366/snap/firefox/1300: Permission denied
Can you paste the corresponding apparmor denial from the logs? (journalctl -t audit, or just the relevant line is enough)
mai 12 08:41:52 jammy audit[759]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/proc/270476/cmdline" pid=759 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
mai 12 08:41:53 jammy audit[759]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/run/systemd/users/82623" pid=759 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
mai 12 08:41:57 jammy audit[270549]: AVC apparmor="DENIED" operation="capable" profile="/usr/lib/snapd/snap-confine" pid=270549 comm="snap-confine" capability=4 capname="fsetid"
mai 12 08:41:57 jammy audit[270638]: AVC apparmor="DENIED" operation="mkdir" profile="snap-update-ns.firefox" name="/usr/share/libreoffice/help/" pid=270638 comm="5" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
mai 12 08:41:57 jammy audit[270549]: AVC apparmor="DENIED" operation="open" profile="/usr/lib/snapd/snap-confine" name="/home/" pid=270549 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=82623 ouid=0
mai 12 08:42:37 jammy audit[270859]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=270859 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:37 jammy audit[270859]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=270859 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:37 jammy audit[270927]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=270927 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:37 jammy audit[270927]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=270927 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:38 jammy audit[270981]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=270981 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:38 jammy audit[270981]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=270981 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:38 jammy audit[271044]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=271044 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:38 jammy audit[271044]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=271044 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:39 jammy audit[271097]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=271097 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
mai 12 08:42:39 jammy audit[271097]: AVC apparmor="DENIED" operation="connect" profile="snap.snapd-desktop-integration.snapd-desktop-integration" name="/run/nscd/socket" pid=271097 comm="getent" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
Why are there lines related to nscd as I don’t use it anymore ? I just switch from nslcd+nscd to sssd.
I don’t see any line related to my directory creation problem with firefox.
That’s strange. Please double-check the permissions on that path, to verify that your user has read/write/execute permissions on it.
Also, those denials on snap-confine might have something to do with the problem. Can you please try manually creating a file /var/lib/snapd/apparmor/snap-confine/nfs-test with these contents:
/home r,
capability fsetid,
then run
sudo apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine.real
and try starting firefox again? (please post the denials logs again, in any case)
Hi @eoli3n! I suspect that that error that you are seeing is https://bugs.launchpad.net/snapd/+bug/1973321
Can you please check, if (after restarting snapd, of course) snaps do work if started from the “/” directory?
Ok, back to work, I need to fix this before the end of the week !
root@jammy:~# cat /var/lib/snapd/apparmor/snap-confine/nfs-test
/home r,
capability fsetid,
root@jammy:~# apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine.real
root@jammy:~#
Then try to start firefox with a user
cannot create user data directory: /home/p00000012366/snap/firefox/1670: Permission denied