What is the correct way to allow a snap package access a filesystem that is mounted via nfs?
I have seen many issues here that deal with the specific problem that the home folder is an nfs mount, but my problem seems to be more basic than that.
Since I am crafting the package myself, I have no problem giving it access to the “network” interface (one of the earlier reported problems seems to be that nfs access requires for this).
But it still does not work for me. I get this on the kernel log
[290079.508354] nfs: RPC call returned error 13
I get this in journalctl: Sep 06 18:17:00 rotatoria audit[29964]: AVC apparmor="DENIED" operation="sendmsg" profile="/snap/snapd/12883/usr/lib/snapd/snap-confine" pid=29964 comm="snap-confine" laddr=192.168.178.35 lport=744 faddr=192.168.178.3 fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"
Also, I have bind-mounted the nfs below /media (this is a trick that I think worked in the past to make it believe that “removable-media” plug is sufficient.)
for sockets you also want the network-bind plug … it also helps to run the snappy-debug tool from the snappy-debug snap in a second terminal while executing your app, that will give you some useful interface hints …
I’m also interested in this topic - I’m trying to set up a remote boot Raspberry PI with Ubuntu on a NFS root partition - everything works fine until I try to install MicroK8s and I get this error in the shell:
ubuntu@node1:~$ sudo snap install microk8s --classic
error: cannot perform the following tasks:
Run install hook of “microk8s” snap if present (run hook “install”: /snap/snapd/14063/usr/lib/snapd/snap-confine: error while loading shared libraries: libudev.so.1: cannot open shared object file: Permission denied)
And this in the audit log:
type=AVC msg=audit(1637626645.144:441): apparmor=“DENIED” operation=“sendmsg” profile="/snap/snapd/14063/usr/lib/snapd/snap-confine" pid=7307 comm=“snap-confine” laddr=xxxx lport=873 faddr=xxxx fport=2049 family=“inet” sock_type=“stream” protocol=6 requested_mask=“send” denied_mask=“send”
I found some articles online about adding ‘network, inet’ the profile but the profile keeps getting overwritten and I haven’t figured out which process is doing that yet
I have heard through the grapevine that these NFS home issues are getting sorted. I wonder if NFS homes that live under non-standard /home are supported now?
Hey James - unfortunately I do not have much insight whether or not changes have been made to snapd; however, I know that there is an agreed upon workaround published to the Snap documentation on how to change the default home directory for a user: https://snapcraft.io/docs/home-outside-home
The issue largely has to do with limitations with AppArmor. You still have to bind mount to /home from the looks of it, but you can store home somewhere else (like on an NFS server). You might just need to make sure that you have the network and network-bind slots connected, but I am not 100% on that.
Thanks for linking this bug in. Looks like the preferred method for setting up non-standard home directories is to use the workaround. It might interesting to try the suggested workaround with my pypi-server snap.
It is a basic snap that I use to test my “snap management” code. It reads packages in from home, so we could use it to test if reading packages on an NFS server actually works or not.
Issue #1 local system users with non-standard $HOME
$ sudo snap install pypi-server
pypi-server 1.5.0 from Jason Nucciarone (nuccitheboss) installed
$ pypi-server
Sorry, home directories outside of /home are not currently supported.
See https://forum.snapcraft.io/t/11209 for details.
$ pwd
/lhome/ubuntu
Issue #2 non-local users NFS homes (ssh into the server as my AD user and create ~/snap/pypi-server
~$ juju ssh jbemfv@100
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-132-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Tue Dec 13 19:28:53 CET 2022
System load: 0.04 Processes: 199
Usage of /: 32.4% of 45.53GB Users logged in: 0
Memory usage: 2% IPv4 address for eth0: 10.104.196.127
Swap usage: 0%
* Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s
just raised the bar for easy, resilient and secure K8s cluster deployment.
https://ubuntu.com/engage/secure-kubernetes-at-the-edge
78 updates can be applied immediately.
To see these additional updates run: apt list --upgradable
New release '22.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
*** System restart required ***
jbemfv@witty-lion:~$ pwd
/home/jbemfv
jbemfv@witty-lion:~$ mount | grep jbemfv
mynfsserver.com:/hpc_home/jbemfv on /home/jbemfv type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.254.253.183,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=10.254.253.183)
jbemfv@witty-lion:~$ pypi-server
WARNING: You are using deprecated arguments to pypiserver.
Please run `pypi-server --help` and update your command to align with the current interface.
In most cases, this will be as simple as just using
pypi-server run [args]
instead of
pypi-server [args]
jbemfv@witty-lion:~$ ls -la snap
total 93
drwxr-sr-x 5 jbemfv hpcusers 73 Dec 13 19:29 .
drwxr-sr-x 22 jbemfv hpcusers 4247 Dec 13 19:28 ..
drwxr-sr-x 4 jbemfv hpcusers 72 Jan 19 2022 lxd
drwxr-sr-x 4 jbemfv hpcusers 68 Dec 13 19:29 pypi-server
drwxr-sr-x 6 jbemfv hpcusers 112 Apr 29 2021 slurm
jbemfv@witty-lion:~$ exit
logout
Connection to 10.104.196.127 closed.
Now that we have used the snap as a non-local user and now have a ~/snap/pypi-server in the nfs we cannot uninstall the snap.
jbemfv@VDL900076:~$ juju ssh 100
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-132-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Tue Dec 13 19:29:38 CET 2022
System load: 0.08 Processes: 195
Usage of /: 32.4% of 45.53GB Users logged in: 0
Memory usage: 2% IPv4 address for eth0: 10.104.196.127
Swap usage: 0%
* Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s
just raised the bar for easy, resilient and secure K8s cluster deployment.
https://ubuntu.com/engage/secure-kubernetes-at-the-edge
78 updates can be applied immediately.
To see these additional updates run: apt list --upgradable
New release '22.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
*** System restart required ***
Last login: Tue Dec 13 19:26:36 2022 from 10.254.224.79
ubuntu@witty-lion:~$ sudo snap remove pypi-server
error: cannot perform the following tasks:
- Remove data for snap "pypi-server" (2) (unlinkat /home/jbemfv/snap/pypi-server/2: permission denied)
Running snappy-debug while trying to remove the snap shows
$ sudo snappy-debug
INFO: Following '/var/log/syslog'. If have dropped messages, use:
INFO: $ sudo journalctl --output=short --follow --all | sudo snappy-debug
kernel.printk_ratelimit = 0
= AppArmor =
Time: Dec 13 19:44:37
Log: apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/proc/2529473/cmdline" pid=1018 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/2529473/cmdline (read)
Suggestion:
* adjust program to not access '@{PROC}/@{pid}/cmdline'
= AppArmor =
Time: Dec 13 19:44:37
Log: apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/proc/2529473/cmdline" pid=1019 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/2529473/cmdline (read)
Suggestion:
* adjust program to not access '@{PROC}/@{pid}/cmdline'
= AppArmor =
Time: Dec 13 19:44:38
Log: apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/proc/2529573/cmdline" pid=1018 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/2529573/cmdline (read)
Suggestion:
* adjust program to not access '@{PROC}/@{pid}/cmdline'
= AppArmor =
Time: Dec 13 19:44:47
Log: apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/proc/2529595/cmdline" pid=1018 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/2529595/cmdline (read)
Suggestion:
* adjust program to not access '@{PROC}/@{pid}/cmdline'
= AppArmor =
Time: Dec 13 19:44:48
Log: apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/proc/851936/cmdline" pid=1018 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/851936/cmdline (read)
Suggestion:
* adjust program to not access '@{PROC}/@{pid}/cmdline'
= AppArmor =
Time: Dec 13 19:44:48
Log: apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/proc/2529646/cmdline" pid=1018 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/2529646/cmdline (read)
Suggestion:
* adjust program to not access '@{PROC}/@{pid}/cmdline'
= AppArmor =
Time: Dec 13 19:44:48
Log: apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/proc/2529647/cmdline" pid=1018 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=339533
File: /proc/2529647/cmdline (read)
Suggestion:
* adjust program to not access '@{PROC}/@{pid}/cmdline'
and
$ journalctl -u snapd
Dec 13 19:29:44 witty-lion runuser[2527380]: pam_unix(runuser:session): session opened for user jbemfv by (uid=0)
Dec 13 19:29:44 witty-lion runuser[2527380]: pam_unix(runuser:session): session closed for user jbemfv
Dec 13 19:29:44 witty-lion snapd[851936]: taskrunner.go:271: [change 37 "Remove data for snap \"pypi-server\" (2)" task] failed: unlinkat /home/jbemfv/snap/pypi-server/2: permission denied
Dec 13 19:44:18 witty-lion snapd[851936]: api_snaps.go:319: Installing snap "snappy-debug" revision unset
Dec 13 19:44:48 witty-lion runuser[2529646]: pam_unix(runuser:session): session opened for user jbemfv by (uid=0)
Dec 13 19:44:48 witty-lion runuser[2529646]: pam_unix(runuser:session): session closed for user jbemfv
Dec 13 19:44:48 witty-lion snapd[851936]: taskrunner.go:271: [change 39 "Remove data for snap \"pypi-server\" (2)" task] failed: unlinkat /home/jbemfv/snap/pypi-server/2: permission denied
Dec 13 19:48:33 witty-lion runuser[2530317]: pam_unix(runuser:session): session opened for user jbemfv by (uid=0)
Dec 13 19:48:33 witty-lion runuser[2530317]: pam_unix(runuser:session): session closed for user jbemfv
Dec 13 19:48:33 witty-lion snapd[851936]: taskrunner.go:271: [change 40 "Remove data for snap \"pypi-server\" (2)" task] failed: unlinkat /home/jbemfv/snap/pypi-server/2: permission denied
Thank you for reporting this issue and sorry that we have let you wait for so long. Please note that we work on a fix for non-local homedirs, there is even a test PPA available in
$ sudo apt-add-repository ppa:mardy/snapd-dev
$ sudo snap set system homedirs=/lhome
This should hopefully give you the ability to run on non-local homedirs.
For the issue of removing snaps for non-local user on NFS home - I wonder if this is related to the NFS root_squash option, so I wonder if you could set “no_root_squash” for the nfs dir and see if that is enough? This is mostly a workaround, we will need to look into this properly.
With the above setup, using /lhome for local system user home, snaps seem to work for me already without using sudo snap set system homedirs=/lhome.
I’m wondering if there is something I’m missing here…
Concerning NFS home dirs, we don’t have the option to enable root squash … but I think when I have tested in the past, that did enable the system root user to remove snaps.
