I would like to request classic confinement for the SPDK snap. Storage Performance Development Kit (SPDK) consists of scripts in bash and python and elf64 binaries, these utilities needs access to system commands and resources. For example the setup script reads/writes to /sys file system, unbinds native modules, sets up hugetlbfs etc. The snap would be used in a cloud/server environment.
For example when the confinement is set to strict. I get the following (please note this snap package was built with kernel-module-control and kernel-module-observe plugs for setup) :
ubuntu@sst100:~$ snap connections
Interface Plug Slot Notes
home spdk:home :home -
network spdk:network :network -
network-bind spdk:network-bind :network-bind -
ubuntu@sst100:~$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
lspci: Cannot open /sys/bus/pci/devices
/snap/spdk/x1/opt/spdk/scripts/setup.sh: line 210: /sbin/modprobe: Permission denied
ubuntu@sst100:~$
The yaml for the snap can be found here: https://github.com/manjo-git/snaps/tree/master/spdk-snap you can also find an example for how to setup NVMe-of over TCP using the snapped (using classic confinement) utilities. This example simply demonstrates how the utilities can be used on both ARM and Intel systems.
These sorts of requests are now meant to be filed in the #store-requests category. I seem to have lost my ability to recategorize posts though, so I can’t move it for you, but you should be able to move it yourself
We have a number of interfaces for interacting with /sys, kernel-module-control for dealing with modules, etc. https://github.com/snapcore/snapd/pull/8271 will add configuration of hugepages (but hardware-observe allows read today). It appears that you didn’t connect all of the interfaces when testing. I suggest adding to plugs ‘kernel-module-control’, installing the snap in devmode, then doing:
$ for i in system-observe system-trace log-observe kernel-module-control ; do sudo snap connect spdk:$i ; done
then posting any policy violations here (preferably installing snappy-debug and examining its recommendations and applying them to the snap first).
Built spdk snap as stable strict, and the snap package installs ok.
ubuntu@sst100:~/Downloads$ sudo snap install --dangerous spdk_20.04_arm64.snap
Mount snap "spdk" (unset) .
Mount snap "spdk" (unset) .
spdk 20.04 installed
ubuntu@sst100:~/Downloads$
Manual connections made as per your suggestion, I also added hardware-observe
ubuntu@sst100:~$ for i in system-observe system-trace log-observe kernel-module-control hardware-observe; do sudo snap connect spdk:$i ; done
[sudo] password for ubuntu:
Need to use vfio-pci module with no-iommu. Load the module before running the setup script. This is documented in my readme.
Run the SPDK setup script. It fails accessing /sys/bus/pci/devices/. One of the functions that the script does is, it disassociates the inbox kernel driver for NVMe and associates it with userspace vfio driver.
ubuntu@sst100:~$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
/snap/spdk/x1/opt/spdk/scripts/setup.sh: line 104: /sys/bus/pci/devices/0007:01:00.0/driver/remove_id: Permission denied
/snap/spdk/x1/opt/spdk/scripts/setup.sh: line 105: /sys/bus/pci/devices/0007:01:00.0/driver/unbind: Permission denied
Thanks for the additional information, but it seems you didn’t install the snap with --devmode. This will allow your snap all the accesses it needs but logging policy violations (which you can paste here) which will allow us to see the bigger picture for what your snap’s requirements are. When installing in devmode, please be sure to connect the interfaces after so the logged items are just new things your snap would need over what currently exists.
oops sorry about that. Yes I can redo with --devmode. Also could you please tell me if my for loop to make the connections above is correct? If I understand you correctly. I have to repeat the steps above after installing the snap with devmode collect the logs and post it here?
ubuntu@sst100:~/Downloads$ for i in system-observe system-trace log-observe kernel-module-control hardware-observe; do sudo snap connect spdk:$i ; done
Modprobe vfio-pci in noiommu mode, and run the setup script. The setup script worked this time, please ignore the modprobe fatal message at the end. That module does not exist on ARM64.
ubuntu@sst100:~/Downloads$ sudo modprobe vfio-pci
ubuntu@sst100:~/Downloads$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
0007:01:00.0 (144d a808): nvme -> vfio-pci
Current user memlock limit: 16 MB
This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as current user.
To change this, please adjust limits.conf memlock limit for current user.
## WARNING: memlock limit is less than 64MB
## DPDK with VFIO may not be able to initialize if run as current user.
modprobe: FATAL: Module msr not found in directory /lib/modules/5.4.0-29-generic
There were no logs in journalctl. Here is what it looks like when I run the snap app.
ubuntu@sst100:~/build/git/build$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup reset
0007:01:00.0 (144d a808): vfio-pci -> nvme
ubuntu@sst100:~/build/git/build$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
0007:01:00.0 (144d a808): nvme -> vfio-pci
Current user memlock limit: 16 MB
This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as current user.
To change this, please adjust limits.conf memlock limit for current user.
## WARNING: memlock limit is less than 64MB
## DPDK with VFIO may not be able to initialize if run as current user.
modprobe: FATAL: Module msr not found in directory /lib/modules/5.4.0-29-generic
ubuntu@sst100:~/build/git/build$
Here is the journalctl output when I run the snap app. There are no violations listed in the journald output.
May 20 20:32:40 sst100 audit[16636]: AVC apparmor="ALLOWED" operation="truncate"
May 20 20:32:40 sst100 audit[17039]: AVC apparmor="ALLOWED" operation="open" pro
May 20 20:32:40 sst100 audit[17040]: SECCOMP auid=1000 uid=0 gid=0 ses=3 pid=170
May 20 20:32:40 sst100 audit[17040]: AVC apparmor="ALLOWED" operation="chown" pr
May 20 20:32:40 sst100 audit[17041]: AVC apparmor="ALLOWED" operation="chmod" pr
May 20 20:32:40 sst100 sudo[16635]: pam_unix(sudo:session): session closed for u
lines 979-1001/1001 (END)[19543.364820] nvme nvme0: failed to set APST feature )
The “ALLOWED” entries are policy violations. Since you installed with --devmode, that puts the apparmor and seccomp profiles into complain mode, which allows but logs policy violations.
These log entries are truncated. Can you provide the full log output?
Sorry about that noise … I was also testing a VPP snap that I was working on. I rebooted the system and ran it fresh.
Snap app setup command.
ubuntu@sst100:~$ sudo modprobe vfio-pci
[sudo] password for ubuntu:
ubuntu@sst100:~$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
0007:01:00.0 (144d a808): nvme -> vfio-pci
Current user memlock limit: 16 MB
This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as current user.
To change this, please adjust limits.conf memlock limit for current user.
## WARNING: memlock limit is less than 64MB
## DPDK with VFIO may not be able to initialize if run as current user.
modprobe: FATAL: Module msr not found in directory /lib/modules/5.4.0-29-generic
ubuntu@sst100:~$
Full journalctl command output:
ubuntu@sst100:~$ journalctl -e
May 20 22:32:57 sst100 vpp.vpp[507]: vpp[507]: dpdk_config: rte_eal_init returne
May 20 22:32:57 sst100 vpp[507]: dpdk_config: rte_eal_init returned -1
May 20 22:33:01 sst100 login[566]: pam_unix(login:session): session opened for u
May 20 22:33:01 sst100 systemd-logind[509]: New session 1 of user ubuntu.
May 20 22:33:01 sst100 systemd[1]: Created slice User Slice of ubuntu.
May 20 22:33:01 sst100 systemd[1]: Started Session 1 of user ubuntu.
May 20 22:33:01 sst100 systemd[1]: Starting User Manager for UID 1000...
May 20 22:33:01 sst100 systemd[783]: pam_unix(systemd-user:session): session ope
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG network certificate mana
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG cryptographic agent and
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG cryptographic agent and
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG cryptographic agent and
May 20 22:33:01 sst100 systemd[783]: Reached target Paths.
May 20 22:33:01 sst100 systemd[783]: Listening on REST API socket for snapd user
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG cryptographic agent (ssh
May 20 22:33:01 sst100 systemd[783]: Reached target Sockets.
May 20 22:33:01 sst100 systemd[783]: Reached target Timers.
May 20 22:33:01 sst100 systemd[783]: Reached target Basic System.
May 20 22:33:01 sst100 systemd[1]: Started User Manager for UID 1000.
May 20 22:33:01 sst100 systemd[783]: Reached target Default.
May 20 22:33:01 sst100 systemd[783]: Startup finished in 83ms.
May 20 22:33:45 sst100 kernel: random: crng init done
May 20 22:33:45 sst100 kernel: random: 7 urandom warning(s) missed due to rateli
lines 1000-1022/1022 (END)[ 108.466045] nvme nvme0: failed to set APST feature)
lines 1000-1022/1022 (END)
The output you posted is still truncated (each line appears to be cut off at the end) and it doesn’t appear to list any policy violations. If you are still experiencing issues, can you please open a terminal, and capture the output - something like the following should work:
Thanks - the output is not truncated this time but I cannot see any evidence that strict confinement is causing any issues for spdk from this output - there are no policy violations - is spdk operating correctly?
@manjo - since we’ve not heard back from you, we are removing this request from our review queue. When you have more time to respond, simply do so here and we can add the request back to the queue. Thanks
Sorry, for the delayed response. With strict confinement SPDK works partially, ie NVMEof over TCP works, but for NVMEof over RDMA to work it needs to access the _re0 devices etc and that fails. But using a classic confinement I am able to use all the features of SPDK.