Request for classic confinement SPDK snap

Hello,

I would like to request classic confinement for the SPDK snap. Storage Performance Development Kit (SPDK) consists of scripts in bash and python and elf64 binaries, these utilities needs access to system commands and resources. For example the setup script reads/writes to /sys file system, unbinds native modules, sets up hugetlbfs etc. The snap would be used in a cloud/server environment.

For example when the confinement is set to strict. I get the following (please note this snap package was built with kernel-module-control and kernel-module-observe plugs for setup) :
ubuntu@sst100:~$ snap connections
Interface Plug Slot Notes
home spdk:home :home -
network spdk:network :network -
network-bind spdk:network-bind :network-bind -

ubuntu@sst100:~$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
lspci: Cannot open /sys/bus/pci/devices
/snap/spdk/x1/opt/spdk/scripts/setup.sh: line 210: /sbin/modprobe: Permission denied
ubuntu@sst100:~$

The yaml for the snap can be found here: https://github.com/manjo-git/snaps/tree/master/spdk-snap you can also find an example for how to setup NVMe-of over TCP using the snapped (using classic confinement) utilities. This example simply demonstrates how the utilities can be used on both ARM and Intel systems.

Thank you.

These sorts of requests are now meant to be filed in the #store-requests category. I seem to have lost my ability to recategorize posts though, so I can’t move it for you, but you should be able to move it yourself :slight_smile:

Sorry about that… I moved it.

Can I please get some review comments on this request ?

We have a number of interfaces for interacting with /sys, kernel-module-control for dealing with modules, etc. https://github.com/snapcore/snapd/pull/8271 will add configuration of hugepages (but hardware-observe allows read today). It appears that you didn’t connect all of the interfaces when testing. I suggest adding to plugs ‘kernel-module-control’, installing the snap in devmode, then doing:

$ for i in system-observe system-trace log-observe kernel-module-control ; do sudo snap connect spdk:$i ; done

then posting any policy violations here (preferably installing snappy-debug and examining its recommendations and applying them to the snap first).

Built spdk snap as stable strict, and the snap package installs ok.

ubuntu@sst100:~/Downloads$ sudo snap install --dangerous spdk_20.04_arm64.snap 
Mount snap "spdk" (unset)                                                      .
Mount snap "spdk" (unset)                                                      .
spdk 20.04 installed
ubuntu@sst100:~/Downloads$

Manual connections made as per your suggestion, I also added hardware-observe

ubuntu@sst100:~$ for i in system-observe system-trace log-observe kernel-module-control hardware-observe; do sudo snap connect spdk:$i ; done
[sudo] password for ubuntu: 

Need to use vfio-pci module with no-iommu. Load the module before running the setup script. This is documented in my readme.

ubuntu@sst100:~$ sudo modprobe vfio-pci
ubuntu@sst100:~$ lsmod | grep vfio
vfio_pci               61440  0
vfio_virqfd            20480  1 vfio_pci
vfio_iommu_type1       36864  0
vfio                   40960  2 vfio_iommu_type1,vfio_pci

Run the SPDK setup script. It fails accessing /sys/bus/pci/devices/. One of the functions that the script does is, it disassociates the inbox kernel driver for NVMe and associates it with userspace vfio driver.

ubuntu@sst100:~$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
/snap/spdk/x1/opt/spdk/scripts/setup.sh: line 104: /sys/bus/pci/devices/0007:01:00.0/driver/remove_id: Permission denied
/snap/spdk/x1/opt/spdk/scripts/setup.sh: line 105: /sys/bus/pci/devices/0007:01:00.0/driver/unbind: Permission denied

Check for connections.

ubuntu@sst100:~$ snap connections | grep spdk
hardware-observe       spdk:hardware-observe       :hardware-observe       manual
home                   spdk:home                   :home                   -
kernel-module-control  spdk:kernel-module-control  :kernel-module-control  manual
log-observe            spdk:log-observe            :log-observe            manual
network                spdk:network                :network                -
network-bind           spdk:network-bind           :network-bind           -
system-observe         spdk:system-observe         :system-observe         manual
system-trace           spdk:system-trace           :system-trace           manual
ubuntu@sst100:~$ 

Thanks for the additional information, but it seems you didn’t install the snap with --devmode. This will allow your snap all the accesses it needs but logging policy violations (which you can paste here) which will allow us to see the bigger picture for what your snap’s requirements are. When installing in devmode, please be sure to connect the interfaces after so the logged items are just new things your snap would need over what currently exists.

oops sorry about that. Yes I can redo with --devmode. Also could you please tell me if my for loop to make the connections above is correct? If I understand you correctly. I have to repeat the steps above after installing the snap with devmode collect the logs and post it here?

Remove existing SPDK snap

ubuntu@sst100:~/Downloads$ sudo snap remove spdk
spdk removed

Install the snap with --devmode

ubuntu@sst100:~/Downloads$ sudo snap install --devmode spdk_20.04_arm64.snap
spdk 20.04 installed

Make connections

ubuntu@sst100:~/Downloads$ for i in system-observe system-trace log-observe kernel-module-control hardware-observe; do sudo snap connect spdk:$i ; done

Modprobe vfio-pci in noiommu mode, and run the setup script. The setup script worked this time, please ignore the modprobe fatal message at the end. That module does not exist on ARM64.

ubuntu@sst100:~/Downloads$ sudo modprobe vfio-pci
ubuntu@sst100:~/Downloads$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
0007:01:00.0 (144d a808): nvme -> vfio-pci

Current user memlock limit: 16 MB

This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as current user.
To change this, please adjust limits.conf memlock limit for current user.

## WARNING: memlock limit is less than 64MB
## DPDK with VFIO may not be able to initialize if run as current user.
modprobe: FATAL: Module msr not found in directory /lib/modules/5.4.0-29-generic

I believe you will have to redo the loop after installing in devmode (but you can always check ‘snap connections spdk’).

What are the security policy violations from journald at the time you ran the snap?

In my post above I did re-run the for loop. Also snap connections for spdk are as follows:

ubuntu@sst100:~/build/git/build$ snap connections | grep spdk
hardware-observe       spdk:hardware-observe       :hardware-observe       manual
home                   spdk:home                   :home                   -
kernel-module-control  spdk:kernel-module-control  :kernel-module-control  manual
log-observe            spdk:log-observe            :log-observe            manual
network                spdk:network                :network                -
network-bind           spdk:network-bind           :network-bind           -
system-observe         spdk:system-observe         :system-observe         manual
system-trace           spdk:system-trace           :system-trace           manual
ubuntu@sst100:~/build/git/build$

There were no logs in journalctl. Here is what it looks like when I run the snap app.

ubuntu@sst100:~/build/git/build$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup reset 
0007:01:00.0 (144d a808): vfio-pci -> nvme

ubuntu@sst100:~/build/git/build$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
0007:01:00.0 (144d a808): nvme -> vfio-pci

Current user memlock limit: 16 MB

This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as current user.
To change this, please adjust limits.conf memlock limit for current user.

## WARNING: memlock limit is less than 64MB
## DPDK with VFIO may not be able to initialize if run as current user.
modprobe: FATAL: Module msr not found in directory /lib/modules/5.4.0-29-generic
ubuntu@sst100:~/build/git/build$

Here is the journalctl output when I run the snap app. There are no violations listed in the journald output.

May 20 20:32:40 sst100 audit[16636]: AVC apparmor="ALLOWED" operation="truncate"
May 20 20:32:40 sst100 audit[17039]: AVC apparmor="ALLOWED" operation="open" pro
May 20 20:32:40 sst100 audit[17040]: SECCOMP auid=1000 uid=0 gid=0 ses=3 pid=170
May 20 20:32:40 sst100 audit[17040]: AVC apparmor="ALLOWED" operation="chown" pr
May 20 20:32:40 sst100 audit[17041]: AVC apparmor="ALLOWED" operation="chmod" pr
May 20 20:32:40 sst100 sudo[16635]: pam_unix(sudo:session): session closed for u
lines 979-1001/1001 (END)[19543.364820] nvme nvme0: failed to set APST feature )

The “ALLOWED” entries are policy violations. Since you installed with --devmode, that puts the apparmor and seccomp profiles into complain mode, which allows but logs policy violations.

These log entries are truncated. Can you provide the full log output?

Sorry about that noise … I was also testing a VPP snap that I was working on. I rebooted the system and ran it fresh.

Snap app setup command.

ubuntu@sst100:~$ sudo modprobe vfio-pci
[sudo] password for ubuntu: 
ubuntu@sst100:~$ sudo HUGEMEM=8192 DRIVER_OVERRIDE=vfio-pci spdk.setup config
0007:01:00.0 (144d a808): nvme -> vfio-pci

Current user memlock limit: 16 MB

This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as current user.
To change this, please adjust limits.conf memlock limit for current user.

## WARNING: memlock limit is less than 64MB
## DPDK with VFIO may not be able to initialize if run as current user.
modprobe: FATAL: Module msr not found in directory /lib/modules/5.4.0-29-generic
ubuntu@sst100:~$ 

Full journalctl command output:

ubuntu@sst100:~$ journalctl -e 
May 20 22:32:57 sst100 vpp.vpp[507]: vpp[507]: dpdk_config: rte_eal_init returne
May 20 22:32:57 sst100 vpp[507]: dpdk_config: rte_eal_init returned -1
May 20 22:33:01 sst100 login[566]: pam_unix(login:session): session opened for u
May 20 22:33:01 sst100 systemd-logind[509]: New session 1 of user ubuntu.
May 20 22:33:01 sst100 systemd[1]: Created slice User Slice of ubuntu.
May 20 22:33:01 sst100 systemd[1]: Started Session 1 of user ubuntu.
May 20 22:33:01 sst100 systemd[1]: Starting User Manager for UID 1000...
May 20 22:33:01 sst100 systemd[783]: pam_unix(systemd-user:session): session ope
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG network certificate mana
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG cryptographic agent and 
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG cryptographic agent and 
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG cryptographic agent and 
May 20 22:33:01 sst100 systemd[783]: Reached target Paths.
May 20 22:33:01 sst100 systemd[783]: Listening on REST API socket for snapd user
May 20 22:33:01 sst100 systemd[783]: Listening on GnuPG cryptographic agent (ssh
May 20 22:33:01 sst100 systemd[783]: Reached target Sockets.
May 20 22:33:01 sst100 systemd[783]: Reached target Timers.
May 20 22:33:01 sst100 systemd[783]: Reached target Basic System.
May 20 22:33:01 sst100 systemd[1]: Started User Manager for UID 1000.
May 20 22:33:01 sst100 systemd[783]: Reached target Default.
May 20 22:33:01 sst100 systemd[783]: Startup finished in 83ms.
May 20 22:33:45 sst100 kernel: random: crng init done
May 20 22:33:45 sst100 kernel: random: 7 urandom warning(s) missed due to rateli
lines 1000-1022/1022 (END)[  108.466045] nvme nvme0: failed to set APST feature)
lines 1000-1022/1022 (END)

Please let me know what the next steps are?