LXD + snaps + nvidia = 😢

snapd
1

I recently started working on a machine that has an nvidia card. As I often do, I launched an ephemeral, unprivileged LXD instance and tried to use a snap:

$ hello-world
cannot remount /tmp/snap.rootfs_o6dcBz/var/lib/snapd/lib/vulkan as read-only: Permission denied

This is caused by an apparmor denial:

[61955.378584] audit: type=1400 audit(1530126985.909:482): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-brave-drum_</var/snap/lxd/common/lxd>" name="/tmp/snap.rootfs_o6dcBz/var/lib/snapd/lib/vulkan/" pid=12945 comm="snap-confine" flags="ro, remount"

It turns out that, when running with nvidia hardware, snapd does a read-only remount in addition to its typical bind mounts, which is denied under LXD’s unprivileged confinement. A work around is to enable nesting, which lets the remount through:

$ lxc config set <container> security.nesting true

Given conversation on IRC with @stgraber and @zyga-snapd, snapd should be using MS_BIND|MS_REMOUNT here.

2 Likes
2

Thanks to @kyrofa and @stgraber for helping in uncovering and debugging the issue!

Thsi is now addressed by https://github.com/snapcore/snapd/pull/5423 - since I cannot test it myself I would like to ask @mborzecki to take this branch, build snapd and, without reexecution, try using snapd inside a container with this snap-confine (you can just build snap-confine and overwrite the executable inside your xenial container running on bionic host, for example). Remember to disable re-execution :slight_smile:

1 Like
3

@zyga-snapd I’ve pushed an update of snap-confine apparmor profile to your PR. The fix works with lxd from edge, version git-fe39a5a (7606). Host nvidia access in snaps works too.

1 Like