Problem with confined nvenc / cuda ffmpeg snap

One for @jdstrand I imagine :D.

The snap for ffmpeg which uses nvenc/cuda I made only works in devmode, but doesn’t spit out anything in snappy-debug security scanlog, so I’m a bit confused why it doesn’t work. I’m keen to incorporate this (or something like it) in a bunch of other snaps.

I’ve made a “simple” test case which I’d like to incorporate in another snap which will not be classic.

I’ve shoved it into launchpad. https://code.launchpad.net/~popey/+snap/nvffmpeg - there should be a snap appear there soon.

Here’s a test case.

$ /snap/bin/ffmpeg -loglevel error -y -i input.mp4 -vcodec nvenc -b:v 5M -acodec copy snap.mp4

The input is whatever mp4 video you have handy. Personally I use a downloaded copy of this video, but use whatever. When installed in devmode, it works, but in strict, you get this:-

[nvenc @ 0x2996180] Cannot init CUDA
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height                                               

I’m doing this on a laptop running Ubuntu 18.04 with nvidia GPU and nvidia binary driver 390.48.

I don’t have nvidia hardware so can’t test this snap.

Two things come to mind though:

  1. hardware necessary for this to work is not present in the device cgroup for the snap. devmode does not enforce device cgroups and there won’t be a logged denied access in strict mode (device cgroups are like traditional UNIX permissions in this regard).
  2. there is a capability apparmor denial whose log message is being rate limited. This is easy to workaround: stop the snap, load the apparmor profile with sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.ffmpeg.ffmpeg then launch the snap and see if a capability denial pops out in the logs

I’ve been working on the ffmpeg snap this evening. My version can be found here:

Like @popey when the snap is devmode or classic it works. It I try and use the accelerated nvidia NVENC (h264) or HEVC (h265) then I see then following on stdout:

[nvenc @ 0x1460380] Cannot init CUDA
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, r$

These are the corresponding denials in syslog:

May  2 18:26:10 skull kernel: [ 7984.861530] audit: type=1400 audit(1525281970.544:585): apparmor="DENIED" operation="open" profile="snap.ffmpeg-nvenc.ffmpeg-nvenc" name="/proc/sys/vm/mmap_min_addr" pid=25709 comm="ffmpeg" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
May  2 18:26:10 skull kernel: [ 7984.861615] audit: type=1400 audit(1525281970.544:586): apparmor="DENIED" operation="open" profile="snap.ffmpeg-nvenc.ffmpeg-nvenc" name="/proc/sys/vm/mmap_min_addr" pid=25709 comm="ffmpeg" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
May  2 18:26:10 skull kernel: [ 7984.861900] audit: type=1400 audit(1525281970.544:587): apparmor="DENIED" operation="open" profile="snap.ffmpeg-nvenc.ffmpeg-nvenc" name="/proc/devices" pid=25709 comm="ffmpeg" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
May  2 18:26:10 skull kernel: [ 7984.862195] audit: type=1400 audit(1525281970.544:588): apparmor="DENIED" operation="open" profile="snap.ffmpeg-nvenc.ffmpeg-nvenc" name="/sys/devices/system/memory/block_size_bytes" pid=25709 comm="ffmpeg" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
May  2 18:26:10 skull kernel: [ 7984.903014] audit: type=1400 audit(1525281970.584:589): apparmor="DENIED" operation="bind" profile="snap.ffmpeg-nvenc.ffmpeg-nvenc" pid=25709 comm="ffmpeg" family="unix" sock_type="seqpacket" protocol=0 requested_mask="bind" denied_mask="bind" addr="@637564612D75766D66642D343032363533313833362D323537303900"

I’ve also built in vaapi support but haven’t tested it yet.

I can add these accesses in the next batch of updates. Can you confirm that adding the below to the apparmor policy for the snap allows it to work?

@{PROC}/sys/vm/mmap_min_addr r,
@{PROC}/devices r,
/sys/devices/system/memory/block_size_bytes r,
unix (bind) type=seqpacket addr="@cuda-uvmfd-[0-9a-f]*",

(I supsect we are going to need some send/receive rules in addition to the above, but we won’t know what they are until we get past the bind denial)

Note, @{PROC}/devices is already under hardware-observe, but we can perhaps adjust that.

Note 2, I found the cuda abstract socket path with the aa-decode command.

@jdstrand Thanks for that. With those applied I now see a new denial:

May  3 16:01:09 skull kernel: [85683.018082] audit: type=1400 audit(1525359669.838:2417): apparmor="DENIED" operation="listen" profile="snap.ffmpeg.ffmpeg" pid=12034 comm="ffmpeg" family="unix" sock_type="seqpacket" protocol=0 requested_mask="listen" denied_mask="listen" addr="@637564612D75766D66642D343032363533313833362D313230333400"

So I change the last line of the AppArmor policy modification you provided to this:

unix (bind,listen) type=seqpacket addr="@cuda-uvmfd-[0-9a-f]*",

With the policy reloaded I now have a strictly confined ffmpeg that can offload video decoding and encoding to nvidia compute :slight_smile:

1 Like

FYI this was merged via https://github.com/snapcore/snapd/pull/5189