EGL-using snaps on impish seem to be broken when using the Nvidia proprietary driver

Saviq · October 20, 2021, 3:41pm

Historically it’s been snapcraft-desktop-helpers that set up the environment:

ubuntu/snapcraft-desktop-helpers/blob/master/common/desktop-exports

###############################################
# Launcher common exports for any desktop app #
###############################################

function prepend_dir() {
  local var="$1"
  local dir="$2"
  if [ -d "$dir" ]; then
    eval "export $var=\"\$dir\${$var:+:\$$var}\""
  fi
}

function append_dir() {
  local var="$1"
  local dir="$2"
  if [ -d "$dir" ]; then
    eval "export $var=\"\${$var:+\$$var:}\$dir\""
  fi
}

This file has been truncated. show original

It’s likely part of most desktop snaps.

alan_g · October 20, 2021, 3:45pm

I don’t know where it comes from, but something is prefixing /var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void: to LD_LIBRARY_PATH even in snaps that avoid snapcraft-desktop-helpers.

E.g.

$ snap run --shell ubuntu-frame -c "echo \$LD_LIBRARY_PATH"
...
/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/ubuntu-frame/489/graphics/lib:/snap/ubuntu-frame/489/usr/lib:/snap/ubuntu-frame/489/usr/lib/x86_64-linux-gnu

I don’t see it anywhere in the snap itself:

$ grep -R gl32 /snap/ubuntu-frame/current/
$

[update]

After poking around the snapd code it comes from SNAP_LIBRARY_PATH via snapcraft-runner:

$ grep -R SNAP_LIBRARY_PATH /snap/ubuntu-frame/current/
/snap/ubuntu-frame/current/snap/command-chain/snapcraft-runner:export LD_LIBRARY_PATH="$SNAP_LIBRARY_PATH${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

alan_g · October 20, 2021, 4:05pm

@Wimpress starting from your original snap recipe, does adding this to your wrapper script help:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH#$SNAP_LIBRARY_PATH:}

(It is probably better not to do this on systems where the Nvidia drivers work, but a useful data point.)

Wimpress · October 20, 2021, 5:01pm

Yes, I am using the helpers from the GNOME extension.

Wimpress · October 20, 2021, 5:03pm

The opengl interface in snapd does that:

ijohnson · October 20, 2021, 5:22pm

Note that while snapd creates those files / mounts, it doesn’t set any env vars related to those

Wimpress · October 20, 2021, 7:06pm

alan_g:

starting from your original snap recipe, does adding this to your wrapper script help:
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH#$SNAP_LIBRARY_PATH:}
(It is probably better not to do this on systems where the Nvidia drivers work, but a useful data point.)

Upgraded my system at home to 21.10, which is also NVIDIA only.

Built the OSB Studio snap (actually a test version called obs-demo) locally from my original yaml and added export LD_LIBRARY_PATH=${LD_LIBRARY_PATH#$SNAP_LIBRARY_PATH:} to the wrapper just before the exec.

OBS Studio is working.

I grabbed the LD_LIBRARY_PATH, before and after adding that export to the wrapper. This is the result. I’ve highlighted what is removed from the Before:

Before: /var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/obs-demo/x18/opt/qt515/lib::/snap/obs-demo/x18/lib:/snap/obs-demo/x18/usr/lib:/snap/obs-demo/x18/lib/x86_64-linux-gnu:/snap/obs-demo/x18/usr/lib/x86_64-linux-gnu:/snap/obs-demo/x18/lib/x86_64-linux-gnu:/snap/obs-demo/x18/usr/lib/x86_64-linux-gnu:/snap/obs-demo/x18/usr/lib:/snap/obs-demo/x18/lib:/snap/obs-demo/x18/usr/lib/x86_64-linux-gnu/dri:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl/vdpau:/snap/obs-demo/x18/usr/lib/x86_64-linux-gnu/pulseaudio
After: /snap/obs-demo/x18/opt/qt515/lib::/snap/obs-demo/x18/lib:/snap/obs-demo/x18/usr/lib:/snap/obs-demo/x18/lib/x86_64-linux-gnu:/snap/obs-demo/x18/usr/lib/x86_64-linux-gnu:/snap/obs-demo/x18/lib/x86_64-linux-gnu:/snap/obs-demo/x18/usr/lib/x86_64-linux-gnu:/snap/obs-demo/x18/usr/lib:/snap/obs-demo/x18/lib:/snap/obs-demo/x18/usr/lib/x86_64-linux-gnu/dri:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl/vdpau:/snap/obs-demo/x18/usr/lib/x86_64-linux-gnu/pulseaudio

The NVIDIA drivers still work and that is due to lines 94 - 107 of the desktop-exports from the GNOME extension in Snapcraft:

https://github.com/snapcore/snapcraft/blob/master/extensions/desktop/common/desktop-exports#L94

@ijohnson What is /var/lib/snapd/void?

tamas · October 20, 2021, 7:12pm

That must be the root cause of How to set LD_LIBRARY_PATH properly? then.

ijohnson · October 20, 2021, 7:52pm

/var/lib/snapd/void is the directory where snap-confine will go when it is not safe/allowed to set the current working directory to anything else. If that directory is showing up in a variable then some part of the wrapper scripts is probably using $PWD (incorrectly).

Wimpress · October 20, 2021, 7:59pm

The wrapper does not use $PWD at all. $SNAP_LIBRARY_PATH appears to be providing /var/lib/snapd/void.

alan_g · October 20, 2021, 7:59pm

It is using SNAP_LIBRARY_PATH, which is set by snapd

Wimpress · October 20, 2021, 8:08pm

I added echo "SNAP_LIB: ${SNAP_LIBRARY_PATH}" to the wrapper, it outputs this:

SNAP_LIB: /var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void

ijohnson · October 20, 2021, 8:10pm

Ah okay, I actually wasn’t aware of SNAP_LIBRARY_PATH, apparently we do set this https://github.com/snapcore/snapd/blob/master/snap/snapenv/snapenv.go#L93 and in fact hard-code /var/lib/snapd/void there for no apparent reason

ijohnson · October 20, 2021, 8:11pm

Ah I guess we include /var/lib/snapd/void there purely to prevent foot shooting mistakes:

Wimpress · October 20, 2021, 8:16pm

Thanks for the info @ijohnson. So, my workaround to get the OBS Studio snap working on 21.10 for NVIDIA users, while preserving $SNAP_LIBRARY_PATH, is adding these two lines to my wrapper:

export LD_LIBRARY_PATH="${LD_LIBRARY_PATH#$SNAP_LIBRARY_PATH:}"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${SNAP_LIBRARY_PATH}"

Perhaps snapd should append SNAP_LIBRARY_PATH to LD_LIBRARY_PATH, rather than prepend it?

Thanks to everyone who commented on this topic

ijohnson · October 20, 2021, 8:23pm

I think you mean the snapcraft extension helpers should be appending?

Wimpress · October 20, 2021, 8:37pm

The desktop helpers and extensions in Snapcraft do not reference SNAP_LIBRARY_PATH at all. Looks like snapcraft does though:

https://github.com/snapcore/snapcraft/blob/fcdd283cdb86ce9ad9f3807765658092ac46f32e/snapcraft/internal/meta/_snap_packaging.py#L581

Tagging @sergiusens

alan_g · October 20, 2021, 8:41pm

It’s this script (in the snap):

snap/command-chain/snapcraft-runner

alan_g · October 20, 2021, 8:48pm

It’s snapcraft setting LD_LIBRARY_PATH, but the point of prepending is to pick up the host Nvidia GL library, not a mesa one from the snap content. Appending would defeat that. (The problem here is that it is problematic when the host and base userspace isn’t compatible.)

Wimpress · October 20, 2021, 8:48pm

That LD_LIBRARY_PATH is baked into the snaps. Which means, if fixing this in snapcraft is the right place to address this, all affected snaps will need to be rebuilt and published in order that NVIDIA users can run them on 21.10 and the next LTS.

Perhaps updating the desktop extensions in Snapcraft and the legacy desktop-helpers, with a workaround like the one I am using, may also help. But still requires snaps are rebuilt and published.