Vulkan is broken on snaps when using Nvidia Proprietary Drivers

Hi everyone,

I’ve been working on packaging some wine applications as snaps and I discovered something interesting.
It looks like Vulkan is broken on many snaps I tried including my own.
I tried the following on Ubuntu 20.04/19.10 and Manjaro.

For example the Retroarch snap has broken vulkan implementation

Steps to reproduce:

  1. Ubuntu 18.04/19.10/20.04
  2. the NVIDIA Proprietary Drivers
  3. sudo apt install vulkan-utils
  4. run vulkaninfo to check that you can get the latest vulkan info also you can run vkcube to make sure your computer can render properly using vulkan.
  5. sudo snap install retroarch
  6. launch retro arch and go to settings>video>output>video and change the driver to vulkan press Esc a few times until the application closes.
  7. try to relaunch the app and see that it crashes right away.

I did some more investigations inside the retroarch snap shell and my own snaps using
snap run --shell retroarch

when running the following command

ls -lh $SNAP/usr/share/vulkan/icd.d/

the output only returns the json files for intel and radeon card

-rw-r--r-- 1 root root 163 Feb 19 07:09 intel_icd.x86_64.json
-rw-r--r-- 1 root root 164 Feb 19 07:09 radeon_icd.x86_64.json

back on my system I ran the following command
cat /usr/share/vulkan/icd.d/nvidia_icd.json
and got the following json output

{
    "file_format_version" : "1.0.0",
    "ICD": {
        "library_path": "libGLX_nvidia.so.0",
        "api_version" : "1.1.119"
    }
}

It looks like the libGLX_nvidia.so.0 is the file its looking for

Back inside the snap I ran the following to try and find the nvidia so file
ls -lh $SNAP/usr/lib/x86_64-linux-gnu |grep -e 'libvul' -e 'libGLX_
and it seems to be missing

lrwxrwxrwx  1 root root    16 Feb 19 07:09 libGLX_indirect.so.0 -> libGLX_mesa.so.0
lrwxrwxrwx  1 root root    20 Feb 19 07:09 libGLX_mesa.so.0 -> libGLX_mesa.so.0.0.0
-rw-r--r--  1 root root  477K Feb 19 07:09 libGLX_mesa.so.0.0.0
lrwxrwxrwx  1 root root    14 Jan 25  2019 libvulkan.so -> libvulkan.so.1
lrwxrwxrwx  1 root root    19 Jan 25  2019 libvulkan.so.1 -> libvulkan.so.1.1.70
-rw-r--r--  1 root root  315K Jan 25  2019 libvulkan.so.1.1.70
-rw-r--r--  1 root root  5.6M Feb 19 07:09 libvulkan_intel.so
-rw-r--r--  1 root root  5.1M Feb 19 07:09 libvulkan_radeon.so

in my own snaps I also tried to add the following since i read it might help

layout:
  /usr/share/vulkan:
    symlink: $SNAP/usr/share/vulkan
  /usr/lib/$SNAPCRAFT_ARCH_TRIPLET/libGLX_nvidia.so.0:
    symlink: $SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/libGLX_nvidia.so.0
  /usr/lib/$SNAPCRAFT_ARCH_TRIPLET/libvulkan_intel.so:
    symlink: $SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/libvulkan_intel.so
  /usr/lib/$SNAPCRAFT_ARCH_TRIPLET/libvulkan_radeon.so:
    symlink: $SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/libvulkan_radeon.so

also inside my own snap i’ve installed the vulkan utils and when running vulkaninfo I get the following

===========
VULKAN INFO
===========

Vulkan Instance Version: 1.1.70

/build/vulkan-UL09PJ/vulkan-1.1.70+dfsg1/demos/vulkaninfo.c:2700: failed with VK_ERROR_INITIALIZATION_FAILED

Additional info about my Nvidia drivers

cat /sys/module/nvidia/version 
440.64

Additional info about snapd

snap --version
snap    2.44.1
snapd   2.44.1
series  16
ubuntu  20.04
kernel  5.4.0-21-generic

Feel free to ask any questions.

-DM

As discussed on IRC, the nvidia’s ICD file is available under /var/lib/snapd/lib/vulkan/icd.d/nvidia_icd.json inside the snap. There is a bunch of other ICD files coming from Mesa under $SNAP/usr/share/vulkan/icd.d/. I think that we should follow https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#icd-discovery and set the VK_ICD_FILENAMES to : delimited glob of all *.json filenames under those two locations., eg:

VK_ICD_FILENAMES=/var/lib/snapd/lib/vulkan/icd.d/nvidia_icd.json:$SNAP/usr/share/vulkan/icd.d/radeon.x86_64.json:...

The ICD vendor can use an absolute path to the given ICD library inside the file. Quick check reveals that nvidia does not, but mesa implementation does. In this case, the snap should use the layouts to place the relevant library files from the $SNAP/usr/lib/... under /usr/lib/...

this might be worth adding to the gnome-3-*, and any KDE/Qt extensions, and documenting in the snapcraft-desktop-helpers repo, too. cc/ @kenvandine, @hellsworth

An alternative is for the VK variable to be set by snapd if it is exposing the host files… cc/ @zyga-snapd