Snap with CUDA 12.9 on Ubuntu Core 24

snapcraft
1

I am trying to make a cuda-utilities snap that gets CUDA working on Ubuntu Core 24. The pc-kernel snap for UC24 comes with nvidia-drivers-550. However, nvidia-drivers-550 are compatible with CUDA 12.4 which is only available for Ubuntu 20 and 22, but not 24.

Therefore, I installed CUDA 12.9 in the snap together with the cuda-compat-12-9 which allows to use the newer CUDA with the older drivers. The cuda-compat-12-9 contains libcuda.so.575.57.08 which replaces (I think, not really sure how this works) libcuda.so.550.163.01 from the pc-kernel snap which is made available through the gpu-2404 slot provided by mesa-2404.

My question: The snap only works with devmode and I cannot figure out which plug/interface/connection I need to fix this. snappy-debug reports that it tries to access /dev/char/... which is not allowed. I guess this is because of the cuda-compat-12-9 package, which does driver related stuff and needs direct access to the GPU.

  1. Is there an interface I can use to get this working?
  2. As far as I know there is no equivalent of classic confinement on Ubuntu Core, right? Are there any updates on this?
  3. What solution do you propose for now?

How to reproduce: You can reproduce this by building the snap and install it on an Ubuntu Core 24 system. The kernel should be installed as follows (instructions: scroll down to Nvidia GPU container pass-through support on Ubuntu Core 24 hosts):

snap refresh pc-kernel --channel 24/edge/nvidia-components-sdp
snap install pc-kernel+nvidia-550-erd-ko
modprobe nvidia_drm modeset=1
sudo modprobe nvidia_uvm
snap install pc-kernel+nvidia-550-erd-user

Running cuda-utilities.deviceQuery with the snap installed in devmode gives a proper output. However, when build without devmode confinement, I get an error about forward compatability is not supported for the hardware. When running the snap built with devmode confinement in jailmode, I get a different error about an incompatible OS or so.

Click to show snapcraft.yaml 
name: cuda-utilities
base: core24
version: "v12.9+build-0.1" # just for humans, typically '1.2+git' or '1.3.2'
summary: Implements the CUDA Samples Utilities
description: |
  This snap implements the Utilities from the CUDA samples repository to be able to run them on
  Ubuntu Core 24 using the `gpu-2404` interface. It also provides the CUDA toolkit on the `cuda`
  slot. This allows other snaps to connect to this slot and access the CUDA files without having
  to install them.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

parts:
  # This part installs CUDA 12.9; The current (09-07-2025) pc-kernel snap for Ubuntu Core 24 uses nvidia-driver-550
  # which requires CUDA 12.4. However, the package `cuda-compat-12-9` allows to use older driver versions with newer 
  # CUDA versions (more info: https://docs.nvidia.com/deploy/cuda-compatibility/#forward-compatibility).
  cuda-12-9:
    plugin: nil
    override-build: |
      # Install the CUDA 12.9 toolkit and add an forward-compatibility package to work with older nvidia drivers
      wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
      dpkg -i cuda-keyring_1.1-1_all.deb
      apt-get update
      apt-get -y install cuda-compat-12-9 cuda-toolkit-12-9
      # The cuda-compat-12-9 package provides the /usr/local/cuda-12.9/compat library which should replace the kernel
      # userspace libraries in $SNAP/gpu-2404-2/usr/lib/x86_64-linux-gnu

      ### cp the necessary files because `apt install` installs in `/usr` etc. and not in
      ### `/root/parts/install-cuda/install`
      mkdir -p $SNAPCRAFT_PART_INSTALL/usr/local/
      mkdir -p $SNAPCRAFT_PART_INSTALL/usr/lib/
      cp -r /usr/local/cuda-12.9 $SNAPCRAFT_PART_INSTALL/usr/local/cuda-12.9
      cp -r /usr/lib/x86_64-linux-gnu/ $SNAPCRAFT_PART_INSTALL/usr/lib/x86_64-linux-gnu/
    build-packages:
      - wget
    prime: # Copies from the install folder to the final snap
      # TODO: Make this more fine-grained. Now it copies everything, not only the CUDA files. Not sure if this is a problem.
      - usr/local/cuda-12.9
      - usr/lib/x86_64-linux-gnu

  # This part installs the utilities from the CUDA samples.
  cuda-samples-utilities:
    after: [cuda-12-9]
    plugin: dump
    source-type: git
    source-branch: v12.9 # Matches the CUDA version
    source-depth: 1
    source: https://github.com/NVIDIA/cuda-samples.git
    override-build: |
      ### Add the nvcc compiler to the PATH
      export PATH=/root/parts/cuda-12-9/install/usr/local/cuda-12.9/bin:$PATH

      ### Build the scripts in Samples/1_Utilities
      cmake $CRAFT_PART_SRC/Samples/1_Utilities
      make -j$(nproc)
      mkdir -p $CRAFT_PART_INSTALL/bin
      cp $CRAFT_PART_BUILD/deviceQuery/deviceQuery $CRAFT_PART_INSTALL/bin/deviceQuery
      cp $CRAFT_PART_BUILD/deviceQueryDrv/deviceQueryDrv $CRAFT_PART_INSTALL/bin/deviceQueryDrv
      cp $CRAFT_PART_BUILD/topologyQuery/topologyQuery $CRAFT_PART_INSTALL/bin/topologyQuery
    build-packages:
      - cmake
      - make
    prime:
      - bin/deviceQuery
      - bin/deviceQueryDrv
      - bin/topologyQuery

  # Run this part as last -- from https://github.com/canonical/gpu-snap
  gpu-2404:
    after: [cuda-12-9, cuda-samples-utilities]
    source: https://github.com/canonical/gpu-snap.git
    plugin: dump
    override-prime: |
      craftctl default
      ${CRAFT_PART_SRC}/bin/gpu-2404-cleanup mesa-2404
      # Workaround for https://bugs.launchpad.net/snapd/+bug/2055273
      mkdir -p "${CRAFT_PRIME}/gpu-2404"
    prime:
      - bin/gpu-2404-wrapper

plugs:
  # The gpu-2404 interface is provided by the `mesa-2404` snap an provides access to two paths: https://github.com/canonical/mesa-2404/blob/b2219e87c52a5bf6b06e5ebe5e43961642885eea/snap/snapcraft.yaml#L470-L475
  # - $SNAP                         => Mounted at the `target: <target_dir>` set below
  # - $SNAP_DATA/kernel-gpu-2404    => Mounted at `<target_dir>-2` by the bin/gpu-2404-wrapper script which calls this `mesa-2404` script: https://github.com/canonical/mesa-2404/blob/b2219e87c52a5bf6b06e5ebe5e43961642885eea/scripts/bin/gpu-2404-provider-wrapper.in#L4-L5
  gpu-2404:
    interface: content
    target: $SNAP/gpu-2404
    default-provider: mesa-2404

slots:
  cuda:
    interface: content
    read:
      - $SNAP/usr/local/cuda12-9
      - $SNAP/usr/lib/x86_64-linux  # Determine which files - i.e., not all of them

apps:
  deviceQuery:
    environment: &cuda-environment
      # LD_LIBRARY_PATH: $LD_LIBRARY_PATH:$SNAP/usr/local/cuda-12.9/targets/x86_64-linux/lib
      LD_LIBRARY_PATH: $SNAP/usr/local/cuda-12.9/compat:$SNAP/usr/local/cuda-12.9/targets/x86_64-linux/lib:$LD_LIBRARY_PATH
    command-chain:
      - bin/gpu-2404-wrapper
    command: bin/deviceQuery
    plugs:
      - opengl
      - system-observe
      - hardware-observe

  deviceQueryDrv:
    environment: *cuda-environment
    command-chain:
      - bin/gpu-2404-wrapper
    command: bin/deviceQueryDrv
    plugs:
      - opengl
      - system-observe
      - hardware-observe

  topologyQuery:
    environment: *cuda-environment
    command-chain:
      - bin/gpu-2404-wrapper
    command: bin/topologyQuery
    plugs:
      - opengl
      - system-observe
      - hardware-observe
2

Never mind, the problem is fixed by adding the network and network-bind plugs. Although I do not really understand what network access has to do with the driver…