Custom ubuntu-image including custom gadget for microk8s strict snap

Hello all, I am wondering if I am missing something. When building a custom image for arm64 that includes a microk8s strict/stable snap and a custom gadget snap with the explicit plugins and implicit slots my image fails to boot properly and restarts infinitely.

My model file:

{
    "type": "model",
    "series": "16",
    "model": "custom-ubuntu-core-26-pi-arm64",
    "architecture": "arm64",
    "authority-id": "my-auth-id",
    "brand-id": "my-brand-id",
    "timestamp": "2026-09-23T15:24:42+00:00",
    "base": "core26",
    "grade": "dangerous",
    "snaps": [
        {
            "name": "my-gadget-pi",
            "type": "gadget"
        },
        {
            "name": "pi-kernel",
            "type": "kernel",
            "default-channel": "26/stable",
            "id": "jeIuP6tfFrvAdic8DMWqHmoaoukAPNbJ"
        },
        {
            "name": "core26",
            "type": "base",
            "default-channel": "latest/stable",
            "id": "cUqM61hRuZAJYmIS898Ux66VY61gBbZf"
        },
        {
            "name": "snapd",
            "type": "snapd",
            "default-channel": "latest/stable",
            "id": "PMrrV4ml8uWuEUDBT8dSGnKUYbevVhc4"
        },
        {
            "name": "console-conf",
            "type": "app",
            "default-channel": "26/stable",
            "id": "ASctKBEHzVt3f1pbZLoekCvcigRjtuqw"
        },
        {
            "name": "microk8s",
            "type": "app",
            "default-channel": "1.36-strict/stable",
            "id": "EaXqgt1lyCaxKaQCU349mlodBkDCXRcg"
        },
        {
            "name": "core22",
            "type": "base",
            "default-channel": "latest/stable",
            "id": "amcUKQILKXHHTlmSa7NMdnXSx02dNeeT"
        }
    ]
}

My gadget.yaml:

volumes:
  pi:
    schema: mbr
    bootloader: piboot
    structure:
      - name: ubuntu-seed
        role: system-seed
        filesystem: vfat
        type: 0C
        size: 1200M
        content:
          # Exposed kernel-assets are not needed since the migration to piboot.
          # However, they can't be fully removed for now as snapd requires to
          # reference at least one of the assets exposed by the kernel.
          # TODO: remove this dtb overlay unnecessary copy when snapd has an
          # option to ignore unused kernel-assets
          - source: $kernel:dtbs/dtbs/overlays/
            target: /overlays
          - source: boot-assets/
            target: /
      - name: ubuntu-boot
        role: system-boot
        filesystem: vfat
        type: 0C
        # whats the appropriate size?
        size: 750M
      # NOTE: ubuntu-save is optional for unencrypted devices like the pi, so
      # this structure can be dropped in favor of a different partition for
      # users who wish to instead use a different partition, since with MBR we
      # are limited to only 4 primary partitions.
      # TODO: look into switching over to GPT, the pi bootloader firmware now
      #       has support for this
      - name: ubuntu-save
        role: system-save
        filesystem: ext4
        type: 83,0FC63DAF-8483-4772-8E79-3D69D8477DE4
        # min-size to allow remodeling from previous releases
        min-size: 16M
        size: 32M
      - name: ubuntu-data
        role: system-data
        filesystem: ext4
        type: 83,0FC63DAF-8483-4772-8E79-3D69D8477DE4
        # XXX: make auto-grow to partition
        size: 1500M
connections:
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:account-control
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:cifs-mount
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:ceph-conf
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:configuration
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:docker-privileged
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:docker-unprivileged
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:firewall-control
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:fuse-support
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:hardware-observe
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:home
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:home-read-all
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:kernel-crypto-api
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:kernel-module-control
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:kernel-module-observe
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:k8s-journald
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:k8s-kubelet
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:k8s-kubeproxy
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:kubernetes-support
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:log-observe
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:login-session-observe
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:mount-observe
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:network
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:network-bind
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:network-control
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:network-observe
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:opengl
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:dot-config-helm
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:dot-kube
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:process-control
  - plug: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg:system-observe

microk8s snap connections:

Interface              Plug                            Slot                    Notes
account-control        microk8s:account-control        :account-control        -
cifs-mount             microk8s:cifs-mount             -                       -
content                -                               microk8s:microk8s       -
content                microk8s:ceph-conf              -                       -
content                microk8s:configuration          -                       -
docker-support         microk8s:docker-privileged      :docker-support         -
docker-support         microk8s:docker-unprivileged    :docker-support         -
firewall-control       microk8s:firewall-control       :firewall-control       -
fuse-support           microk8s:fuse-support           -                       -
hardware-observe       microk8s:hardware-observe       :hardware-observe       -
home                   microk8s:home                   :home                   -
home                   microk8s:home-read-all          :home                   -
kernel-crypto-api      microk8s:kernel-crypto-api      -                       -
kernel-module-control  microk8s:kernel-module-control  -                       -
kernel-module-observe  microk8s:kernel-module-observe  :kernel-module-observe  -
kubernetes-support     microk8s:k8s-journald           :kubernetes-support     -
kubernetes-support     microk8s:k8s-kubelet            :kubernetes-support     -
kubernetes-support     microk8s:k8s-kubeproxy          :kubernetes-support     -
kubernetes-support     microk8s:kubernetes-support     :kubernetes-support     -
log-observe            microk8s:log-observe            :log-observe            -
login-session-observe  microk8s:login-session-observe  :login-session-observe  -
mount-observe          microk8s:mount-observe          :mount-observe          -
network                microk8s:network                :network                -
network-bind           microk8s:network-bind           :network-bind           -
network-control        microk8s:network-control        :network-control        -
network-observe        microk8s:network-observe        :network-observe        -
opengl                 microk8s:opengl                 :opengl                 -
personal-files         microk8s:dot-config-helm        :personal-files         -
personal-files         microk8s:dot-kube               :personal-files         -
process-control        microk8s:process-control        :process-control        -
system-observe         microk8s:system-observe         :system-observe         -

My gadget snap is built without errors and ubuntu-image includes the custom gadget snap without issue. Any ideas?

Are you seeding the core22 snap to fulfill microk8s needs? Note that snapd processes the model from top to bottom in order during first boot so you need to make sure it gets listed (and installed) before the microk8s snap in the model…

(If a single snap fails to install during first boot, the complete boot will fail)

Thanks for the insight and reply! I changed the order of snaps as suggested in my model file to the following:

    "snaps": [
        {
            "name": "my-gadget-pi",
            "type": "gadget"
        },
        {
            "name": "pi-kernel",
            "type": "kernel",
            "default-channel": "26/stable",
            "id": "jeIuP6tfFrvAdic8DMWqHmoaoukAPNbJ"
        },
        {
            "name": "core26",
            "type": "base",
            "default-channel": "latest/stable",
            "id": "cUqM61hRuZAJYmIS898Ux66VY61gBbZf"
        },
        {
            "name": "snapd",
            "type": "snapd",
            "default-channel": "latest/stable",
            "id": "PMrrV4ml8uWuEUDBT8dSGnKUYbevVhc4"
        },
        {
            "name": "console-conf",
            "type": "app",
            "default-channel": "26/stable",
            "id": "ASctKBEHzVt3f1pbZLoekCvcigRjtuqw"
        },
        {
            "name": "core22",
            "type": "base",
            "default-channel": "latest/stable",
            "id": "amcUKQILKXHHTlmSa7NMdnXSx02dNeeT"
        },
        {
            "name": "microk8s",
            "type": "app",
            "default-channel": "1.36-strict/stable",
            "id": "EaXqgt1lyCaxKaQCU349mlodBkDCXRcg"
        }
    ]

However, I am still getting the infinite reboot issue. A few errors I was able to make note of right before the reboot seemed to be related to disk mounts like the following:

[FAILED] Failed unmounting run-mnt-ubuntu\x2dseed.mount - /run/mnt/ubuntu-seed.
[FAILED] Failed to start remount-seed.service - Remounting /run/mnt/ubuntu-seed read-only.
See 'systemctl status remount-seed.service ' for details
[FAILED] Failed to start remount-data.service - Remounting /run/mnt/data read-only.
See 'systemctl status remount-data.service ' for details
reboot: Restarting sys
shutdown[1]: Failed to parse /proc/self/mountinfo: Operation not supported
shutdown[1]: Unable to unmount file systems: Operation not supported
shutdown[1]: Cannot enumerate swap partitions, no libmount support.
shutdown[1]: Unable to deactivate swaps: Operation not supported
shutdown[1]: Failed to parse /proc/self/mountinfo: Operation not supported
shutdown[1]: Unable to unmount file systems: Operation not supported
shutdown[1]: Cannot enumerate swap partitions, no libmount support.
shutdown[1]: Unable to deactivate swaps: Operation not supported
shutdown[1]: Could not detach loopback /dev/loop1: Device or resource busy
shutdown[1]: Unable to finalize remaining file systems, swap devices, loop devices, ignoring.

The system never finalizes a boot, so I never get to check the service logs. I am wondering if I need to explicitly list the slots in my gadget file and if so what would be the correct syntax for this use case? Also, if there is anything else I might be missing?

You only need to list slots for connections where the slot is provided by another snap instead of the system, but I’d leave out all of the ones that are not connected in your snap connections output above…