Changes in pi/pi-kernel DTB handling

snapd
4

Hi,

It has come to our attention that in a certain situation, the combination of gadget asset updates and switching to consume DTB’s from the kernel snap can leave a device in an un-upgradable state that requires changes to the gadget to be able to update the device again.

Specifically, when a device has installed a pi gadget (or a fork of the pi gadget) where the pi-gadget has a gadget.yaml like this:

volumes:
    volume-id:
        schema: mbr
        bootloader: u-boot
        structure:
          - name: ubuntu-seed
            filesystem: vfat
            type: 0C
            size: 1200M
            content:
              - source: boot-assets/
                target: /`

(trimmed for clarity, but main thing is the absence of $kernel references) and a revision of the pi-kernel snap where it does not contain a kernel.yaml referencing the DTB assets it wishes to expose, then the device is unable to refresh the gadget snap or the kernel snap if the following is also true on the store side:

  • the pi-kernel snap on the tracking channel has a kernel.yaml referencing the DTB assets it wishes to export as boot assets
  • the pi (or fork) gadget has both an update: edition setting AND a reference to a kernel asset like in the following gadget.yaml:
volumes:
    volume-id:
        schema: mbr
        bootloader: u-boot
        structure:
          - name: ubuntu-seed
            filesystem: vfat
            type: 0C
            size: 1200M
            update:
              edition: 1
            content:
              - source: boot-assets/
                target: /
              - source: $kernel:pidtbs/dtbs/broadcom/
                target: /
              - source: $kernel:pidtbs/dtbs/overlays/
                target: /overlays

What happens is that attempting to refresh the pi gadget snap at this point will fail with a message like:

cannot resolve content for structure #0 \("ubuntu-seed"\) at index 1: cannot find "pidtbs" in kernel info

And attempting to refresh the pi-kernel snap will fail with a message like:

gadget does not consume any of the kernel assets needing synced update "pidtbs"

This is because there is now a circular dependency created, where in order to process the gadget asset updates for the new gadget, the device must have a kernel that exposes pidtbs, which there is not, and in order to process the kernel there must be a gadget referencing the pidtbs for the kernel to export to.

The solution out of this situation is to publish 2 new gadget snaps with epochs defined in their snapcraft.yaml’s (or snap.yaml if you are not using snapcraft to build the gadget snap). This assumes that your gadget snap has never defined an epoch before at any point and thus has the implicit epoch: 0 setting.

The first gadget snap to publish is the a modified version of the gadget.yaml above but importantly without the update: edition setting. This is because the update: edition is what prevents snapd from being able to refresh the gadget and we need to refresh the gadget first here in order to refresh the kernel. So add something like this to your snapcraft.yaml to create a new epoch transitioning snap:

epoch: 1*

And the gadget.yaml for clarity of this revision without the update: edition in it:

volumes:
    volume-id:
        schema: mbr
        bootloader: u-boot
        structure:
          - name: ubuntu-seed
            filesystem: vfat
            type: 0C
            size: 1200M
            content:
              - source: boot-assets/
                target: /
              - source: $kernel:pidtbs/dtbs/broadcom/
                target: /
              - source: $kernel:pidtbs/dtbs/overlays/
                target: /overlays

Build and publish this snap so that devices can then refresh to this revision, then they will be able to refresh to the pi-kernel snap, but the assets that do not come from the kernel will not be updated (since we didn’t define the update: edition) in the gadget snap.

Now change the epoch in your snapcraft.yaml to 1:

epoch: 1

and add the update: edition back to your gadget.yaml:

volumes:
    volume-id:
        schema: mbr
        bootloader: u-boot
        structure:
          - name: ubuntu-seed
            filesystem: vfat
            type: 0C
            size: 1200M
            update:
              edition: 1
            content:
              - source: boot-assets/
                target: /
              - source: $kernel:pidtbs/dtbs/broadcom/
                target: /
              - source: $kernel:pidtbs/dtbs/overlays/
                target: /overlays

The epoch: 1 means that essentially in order for a device with a gadget snap with epoch: 0 (the default if not defined), to be able to refresh to the epoch: 1 revision, the device must first refresh to the specific revision with epoch: 1* (the one above without the update: edition in it).

After these have been published, devices will always go through this refresh strategy:

  1. device sees refreshes for both pi-kernel and pi gadget snap available from the store.
  2. The newest revision for the pi gadget snap has epoch: 1, but the current device version has epoch: 0, so snapd will ask the store for a revision that can do both epochs, which is epoch: 1*, and snapd will refresh to this revision.
  3. After refreshing to the new gadget snap with epoch: 1*, the device now can either attempt to refresh to epoch: 1 gadget snap or it can now refresh to the pi-kernel snap. If it tries the refresh to the epoch: 1 gadget snap, that will fail because there is still not a kernel installed to resolve the $kernel references in the gadget.yaml. Thus the device will fail on that refresh and at some point attempt the refresh of the kernel snap instead.
  4. The kernel snap refresh will proceed because the device now has a gadget.yaml which references all the assets that the kernel snap wants to expose, so that will finish.
  5. After this, the only refresh left available is the one to epoch: 1 that also has the update: edition in it, and at this point the gadget asset update will take place updating relevant firmware, etc.

Hopefully this helps anyone who has gotten into this situation get out of it by publishing 2 new snap revisions of the gadget snap using epochs.

Thanks,

Ian