Changes in pi/pi-kernel DTB handling

Dear Ubuntu Core Raspberry Pi community,

The latest snapd 2.50 release (that is currently being released) contains a new feature that allows the gadget snap to reference DTB files that are part of the kernel snap. This fixes a long-standing issue that formerly required synchronous kernel and DTB updates.

The snapd version for this feature is in its stable channel, while the required pi gadget and pi-kernel kernel snaps are currently in their respective beta channels. If you wish to test this feature, please switch to the beta channels for the pi/pi-kernel and refresh your snaps. We welcome all feedback on the forum.

Important: for devices using a custom gadget with the pi-kernel, ensure the following is added to snapcraft.yaml or meta/snap.yaml:

assumes: [kernel-assets]

The above ensures the gadget is only refreshed when a new enough version of snapd is already in use.

In addition, the pi-kernel now declares that it provides “dtbs” assets in its “meta/kernel.yaml”. This means any custom gadgets need a corresponding consuming declaration in their “meta/gadget.yaml”.

The pi-kernel layout for the “dtbs/” folder is slightly different for arm64 and armhf architectures:

For arm64:

--- a/gadget.yaml
+++ b/gadget.yaml
@@ -10,6 +9,10 @@ volumes:
         type: 0C
         size: 1200M
         content:
+          - source: $kernel:dtbs/dtbs/broadcom/
+            target: /
+          - source: $kernel:dtbs/dtbs/overlays/
+            target: /overlays
           - source: boot-assets/
             target: /
       - name: ubuntu-boot

And for armhf:

--- a/gadget.yaml
+++ b/gadget.yaml
@@ -10,6 +9,8 @@ volumes:
         type: 0C
         size: 1200M
         content:
+          - source: $kernel:dtbs/dtbs/
+              target: /
           - source: boot-assets/
               target: /
       - name: ubuntu-boot

Technical details:

The kernel snap can now declare in its meta/kernel.yaml that it contains “assets” that should be updated in sync with the kernel. This could look like this:

assets:
 <name>:
   update: true 
   content:
a.bin
...

A gadget can now reference these kernel assets in the meta/gadget.yaml of the gadget like this:

volumes:
 Pi:
  Schema: mbr
  Structure:
Name: ubuntu-boot
Content:
    - source: $kernel:<name>/a.bin
      Target: ...
    - source: b.bin
      Target: ...

Note that a kernel that declares an asset with “update: true” requires a gadget that puts at least one of these assets into place via a gadget $kernel: reference.

4 Likes

Hi,

It has come to our attention that in a certain situation, the combination of gadget asset updates and switching to consume DTB’s from the kernel snap can leave a device in an un-upgradable state that requires changes to the gadget to be able to update the device again.

Specifically, when a device has installed a pi gadget (or a fork of the pi gadget) where the pi-gadget has a gadget.yaml like this:

volumes:
    volume-id:
        schema: mbr
        bootloader: u-boot
        structure:
          - name: ubuntu-seed
            filesystem: vfat
            type: 0C
            size: 1200M
            content:
              - source: boot-assets/
                target: /`

(trimmed for clarity, but main thing is the absence of $kernel references) and a revision of the pi-kernel snap where it does not contain a kernel.yaml referencing the DTB assets it wishes to expose, then the device is unable to refresh the gadget snap or the kernel snap if the following is also true on the store side:

  • the pi-kernel snap on the tracking channel has a kernel.yaml referencing the DTB assets it wishes to export as boot assets
  • the pi (or fork) gadget has both an update: edition setting AND a reference to a kernel asset like in the following gadget.yaml:
volumes:
    volume-id:
        schema: mbr
        bootloader: u-boot
        structure:
          - name: ubuntu-seed
            filesystem: vfat
            type: 0C
            size: 1200M
            update:
              edition: 1
            content:
              - source: boot-assets/
                target: /
              - source: $kernel:pidtbs/dtbs/broadcom/
                target: /
              - source: $kernel:pidtbs/dtbs/overlays/
                target: /overlays

What happens is that attempting to refresh the pi gadget snap at this point will fail with a message like:

cannot resolve content for structure #0 \("ubuntu-seed"\) at index 1: cannot find "pidtbs" in kernel info

And attempting to refresh the pi-kernel snap will fail with a message like:

gadget does not consume any of the kernel assets needing synced update "pidtbs"

This is because there is now a circular dependency created, where in order to process the gadget asset updates for the new gadget, the device must have a kernel that exposes pidtbs, which there is not, and in order to process the kernel there must be a gadget referencing the pidtbs for the kernel to export to.

The solution out of this situation is to publish 2 new gadget snaps with epochs defined in their snapcraft.yaml’s (or snap.yaml if you are not using snapcraft to build the gadget snap). This assumes that your gadget snap has never defined an epoch before at any point and thus has the implicit epoch: 0 setting.

The first gadget snap to publish is the a modified version of the gadget.yaml above but importantly without the update: edition setting. This is because the update: edition is what prevents snapd from being able to refresh the gadget and we need to refresh the gadget first here in order to refresh the kernel. So add something like this to your snapcraft.yaml to create a new epoch transitioning snap:

epoch: 1*

And the gadget.yaml for clarity of this revision without the update: edition in it:

volumes:
    volume-id:
        schema: mbr
        bootloader: u-boot
        structure:
          - name: ubuntu-seed
            filesystem: vfat
            type: 0C
            size: 1200M
            content:
              - source: boot-assets/
                target: /
              - source: $kernel:pidtbs/dtbs/broadcom/
                target: /
              - source: $kernel:pidtbs/dtbs/overlays/
                target: /overlays

Build and publish this snap so that devices can then refresh to this revision, then they will be able to refresh to the pi-kernel snap, but the assets that do not come from the kernel will not be updated (since we didn’t define the update: edition) in the gadget snap.

Now change the epoch in your snapcraft.yaml to 1:

epoch: 1

and add the update: edition back to your gadget.yaml:

volumes:
    volume-id:
        schema: mbr
        bootloader: u-boot
        structure:
          - name: ubuntu-seed
            filesystem: vfat
            type: 0C
            size: 1200M
            update:
              edition: 1
            content:
              - source: boot-assets/
                target: /
              - source: $kernel:pidtbs/dtbs/broadcom/
                target: /
              - source: $kernel:pidtbs/dtbs/overlays/
                target: /overlays

The epoch: 1 means that essentially in order for a device with a gadget snap with epoch: 0 (the default if not defined), to be able to refresh to the epoch: 1 revision, the device must first refresh to the specific revision with epoch: 1* (the one above without the update: edition in it).

After these have been published, devices will always go through this refresh strategy:

  1. device sees refreshes for both pi-kernel and pi gadget snap available from the store.
  2. The newest revision for the pi gadget snap has epoch: 1, but the current device version has epoch: 0, so snapd will ask the store for a revision that can do both epochs, which is epoch: 1*, and snapd will refresh to this revision.
  3. After refreshing to the new gadget snap with epoch: 1*, the device now can either attempt to refresh to epoch: 1 gadget snap or it can now refresh to the pi-kernel snap. If it tries the refresh to the epoch: 1 gadget snap, that will fail because there is still not a kernel installed to resolve the $kernel references in the gadget.yaml. Thus the device will fail on that refresh and at some point attempt the refresh of the kernel snap instead.
  4. The kernel snap refresh will proceed because the device now has a gadget.yaml which references all the assets that the kernel snap wants to expose, so that will finish.
  5. After this, the only refresh left available is the one to epoch: 1 that also has the update: edition in it, and at this point the gadget asset update will take place updating relevant firmware, etc.

Hopefully this helps anyone who has gotten into this situation get out of it by publishing 2 new snap revisions of the gadget snap using epochs.

Thanks,

Ian

There is no doc describes the details of kernel-assets. We should have one which lists the things should be added into gadget snap and kernel snap.

@degville could you please help on this? Thank you.

Hi,

I would also like to know what’s required in the kernel snap to make this work. It’s not clear something [ at least the assets: stanza ] has to declared in the kernel snapcraft.yaml directly, or something else.

Is there a document I’m missing that already describes this ?

Cheers, Just

According to this git commit in the PI kernel snap, the assets are manually defined in a meta/kernel.yaml , but I can’t find anything else much about it to verify that.

Cheers, Just

It does seem to be supported in the snapd code:

https://github.com/snapcore/snapd/blob/master/kernel/kernel.go#L67

After some trial and error testing - I can confirm this works for me. Declaring the assets in kernel.yaml in the kernel, then using the $kernel:ref as a source in the gadget.yaml of the gadget pulls in the assets when you build the image.

That’s great :+1:

It’s a shame it’s not documented well anywhere that I can find.

Cheers, Just