Guidance on snap interface to load a device-tree overlay on RPi Core18

Is the plug connected? It isn’t enough for it to simply be present on the snap.

This sounds like the kind of thing that would necessitate building your own gadget/image. I personally have never fiddled with creating a gadget snap with a different device tree overlay, but you can see how it’s done for a raspberry pi 3 here: https://github.com/snapcore/pi3-gadget/blob/master/snapcraft.yaml#L58
I imagine all you would need to do is stage the overlay you need into $SNAPCRAFT_PART_INSTALL/boot-assets/overlays/your-favorite-overlay.dtb.
There you can also see how to modify the /boot/config.txt, though admittedly I’m not sure if that is different from the /boot/u-boot/config.txt (I think it’s the same thing but I’m not sure).

1 Like

No since I can’t actually successfully install the snap with strict confinement (since I want this to be a daemon oneshot service - so on snap install it tries to run the service, this fails due to AppArmor denials, so the snap is considered failed to install) - and so then I can’t even manually connect the interface. So the other option then is to not make it a oneshot service but then we lose the automagic hardware enablement which is the whole point of the snap.

Sure but do we really envision N different custom gadget snaps for the RPi for all the various N hardware devices which people might want to add on (and each published by some random publisher)? I am pretty close to achieving this - just hindered by not having the store assertions in place from what I can tell to get it to autoconnect on install and allow the system-files / module-control permissions.

1 Like

For testing, your install hook should be able to do a snapctl stop --disable the-daemon, if that helps.

1 Like

Thanks for the suggestion - with this in place it now works as strictly confined after manually installing the snap, connecting the interfaces and then reenabling:

snap install ./wfuhuMg7FCMYJGocE7AUGCdiW6vAkiH8_19.snap --dangerous
snap connect rpi-tpm2-slb9670-hwe:system-files :system-files
snap connect rpi-tpm2-slb9670-hwe:kernel-module-control :kernel-module-control
snap start --enable rpi-tpm2-slb9670-hwe

And then it automatically loads on boot as expected too!

1 Like

@jdstrand can you comment on whether system-files for write to all of the device-trees overlays path is reasonable in this case (as @jamesh suggested in comment 4 above Guidance on snap interface to load a device-tree overlay on RPi Core18)

I am worried about the consequences of using system files and for injecting the dbt this way. What is the impact on the SPI interface in your overlay? What is the general impact in case a different overlay is used?

While inconvenient I strongly prefer to force this to go through a new model, a new gadget and, if required, changes to snapd to inhibit creation of implicit slots via gadget language.

Well it makes sense to me that you would have your own gadget for this specific type of device, since it requires such extensive changes like loading kernel modules, modifying the boot config.txt, and adding it’s own device tree overlay. Other simpler devices that just use pre-existing /dev/xyz should be usable from the slot interfaces exposed by the gadget such as gpio, i2c, etc.

To me, the ideal thing would be to have all the support for this device baked into the gadget (either a custom gadget or perhaps upstream support for this device into the official gadget since it seems well supported upstream), because providing all these permissions to a snap seems excessive. If necessary, one could set something up such that the device tree overlay is only loaded dynamically by the gadget snap or by snapd when a slot exposed by the gadget for this specific device is connected to something so that a snap could consume this device specifically. This way the snap that wants to talk to this TPM module would simply plug spi-tpm (strawman name) which is slotted by the gadget snap, and upon connection the device tree overlay is loaded and this driver takes over the SPI bus.

This point specifically is why I think that the behavior of loading this driver and device tree module shouldn’t be allowed by a snap, because if a user has a snap installed that’s using the SPI bus and is connected to the spi slot from the gadget snap, why should another snap be allowed to “take over” the SPI bus by loading this driver + device tree overlay? I think that since there’s a conflict between normal SPI usage and this device that conflict should be mediated by snapd in some way, or at least handled in the gadget so that the only SPI slot exposed is for this device and only is connected to the specific consumer of this snap and there’s no opportunity for another snap to connect to the SPI bus normally.

Also note that last I heard, the general recommendation was never to use kernel-module-control, and if you need a custom kernel module, you needed to put the kernel module into a custom kernel snap and use that on the core device, simply because kernel-module-control is so powerful.

1 Like

@ijohnson - thanks for the advice. I am still having trouble understanding how a specific gadget snap is the best idea here - so in this case we added a TPM module so we need to use gadget snap pi-tpm (strawman name) - but then if I add some other module as well which needs some custom device-tree / driver elements etc - then surely we should have a second gadget snap which adds support for that device in the same manner as the proposal for a specific gadget snap for this device - but we can only have 1 gadget snap installed - so then we have a combinatorial explosion of NxN gadget snaps needed to be produced to support all the different hardware elements which might be added. So hence why I really want to try and pursue this idea of a single snap being able to do hardware enablement on its own to support cases like this where the devices are removable etc.

As for the device claiming the SPI bus etc - there can only be one device connected to the SPI interface - and so then it is up to the user to ensure they don’t install a hardware enablement snap for a piece of hardware which is not connected - OR in that case to disable it.

Finally, for kernel-module-control - I really want to try and avoid having to have custom kernels / gadgets just to enable a particular hardware add-on module - I just can’t see how this is scalable - whereas the case of an additional snap which simply does the small amount of work to enable the hardware (as is the case here) seems like a much simpler solution.

1 Like

Well, that is up for discussion with the reviewers, but I would actually kick this up to an architect and have @pedronis and/or @niemeyer comment.

@pedronis/@niemeyer: the short summary is basically if it is ok for a snap to modify the device-tree overlay and if so, how this is exposed to snaps? If not, how to support @alexmurray’s use case?

For the first question, AIUI, while the device tree mechanism is path-based and therefore potentially apparmor-friendly, the paths used can be arbitrary (@alexmurray - correct me here) such that different tpm add on boards will use different overlay paths. This means we can’t really have a function-specific interface like device-tree-tpm2 or similar (the idea being that the snap would have access to only this area, or ideally snapd would create the overlay on behalf of the snap). Because we can’t easily do this, the result is we end up granting very powerful rules for modifying many things in the device tree, which is something we’ve avoided since, again AIUI, you can destroy the system and/or completely take it over with arbitrary device tree overlay access.

For the second question, I agree with @zyga-snapd that something else is probably required, that needs design. Perhaps in the shortest of terms we can grant @alexmurray’s snap the ability to write to the specific area his snap needs to write to for this add on board via the system-files interface. This is not maintainable long term though and there is no mechanism for saying that the other spi devices will be inaccessible to snaps that might have the spi interface connected. So there are several things to consider IME:

  1. gracefully dealing with device overlays that break other interfaces (eg, this tpm device overlay claiming spi)
  2. how to support device overlays for arbitrary add on boards for IoT tinkerers
  3. how to expose this to snaps

When considering this, I think it is vitally important to acknowledge that we have the gadget snap mechanism that should work perfectly for this for production IoT devices that will be deployed in the field. What is missing is something to accommodate developers/tinkerers who are using the raspi or similar to prove their designs/etc and are using flexible hardware that supports add-on boards.

While writing this, for ‘2’ and ‘3’ I was thinking that snapd should do the device overlay operation on behalf of the snap. Perhaps this is a new device-overlay backend that interfaces could tap into where snapd upstream enumerates the add-on boards that we support and their corresponding overlays which might tie into the warnings system for things like claiming all of spi. Another idea is to simply acknowledge that this is all officially supported via the gadget snap for proper IoT devices, but expose a snap set command in the core/snapd snap that tinkerers/developers can use to feed a device overlay to the device that snapd will manage and create on boot (this could even be part of the revert logic such that snapd would back out a device overlay if the system failed to boot). There are likely other ideas…

So the way that I have seen this work in commercial deployments is that you have a single logical “device” with all the “sub-devices” added to the gadget snap - i.e. your single gadget specific to this deployment has the TPM drivers, maybe some special GPIO pins, etc. all put into the gadget snap (or possibly the kernel snap for out-of-tree drivers and such) and exposed via slots in the gadget snap for userspace snaps to connect to. So this way instead of having all the combinatorial explosion gadget snaps, you would only have as many gadget snaps as you have actual “sub-devices” connected to the logical “device”.

This handles both the case of prototyping/development, as you can easily create your own image from the gadget with ubuntu-image, as well as commercial deployment as with a commercial deployment you will have a finite set of actual models you deploy and you can either handle all cases with a single gadget or maintain separate gadgets for the set of models you maintain.

I will grant that building a custom image/gadget snap is potentially more work than it needs to be, but better tooling around that is IMO the better, more maintainable solution than providing all these accesses to snaps.

Maybe I’m missing something but the issue isn’t about connecting the interfaces, it is about conditionally applying the device tree overlay dependent on when the tpm add-on board is present. This is needed because not doing so means that SPI will be broken for everyone that uses this gadget. I’m told (@alexmurray, correct me), that board presence is not discoverable before the device tree overlay is applied. This makes it impossible to have a single gadget snap that includes this overlay that has functional spi for anything other than tpm.

(In saying that, I wonder if it is possible to unconditionally apply the overlay, then detect if the device is there, then remove the overlay. Does this make SPI functional again? That is admittedly a bit wonky.)

1 Like

So there are a couple issues here as @jdstrand highlights - do we want to support allows non-gadget snaps acting in this ‘dynamic’ gadget role? @pedronis @niemeyer can you comment on this at all? If not then the only way forward is a new gadget snap specifically to enable this hardware device, however as outlined above I can’t see how this is scalable across the RPi ecosystem where there are potentially many different add-on devices that might be used and combined in arbitrary ways. Also if a gadget snap is the only way forward, would Canonical publish this (so that it could get maintained along with the standard pi gadget snap)?

Much easier IMO if we can find a way for snaps to perform this hardware enablement role directly - is a lot more flexible for this kind of use-case. If we do then want snaps to be able to perform some of this functionality, what kind of interface do we expose? @jdstrand is right that allowing dynamic loading of arbitrary device-tree overlays is quite a powerful functionality that we can’t easily constrain.

The other option is to make this not so dynamic, where a snap could install a device-tree overlay (these live under /boot/uboot/overlays) and then snapd could enable this by directly editing /boot/uboot/config.txt to have the boot process load the new overlay at boot time, rather than leaving it up to a snap to do it at runtime. This then makes the device available earlier in the boot sequence so could be more flexible. Also this allows an end-device user to more easily introspect the installed overlays by simply looking in the usual location for them. From a security and confinement point-of-view, this is still no less powerful than the dynamic approach BUT it does mean snapd could have some say in the process if desired (introspect the overlay etc?).

god forbid no ! :slight_smile:

there are cases where you can damage or even completely fry your hardware, no application snap should be allowed to load overlays …

application snaps are not limitable by board, they sit in the store for a specific architecture and while you can namespace them like i.e. “foobar-pi” they will still be installable on any other armhf system … so installing teh foobar-pi (shipping an overlay for the RPi only) snap on some allwinner or imx board will probably set some voltage on some regulator to a value that makes the chip go up in flames … this really needs to be done in the gadget …

that said, the Pi being a special case here with lots of extra hacks like the snap system keys to mangle config.txt via snap set/get, there could surely be a mechanism to set a dtoverlay value from some pi specific interface.

We could have the super-privileged interface but it would strictly be for experimentation only.

If we want first class support for this it would need to be mediated by snapd itself or at least specific gadgets.

I could imagine using the content interface or an interface like it, together with the declaration language that let us control it quite precisely, to provide the DTBs to snapd/the gadget for loading on behalf of the snap but not directly by the snap. In that approach the store/gadget brands and to some extent users but not the snaps would have a voice about what to load.

Is there a way or could a way be conceivable to check for conflict between DTBs before loading them?

@ogra - ok that is a good point re hardware incompatibility etc.

The other option for this is to get the overlay merged into the pi kernel itself, so then only a change is needed to config.txt to specify this to be loaded. This could be mediated and managed by snapd - and if needed down the track perhaps could be extended to allow snaps to ship overlays themselves for snapd to managed as well.

Either way it is probably useful to have this overlay in the standard core 18 pi kernel (since the driver is already there) so I’ll see if I can push that through an upcoming SRU cycle.

Regarding management of overlays via snapd, since we already have various pi config.txt settings accessible via snap set system pi-config.foo - perhaps we could just add pi-config.dtoverlay (and may as well then expose dtparam as well - https://elinux.org/RPiconfig#Device_Tree)?

1 Like

On a related note for those creating gadget snaps for their devices, note that gadgets can have services (even with plugs) and for the case of say, a device that needs tpm_tis_spi with device tree overlay, one can:

  1. create the device tree overlay like normal in the gadget
  2. do a one-shot daemon in the gadget that writes out tpm_tis_spi to /etc/modules-load.d/<your filename>.conf (just make sure that <your filename> doesn’t start with 'snap.
  3. plugs system-files in your gadget that allows writing to /etc/modules-load.d/<your filename>.conf
  4. setup your gadget to auto-connect system-files for itself
  1. create the device tree overlay like normal in the gadget

Is the normal way to do this what was suggested by @ijohnson above, simply make sure to stage the binary .dtbo file to $SNAPCRAFT_PART_INSTALL/boot-assets/overlays/your-favorite-overlay.dtbo?

Also, and maybe I missing something, but doesn’t the inclusion of the correct DTB overlay for this TPM module automatically trigger the module loading? I’m not sure why you’re additionally adding a conf file to /etc/module-load.d/ to do this?

@ijohnson also mentioned modifying /boot/config.txt, however again with the correct overlay, I’m not sure why this would be required?

1 Like