This is a long standing topic that we have discussed at several sprints before but nobody got around to actually implement it (until now 
)
The problem
We need to use the initramfs-tools infrastructure from Ubuntu to generate our initrds to benefit from fixes, security and general maintenance done in the distribution. Hooks, scripts and binaries typically get put into place during package install when the core snap (or in the future the base snap) is assembled and at the end a binary initrd.img is created with the content of these package snippets.
We also use this hook mechanism to additionally include the Ubuntu Core specific bits needed to assemble a properly usable rootfs from the readonly core (or base) snap and the writable system partition during boot.
On a desktop machine, where you also have a deb of the kernel in place and /lib/modules is populated already, the modules and firmware are automatically included in this initrd.img
In the snap world, where we separate the kernel and use a generic rootfs snap this assembling process is done in two steps, the core build creates a generic initrd.img including all bits that are not specific to the particular target hardware (i.e. no firmware or modules are available at creation time).
To create the kernel snap we download the latest core from the store and re-pack the included generic initrd.img file to add the hardware specific bits like firmware and modules that are defined in snapcraft.yaml and ship it along with the kernel snap.
During boot the re-packed initrd from the kernel snap is used.
Now, if a (security) fix lands in the generic bits of the initrd or in one of the packages that serve the bits included there, you actually have to re-build all the kernel snaps to make them pick up the change from the new core snap.
This exposes a binding between kernel and core that we want to get rid of, both snaps should be completely independent and so should the initrd.img contents…
The solution
The agreed concept we came up with during several sprints and discussions on the topic was to use a well maintained kernel feature that allows to concatenate multiple initrd.img files into one during boot.
To achieve this, the initrd.img will be directly used from the core/base snap while the kernel will ship an additional modules.img with the content of /lib/modules, /lib/firmware (selected via the initrd-modules: and initrd-firmware: options in the kernel package snapcraft.yaml) and the matching databases (modules.dep etc). Both files will be merged when being loaded by the boot loader.
While it was clear that this works with grub by simply specifying multiple initrd lines in the configuration, the way forward on the u-boot driven devices was unclear and untested.
Over the course of several months i have now tested a prototype setup for our u-boot driven devices (both Rpi’s, dragonboard and beaglebone) without finding any issues. A prototype script (I ran something similar during the test period via a manually put in place systemd unit) for the u-boot setup can be found at Ubuntu Pastebin …
The way foward
PRs for the u-boot driven gadget snaps are now in the works, the implementation will check for the existence of a modules.img in the kernel snap, if this file does not exist, it will fall back to simply loading an initrd.img using the old code (to provide full backwards compatibility).
If a modules.img is found, it will also be looked for a /boot/uboot/<core snap>/initrd.img to load the generic parts of the initrd.
Requied changes to snapd and snapcraft
When installing a kernel snap, snapd populates /boot/uboot/<kernel snap>/ with the necessary files (by copying them from the snap). This needs to be extended to copy a modules.img and to not consider a missing initrd.img as a failure.
At the same time upgrading the core/base snap will need to change to provide a similar behaviour to copy /boot/uboot/<core snap>/initrd.img in place for the bootloader.
The snapcraft kernel plugin needs to change to stop downloading the core/base snaps and grow code similar to the prototype script above to generate a modules.img file at build time.
Grub based gadget snaps need to grow some additional scripting to provide a similar fallback behaviour for kernels created with an actual initrd.img and without modules.img.