Proposal to enable pi2/3 major kernel updates

At the moment we are limited when updating kernel by version of dtb files presented in system-boot partition. Those dtb flies are read by pi’s firmware therefore they cannot be scripted and need to be compatible with all kernel revision presented on device. This limits us when updating kernel as we cannot do major kernel updates, let alone dtb changes…

Proposed solution to get out of this lock down.
With combination of pre and post refresh hooks we can actually overcome this limitation and freely update between major kernel versions, it would work as following:

  • new kernel revision is being installed
  • pre-refresh hook is invoked on current kernel revision, in this hook we extract current run time dtb (this has all overlays from config.txt and other data filled in by pi’s firmware). After bootargs cleanup (removing part which has been added by u-boot such as snap_kernel snap_core) we store this dtb to /boot/uboot/pi2-kernel_\<current rev>.snap/raspi2.dtb in case we need it later
  • snapd introduces new kernel revision to the system
  • post-refresh hook is invoked for new kernel revision. This hook copies dtbs and overlay dtbo files to /boot/uboot/ so at next boot firmware will use new updated files related to kernel we are trying to boot

Changes in u-boot:
u-boot always tries to load /uboot/${snap_kernel}/raspi2.dtb
In normal situation this always fails, as file does not exist and dtb prepared by firmware is used instead. Only situation this actually kicks in is when there was kernel rollback. In such a situation it ensures that even if we rollbacked because of boot fail and therefore raspi firmware is now using dtb from borked kernel, dtb prepared by firmware is ignored and backed up one is used instead, one created for kernel we are now trying to boot.

Additionally to ensure safe rollout, post-refresh hook will only copy new dtbs in if pre-refresh hook made successful dtb backup. So if current kernel revision does not have pre-refresh hook post-refresh hook on new kernel revision will do nothing.

I have prototyped this and so far it functions as expected, I have been switching between 4.4 and 4.15 kernels, interrupting boots in the middle to force rollback and system recovers as desired.

Related changes are here:

Kernel patch introducing hooks:
https://git.launchpad.net/~ondrak/ondras-snaps/+git/linux-kernel/commit/?id=e768201d68d458d4852a52ced44fbb7301cd10f5

Gadget u-boot tweak:

snapd changes:
To support this functionality, we need tweak one existing interface and add one new one

  • update to hardware-observe to allow reading of device tree:
  • new boot-update interface allowing us update of boot components:
4 Likes

This is just awesome, we have been waiting for such an option for so long and it can definitely finally help with pi3 kernel updates …

one little nitpick is the use of the “raspi2.dtb” name, that file shopuld be unversioned since we will likely use this setup across multiple HW versions and should not need to special-case each uboot script per device…

big plus 1 to merge this ASAP

1 Like

Good point, would kernel.dtb be better fit here?

whatever you like to name it. as long as we can use the same name across all revisions of one board without having to do extra per-board maintenance of the script :slight_smile:

@ondra My understanding is that we have updates to pi2 boards flowing, and that the dtb problem in that specific case was a red-herring. Is that not the case? Didn’t we update pi2-kernel just a few weeks ago?

Nevertheless, we already have a design for the proper representation of dtbs that can be updated. Let’s please not cook a different solution without looking at that design first. Even if it’s incomplete for your needs, we should improve on it instead of besides it. Happy to discuss this in our next meeting.

@niemeyer the boot-update interface goes beyond dtb updates, where do we stand with:

since @ondra solves exactly this bit in a rather elegant way while at the same time avoiding that porters do bootloader updates via uboot-shell hacks in uboot.env that we do not have any control over today …

his proposal is to limit the use of this interface to gadget (and/or kernel) snaps, it will simply move the update process to a more visible place into a snap upgrade hook from a u-boot hush-shell script and will avoid any potential issues that could be caused by the kernel vs u-boot FAT implementation (writes will happen from linux, not via the u-boot implementation with this).

i think @mvo will agree that this is a viable solution to the long standing problem of optional bootloader upgrades managed via the snap side.

@ogra If you perform system-critical updates in hooks, snapd has absolutely no idea of what is going on, and it will definitely fall apart at some point.

As mentioned:

I was referring to bootloader asset updates which are controlled by the porter today already but in an invisible and risky way via hush shell scripts in uboot.env and i think we said we wanted to give porters an ability to do this via snaps. if you discuss this topic in said meeting, please take the wider use of this interface over dtb files into account.

I have tested this very use case , and unless you craft your kernel in a way that you do NOT need to update dtbs it will fail.
dtbs are build as part of kernel build and are therefore directly related to actual kernel release. The fact we update kernel.img and do not update dtb and things are working is just because we are very careful with what we change in kernel. If there is tomorrow kernel update which WILL require dtbs being updated as well, do we have solution for this? If so can you please point me to it?

With this proposal you can jump even between 4.4 and 4.15 kernel with working fallback.

As mentioned above:

  1. The pi2 kernel had a major update just weeks ago. That disagrees with what was presented in the first few sentences of the top message in this topic (it can be updated, at least in some cases - we did it)
  2. We have a design for dtb updates in place
  3. If your needs are different, they need to be put into that design instead of besides it
  4. If you do critical updates like this in hooks it will likely destroy systems in the future
  5. We should have a meeting to discuss this

are you aware that this is a sheer matter of luck ? the devicetree needs to be in sync with the enabled kernel configuration options, builtin features and created modules, if one of them changes you can cause an unbootable system or physically destroy attached addon board hardware due to wrong values being pushed into the wrong registers, ports being wrongly toggled or supplied with wrong voltage or current.

where is this design written down ?

@ogra Given the results of our prior communication and the confrontational way you start, I’m not optimistic that discussing this here will help. I’ll catch up with @ondra in our next call.

so there is no write up of the dtb upgrade design ?

I’m actually serious. We won’t be discussing this here.

Had a great conversation with @ondra today and we understand each other and are in agreement about the path forward. We’ll jointly review the original design, and will adapt it to the new needs as necessary.

any update on this topic? :confused: