well, that wont scale at all … perhaps we could hire a specific “porter patch maintainer” for it, once we have enough boards that might become a fulltime job … 
i think leaving it to the porter and taking the risk of blame is the only scalable option.
I’m not sure we can afford to require manual actions. People will have devices in the roof that they won’t want to manually do anything about, yet the device manufacturer will want to update these details. Yes, we should be extremely careful, but there’s probably no way around it.
We’ll probably need this at some point for other reasons, but for the assets that we manage and that are a standard part of the gadget format, we need to handle it. It’s also much safer to do it well once than to expect everybody to be careful every time on their own custom scripts.
That’s what we need I think.
Can’t we hook this process into the snap refresh procedure itself?
I’d prefer to engineer it in a way that updating the gadget and updating these bootloader files is perceived as the same thing on the user end. People are already surprised today that this isn’t the case, which is a great indication that this is a path of least surprise.
well, would you prefer to brick the device automatically or rather have someone do it manually through a remote connection and be aware of the risk …
manufacturers not supporting all builtin HW on new systems providing enablement for this additional HW via an update of the bootloader binaries is not a rare case in the embedded world.
having a human involved here who sees a massage about the possibility that the device is bricked when powering it off during the flash process will at least prevent you from silent bricking.
… or the biggest surprise if their device on the roof that uses NAND storage and has no way to recover from a broken bootloader hard-bricks itself over night.
there is no proper solution to this problem in any case, only compromises …
@ogra There’s no option here. We will enable manufacturers to automatically update their devices, as otherwise we’re not delivering on the promise of helping people be up-to-date and out of security risks. We need to do a good job on the update procedure, and make the manufacturer aware of what risks are involved in the process.
I think updating the firmware is a really special case. It’s rarely security sensitive and should not happen outside of assisted and planned scenario. Think about this in another context. Would you be comfortable as a device manufacturer if the software you need to use would write to UEFI persistent storage each time you ship an updated gadget snap? The risk taken here is mostly similar to flashing BIOS on a motherboard or doing bootloader update on a phone. Those are not casual experiences and the cost of failure is immense. I bet there will be commercial feedback on this front as soon as more people doing ARM devices (where this is more common) realize this takes place. We’re not re-installing GRUB here, we’re updating BIOS transparently.
I bet this cannot be done, for legal, warranty and manufacturer cost reasons. Let’s please reconsider what is updated, how and how much the manufacturer is in control.
1 Like
Could we imagine delivering firmware updates via a separate (privileged) snap? For instance, we don’t ship a BIOS image in the gadget snap/images, but you might want to update your BIOS.
This thread is about updating bootloader assets already in the gadget snap / in images which is more specific; I guess we could limit the change to that, but then it wouldn’t apply to other firmware update situations.
On the actual updates, I join Zyga’s concerns that we don’t want to flash the bootloader assets every time the gadget snap is updated as that introduces more risk.
I also wonder if these updates should be done in a relatively controlled environment, e.g. while no snap provided-services are started. For instance: stop all snaps, write bootloader bits carefully, start all snaps. That’s intrusive, but that would bring the system to a quiet state with predictable performance and no interfering load while doing the critical update.
probably even a special bootmode that can trigger the update hook from the initrd (similar to the bootreason variable on android that triggers recovery mode to do certain risky updates). that way you would not even have init running and could mount the root partition readonly (in case you need to copy from there). and typically you want to reboot anyway after updating such a part of the system.
We shouldn’t be discussing how far we need to go. We’ll go as far as we need to, and no further. I’m all for not taking risks we don’t have to. As such, we can start from the real use cases we have today, which I assume is what @mvo is really interested in getting opinions on. The point I’ve raised above is that for those cases we need to design solutions which can be automated from the get go, since snapd is not just phones and desktop.
I updated my BIOS on all my devices and each time it was to support a new CPU or to fix ACPI on Linux. The only exception I can think of is Apple releasing a iOS update to stop jailbreaking but even that required me to manually agree as it was still a risky operation.
As to the issue at hand: the suggestion to update boot assets as they are descried in gadget.yaml does bring in the bricking risk so I think we need to have a way to say “we really want to write those files / raw partitions” (perhaps other than trying to read the original.
That’s exactly the point. You and the vast majority of people in the world only update anything at all when there are new visible benefits to be had. Meanwhile, security issues thrive.
So yes, let’s please take a simple case to handle, and then go into specifics about it.
May I also say that I’m all up-to-date on all my machines and BIOS (because updates are infrequent).
Still I agree. Let’s work on the reference platforms and see what does it mean to update gadget snaps there. I also wonder if we support UEFI boot on amd64 (and soon arm64).
@ogra can you please tell us what lives in the pi2/3 gadget snap and what would be the correct way to determine that we need to rewrite some files / raw partitions and the correct sequence of doing that? Ideally we’d do the same for the dragonboard as well as (AFAIR) it has way more “interesting” partitions and other magic.
yes, the pi is completely boring regarding the setup, all files simply sit on the vfat partition, OTOH i dont see a reason to update the dragonboard bootloader files (i doubt qualcomm has even changed them since the board was released).
if we just want to have a target for a complex update exercise the dragonboard is definitely the right device though.
regarding the changes … the pi does all the HW initialization in the first stage bootloader blob and also ships additional features …
is a very good example where power management features were changed …
additionally the pi boot firmware also requires updating for major kernel version bumps and, given that the bootloader is actually the graphics driver, there can be driver related fixes too that you need to update for. specifically on the pi’s the bootloader blob files are way more than a BIOS, while on other boards the BIOS comparison is actually true. in general, the update process here is simply “copy”.
for UEFI we actually ship all bits and pieces from the fwupdate setup we also use on the normal ubuntu images, the process there should be identical and is AFAIK actually used on the dell gateways (which also use secureboot)
so how about we provide an opt-in upgrade feature for bootloader bits …
i.e. the feature is there by default but you need to explicitly enable it through a core setting, via a snap command or through a gadget setting. that leaves us the wiggle room to actually warn about the risks with power outages etc when the user enables it on a device that does not have it by default and gives vendors the ability to turn it on by default via a gadget setting on images for devices they know will be always on and never suffer from potential power loss.
this still doesnt bring us rollback but sounds like it would cause the least surprises for all sides.
Yes, having a setting inside the gadget which allows the publisher to explicitly state updating of resources is intended in that particular gadget seems like a good idea.
1 Like
Adding an additional comment here for posterity and searchability since AIUI this is being worked on with priority.
I’m told that this topic is what is preventing ARM kernels like the pi2-kernel snap from getting updates to the stable channel. Obviously, this is not great because users of those kernels aren’t getting security updates or bug fixes. One bug that is fixed in recent kernels but broken in the pi2-kernel in the stable channel is that AppArmor complain mode (what we use with devmode) doesn’t always work correctly. This manifested itself with apt-secure not working properly within the classic snap. See https://bugs.launchpad.net/snappy/+bug/1670475/comments/13 for details.
I wanted to explicitly ask in light of the issue of Skylake & Kaby Lake hyperthreading with the Kaby Lake processors needing a BIOS update.
https://lists.debian.org/debian-devel/2017/06/msg00308.html
Are gadget snaps currently capable of updating BIOS as discussed here?
I am wondering whether grub.cfg would be updated as part of this. What would happen to, say, custom modifications to the kernel command line? I think that a mechanism like the one in classic images, where you have scripts in /etc/grub.d/ that are run after upgrading grub should be in place so custom modifications can take place after gadget snap updates.
Just wanted to know what is the status of this.
In the meanwhile, what options do I have to update the kernel in my raspi3 with Ubuntu core? Make and installation from scratch using the daily version as @ogra suggested?
Hi everyone,
I’d like to know if there are any updates about this issue with the release of Ubuntu Core 18. Is it possible to update the kernel in the RPi 3 or is it still impossible to do it? Are @ogra’s images able to do so?
Thanks for your replies. I deeply appreciate your help.
the kernel could always be updated, but parts of the gadget snap still can not, if a gadget gets updated only the meta-data will be updated … i.e. gadget.yaml and snapcraft.yaml, so you can add/remove config defaults and interfaces, but you will i.e. not be able to update the u-boot binary or the Pi bootloader firmware, snapd will ignore these bits.