Creating custom image based on UC20

Hi,

I’m looking at creating a custom image for deployment, but based on UC20. Initially just locally testing, but will be using a Brand Store eventually. We are particularly interested in the FDE in combination with a TPM2 device.

Trying everything in stages, I have so far:

Although the final test image built fine, it does not finish installing properly on a test system with a TPM2 device [ where all the previous test images installed ok with FDE ].

After initial boot and setup [ including the checks for Secure Boot enabled and TPM available etc ], it reboots. On reaching target Basic System it then moves onto mounting ubuntu-boot, ubuntu-seed and finally on mounting ubuntu-data-enc there is a prompt for a recovery key:

Please enter the recovery key for disk /dev/disk/by-label/ubuntu-data-enc: (press TAB for no echo)

…as I don’t have any input, it eventually times out with:

[FAILED] Failed to start the-tool.service.

The image works fine in a VM without a TPM device.

Any idea what the issue could ?

Thanks in advance,
Just

Are you building your own kernel snap as well?

No, I want to avoid modifying the kernel snap if possible, the only snap I am modifying is the gadget snap [ although so far it is only very light modification ]. This is because I want to try and take advantage of Secure Boot without needing to sign anything ourselves if possible.

Just to try and rule out a bug around FDE if the image is grade dangerous , I just tested with http://cdimage.ubuntu.com/ubuntu-core/20/dangerous-edge/20200915/ubuntu-core-20-amd64.img.xz , seems to install fine.

I have identified the source of the issue - but I don’t yet know the underlying reason, hopefully one of the Devs here can fill in gaps.

With my debugging, I reached the point where it really was only the contents of the gadget snap that differed. So I tried extracting the contents of mine and the published pc snap, and diffed them. The differences included some metadata [ like snap name and description etc ], but also the grub and shim executable. That got me thinking …

As I noted in my initial post, I had to modify the gadget snap so that it built without error. I linked to the details of that here: https://github.com/snapcore/pc-amd64-gadget/issues/49

So, I tried replacing the grubx64.efi in my snap with the one from the published snap. Re-created the squashfs image, the re-built the uc20 image. Same issue.

Then I tried the same process, but with replacing the shim.efi.signed in my snap with the one from the published snap. That new image worked !

In the snap, shim.efi.signed is just a copy of shimx64.efi.dualsigned from the shim-signed deb package. Please note, shimx64.efi.dualsigned is not in the stock 20.04 package, it only seems to be in the 20.10 package.

I have two primary questions:

  • Why do I need the dual-signed shim for this to work properly with FDE ?
  • How do I include the dual-signed shim in the build ?

For the build related issue, I’m using the default multipass approach. How do I make that process include the package I need, do I have to switch the build multipass instance to be 20.10 somehow, or somehow fiddle the apt sources, or something else ?

Thanks in advance,
Just

Ping @xnox for more information about the dual-signed shim, but in the meantime I can advice what we do for tests locally while working on uc20 itself.

If all we need to change is something in the gadget.yaml like defaults or connections, we will just unpack the upstream pc gadget snap (downloaded from the 20/edge channel) and modify that file and repack the rest of the as-is (including the grub and shim files).

If however we need to also change the kernel or something in the initramfs, we will end up needing to re-sign the kernel with snakeoil keys from the ubuntu-core-initramfs package (which comes from the snappy-dev/image PPA), then resign shim specifically in the gadget snap with snakeoil keys as well. Then finally, we need to switch the device to trust the snakeoil keys, which for a VM involves changing the OVMF vars file to OVMF_VARS.snakeoil.fd instead of the normal Microsoft keys version with @ OVMF_VARS.ms.fd. If you are booting/testing on a real device then you may want to enroll the snakeoil keys to your device’s firmware (details depend on the UEFI/motherboard vendor), but note that if this is a production device then enrolling the snakeoil keys will allow any attacker to bypass secureboot on that machine then, so only do that for development purposes.

Thanks for info @ijohnson

For the time being I have added override-stage to grub-prepare that wgets shim-signed_1.43+15+1552672080.a4a1fbe-0ubuntu2_amd64.deb - it’s not aewsome to hard code things like that, but it enables to build normally again for now.

If @xnox has any extra info on the dual-signed shim, and why including only the single-signed shim breaks install when using FDE, I would of course be very interested.

Cheers,
Just

1 Like

the security of FDE relies on inability to boot things that are not measured, which can fake a TPM prefix attack. Meaning if one trusts keys, that one cannot fully control there is risk that the owner of the key has signed something that may be abused to bypass FDE.

Thus to achieve full FDE security, UC20 gadget uses shim signed by an Online UC20 key chained to Canonical controlled UC20 CA. Thus a device manufacturer can ship UC20 key in KEK & DB, and be sure of uncompromisable security of the system. No other systems, but UC20 will able to even start bootloader on such systems.

However, majority of hardware out there do not have UC20 key in db, and thus the default gadget with such UC20 signature will not boot at all anywhere. Therefore we made a dual-signed shim, which in addition to the UC20 signature, has a signature chained to teh Microsoft 3rd Party Root CA (aka UEFI 2011 key). This key is widely shipped on many SKUs out there in db, allowing to boot & install UC20 out of the box. However, such systems rely on cooperation between Canonical & Microsoft to sign the shim, but also to sign dbx updates with revocations of vulnerable binaries that allow FDE bypass.

If your goal of your device to boot with MS UEFI 2011, you can choose to strip Canonical UC20 signature, or only ship shim signed by MS UEFI 2011 key like it is shipped on classic ubuntu.

The dual signed shim signed by both Microsoft and Canonical UC20 key is available from https://launchpad.net/~canonical-foundations/+archive/ubuntu/uc20-staging-ppa PPA, in deb format only. This is the archive that is used during gadget building.

So dual-signed shim is used to boot on different hardware - secure hardware that only allows booting UC20 and nothing else; and more widely available hardware that will boot anything signed by MS ever including UC20 (there really is a lot of things singed by MS UEFI 2011 key, like every Ubuntu release since 12.04 LTS!)

1 Like

Thanks for the detailed explanation @xnox - great post :+1:

Cheers, Just

Maybe it would make sense to encode that PPA into the snapcraft.yaml file using the package-repositories feature? The PPA is clearly necessary to build a working snap, so it seems like that should be expressed in the project rather than a Launchpad build recipe.

2 Likes

As I have mentioned above, I would do that, if that was stable feature, which it still is not? As doing that, would make the builds fail in launchpad where we build the production snaps and upload to the snapstore.

$ snapcraft --provider=lxd
*EXPERIMENTAL* 'package-repositories' configured, but not enabled. Enable with '--enable-experimental-package-repositories' flag.

Thus even if I do add that, it will not build =(

just use a part (before all others) that calls add-apt-repository like everyone has done for years now ? (package-repositories is really just a convenience to avoid having an extra part to add a PPA)

Just one heads-up for you on UC20, the FDE TPM key management bits are still subject to change, so I would not build UC20 images with FDE until UC20 is GA. We’re being very conservative with GA because we don’t want to rush the key handling specifically (there are a million ways to get it wrong).

2 Likes

Thanks for the heads up Mark - we don’t currently plan to use it in production until GA :+1:

Cheers, Just