Changes in shim-signed packaging and breakages in UC20 secured grade model images on amd64

device
1

Hi,

I have just discovered a breakage in secure boot amd64. I haven’t dug into the details on the breakage exact reason yet, but it is related to package versions and/or dependencies.

We build a gadget snap, which follows the reference snap here: https://github.com/snapcore/pc-amd64-gadget/blob/20/snapcraft.yaml

In fact it’s slightly different, but we have a similar grub-prepare stage which installs packages required to get secure boot working. The difference is in our package sources, as the reference pc gadget snap uses some PPAs for sources. So our section looks like this:

  grub-prepare:
    plugin: nil
    # Not required - we do this in the docker build image #
    #build-snaps: [snapd/latest/edge]
    build-packages:
      - wget
    stage-packages:
      - grub-efi-amd64-signed
      - grub-pc-bin
      - shim-signed
      - sbsigntool
    override-stage: |
      SHIM_PKG="shim-signed_1.45+15+1552672080.a4a1fbe-0ubuntu2_amd64.deb"
      wget http://archive.ubuntu.com/ubuntu/pool/main/s/shim-signed/${SHIM_PKG} -O ${SHIM_PKG}
      ar x ${SHIM_PKG}
      tar xvf data.tar.xz ./usr/lib/shim/shimx64.efi.dualsigned
      snapcraftctl stage
    prime: [ -* ]

It’s been working fine for us.

Our build pipeline uses ubuntu 20.04, which includes the focal-updates suite, the same as snapcraft build does with the multipass images.

We found that shim-signed was recently bumped in focal-updates, and the new version no longer Depends on the shim package.

It’s gone from:

shim-signed=1.40.4+15+1552672080.a4a1fbe-0ubuntu2

to

shim-signed=1.40.6+15.4-0ubuntu7

The gadget build now produces the situation I described here, where after intial image install the keys for the encrypted volumes aren’t unsealed: Debugging installation issue with secured image
The system installs without error [as far as I can tell ], but fails to extract the key from the TMP.

Adding the shim package manually to the gadget snap part did not seem to solve the issue.

After looking at the change log, I noticed that the shim-signed package now includes the parts of shim directly, and also includes the dualsigned efi shim. So I removed our hack that adds the dualsigned shim, still didn’t work, same error. Then I tried with only the new shim-signed package [ not including shim too ] , same error.

Currently the grub-prepare part looks like this:

  grub-prepare:
    plugin: nil
    # Not required - we do this in the docker build image #
    #build-snaps: [snapd/latest/edge]
    build-packages:
      - wget
    stage-packages:
      - grub-efi-amd64-signed
      - grub-pc-bin
      - shim
      - shim-signed
      - sbsigntool
    prime: [ -* ]

Can anyone help here ? @ijohnson

The reference pc gadget on github hasn’t changed recently, so presumably that’s still using a version of shim-signed from the ppa.

@xnox I know you have worked on the pc gadget. Any ideas ?

Currently I’m out of ideas, but will try and go back to our hacked inclusion of the dualsigned shim, and the older shim-signed package from focal not in updates, 1.40.3+15+1533136590.3beb971-0ubuntu1

Cheers,
Just

2

Can you try using stage-snaps with the pc gadget snap to just reuse the binaries from the upstream gadget snap instead of manually pulling all the debian packages that the pc gadget snap uses?

3

It’s not a bad idea, but it’s harder for us because we’re using docker as the build env, and snapd isn’t running.

I’m just trying the non focal-updates package. If that doesn’t work I will have a rethink.

Cheers,
Just

4

I got this going again. In the end, after taking a proper look at the Makefile used in the grub part I realised that the only file that was required was the dualsigned shim and the grub signed executable.

Now the part looks like this:

  grub-prepare:
    plugin: nil
    # Not required - we do this in the docker build image #
    #build-snaps: [snapd/latest/edge]
    build-packages:
      - wget
    stage-packages:
      - grub-efi-amd64-signed
      - grub-pc-bin
    override-stage: |
      SHIM_PKG="shim-signed_1.45+15+1552672080.a4a1fbe-0ubuntu2_amd64.deb"
      wget http://archive.ubuntu.com/ubuntu/pool/main/s/shim-signed/${SHIM_PKG} -O ${SHIM_PKG}
      ar x ${SHIM_PKG}
      tar xvf data.tar.xz ./usr/lib/shim/shimx64.efi.dualsigned
      snapcraftctl stage

…and it’s all working again ! :sweat_smile:

I will probably make one further “enhancement” and just put the deb package in our git repo, I can’t risk loosing the one dualsigned shim that I know works !

It may be possible to get things working with the focal-updates package, but I very short on time right now.

I really think it would be useful to have some detailed docs on what is really required here for a amd64 gadget to support Secure Boot properly. I feel like I’m kind of just poking around in the dark, this shim works, this one doesn’t. I’m not a SB expert by any means [ obviously ! ], and some pointers would be very welcome :+1:

Cheers, Just

5

Well in that case you could skip the stage-snaps and just use the snap command to download the snap manually, snap download does not require snapd to be up and running

6

Ahh, yes, I had forgotten about snap download. Good point. I will bear that in mind for the future.

Cheers,
Just

7

Irrespective of the issue you are discussing the above commands are insecure. A download is performed over HTTP and the apt archive metadata is not verified with GPG, meaning one could potentially download a compromised / modified shim, that is either malicious or an old revoked one.

Instead one can use apt download that verifies GPG metadata, use HTTPS launchpad librarian URLs, or for example use pull-pkg (pull-lp-debs pull-ppa-debs …) from ubuntu-dev-tools that talks to Launchpad api, finds the right debs, downloads them over HTTPS and verifies that the checksums of those files are the expected ones. Or just use stage-packages in snapcraft yaml with archive declarations you care about. All of these methods are more secure preventing machine in the middle attacks.

1 Like
8

@jocado Ubuntu 20.04 (aka classic) and Ubuntu Core 20 use different shims, different grubs, and different kernels, which are signed with different keys and have different properties.

One cannot use shim from the Ubuntu 20.04 archive for Ubuntu Core 20.

It will lead to boot, install, upgrade, recover, unseal issues.

Please only use stage-snap / snap download && unsquashfs when building a gadget based on the pc one. I.e. one really really really wants the matching shim & grub combo from the pc gadget itself.

10

I was not aware of pull-pkg command - that’s cool.

As it happens, I changed the snap to include the deb file without wget, and verified hashes and sigs.

Cheers,
Just

1 Like
11

that works too. You are using the one signed with UC20 CA right? sbverify --list should have UC20 CA in the list on the shim.

12

Yes, I agree that’s probably best in the long run. I will look at getting it changed in the near future.

It does feel like something that should documented a bit better however. In all the docs I read I never got the impression that was the best approach.

If you read this: Building a gadget snap | Ubuntu …it really just directs you to here: https://github.com/snapcore/pc-amd64-gadget

The README doesn’t indicate that you should clone the repo but change the part to include the shim from the snap in the store :slight_smile:

Perhaps a more “end user” focused reference snap project in guthub should be created, or the README updated, or something.

Cheers, Just

1 Like
13

sbverify --list should have UC20 CA in the list on the shim.

Thanks, that’s a good tip.

Strangely the working dual signed shim it doesn’t have UC20 CA in the subject string.

image signature certificates:
 - subject: /C=GB/ST=Isle of Man/O=Canonical Ltd./OU=Secure Boot/CN=Canonical Ltd. Secure Boot Signing (2017)
   issuer:  /C=GB/ST=Isle of Man/L=Douglas/O=Canonical Ltd./CN=Canonical Ltd. Master Certificate Authority

Cheers,
Just

14

As I said Ubuntu 20.04 (focal / archive.ubuntu.com) and Ubuntu Core 20 (pc gadget / core-specific PPAs) use different shims signed with different keys.

Hence you must use shim & grub & related efi binaries from the pc gadget.

15

ok, but that shim is working.

If I download the pc snap, extract it, and examine the shim in that, it only seems to have the MS signature. The full output is this:

$ sbverify --list shim.efi.signed 
warning: data remaining[1171248 vs 1334816]: gaps between PE/COFF sections?
signature 1
image signature issuers:
 - /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
image signature certificates:
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Windows UEFI Driver Publisher
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation Third Party Marketplace Root

That seems odd.

Chers,
Just

16

horum, that does not look quite right.

17

Ahh - hang on , wrong channel.

18

Ok, that’s better:

$ sbverify --list shim.efi.signed 
warning: data remaining[1179872 vs 1343496]: gaps between PE/COFF sections?
signature 1
image signature issuers:
 - /C=GB/ST=Isle of Man/L=Douglas/O=Canonical Ltd./CN=Canonical Ltd. Master Certificate Authority
image signature certificates:
 - subject: /C=GB/ST=Isle of Man/O=Canonical Ltd./OU=Secure Boot/CN=Canonical Ltd. Secure Boot Signing (Ubuntu Core 2019)
   issuer:  /C=GB/ST=Isle of Man/L=Douglas/O=Canonical Ltd./CN=Canonical Ltd. Master Certificate Authority
signature 2
image signature issuers:
 - /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
image signature certificates:
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Windows UEFI Driver Publisher
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation Third Party Marketplace Root
19

@xnox Can I ask in that case, two things:

Why does the shim with the “wrong” signature currently work for us ?

What are the implications of changing the shim on a currently built device ? If I update the shim in the gadget, will it just work or break something ?

Cheers,
Just

20

It depends on what the device had there before. And what you have in the recovery partition. Since reinstall option will bring back the shim that the device was initially provisioned with / how the initial image looked like. It is best to reinstall.

For example, the image with classic shim may fail to upgrade and reseal in some future update of the gadget, it might get revoked by UEFI revocation lists, it may fail to boot on Ubuntu Core certified devices and platforms.

21

ok , thanks @xnox

Sounds like something we should only do as part of a new image version roll out. So I will leave it as is for now, and look at changing in the future.