Debugging installation issue with secured image

Hi,

One of our image latest builds is failing to install/bootstrap properly. It’s UC in secured mode. After resetting the TPM, we write the image to disk and reboot, It initially boots, runs through all the setup including encrypting the volumes, I don’t see any errors [ but some of the test goes by pretty fast ], and when it reboots for the first time it fails to mount the ubuntu-seed volume.

It waits promoting for:

Please enter the recovery key for disk /dev/disk/by-partuuid/blah-blah

Then eventually there is an error from the-tool:

error: cannon activate encrypted device "/dev/disk/by-partuuid/blah-blah": cannot activate with sealed key (cannon unseal key: invalid key data file: cannot complete authorization policy assertions: cannot complete OR assertions: current session digest not found in policy data)  and activation with recovery key failed (cannot obtain recovery key: /usr/sbin/systemd-ask-password failed: exit status 1)

How can I debug this ? There may be a relevant message in the initial boot and setup, but some of it goes past very fast. Is there a log of the initial boot kept somewhere ? Anyway I can interrogate the tpm for anything that may help ?

Feels like something in a snap has changed, but I have a working image with the same gadget snap content and snapd revision on the same hardware. I’m really quite stuck for ideas at this point.

Cheers,
Just

Are you sure it’s ubuntu-seed and not ubuntu-data? ubuntu-seed is not encrypted so mounting that should not be a problem, ubuntu-data is the more likely suspect in my mind since you are using grade: secured.

Most likely the earlier logs that went by fast have a clue, what I’ve done before is to record a video with the slow motion video feature on my phone and go by frame by frame :slight_smile: There’s probably smarter ways but that has been reliable for me

You’re right @ijohnson, it’s not ubuntu-seed . It was late last night!
The seed volume is mentioned just before as being mounted. The voume that fails to mount is only mentioned by uuid.

I had tried a video but it was a bit too quick in places still, but I will try a slow motion one, that’s a good idea.

Cheers,
Just

Ahh - I found the issue I believe, but only by binary searching by changing the included snaps.

I will create a new thread for it describing the issue.

For ref: Changes in shim-signed packaging and breakages in UC20 secured grade model images on amd64