Hey, we are trying to have Full disk encryption with ubuntu core 22 based custom image(have our own model/assertion see below). we have UEFI+TPM based amd64 device.
When we try to flash and boot with the image generated below, the model assertion, it fails to boot up our image only thing we can see is tty login screen, which our user(added through user assertion USB) is not valid anymore, so we can’t ssh in at all.
Update:
The only partition we see after flashing this faulty image is ubuntu-seed. The remaining expected partitions (ubuntu-boot, ubuntu-data, and ubuntu-save) are not being created. Here’s the output from lsblk:
It’s normal for ubuntu seed to be the only volume in the install image. The others are created during the installation process, which is itself seeming to fail for some reason.
In order to debug failed installs and find the reason, you can interrupt the first boot after writing the image, and enter some custom kernel command line options to force a rescue shell to be created, then enter that shell and look investigate or setup remote access.
Instructions follow.
Press Esc key as soon as the BIOS is done to open the grub menu [ you can hit Esq multiple times without issue ]
Hit “e” to edit the kernel cmdline for the “Install …” option in grub
At the end of the line that begins with “chainloader (loop)/kernel.efi …” add the following:
dangerous systemd.debug-shell=1
Hit Ctrl+X
Once the image gets to the “Installing the system, …” message, press the Ctrl + Alt + F9 keys to switch TTY shells and get the debug shell.
It turns out that the TPM had to be cleared using the following command, since it was entering lockout mode:
echo 5 | sudo tee /sys/class/tpm/tpm0/ppi/request
After that, the previously missing partitions ubuntu-boot, ubuntu-save, and ubuntu-data are now being created.
Both ubuntu-save and ubuntu-data are correctly encrypted, as expected. I’m unable to mount them directly since they use LUKS, which is also expected.
However, UC boot still doesn’t complete. Instead, I’m prompted with the following message:
Please enter the recovery key for disk /dev/disk/by-partuuid/<UUID>: (press TAB for no echo)
This happens even when the device is connected to the internet during boot, so I assume snapd should be able to fetch any required assertions if needed.
I’m using a custom UC22 image with "storage-safety": "secured" (also tried ‘signed’) in the model assertion.
The board has Firmware TPM (PTT) enabled in the BIOS. From the OS side, the TPM is visible and functional (tpm2_getcap properties-fixed returns expected value TPM2 manufacturer ‘INTC’).
The TPM is no longer in lockout mode (inLockout: 0), and all tpm2_getcap properties-variable values appear normal.
What I’m trying to understand:
Is there any limitation when using firmware TPM compared to discreet?
Could an invalid or missing user.assert cause the system to fall back to the recovery key prompt?
Is there any log or file I can inspect (via Ubuntu Live or recovery mode) to understand why the system is asking for the recovery key?
Is it possible to inject debug hooks or scripts into the recovery or install phase of UC to capture key sealing events or potential errors?
Is there a way to preload or retrieve the recovery key outside the device before the FDE step completes?
Any guidance would be much appreciated. I’m currently blocked by this.
I’m not sure about hooks, perhaps someone else knows. But, you can try the method I poisted above to get a recovery shell during install and look at the snap changes. However, in my experience if the install errors it’s normally visible at the time. You may have to try and grab a video of the screen or capture console output somehow in order play it back and spot it.
Not to my knowledge.
Generally, I’ve had similar issues in the past, where the install happened but the disk failed to unseal immediately afterwards. In those cases it was found to be not properly implemented firmware, which required some special handling to be added.
In case it’s the same issue, you can try adding this kernel commandline via the your gadget.
Tried using the ubuntu-core.force-old-efi-handover kernel command-line option, but unfortunately it didn’t resolve the issue, and the recovery key prompt still appears during boot.
The logs don’t point to any specific failure or explanation for why the keys are being requested. On the other hand, I can confirm that both ubuntu-data and ubuntu-save are being encrypted as expected (logs show messages indicating the sealing process of encryption keys to the TPM for both partitions).
In addition to clearing the TPM (via BIOS and echo 5 | sudo tee /sys/class/tpm/tpm0/ppi/request), I also tried updating to kernel 6.8.0-64.67~22.04.1 (rev 2604) and flashing the latest BIOS firmware. The outcome remains unchanged: recovery keys are still being requested on boot.
Is there anything else I could try to unblock this?
Yes, I tried using snapd.debug=1 systemd.journald.forward_to_console=1, but nothing relevant to why the keys aren’t being unsealed is shown in the output.
Please check the docs to confirm, but I’m pretty sure that kernel-initrd-modules is not a gadget configuration item, but something that you would use in a kernel snap configuration [ when creating a kernel snap ].
In ubuntu core, the kernel snap contains the initrd, and it’s not possible to modify it on an installed system in the same way you might do on more traditional linux distros. This is mostly relating to security [ it’s part of the signed efi executable ]. Therefor, if you need to modify the initrd you have to create a your own kernel snap. As you are trying to get FDE to work, you will then have to manage the keys and signing too in order to for secure boot to work with a custom snap [ which is required for FDE ].
I’ve not tried to get netcondsole to work on UC, but it might be worth just checking if the standard pc-kernel has netconsole enable or not in the initird or not. You can extract the initrd from the pc-kernel snap, and check it’s contents.
If it’s not included, you could try a feature request or a bug report to add it, it seems like a reasonable enhancement, although that’s not going to help you in the short term.
