Custom UC20 Image for RPi4 fails

device
#1

Hi all,

I was experimenting with the custom UC20 image creation process for my RPi4.
After creating and flashing the image, I have an expired public key error which makes the system unbootable.
It is quite strange that the mentioned keys are just registered and everything seems fine. In addition to that, I did not really understand how the system checks if these public keys are valid or not without internet access.

Here is the logs;

Here you can also see the model assertion file.

Any idea about this problem?

PS: If I try to boot from the pre-built Ubuntu-Core RPi4 image(arm64), everything is perfect.

#2

Hi unfortunately this is a known bug, the issue is that the time in the initrd is stuck at Nov 2019, so if you created the key that you signed your model assertion with after that date, then the image is unable to boot. You can check when your key was created with something like this (assuming that you replace NAME with the name of the key you used with snap sign -k, if you didn’t use -k, then the name is default):

snap known account-key --remote public-key-sha3-384=$(snap keys | grep -Po '^NAME\s+\K.*') | grep since

This issue is tracked in a few different bugs, mostly assigned to the foundations team / @xnox to work on at some point, I will let @xnox comment on a timeline to addressing those bugs. The bugs are:

Built UC20 Rasperry Pi image hangs on boot
UC20: enabling Secure Boot + FDE on Raspberry PI4
#3

@ijohnson As you mentioned, unfortunately, I created and registered the key today.
Just out of curiosity, is there any workaround or dirty-hack that I could do to get rid of this problem?

1 Like
#4

You could try to attach a RTC to the pi which has an associated kernel module that is built into the kernel, then the kernel would refresh the time to whatever the RTC says, but I’m not sure which RTC’s are supported builtin to the kernel unfortunately…

Alternatively you could rebuild the kernel with a newer timestamp that systemd reads from, but this is fiddly and time consuming unfortunately.

#5

but timestamp is a free form field no? you can make your key be since anytime you want?

#6

@mvo please explain this to me again sometime. snapd in the initrd will never have the right time, and it can be in the past or the future relative a fixed point in time, whilst the models & keys can come from the past or the future.

Checking for since, without until is kind of obsurd. I am at the point where i’m going to run snap-bootstrap under faketime set to year 3000.

#7

given timestamp is a freeform field, can you set to 2018-01-01? resign the model and try again?

#8

I already tried that, but ubuntu-image tool basically complains that the timestamp is outside of signing key validity and stops working.

error: cannot fetch and check prerequisites for the model assertion: cannot add assertion model  model assertion timestamp outside of signing key validity (key valid since "2021-02-25 09:18:29 +0000 UTC")
COMMAND FAILED: snap prepare-image ubuntu-core-bugra.model /tmp/tmpo_551nkn/unpack
2 Likes
#9

Ok, this is actually very useful information!

I think I can make initrd never fail to boot, and yet have more reliable timesource.

grep -e 'since:' -e 'timestamp:' *
model-etc:since: 2016-08-31T00:00:00.0Z
snaps:since: 2016-04-01T00:00:00.0Z
snaps:timestamp: 2018-05-22T17:54:12.521666Z
snaps:timestamp: 2020-12-02T06:49:21.858897Z
snaps:timestamp: 2016-09-05T18:41:45.924828Z
snaps:timestamp: 2020-11-19T02:08:10.474594Z
snaps:timestamp: 2020-10-06T15:02:34.390372Z
snaps:timestamp: 2020-12-01T22:56:02.766604Z
snaps:timestamp: 2016-09-05T18:39:57.749968Z
snaps:timestamp: 2020-11-02T17:47:32.264069Z

If any of these timestamps are tampered with the digital signatures will fail, it means in grade: signed model can set system clock to at least any of those.

UC20 by default cannot trust any unverified things that can be tampered with for time validity.

#10

@xnox as you have found you can’t simply change the timestamp, since you need to change the timestamp on the account-key assertion which is what signs the model assertion, and account-key assertions are signed by canonical store keys, which will not sign something in the past for you.

I think at a minimum the first thing that we can do is have ubuntu-core-initramfs’s systemd bump it’s epoch every time it gets SRU’d, that would at least help in this case where someone just wants to build an image, the longest someone would be stuck waiting to be able to boot their image on the Pi if they register a new key would be until the next systemd SRU (which is not ideal but better than the current situation). Perhaps this kind of feature could even be extended to fast forward the time in the initrd to when the kernel initrd was built, by dropping some sort of epoch file in the initramfs that isn’t tied to systemd’s SRU cycle and thus have the time in the initrd monotonic and increase basically every kernel SRU cycle.

I think there are two issues with your approach of getting the time from these assertions:

  1. the initrd shouldn’t be using information from assertions if it is not fully parsing them and verifying their signatures, which currently is the role of snap-bootstrap/snapd (unless you want to write an assertion library in shell :upside_down_face:), I don’t know exactly what kind of attack could be formulated here, but I imagine there is one, where the initrd reading info blindly out of assertions leads to bad things
  2. This creates a circular root of trust in that snap-bootstrap trusts that the time is correct in order to validate things like the model assertion before considering the individual snap-declaration assertions IIRC, so if the initrd already got it’s time from these assertions OOTB from snap-bootstrap, then when snap-bootstrap is validating that the assertions are correct it’s root of trust to time actually comes from the thing that it is verifying, creating a circular root of trust.

I think a more reasonable solution to our time woes in the initrd is to do what Ubuntu Core 16/18 did (and was discussed on the bug) and use something like fixrtc-mount to inspect the time that the superblock was last mounted, or alternatively start injecting some sort of file upon image build time that serves the same basic purpose.

#11

that must not be done under secureboot or any other type of signed boot.

re circular loop => it’s not quite circular. We know that root key is created in the past, and we are looking at signatures. Think of it as a signed time source stamp file basically. Similar to /var/lib/systemd/timesync/clock but signed.

#12

Well the other issue with looking at timestamp signatures of i.e. snap-declaration assertions is that it still wouldn’t move the clock forward enough for @bugraaydogar to have booted their image, since the most recent timestamp from your image is from 2020-12-02 and you would have folks who created their keys recently still need to wait until a new snap-declaration or snap-revision assertion was issued after they created their keys. If that’s the best we can do I think just bumping the epoch time every time we build the initrd is a better solution and it also avoids having some other service in the initrd parse assertions and figure out what time to use.

#13

@ijohnson

But the seed partition not only has snap revisions, but also the key-account assertions and models. I.e. there is timestamp of when key was created too, chained to the root of trust. If I’m reading things right.

#14

@xnox yes after discussion @pedronis is implementing the feature you requested using timestamps from assertions, it requires refactoring our assertions codebase to use the timestamp from the kernel as a lower bound on the time, and validate that all of the assertions are valid within the open ended time interval starting at the kernel timestamp. This ensures that we are not still vulnerable to the issue around someone booting a super old image with revoked root of trust using a newer kernel.efi that we signed and the system trusts, and that in order to boot that super old image it has to be booted as-is with the old kernel.efi that came with it.

#15

I’ve got the similar question :slight_smile: Do we have an issue to follow up?

1 Like
#16

Relevant snapd PR’s have been merged, so it’s just waiting for the next snapd release, and then after that for the initramfs to be rebuilt and then the kernel to be rebuilt. Moving time forward to when the kernel was built was already landed in the edge kernels AFAIK, but moving the time even more forward to when the keys were created will not be released until snapd 2.50 (the next snapd release) in a few weeks.

1 Like
#17

Thanks @ijohnson! Also found this commit should be one of the fixes, https://github.com/snapcore/snapd/commit/0409b0d0e9637964cfe6639abc428e5967725a11. :slight_smile:

#18

I don’t find the commit for ubuntu-core-initramfs project. The latest commit is last May.

#19

I used the stable core20 image from the site cdimages which cannot boot. Will give it a try for edge image and see how it goes.

#20

This commit is part of the fix, but the final part which ties everything together in snapd is from this PR: https://github.com/snapcore/snapd/pull/10051 which is on edge snapd now, but won’t be included into the initramfs until snapd is released to stable, at which point rebuilding the initramfs will pick up the fix.

The commit for the initramfs you are looking for is actually now in the core-initrd project: https://github.com/snapcore/core-initrd/commit/c8cba5b6b6d97b3ea8d723f7951da500be348777.