Built UC20 Rasperry Pi image hangs on boot

I am struggling to make a working UC20 image for a Raspberry pi 4 using ubuntu-image. After an img is created and flashed on a card, I’m left with a boot process that hangs at “starting kernel…”. So thinking it’s the fault of my gadget, I removed all the customizations to a base “vanilla” image which is all made of official snaps.
When I sign and built the image this time, the raspberry pi gets stuck at a black screen on boot (nothing at all on the screen). I tried with an image composed of stable and edge (core20, snapd, pi-kernel and pi gadget) snaps but obtained the same results.

After the img is created, I am write it to an SD card using balenaEtcher, (I also tried with dd with no avail).

What model assertion are you using to create the image? And if you are using your own model assertion, when did you create the key that you signed your model assertion with?

For the stable and edge “vanilla” images I used all snaps from the store:

model.json
{
    "type": "model",
    "series": "16",
    "authority-id": "BZn14KGiwBc3TVYC0rgBvDtkHrbYbspY",
    "brand-id": "BZn14KGiwBc3TVYC0rgBvDtkHrbYbspY",
    "model": "iris",
    "architecture": "arm64",
    "timestamp": "2021-04-14T15:55:57+00:00",
    "base": "core20",
    "grade": "dangerous",
    "snaps": [
        {
            "name": "pi",
            "type": "gadget",
            "default-channel": "20/stable",
            "id": "YbGa9O3dAXl88YLI6Y1bGG74pwBxZyKg"
        },
        {
            "name": "pi-kernel",
            "type": "kernel",
            "default-channel": "20/stable",
            "id": "jeIuP6tfFrvAdic8DMWqHmoaoukAPNbJ"
        },
        {
            "name": "core20",
            "type": "base",
            "default-channel": "latest/stable",
            "id": "DLqre5XGLbDqg9jPtiAhRRjDuPVa5X1q"
        },
        {
            "name": "snapd",
            "type": "snapd",
            "default-channel": "latest/stable",
            "id": "PMrrV4ml8uWuEUDBT8dSGnKUYbevVhc4"
        }
    ]
}

I used the output of date -Iseconds --utc, so a recent date.

and how old is the key you are signing this with ? there is currently a bug where too new keys make the first boot fail on Pi’s du to RTC issues …

1 Like

If you mean the date of creation (or registration) of the key itself I’d say sometime in the last two weeks (snap keys doesn’t give much more info). If you mean the timestamp pf the model assertion it’s definitely the same day (2021-04-14T15:55:57+00:00).

Is there a way to confirm that what I’m getting is due to the bug? If you mind, can you pointing me to the bug so I can watch it?

EDIT: From Custom UC20 Image for RPi4 fails
and using the command snap known account-key --remote public-key-sha3-384=$(snap keys | grep -Po '^NAME\s+\K.*') | grep since

The key dates from 2021-04-08T17:57:23Z, which is AFTER Nov2019.

The linked forum post also have bug pages, I’ll keep and eye out for when it gets fixed.

1 Like

Yes then you are running into the bug mentioned. You will need to wait until the fix has made it’s way through the release process, which will take a little bit unfortunately.

well, could he not set his PC clock to 2019, create a new key, move the clock back forward and have it worked around that way by using the key created with cheated date ? (not sure you feel this is feasible @dav)

or is it the upload date that counts ?

No because his key needs to be signed by the canonical key when he uploads the key (unless you can move the time backwards on the server that hosts the canonical key somehow - please don’t try :slight_smile: )

1 Like

LOL, i’m not that powerful

Thanks for the info, unless I find the motivation to re build the kernel snap (or buy a compatible RTC), I’ll put this on hold for now.
@ijohnson Sounds like the fix exists but is making its way through the release pipeline. If that’s the case, any chance you can share the estimated release date (this answer may influence buying an RTC module) ?

Hey @dav,
Recently, I tried the edge channel and it works. I hope, it helps you with your exploration of Ubuntu Core.

2 Likes

Thanks, I’ll give it a go.

It looks good until the very end:

Fetching snapd
Fetching pi-kernel
Fetching core20
Fetching pi
Fetching htop
Crash in state machine
Traceback (most recent call last):
  File "/snap/ubuntu-image/215/lib/python3/site-packages/ubuntu_image/__main__.py", line 393, in main
    list(state_machine)
  File "/snap/ubuntu-image/215/lib/python3/site-packages/ubuntu_image/state.py", line 82, in __next__
    step()
  File "/snap/ubuntu-image/215/lib/python3/site-packages/ubuntu_image/common_builder.py", line 348, in populate_bootfs_contents
    self._populate_one_bootfs(name, volume)
  File "/snap/ubuntu-image/215/lib/python3/site-packages/ubuntu_image/common_builder.py", line 329, in _populate_one_bootfs
    for filename in os.listdir(src):
FileNotFoundError: [Errno 2] No such file or directory: '/sharedmount/build/unpack/gadget/$kernel:dtbs/dtbs/broadcom/'

note:

  • /sharedmount is the shared directory between the docker container that builds the image and the host.
  • It’s running ubuntu-image 1.11+snap1
  • /sharedmount/build/unpack/gadget only contains boot-assets, meta and snap directories

EDIT: exploring the file in question (common_builder.py) , I ran into this comment block which sounds

                    # Not using "snap prepare-image" (or old snap
                    # binary) - the below code path will not
                    # understand new style "$kernel:" references in
                    # gadget.yaml and will fail if used with these
                    # references.

Makes me think I am somehow using an old binary…
I am building using this command: export UBUNTU_STORE_AUTH_DATA_FILENAME=/root/mycreds; ubuntu-image snap -w /sharedmount/build -O /sharedmount /sharedmount/iris-model.model

1 Like

@dav, I just gave it a try now and it worked here. I built the image on native Ubuntu Desktop 20.04.

ubuntu-image snap pi-model.assert

Probably, you are having an environment-related issue. Sorry it is hard to help without having a similar environment :slight_smile:

Well that encourages me to try harder. I’ll make a fresh Focal docker container and try again with as little tweaks as possible.

Got it to work. Not sure what did it, but I think upgrading snapd to --edge was the winning move.

2 Likes

Hi,
Thanks for those hints, I also observed the same probem with host’s snapd (stable/11402) :

No such file or directory: '/tmp/tmpcp3sp3cn/unpack/gadget/$kernel:dtbs/dtbs/broadcom/'

Update to edge/2.50+git1666.g799fa37 fixed the issue.

ubuntu-image --version
ubuntu-image 1.11+snap1
snap --version
snap    2.50+git1666.g799fa37
snapd   2.50+git1666.g799fa37
series  16
ubuntu  20.10

Now by setting pi and pi-kernel to “edge” channel, I used this recipe as source:

The pc one works as expected (on vm) not the pi one:

There is signal on HDMI showing black screen
then signal is gone (rebooting?)
and signal is back with black display

I tried combination of various channels but not luck yet.

Did I miss something ?

Related links:

The DTB handling changed in the edge channel … it will still take a bit until this fully lands on stable so better build with everything (kernel, gadget in the image … and on the build host: snapd, ubuntu-image) from edge to get properly working images ATM …

1 Like

Thanks for the step by step tutorial!