Barebones Core22 image fails to boot using pc-kernel 22-hwe/stable

I’ve been having issues with the pc-kernel 22-hwe track images failing to boot. The boot failure occurs due to a kernel panic when opening the root device. This occurs both on an Intel NUC13anhi7 and qemu.

Here’s the qemu trace:

[    0.651875] /dev/root: Can't open blockdev
[    0.652808] VFS: Cannot open root device "" or unknown-block(0,0): error -6
[    0.653749] Please append a correct "root=" boot option; here are the available partitions:
[    0.654822] fd00         3097600 vda 
[    0.654829]  driver: virtio_blk
[    0.658604]   fd01            1024 vda1 4fa06cee-60c7-4be7-a22c-d4cb8e2fff59
[    0.658607] 
[    0.660521]   fd02         1228800 vda2 ac28c6e8-afaa-489b-b256-aeb617d68a18
[    0.660523] 
[    0.661772] List of all bdev filesystems:
[    0.662408]  ext3
[    0.662409]  ext2
[    0.662752]  ext4
[    0.663180]  squashfs
[    0.663528]  vfat
[    0.663940]  fuseblk
[    0.664283] 
[    0.665167] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.666625] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.8.0-44-generic #44~22.04.1-Ubuntu
[    0.667731] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[    0.668836] Call Trace:
[    0.669352]  <TASK>
[    0.669817]  dump_stack_lvl+0x27/0xa0
[    0.670462]  dump_stack+0x10/0x20
[    0.671866]  panic+0x366/0x3c0
[    0.672402]  mount_root_generic+0x1a5/0x350
[    0.673165]  mount_root+0x98/0x100
[    0.673972]  ? __pfx_kernel_init+0x10/0x10
[    0.674608]  prepare_namespace+0x6c/0x2f0
[    0.675224]  kernel_init_freeable+0x1c6/0x210
[    0.676350]  kernel_init+0x1b/0x200
[    0.676923]  ret_from_fork+0x44/0x70
[    0.677474]  ? __pfx_kernel_init+0x10/0x10
[    0.678112]  ret_from_fork_asm+0x1b/0x30
[    0.678682]  </TASK>
[    0.679169] Kernel Offset: 0x2ea00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

Here’s my seed.manifest:

bare 5
core22 1621
pc 146
pc-kernel 1985
snapd 21759

Notice it is using pc-kernel 1985 which is the 22-hwe/stable revision at the time of this post.

$ snap info pc-kernel
...
  22-hwe/stable:    6.8.0-44.44~22.04.1 2024-09-19 (1985) 293MB -

If use an old 22-hwe revision, 1921, the boot is successful.

$ snap download pc-kernel --revision 1921
$ ubuntu-image snap model.assert --snap pc-kernel_1921.snap
$ qemu-system-x86_64 ... pc.img
$ ssh localhost -p 8022
$ snap info pc-kernel
...
installed:          6.5.0.44.44~22.04.1            (1921) 285MB kernel

If i then run snap refresh pc-kernel and let the machine reboot, the next boot fails and kernel snap is reverted.

I can successfully change the channel to 22-oem/stable which is revision 1954 and still the 6.5 kernel series

Reproduction is simple; just create a basic image using ubuntu-image and boot using qemu from the docs

Here’s my model.json:

{
    "type": "model",
    "series": "16",
    "authority-id": "***",
    "brand-id": "***",
    "system-user-authority": "*",
    "model": "ubuntu-core-22-amd64",
    "architecture": "amd64",
    "timestamp": "2024-09-25T13:52:46+00:00",
    "base": "core22",
    "grade": "dangerous",
    "snaps": [
        {
            "name": "pc",
            "type": "gadget",
            "default-channel": "22/stable"
        },
        {
            "name": "pc-kernel",
            "type": "kernel",
            "default-channel": "22-hwe/stable"
        },
        {
            "name": "bare",
            "type": "base",
            "default-channel": "latest/stable"
        },
        {
            "name": "core22",
            "type": "base",
            "default-channel": "latest/stable"
        },
        {
            "name": "snapd",
            "type": "snapd",
            "default-channel": "latest/stable"
        }
    ]
}

Please advise if I should revert to the 22-oem series for the 6.5 kernel instead of the 6.8 kernel. These images have been built using the 22-hwe/stable series for the past ~2 years in order to use some networking hardware not available in the 5.15 kernel.

Hi @cparich

Did you ever get to the bottom of this ?

I’m also just trying 22-hwe/stable and it fails to boot. I don’t get the same panic however, indeed there is no obvious message relating to the reason, but the devise resets about 1 second into early boot.

This is on some dell x86 hardware.

Cheers, Just

That’s the same revision I tried. I also tried the current revision in edge, which is 2006.

Do you happen to know what the last working revision you used was ?

Cheers, Just

Ahh, sorry, I missed that point :slight_smile:

Just to confirm, 1921 works for me too.

Cheers, Just

Did you ever get to the bottom of this ?

My best guess is it’s missing a filesystem driver for ext4 or the appropriate driver isn’t signed or some such considering the error line

[    0.651875] /dev/root: Can't open blockdev
[    0.652808] VFS: Cannot open root device "" or unknown-block(0,0): error -6
[    0.653749] Please append a correct "root=" boot option; here are the available partitions:

I switched my builds to 22-oem in the meantime, which is the 6.5 kernel; which seems to be good enough for my NUC 13ANHI7 devices.

Unfortunately, I need to use the Canonical kernels to get FDE support without a whole load of extra work on my end, so hopefully somebody from the kernel team sees this thread and fixes the issue using my repro above.