Boot.sel bad CRC after kernel refresh

gbisson · January 24, 2023, 10:25am

Hi,

I’m having an issue I can’t explain when using ‘snap refresh’ related to the state of boot.sel. First, I’m using a Nitrogen i.MX 8M Plus platform with those 2 snaps:

Yesterday I made an update to the kernel (6) and generated a new core20 using ‘ubuntu-image’ as defined here:

github.com

boundarydevices/ubuntu-core/blob/20-arm64/snappy.mk#L14


UBUNTU_CORE_CHANNEL ?= stable
UBUNTU_IMAGE := /snap/bin/ubuntu-image
SNAPPY_IMAGE := ubuntu-core20-arm64-nitrogen-$(UBUNTU_CORE_CHANNEL)-`date +%Y%m%d`.img
all: build
clean:
	rm -f $(OUTPUT_DIR)/*.img.gz
distclean: clean
build-snappy:
	@echo "build snappy..."
	$(UBUNTU_IMAGE) snap \
		-O $(OUTPUT_DIR) \
		$(IMAGE_MODEL)
pack: build-snappy
	mv $(DEFAULT_IMAGE) $(SNAPPY_IMAGE)
	gzip $(SNAPPY_IMAGE)
build: build-snappy pack

The resulting image is working fine. The previous image I had with the previous kernel (5) is also working fine.

However, when trying to ‘snap refresh’ on a system that had the previous kernel (5) installed, the system fails to boot because of a “bad CRC”:

error: fallback kernel snap unusable: cannot open "/run/mnt/ubuntu-boot/uboot/ubuntu/boot.sel": bad CRC 4101280109 != 950455135

When looking into the boot.sel generated by snapd during the refresh, it does include a CRC32 of 4101280109 (0xf474916d) which I can compute again myself:

dd if=boot.sel.update of=boot.sel.update2 bs=4 skip=1
crc32 boot.sel.update2
f474916d

However, it looks like the original boot.sel generated so not use the same method to generate the CRC and clearly snapd computes it differently too. I’m unclear what needs to be done at this stage to fix the corrupted boot.sel but I don’t know where to go from here as snapd should be the one generating the boot.sel in all cases.

Regards, Gary

valentind · January 24, 2023, 5:36pm

I think there should be one more byte for the flags. See what you get if you skip 5 instead of 4 bytes. If you then get 38a6cb5f, then it means the file is fine but u-boot is not reading the flag as a flag.

Have you configured your u-boot with CONFIG_SYS_REDUNDAND_ENVIRONMENT=n?

gbisson · January 24, 2023, 5:56pm

Do you mean the opposite? I skip 4 instead of 5?

I confirm that I don’t have REDUNDAND_ENV enabled. Here is how I import the env:

github.com

boundarydevices/ubuntu-core/blob/20-arm64/gadget/bootscript.txt#L49


# Import U-Boot variables set at build time
setenv core_state "/uboot/ubuntu/boot.sel"
if load ${devtype} ${devnum}:${distro_bootpart} ${a_script} ${core_state}; then
	# Core image; set defaults for core's state then import from the first
	# boot.sel file. Then load the mutable boot.sel file from the next partition
	setenv kernel_vars "snap_kernel snap_try_kernel kernel_status"
	setenv recovery_vars "snapd_recovery_mode snapd_recovery_system snapd_recovery_kernel"
	setenv snapd_recovery_mode "install"
	setenv snapd_standard_params "panic=-1 systemd.gpt_auto=0 rd.systemd.unit=basic.target net.ifnames=0"
	# Remove CRC check for now as u-boot might not have redundant env
	env import ${a_script} ${filesize} ${recovery_vars}
	setenv bootargs "${bootargs} snapd_recovery_mode=${snapd_recovery_mode} snapd_recovery_system=${snapd_recovery_system} ${snapd_standard_params}"
	if test "${snapd_recovery_mode}" = "run"; then
		setexpr distro_bootpart ${distro_bootpart} + 1
		load ${devtype} ${devnum}:${distro_bootpart} ${a_script} ${core_state}
		env import ${a_script} ${filesize} ${kernel_vars}
		setenv kernel_name "${snap_kernel}"
		if test -n "${kernel_status}"; then

What’s funny is that I can see a 5th bit in the original boot.sel:

$ xxd boot.sel.old
00000000: 48c0 30b8 0073 6e61 705f 6b65 726e 656c  H.0..snap_kernel
00000010: 3d6e 6974 726f 6765 6e2d 6b65 726e 656c  =nitrogen-kernel
00000020: 5f35 2e73 6e61 7000 00ff ffff ffff ffff  _5.snap.........

However in the boot.sel after the upgrade it isn’t there:

$ xxd boot.sel.update
00000000: 6d91 74f4 6b65 726e 656c 5f73 7461 7475  m.t.kernel_statu
00000010: 733d 7472 7969 6e67 0073 6e61 705f 6b65  s=trying.snap_ke
00000020: 726e 656c 3d6e 6974 726f 6765 6e2d 6b65  rnel=nitrogen-ke
00000030: 726e 656c 5f35 2e73 6e61 7000 736e 6170  rnel_5.snap.snap
00000040: 5f74 7279 5f6b 6572 6e65 6c3d 6e69 7472  _try_kernel=nitr
00000050: 6f67 656e 2d6b 6572 6e65 6c5f 362e 736e  ogen-kernel_6.sn
00000060: 6170 0000 0000 0000 0000 0000 0000 0000  ap..............

Any idea why the boot.sel format seems to change between original boot & after refresh?

valentind · January 24, 2023, 6:53pm

When does it fail? Where do you get the “bad CRC” error message? Is that snapd that gives this error?

I wonder if the file is being written somewhere by u-boot and snapd cannot read it. I think it needs to read kernel_status, snap_kernel and snap_try_kernel.

gbisson · January 24, 2023, 7:11pm

It fails after the reboot triggered by snapd after the refresh.

Yes the bad CRC is a snapd message:

github.com

snapcore/snapd/blob/master/bootloader/ubootenv/env.go#L113


	if len(contentWithHeader) < headerSize {
		return nil, fmt.Errorf("cannot open %q: smaller than expected header", fname)
	}
	crc := readUint32(contentWithHeader)
	payload := contentWithHeader[headerSize:]
	actualCRC := crc32.ChecksumIEEE(payload)
	if crc != actualCRC {
		return nil, fmt.Errorf("cannot open %q: bad CRC %v != %v", fname, crc, actualCRC)
	}
	if eof := bytes.Index(payload, []byte{0, 0}); eof >= 0 {
		payload = payload[:eof]
	}
	data, err := parseData(payload, flags)
	if err != nil {
		return nil, err
	}

And no this is not U-Boot creating that file, it only reads it.

valentind · January 25, 2023, 8:50am

It must be writing somewhere. It has to. Otherwise the update would not work.

Your issue is that snapd does not support CONFIG_SYS_REDUNDAND_ENVIRONMENT=n.

There is a PR that has been opened recently. This should fix your issue:

gbisson · January 25, 2023, 9:17am

Thanks, it looks like this would indeed solve my issue! I’ll wait until this gets merged and update the thread then.