Device gets bricked after trying to upgrade Core snap

Hi,

This seems true, but…
After some tests I found that uboot version 2013.01.01 still can sometimes brick a device.
I can’t say wether it’s this concrete version or all of them including 2017, but power-cycling the device when bootloader saves environment variables leads to corruption of uboot.env file.
After that you see:

reading uboot.env
FAT: Misaligned buffer address (3fed6738)
Invalid FAT entry
*** Warning - bad CRC, using default environment

Default environment doesn’t have snappy_boot variable and as a result will not boot UbuntuCore at all, neither kernel nor initrd.

Standard UbuntuCore snappy_boot saves environment every time it boots, not only when something gets updated. So that the probability of that fail is very high. I had some tests when I tried to catch the moment of writing environment to FAT and trigger the power off. And I can say that it’s not difficult to achieve, I get this in 5th trial.

Moreover the first time I had this situation was a casual work with device when I turned it on, changed my mind, turned it off in 1.5 second and I got precisely in that moment of saving uboot.env.

After that disaster I put my SD card in a card-reader and:

denis@denis-m1:~$ sudo fsck.fat /dev/sdc1
[sudo] password for denis: 
fsck.fat 3.0.28 (2015-05-16)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
1) Remove dirty bit
2) No action
? 1
/UBOOT.ENV
  Contains a free cluster (49455). Assuming EOF.
/UBOOT.ENV
  File size is 131072 bytes, cluster chain length is 0 bytes.
  Truncating file to 0 bytes.
Perform changes ? (y/n) y
/dev/sdc1: 91 files, 26551/258078 clusters

Well, the result is sad, uboot.env is lost and so the ability to boot.

Is it inevitable behavior or I just miss something?

P.S. And BTW, I don’t see how “RUDUNDANT” works here in the case of FAT write to uboot.env. For example when you use SPI-NOR flash or NAND flash with CONFIG_SYS_REDUNDAND_ENVIRONMENT, you also have CONFIG_ENV_OFFSET_REDUND. And there are 2 copies present. And I suppose it can revert back to healthy environment every time something happens. I never saw it in action but at least this is present in uboot sources. In env_fat.c OFFSET is not used.

env_nand.c:

#ifdef CONFIG_ENV_OFFSET_REDUND
void env_relocate_spec(void)
{
// --------------- cut--------------------------------------
	read1_fail = readenv(CONFIG_ENV_OFFSET, (u_char *) tmp_env1);
	read2_fail = readenv(CONFIG_ENV_OFFSET_REDUND, (u_char *) tmp_env2);

	if (read1_fail && read2_fail)
		puts("*** Error - No Valid Environment Area found\n");
	else if (read1_fail || read2_fail)
		puts("*** Warning - some problems detected "
		     "reading environment; recovered successfully\n");

Then it checks crc32 and handles env_valid flags.

2 Likes

Hello,
I faced same issue, can you tell me how to solve it?
thank you very much!

Best Regards
Qing

note that the basic issue with the corruption seems to have been a bug in fsck from dosfstools. it was fixed in:

@chang.qing you really need to provide some more info than that, what device, which core snap version has been there before the upgrade, was it only core that got upgraded or also the kernel etc etc …

Dear ogra:
I say the CONFIG_SYS_REDUNDAND_ENVIRONMENT in uboot. I found it do nothing for saveenv() and is follow:
#ifdef CONFIG_CMD_SAVEENV
#ifdef CONFIG_SYS_REDUNDAND_ENVIRONMENT
static unsigned char env_flags;

int saveenv(void)
{

if (gd->env_valid == 1) {
puts("Writing to redundant UBI… “);
if (ubi_volume_write(CONFIG_ENV_UBI_VOLUME_REDUND,
** (void *)env_new, CONFIG_ENV_SIZE)) {
printf("\n
Unable to write env to %s:%s **\n”,
CONFIG_ENV_UBI_PART,
CONFIG_ENV_UBI_VOLUME_REDUND);
return 1;
}
}

return 0;

}
#else

but it did’nt add the special header in env file as you described on your blog. and did’t save any special header. I used the uboot version is 2016.5 for imx6q
soc.

And another issue: If I don’t want to use initrd to boot
the ubuntu core and want to use uboot boot kernel , then use kernel to boot ubuntu core directly, then how can I do?

Thanks!

Best Regards
Qing

it does something in the upstream code, there is definitely another place where the define is used when you have properly enabled FAT support and an uboot.env file in your system setup for the board.

This is a hard requirement, snapd and all other system tools in userspace require the file to be in the correct format. If you do not use this, writing to the environment on kernel or core updates will fail and your system will not properly boot.

This is not possible, the rootfs gets assembled from a readonly squashfs and multiple writable directories that get bind-mounted on top of the readonly parts. This works through initramfs scripts that are essential to the system setup so you can not boot Ubuntu Core without initrd.

It would be nice if you could actually open a new topic specific to your board problems in the device category where we can go on discussing this, since this topic is specifically about a failed core snap upgrade and not really related to imx6 boards or any other specific devices.