Why snap_core and snap_kernel in boot cmdline

Hello All,

I am trying to build a custom ubuntu-core image for Olimex Lime2 platform board with a custom gadget snap and kernel snap. I have also managed to create a boot-able image.
I was referring to the U-boot environment variables from pi3-gadget snap (github) where I found that “snap_core=${snap_core} snap_kernel=${snap_kernel}” variables are passing as Kernel Command line arguments.
I tried to search to find out why they are used and which particular module in kernel use that but unable to find it.
Any help regarding this info would be greatly appreciated.

2 Likes

The rootfs is dynamically assembled on every boot out of the readonly core snap and the /writable/system-data directories by the initrd. To do this the info which core snap to use is read from the kernel cmdline.

Beyond this, there is logic in snapd that sets “snap_try”, “snap_try_kernel” and “snap_try_core” when the kernel or core snap get upgraded. The snappy_boot script in the bootloader then reads these variables, does a boot in “try” mode and unsets all the “*_try” variables if the boot succeeded … does it not succeed, the variables will be set back to their former state (i.e. your system automatically goes back to the working version).

The original values for these variables will be injected into the bootloader environment by ubuntu-image at image creation time.

2 Likes

@ogra Thanks for reply. Please correct my understanding.

Rootfs is dynamically assembled from readable core snap and the /writable/system-data directories. It is done by initrd image.

I have below further queries:
a) Which program/binary/module in initrd image perform this assembly by using snap_core kernel command line argument?
b) There is also kernel.snap available in /writable/system-data/. Which module shall use that kernel.snap? Is it snapd daemon which shall use that kernel.snap? Or any other module.

Appreciate your support here.

The mounting of the various bits is done in the ubuntu-core-rootfs script the initrd ships.

The process is like:

  • Look for the partition that ubuntu-image labeled “writable” and mount it under a temporary dir in the initrd.
  • Find the core snap on that partition and mount it under /root …
  • The core snap has a “/writable” dir inside to serve as a mountpoint, move the mount from the first step from the temporary dir to /root/writable
  • The /writable dir contains /witable/system-data which has all the writable data your rootfs needs and a file describing them (/etc/system-image/writable-paths)
  • Process /etc/system-image/writable-paths to create bind mounts from /writable/system-data/* to /root/*

Now you have a complete rootfs underneath /root, combined from the readonly snap (which is simply a signed squashfs file) and the dirs under /writable/system-data.

When you boot you also want the system to be able to load kernel modules and firmware for your hardware. For this the kernel snap gets mounted in the location that ubuntu-image initially prepared for it under /writable/system-data/snap/$snap_kernel and the “modules” and “firmware” dirs in there get bind mounted to /root/lib/*

Now that you have a fully assembled rootfs with modules and firmware underneath /root, the system re-execs with /root as /, removes the initrd from ram and boots with systemd as PID 1.

5 Likes

Thanks @ogra for reply. As you have mentioned above mounting of various bits is done in “ubuntu-core-rootfs” script the initrd ships. Do we need to perform some modifications in kernel source code in order to execute the script while booting?

Now, My understanding is that above mentioned items shall be handled by “ubuntu-core-rootfs” script. Please correct me if I am wrong here.

In order to perform this do we need some modification in kernel source code?
The initrd ships lib/systemd/systemd-udevd so kernel will boot with it (“lib/systemd/systemd-udevd”) as PID 1. This we need to pass as “init=” command line arguments else kernel will look for other process.

Note: I am using Linux Kernel v4.10.10 for this activity. In order to support the above requirements do I need to perform some modifications in kernel source code.

Appreciate your support here.

The writable partition uses an ext4 filesystem, the system-boot partition uses vfat and the snap packages are squashfs files, so you need support for these four filesystems at least. There are a bunch of other config options that need to be enabled but i think the snapcraft kernel plugin properly checks for all of them and warns you when you build the kernel snap (@ppisati may correct me here).

… exactly … the above steps were describing what the ubuntu-core-rootfs script does.

Apart from all the general bits you would enable in any distro you will not need any kernel modifications to switch to /

The kernel commandline and bootloader config are all handled automatically through ubuntu-image and the defaults the gadget snap ships in uboot.env.in. If you use one of the existing gadgets as template, you should not need to touch the cmdline manually unless you need some special hardware options related to your board.

Thanks @ogra for your support.

As of now when we build kernel snap using kernel.yaml then ramdisk image (initrd) is build in “.img” format but u-boot bootloader expect the ramdisk image format in “uInitrd”. I checked the kernel plugin script currently there is no support to create initrd image in “uInitrd” format. Please correct me if I am wrong here and provide me an option to use if any to build initrd image in “uInitrd” format.

Appreciate your support here.

Well, so fix uboot to actually use initrd.img files … you can follow my blog entry about this.

Since the Olimex is an allwinner chipset as far as i know, i have a few allwinner gadget snaps here and here that you might find helpful to use as templates.

1 Like

Thanks @ogra for your support. I will modify u-boot and try to boot the board

@ogra: I tried to fix the u-boot boot-loader and managed to boot the ubuntu-core image on Olimex Lime 2 for first time. Now when I restart it again then system is not able to come up. I am getting below error in boot logs

“mount: mounting /dev/loop0 on /root failed: Invalid argument”

I have a doubt that which binary/module is trying to mount /dev/loop0 on /root. And if it gets failed even then it is allowing (from ubuntu-core-rootfs) to mount core_1690.snap to /root but no rootfs is present in /root directory.

Is it possible that mounting of /dev/loop0 failed on /root will result in failure of mount core_1690.snap on /root (Though I am not getting any error in boot logs when try to mount core_1690.snap to /root). I checked that core1690.snap was present in /tmpmnt_writable/system-data/var/lib/snapd/snaps.

Even I checked that kernel.snap is present in /tmpmnt_writable/system-data/var/lib/snapd/snaps and I am not getting error while mounting kernel.snap to /tmpmnt_kernel. But when it goes further then I get below errors which indicates that kernel.snap was not mounted properly /tmpmnt_kernel.

“mount: mounting /tmpmnt_kernel/modules on /root/lib/modules failed: No such file or directory”
“mount: mounting /tmpmnt_kernel/firmware on /root/lib/firmware failed: No such file or directory”

Appreciate your support here:

PS: Kernel command line arguments
Kernel command line: console=ttyS0,115200 rootfstype=ext4 root=/dev/mmcblk0p2 net.ifnames=0 panic=-1 init=/lib/systemd/systemd ro panic=-1 fixrtc snap_core=core_1690.snap snap_kernel=olimex-a20-olinuxino-lime2-linux_x1.snap rootwait

this line is definitely not proper, check what uboot.env.in in my example gadgets uses for root= … you also do not need rootfstype or rootwait, remove them from you config.

i’m pretty sure there is another error before the /dev/loop0 about it not finding the writable partition.

Thanks @orga for you prompt response. I modified the “root=” parameter as mentioned in you gadget-snaps examples.
But still I am facing same problem. I am also providing below boot logs for your reference.

Boot Logs:
[ 5.650078] usb0-vbus: disabling
[ 5.655102] Freeing unused kernel memory: 1024K
Loading, please wait…
starting version 229
Begin: Loading essential drivers … done.
Begin: Running /scripts/init-premount … done.
Begin: Mounting root file system … Begin: Running /scripts/nfs-top … done.
Begin: Running /scripts/nfs-premount … done.
Begin: Running /scripts/local-top … done.
Begin: Running /scripts/local-premount … ext4
done.
[ 7.282867] initrd: checking filesystem for writable partition
[ 7.298463] EXT4-fs (mmcblk0p2): couldn’t mount as ext3 due to feature incompatibilities
[ 7.308074] EXT4-fs (mmcblk0p2): couldn’t mount as ext2 due to feature incompatibilities
[ 7.345587] EXT4-fs (mmcblk0p2): recovery complete
[ 7.351974] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
ext4
[ 7.466606] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 7.519176] EXT4-fs (mmcblk0p2): couldn’t mount as ext3 due to feature incompatibilities
[ 7.528576] EXT4-fs (mmcblk0p2): couldn’t mount as ext2 due to feature incompatibilities
[ 7.553185] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 7.932370] UDF-fs: warning (device loop0): udf_fill_super: No partition found (2)
[ 8.062261] F2FS-fs (loop0): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[ 8.069140] F2FS-fs (loop0): Can’t find valid F2FS filesystem in 1th superblock
[ 8.076592] F2FS-fs (loop0): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[ 8.083498] F2FS-fs (loop0): Can’t find valid F2FS filesystem in 2th superblock
[ 8.090823] F2FS-fs (loop0): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[ 8.097707] F2FS-fs (loop0): Can’t find valid F2FS filesystem in 1th superblock
[ 8.105021] F2FS-fs (loop0): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[ 8.111916] F2FS-fs (loop0): Can’t find valid F2FS filesystem in 2th superblock
mount: mounting /dev/loop0 on /root failed: Invalid argument
[ 8.427497] UDF-fs: warning (device loop0): udf_fill_super: No partition found (2)
[ 8.492343] F2FS-fs (loop0): Magic Mismatch, valid(0xf2f52010) - read(0x2491dba5)
[ 8.499831] F2FS-fs (loop0): Can’t find valid F2FS filesystem in 1th superblock
[ 8.507267] F2FS-fs (loop0): Magic Mismatch, valid(0xf2f52010) - read(0xe68730cb)
[ 8.514769] F2FS-fs (loop0): Can’t find valid F2FS filesystem in 2th superblock
[ 8.522110] F2FS-fs (loop0): Magic Mismatch, valid(0xf2f52010) - read(0x2491dba5)
[ 8.529582] F2FS-fs (loop0): Can’t find valid F2FS filesystem in 1th superblock
[ 8.536895] F2FS-fs (loop0): Magic Mismatch, valid(0xf2f52010) - read(0xe68730cb)
[ 8.544383] F2FS-fs (loop0): Can’t find valid F2FS filesystem in 2th superblock
mount: mounting /tmpmnt_kernel/modules on /root/lib/modules failed: No such file or directory
mount: mounting /tmpmnt_kernel/firmware on /root/lib/firmware failed: No such file or directory
[ 8.822511] initrd: mounting /run
mount: mounting tmpfs on /root/run failed: No such file or directory
mount: mounting /tmpmnt_writable on /root/writable failed: No such file or directory
mount: mounting /root/writable/system-data/etc/systemd/system on /root/etc/systemd/system failed: No such file or directory
mount: mounting /root/writable/system-data/etc/machine-id on /root/etc/machine-id failed: No such file or directory
Begin: Running /scripts/local-bottom … done.
done.
Begin: Running /scripts/nfs-bottom … done.
Begin: Running /scripts/init-bottom … mount: mounting /dev on /root/dev failed: No such file or directory
done.
mount: mounting /run on /root/run failed: No such file or directory
run-init: current directory on the same filesystem as the root: error 0
Target filesystem doesn’t have requested /lib/systemd/systemd.
run-init: current directory on the same filesystem as the root: error 0
run-init: current directory on the same filesystem as the root: error 0
run-init: current directory on the same filesystem as the root: error 0
run-init: current directory on the same filesystem as the root: error 0
run-init: current directory on the same filesystem as the root: error 0
No init found. Try passing init= bootarg.

BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
Enter ‘help’ for a list of built-in commands.

PS: Kernel Command Line args
Kernel command line: console=ttyS0,115200 root=/dev/disk/by-label/writable net.ifnames=0 panic=-1 init=/lib/systemd/systemd ro panic=-1 fixrtc snap_core=core_1690.snap snap_kernel=olimex-a20-olinuxino-lime2-linux_x1.snap

hmm, does your kernel have support for loop block devices ?

@ogra Yes kernel have support for loop block devices. Below flags are enabled in .config file
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8

that looks correct… if you are in that initramfs shell, do you actually see something in /dev/disk/by-label/writable ?
(could you pastebin the output of “mount” in that state)

@ogra: yes I am able to see that /dev/disk/by-label/writable is a soft link to mmblk0p2 (second partition in SD card). Please find below output of
a) ls -l /dev/disk/by-label/writable
(initramfs) ls -l /dev/disk/by-label/writable
lrwxrwxrwx 1 15 /dev/disk/by-label/writable -> …/…/mmcblk0p2

b) mount
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=502436k,nr_inodes=125609,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=102692k,mode=755)
/dev/mmcblk0p2 on /tmpmnt_writable type ext4 (rw,relatime,data=ordered)

this looks all fine as well …

does core_1690.snap exist in /tmpmnt_writable/var/lib/snapd/snaps/

(on a fresh install there should be at least gadged, core and kernel snap in this dir)

@orga: yes core_1690.snap exist in /tmpmnt_writable/var/lib/snapd/snaps/. Please find below outcome of ls in /tmpmnt_writable/var/lib/snapd/snaps/ directory.

Output:

(initramfs) cd tmpmnt_writable/system-data/var/lib/snapd/snaps/
(initramfs) pwd
/tmpmnt_writable/system-data/var/lib/snapd/snaps
(initramfs) ls -l
-rw-r--r--    1         0 core_1690.snap
-rw-r--r--    1  11005952 olimex-a20-olinuxino-lime2-linux_x1.snap

it does exist … but is 0 bytes big … something went clearly wrong during image creation i guess … did you just use ubuntu-image to build it or did you do anything manually ?

@orga: yes I used ubuntu-image tool to create a build. Anyways I tried to create a fresh build again and boot the system. The system was able to boot it successfully with ubuntu-core image and configuration settings window appeared. I tried to configure ethernet eth0 static IP address but I got a message (what could be the reason of this message).
" Network configuration timed out; please verify your settings."

Anyways then I tried to re-boot the system now this time booting failed with error message
mount: mounting /tmpmnt_writable/system-data/var/lib/snapd/snaps/core_2314.snap on /root failed: No such file or directory

I also checked that core_2314.snap was not present in /tmpmnt_writable/system-data/var/lib/snapd/snaps. What could be the reason for that? Is that my network configurations failed? or any other reasons?