Network Configuration :: First Ubuntu Core boot in custom hardware

ogra · June 23, 2021, 9:50am

ah, during first boot that partition should have been created already, you should only have to create the “/journal” directory in the existing var/log … that points to a problem with the initramfs your kernel snap uses (which is responsible for creating all the additional partitions beyond ubuntu-seed during the first boot …

is this UC20 ? if so, you should use the v2 kernel plugin to build your kernel snap:

jithcr · June 23, 2021, 10:06am

I am trying on UC18…

ogra · June 23, 2021, 10:08am

oops, sorry then (i was blindly assuming UC20), the data partition is called writable on UC18, check if there is a system-data/var/log/ in it …

jithcr · June 23, 2021, 10:11am

Yes, writable/system-data/var/log folder is there

ogra · June 23, 2021, 10:14am

so create a journal subdir in there, boot again and you should have binary journals you can read with journalctl on your desktop using the --file or --directory options (see journalctl --help)

jithcr · June 24, 2021, 11:18am

I could see the log as you suggested.I am seeing multiple errors like can not mark boot success,system-key missing on disk, apparmor not enabled etc.

Below are the main errors.

Jun 24 15:36:40 localhost systemd[1]: Starting OpenBSD Secure Shell server...
Jun 24 15:36:40 localhost snapd[745]: AppArmor status: apparmor not enabled
Jun 24 15:36:40 localhost sshd[778]: Server listening on 0.0.0.0 port 22.
Jun 24 15:36:40 localhost sshd[778]: Server listening on :: port 22.
Jun 24 15:36:40 localhost systemd[1]: Started OpenBSD Secure Shell server.
Jun 24 15:36:40 localhost snapd[745]: daemon.go:248: started snapd/2.51 (series 16; devmode) ubuntu-core/18 (arm64) linux/5.4.85-gab1e81e16530
Jun 24 15:36:40 localhost snapd[745]: daemon.go:341: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
Jun 24 15:36:41 localhost snapd[745]: helpers.go:105: error trying to compare the snap system key: system-key missing on disk
Jun 24 15:36:41 localhost systemd[1]: Started /tmp/tmp.kVRe1vVFMc/usr/lib/snapd/snapd.
Jun 24 15:36:41 localhost systemd-networkd[698]: eth0: DHCPv4 address 192.168.1.7/24 via 192.168.1.1
Jun 24 15:36:41 localhost systemd-timesyncd[708]: Network configuration changed, trying to establish connection.
Jun 24 15:36:41 localhost systemd-timesyncd[708]: Network configuration changed, trying to establish connection.
Jun 24 15:36:41 localhost systemd-networkd[698]: eth0: Configured
Jun 24 15:36:41 localhost systemd-timesyncd[708]: Network configuration changed, trying to establish connection.
Jun 24 15:36:42 localhost systemd[1]: Created slice system-serial\x2dconsole\x2dconf.slice.
Jun 24 15:36:42 localhost systemd[1]: Created slice system-console\x2dconf.slice.
Jun 24 15:36:42 localhost set-cpufreq[719]: Setting ondemand scheduler for all CPUs
Jun 24 15:36:48 localhost snapd[745]: stateengine.go:150: state ensure error: devicemgr: cannot mark boot successful: cannot open "/boot/uboot/uboot.env": bad CRC 2742774496 != 1902643021
Jun 24 15:36:58 localhost snapd[1343]: patch.go:63: Patching system state level 6 to sublevel 1...
Jun 24 15:36:59 localhost snapd[1343]: patch.go:63: Patching system state level 6 to sublevel 2...
Jun 24 15:36:59 localhost snapd[1343]: patch.go:63: Patching system state level 6 to sublevel 3...
Jun 24 15:36:59 localhost snapd[1343]: daemon.go:248: started snapd/2.51 (series 16; devmode) ubuntu-core/18 (arm64) linux/5.4.85-gab1e81e16530
Jun 24 15:46:42 localhost snapd[1422]: handlers.go:511: Reported install problem for "variscite-imx8mmini-kernel" as Crash report successfully
Jun 24 15:48:16 localhost systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.
Jun 24 15:48:16 localhost systemd[1]: snapd.service: Killing process 1422 (snapd) with signal SIGKILL.
Jun 24 15:48:17 localhost snap-failure[1649]: cmd_snapd.go:136: stopping snapd socket
Jun 24 15:48:17 localhost snap-failure[1649]: cmd_snapd.go:143: restoring invoking snapd from: /snap/core/current/usr/lib/snapd/snapd
Jun 24 15:48:17 localhost snap-failure[1649]: error: snapd failed: fork/exec /snap/core/current/usr/lib/snapd/snapd: no such file or directory
Jun 24 15:48:16 localhost systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL
Jun 24 15:48:16 localhost systemd[1]: snapd.service: Failed with result 'timeout'.
Jun 24 15:51:19 localhost kernel: cfg80211: Loading compiled-in X.509 certificates for regulatory database
Jun 24 15:51:19 localhost systemd-udevd[639]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Jun 24 15:51:19 localhost kernel: cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
Jun 24 15:51:19 localhost kernel: platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Jun 24 15:51:19 localhost kernel: cfg80211: failed to load regulatory.db

Could help me to narrow down the problem?

ogra · June 24, 2021, 11:28am

lets start with your kernel, you are obviously missing apparmor patches and probably some default config options.
here is an older rpi kernel i worked on using the pi foundation tree as upstream and adding the needed configs and apparmor patches for an initial pi4 kernel (5.1 though):

try to make sure your kernel uses the same configs, there is also this thread which is more specifically for 5.4 kernels:

jithcr · June 25, 2021, 7:21am

I am using 5.4.85 kernel which has ported for custom hardware. This is non ubuntu kernel. So i applied the apparmor patches and added configs (snappy/generic.config, snappy/security.config, snappy/systemd.config, snappy/snappy.config, snappy/containers.config)

Mainly referred below link

Is there any patch required for UC other than AppArmor kernel patches?

ogra · June 25, 2021, 8:23am

this should be enough and if it worked properly the

AppArmor status: apparmor not enabled

should be gone …

jithcr · June 25, 2021, 8:43am

Thank you , then I will crosscheck on the AppArmor patches in my kernel.

ogra · June 25, 2021, 8:45am

also make sure that apparmor is the default LSM in your security config settings of the kernel … i have seen it default to selinux before on some kernels even with the config patches …

jithcr · June 26, 2021, 6:17am

I could find CONFIG_LSM=“lockdown,yama,loadpin,safesetid,integrity” in my kernel config

ogra · June 26, 2021, 8:07am

i guess you want the below:

$ grep CONFIG_LSM= /snap/pi-kernel/current/config-5.4.0-1036-raspi 
CONFIG_LSM="yama,integrity,apparmor"
$

jithcr · June 26, 2021, 10:32am

After adding suggested CONFIG_LSM as you suggested, AppArmor Not enabled log has gone.

Jun 26 14:34:20 localhost systemd[1]: Starting Load AppArmor profiles managed internally by snapd...
Jun 26 14:34:20 localhost systemd[1]: Started Load AppArmor profiles managed internally by snapd.
Jun 26 14:34:22 localhost snapd[1385]: AppArmor status: apparmor is enabled and all features are available

More repeated error log is mentioned below.
Jun 26 14:43:04 localhost snapd[1465]: stateengine.go:150: state ensure error: devicemgr: cannot mark boot successful: cannot open "/boot/uboot/uboot.env": bad CRC 2742774496 != 1902643021

As you told in previous comments, there can be issues with config which I used for kernel. I used below link for extra configs which I applied on top of my board config.

Where, CONFIG_LSM not mentioned. I understand I used 4.4.y kernel snappy reference for 5.4.85 which led into this problems.

I am bit confused now.

Can you suggest me the approach for getting the required Ubuntu core configs for 5.4.x? How can I get know the required configs for UC.

ogra · June 26, 2021, 10:50am

your kernel config seems to be fine now, the next issue is your gadget which ships the uboot.env file in a wrong format or size …

jithcr · June 28, 2021, 6:26am

uboot.env file size is 131072. There is some issues with redund config/ redundancy in gadget/bootloader. I think that could be the problem. I understand UC expects CONFIG_SYS_REDUNDAND_ENVIRONMENT in gadget.

ogra · June 28, 2021, 8:23am

yep, it does expect CONFIG_SYS_REDUNDAND_ENVIRONMENT and that you use the -r option to mkimage when creating the .env file …

jithcr · June 29, 2021, 1:42pm

Yes…Problem was there with CONFIG_SYS_REDUNDAND_ENVIRONMENT.I am able to complete the network configurations.

jithcr · July 1, 2021, 1:24pm

Hi,

I am seeing Ubuntu Core 16 in debug console message for UC18 image as below.
Actually I generated UC18 image.

Ubuntu Core 16 on 192.168.1.4 (ttymxc3)

The host key fingerprints are:

    RSA     SHA256:fpULzXlkbMS61OzUH3R9fh1c727VUI6qU9hZppQM
    ECDSA   SHA256:+0W0wgXV0liOvudXxLitHuCDSyWGIqU3S45uypw
    ED25519 SHA256:KdlSqUkqhTDlQup0FNt06x/i3FRe+hF4xl4V58Odo

To login:

    ssh user@192.168.1.4

During ssh login, I am seeing Ubuntu core 18 as below.

    Welcome to Ubuntu Core 18 (GNU/Linux 5.4.85-gab1e81e16530 aarch64)
     *Ubuntu Core:     https://www.ubuntu.com/core
     *Community:       https://forum.snapcraft.io
     *Snaps:           https://snapcraft.io

    This Ubuntu Core 18 machine is a tiny, transactional edition of Ubuntu,
    designed for appliances, firmware and fixed-function VMs.


snap list
Name                        Version    Rev    Tracking       Publisher   Notes
core18                      20210611   2073   latest/stable  canonical✓  base
snapd                       2.51.1     12397  latest/stable  canonical✓  snapd
im8xmini                    16-0.1     x1     -              -           gadget
imx8mmini-kernel            5.4.85     x1     -              -           kernel

Can you help me to understand the reason for version change in logs.

ogra · July 1, 2021, 2:55pm

smells like a bug to me … (i.e. not your fault)

check with snap version when logged in what it really is … that should always tell the truth …