Network Configuration :: First Ubuntu Core boot in custom hardware

jithcr · June 22, 2021, 1:20pm

Hi,

I am getting connection refused message on Profile set up. (Doing via serial console)

Selected dhcp option network connection and proceeding to Profile set up as below

Network connections

    Configure at least one interface this server can use to talk to
    other machines, and which preferably provides sufficient access
    for updates.

      eth0     >  Will use DHCP for IPv4, currently has address:
                  192.168.1.5
                  Has no IPv6 configuration, currently has address:
                  fe80::219:b8ff:fe00:f12b
                  00:19:b8:00:f1:2b (100M)

Profile setup

Enter an email address from your account in the store.

         Email address:    myname@mydomain.com

If you do not have an account, visit https://login.ubuntu.com
to create one.

                      Creating user failed:
error: while creating user: cannot communicate with server: Post
http://localhost/v2/users: dial unix /run/snapd.socket: connect:
                       connection refused

I can only check my network, at gadget snap/bootloader level and that confirmed. Able to ping from device to 8.8.8.8.

u-boot=> ping 8.8.8.8
Using ethernet@30be0000 device
host 8.8.8.8 is alive

Can you please help me to understand the problem.

ogra · June 22, 2021, 1:28pm

this means there is no snapd running on your device (note it is talking to localhost) …

jithcr · June 22, 2021, 1:33pm

Can you please direct me what I am missing or need to be taken care?

ogra · June 22, 2021, 1:40pm

not without any logs … if your system runs from SD card or USB key you could try to create a system-data/var/log/journal directory on the ubuntu-data partition to get a persistent journald log

jithcr · June 22, 2021, 2:04pm

Thank you !!
Sure, I was not aware this method. I am using SD card as boot media.

jithcr · June 23, 2021, 9:37am

I have added ubuntu-data partition into SDcard and created var/log/journal directory.

Then repeated the boot procedure, I could not find log files under journal.
Can you share me link to avail persistent journald log feature.

Apart from that, I could see writable/system-data/var/log folder in SD card which contains console-conf/subiquity-debug.log(16MB) and lastlog(0MB).

ogra · June 23, 2021, 9:50am

ah, during first boot that partition should have been created already, you should only have to create the “/journal” directory in the existing var/log … that points to a problem with the initramfs your kernel snap uses (which is responsible for creating all the additional partitions beyond ubuntu-seed during the first boot …

is this UC20 ? if so, you should use the v2 kernel plugin to build your kernel snap:

jithcr · June 23, 2021, 10:06am

I am trying on UC18…

ogra · June 23, 2021, 10:08am

oops, sorry then (i was blindly assuming UC20), the data partition is called writable on UC18, check if there is a system-data/var/log/ in it …

jithcr · June 23, 2021, 10:11am

Yes, writable/system-data/var/log folder is there

ogra · June 23, 2021, 10:14am

so create a journal subdir in there, boot again and you should have binary journals you can read with journalctl on your desktop using the --file or --directory options (see journalctl --help)

jithcr · June 24, 2021, 11:18am

I could see the log as you suggested.I am seeing multiple errors like can not mark boot success,system-key missing on disk, apparmor not enabled etc.

Below are the main errors.

Jun 24 15:36:40 localhost systemd[1]: Starting OpenBSD Secure Shell server...
Jun 24 15:36:40 localhost snapd[745]: AppArmor status: apparmor not enabled
Jun 24 15:36:40 localhost sshd[778]: Server listening on 0.0.0.0 port 22.
Jun 24 15:36:40 localhost sshd[778]: Server listening on :: port 22.
Jun 24 15:36:40 localhost systemd[1]: Started OpenBSD Secure Shell server.
Jun 24 15:36:40 localhost snapd[745]: daemon.go:248: started snapd/2.51 (series 16; devmode) ubuntu-core/18 (arm64) linux/5.4.85-gab1e81e16530
Jun 24 15:36:40 localhost snapd[745]: daemon.go:341: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
Jun 24 15:36:41 localhost snapd[745]: helpers.go:105: error trying to compare the snap system key: system-key missing on disk
Jun 24 15:36:41 localhost systemd[1]: Started /tmp/tmp.kVRe1vVFMc/usr/lib/snapd/snapd.
Jun 24 15:36:41 localhost systemd-networkd[698]: eth0: DHCPv4 address 192.168.1.7/24 via 192.168.1.1
Jun 24 15:36:41 localhost systemd-timesyncd[708]: Network configuration changed, trying to establish connection.
Jun 24 15:36:41 localhost systemd-timesyncd[708]: Network configuration changed, trying to establish connection.
Jun 24 15:36:41 localhost systemd-networkd[698]: eth0: Configured
Jun 24 15:36:41 localhost systemd-timesyncd[708]: Network configuration changed, trying to establish connection.
Jun 24 15:36:42 localhost systemd[1]: Created slice system-serial\x2dconsole\x2dconf.slice.
Jun 24 15:36:42 localhost systemd[1]: Created slice system-console\x2dconf.slice.
Jun 24 15:36:42 localhost set-cpufreq[719]: Setting ondemand scheduler for all CPUs
Jun 24 15:36:48 localhost snapd[745]: stateengine.go:150: state ensure error: devicemgr: cannot mark boot successful: cannot open "/boot/uboot/uboot.env": bad CRC 2742774496 != 1902643021
Jun 24 15:36:58 localhost snapd[1343]: patch.go:63: Patching system state level 6 to sublevel 1...
Jun 24 15:36:59 localhost snapd[1343]: patch.go:63: Patching system state level 6 to sublevel 2...
Jun 24 15:36:59 localhost snapd[1343]: patch.go:63: Patching system state level 6 to sublevel 3...
Jun 24 15:36:59 localhost snapd[1343]: daemon.go:248: started snapd/2.51 (series 16; devmode) ubuntu-core/18 (arm64) linux/5.4.85-gab1e81e16530
Jun 24 15:46:42 localhost snapd[1422]: handlers.go:511: Reported install problem for "variscite-imx8mmini-kernel" as Crash report successfully
Jun 24 15:48:16 localhost systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.
Jun 24 15:48:16 localhost systemd[1]: snapd.service: Killing process 1422 (snapd) with signal SIGKILL.
Jun 24 15:48:17 localhost snap-failure[1649]: cmd_snapd.go:136: stopping snapd socket
Jun 24 15:48:17 localhost snap-failure[1649]: cmd_snapd.go:143: restoring invoking snapd from: /snap/core/current/usr/lib/snapd/snapd
Jun 24 15:48:17 localhost snap-failure[1649]: error: snapd failed: fork/exec /snap/core/current/usr/lib/snapd/snapd: no such file or directory
Jun 24 15:48:16 localhost systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL
Jun 24 15:48:16 localhost systemd[1]: snapd.service: Failed with result 'timeout'.
Jun 24 15:51:19 localhost kernel: cfg80211: Loading compiled-in X.509 certificates for regulatory database
Jun 24 15:51:19 localhost systemd-udevd[639]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Jun 24 15:51:19 localhost kernel: cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
Jun 24 15:51:19 localhost kernel: platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Jun 24 15:51:19 localhost kernel: cfg80211: failed to load regulatory.db

Could help me to narrow down the problem?

ogra · June 24, 2021, 11:28am

lets start with your kernel, you are obviously missing apparmor patches and probably some default config options.
here is an older rpi kernel i worked on using the pi foundation tree as upstream and adding the needed configs and apparmor patches for an initial pi4 kernel (5.1 though):

try to make sure your kernel uses the same configs, there is also this thread which is more specifically for 5.4 kernels:

jithcr · June 25, 2021, 7:21am

I am using 5.4.85 kernel which has ported for custom hardware. This is non ubuntu kernel. So i applied the apparmor patches and added configs (snappy/generic.config, snappy/security.config, snappy/systemd.config, snappy/snappy.config, snappy/containers.config)

Mainly referred below link

Is there any patch required for UC other than AppArmor kernel patches?

ogra · June 25, 2021, 8:23am

this should be enough and if it worked properly the

AppArmor status: apparmor not enabled

should be gone …

jithcr · June 25, 2021, 8:43am

Thank you , then I will crosscheck on the AppArmor patches in my kernel.

ogra · June 25, 2021, 8:45am

also make sure that apparmor is the default LSM in your security config settings of the kernel … i have seen it default to selinux before on some kernels even with the config patches …

jithcr · June 26, 2021, 6:17am

I could find CONFIG_LSM=“lockdown,yama,loadpin,safesetid,integrity” in my kernel config

ogra · June 26, 2021, 8:07am

i guess you want the below:

$ grep CONFIG_LSM= /snap/pi-kernel/current/config-5.4.0-1036-raspi 
CONFIG_LSM="yama,integrity,apparmor"
$

jithcr · June 26, 2021, 10:32am

After adding suggested CONFIG_LSM as you suggested, AppArmor Not enabled log has gone.

Jun 26 14:34:20 localhost systemd[1]: Starting Load AppArmor profiles managed internally by snapd...
Jun 26 14:34:20 localhost systemd[1]: Started Load AppArmor profiles managed internally by snapd.
Jun 26 14:34:22 localhost snapd[1385]: AppArmor status: apparmor is enabled and all features are available

More repeated error log is mentioned below.
Jun 26 14:43:04 localhost snapd[1465]: stateengine.go:150: state ensure error: devicemgr: cannot mark boot successful: cannot open "/boot/uboot/uboot.env": bad CRC 2742774496 != 1902643021

As you told in previous comments, there can be issues with config which I used for kernel. I used below link for extra configs which I applied on top of my board config.

Where, CONFIG_LSM not mentioned. I understand I used 4.4.y kernel snappy reference for 5.4.85 which led into this problems.

I am bit confused now.

Can you suggest me the approach for getting the required Ubuntu core configs for 5.4.x? How can I get know the required configs for UC.