Uc20 firstboot configuration error

Hi,

I am trying to load UC20 on arm64 (iMx8M plus) based custom hardware.
Used custom kernel plugin for building kernel snap by referring to https://github.com/ogra1/rockpi-n10-kernel. Generated uc20 image and tried to boot. Board is booting and I am getting login prompt with below boot log.
It looks like due to some error (snapd or permission error?), boot flow directly going to login prompt without Firstboot configuration (Not asking email for SSH login).
What could be the issue? What am I missing here?

====Log===
[ 118.834888] snapd[1579]: /snap/gadget/x1/meta/hooks/install: 3: cannot create /etc/netplan/00-snapd-config.yaml: Permission denied
[ 118.835177] snapd[1579]: /snap/gadget/x1/meta/hooks/install: 17: cannot create /etc/systemd/network/70-hon-fec.link: Permission denied
[ 118.835452] snapd[1579]: /snap/gadget/x1/meta/hooks/install: 25: cannot create /etc/systemd/network/70-hon-dwmac.link: Permission denied
[ 118.835740] snapd[1579]: -----
[ 119.136137] systemd[1]: run-snapd-ns-gadget.mnt.mount: Succeeded.
[ 119.163006] systemd[1]: Reloading.
[ 119.604824] systemd[1]: emergency.target: Requested dependency OnFailure=reboot.target ignored (target units cannot fail).
[ 121.902914] systemd[1]: Reloading.
[ 122.338170] systemd[1]: emergency.target: Requested dependency OnFailure=reboot.target ignored (target units cannot fail).
[ 124.729025] systemd[1]: writable-system\x2ddata-snap-gadget-x1.mount: Succeeded.
[ 124.732483] systemd[1]: snap-gadget-x1.mount: Succeeded.
[ 125.138046] systemd[1]: Reloading.
[ 125.604804] systemd[1]: emergency.target: Requested dependency OnFailure=reboot.target ignored (target units cannot fail).
[ 127.846077] systemd[1]: getty@tty1.service: start-pre operation timed out. Terminating.
[ 127.848263] systemd[1]: getty@tty1.service: Control process exited, code=killed, status=15/TERM
[ 127.848769] systemd[1]: getty@tty1.se[ 127.874286] systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 1.
[ OK ] Stopped Getty on tty1.
[ 127.886873] systemd[1]: Stopped Getty on tty1.
[ OK ] Started Getty on tty1.
[ 127.895570] systemd[1]: Started Getty on tty1.
[ 127.898006] systemd[1]: Reloading.
[ 128.288156] systemd[1]: emergency.target: Requested dependency OnFailure=reboot.target ignored (target units cannot fail).
[ 130.437546] systemd[1]: serial-getty@ttymxc1.service: Scheduled restart job, restart counter is at 1.
[ OK ] Stopped Serial Getty on ttymxc1.
[ 130.454966] systemd[1]: Stopped Serial Getty on ttymxc1.
[ OK ] Started Serial Getty on ttymxc1.
[ 130.463887] systemd[1]: Started Serial Getty on ttymxc1.
[ 130.515062] systemd[1]: Cannot find unit for notify message of PID 2141, ignoring.
[ 132.870877] snapd[1579]: handlers.go:511: Reported install problem for “gadget” as 7698a94a-178e-11ec-bc0e-fa163e983629 OOPSID
[ 133.214770] systemd[1]: writable-system\x2ddata-snap-core20-1084.mount: Succeeded.
[ 133.218372] systemd[1]: snap-core20-1084.mount: Succeeded.
[ 133.602482] systemd[1]: Reloading.
[ 134.106367] systemd[1]: emergency.target: Requested dependency OnFailure=reboot.target ignored (target units cannot fail).
[ 136.403433] systemd[1]: Reloading.
[ 136.852363] systemd[1]: emergency.target: Requested dependency OnFailure=reboot.target ignored (target units cannot fail).
[ 139.151426] systemd[1]: Cannot find unit for notify message of PID 2196, ignoring.
[ 140.543389] snapd[1579]: handlers.go:511: Reported install problem for “core20” as 7b682f4a-178e-11ec-bc0e-fa163e983629 OOPSID
[ 140.856447] systemd[1]: writable-system\x2ddata-snap-kernel-x1.mount: Succeeded.
[ 140.860065] systemd[1]: snap-kernel-x1.mount: Succeeded.
[ 141.234039] systemd[1]: Reloading.
[ 141.668117] systemd[1]: emergency.target: Requested dependency OnFailure=reboot.target ignored (target units cannot fail).
[ 143.912223] systemd[1]: Reloading.
[ 144.337937] systemd[1]: emergency.target: Requested dependency OnFailure=reboot.target ignored (target units cannot fail).
[ 146.602400] systemd[1]: Cannot find unit for notify message of PID 2246, ignoring.
[ 148.538536] snapd[1579]: handlers.go:511: Reported install problem for kernel" as 7fe42254-178e-11ec-bc0e-fa163e983629 OOPSID
[ 148.748963] dbus-daemon[771]: [system] Reloaded configuration
[ 148.806884] systemd[1]: snapd.seeded.service: Main process exited, code=killed, status=15/TERM
[ 148.807385] systemd[1]: snapd.seeded.service: Failed with result ‘signal’.
[ OK ] Stopped Wait until snapd is fully seeded.
[ 148.811608] systemd[1]: Stopped Wait until snapd is fully seeded.
[ 148.816581] snapd[1579]: main.go:155: Exiting on terminated signal.

  •     Stopping Snap Daemon...*
    

[ 148.817555] systemd[1]: Stopping Snap Daemon…
[ 218.004801] systemd[1]: getty@tty1.service: start-pre operation timed out. Terminating.
[ 218.006769] systemd[1]: getty@tty1.service: Control process exited, code=killed, status=15/TERM
[ 218.007261] systemd[1]: getty@tty1.service: Failed with result ‘timeout’.
[ 218.016797] systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 2.
[ OK ] Stopped Getty on tty1.
[ 218.022662] systemd[1]: Stopped Getty on tty1.
[ 218.032251] systemd[1]: Started Getty on tty1.
[ OK ] Started Getty on tty1.
[ 220.504852] systemd[1]: serial-getty@ttymxc1.service: start-pre operation timed out. Terminating.
[ 220[** ] (1 of 2) A stop job is running for Snap Daemon (1min 11s / 1min 30s)*
[ 220.758613] systemd[1]: serial-getty@ttymxc1.service: Scheduled restart job, restart counter is at 2.
[ OK ] Stopped Serial Getty on ttymxc1.
[ 220.760097] systemd[1]: Stopped Serial Getty on ttymxc1.
[ OK ] Started Serial Getty on ttymxc1.
[ 220.769236] systemd[1]: Started Serial Getty on ttymxc1.
[ 239.004835] systemd[1]: snapd.service: State ‘stop-sigterm’ timed out. Killing.
[ 239.005400] systemd[1]: snapd.service: Killing process 1579 (snapd) with signal SIGKILL.
[ 239.017714] systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL
[ 239.018268] systemd[1]: snapd.service: Failed with result ‘timeout’.
[ 239.018909] systemd[1]: Stopped Snap Daemon.
[ 239.021506] systemd[1]: snapd.service: Triggering OnFailure= dependencies.
[ 239.028569] systemd[1]: snapd.socket: Succeeded.
[ 239.033857] systemd[1]: Closed Socket activation for snappy daemon.
[ 239.044653] systemd[1]: Starting Auto import assertions from block devices…
[ 239.056603] systemd[1]: Starting Failure handling of the snapd snap…
[ 239.115101] snap[2269]: auto-import is disabled in install-mode
[ 239.123035] systemd[1]: snapd.autoimport.service: Succeeded.
[ 239.127110] systemd[1]: Finished Auto import assertions from block devices.
[ 239.389528] snap-failure[2270]: cmd_snapd.go:136: stopping snapd socket
[ 239.412402] snap-failure[2270]: cmd_snapd.go:143: restoring invoking snapd from: /snap/core/current/usr/lib/snapd/snapd
[ 239.414804] snap-failure[2270]: error: snapd failed: fork/exec /snap/core/current/usr/lib/snapd/snapd: no such file or directory
[ 239.417634] systemd[1]: snapd.failure.service: Main process exited, code=exited, status=1/FAILURE
[ 239.419271] systemd[1]: snapd.failure.service: Failed with result ‘exit-code’.
[ 239.423673] systemd[1]: Failed to start Failure handling of the snapd snap.
Waiting for server to restart |[ 308.254765] systemd[1]: getty@tty1.service: start-pre operation timed out. Terminating.
[ 308.256222] systemd[1]: getty@tty1.service: Control process exited, code=killed, status=15/TERM
[ 308.256724] systemd[1]: getty@tty1.service: Failed with result ‘timeout’.
[ 308.266018] systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 3.
[ 308.271367] systemd[1]: Stopped Getty on tty1.
[ 308.280720] systemd[1]: Started Getty on tty1.
[ 311.004845] systemd[1]: serial-getty@ttymxc1.service: start-pre operation [ *** ] A start job is running for Start th…e snapd snap (4min 43s / no limit)
[ 311.256129] systemd[1]: serial-getty@ttymxc1.service: Scheduled restart job, restart counter is at 3.
[ OK ] Stopped Serial Getty on ttymxc1.
[ 311.256757] systemd[1]: Stopped Serial Getty on ttymxc1.
[ OK ] Started Serial Getty on ttymxc1.
[ 311.265330] systemd[1]: Started Serial Getty on ttymxc1.
[ 388.922277] run-snapd-from-snap[1609]: Automatically connect eligible plugs and slots of snap “snapd” -Automatically connect eligible plugs and slots of snap “snapd” \Automatically connect eligible plugs and slots of snap “snapd” |Automatically connect eligible plugs and slots of snap “snapd” /Automatically connect eligible plugs and slots of snap “snapd” -Automatically connect eligible plugs and slots of snap “snapd” \Automatically connect eligible plugs and slots of snap “snapd” |Automatically connect eligible plugs and slots of snap “snapd” /Automatically connect eligible plugs and slots of snap “snapd” -Automatically connect eligible plugs and slots of snap “snapd” \Setup snap “snapd” aliases |Mount snap “kernel” (unset) /Mount snap “kernel” (unset) -Mount snap “kernel” (unset) mx-kernel" (unset) |Mount snap “kernel” (unset) /Mount snap “kernel” (unset) -Mount snap “kernel” (unset) \Mount snap “kernel” (unset) |Mount snap “kernel” (unset) /Mount snap “kernel” (unset) -Mount snap “kernel” (unset) \Mount snap “kernel” (unset) |Mount snap “kernel” (unset) /Mount snap “kernel” (unset) -Mount snap “kernel” (unset) \Mount snap “kernel” (unset) |Mount snap “kernel” (unset) /Mount snap “kernel” (unset) -Mount snap “kernel” (unset) \Make snap “kernel” (unset) available to the system |Run install hook of “kernel” snap if present /Mount snap “core20” (1084) -Mount snap “core20” (1084) \Mount snap “core20” (1084) |Mount snap “core20” (1084) /Mount snap “core20” (1084) -Mount snap “core20” (1084) \Mount snap “core20” (1084) |Mount snap “core20” (1084) /Mount snap “core20” (1084) -Mount snap “core20” (1084) \Mount snap “core20” (1084) |Mount snap “core20” (1084) /Mount snap “core20” (1084) -Mount snap “core20” (1084) \Mount snap “core20” (1084) |Mount snap “core20” (1084) /Mount snap “core20” (1084) -Mount snap “core20” (1084) \Mount snap “core20” (1084) |Mount snap “core20” (1084) /Mount snap “core20” (1084) -Mount snap “core20” (1084) \Mount snap “core20” (1084) |Mount snap “core20” (1084) /Mount snap “core20” (1084) -Mount snap “core20” (1084) \Mount snap “core20” (1084) |Mount snap “core20” (1084) /Mount snap “core20” (1084) -Mount snap “core20” (1084) \Mount snap “core20” (1084) |Mount snap “core20” (1084) /Mount snap “core20” (1084) [ 389.401532] [ 389.403514] systemd[1]: Finished Start the snapd services from the snapd snap.
[ 389.433525] [ 389.434603] systemd[1]: Reached target Multi-User System.
[ 389.436195] systemd[1]: Reached target Graphical Interface.
[ 389.437242] systemd[1]: Condition check resulted in Ubuntu Core Firstboot Configuration tty1 being skipped.
[ 389.437875] systemd[1]: Condition check resulted in Ubuntu Core Firstboot Configuration ttymxc1 being skipped.
[ 389.438266] systemd[1]: Starting Update UTMP about System Runlevel Changes…
[ 389.438658] systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
[ 389.439027] systemd[1]: Finished Update UTMP about System Runlevel Changes.
[ 389.439397] systemd[1]: Startup finished in 25.755s (kernel) + 6min 3.264s (userspace) = 6min 29.020s.

Ubuntu Core 20 on 192.168.1.102 (ttymxc1)

ubuntu login:
====== Log =====

that is a pretty clear message i’d say, a hook of your gadget fails due to missing plug connections which makes the snap refuse to install … i think all follow-up errors are just fallout of that (or perhaps not, but you will only be able to debug the stuff further down once the gadget installs properly) …