No snaps installed on a new ubuntu core port (kernel, gadget)

DC-fgervais · August 7, 2018, 6:12pm

I did a kernel and gadget snaps for a new hardware platform and I can now boot all the way and ssh on the target.

I feel like this is a good achievement

I’m new to ubuntu core but the way I understood it is that I should see the kernel, gadget and core snaps installed and be able to update them.

However this is what I get:

dc-fgervais@localhost:~$ snap list
No snaps are installed yet. Try 'snap install hello-world'.
dc-fgervais@localhost:~$

Is my understanding correct that I should see my base snaps listed there?

ogra · August 8, 2018, 11:38am

Yes, if the image, gadget and kernel are built correctly you should see the list of snaps installed. what is the output of:

snap changes

also pushing your journald output to a pastebin and linking it here would be helpful to see any possible boot errors .

DC-fgervais · August 9, 2018, 3:37pm

Good thanks for the confirmation.

Here’s what I have on my end:

dc-fgervais@localhost:~$ snap changes
error: no changes found
dc-fgervais@localhost:~$

ogra · August 9, 2018, 3:46pm

Aug 09 14:39:57 localhost.localdomain snapd[1843]: 2018/08/09 14:39:57.175787 stateengine.go:101: state ensure error: devicemgr: need a model assertion

how was that image created exactly, how did you create and sign the model assetion for it ?

DC-fgervais · August 9, 2018, 3:56pm

The model was created like this:

cat board-model.json | snap sign -k my-key &> board.model

and the image like this:

UBUNTU_IMAGE_SKIP_COPY_UNVERIFIED_MODEL=1 ubuntu-image snap -d \
	-c stable \
	--image-size 4G \
	--extra-snaps ../gadget/gadget_0_amd64.snap \
	--extra-snaps ../kernel/kernel_4.8_armhf.snap \
	-O . \
	board.model

ogra · August 9, 2018, 4:36pm

have you tried without setting the env var there ?

DC-fgervais · August 9, 2018, 5:33pm

Yes but it fails on account key not found. I’m guessing it’s because I skipped the register-key step.

I trying to stay as decoupled as possible from the store because in the final use-case, the device will be offline.

DEBUG:ubuntu-image:-> [ 0] make_temporary_directories
DEBUG:ubuntu-image:-> [ 1] prepare_gadget_tree
DEBUG:ubuntu-image:-> [ 2] prepare_image
error: cannot fetch and check prerequisites for the model assertion: account-key (<I removed it, I'm not sure if this is public or private info>) not found
ERROR:ubuntu-image:COMMAND FAILED: snap prepare-image --channel=stable --extra-snaps=../gadget/gadget_0_amd64.snap --extra-snaps=../kernel/kernel_4.8_armhf.snap board.model /tmp/tmpa_6cnbiz/unpack
ERROR:ubuntu-image:Full debug traceback follows
Traceback (most recent call last):
  File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/assertion_builder.py", line 24, in prepare_image
    self.args.channel, self.args.extra_snaps)
  File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/helpers.py", line 131, in snap
    run(cmd, stdout=None, stderr=None, env=os.environ)
  File "/snap/ubuntu-image/95/lib/python3/site-packages/ubuntu_image/helpers.py", line 112, in run
    proc.check_returncode()
  File "/snap/ubuntu-image/95/usr/lib/python3.5/subprocess.py", line 349, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['snap', 'prepare-image', '--channel=stable', '--extra-snaps=../gadget/gadget_0_amd64.snap', '--extra-snaps=../kernel/kernel_4.8_armhf.snap', 'board.model', '/tmp/tmpa_6cnbiz/unpack']' returned non-zero exit status 1

ogra · August 10, 2018, 8:20am

well, i doubt this will work then, snapd checks all signatures on first boot against the public key of the master signature it has builtin , so you need a properly signed key for the model assertion.

the same would be true for the gadget and kernel snaps if you would not use --extra-snaps but have them come from the store (they would have to be signed by the same uploader key that is used in the model signature). this is a security feature to make sure only the actual owner of a device can provide images or these security sensitive packages (that essentially make you fully own a device).

DC-fgervais · August 10, 2018, 2:21pm

Oh I see

But doesn’t it have a public key embedded somewhere in the device to verify the signature of the packages it installs?

Me signing the packages with my private key already ensures packages are coming from me wouldn’t you say?

ogra · August 10, 2018, 3:55pm

how would the store know about your private key or how would snapd know about it ? the authority is the store master key that is getting used to sign everything, snaps, assertions (there are more than just model, i.e. to authenticate local users you use a system-user assertion), sub-stores.

the only signature snapd itself knows about is the public store key so the most minimal signature bit you have to provide is a model assertion signed by a valid store user that was signed with the same key.

DC-fgervais · August 10, 2018, 5:35pm

Ok I think I get what you mean.

I’ll do more test and see if I can get something that fits my needs.

Thank you

DC-fgervais · August 13, 2018, 8:23pm

I decided to go on and register the key to at least get to a working state but it looks like I still need a bit of help to get things running.

So after registering my key, I was able to build without using:
UBUNTU_IMAGE_SKIP_COPY_UNVERIFIED_MODEL=1

Still, when I boot it looks like I get stuck in some kind of initialization loop as snap changes keeps showing failures over and over:

dc-fgervais@localhost:~$ snap changes
ID   Status  Spawn               Ready               Summary
1    Error   today at 20:09 UTC  today at 20:10 UTC  Initialize system state
2    Error   today at 20:11 UTC  today at 20:11 UTC  Initialize system state
3    Error   today at 20:12 UTC  today at 20:13 UTC  Initialize system state
4    Error   today at 20:13 UTC  today at 20:14 UTC  Initialize system state
5    Error   today at 20:14 UTC  today at 20:15 UTC  Initialize system state
6    Error   today at 20:15 UTC  today at 20:16 UTC  Initialize system state
7    Error   today at 20:17 UTC  today at 20:17 UTC  Initialize system state
8    Error   today at 20:18 UTC  today at 20:19 UTC  Initialize system state
9    Error   today at 20:19 UTC  today at 20:20 UTC  Initialize system state
10   Error   today at 20:20 UTC  today at 20:21 UTC  Initialize system state
11   Error   today at 20:22 UTC  today at 20:23 UTC  Initialize system state
12   Error   today at 20:23 UTC  today at 20:24 UTC  Initialize system state
13   Error   today at 20:24 UTC  today at 20:25 UTC  Initialize system state
14   Do      today at 20:26 UTC  -                   Initialize system state

These repeated failures also show in the snapd log:

dc-fgervais@localhost:~$ sudo journalctl -u snapd
-- Logs begin at Mon 2018-08-13 20:08:44 UTC, end at Mon 2018-08-13 20:22:32 UTC. --
Aug 13 20:09:12 localhost.localdomain systemd[1]: Starting Snappy daemon...
Aug 13 20:09:17 localhost.localdomain snapd[2136]: AppArmor status: apparmor is enabled but some features are missing: dbus, mount, namespaces, network, ptrace, signal
Aug 13 20:09:18 localhost.localdomain snapd[2136]: 2018/08/13 20:09:18.281397 helpers.go:119: error trying to compare the snap system key: system-key missing on disk
Aug 13 20:09:18 localhost.localdomain snapd[2136]: 2018/08/13 20:09:18.485912 daemon.go:343: started snapd/2.34.3 (series 16; devmode) ubuntu-core/16 (armhf) linux/4.8.7.
Aug 13 20:09:18 localhost.localdomain systemd[1]: Started Snappy daemon.
Aug 13 20:10:17 localhost.localdomain snapd[2136]: 2018/08/13 20:10:17.632664 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:10:31 localhost.localdomain snapd[2136]: 2018/08/13 20:10:31.296156 handlers.go:388: Reported install problem for "gadget" as dba6d668-9f32-11e8-b613-fa163e0ec2f1 OOPSID
Aug 13 20:10:39 localhost.localdomain snapd[2136]: 2018/08/13 20:10:39.019852 handlers.go:388: Reported install problem for "kernel" as e0e383ce-9f32-11e8-9f9e-fa163e54c21f OOPSID
Aug 13 20:10:45 localhost.localdomain snapd[2136]: 2018/08/13 20:10:45.959244 handlers.go:388: Reported install problem for "core" as e50108c8-9f32-11e8-b7ea-fa163e8d4bab OOPSID
Aug 13 20:11:34 localhost.localdomain snapd[2136]: 2018/08/13 20:11:34.765804 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:11:44 localhost.localdomain snapd[2136]: 2018/08/13 20:11:44.306608 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:11:50 localhost.localdomain snapd[2136]: 2018/08/13 20:11:50.727799 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:11:54 localhost.localdomain groupadd[2962]: group added to /var/lib/extrausers/group: name=dc-fgervais, GID=1000
Aug 13 20:11:54 localhost.localdomain groupadd[2962]: group added to /var/lib/extrausers/gshadow: name=dc-fgervais
Aug 13 20:11:54 localhost.localdomain groupadd[2962]: new group: name=dc-fgervais, GID=1000
Aug 13 20:11:54 localhost.localdomain useradd[2966]: new user: name=dc-fgervais, UID=1000, GID=1000, home=/home/dc-fgervais, shell=/bin/bash
Aug 13 20:11:55 localhost.localdomain usermod[2975]: change user 'dc-fgervais' password
Aug 13 20:11:56 localhost.localdomain chfn[2980]: changed user 'dc-fgervais' information
Aug 13 20:11:59 localhost.localdomain snapd[2136]: 2018/08/13 20:11:59.576382 handlers.go:388: Reported install problem for "core" as already-reported
Aug 13 20:12:56 localhost.localdomain snapd[2136]: 2018/08/13 20:12:56.568557 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:13:05 localhost.localdomain snapd[2136]: 2018/08/13 20:13:05.288977 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:13:11 localhost.localdomain snapd[2136]: 2018/08/13 20:13:11.579459 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:13:17 localhost.localdomain snapd[2136]: 2018/08/13 20:13:17.813133 handlers.go:388: Reported install problem for "core" as already-reported
Aug 13 20:13:59 localhost.localdomain snapd[2136]: 2018/08/13 20:13:59.935583 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:14:09 localhost.localdomain snapd[2136]: 2018/08/13 20:14:09.417519 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:14:16 localhost.localdomain snapd[2136]: 2018/08/13 20:14:16.119468 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:14:23 localhost.localdomain snapd[2136]: 2018/08/13 20:14:23.141433 handlers.go:388: Reported install problem for "core" as already-reported
Aug 13 20:15:06 localhost.localdomain snapd[2136]: 2018/08/13 20:15:06.196386 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:15:15 localhost.localdomain snapd[2136]: 2018/08/13 20:15:15.988297 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:15:23 localhost.localdomain snapd[2136]: 2018/08/13 20:15:23.107987 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:15:30 localhost.localdomain snapd[2136]: 2018/08/13 20:15:30.012712 handlers.go:388: Reported install problem for "core" as already-reported
Aug 13 20:16:13 localhost.localdomain snapd[2136]: 2018/08/13 20:16:13.917694 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:16:24 localhost.localdomain snapd[2136]: 2018/08/13 20:16:24.488482 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:16:32 localhost.localdomain snapd[2136]: 2018/08/13 20:16:32.476676 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:16:39 localhost.localdomain snapd[2136]: 2018/08/13 20:16:39.917221 handlers.go:388: Reported install problem for "core" as already-reported
Aug 13 20:17:30 localhost.localdomain snapd[2136]: 2018/08/13 20:17:30.278286 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:17:41 localhost.localdomain snapd[2136]: 2018/08/13 20:17:41.907256 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:17:50 localhost.localdomain snapd[2136]: 2018/08/13 20:17:50.655096 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:17:58 localhost.localdomain snapd[2136]: 2018/08/13 20:17:58.617064 handlers.go:388: Reported install problem for "core" as already-reported
Aug 13 20:18:45 localhost.localdomain snapd[2136]: 2018/08/13 20:18:45.577465 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:18:59 localhost.localdomain snapd[2136]: 2018/08/13 20:18:59.188330 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:19:08 localhost.localdomain snapd[2136]: 2018/08/13 20:19:08.276128 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:19:16 localhost.localdomain snapd[2136]: 2018/08/13 20:19:16.331569 handlers.go:388: Reported install problem for "core" as already-reported
Aug 13 20:20:02 localhost.localdomain snapd[2136]: 2018/08/13 20:20:02.195724 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:20:13 localhost.localdomain snapd[2136]: 2018/08/13 20:20:13.772045 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:20:22 localhost.localdomain snapd[2136]: 2018/08/13 20:20:22.311631 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:20:30 localhost.localdomain snapd[2136]: 2018/08/13 20:20:30.712040 handlers.go:388: Reported install problem for "core" as already-reported
Aug 13 20:21:17 localhost.localdomain snapd[2136]: 2018/08/13 20:21:17.466928 devicestate.go:170: installing unasserted kernel "kernel"
Aug 13 20:21:29 localhost.localdomain snapd[2136]: 2018/08/13 20:21:29.757187 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 13 20:21:38 localhost.localdomain snapd[2136]: 2018/08/13 20:21:38.577900 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 13 20:21:47 localhost.localdomain snapd[2136]: 2018/08/13 20:21:47.585770 handlers.go:388: Reported install problem for "core" as already-reported

Another thing that might be of interest, I cannot get the time synced due to network restrictions:

Aug 13 20:09:25 localhost.localdomain systemd-timesyncd[1998]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com).
Aug 13 20:09:35 localhost.localdomain systemd-timesyncd[1998]: Timed out waiting for reply from 91.189.89.198:123 (ntp.ubuntu.com).
Aug 13 20:09:46 localhost.localdomain systemd-timesyncd[1998]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).

Any idea what the problem could be at this point?

ogra · August 13, 2018, 8:50pm

this is normal if you do not use a gadget from the store … it just means the image can not be fully validated (it is expected that gadget and model are signed with the same key and that kernel, and gadget come from the store, but since you do not upload your gadget it wont be signed, this is fine for out-of-bound built images and does no harm (snapd will eventually stop trying to initialize the state))

This is pretty serious, your kernel seems to be missing a lot of security features, this will have quite some impact (and obviously causes the automated error reporting to send oopses to errors.ubuntu.com). See:

@ppisati usually keeps the info and patches there up to date …

(i guess the timesync stuff is just fallout/side-effect of some of the above)

DC-fgervais · August 22, 2018, 8:39pm

I think I have apparmor running correctly now.

AppArmor status: apparmor is enabled and all features are available

However is seem I’ll still need a bit of help as I get the same abnormal behaviors as before.

I feel like this might be the problem:

helpers.go:119: error trying to compare the snap system key: system-key missing on disk

The file does in fact exist but maybe there is something wrong with it?

ls -l /var/lib/snapd/system-key 
-rw-r--r-- 1 root root 316 Aug 22 20:20 /var/lib/snapd/system-key

Any idea?

See the full snapd journal here

sudo journalctl -u snapd
-- Logs begin at Wed 2018-08-22 20:19:55 UTC, end at Wed 2018-08-22 20:23:42 UTC. --
Aug 22 20:20:29 localhost.localdomain systemd[1]: Starting Snappy daemon...
Aug 22 20:20:34 localhost.localdomain snapd[1146]: AppArmor status: apparmor is enabled and all features are available
Aug 22 20:20:35 localhost.localdomain snapd[1146]: 2018/08/22 20:20:35.485995 helpers.go:119: error trying to compare the snap system key: system-key missing on disk
Aug 22 20:20:35 localhost.localdomain snapd[1146]: 2018/08/22 20:20:35.618862 daemon.go:343: started snapd/2.34.3 (series 16) ubuntu-core/16 (armhf) linux/4.14.65.
Aug 22 20:20:35 localhost.localdomain systemd[1]: Started Snappy daemon.
Aug 22 20:21:36 localhost.localdomain snapd[1146]: 2018/08/22 20:21:36.190799 devicestate.go:170: installing unasserted kernel "kernel"
Aug 22 20:21:51 localhost.localdomain snapd[1146]: 2018/08/22 20:21:51.577850 handlers.go:388: Reported install problem for "gadget" as e9463e42-a646-11e8-b86d-fa163ef911dc OOPSID
Aug 22 20:22:00 localhost.localdomain snapd[1146]: 2018/08/22 20:22:00.977921 handlers.go:388: Reported install problem for "kernel" as ef49cfd4-a646-11e8-b9be-fa163e30221b OOPSID
Aug 22 20:22:08 localhost.localdomain snapd[1146]: 2018/08/22 20:22:08.459001 handlers.go:388: Reported install problem for "core" as f3d130d8-a646-11e8-8965-fa163e0ec2f1 OOPSID
Aug 22 20:23:07 localhost.localdomain snapd[1146]: 2018/08/22 20:23:07.526369 devicestate.go:170: installing unasserted kernel "kernel"
Aug 22 20:23:13 localhost.localdomain groupadd[1940]: group added to /var/lib/extrausers/group: name=dc-fgervais, GID=1000
Aug 22 20:23:14 localhost.localdomain groupadd[1940]: group added to /var/lib/extrausers/gshadow: name=dc-fgervais
Aug 22 20:23:14 localhost.localdomain groupadd[1940]: new group: name=dc-fgervais, GID=1000
Aug 22 20:23:14 localhost.localdomain useradd[1947]: new user: name=dc-fgervais, UID=1000, GID=1000, home=/home/dc-fgervais, shell=/bin/bash
Aug 22 20:23:15 localhost.localdomain usermod[1960]: change user 'dc-fgervais' password
Aug 22 20:23:16 localhost.localdomain chfn[1965]: changed user 'dc-fgervais' information
Aug 22 20:23:20 localhost.localdomain snapd[1146]: 2018/08/22 20:23:20.031663 handlers.go:388: Reported install problem for "gadget" as already-reported
Aug 22 20:23:27 localhost.localdomain snapd[1146]: 2018/08/22 20:23:27.095814 handlers.go:388: Reported install problem for "kernel" as already-reported
Aug 22 20:23:34 localhost.localdomain snapd[1146]: 2018/08/22 20:23:34.702780 handlers.go:388: Reported install problem for "core" as already-reported

ogra · August 22, 2018, 9:51pm

checking:

shows that your error seems to be:

ERROR snap "gadget" supported architectures (amd64) are incompatible with this system (armhf)

make sure to build it for armhf …

in older snapcraft this requires the --target-arch switch, in newer you need to set the proper “architectures:” with “build-on” and “run-on” in your snapcraft.yaml:

here is an example where i use the new architectures setup:

https://github.com/ogra1/pi-kiosk-gadget/blob/master/snapcraft.yaml#L11

DC-fgervais · August 23, 2018, 3:48pm

Thanks a lot @ogra for your sustained support.

I though this architecture mismatch was some minor problem that I could fix later on but you were right, this was in fact preventing correct system initialization.

It seems to work fine now.

rarep · November 14, 2018, 1:10pm

Continuing the discussion from No snaps installed on a new ubuntu core port (kernel, gadget):

Hi,
I also ran into the same problem with no snaps installed after bootup. I built the kernel and gadget snaps for pi3b+ from source (no modification).

Kernel: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic; branch=raspi2
Gadget: https://github.com/snapcore/pi3-gadget; branch=18

And followed the ubuntu tutorial for creating the ubuntu image.
I could see from the logs that apparmor is enabled and all features are available.
But this:

snapd[1235]: daemon.go:344: started snapd/2.35.5 (series 16) ubuntu-core/16 (armhf) linux/4.15.18

series 16?
Am I checking out the wrong source for kernel and gadget for series 18?

ogra · November 14, 2018, 2:06pm

i think you need a special model assertion for core 18
(i dont think core18 images are stable enough yet for production use though … i havent touched them yet, perhaps @mvo can help out here)

rarep · November 15, 2018, 12:05pm

@ogra Thanks for the reply.
I remember I did change the model definition from 18 to 16 to pass the build since I encountered a problem when building:

error: model with series “18” != “16” unsupported

And I tried looking for model assertions for 18 from ubuntu site but to no avail.
I’ll just go with 16 then and checkout xenial. Thanks.

mpoid · February 3, 2023, 4:20pm

I am a new user in this forum. I try to build custom image Ubuntu Core based on ubuntu Core22 model assertion :

{ “type”: “model”, “series”: “16”, “authority-id”: “my-developer-id”, “brand-id”: “my-developer-id”, “model”: “ubuntu-core-22-amd64”, “architecture”: “amd64”, “timestamp”: “2023-02-03T14:51:34+00:00”, “base”: “core22”, “grade”: “dangerous”, “snaps”: [ { “name”: “pc”, “type”: “gadget”, “default-channel”: “22/stable”, “id”: “UqFziVZDHLSyO3TqSWgNBoAdHbLI4dAH” }, { “name”: “pc-kernel”, “type”: “kernel”, “default-channel”: “22/stable”, “id”: “pYVQrBcKmBa0mZ4CCN7ExT6jH8rY1hza” }, { “name”: “core22”, “type”: “base”, “default-channel”: “latest/stable”, “id”: “DLqre5XGLbDqg9jPtiAhRRjDuPVa5X1q” }, { “name”: “snapd”, “type”: “snapd”, “default-channel”: “latest/stable”, “id”: “PMrrV4ml8uWuEUDBT8dSGnKUYbevVhc4” } ] }

And I build a custom kernel snap based on : git clone -b v5.18 https://github.com/torvalds/linux.git

Here my snapcraft.yaml file:

name: pc-kernel-custom version: 5.18.0-051800-generic summary: kernel snap quick description description: kernel snap longer description grade: stable confinement: strict type: kernel build-base: core22 architectures:

build-on: amd64 build-for: amd64 parts: kernel: plugin: make source: ./linux/ firmware: plugin: nil stage-packages:

linux-firmware organize: lib/firmware: firmware prime:

-usr

-lib build-packages:

cpio

libssl-dev

And the build passed and I have this output: pc-kernel-custom_5.18.0-051800-generic_amd64.snap

–Building custom image by this command :

ubuntu-image snap custom-model.model --snap ./pc-kernel-custom_5.18.0-051800-generic_amd64.snap Blockquote

But when I test the resulting image It’s not boot. I have this error : error : no such device ubuntu-boot.

Can anyone help me ?