Customised cm3 image fails on reboot

seanlano · May 9, 2018, 6:01am

I’ve also posted this as a bug on the cm3-gadget page, but I wanted to ask here too because it seems like something else might be happening that isn’t necessarily related to the cm3-gadget Snap.

I’m trying to make a custom gadget Snap for a Compute Module 3, so that I can change the default device tree.

As an example, I’ve created a model assertion, which contains this:

type: model
authority-id: JUVaYs8gDh212C6nrFVN4CYsuPo6kixd
series: 16
brand-id: JUVaYs8gDh212C6nrFVN4CYsuPo6kixd
model: cm3-stock
architecture: armhf
gadget: cm3
kernel: pi2-kernel
timestamp: 2018-05-07T01:24:38+00:00
sign-key-sha3-384: u1OFY0uzgzlU7xZ1P3Sr4UavaF3I6Wjq3cNxA1OaUQ4pt8e213tH-uuwQUW2jD6Y

AcLBXAQAAQoABgUCWvJqwAAKCRC5TOT5ixabevKvD/46GvlxM6/v1eqLyycXXxmoD5ljv9dusTDB
E/kFM7wlp912qvizbH9grxS0IXImlRqIl7ty5mcAob3wzV0ZrgLhWmbYtO26lDYN/gv93Y5UR2zN
O5bq3Fq9b74lzKxRDNVo6DKz+yOqXvl7sqBBh2jmpilsk6WWAkXNU2vT0V0fJ1IuVJef+xfK/xAx
lcrLeJZMolYWC3QilSTeFbjNwLpyD4pihtVg0wmCq8F6PAy4sFST+FCb1cCx+DVSdzDCubE1siX7
z0hIC2ATWFPK4/88Vbq/OHriQ7/UhnRle4/bXMCvZralmJA0JfJvbR+BXXN0OanVTSiLDXZmuVDY
6cq2Ca5KH3ocLcuuzbIqjllv5HmcKmcj57DPkUquUiw83UZ7xiMiqlwlwgDXGHmjKL6KPZ9+F9KL
HSNX0UCwjBrRcqFplcj3w+O/Bkm3iWqcrMeEN8kzCGRnRvoMV3tkt8VdjEUU97pSlOXegt8Oy5xm
icdnVlCzdMebJHVgUhXuNNeRkjkhk/pNm4+T+GCObmmgTIphbPHmZKtkAb4sEl3h3t5PAPk6IDiC
bu/30+5Wi193TCTtjlhe8VCv5lzVf0e9p59bbpFfTjc8RrESbYQKTSTQAlrN/YpBCOXCj4fxNv37
V+yWttReQjZlj9jKNMZgNBDCFwHhA3ZJJB3c2dkrMg==

If I create an Ubuntu Core image using this model assertion with the kernel and gadget Snaps from the global store, it works fine. This is the command to make the image:

$ sudo ubuntu-image snap -c stable cm3-stock.model

I can copy this to the Compute Module, and it works fine.

However, if I clone the cm3-gadget repo and build it with Snapcraft, like this:

$ git clone https://github.com/snapcore/cm3-gadget.git
...
$ cd cm3-gadget
$ snapcraft
...

I get a (seemingly) correctly built Snap package, cm3_16.04-0.5_amd64.snap (I realise it says amd64 but I presume this is because that is my host system architecture). I then build a new image with:

$ sudo ubuntu-image snap -c stable --extra-snaps cm3_16.04-0.5_amd64.snap cm3-stock.model

I can copy this to the Compute Module, and it works and correctly boots - the first time. But as soon as I reboot, regardless of if done with sudo reboot or by a power cycle, the subsequent boots never work. U-boot gives output like this:

U-Boot 2017.05-dirty (May 09 2018 - 14:35:32 +1000)

DRAM:  944 MiB
RPI Compute Module 3 (0xa020a0)
MMC:   bcm2835_sdhci: 0
reading uboot.env
In:    serial
Out:   lcd
Err:   lcd
Net:   Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0(part 0) is current device
LOADBOOTENV
reading uEnv.txt
** Unable to read file uEnv.txt **
Running uenvcmd ...
ENVCMD
reading pi2-kernel_52.snap/kernel.img
** Unable to read file pi2-kernel_52.snap/kernel.img **
reading pi2-kernel_52.snap/initrd.img
** Unable to read file pi2-kernel_52.snap/initrd.img **
Bad Linux ARM zImage magic!
Scanning mmc 0:1...
Scanning mmc 0:2...
starting USB...
USB0:   Core Release: 2.80a
scanning bus 0 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
       scanning usb for ethernet devices... 0 Ethernet Device(s) found

USB device 0: unknown device
No ethernet found.

It somehow is unable to read the kernel image. If I mount the CM3 storage, it looks like the kernel Snap folder has been completely removed!

This is the initial, working, system-boot folder:

$  ls -lh system-boot
total 13M
-rw-r--r-- 1 sean sean  15K May 10  2018 bcm2709-rpi-2-b.dtb
-rw-r--r-- 1 sean sean  16K May 10  2018 bcm2710-rpi-3-b.dtb
-rw-r--r-- 1 sean sean  15K May 10  2018 bcm2710-rpi-cm3.dtb
-rw-r--r-- 1 sean sean  50K May 10  2018 bootcode.bin
-rw-r--r-- 1 sean sean  104 May 10  2018 cmdline.txt
-rw-r--r-- 1 sean sean 1.6K May 10  2018 config.txt
-rw-r--r-- 1 sean sean  19K May 10  2018 COPYING.linux
-rw-r--r-- 1 sean sean 2.6K May 10  2018 fixup_cd.dat
-rw-r--r-- 1 sean sean 6.6K May 10  2018 fixup.dat
-rw-r--r-- 1 sean sean 9.6K May 10  2018 fixup_db.dat
-rw-r--r-- 1 sean sean 9.6K May 10  2018 fixup_x.dat
-rw-r--r-- 1 sean sean 1.5K May 10  2018 LICENCE.broadcom
drwxr-xr-x 2 sean sean  10K May 10  2018 overlays
drwxr-xr-x 3 sean sean  512 May 10  2018 pi2-kernel_52.snap
-rw-r--r-- 1 sean sean 643K May 10  2018 start_cd.elf
-rw-r--r-- 1 sean sean 4.8M May 10  2018 start_db.elf
-rw-r--r-- 1 sean sean 2.8M May 10  2018 start.elf
-rw-r--r-- 1 sean sean 3.8M May 10  2018 start_x.elf
-rw-r--r-- 1 sean sean 352K May 10  2018 uboot.bin
-rw-r--r-- 1 sean sean 128K May 10  2018 uboot.env

And this is after the first boot and reboot:

$  ls -lh system-boot
total 13M
-rw-r--r-- 1 sean sean  15K May 10  2018 bcm2709-rpi-2-b.dtb
-rw-r--r-- 1 sean sean  16K May 10  2018 bcm2710-rpi-3-b.dtb
-rw-r--r-- 1 sean sean  15K May 10  2018 bcm2710-rpi-cm3.dtb
-rw-r--r-- 1 sean sean  50K May 10  2018 bootcode.bin
-rw-r--r-- 1 sean sean  104 May 10  2018 cmdline.txt
-rw-r--r-- 1 sean sean 1.6K May 10  2018 config.txt
-rw-r--r-- 1 sean sean  19K May 10  2018 COPYING.linux
-rw-r--r-- 1 sean sean 2.6K May 10  2018 fixup_cd.dat
-rw-r--r-- 1 sean sean 6.6K May 10  2018 fixup.dat
-rw-r--r-- 1 sean sean 9.6K May 10  2018 fixup_db.dat
-rw-r--r-- 1 sean sean 9.6K May 10  2018 fixup_x.dat
-rw-r--r-- 1 sean sean 1.5K May 10  2018 LICENCE.broadcom
drwxr-xr-x 2 sean sean  10K May 10  2018 overlays
-rw-r--r-- 1 sean sean 643K May 10  2018 start_cd.elf
-rw-r--r-- 1 sean sean 4.8M May 10  2018 start_db.elf
-rw-r--r-- 1 sean sean 2.8M May 10  2018 start.elf
-rw-r--r-- 1 sean sean 3.8M May 10  2018 start_x.elf
-rw-r--r-- 1 sean sean 352K May 10  2018 uboot.bin
-rw-r--r-- 1 sean sean 128K May 10  2018 uboot.env

The pi2-kernel_52.snap folder is entirely gone! Why would this be? This doesn’t happen with the cm3 gadget Snap from the global store, only when I use my self-built Snap - and everything else is the same.

seanlano · May 9, 2018, 6:58am

I’ve tried it again, this time I thought I’d re-install the kernel snap just after the console-config section and before the reboot. I SCP’ed the pi2-kernel snap over to the CM3, then tried to install it, but I get this error:

$ sudo snap install --dangerous pi2-kernel_52.snap 
error: cannot install snap file: snap "pi2-kernel" has "seed" change in progress

What is the “seed” change?

ogra · May 9, 2018, 9:29am

see with:

snap changes

(and if you see an error:

snap change $number

for details about the change)

chipaca · May 9, 2018, 10:25am

or just snap tasks --last=seed

seanlano · May 9, 2018, 11:09pm

Ah, OK - that seems to be it. The above commands give this:

$ snap changes
ID   Status  Spawn                 Ready                 Summary
1    Error   2018-05-09T22:57:29Z  2018-05-09T22:58:01Z  Initialize system state
2    Error   2018-05-09T22:58:48Z  2018-05-09T22:59:12Z  Initialize system state
3    Error   2018-05-09T23:00:00Z  2018-05-09T23:00:20Z  Initialize system state
4    Do      2018-05-09T23:01:08Z  -                     Initialize system state

and then:

$ snap change 1
Status  Spawn                 Ready                 Summary
Done    2018-05-09T22:56:55Z  2018-05-09T22:58:01Z  Ensure prerequisites for "core" are available
Undone  2018-05-09T22:56:55Z  2018-05-09T22:58:01Z  Prepare snap "/var/lib/snapd/seed/snaps/core_4573.snap" (4573)
Undone  2018-05-09T22:56:55Z  2018-05-09T22:58:00Z  Mount snap "core" (4573)
Undone  2018-05-09T22:56:55Z  2018-05-09T22:57:58Z  Copy snap "core" data
Undone  2018-05-09T22:56:55Z  2018-05-09T22:57:58Z  Setup snap "core" (4573) security profiles
Undone  2018-05-09T22:56:55Z  2018-05-09T22:57:58Z  Make snap "core" (4573) available to the system
Undone  2018-05-09T22:56:55Z  2018-05-09T22:57:58Z  Setup snap "core" (4573) security profiles (phase 2)
Undone  2018-05-09T22:56:55Z  2018-05-09T22:57:58Z  Automatically connect eligible plugs and slots of snap "core"
Undone  2018-05-09T22:56:55Z  2018-05-09T22:57:58Z  Set automatic aliases for snap "core"
Undone  2018-05-09T22:56:55Z  2018-05-09T22:57:57Z  Setup snap "core" aliases
Done    2018-05-09T22:56:55Z  2018-05-09T22:57:57Z  Run install hook of "core" snap if present
Undone  2018-05-09T22:56:55Z  2018-05-09T22:57:57Z  Start snap "core" (4573) services
Done    2018-05-09T22:57:29Z  2018-05-09T22:57:57Z  Ensure prerequisites for "pi2-kernel" are available
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:57Z  Prepare snap "/var/lib/snapd/seed/snaps/pi2-kernel_52.snap" (52)
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:56Z  Mount snap "pi2-kernel" (52)
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:55Z  Copy snap "pi2-kernel" data
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:55Z  Setup snap "pi2-kernel" (52) security profiles
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:55Z  Make snap "pi2-kernel" (52) available to the system
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:54Z  Setup snap "pi2-kernel" (52) security profiles (phase 2)
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:54Z  Automatically connect eligible plugs and slots of snap "pi2-kernel"
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:54Z  Set automatic aliases for snap "pi2-kernel"
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:54Z  Setup snap "pi2-kernel" aliases
Done    2018-05-09T22:57:29Z  2018-05-09T22:57:54Z  Run install hook of "pi2-kernel" snap if present
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:54Z  Start snap "pi2-kernel" (52) services
Done    2018-05-09T22:57:29Z  2018-05-09T22:57:53Z  Ensure prerequisites for "cm3" are available
Undone  2018-05-09T22:57:29Z  2018-05-09T22:57:53Z  Prepare snap "/var/lib/snapd/seed/snaps/cm3_x1.snap" (unset)
Error   2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Mount snap "cm3" (unset)
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Copy snap "cm3" data
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Setup snap "cm3" (unset) security profiles
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Make snap "cm3" (unset) available to the system
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Setup snap "cm3" (unset) security profiles (phase 2)
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Automatically connect eligible plugs and slots of snap "cm3"
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Set automatic aliases for snap "cm3"
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Setup snap "cm3" aliases
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Run install hook of "cm3" snap if present
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Start snap "cm3" (unset) services
Hold    2018-05-09T22:56:55Z  2018-05-09T22:57:46Z  Run configure hook of "core" snap if present
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Run configure hook of "pi2-kernel" snap if present
Hold    2018-05-09T22:57:29Z  2018-05-09T22:57:46Z  Run configure hook of "cm3" snap if present
Hold    2018-05-09T22:56:34Z  2018-05-09T22:57:46Z  Mark system seeded
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Run hook prepare-plug-core-support-plug of snap "core"
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Run hook prepare-slot-core-support of snap "core"
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Connect core:core-support-plug to core:core-support
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Run hook connect-slot-core-support of snap "core"
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Run hook connect-plug-core-support-plug of snap "core"
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Run hook prepare-plug-core-support-plug of snap "core"
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Run hook prepare-slot-core-support of snap "core"
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Connect core:core-support-plug to core:core-support
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Run hook connect-slot-core-support of snap "core"
Done    2018-05-09T22:57:36Z  2018-05-09T22:57:58Z  Run hook connect-plug-core-support-plug of snap "core"

......................................................................
Mount snap "cm3" (unset)

2018-05-09T22:57:46Z ERROR snap "cm3" supported architectures (amd64) are incompatible with this system (armhf)

So, my method of building the gadget snap is the issue! Turns out that stray amd64 in my first post can’t just be ignored.

OK, so if I try to create the gadget snap with snapcraft --target-arch=armhf, I get

NotImplementedError: The plugin used by 'uboot' does not support cross-compiling to a different target architecture

What is the easiest way to make the gadget snap for armhf from my amd64 machine?

seanlano · May 9, 2018, 11:17pm

I just tried doing snapcraft prime, and then manually editing the meta/snap.yaml to say armhf instead of amd64, but when I then run snapcraft snap it seems to detect this and revert it, such that the resulting .snap file says amd64 again.

seanlano · May 10, 2018, 2:38am

OK, success! I built the gadget snap on the Pi itself, then copied it and built my image. Now it works! I still can’t figure out how to cross-compile the gadget snap, but creating it on-target seems to work.

ogra · May 10, 2018, 12:42pm

Here is a fix for the cross compilation (snapcraft changed and the snap had not been updated for it)

https://github.com/snapcore/cm3-gadget/pull/4