Cleanbuild remote on pi grabs wrong arch lxc image

[popey]

I’ve setup Ubuntu Core and lxd on a remote Pi. If I use “snapcraft cleanbuild --remote pi” on my laptop, it ships the build over to the pi, but errors with:-

$ snapcraft cleanbuild --remote pi2
Creating snapcraft-mildly-pumped-liger
error: Requested architecture isn't supported by this host
Stopping pi2:snapcraft-mildly-pumped-liger
error: not found

Looking at the lxd logs on the pi, I see:-

alias=xenial/amd64 image=8fa08537ae51c880966626561987153e72d073cbe19dfe5abc062713d929254d lvl=info msg="Downloading image" operation=2473916e-5f52-49f7-b9de-d1399487f25c server=https://cloud-images.ubuntu.com/releases t=2017-05-19T09:02:39+0000 trigger=/1.0/operations/2473916e-5f52-49f7-b9de-d1399487f25c
alias=xenial/amd64 image=8fa08537ae51c880966626561987153e72d073cbe19dfe5abc062713d929254d lvl=info msg="Image downloaded" operation=2473916e-5f52-49f7-b9de-d1399487f25c server=https://cloud-images.ubuntu.com/releases t=2017-05-19T09:03:11+0000 trigger=/1.0/operations/2473916e-5f52-49f7-b9de-d1399487f25c

Why is it downloading an amd64 image on a pi (armhf) device?

[evan]

This was resolved by using a newer snapcraft (snap refresh --edge snapcraft). It still isn’t happy though:

$ snapcraft cleanbuild --remote rpi
…
Setting up python3-magic (1:5.25-2ubuntu1) …
Setting up snapcraft (2.29) …
Processing triggers for libc-bin (2.23-0ubuntu7) …
Setting target machine to 'amd64’
Building for a different target architecture requires a plugin specific implementation in the ‘emoji’ plugin
Stopping rpi:snapcraft-superelaborately-nondisparate-carley
Command ‘[‘lxc’, ‘exec’, ‘rpi:snapcraft-superelaborately-nondisparate-carley’, ‘–env’, ‘HOME=/build_emoji’, ‘–’, ‘snapcraft’, ‘snap’, ‘–output’, ‘emoji_0.3.0_amd64.snap’, ‘–target-arch’, ‘amd64’]’ returned non-zero exit status 1.

[evan]

@sergiusens, @kalikiana this doesn’t seem like a safe assumption, given lxd remotes. Could we make it conditional on using local lxd?

[sergiusens]

Yes, we can fix this.

[kalikiana]

What exactly isn’t a safe assumption?
That code gets the architecture of the LXD host. armhf in the case of the Pi. By default snapcraft builds for the client architecture, which is amd64 here. So we end up cross-compiling on armhf for amd64.
If the intention was to build for armhf one would use --target-arch=armhf.

[evan]

Right, I don’t think that’s the right default. If I call snapcraft cleanbuild --remote rpi, I would expect it to produce an armhf build of the snap on the Raspberry Pi and then ferry it back to my amd64 machine.

[sergiusens]

I think this should be the case as well; as in native to the target and not native to the source.

[kalikiana]

So we want the default to be the LXD host architecture? In that case we need to figure out how to pass this on cleanly, because ProjectOptions gets its default from the client, before LXD comes into the picture.

[kalikiana]

lxd: only pass target arch if specified explicitly

[kalikiana]

Aaaaaand the code was merged!

[daniel]

Aaaaaand the code was merged!

I just tried using snapcraft from edge (2.34+git28.bf216fd) on my x86_64 PC to connect to my raspberry pi 3 running lxd in snappy Ubuntu core. It failed complaining that armhf and amd64 are not equal:

$ snapcraft cleanbuild --remote pi
Creating snapcraft-temptedly-defunct-ashlee
Starting snapcraft-temptedly-defunct-ashlee   
Waiting for a network connection...
Network connection established
<<<snipped>>>
Looking up assertion ['account-key', 'public-key-sha3-384=<<<snipped>>>']
Looking up assertion ['snap-declaration', 'snap-name=core']
Looking up assertion ['snap-revision', 'snap-revision=2898', 'snap-id=99T7MUlRhtI3U0QFgl5mXXESAiSwt776']
Adding assertion core_2898.assert
Installing /run/core_2898.snap
error: cannot perform the following tasks:
- Mount snap "core" (2898) (snap "core" supported architectures (amd64) are incompatible with this system (armhf))
Stopping pi:snapcraft-temptedly-defunct-ashlee
Traceback (most recent call last):
<<<snipped>>>
subprocess.CalledProcessError: Command '['lxc', 'exec', 'pi:snapcraft-temptedly-defunct-ashlee', '--', 'snap', 'install', '/run/core_2898.snap']' returned non-zero exit status 1.

[kalikiana]

Thanks for testing! I’ll have a look at that shortly. From the looks of it, Snapcraft mistakenly tries to inject the core snap of the host even though it has a different architecture. This is an interesting case as revisions of architectures don’t match. I’ll have to find out if I can map the revision or find the closest match, like latest on the same channel.

[daniel]

Surely for cleanbuilds it shouldn’t matter what is on the host. Why is snapcraft matching host to guest for cleanbuilds, even when they’re both the same PC/Arch?? The point of cleanbuild I thought was that it was an entirely isolated instance with nothing depending on your host system!

[kalikiana]

What Snapcraft does here is install the same versions of core snap and itself for predictable behavior. Since Snapcraft re-runs itself in the container, an older Snapcraft in the container eg. from apt would lead to potentially unexpected behavior. Fixes for plugins wouldn’t work even if the host has a newer version. And if you’re running from a specific channel, you want the version from that channel.

[kalikiana]

lxd: don't inject local snaps on a different arch

[kalikiana]

The PR was just merged!

[daniel]

What Snapcraft does here is install the same versions of core snap and itself for predictable behavior. Since Snapcraft re-runs itself in the container, an older Snapcraft in the container eg. from apt would lead to potentially unexpected behavior.

but is not the whole point of cleanbuild that it builds in a completely virgin environment with no ties to the host?!