We have been unable to build snaps on Ubuntu 20.04 LTS the last few days due to “Waiting for SSH to be up”. We went back to basics and used the “hello” example with the following snapcraft.yaml file:
name: hello
base: core20
version: '2.20'
summary: GNU Hello, the "hello world" snap
description: |
GNU hello prints a friendly greeting.
grade: devel
confinement: devmode
apps:
hello:
command: bin/hello
parts:
gnu-hello:
source: http://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz
plugin: autotools
On an Ubuntu 20.04 workstation (with all the latest updates) we cannot build the snap and the output of the ‘snapcraft --debug’ command is:
Launching a VM.
launch failed: The following errors occurred:
snapcraft-hello: timed out waiting for response
An error occurred with the instance when trying to launch with 'multipass': returned exit code 2.
Ensure that 'multipass' is setup correctly and try again.
The multipass log file shows:
2023-07-12T22:02:20-04:00 multipassd[4162797]: Waiting for SSH to be up
...
2023-07-12T22:09:21-04:00 multipassd[4162797]: Cannot open ssh session on "snapcraft-hello" shutdown: failed to determine IP address
and there is no trace of any DHCPOFFER entries. When we replace ‘core20’ in the snapcraft.yaml file with ‘core18’ or ‘core22’ it all works fine.
I am pulling out my last hairs trying to understand even why the official “hello” snap fails to build. This problem is reproducible on all our Ubuntu 20.04 workstations across different locations.
Who can help and what other information can I supply?
this is surely weird since snapcraft is supposed since a while already to default to use lxd containers instead of multipass VMs for builds, what version of the snapcraft snap do you have installed ? (snapcraft version should tell) …
also, do you have by chance the SNAPCRAFT_BUILD_ENVIRONMENT variable set and pointing to multipass ?
For our own builds (like the optumi-agent that we cannot build either) we use ‘classic’ confinement but for the hello example it is ‘devmode’. I am at a complete loss that this happened out of the blue on multiple workstations on last Monday (July 10). I would have assumed that many users would be seeing this. Can you try to run this hello example on your machine?
well, your issue is clearly not related to the source code you are trying to build, but to the multipass backend not properly starting, the error message is a clear indicator here:
An error occurred with the instance when trying to launch with 'multipass': returned exit code 2.
Ensure that 'multipass' is setup correctly and try again.
As other snippets of information, I see the same behavior when building the hello snap on an Ubuntu 22.04 LTS machine (i.e. core22 works but core20 does not). And trying to get the snapcraft configuration yields nothing (assuming I used the correct command ;):
$ sudo snap get snapcraft
error: snap "snapcraft" has no configuration
snapcraft has no configuration by default unless you set any … try using the env variable (after you have set up lxd (in case you have not used it yet, you want at least sudo lxd init --auto to set it up)) or the runtime option:
I used the command “snapcraft --debug --use-lxd” and am getting a lot further with the hello snap. It indeed starts with “Launching a container” and after lots of output lines, it ends with:
make[3]: Leaving directory '/root/parts/gnu-hello/build'
make[2]: Leaving directory '/root/parts/gnu-hello/build'
make[1]: Leaving directory '/root/parts/gnu-hello/build'
Staging gnu-hello
+ snapcraftctl stage
Priming gnu-hello
+ snapcraftctl prime
Failed to generate snap metadata: The specified command 'bin/hello' defined in the app 'hello' does not exist.
Ensure that 'bin/hello' is installed with the correct path.
snapcraft-hello #
I will also try to build our own snap with the --use-lxd option to see how that fares but that will take longer. Thanks for your help so far!
btw, trying the hello example copy/pasted from the website you linked above:
$ snapcraft --use-lxd
Launching a container.
Waiting for container to be ready
[...]
make[3]: Leaving directory '/root/parts/gnu-hello/build'
make[2]: Leaving directory '/root/parts/gnu-hello/build'
make[1]: Leaving directory '/root/parts/gnu-hello/build'
Staging gnu-hello
+ snapcraftctl stage
Priming gnu-hello
+ snapcraftctl prime
Snapping |
Snapped hello_2.10_amd64.snap
that said … looking at your snapcraft.yaml above, you edited base: in there … please note that snapcraft plugins change behavior and required syntax between different base versions …
in your particular case the base: core18 autotools plugin did auto-set --prefix=/usr … while base: core20 drops that behavior and lets the upstream default in place …
if i do the same change you did to the proposed snapcraft.yaml from the example, i see:
so when changing to core20 you should also have changed either the executable path in command: or set a --prefix=/usr option for the plugin to have the binary end up in /usr/bin instead of /usr/local/bin
Thank you for the explanation on --prefix, I learned something ;). And to follow-up, switching to using LXD instead of Multipass allowed us to build our own snaps again - very helpful!
Of courseI question is why all our developer workstations (almost all on Ubuntu 20.04) were configured to use Multipass when LXD is the default. Or is LXD the default on Ubuntu 22.04 but not on Ubuntu 20.04? If I remember correctly I had to set up LXD at some point when I was trying to build snaps using core22.
And the fact that using Multipass (instead of LXD) is not working - it that a problem on our workstations or can you reproduce that too with the hello snap?
LXD is only default for Core22 snaps, snaps using earlier bases still use Multipass unless overridden.
I think what makes this confusing, is LXD used to be default on earlier bases historically, until it was swapped to Multipass; and then back to LXD for Core22 snaps.
Ok, so that leaves the one question why using Multipass and core20 is not working for the hello snap due to some SSH/IP problem. Is that a known problem? Can I provide more information or can you reproduce it too?
Multipass uses “special” images for Snapcraft builds and it looks like the images were updated in the last day or two depending on the release which correlates to a few reports we’ve been seeing that Snapcraft is failing when using Multipass.
We will have to investigate to see what in the new images has caused this issue. These images are maintained by a different team, so I’ll follow up with them.
Just an FYI and for anyone else who may stumble on this thread due to this issue.
I have spotted the symptom affecting this and the team responsible for creating the images is now aware of the issue and will be working on a resolution. At this time, this only affects Focal based builds (core20).
