Building for core18: multipass issue

Hi

I’m in the process of porting my gnucash-jz package to core18. When I attempt any build, snapcraft (after prompting me to install multipass) attempts to start a VM, then hangs:

Launching a VM.
start failed: timed out waiting for instance to respond                         
An error occurred when trying to start the instance with 'multipass': returned exit code 2.
Ensure that 'multipass' is setup correctly and try again.

Rebooting, reinstalling multipass etc. doesn’t seem to fix the problem.
Is there some way to troubleshoot this?

NB: Ubuntu 18.10, amd64, kernel: 4.18.0-12-lowlatency

Thanks & Regards
Jacob

1 Like

What happens if you multipass launch? It should launch a vm.

Hi @jzimm can you have a look at snap logs -f multipass when running snapcraft? It may have pointers on what’s going on with Multipass.

$ multipass launch
Starting meticulous-gryphon

Hanging forever…

$ snap logs -f multipass
2018-12-12T22:05:08Z multipassd[20670]: QSslSocket: cannot resolve X509_getm_notAfter
2018-12-12T22:05:08Z multipassd[20670]: QSslSocket: cannot resolve X509_get_version
2018-12-12T22:05:08Z multipassd[20670]: QSslSocket: cannot resolve OpenSSL_version_num
2018-12-12T22:05:08Z multipassd[20670]: QSslSocket: cannot resolve OpenSSL_version
2018-12-12T22:05:08Z multipassd[20670]: Incompatible version of OpenSSL
2018-12-12T22:05:12Z multipass.multipassd[20670]: E1213 09:05:12.159723474   20670     uri_parser.cc:46]           bad uri.scheme: ''
2018-12-12T22:05:12Z multipass.multipassd[20670]: E1213 09:05:12.160181771   20670 uri_parser.cc:52]                            ^ here
2018-12-12T22:05:12Z multipass.multipassd[20670]: E1213 09:05:12.160317400   20670     http_proxy.cc:54]           cannot parse value of 'http_proxy' env var
2018-12-12T22:05:12Z multipassd[20670]: gRPC listening on unix:/run/multipass_socket, SSL:on
2018-12-12T22:05:12Z multipassd[20670]: QIODevice::write (QFile, "/var/snap/multipass/common    /cache/multipassd/vault/multipassd-image-records.json"): device not open
2018-12-12T22:06:03Z multipassd[20670]: process program 'qemu-system-x86_64'
2018-12-12T22:06:03Z multipassd[20670]: process arguments '--enable-kvm, -hda, /var/snap/multipass/common/data/multipassd/vault/instances/meticulous-gryphon/ubuntu-18.04-server-    cloudimg-amd64.img, -drive, file=/var/snap/multipass/common/data/multipassd/vault/instances/meticulous-gryphon/cloud-init-config.iso,if=virtio,format=raw, -smp, 1, -m, 1G, -device, virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:f2:35:28, -netdev, tap,id=hostnet0,ifname=tap-d0287099bf7,script=no,downscript=no, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic'
**strong text**2018-12-12T22:06:03Z multipassd[20670]: process state changed to Starting
2018-12-12T22:06:03Z multipassd[20670]: process state changed to Running
2018-12-12T22:06:03Z multipassd[20670]: process started
2018-12-12T22:06:04Z multipassd[20670]: Failed to open module: /usr/lib/x86_64-linux-gnu/qemu/block-iscsi.so: undefined symbol: scsi_sense_to_errno
Failed to open module: /usr/lib/x86_64-linux-gnu/qemu/block-curl.so: undefined symbol: qdict_put_str
Failed to open module: /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so: undefined symbol: aio_bh_schedule_oneshot

UPDATE: I think I isolated the problem. Multipass creates the virtual network interfaces mpqemubr0 and tap-XXXXXXXX. With firewalld installed on my machine, these were automatically assigned to the “public” zone and that apparently prevents the VM created by multipass from performing DHCP and initialising its network connection. The VM then just waits forever for the network to come up.

Manually switching these interfaces to “trusted” seems to fix it. Maybe multipass should set the correct firewall rules automatically?

1 Like

Hi @jzimm glad you worked it out. The problem is there’s so many firewall solutions out there it’s impossible to deal with them all. We do set up rules for the instances, but apparently your firewall overrides those.

I think the only meaningful thing we could do is report a better error, suggesting firewall issues when we can’t reach the instance.

A better error message, or some log message ("Waiting for VM network to initialize…) would certainly help. More people will unavoidably run into this problem.

1 Like

I ran into this problem earlier today and yesterday as well. I had to snapcraft clean and restart my computer to be able to build again. I have a fast internet connection and no special configuration for my network or firewall. It feels like Snapcraft sometimes waits for a failed network request and doesn’t retry or recognize that it’s dead.

In fact this also brings the question why exactly does snapcraft need to use a VM? A standardised build container, like snap cleanbuild but updated for core18, would be much more efficient.

1 Like

The release notes explain it better, but the gist of why VM’s are being used is because Snapcraft is now supported on MacOS and the experience is seamless between Linux and MacOS due to the use of Multipass.

@townsend VM does not solve the problem when we want to build process to consume all system resources. Unless snap is not meant to be a solution to create reproducible build environment. In that case, we then need to maintain two build system, snap and perhaps another container solution.

If Mac is the concern, why not support docker, which works on Linux and Mac?

Also, what’s the reason to retire lxd as a build environment? It was not in the release note and on this forum anywhere.

@townsend As I see it, this causes a regression for package maintainers. With the previous approach it was very convenient to keep a “prime” directory, possibly mounted into a build container, that could be mounted as a snap through “snap try”, and work iteratively in it until all necessary adjustments were made and integrated into snapcraft.yaml. Now with multipass, builds are slow, there is no possibility to just hack and try some stuff in a half-working snap to work it out, and it generally goes backwards in terms of user friendliness.

Two comments re the Mac concern. If that’s really the motivation, couldn’t the same build environment be provided as a container for Linux systems and as a VM image on Mac? For snapcraft maintainers, doesn’t it ultimately come down to maintaining a tarball that can be extracted either in a container or on a virtual disk?

And second, I don’t know if I’m the only one who thinks this, but is it fair and reasonable to annoy Linux developers in order to facilitate support of a proprietary, closed source platform that many Linux users frankly don’t give a flick about, to put it mildly?

@yihuaf, @jzimm,

Hey all! I’m a Multipass dev and not a Snapcraft dev, so I’ll leave up to them to answer your questions. I was just trying to give my opinion about why the change, but it seems to have lead to more questions that I can’t answer:)

1 Like

It was exactly what happened to me. I uninstalled ubuntu firewall and it solved the problem.
In addition I did some instructions from below (snapcraft clean; multipass launch; reboot) and now is working fine again

Hi all,

I also encountered the similar the issue when I used snapcraft to create a uc18 gadget for arm64:

    $ snapcraft cleanbuild --target-arch=arm64
    Setting target machine to 'arm64'
    Creating snapcraft-loudly-fresh-mole
    Starting snapcraft-loudly-fresh-mole
    ...
    You need multipass installed to build snaps which use the base keyword.
    Would you like to install it now? [y/N]: y
    snapd is not logged in, snap install commands will use sudo
    multipass (beta) 2018.12.1 from Canonical✓ installed
    Channel latest/beta for multipass is closed; temporarily forwarding to beta.
    Waiting for multipass...
    Setting target machine to 'arm64'
    Launching a VM.
    launch failed: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.15.0-1030-oem/modules.dep.bin'
    modprobe: FATAL: Module msr not found in directory /lib/modules/4.15.0-1030-oem
    An error occurred when trying to launch the instance with 'multipass': returned exit code 2.
    Ensure that 'multipass' is setup correctly and try again.
    Error: not found
    Stopping local:snapcraft-loudly-fresh-mole
    An error occurred when trying to copy files using 'lxd': returned exit code 1

I ensure that multipass can be launched formally under the host, however, the snapcraft cleanbuild always failed to trigger the multipass/lxd.

Hi @woodrow, by saying

do you mean that multipass launch works fine on the host normally?

The following messages:

    launch failed: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.15.0-1030-oem/modules.dep.bin'
    modprobe: FATAL: Module msr not found in directory /lib/modules/4.15.0-1030-oem
    An error occurred when trying to launch the instance with 'multipass': returned exit code 2.

suggest there’s something weird going on with your kernel packages. We’re using a tweaked kvm-ok script to check whether your host can do KVM. It fails loading the msr module that allows checking for BIOS support of virtualization. Can you check if you can modprobe msr and run the upstream kvm-ok script from the cpu-checker package?

If you would also please file an issue on our GitHub we can probably make this a non-fatal error in the check script.

Hi @Saviq,

Yes, after I met the issue, I tried to install mulitpass snap on host, and it did work with launch. There are actions I just went through:

$ lsmod | grep msr
msr                    16384  0
$ snap list
Name                  Version         Rev   Tracking  Publisher   Notes
core                  16-2.36.3       6130  stable    canonical✓  core
gnome-3-26-1604       3.26.0          74    stable/…  canonical✓  -
gnome-calculator      3.30.1          260   stable/…  canonical✓  -
gnome-characters      3.30.0          139   stable/…  canonical✓  -
gnome-logs            3.30.0          45    stable/…  canonical✓  -
gnome-system-monitor  3.30.0          57    stable/…  canonical✓  -
gtk-common-themes     0.1-4-g88bc1b2  818   stable/…  canonical✓  -
hw-probe              1.4-10          153   stable    linuxhw     -
multipass             2018.12.1       572   beta      canonical✓  classic
ubuntu-image          1.4+snap3       104   stable    canonical✓  classic
$ multipass launch
Launched: full-ribbonfish
$ multipass list
Name                    State             IPv4             Release
full-ribbonfish         RUNNING           10.127.219.211   Ubuntu 18.04 LTS
$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Like above situation, the thing is very strange to me. Why does “snapcraft cleanbuild” need to trigger multipass and lxd at the same time?

Thanks, I will do that if someone also affected this symptom.

That one sounds like a snapcraft issue, cleanbuild is a legacy command that should have nothing to do with Multipass.

Ok, I filed a bug on launchpad to track the case I hit, https://bugs.launchpad.net/snapcraft/+bug/1811208

I’m getting this issue when trying to build a package:

$ multipass launch
One quick question before we launch … Would you like to help                    
the Multipass developers, by sending anonymous usage data?
This includes your operating system, which images you use,
the number of instances, their properties and how long you use them.
We’d also like to measure Multipass’s speed.

Send usage data (yes/no/Later)? no

Launched: spruce-dugong                                   
# multipass itself is working fine                      
$ snapcraft 
Launching a VM.
launch failed: Downloaded image hash does not match                             
An error occurred when trying to launch the instance with 'multipass': returned exit code 2.
Ensure that 'multipass' is setup correctly and try again.

The most annoying thing is that it redownloads the image each time so that debugging this is really painful. Are there workarounds for this issue?

I can’t even use cleanbuild instead :’(

snapcraft cleanbuild
The cleanbuild command is no longer supported when using the base keyword.

How do I directly build on the host system, nowadays?

Thanks a lot for your help!