Armhf builds on Launchpad time out while running rustup

Experiencing some strange behavior on Launchpad armhf builds triggered via A couple of days ago I had a build that failed after ~49 hours (!), with no build log available (launchpad link). That same build ran on arm64 in 20 minutes so I figured it must be a one-off glitch, and retried.

The build is currently running (launchpad link), but has been going for 17 hours and appears to have hung. I don’t know to what extent the log tail I can see at the above link is representative of its status. The last couple of lines are the start of the Rust compiler installation:

Executing parts lifecycle: pull rust-deps [24/Jul/2022:10:38:57 +0000] "CONNECT HTTP/1.1" 200 28838 "-" "curl/7.81.0"
[24/Jul/2022:10:39:01 +0000] "CONNECT HTTP/1.1" 200 14959749 "-" "curl/7.81.0"

The Rust install command (snapcraft.yaml link) does work on armhf, so I can’t imagine that being the actual problem, but who knows.

Any ideas?

I’m seeing a similar problem with all armhf builds of the firefox snap: they hang (seemingly) forever here:

Building rust
+ sh -q -y --default-toolchain 1.61.0
info: downloading installer
[25/Jul/2022:12:52:12 +0000] "CONNECT HTTP/1.1" 200 15000942 "-" "curl/7.68.0"
info: profile set to 'default'
info: default host triple is armv7-unknown-linux-gnueabihf
info: syncing channel updates for '1.61.0-armv7-unknown-linux-gnueabihf'

The last successful armhf build I could find is from 2022-07-11, so the regression happened in the past two weeks.

I’ve passed the -v flag to in an attempt to understand better where the hang is happening:

2022-07-25 16:13:39 (9.00 MB/s) - ‘/tmp/tmp.li3KklcYXv/rustup-init’ saved [14910276/14910276]
[25/Jul/2022:16:13:39 +0000] "CONNECT HTTP/1.1" 200 14943297 "-" "Wget/1.20.3 (linux-gnueabihf)"
info: profile set to 'default'
info: default host triple is armv7-unknown-linux-gnueabihf
verbose: creating update-hash directory: '/root/.rustup/update-hashes'
verbose: installing toolchain '1.61.0-armv7-unknown-linux-gnueabihf'
verbose: toolchain directory: '/root/.rustup/toolchains/1.61.0-armv7-unknown-linux-gnueabihf'
info: syncing channel updates for '1.61.0-armv7-unknown-linux-gnueabihf'
verbose: creating temp root: /root/.rustup/tmp
verbose: creating temp file: /root/.rustup/tmp/o4mng72qw59pgfvy_file
verbose: downloading file from: ''
verbose: downloading with reqwest

Hm interesting - thanks for sharing your experience @oSoMoN. So it sounds like in both of our cases it’s happening while running rustup…but for no discernible reason.

And I cannot reproduce locally when building the firefox snap in a focal/armhf container on a raspberry pi 4.

This is probably

At Colin’s suggestion, I’ve kicked off a test build where is running under strace. The build is currently hung, waiting to time out, and here is the output I can see for now:

[pid 11863] mprotect(0xf7391000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7392000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7393000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7394000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7395000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7396000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] read(7, "", 4096) = 0
[pid 11863] close(7) = 0

I am now working around the problem in the firefox snap by not using rustup to pull the rust toolchain.

1 Like

Huh, I had no idea the underlying issues here ran so deep - thanks for the pointer (and attempted solutions) @cjwatson . Given that it’s been open for about 3 years now I suppose it’s unlikely for there to be a quick fix at this point. Thanks for sharing your workaround @oSoMoN - will probably just need to go with that.