Armhf builds on Launchpad time out while running rustup

snapcraft
#1

Experiencing some strange behavior on Launchpad armhf builds triggered via build.snapcraft.io: A couple of days ago I had a build that failed after ~49 hours (!), with no build log available (launchpad link). That same build ran on arm64 in 20 minutes so I figured it must be a one-off glitch, and retried.

The build is currently running (launchpad link), but has been going for 17 hours and appears to have hung. I don’t know to what extent the log tail I can see at the above link is representative of its status. The last couple of lines are the start of the Rust compiler installation:

Executing parts lifecycle: pull rust-deps [24/Jul/2022:10:38:57 +0000] "CONNECT sh.rustup.rs:443 HTTP/1.1" 200 28838 "-" "curl/7.81.0"
[24/Jul/2022:10:39:01 +0000] "CONNECT static.rust-lang.org:443 HTTP/1.1" 200 14959749 "-" "curl/7.81.0"

The Rust install command (snapcraft.yaml link) does work on armhf, so I can’t imagine that being the actual problem, but who knows.

Any ideas?

#2

I’m seeing a similar problem with all armhf builds of the firefox snap: they hang (seemingly) forever here:

Building rust
+ REQUIRED_RUST_VERSION=1.61.0
+ sh rustup.sh -q -y --default-toolchain 1.61.0
info: downloading installer
[25/Jul/2022:12:52:12 +0000] "CONNECT static.rust-lang.org:443 HTTP/1.1" 200 15000942 "-" "curl/7.68.0"
info: profile set to 'default'
info: default host triple is armv7-unknown-linux-gnueabihf
info: syncing channel updates for '1.61.0-armv7-unknown-linux-gnueabihf'

The last successful armhf build I could find is from 2022-07-11, so the regression happened in the past two weeks.

#3

I’ve passed the -v flag to rustup.sh in an attempt to understand better where the hang is happening:

2022-07-25 16:13:39 (9.00 MB/s) - ‘/tmp/tmp.li3KklcYXv/rustup-init’ saved [14910276/14910276]
[25/Jul/2022:16:13:39 +0000] "CONNECT static.rust-lang.org:443 HTTP/1.1" 200 14943297 "-" "Wget/1.20.3 (linux-gnueabihf)"
info: profile set to 'default'
info: default host triple is armv7-unknown-linux-gnueabihf
verbose: creating update-hash directory: '/root/.rustup/update-hashes'
verbose: installing toolchain '1.61.0-armv7-unknown-linux-gnueabihf'
verbose: toolchain directory: '/root/.rustup/toolchains/1.61.0-armv7-unknown-linux-gnueabihf'
info: syncing channel updates for '1.61.0-armv7-unknown-linux-gnueabihf'
verbose: creating temp root: /root/.rustup/tmp
verbose: creating temp file: /root/.rustup/tmp/o4mng72qw59pgfvy_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-1.61.0.toml.sha256'
verbose: downloading with reqwest
#4

Hm interesting - thanks for sharing your experience @oSoMoN. So it sounds like in both of our cases it’s happening while running rustup…but for no discernible reason.

#5

And I cannot reproduce locally when building the firefox snap in a focal/armhf container on a raspberry pi 4.

#6

This is probably https://github.com/seanmonstar/reqwest/issues/642.

#7

At Colin’s suggestion, I’ve kicked off a test build where rustup.sh is running under strace. The build is currently hung, waiting to time out, and here is the output I can see for now:

[pid 11863] mprotect(0xf7391000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7392000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7393000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] read(7, "fySU0\nWXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS\nfvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB\nzhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq\nhkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB\nCUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD\n+JbNR6iC8hZVdyR+EhCVBCyj\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDczCCAlugAwIBAgILAK7PALrEzzL4Q7IwDQYJKoZIhvcNAQELBQAwVjELMAkG\nA1UEBhMCVVMxEzARBgNVBAsTCmVtU2lnbiBQS0kxFDASBgNVBAoTC2VNdWRocmEg\nSW5jMRwwGgYDVQQDExNlbVNpZ24gUm9vdCBDQSAtIEMxMB4XDTE4MDIxODE4MzAw\nMFoXDTQzMDIxODE4MzAwMFowVjELMAkGA1UEBhMCVVMxEzARBgNVBAsTCmVtU2ln\nbiBQS0kxFDASBgNVBAoTC2VNdWRocmEgSW5jMRwwGgYDVQQDExNlbVNpZ24gUm9v\ndCBDQSAtIEMxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAz+upufGZ\nBczYKCFK83M0UYRWEPWgTywS4/oTmifQz/l5GnRfHXk5/Fv4cI7gklL35CX5VIPZ\nHdPIWoU/Xse2B+4+wM6ar6xWQio5JXDWv7V7Nq2s9nPczdcdioOl+yuQFTdrHCZH\n3DspVpNqs8FqOp099cGXOFgFixwR4+S0uF2FHYP+eF8LRWgYSKVGczQ7/g/IdrvH\nGPMF0Ybzhe3nudkyrVWIzqa2kbBPr"..., 4096) = 2941
[pid 11863] mprotect(0xf7394000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7395000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] mprotect(0xf7396000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 11863] read(7, "", 4096) = 0
[pid 11863] close(7) = 0
#8

I am now working around the problem in the firefox snap by not using rustup to pull the rust toolchain.

1 Like
#9

Huh, I had no idea the underlying issues here ran so deep - thanks for the pointer (and attempted solutions) @cjwatson . Given that it’s been open for about 3 years now I suppose it’s unlikely for there to be a quick fix at this point. Thanks for sharing your workaround @oSoMoN - will probably just need to go with that.