Launchpad infra build fails

store
1

Is there something unwell going on at the backend of the snapcraft build environment? I set aside some time to update some snaps. I have had a few failures not of my doing:

For example: From this buildlog

E: Failed to fetch http://ftpmaster.internal/ubuntu/pool/main/g/glibc/libc6-dev_2.31-0ubuntu9.2_amd64.deb  Temporary failure resolving 'ftpmaster.internal'

From this buildlog.

error: cannot perform the following tasks:
- Download snap "core18" (2246) from channel "stable" (received an unexpected http response code (401) when trying to download https://storage.snapcraftcontent.com/download-origin/local/CSO04Jhav2yK0uz97cr0ipQRyqg0qQL6_2246.snap?interactive=1&sig=IkNTTzA0SmhhdjJ5SzB1ejk3Y3IwaXBRUnlxZzBxUUw2XzIyNDYuc25hcCI.FGHm0A.Zux8cUWl2Qi94daD2edv4Xpn7tQ)
Install failed

Edit: 30 mins later, it fails again, differently. From this buildlog:

fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
[01/Nov/2021:23:37:33 +0000] "CONNECT github.com:443 HTTP/1.1" 200 1105119471 "-" "git/2.25.1"
Failed to pull source, command 'git clone --recursive https://github.com/Anuken/Mindustry.git /build/mindustry/parts/mindustry/src' exited with code 128.
Build failed

It’s somewhat demotivating to try and update snaps and find they fail to build multiple times in different ways I can’t fix other than by stabbing the “Trigger new build” button.

2

Back to failing to fetch archive builds. From this buildlog

E: Failed to fetch http://ftpmaster.internal/ubuntu/pool/universe/o/openjdk-16/openjdk-16-jdk-headless_16.0.1+9-1~20.04_amd64.deb  Connection timed out [IP: 10.10.10.1 8222]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Could not install all requested build packages: adwaita-icon-theme=3.36.1-2ubuntu0.20.04.2 ca-certificates-java=20190405ubuntu1 fontconfig=2.13.1-2ubuntu3 fontconfig-config=2.13.1-2ubuntu3 fonts-dejavu-core=2.37-1 gir1.2-atk-1.0=2.35.1-1ubuntu2 gir1.2-freedesktop=1.64.1-1~ubuntu20.04.1 gir1.2-gdkpixbuf-2.0=2.40.0+dfsg-3ubuntu0.2 gir1.2-glib-2.0=1.64.1-1~ubuntu20.04.1 gir1.2-gtk-2.0=2.24.32-4ubuntu4 gir1.2-harfbuzz-0.0=2.6.4-1ubuntu4 gir1.2-pango-1.0=1.44.7-2ubuntu4 gtk-update-icon-cache=3.24.20-0ubuntu1 hicolor-icon-theme=0.17-2 humanity-icon-theme=0.6.15 icu-devtools=66.1-2ubuntu2 java-common=0.72 jq=1.6-1ubuntu0.20.04.1 libasound2=1.2.2-2.1ubuntu2.4 libasound2-data=1.2.2-2.1ubuntu2.4 libasound2-dev=1.2.2-2.1ubuntu2.4 libatk1.0-0=2.35.1-1ubuntu2 libatk1.0-data=2.35.1-1ubuntu2 libatk1.0-dev=2.35.1-1ubuntu2 libavahi-client3=0.7-4ubuntu7.1 libavahi-common-data=0.7-4ubuntu7.1 libavahi-common3=0.7-4ubuntu7.1 libblkid-dev=2.34-0.1ubuntu9.1 libcairo-gobject2=1.16.0-4ubuntu1 libcairo-script-interpreter2=1.16.0-4ubuntu1 libcairo2=1.16.0-4ubuntu1 libcairo2-dev=1.16.0-4ubuntu1 libcups2=2.3.1-9ubuntu1.1 libdatrie-dev=0.2.12-3 libdatrie1=0.2.12-3 libdbus-1-3=1.12.16-2ubuntu2.1 libelf1=0.176-1.1build1 libexpat1-dev=2.2.9-1build1 libffi-dev=3.3-4 libfontconfig1=2.13.1-2ubuntu3 libfontconfig1-dev=2.13.1-2ubuntu3 libfreetype-dev=2.10.1-2ubuntu0.1 libfreetype6=2.10.1-2ubuntu0.1 libfreetype6-dev=2.10.1-2ubuntu0.1 libfribidi-dev=1.0.8-2 libfribidi0=1.0.8-2 libgdk-pixbuf2.0-0=2.40.0+dfsg-3ubuntu0.2 libgdk-pixbuf2.0-bin=2.40.0+dfsg-3ubuntu0.2 libgdk-pixbuf2.0-common=2.40.0+dfsg-3ubuntu0.2 libgdk-pixbuf2.0-dev=2.40.0+dfsg-3ubuntu0.2 libgirepository-1.0-1=1.64.1-1~ubuntu20.04.1 libglib2.0-0=2.64.6-1~ubuntu20.04.4 libglib2.0-bin=2.64.6-1~ubuntu20.04.4 libglib2.0-data=2.64.6-1~ubuntu20.04.4 libglib2.0-dev=2.64.6-1~ubuntu20.04.4 libglib2.0-dev-bin=2.64.6-1~ubuntu20.04.4 libgraphite2-3=1.3.13-11build1 libgraphite2-dev=1.3.13-11build1 libgtk2.0-0=2.24.32-4ubuntu4 libgtk2.0-common=2.24.32-4ubuntu4 libgtk2.0-dev=2.24.32-4ubuntu4 libharfbuzz-dev=2.6.4-1ubuntu4 libharfbuzz-gobject0=2.6.4-1ubuntu4 libharfbuzz-icu0=2.6.4-1ubuntu4 libharfbuzz0b=2.6.4-1ubuntu4 libice-dev=2:1.0.10-0ubuntu1 libice6=2:1.0.10-0ubuntu1 libicu-dev=66.1-2ubuntu2 libicu66=66.1-2ubuntu2 libjbig0=2.1-3.1build1 libjpeg-turbo8=2.0.3-0ubuntu1.20.04.1 libjpeg8=8c-2ubuntu8 libjq1=1.6-1ubuntu0.20.04.1 liblcms2-2=2.9-4 libmount-dev=2.34-0.1ubuntu9.1 libnspr4=2:4.25-1 libnss3=2:3.49.1-1ubuntu1.5 libonig5=6.9.4-1 libpango-1.0-0=1.44.7-2ubuntu4 libpango1.0-dev=1.44.7-2ubuntu4 libpangocairo-1.0-0=1.44.7-2ubuntu4 libpangoft2-1.0-0=1.44.7-2ubuntu4 libpangoxft-1.0-0=1.44.7-2ubuntu4 libpcre16-3=2:8.39-12build1 libpcre2-16-0=10.34-7 libpcre2-32-0=10.34-7 libpcre2-dev=10.34-7 libpcre2-posix2=10.34-7 libpcre3-dev=2:8.39-12build1 libpcre32-3=2:8.39-12build1 libpcrecpp0v5=2:8.39-12build1 libpcsclite1=1.8.26-3 libpixman-1-0=0.38.4-0ubuntu1 libpixman-1-dev=0.38.4-0ubuntu1 libpng-dev=1.6.37-2 libpthread-stubs0-dev=0.4-1 librsvg2-2=2.48.9-1ubuntu0.20.04.1 librsvg2-common=2.48.9-1ubuntu0.20.04.1 libselinux1-dev=3.0-1build2 libsepol1-dev=3.0-1 libsm-dev=2:1.2.3-1 libsm6=2:1.2.3-1 libthai-data=0.1.28-3 libthai-dev=0.1.28-3 libthai0=0.1.28-3 libtiff5=4.1.0+git191117-2ubuntu0.20.04.2 libwebp6=0.6.1-2ubuntu0.20.04.1 libx11-6=2:1.6.9-2ubuntu1.2 libx11-data=2:1.6.9-2ubuntu1.2 libx11-dev=2:1.6.9-2ubuntu1.2 libxau-dev=1:1.0.9-0ubuntu1 libxau6=1:1.0.9-0ubuntu1 libxcb-render0=1.14-2 libxcb-render0-dev=1.14-2 libxcb-shm0=1.14-2 libxcb-shm0-dev=1.14-2 libxcb1=1.14-2 libxcb1-dev=1.14-2 libxcomposite-dev=1:0.4.5-1 libxcomposite1=1:0.4.5-1 libxcursor-dev=1:1.2.0-2 libxcursor1=1:1.2.0-2 libxdamage-dev=1:1.1.5-2 libxdamage1=1:1.1.5-2 libxdmcp-dev=1:1.1.3-0ubuntu1 libxdmcp6=1:1.1.3-0ubuntu1 libxext-dev=2:1.3.4-0ubuntu1 libxext6=2:1.3.4-0ubuntu1 libxfixes-dev=1:5.0.3-2 libxfixes3=1:5.0.3-2 libxft-dev=2.3.3-0ubuntu1 libxft2=2.3.3-0ubuntu1 libxi-dev=2:1.7.10-0ubuntu1 libxi6=2:1.7.10-0ubuntu1 libxinerama-dev=2:1.1.4-2 libxinerama1=2:1.1.4-2 libxml2=2.9.10+dfsg-5ubuntu0.20.04.1 libxml2-utils=2.9.10+dfsg-5ubuntu0.20.04.1 libxrandr-dev=2:1.5.2-0ubuntu1 libxrandr2=2:1.5.2-0ubuntu1 libxrender-dev=1:0.9.10-1 libxrender1=1:0.9.10-1 openjdk-16-jdk-headless=16.0.1+9-1~20.04 openjdk-16-jre-headless=16.0.1+9-1~20.04 pango1.0-tools=1.44.7-2ubuntu4 pkg-config=0.29.1-0ubuntu4 python3-distutils=3.8.10-0ubuntu1~20.04 python3-lib2to3=3.8.10-0ubuntu1~20.04 shared-mime-info=1.15-1 ubuntu-mono=19.04-0ubuntu3 ucf=3.0038+nmu1 uuid-dev=2.34-0.1ubuntu9.1 wget=1.20.3-1ubuntu1 x11-common=1:7.7+19ubuntu14 x11proto-core-dev=2019.2-1ubuntu1 x11proto-dev=2019.2-1ubuntu1 x11proto-input-dev=2019.2-1ubuntu1 x11proto-randr-dev=2019.2-1ubuntu1 x11proto-xext-dev=2019.2-1ubuntu1 x11proto-xinerama-dev=2019.2-1ubuntu1 xorg-sgml-doctools=1:1.11-1 xtrans-dev=1.4.0-1 zlib1g-dev=1:1.2.11.dfsg-2ubuntu1.2
Build failed
3

I see this type of error quite frequently as well. Most usually DNS problems.

4

This mostly seems to be at the layer below Launchpad: DNS, connection timeouts, that sort of thing. (I can’t speak for the GitHub clone error; GitHub could have been flaky at the time, or it could have been a networking issue.) I only have indirect visibility into that sort of thing, but I’ve looked around things like firewall graphs and incident logs and I can’t see anything relevant at that time, and our general rate of build failures doesn’t seem to have been unusually high then either, so I don’t have a lot of leads.

However, one thing that would fairly clearly help here would be telling apt to retry failed connections a few times (it was changed to do that by default recently, but after focal), so I’ve filed a buildd bug to do that.

The snap download failure you linked seems likely to be a store failure of some kind; perhaps a store developer can speak to that.

1 Like