Launchpad builders failing to clone source for part - Received HTTP code 407 from proxy after CONNECT

snapcraft
1

I’ve recently been working on creating a content snap containing Qt 5.15.2 and KDE Frameworks 5.85 both compiled form source (as a result the snap does take a long time to build with qtwebengine being the primary culprit for long build times) with the purpose being to resolve issues such as the following:


With the former requiring a patch to qtbase.

I’m able to successfully build the snap on my local machines (amd64 and arm64 rpi4) but when using the launchpad builders to build the snap, none of my builds have managed to clone the source for extra-cmake-modules from either https://invent.kde.org/frameworks/extra-cmake-modules or the KDE Github mirror https://github.com/KDE/extra-cmake-modules. The exact error I’m getting on the amd64, arm64 and armhf builders is the following:

Pulling extra-cmake-modules 
+ snapcraftctl pull
Cloning into '/build/nightmayr-kf5-qt-5-15-2-core20/parts/extra-cmake-modules/src'...
[06/Sep/2021:03:38:43 +0000] "CONNECT github.com:443 HTTP/1.1" 407 1782 "-" "git/2.25.1"
fatal: unable to access 'https://github.com/KDE/extra-cmake-modules.git/': Received HTTP code 407 from proxy after CONNECT
Failed to pull source, command 'git clone --recursive --branch v5.85.0 https://github.com/KDE/extra-cmake-modules.git /build/nightmayr-kf5-qt-5-15-2-core20/parts/extra-cmake-modules/src' exited with code 128.
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 230, in run
    self.pull()
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 196, in pull
    env=env)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 100, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py", line 537, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'exec', 'lp-focal-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_LOCAL_SOURCES=1', '--env', 'SNAPCRAFT_SETUP_CORE=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-65891863", "build-request-timestamp": "2021-09-05T18:01:14Z", "build_url": "https://launchpad.net/~build.snapcraft.io/+snap/7e2da169bd9488c3a7d2eaf617d9e411/+build/1514740"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/lpbuildd-git-proxy', '--', '/bin/sh', '-c', 'cd /build/nightmayr-kf5-qt-5-15-2-core20 && linux64 snapcraft pull']' returned non-zero exit status 2.
Revoking proxy token...
Unable to revoke token for SNAPBUILD-1514740-1630864885: HTTP Error 401: UnauthorizedRUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=focal --arch=amd64 SNAPBUILD-1514740
Scanning for processes to kill in build SNAPBUILD-1514740

Snapcraft yaml: https://github.com/Nightmayr-snaps/nightmayr-kf5-qt-5-15-2-core20-snap/blob/beb2d8426f5d6e2e0b8d836c1d31273346728fe4/snap/snapcraft.yaml
Launchpad builds: https://launchpad.net/~build.snapcraft.io/+snap/7e2da169bd9488c3a7d2eaf617d9e411

2

This is caused by a particularity of Launchpad’s build environment for snaps: it has a timeout for how long it is allowed to access the internet (through a proxy) starting from the start of the build.

I know I’ve been working around this feature in several of the snaps I maintain (e.g. in the chromium snap) by ensuring all the sources are pulled before building parts.

This is a bit annoying as it artificially adds complexity to the project’s snapcraft.yaml.

3

Ah that’s unfortunate, so the solution is to build the part that takes the longest last? I can look into building QtWebengine last but unless there’s a KF5 part that doesn’t use it as a dependency once staged I dont know if it would be possible in the case of my snap.

Do you know if this is something that is likely to be sorted/changed for the Launchpad builders in future? I’m assuming the timeout doesn’t affect pushing the snap to the store. How long does it take for the timeout to kick in?

4

if it did not change recently it should be 3h …

1 Like
5

Or split parts into two, one that pulls sources, and another one that builds without the need to access the internet. Then use ordering (the after keyword) to make sure all source parts are processed first. This is a bit cumbersome, but it works.

1 Like
6

As a side note: these packages you’re building could possibly solve issues as the ones I raised here.

1 Like
7

Thanks for the suggestion, that ended up doing the trick! I ended pulling all sources into a ‘sources’ part and ended up doing an override-pull (tried using the source and source-type: local but wasn’t able to get it to expand the SNAPCRAFT_PART environment variables) for each part looking like:

override-pull: |
            cp -a $SNAPCRAFT_PART_SRC/../../sources/src/qtwebengine/. $SNAPCRAFT_PART_SRC

In the case of my build qtwebengine is basically compiling chromium during its build step so takes a long time (especially on Launchpad’s arm64 and armhf builders), is it possible to build qtwebengine in a separate intermediate snap and then copy the contents to this content snap or are there problems with this approach (is this what stage-snaps is used for)?

8

That works but it makes too many assumptions on snapcraft’s internal directory structure. You could instead pull your sources in $SNAPCRAFT_STAGE, which is a shared area between all parts. In that case though, make sure the parts that do that also have an empty override-prime stanza to ensure the sources don’t end up in the final snap.

If qtwebengine is updated less often than your final content snap, then yes this makes sense.

1 Like