Problems with remote build?

Hi,

Anyone else experiencing problems with remote builds? I’m using GitHub + Snapcraft.io.

I normally have to trigger a build multiple times to succeed for every architecture.

Today I’ve also experienced failed builds (constantly, for every architecture) for one of the snaps I’m maintaining and I’m not sure what’s wrong. It’s building fine on my local machines.

On remote it (home-assistant-snap) fails with:

Check the build logs and ensure the part's configuration and sources are correct.
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 258, in run
    self.build()
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 247, in build
    env=env)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 100, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py", line 537, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'exec', 'lp-focal-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-63549903", "build-request-timestamp": "2021-05-11T19:17:18Z", "build_url": "https://launchpad.net/~build.snapcraft.io/+snap/c8c030b03a17f51bd10e02ab2f95e72b/+build/1405599"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/home-assistant-snap && linux64 snapcraft']' returned non-zero exit status 2.
Revoking proxy token...

Unsure if it’s a proxy issue again?

The other (zwavejs2mqtt) is failing of various reasons:

[11/May/2021:16:32:41 +0000] "CONNECT registry.npmjs.org:443 HTTP/1.1" 200 10998 "-" "-"
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated @babel/polyfill@7.12.1: 🚨 This package has been deprecated in favor of separate inclusion of a polyfill and regenerator-runtime (when needed). See the @babel/polyfill docs (https://babeljs.io/docs/en/babel-polyfill) for more information.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated core-js@2.6.12: core-js@<3.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
npm notice 
npm notice New minor version of npm available! 7.7.6 -> 7.12.1
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v7.12.1>
npm notice Run `npm install -g npm@7.12.1` to update!
npm notice 
npm ERR! code ECONNRESET
npm ERR! errno ECONNRESET
npm ERR! network request to https://registry.npmjs.org/which-module/-/which-module-2.0.0.tgz failed, reason: socket hang up
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-05-11T16_34_40_324Z-debug.log
Failed to run 'override-build': Exit code was 1.
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 258, in run
    self.build()

However this will complete after X tries, based on trials and errors the past weeks.

Build duration is only between 7 - 25 minutes, depending of the architecture.

probably https://bugs.launchpad.net/bugs/1886861 ?

there is a workaround:

Hi @ogra,

Thanks for this. I’m not using electron. The first is using the python plugin and the other npm.

But I guess I can just skip the first env.variable, and use it like this for npm:

if [ -n "$http_proxy" ]; then
    export GLOBAL_AGENT_HTTP_PROXY="${http_proxy}"
    export GLOBAL_AGENT_HTTPS_PROXY="${http_proxy}"
fi

But what can I use for the python plugin?

since this is npm specific behaviour i dont think you need anything for the python plugin …

Here’s the full build log of «home-assistant-snap» (python) here: https://launchpadlibrarian.net/538313784/buildlog_snap_ubuntu_focal_ppc64el_snapcraft-home-assistant-snap-496937_BUILDING.txt.gz

    ERROR: Command errored out with exit status 1:
     command: /build/snapcraft-home-assistant-snap-496937/parts/homeassistant/install/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xhq54ey9/freesms/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xhq54ey9/freesms/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-8z_dq6ts
         cwd: /tmp/pip-install-xhq54ey9/freesms/
    Complete output (24 lines):
    running egg_info
    creating /tmp/pip-pip-egg-info-8z_dq6ts/freesms.egg-info
    writing /tmp/pip-pip-egg-info-8z_dq6ts/freesms.egg-info/PKG-INFO
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-xhq54ey9/freesms/setup.py", line 18, in <module>
        setup(
      File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/build/snapcraft-home-assistant-snap-496937/parts/homeassistant/install/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 292, in run
        writer(self, ep.name, os.path.join(self.egg_info, ep.name))
      File "/build/snapcraft-home-assistant-snap-496937/parts/homeassistant/install/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 628, in write_pkg_info
        metadata.write_pkg_info(cmd.egg_info)
      File "/usr/lib/python3.8/distutils/dist.py", line 1117, in write_pkg_info
        self.write_pkg_file(pkg_info)
      File "/build/snapcraft-home-assistant-snap-496937/parts/homeassistant/install/lib/python3.8/site-packages/setuptools/dist.py", line 172, in write_pkg_file
        license = rfc822_escape(self.get_license())
      File "/usr/lib/python3.8/distutils/util.py", line 475, in rfc822_escape
        lines = header.split('\n')
    TypeError: a bytes-like object is required, not 'str'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 20.2.4; however, version 21.1.1 is available.
You should consider upgrading via the '/build/snapcraft-home-assistant-snap-496937/parts/homeassistant/install/bin/python3 -m pip install --upgrade pip' command.
[12/May/2021:09:15:04 +0000] "CONNECT pypi.org:443 HTTP/1.0" 200 2132036 "-" "-"
[12/May/2021:09:15:04 +0000] "CONNECT files.pythonhosted.org:443 HTTP/1.0" 200 100299510 "-" "-"
Failed to build 'homeassistant'.

Recommended resolution:
Check the build logs and ensure the part's configuration and sources are correct.
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 258, in run
    self.build()
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 247, in build
    env=env)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 100, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py", line 537, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'exec', 'lp-focal-ppc64el', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-63563508", "build-request-timestamp": "2021-05-12T08:54:37Z", "build_url": "https://launchpad.net/~giaever-online/+snap/snapcraft-home-assistant-snap-496937/+build/1406316"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/snapcraft-home-assistant-snap-496937 && linux64 snapcraft']' returned non-zero exit status 2.
Revoking proxy token...

This is where it fails for every architecture. It hasn’t been an update to the package «freesms» since 2017, and I’m not sure how to interpret the error; is something missing in the header-variable that is split? Is this holding environment data, that is missing?

I tried to roll back to a previous version that was previously (remote) built without any errors, and this also fails now…

Any version builds fine on my local machines.

well, you get a HTTP 200 there … that does not look like a connection problem to me …

Note this is two different kind of snap packages

  1. home-assistant-snap (built with python)
  2. zwavejs2mqtt (built with npm)

Yeah, but I really don’t understand why this would successfully build locally (with multipass) and that an untouched git repository now fails to remote build when it built fine just a short week ago.

Hi,

Someone wrote in an issue on github as the snap-package home-assistant-snap haven’t been updated reasonly:

I also had some snap-store connections issues yesterday. The culprit is most likely that yesterday, they finally resolved the Ubuntu 21.04 ship-blocker and publicly released it. The 21.04 downloads apparently affected the snap store availability.

Could it be that there’s been changes to requirements etc that makes the build process to fail suddenly under Ubuntu 21.04? Anyone know how I can have a local build env close to the remote build to debug this issue or a tip on how I can start debug why it fails on remote?

there is no “build process under 21.04” at all … your snaps are only building on LTS releases that match the base: in your snapcraft.yaml so for base: core this is 16.04, for core18 it is 18.04 and for core20 it is 20.04 …

the remote builds on LP are using an lxd container to build the snaps, this container is in turn hosted behind a proxy …

the general setup is similar to (ignore the Ubuntu Core part):