"ModuleNotFoundError: No module named 'distutils.util'" when trying to build snaps on Launchpad (but works locally)

snapcraft
1

I’m trying to build a new version of my app (snapcraft.yaml file here). It works locally (amd64), but when I push the code to Launchpad and it triggers automatic builds, it fails on all the archs (amd64, armhf, arm64). Requesting manual builds fails equally.

On my laptop (running 18.04), I’m using the latest snapcraft (3.9.8) and multipass (1.0.2) from stable.

The failed build logs all show the same error:

Building qabro 
Traceback (most recent call last):
  File "/build/qabro/stage/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/build/qabro/stage/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/build/qabro/parts/qabro/install/lib/python3.6/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal.cli.main import main as _main  # isort:skip # noqa
  File "/build/qabro/parts/qabro/install/lib/python3.6/site-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/build/qabro/parts/qabro/install/lib/python3.6/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/build/qabro/parts/qabro/install/lib/python3.6/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/build/qabro/parts/qabro/install/lib/python3.6/site-packages/pip/_internal/cli/cmdoptions.py", line 19, in <module>
    from distutils.util import strtobool
ModuleNotFoundError: No module named 'distutils.util'
Failed to run '/build/qabro/stage/usr/bin/python3 -m pip wheel --no-index --find-links /build/qabro/parts/qabro/python-packages --wheel-dir /tmp/tmpbvm2ybaf launchpadlib': Exited with code 1.
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 266, in run
    self.build()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 255, in build
    env=env)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 102, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 536, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-bionic-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build_url": "https://launchpad.net/~pieq/+snap/qabro/+build/831264"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/qabro && linux64 snapcraft']' returned non-zero exit status 2

I really don’t understand what’s going on… How come it says No module named 'distutils.util when, a few lines above in the build log, I can see:

Get:1 python3-distutils_3.6.9-1~18.04_all.deb [144 kB]

I tried to add python3-distutils to the build-packages list in my snapcraft.yaml file for the qabro part, but it fails with the same results…

Does anyone know what’s going on?

Thanks!

2

@sergiusens or @cjp256, it looks like the launchpad version of Snapcraft is not copying python3-setuptools’ dependency of python3-distutils into $SNAPCRAFT_PART_INSTALL during building of a python project. This doesn’t fail in multipass, or LXD, only the launchpad system (I’ve verified each of the three scenarios).

With distutils missing, some pip installations will fail. Installing python3-distutils in the host as a build-package does not help, and this is done by snapcraft anyway when it installs python3-setuptools because setuptools has a dependency on distutils.

This is with core18 as the base snap. I’ve not yet tested core.

3

It looks like the problem does not occur on launchpad when using core as the base snap. So this is specific to core18 builds on launchpad.

4

hmmm, strange… can you try edge?
If this fails we might be having a bootstrap problem, but I have not recieved any recipe build failures from snapcraft itself, which is written in python, when building with snapcraft from stable using core18

5

Hi, I just tried building from Launchpad using snapcraft edge, and it fails with the same error.

6

I tested Launchpad builds with snapcraft 3.8, edge, and stable. All failed, so it doesn’t seem to be a (recent) regression as far as snapcraft is concerned…?

Typically in the past, the biggest difference that I’ve noticed between the build service and local builds is that Launchpad’s images don’t automatically install “Recommended packages”. So I fiddled for a bit to bring those closer to parity but didn’t have any luck there (or I missed something).

I’m going to see if I can better reproduce Launchpad’s environment to debug further.

7

this reproduces with local snapcraft easily enough:

snapcraft pull
snapcraft build

Something is breaking when the two phases are not in the same run…

8

I think I sorted this out a little bit.

With this project, there are ‘after’ dependencies for sosreport and patches.

This causes is that sosreport and patches to both build to completion before building qabro part.

This means that the stage step gets run for those parts resulting in some python bits being staged into the /root/stage tree.

The PYTHON_HOME path that snapcraft uses is cached [1]. When it is not cached, it does a query and prefers the stage directory over install directory if python is available [2]. In your normal local instance running snapcraft, I don’t think it ever in fact uses the stage directory. But in the Launchpad case, where the steps are split up in discrete commands, it will in fact use the stage installation for the second build step, which happens to be not as featured as that found in the install directory.

Specifically, when the steps are split up, the cache behavior is affected:

  • when the snapcraft pull step runs, it uses the install directory (/root/parts/qabro/install/usr) for PYTHON_HOME.
  • when the snapcraft build step runs, it pivots to the stage directory (/root/stage/usr) which does not in fact, contain the distutils.util at this time.

I don’t have 100% confidence on what I just stated, but figured I’d post an update and see if anyone has any thoughts before I get back to it tomorrow :smiley:

[1] https://github.com/snapcore/snapcraft/blob/master/snapcraft/plugins/_python/_pip.py#L154 [2] https://github.com/snapcore/snapcraft/blob/master/snapcraft/plugins/_python/_python_finder.py#L81

9

@cjp256 @sergiusens I went ahead and created an issue in Launchpad to track this. I also mentioned the workaround provided by cjp256 on IRC to make it work (adding python3-distutils to sosreport’s stage-packages section).

1 Like