Strange snapcraft build error


Sorry for the vague title. I have a snapcraft build that’s failing, but the annoying thing is it’s only failing in our CI env, and I’m not exactly sure why.

It works locally using snpacraft and multipass. The CI system uses docker, and we have a snapcraft docker image which generally works fine. The really annoying thing is, if I run the docker image locally and build the snap, it also works fine !

Even more annoyingly, if I strace the command, snapcraft seems to hang in the build env, and I can never find out why. CI output logs are restricted in length, and if I write strace output to a log file I never get to see it as the CI fails and I can’t inspect it. Yet again, even with strace it works fine locally in docker. I suspect some kind of TTY issue there but I can’t be sure.

Anyway, the actual snapcraft build error I’m trying to dig into.

This is the part that’s failing:

    plugin: nil
      - journalbeat=7.12.0
      - '-etc/journalbeat/journalbeat.yml'
      - '-etc/init.d/journalbeat'

I’m not sure if it’s relevant, but I’ll mention it just in case. I’m also using the package repos support. I have a section like this:

  - type: apt
    components: [main]
    suites: [stable]
    key-id: 46095ACC8548582C1A2699A9D27D666CD88E42B4

…and I’m running snapcraft with --enable-experimental-package-repositories

Here is the full command:

SNAPCRAFT_ENABLE_DEVELOPER_DEBUG=yes snapcraft --enable-experimental-package-repositories --debug

Finally, here is the error I’m getting:

Preparing to pull journalbeat 
Requested stage-packages: ['journalbeat=7.12.0']
snapcraft is running as a snap True, SNAP_NAME set to 'snapcraft'
Get:1 focal InRelease [265 kB]
Get:2 focal-updates InRelease [114 kB]
Get:3 focal-backports InRelease [101 kB]
Get:4 stable InRelease [10.4 kB]
Get:5 focal-security InRelease [109 kB]
Get:6 focal/multiverse amd64 Packages [177 kB]
Get:7 focal/restricted amd64 Packages [33.4 kB]
Get:8 focal/main amd64 Packages [1275 kB]
Get:9 focal/universe amd64 Packages [11.3 MB]
Get:10 focal-updates/multiverse amd64 Packages [29.6 kB]
Get:11 focal-updates/universe amd64 Packages [950 kB]
Get:12 focal-updates/restricted amd64 Packages [271 kB]
Get:13 focal-updates/main amd64 Packages [1188 kB]
Get:14 focal-backports/universe amd64 Packages [4305 B]
Get:15 stable/main amd64 Packages [66.1 kB]
Get:16 focal-security/multiverse amd64 Packages [21.6 kB]
Get:17 focal-security/main amd64 Packages [773 kB]
Get:18 focal-security/restricted amd64 Packages [239 kB]
Get:19 focal-security/universe amd64 Packages [683 kB]
Fetched 17.7 MB in 0s (0 B/s)
Marking 'journalbeat=7.12.0' (and its dependencies) to be fetched
package: <Package: name:'journalbeat' architecture='amd64' id:91976>
Skipping filtered manifest packages: ['adduser', 'apparmor', 'apt', 'base-files', 'base-passwd', 'bash', 'bash-completion', 'bsdutils', 'bzip2', 'ca-certificates', 'cloud-guest-utils', 'cloud-init', 'console-conf', 'coreutils', 'cryptsetup', 'cryptsetup-bin', 'dash', 'dbus', 'dbus-user-session', 'debconf', 'debianutils', 'diffutils', 'distro-info-data', 'dmsetup', 'dosfstools', 'dpkg', 'e2fsprogs', 'fdisk', 'finalrd', 'findutils', 'gcc-10-base', 'gdbserver', 'gpgv', 'grep', 'gzip', 'hostname', 'init-system-helpers', 'iproute2', 'iptables', 'iputils-ping', 'isc-dhcp-client', 'kmod', 'less', 'libacl1', 'libapparmor1', 'libapt-pkg6.0', 'libargon2-1', 'libattr1', 'libaudit-common', 'libaudit1', 'libblkid1', 'libbsd0', 'libbz2-1.0', 'libc-bin', 'libc6', 'libcap-ng0', 'libcap2', 'libcap2-bin', 'libcbor0.6', 'libcom-err2', 'libcrypt1', 'libcryptsetup12', 'libdb5.3', 'libdbus-1-3', 'libdebconfclient0', 'libdevmapper1.02.1', 'libdns-export1109', 'libedit2', 'libelf1', 'libexpat1', 'libext2fs2', 'libfdisk1', 'libffi7', 'libfido2-1', 'libgcc-s1', 'libgcrypt20', 'libglib2.0-0', 'libgmp10', 'libgnutls30', 'libgpg-error0', 'libgssapi-krb5-2', 'libhogweed5', 'libidn2-0', 'libip4tc2', 'libip6tc2', 'libisc-export1105', 'libjson-c4', 'libk5crypto3', 'libkeyutils1', 'libkmod2', 'libkrb5-3', 'libkrb5support0', 'liblz4-1', 'liblzma5', 'liblzo2-2', 'libmnl0', 'libmount1', 'libmpdec2', 'libncurses6', 'libncursesw6', 'libnetfilter-conntrack3', 'libnetplan0', 'libnettle7', 'libnfnetlink0', 'libnftnl11', 'libnl-3-200', 'libnl-genl-3-200', 'libnl-route-3-200', 'libnss-extrausers', 'libp11-kit0', 'libpam-modules', 'libpam-modules-bin', 'libpam-runtime', 'libpam-systemd', 'libpam0g', 'libpcre2-8-0', 'libpcre3', 'libpcsclite1', 'libpopt0', 'libprocps8', 'libpython3-stdlib', 'libpython3.8-minimal', 'libpython3.8-stdlib', 'libreadline8', 'libseccomp2', 'libselinux1', 'libsemanage-common', 'libsemanage1', 'libsepol1', 'libsmartcols1', 'libsqlite3-0', 'libss2', 'libssl1.1', 'libstdc++6', 'libsystemd0', 'libtasn1-6', 'libtinfo6', 'libudev1', 'libunistring2', 'libuuid1', 'libwrap0', 'libxtables12', 'libyaml-0-2', 'libzstd1', 'login', 'logsave', 'lsb-base', 'mawk', 'mime-support', 'mount', 'ncurses-base', 'ncurses-bin', 'netbase', 'netcat-openbsd', '', 'openssh-client', 'openssh-server', 'openssh-sftp-server', 'openssl', 'passwd', 'perl-base', 'probert-common', 'probert-network', 'procps', 'python3', 'python3-attr', 'python3-blinker', 'python3-certifi', 'python3-cffi-backend', 'python3-chardet', 'python3-configobj', 'python3-cryptography', 'python3-distutils', 'python3-idna', 'python3-importlib-metadata', 'python3-jinja2', 'python3-json-pointer', 'python3-jsonpatch', 'python3-jsonschema', 'python3-jwt', 'python3-lib2to3', 'python3-markupsafe', 'python3-minimal', 'python3-more-itertools', 'python3-netifaces', 'python3-oauthlib', 'python3-pkg-resources', 'python3-pyrsistent', 'python3-pyudev', 'python3-requests', 'python3-requests-unixsocket', 'python3-serial', 'python3-setuptools', 'python3-six', 'python3-urllib3', 'python3-urwid', 'python3-yaml', 'python3-zipp', 'python3.8', 'python3.8-minimal', 'readline-common', 'rfkill', 'sbsigntool', 'secureboot-db', 'sed', 'sensible-utils', 'squashfs-tools', 'subiquitycore', 'sudo', 'systemd', 'systemd-sysv', 'systemd-timesyncd', 'sysvinit-utils', 'tar', 'tzdata', 'ubuntu-keyring', 'ucf', 'udev', 'util-linux', 'vim-common', 'vim-tiny', 'wpasupplicant', 'xxd', 'zlib1g']
Get:1 journalbeat-7.12.0-amd64.deb [24.9 MB]                                    
Fetched 24.9 MB in 0s (0 B/s)                                                   
Extracting stage package: journalbeat
/usr/bin/bash: line 159:   313 Killed                  SNAPCRAFT_ENABLE_DEVELOPER_DEBUG=yes snapcraft --enable-experimental-package-repositories --debug

It’s certainly something about that package. If I replace that package with other random packages names it works fine.

I found I could work around the issue with some manual overrides instead:

    plugin: nil
    override-pull: |
      snapcraftctl pull
      apt-get download journalbeat=7.12.0
    override-build: |
      snapcraftctl build
      echo BUILD journalbeat
      dpkg -x ${SNAPCRAFT_PART_BUILD}/journalbeat*.deb ${SNAPCRAFT_PART_INSTALL}

One strange thing I’ve noticed, the package filename from the repo is not correct. It’s journalbeat_7.12.0_amd64.deb , but I beleive the underscore there is wrong, and it should be journalbeat-7.12.0_amd64.deb.

But, that doesn’t stop the build working locally.

I will use the workaround above, but I’m curious and want to know what the cause of the issue is in case it bites in some other way in the future.
Any ideas why I don’t get a python stack trace, and how I could dig into the cause a bit more?


The output indicates that something “killed” snapcraft during the build. This might be the system’s Out of Memory daemon that kicked in and killed your process. Alternatively some mechanism on the CI server decides that you’ve used enough time or resources, perhaps?

I am following that angle up with the people that administer the CI system. However, I think it’s unlikely, as simply changing the package to something else fixes the issue.


Perhaps try to bump memory in the Multipass VM:

Alternatively consider LXD if using Linux for CI:
snapcraft ... --use-lxd

The CI uses docker :slight_smile:

OK, I got the CI admins to check the host logs for me in the end, and to my surprise it was indeed being OOM killed.

It’s strange that a different package name fixed the issue. Memory usage just must have been right on the cusp !



Did you get any proper solution?, I am also facing the same issue!