Hi,
Sorry for the vague title. I have a snapcraft build that’s failing, but the annoying thing is it’s only failing in our CI env, and I’m not exactly sure why.
It works locally using snpacraft and multipass. The CI system uses docker, and we have a snapcraft docker image which generally works fine. The really annoying thing is, if I run the docker image locally and build the snap, it also works fine !
Even more annoyingly, if I strace the command, snapcraft seems to hang in the build env, and I can never find out why. CI output logs are restricted in length, and if I write strace output to a log file I never get to see it as the CI fails and I can’t inspect it. Yet again, even with strace it works fine locally in docker. I suspect some kind of TTY issue there but I can’t be sure.
Anyway, the actual snapcraft build error I’m trying to dig into.
This is the part that’s failing:
parts:
journalbeat:
plugin: nil
stage-packages:
- journalbeat=7.12.0
stage:
- '-etc/journalbeat/journalbeat.yml'
- '-etc/init.d/journalbeat'
I’m not sure if it’s relevant, but I’ll mention it just in case. I’m also using the package repos support. I have a section like this:
package-repositories:
- type: apt
url: https://artifacts.elastic.co/packages/7.x/apt
components: [main]
suites: [stable]
key-id: 46095ACC8548582C1A2699A9D27D666CD88E42B4
…and I’m running snapcraft with --enable-experimental-package-repositories
Here is the full command:
SNAPCRAFT_ENABLE_DEVELOPER_DEBUG=yes snapcraft --enable-experimental-package-repositories --debug
Finally, here is the error I’m getting:
Preparing to pull journalbeat
Requested stage-packages: ['journalbeat=7.12.0']
snapcraft is running as a snap True, SNAP_NAME set to 'snapcraft'
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:4 https://artifacts.elastic.co/packages/7.x/apt stable InRelease [10.4 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [29.6 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [950 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [271 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1188 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4305 B]
Get:15 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 Packages [66.1 kB]
Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [21.6 kB]
Get:17 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [773 kB]
Get:18 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [239 kB]
Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [683 kB]
Fetched 17.7 MB in 0s (0 B/s)
Marking 'journalbeat=7.12.0' (and its dependencies) to be fetched
package: <Package: name:'journalbeat' architecture='amd64' id:91976>
Skipping filtered manifest packages: ['adduser', 'apparmor', 'apt', 'base-files', 'base-passwd', 'bash', 'bash-completion', 'bsdutils', 'bzip2', 'ca-certificates', 'cloud-guest-utils', 'cloud-init', 'console-conf', 'coreutils', 'cryptsetup', 'cryptsetup-bin', 'dash', 'dbus', 'dbus-user-session', 'debconf', 'debianutils', 'diffutils', 'distro-info-data', 'dmsetup', 'dosfstools', 'dpkg', 'e2fsprogs', 'fdisk', 'finalrd', 'findutils', 'gcc-10-base', 'gdbserver', 'gpgv', 'grep', 'gzip', 'hostname', 'init-system-helpers', 'iproute2', 'iptables', 'iputils-ping', 'isc-dhcp-client', 'kmod', 'less', 'libacl1', 'libapparmor1', 'libapt-pkg6.0', 'libargon2-1', 'libattr1', 'libaudit-common', 'libaudit1', 'libblkid1', 'libbsd0', 'libbz2-1.0', 'libc-bin', 'libc6', 'libcap-ng0', 'libcap2', 'libcap2-bin', 'libcbor0.6', 'libcom-err2', 'libcrypt1', 'libcryptsetup12', 'libdb5.3', 'libdbus-1-3', 'libdebconfclient0', 'libdevmapper1.02.1', 'libdns-export1109', 'libedit2', 'libelf1', 'libexpat1', 'libext2fs2', 'libfdisk1', 'libffi7', 'libfido2-1', 'libgcc-s1', 'libgcrypt20', 'libglib2.0-0', 'libgmp10', 'libgnutls30', 'libgpg-error0', 'libgssapi-krb5-2', 'libhogweed5', 'libidn2-0', 'libip4tc2', 'libip6tc2', 'libisc-export1105', 'libjson-c4', 'libk5crypto3', 'libkeyutils1', 'libkmod2', 'libkrb5-3', 'libkrb5support0', 'liblz4-1', 'liblzma5', 'liblzo2-2', 'libmnl0', 'libmount1', 'libmpdec2', 'libncurses6', 'libncursesw6', 'libnetfilter-conntrack3', 'libnetplan0', 'libnettle7', 'libnfnetlink0', 'libnftnl11', 'libnl-3-200', 'libnl-genl-3-200', 'libnl-route-3-200', 'libnss-extrausers', 'libp11-kit0', 'libpam-modules', 'libpam-modules-bin', 'libpam-runtime', 'libpam-systemd', 'libpam0g', 'libpcre2-8-0', 'libpcre3', 'libpcsclite1', 'libpopt0', 'libprocps8', 'libpython3-stdlib', 'libpython3.8-minimal', 'libpython3.8-stdlib', 'libreadline8', 'libseccomp2', 'libselinux1', 'libsemanage-common', 'libsemanage1', 'libsepol1', 'libsmartcols1', 'libsqlite3-0', 'libss2', 'libssl1.1', 'libstdc++6', 'libsystemd0', 'libtasn1-6', 'libtinfo6', 'libudev1', 'libunistring2', 'libuuid1', 'libwrap0', 'libxtables12', 'libyaml-0-2', 'libzstd1', 'login', 'logsave', 'lsb-base', 'mawk', 'mime-support', 'mount', 'ncurses-base', 'ncurses-bin', 'netbase', 'netcat-openbsd', 'netplan.io', 'openssh-client', 'openssh-server', 'openssh-sftp-server', 'openssl', 'passwd', 'perl-base', 'probert-common', 'probert-network', 'procps', 'python3', 'python3-attr', 'python3-blinker', 'python3-certifi', 'python3-cffi-backend', 'python3-chardet', 'python3-configobj', 'python3-cryptography', 'python3-distutils', 'python3-idna', 'python3-importlib-metadata', 'python3-jinja2', 'python3-json-pointer', 'python3-jsonpatch', 'python3-jsonschema', 'python3-jwt', 'python3-lib2to3', 'python3-markupsafe', 'python3-minimal', 'python3-more-itertools', 'python3-netifaces', 'python3-oauthlib', 'python3-pkg-resources', 'python3-pyrsistent', 'python3-pyudev', 'python3-requests', 'python3-requests-unixsocket', 'python3-serial', 'python3-setuptools', 'python3-six', 'python3-urllib3', 'python3-urwid', 'python3-yaml', 'python3-zipp', 'python3.8', 'python3.8-minimal', 'readline-common', 'rfkill', 'sbsigntool', 'secureboot-db', 'sed', 'sensible-utils', 'squashfs-tools', 'subiquitycore', 'sudo', 'systemd', 'systemd-sysv', 'systemd-timesyncd', 'sysvinit-utils', 'tar', 'tzdata', 'ubuntu-keyring', 'ucf', 'udev', 'util-linux', 'vim-common', 'vim-tiny', 'wpasupplicant', 'xxd', 'zlib1g']
Get:1 journalbeat-7.12.0-amd64.deb [24.9 MB]
Fetched 24.9 MB in 0s (0 B/s)
Extracting stage package: journalbeat
/usr/bin/bash: line 159: 313 Killed SNAPCRAFT_ENABLE_DEVELOPER_DEBUG=yes snapcraft --enable-experimental-package-repositories --debug
It’s certainly something about that package. If I replace that package with other random packages names it works fine.
I found I could work around the issue with some manual overrides instead:
parts:
journalbeat:
plugin: nil
override-pull: |
snapcraftctl pull
apt-get download journalbeat=7.12.0
override-build: |
snapcraftctl build
echo BUILD journalbeat
dpkg -x ${SNAPCRAFT_PART_BUILD}/journalbeat*.deb ${SNAPCRAFT_PART_INSTALL}
One strange thing I’ve noticed, the package filename from the repo is not correct. It’s journalbeat_7.12.0_amd64.deb , but I beleive the underscore there is wrong, and it should be journalbeat-7.12.0_amd64.deb.
But, that doesn’t stop the build working locally.
I will use the workaround above, but I’m curious and want to know what the cause of the issue is in case it bites in some other way in the future.
Any ideas why I don’t get a python stack trace, and how I could dig into the cause a bit more?
Cheers,
Just
