Revert snapcraft to older version?

Our CI pipeline that would build our custom snap broke with a recent release of snapcraft 4.6.2 (never tested 4.6.1, but 4.6 works). Until I can figure out what happened and (update it), is there a way to temporarily install snapcraft by revision number (all channels now point to 4.6.2)?

I tried installing snapcraft from apt (hoping to get a different version), but turns out it points to the store…

Alternatively, I can try to bulid snapcraft myself from github’s 4.6 release. Once installed from a .snap file, I hope automatic updates won’t break our CI. Any chance this is feasable?

Sorry about the inconvenience - ping @sergiusens / @cjp256.

If you build an older release of snapcraft as a snap and install using the --dangerous tag, it will not get updated by any kind of refresh. You’re effectively “pinning” the version to whatever you build and install. As a short term solution, this would work. Standard disclaimer that you won’t get security updates (or indeed broken updates :wink: ) if you do.

This should be enough…

$ git clone https://github.com/snapcore/snapcraft
$ cd snapcraft
$ git checkout "4.6"
$ snapcraft --use-lxd
$ snap install snapcraft*.snap --dangerous

You could snapcraft remote-build in place of snapcraft --use-lxd to save yourself some compile time, but it doesn’t take a huge while (less than 10 mins on my 5 year old ThinkPad) to build.

2 Likes

Trying now, thanks!
I want our CI to be stable (and not break on us), pinning it this way and periodically updating snapcraft ourselves may well be way to go.

1 Like

The remote-build option did not play nice.

...
Build status as of 2021-04-16 17:36:10.060642:
	arch=arm64	state=Currently building
Build status as of 2021-04-16 17:37:11.176946:
	arch=arm64	state=Failed to build
Snap file not available for arch 'arm64'.
Build log available at 'snapcraft_arm64.txt'
Build failed for arch 'arm64'.
Build complete.

When I try to build from inside a Server20.04 lxd container (using snapcraft snap --destructive-mode), I get:

Failed to fetch stage packages: Error downloading packages for part 'snapcraft-libs': The package 'libffi6' was not found..

I’m trying with --use-lxd now…

EDIT: I am using a UbuntuServer 20.04 container (not 18.04) which may be why libffi6 isn’t found.

You should be using an 18.04 container, ideally just “–use-lxd”

Can you provide a bit more information around the error, is it around stage-packages or python packages missing through the use of stage-pacakges?

Can you provide a bit more information around the error

For the remote-build (trying to build snapcraft 2.6)? I can provide with the build log.

For our custom snap (that used to work)? I think it’s a linking problem, logs are filled with undefined symbols errors. There is also a swarm of warnings like:

Copying needed symlink target '/lib/aarch64-linux-gnu/libselinux.so.1' from host to satisfy '/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libselinux.so'.

which I find suspicious.

Sorry if I was unclear, I want to know what the error is when using Snapcraft 4.6.2

Got it, here is a selected snippet from the failed build logs.
The stage was BUILD (and it what it does is compiles/links some C code).

/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(gzlibdecompressor.c.o): in function `g_zlib_decompressor_reset':
(.text+0x338): undefined reference to `inflateReset'
[6662] 2)/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(gzlibdecompressor.c.o): in function `g_zlib_decompressor_convert':
[6663] 3)(.text+0x3e4): undefined reference to `inflate'
[6664] 4)/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(gzlibdecompressor.c.o): in function `g_zlib_decompressor_constructed':
[6665] 5)(.text+0x838): undefined reference to `inflateInit_'
[6666] 6)/root/parts/tis/install/usr/bin/ld: (.text+0x86c): undefined reference to `inflateInit2_'
[6667] 7)/root/parts/tis/install/usr/bin/ld: (.text+0x8d0): undefined reference to `inflateInit2_'
[6668] 8)/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(gzlibdecompressor.c.o): in function `g_zlib_decompressor_finalize':
[6669] 9)(.text+0x900): undefined reference to `inflateEnd'
[6670] 70)/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(glocalfileinfo.c.o): in function `set_selinux_context':
[6671] 71)(.text+0x988): undefined reference to `is_selinux_enabled'
[6672] 72)/root/parts/tis/install/usr/bin/ld: (.text+0x9a8): undefined reference to `setfilecon_raw'
[6673] 73)/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(glocalfileinfo.c.o): in function `_g_local_file_info_get':
[6674] 74)(.text+0x2a7c): undefined reference to `is_selinux_enabled'
[6675] 75)/root/parts/tis/install/usr/bin/ld: (.text+0x2a90): undefined reference to `lgetfilecon_raw'
[6676] 76)/root/parts/tis/install/usr/bin/ld: (.text+0x2ab4): undefined reference to `freecon'
[6677] 77)/root/parts/tis/install/usr/bin/ld: (.text+0x3098): undefined reference to `getfilecon_raw'

It’s currently re-building now with my new snapcraft_2.6.snap installed. I’ll update here if it succeeds. I can provide with a minimal snapcraft.yaml once I finish getting our CI back up.

EDIT: It works with snapcraft 2.6. Do let me know if you want a minimal snapcraft.yaml to reproduce the problem.

I sent this as a message to the two gentlemen responsible for this PR: https://github.com/snapcore/snapcraft/pull/3497/files#diff-0a723460ca166a14406f4f36ac1e0a6857ce9b47a2961f99aac99ad481e72a5d. I am posting here for others involved.

We have an issue building our snap on our pipeline which previously worked on snapcraft version 4.6. We are building a library called TIS using cmake and we are getting the following symlink warnings:

Copying needed symlink target ‘/lib/aarch64-linux-gnu/libblkid.so.1.1.0’ from host to satisfy ‘/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libblkid.so’.

6522Copying needed symlink target ‘/lib/aarch64-linux-gnu/libcrypt.so.1.1.0’ from host to satisfy ‘/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libcrypt.so’.

6523Copying needed symlink target ‘/lib/aarch64-linux-gnu/libpcre.so.3.13.3’ from host to satisfy ‘/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libpcre.so’.

6524Copying needed symlink target ‘/lib/aarch64-linux-gnu/libsepol.so.1’ from host to satisfy ‘/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libsepol.so’.

6525Copying needed symlink target ‘/lib/aarch64-linux-gnu/libuuid.so.1.3.0’ from host to satisfy ‘/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libuuid.so’.

6526Copying needed symlink target ‘/lib/aarch64-linux-gnu/libselinux.so.1’ from host to satisfy ‘/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libselinux.so’.

6527Copying needed symlink target ‘/lib/aarch64-linux-gnu/libmount.so.1.1.0’ from host to satisfy ‘/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libmount.so’.

6528Copying needed symlink target ‘/lib/aarch64-linux-gnu/libz.so.1.2.11’ from host to satisfy ‘/root/parts/tis/install/usr/lib/aarch64-linux-gnu/libz.so’.

As a result when we build using cmake we are getting a number of linker warnings:

/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(gzlibdecompressor.c.o): in function `g_zlib_decompressor_set_gzheader’:

6659(.text+0x254): undefined reference to `inflateGetHeader’

6660/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(gzlibdecompressor.c.o): in function `g_zlib_decompressor_reset’:

6661(.text+0x338): undefined reference to `inflateReset’

6662/root/parts/tis/install/usr/bin/ld: /root/parts/tis/install/usr/lib/aarch64-linux-gnu/libgio-2.0.a(gzlibdecompressor.c.o): in function `g_zlib_decompressor_convert’:

6663(.text+0x3e4): undefined reference to `inflate’

For reference here is our snapcraft.yaml:

We are on a deadline to get our snap building properly again so any insight would a massive help.

That would be great, yes

the the build environment an actual Ubuntu 20.04 image with no additional repositories added?

here is a snapcraft.yaml that should work with 4.6 and fail with 4.62:

snapcraft.yaml
---
title: IRIS by Incuvers
name: iris-incuvers
version: "1.0"
summary: Incuvers Real-time Imaging System
description: |
  Accompanying software application for IRIS (Incuvers Real-time Imaging System);
  a personalized, IoT cell incubator with imaging capabilities.
grade: devel
architectures:
  - build-on: arm64
    run-on: arm64
base: core20
confinement: devmode

apps:
  daemon:
    command: echo "hi mom"
    daemon: oneshot

parts:
  tis:
    source-type: git
    source: https://github.com/TheImagingSource/tiscamera
    source-tag: v-tiscamera-0.13.1
    plugin: cmake
    cmake-parameters: [-DBUILD_GST_1_0=ON, -DBUILD_TOOLS=ON, -DBUILD_V4L2=ON, -DBUILD_LIBUSB=ON]
    build-packages:
      - g++
      - pkg-config
      - uuid-dev
      - libudev-dev
      - libgstreamer-plugins-base1.0-dev
      - libglib2.0-dev
      - libusb-1.0-0-dev
      - libzip-dev
      - python3-setuptools  # may conflict with python3-wheel (bdist_wheel)
      - libgl1
      - libglvnd0
      - libgirepository1.0-dev
    stage-packages:
      - gstreamer1.0-tools
      - libusb-1.0-0
      - libgstreamer1.0-0
      - libgstreamer1.0-dev
      - libgstreamer-plugins-base1.0-0
      - libgirepository1.0-dev
      - gstreamer1.0-plugins-good
      - gstreamer1.0-plugins-bad
      - liborc-0.4-0
      - libpcre2-8-0
      - udev
      - uvcdynctrl
      - libgpm2
      - libslang2
      - freeglut3
      - gir1.2-gstreamer-1.0
      - gir1.2-gst-plugins-base-1.0
      - gir1.2-gtk-3.0
    organize:
      gst-inspect-1.0: bin/gst-inspect-1.0
      $SNAPCRAFT_PART_SRC/data/uvc-extensions/usb3.xml: ./usr/share/uvcdynctrl/data/199e/usb3.xml

we are building the snap in an lxd container on a rasberry pi running 20.04. We are also cleaning between builds so the packages from the previous successful build and this one have not changed. The only change was the snapcraft version which auto refreshed on us.

I have published Snapcraft 4.6 to stable/4.6 to unblock you, snap install snapcraft --classic --channel stable/4.6

2 Likes

this was supposed to be posted on Friday :man_shrugging:

The source of the problem is actually https://github.com/snapcore/snapcraft/commit/0bf7a2e6619b0037a50caeb49d28788c021d0921

The filtering for stage-packages has been fixed to not include libraries from the base that have an architecture qualifier.

For this specific case, you have -dev packages in stage-packages, his ends up bringing ffi-dev but not libffi (as it is part of the base), which leaves dangling symlink (the .so) and causes all the linking errors.

The simple solution is to remove -dev packages from stage-packages and leave them in build-packages, if you really want to include a package that is part of the base, explicitly mentioning it in stage-packages will bring it in.

We are going to be working on a fix that allows for a progressive change for this next week without altering existing snapcraft.yaml’s.

1 Like

I just tested removing the *-dev packages from stage-packages and the part now compiles using snapcraft 4.6.2. I’ve yet to determine if it affects the functionality of the full snap, but it appears that your filtering hypothesis is correct.

1 Like