Snapcraft is suddenly unstable, cannot identify dependencies and has unexpected buffer overflows

Are you also using the GNOME extension or is this something else?

I think I found the problem. The GNOME sdk snap includes libc6, the extension exports LD_LIBRARY_PATH to those paths to build, which of course affects runtime:

It probably worked before

snapcraft-snapcraft-test # gdb /bin/sed core_9519
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /bin/sed...
(No debugging symbols found in /bin/sed)
[New LWP 9519]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `sed s/'/'\\''/g;1s/^/'/;$s/$/' \\/'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f623cd9018b in raise () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007f623cd9018b in raise () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f623cd6f859 in abort () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f623cdda3ee in ?? () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f623ce7cb4a in __fortify_fail () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f623ce7cb16 in __stack_chk_fail () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f623cca6118 in __pthread_tunables_init ()
   from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f623cc970d4 in __pthread_initialize_minimal ()
   from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007f623cc9600d in _init () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libpthread.so.0
#8  0x00007ffc6e7b0430 in ?? ()
#9  0x00007f623cf85c0c in ?? () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f623cf85dc6 in ?? () from /lib64/ld-linux-x86-64.so.2
#11 0x00007f623cf7513a in ?? () from /lib64/ld-linux-x86-64.so.2
#12 0x0000000000000002 in ?? ()
#13 0x00007ffc6e7b156d in ?? ()
#14 0x00007ffc6e7b1571 in ?? ()
#15 0x0000000000000000 in ?? ()

As a quick test I ran

unsquashfs /var/lib/snapd/snaps/gnome-3-38-2004-sdk_47.snap
for i in $(dpkg -L libc6); do [ -f squashfs-root$i ] && rm squashfs-root$i ; done
snap pack squashfs-root
snap install gnome-3-38-2004-sdk_0+git.99b816f_amd64.snap --dangerous

and it solved the issue. I will discuss with @kenvandine

Since this SDK snap is not meant to work in isolation and LD_LIBRARY_PATH exports are expected, libc6 should not be there.

2 Likes

Thanks ! While we wait for the fix to be pushed, is you unsquashfs magic a suitable workaround ?

Hi Sergio,

is there any explanation for the behaviour with the dependency detection suddenly being broken? This occurs even if the Gnome extension isn’t used, it implies there might be another change elsewhere that isn’t related solely to the Gnome extension. (The snaps I tested did run fine but for nitpicking reasons I just thought I’d doublecheck)

Do you have an example of this broken without using the GNOME extension? I would like to see it.

Just add python3-yaml to the stage-packages :

name: snapcraft-test
description: snapcraft-test
confinement: strict
version: '0.1'
summary: snapcraft-test
base: core20
grade: stable

parts:
    snapcraft-test:
        source: src
        plugin: dump
        stage-packages:
          - python3-yaml

apps:
    snapcraft-test:
        command: usr/bin/hello.sh
......
Priming snapcraft-test 
+ snapcraftctl prime
Unable to determine library dependencies for '/root/prime/usr/lib/python3/dist-packages/_yaml.cpython-38-x86_64-linux-gnu.so'
Snapping |                                                                                                                                                    
Snapped snapcraft-test_0.1_amd64.snap
2 Likes

Yes, https://github.com/MirServer/egmde-snap/pull/57

The logs shows it built, is this a case of not being able to run what was built?

Yes, it built, but with loads of “Unable to determine library dependencies for…” warnings. (Which are problematic for a classic snap.) But installing “stack smashes” in the install/post-refresh hook (except on 20.04 where it crashes when run).

This started during the UK day yesterday.

I think it has something to do with archive changes: today, with the same toolchain I couldn’t reproduce until I “cleaned” the multipass VM I was using.

1 Like

I’ve had no issues running the snap in the original post above in strict confinement. There wasn’t any obvious defects, but stable functionality in Snapcraft suddenly breaking alongside the Gnome extension also breaking just raises a red flag to me since there wasn’t a change to Snapcraft itself to explain it.

This issue should only affect snaps that are bundling libc6 or those that are doing classic confinement and have a mix of libraries from the system mixed with those from the host.

I’ve been hitting this with a snap which doesn’t (directly) bundle libc and is under strict confinement, which builds OK on core18. I’m just running another build just now to check, but in the meantime the repo is at https://github.com/mcphail/boa/tree/core20 if you want to take a look?

Pulling gzdoom 
+ snapcraftctl pull
*** stack smashing detected ***: terminated
Aborted (core dumped)
Usage: snapcraftctl [OPTIONS] COMMAND [ARGS]...

  snapcraftctl is how snapcraft.yaml can communicate with snapcraft

Options:
  -d, --debug
  --help       Show this message and exit.

Commands:
  build        Run the 'build' step of the calling part's lifecycle
  prime        Run the 'prime' step of the calling part's lifecycle
  pull         Run the 'pull' step of the calling part's lifecycle
  set-grade    Set the grade of the snap
  set-version  Set the version of the snap
  stage        Run the 'stage' step of the calling part's lifecycle
+ git cherry-pick df4f41f4e8d9b1d0372b6e814bb1a0c091a59f23
*** stack smashing detected ***: terminated
/bin/bash: line 43: 12942 Aborted                 (core dumped) git cherry-pick df4f41f4e8d9b1d0372b6e814bb1a0c091a59f23
Failed to run 'override-pull': Exit code was 134.
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 257, in run
    self.pull()
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 223, in pull
    env=env)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 100, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py", line 537, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'exec', 'lp-focal-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_LOCAL_SOURCES=1', '--env', 'SNAPCRAFT_SETUP_CORE=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-63273093", "build-request-timestamp": "2021-04-27T15:34:49Z", "build_url": "https://launchpad.net/~njmcphail/+snap/snapcraft-boa-b98bae/+build/1390976"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/snapcraft-boa-b98bae && linux64 snapcraft pull']' returned non-zero exit status 2.
Revoking proxy token...
RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=focal --arch=amd64 SNAPBUILD-1390976
Scanning for processes to kill in build SNAPBUILD-1390976

The root of you issue is the libc6 update causing snapcraft to segfault when running ldd, we are working on a work around in snapcraft for this which will come out as 4.6.3.

I have spoken to foundations to see how soon we can get a spin of core20 as well.

1 Like

Is there a workaround to this issue? Otherwise, how long might we expect it to take to have a fix released for core20?

Thank you.

If you are affected by the “Unable to determine libraries …” issue with classic confinement, there is a PR in progress which you can test by using snapcraft from the edge/pr-3514 channel. This will be out today.

core20 will be updated sometime tomorrow or the day after depending on testing results.

egmde is fixed when using Snapcraft from edge/pr-3514

edge/pr-3514 fixes the dependency problems for me too, thanks for all the help

2 Likes

I created a patchset for the gnome-3-38-2004-sdk snap which hopefully would get reviewed tomorrow.

I made a call for testing for Snapcraft to solve one of the common issues mentioned here Call for testing: Snapcraft 4.6.3