Today my Snapcraft has been acting funny, but not just my snapcraft on my local machine, it’s been acting funny on Github too, see here:
It appears that Snapcraft has suddenly lost the ability to intelligently identify linkage entirely. If you look at the build log from 2 days ago, there wasn’t any issue there. It’s doing this to everything I build locally.
Now what’s even stranger, adding the Gnome extensions to one of my other snaps causes stack smashing errors and then kills snapcraft. I don’t have a build log for this but you can reproduce it yourself by taking the latest commit at https://github.com/mrCarroll/runescape-snap and simply adding the extension to both the rs3 and osrs parts.
Updating pull step for rs3 (source changed)
+ snapcraftctl pull
*** stack smashing detected ***: terminated
Aborted (core dumped)
Usage: snapcraftctl [OPTIONS] COMMAND [ARGS]...
snapcraftctl is how snapcraft.yaml can communicate with snapcraft
Options:
-d, --debug
--help Show this message and exit.
Commands:
build Run the 'build' step of the calling part's lifecycle
prime Run the 'prime' step of the calling part's lifecycle
pull Run the 'pull' step of the calling part's lifecycle
set-grade Set the grade of the snap
set-version Set the version of the snap
stage Run the 'stage' step of the calling part's lifecycle
Updating pull step for rs3-gpg (source changed)
+ wget https://content.runescape.com/downloads/ubuntu/runescape.gpg.key -O /root/parts/rs3-gpg/src/runescape.gpg.key
*** stack smashing detected ***: terminated
/bin/bash: line 43: 4876 Aborted (core dumped)
(This actually happens 3 times in the same build for different stages)
This is on a clean Ubuntu 20.04.2 VM (HyperV) with nothing but Snapcraft/LXD installed, Snapcraft 4.6.2 (6350), LXD (20037).
james@james-Virtual-Machine:~/runescape-snap$ snap version
snap 2.49.2
snapd 2.49.2
series 16
ubuntu 20.04
kernel 5.8.0-50-generic
Adding the Gnome extensions to RuneScape in the past never caused any build time issues, the runtime didn’t work, but this new snapcraft behaviour is new.
Has anyone else noticed anything similarly suspicious with Snapcraft lately and is there any info that might be relevant to help debugging? I’ve deployed some snaps with this potentially broken build system earlier so might need to redeploy them, but it’s hard to determine the issue at first glance.
Edit: Stack smashing occurs in the edge release of Snapcraft 4.7 too
I wasn’t able to build any snap, even snaps that built properly a week or so ago.
However, it seems like this only happens with core20. I was able to rebuild a core18 snap without problem. I don’t have too many core18 snaps on my machine, but I can investigate more if needed.
Unfortunately, the snap I’m working on right now does need core20.
$ cat src/usr/bin/hello.sh
#! /bin/sh
echo "It worked !"
Now, the interesting part is this : running snapcraft with this snapcraft.yaml, it crashes :
$ snapcraft clean snapcraft-test
Launching a VM.
Cleaning pull step (and all subsequent steps) for snapcraft-test
$ snapcraft
Launching a VM.
snapd is not logged in, snap install commands will use sudo
snap "core20" has no updates available
Skipping pull gnome-3-38-extension (already ran)
Pulling snapcraft-test
+ snapcraftctl pull
*** stack smashing detected ***: terminated
Aborted
(more output removed)
but when I remove the extensions: [gnome-3-38] from the app, it works fine :
snapcraft clean snapcraft-test
Launching a VM.
Cleaning pull step (and all subsequent steps) for snapcraft-test
Cleaning up staging area
Cleaning up parts directory
$ snapcraft
Launching a VM.
snapd is not logged in, snap install commands will use sudo
snap "core20" has no updates available
Pulling snapcraft-test
+ snapcraftctl pull
Building snapcraft-test
+ snapcraftctl build
+ cp --archive --link --no-dereference . /root/parts/snapcraft-test/install
Staging snapcraft-test
+ snapcraftctl stage
Priming snapcraft-test
+ snapcraftctl prime
Snapping |
Snapped snapcraft-test_0.1_amd64.snap
@James-Carroll could you run this test on your machine to confirm it’s not just me ? Thanks !
Same results here, the tiny hello world snap crashes With Core20 + Gnome-3-38, works fine with just Core20 (I’d imagine it’d complain about libraries if it had any), and has no problems in Core18 + Gnome 3-34.
I’d say glad to see it’s not just me, I was considering everything from faulty RAM and an unreliable CPU after some issues with this computer lately, but if it’s as wide spread as it looks it might end up being there might be a lot of snaps that are potentially defective being spread.
I think I found the problem. The GNOME sdk snap includes libc6, the extension exports LD_LIBRARY_PATH to those paths to build, which of course affects runtime:
It probably worked before
snapcraft-snapcraft-test # gdb /bin/sed core_9519
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /bin/sed...
(No debugging symbols found in /bin/sed)
[New LWP 9519]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `sed s/'/'\\''/g;1s/^/'/;$s/$/' \\/'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f623cd9018b in raise () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f623cd9018b in raise () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f623cd6f859 in abort () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f623cdda3ee in ?? () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f623ce7cb4a in __fortify_fail () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f623ce7cb16 in __stack_chk_fail () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f623cca6118 in __pthread_tunables_init ()
from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f623cc970d4 in __pthread_initialize_minimal ()
from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007f623cc9600d in _init () from /snap/gnome-3-38-2004-sdk/current/lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007ffc6e7b0430 in ?? ()
#9 0x00007f623cf85c0c in ?? () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f623cf85dc6 in ?? () from /lib64/ld-linux-x86-64.so.2
#11 0x00007f623cf7513a in ?? () from /lib64/ld-linux-x86-64.so.2
#12 0x0000000000000002 in ?? ()
#13 0x00007ffc6e7b156d in ?? ()
#14 0x00007ffc6e7b1571 in ?? ()
#15 0x0000000000000000 in ?? ()
As a quick test I ran
unsquashfs /var/lib/snapd/snaps/gnome-3-38-2004-sdk_47.snap
for i in $(dpkg -L libc6); do [ -f squashfs-root$i ] && rm squashfs-root$i ; done
snap pack squashfs-root
snap install gnome-3-38-2004-sdk_0+git.99b816f_amd64.snap --dangerous
and it solved the issue. I will discuss with @kenvandine
Since this SDK snap is not meant to work in isolation and LD_LIBRARY_PATH exports are expected, libc6 should not be there.
is there any explanation for the behaviour with the dependency detection suddenly being broken? This occurs even if the Gnome extension isn’t used, it implies there might be another change elsewhere that isn’t related solely to the Gnome extension. (The snaps I tested did run fine but for nitpicking reasons I just thought I’d doublecheck)
Yes, it built, but with loads of “Unable to determine library dependencies for…” warnings. (Which are problematic for a classic snap.) But installing “stack smashes” in the install/post-refresh hook (except on 20.04 where it crashes when run).
This started during the UK day yesterday.
I think it has something to do with archive changes: today, with the same toolchain I couldn’t reproduce until I “cleaned” the multipass VM I was using.
I’ve had no issues running the snap in the original post above in strict confinement. There wasn’t any obvious defects, but stable functionality in Snapcraft suddenly breaking alongside the Gnome extension also breaking just raises a red flag to me since there wasn’t a change to Snapcraft itself to explain it.
This issue should only affect snaps that are bundling libc6 or those that are doing classic confinement and have a mix of libraries from the system mixed with those from the host.
I’ve been hitting this with a snap which doesn’t (directly) bundle libc and is under strict confinement, which builds OK on core18. I’m just running another build just now to check, but in the meantime the repo is at https://github.com/mcphail/boa/tree/core20 if you want to take a look?
Thanks !