_az
December 10, 2022, 9:21pm
1
We’ve seen reports that our users on CentOS 7 are unable to install the certbot snap after upgrading snapd.
It appears to be encountering a segfault when running the configure hook.
Digging in to this, it appears that running snap-exec (unconfined) is enough to trigger the segfault:
[root@armor-client ~]# /usr/libexec/snapd/snap-exec
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
runtime stack:
runtime.throw({0x7257cd, 0x0})
/usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0x7ffc1b3b41d0 sp=0x7ffc1b3b41a0 pc=0x435291
runtime.sigpanic()
/usr/lib/golang/src/runtime/signal_unix.go:719 +0x396 fp=0x7ffc1b3b4220 sp=0x7ffc1b3b41d0 pc=0x44aa16
goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x5e9a70, 0xc0000ab6c0)
/usr/lib/golang/src/runtime/cgocall.go:156 +0x5c fp=0xc0000ab698 sp=0xc0000ab660 pc=0x40551c
crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL()
_cgo_gotypes.go:603 +0x49 fp=0xc0000ab6c0 sp=0xc0000ab698 pc=0x5366c9
crypto/internal/boring.init.0()
/usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x45 fp=0xc0000ab6f8 sp=0xc0000ab6c0 pc=0x537445
runtime.doInit(0xad84a0)
/usr/lib/golang/src/runtime/proc.go:6498 +0x123 fp=0xc0000ab830 sp=0xc0000ab6f8 pc=0x4449a3
runtime.doInit(0xad61a0)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000ab968 sp=0xc0000ab830 pc=0x4448f1
runtime.doInit(0xad9280)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000abaa0 sp=0xc0000ab968 pc=0x4448f1
runtime.doInit(0xad8260)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000abbd8 sp=0xc0000abaa0 pc=0x4448f1
runtime.doInit(0xad9bc0)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000abd10 sp=0xc0000abbd8 pc=0x4448f1
runtime.doInit(0xad7220)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000abe48 sp=0xc0000abd10 pc=0x4448f1
runtime.doInit(0xad7fe0)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000abf80 sp=0xc0000abe48 pc=0x4448f1
runtime.main()
/usr/lib/golang/src/runtime/proc.go:238 +0x1e6 fp=0xc0000abfe0 sp=0xc0000abf80 pc=0x437926
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000abfe8 sp=0xc0000abfe0 pc=0x464361
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000034fb0 sp=0xc000034f90 pc=0x437d36
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.forcegchelper()
/usr/lib/golang/src/runtime/proc.go:306 +0xad fp=0xc000034fe0 sp=0xc000034fb0 pc=0x437bcd
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000034fe8 sp=0xc000034fe0 pc=0x464361
created by runtime.init.8
/usr/lib/golang/src/runtime/proc.go:294 +0x25
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc0000357b0 sp=0xc000035790 pc=0x437d36
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.bgsweep()
/usr/lib/golang/src/runtime/mgcsweep.go:163 +0x88 fp=0xc0000357e0 sp=0xc0000357b0 pc=0x4243a8
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000357e8 sp=0xc0000357e0 pc=0x464361
created by runtime.gcenable
/usr/lib/golang/src/runtime/mgc.go:181 +0x55
goroutine 4 [GC scavenge wait]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000035f80 sp=0xc000035f60 pc=0x437d36
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.bgscavenge()
/usr/lib/golang/src/runtime/mgcscavenge.go:265 +0xcd fp=0xc000035fe0 sp=0xc000035f80 pc=0x4224ad
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000035fe8 sp=0xc000035fe0 pc=0x464361
created by runtime.gcenable
/usr/lib/golang/src/runtime/mgc.go:182 +0x65
goroutine 5 [finalizer wait]:
runtime.gopark(0xaec640, 0xc000034770, 0xf1, 0x48, 0xad75e0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000034630 sp=0xc000034610 pc=0x437d36
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.runfinq()
/usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000347e0 sp=0xc000034630 pc=0x419f33
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000347e8 sp=0xc0000347e0 pc=0x464361
created by runtime.createfing
/usr/lib/golang/src/runtime/mfinal.go:157 +0x45
Aborted
The crashing process is this one:
snap-exec 5540 2447 0 /usr/libexec/snapd/snap-exec --hook=configure certbot
Version info:
[root@armor-client ~]# snap version
snap 2.57.6-1.el7
snapd 2.57.6-1.el7
series 16
centos 7
kernel 3.10.0-1160.45.1.el7.x86_64
[root@armor-client ~]# rpm -q snapd
snapd-2.57.6-1.el7.x86_64
[root@armor-client ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
Hello, does anyone have a solution to this problem?
gvde
December 12, 2022, 6:01am
3
No solution, but temporary workaround: downgrade to the previous version.
# yum downgrade snapd snap-confine snapd-selinux
...
---> Package snap-confine.x86_64 0:2.55.3-1.el7 will be a downgrade
---> Package snap-confine.x86_64 0:2.57.6-1.el7 will be erased
---> Package snapd.x86_64 0:2.55.3-1.el7 will be a downgrade
---> Package snapd.x86_64 0:2.57.6-1.el7 will be erased
---> Package snapd-selinux.noarch 0:2.55.3-1.el7 will be a downgrade
---> Package snapd-selinux.noarch 0:2.57.6-1.el7 will be erased
--> Finished Dependency Resolution
...
I think it’s an issue with 2.57 as there is also the same report for ubuntu: https://bugs.launchpad.net/snapd/+bug/1995102
1 Like
_az
December 12, 2022, 6:26am
4
The Ubuntu report looks to have a different cause, something in the CLI parser. It is not the case on CentOS 7 that every snap command segfaults - it is possible to install and run the hello-world snap, for example.
Unfortunately EPEL7 does not keep older versions of packages uploaded, so a yum downgrade would only be possible if it was locally cached, I guess.
gvde
December 12, 2022, 6:45am
5
If you don’t have it in the cache you should still find those three rpms in a mirror which doesn’t delete old version like https://mirror.lt.ucsc.edu/epel/7/x86_64/Packages/s/.
1 Like
Hello, thank you very much for your help. I did the following:
yum remove snapd
yum remove snap-confine.x86_64
yum remove snapd-selinux.noarch
then I downloaded from https://mirror.lt.ucsc.edu/epel/7/x86_64/Packages/s/ snapd-2.54.4-1.el7.x86_64.rpm, snapd-selinux-2.54.4- 1.el7.noarch.rpm, snap-confine-2.54.4-1.el7.x86_64.rpm
i installed .rpm yum localinstall …
Thank you very much again!
boutell
December 12, 2022, 3:26pm
7
Seeing the same issue. This is really bad news. Can anything be done to fix this issue upstream?
bmw
December 12, 2022, 3:57pm
8
Just to help show it’s not an obscure Certbot issue, here’s the same thing happening when installing the snapcraft snap:
# snap install snapcraft --classic
error: cannot perform the following tasks:
- Run configure hook of "snapcraft" snap if present (run hook "configure":
-----
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
runtime stack:
runtime.throw({0x7257cd, 0x0})
/usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0x7ffdc76e55f0 sp=0x7ffdc76e55c0 pc=0x435291
runtime.sigpanic()
/usr/lib/golang/src/runtime/signal_unix.go:719 +0x396 fp=0x7ffdc76e5640 sp=0x7ffdc76e55f0 pc=0x44aa16
goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x5e9a70, 0xc0000a76c0)
/usr/lib/golang/src/runtime/cgocall.go:156 +0x5c fp=0xc0000a7698 sp=0xc0000a7660 pc=0x40551c
crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL()
_cgo_gotypes.go:603 +0x49 fp=0xc0000a76c0 sp=0xc0000a7698 pc=0x5366c9
crypto/internal/boring.init.0()
/usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x45 fp=0xc0000a76f8 sp=0xc0000a76c0 pc=0x537445
runtime.doInit(0xad84a0)
/usr/lib/golang/src/runtime/proc.go:6498 +0x123 fp=0xc0000a7830 sp=0xc0000a76f8 pc=0x4449a3
runtime.doInit(0xad61a0)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000a7968 sp=0xc0000a7830 pc=0x4448f1
runtime.doInit(0xad9280)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000a7aa0 sp=0xc0000a7968 pc=0x4448f1
runtime.doInit(0xad8260)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000a7bd8 sp=0xc0000a7aa0 pc=0x4448f1
runtime.doInit(0xad9bc0)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000a7d10 sp=0xc0000a7bd8 pc=0x4448f1
runtime.doInit(0xad7220)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000a7e48 sp=0xc0000a7d10 pc=0x4448f1
runtime.doInit(0xad7fe0)
/usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000a7f80 sp=0xc0000a7e48 pc=0x4448f1
runtime.main()
/usr/lib/golang/src/runtime/proc.go:238 +0x1e6 fp=0xc0000a7fe0 sp=0xc0000a7f80 pc=0x437926
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000a7fe8 sp=0xc0000a7fe0 pc=0x464361
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000032fb0 sp=0xc000032f90 pc=0x437d36
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.forcegchelper()
/usr/lib/golang/src/runtime/proc.go:306 +0xad fp=0xc000032fe0 sp=0xc000032fb0 pc=0x437bcd
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000032fe8 sp=0xc000032fe0 pc=0x464361
created by runtime.init.8
/usr/lib/golang/src/runtime/proc.go:294 +0x25
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc0000337b0 sp=0xc000033790 pc=0x437d36
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.bgsweep()
/usr/lib/golang/src/runtime/mgcsweep.go:163 +0x88 fp=0xc0000337e0 sp=0xc0000337b0 pc=0x4243a8
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000337e8 sp=0xc0000337e0 pc=0x464361
created by runtime.gcenable
/usr/lib/golang/src/runtime/mgc.go:181 +0x55
goroutine 4 [GC scavenge wait]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000033f80 sp=0xc000033f60 pc=0x437d36
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.bgscavenge()
/usr/lib/golang/src/runtime/mgcscavenge.go:265 +0xcd fp=0xc000033fe0 sp=0xc000033f80 pc=0x4224ad
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000033fe8 sp=0xc000033fe0 pc=0x464361
created by runtime.gcenable
/usr/lib/golang/src/runtime/mgc.go:182 +0x65
goroutine 5 [finalizer wait]:
runtime.gopark(0xaec640, 0xc000032770, 0xf1, 0x48, 0xad75e0)
/usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000032630 sp=0xc000032610 pc=0x437d36
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:372
runtime.runfinq()
/usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000327e0 sp=0xc000032630 pc=0x419f33
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000327e8 sp=0xc0000327e0 pc=0x464361
created by runtime.createfing
/usr/lib/golang/src/runtime/mfinal.go:157 +0x45
-----)
mvo
December 12, 2022, 4:31pm
9
Thanks for raising this issue. We are currently looking into the certbot issue.
However please note that the segfault of “snap-exec” is most likely unrelated to the certbot issue. The snap-exec tool is run inside the snap namespace so running it directly from the host is not how it’s usually executed.
[edit: looks like that is exactly the issue]
1 Like
mvo
December 12, 2022, 5:17pm
10
I wonder if this is somehow related to how the rpm was build. When I our google:centos-7-64:tests/main/classic-confinement spread test and snap install --clasic snapcraft and snap install --classic certbox there I see no failure. During that test the snapd rpm is build in on the test VM so I wonder if something is off with the rpm build somehow ?
_az
December 12, 2022, 11:14pm
11
Do you use mainline Go to build snapd in those tests?
Based on the segfault’s stack trace, something is going wrong with Go trying to do runtime linking to BoringSSL, which would not happen in mainline Go:
It looks like EPEL7 might be building snapd using the boringcrypto branch of Go .
That could explain a coverage gap in testing.
1 Like
mvo
December 13, 2022, 8:25am
12
Indeed, thanks _az! I wonder how to disable this boringcrypto during the build, that might be all that is needed to fix this issue.
1 Like
bmw
December 13, 2022, 7:38pm
13
To help make sure we’re all on the same page about the severity here, if a CentOS 7 snapd user upgrades the packages on their system to the newest snap RPM packages in EPEL 7, any classic snap becomes completely unrunnable on their system. The Certbot snap alone has over 50k users on CentOS 7.
I really hope this can be fixed soon.
3 Likes
_az
December 13, 2022, 9:07pm
14
mvo
December 16, 2022, 8:54am
15
Thanks everyone for reporting this and for your patience. The amazing @mborzecki has fixed this by building snapd without the boringssl change. The build is in https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-4790972cca and should be generally now.
1 Like
The update should hit testing soon. For the time being, you can switch to the builds tab in bodhi and grab the RPMs directly from the build system.
