Artful i386 panics

The autopkgtest tests on artful/i386 are currently breaking. The full logs can be found here http://autopkgtest.ubuntu.com/packages/s/snapd/artful/i386 (warning, logs are ~20MB or so).

What happens is that snapd on artful (2.26.10) crashes with a panic like runtime: pointer 0x11673605 to unused region of span idx=0x8b39 span.base()=0x14654000 span.limit=0x156e0abc span.state=1 (full panic log is at http://paste.ubuntu.com/25318446/).

To reproduce:

  • use artful i386 VM (architecture seems to be important)
  • sudo apt install snapd
  • sudo snap install --beta core
  • wait for error from install (misleading 416 error)

check journalctl -u snapd - it will contain the error trace.

Running snapd from git master inside artful/i386 did not trigger the bug.

The panic happens also when I push the current master into the ppa:snappy-dev/edge PPA and use the deb produced by this (2.27.1+git321) or with a manually build (dpkg-buildpackage) version of snapd. It seems to be not happening go build github.com/snapcore/snapd/cmd/snapd && sudo cp snapd /usr/lib/snapd/snapd. So there might be differences in how golang compiles in the deb building?

It could well be the -buildmode=pie? The code in the go compiler to generate position independent code for i386 is a total hack (I feel safe saying this because I wrote it).

1 Like

And it is apparently exactly this. When I build my local git checkout with “-buildmode=pie” I get the same crash in artful.

@mwhudson can you please expand on the hack aspect. Should we consider building non-pie executables?

I created branches for 2.27 and master that disables -buildmode=pie on i386: https://github.com/snapcore/snapd/pull/3737 and https://github.com/snapcore/snapd/pull/3738

I explained the nature of the hack on IRC:

<mwhudson> zyga-ubuntu: and the "it's a hack" thing
<mwhudson> zyga-ubuntu: well you know you can't access the instruction pointer directly in i386 right?
<mwhudson> so whenever there is access to global data, which you want to do ip-relative
<mwhudson> the go compiler inserts a call to a function that copies the return address (i.e. the callers' ip) into bx and uses that
<mwhudson> the hack part is that it does this for _every_ access, there's no cleverness to avoid calling the stubs all the time
<mwhudson> so it should work, just with large and slow code
<zyga-ubuntu> mwhudson: I see, so it's not a hack per se that it's all barely holding together,
<zyga-ubuntu> mwhudson: more of a creative coding on your part
<zyga-ubuntu> mwhudson: and the true cause of the bug is still a mystery
<mwhudson> zyga-ubuntu: yeah

Could this be related to https://lists.ubuntu.com/archives/ubuntu-devel/2017-August/039946.html ?

I don’t think so, Go compilation ignores LDFLAGS etc, But I’m not 100% sure exactly what changed recently here. I have it somewhere on my TODO list to interpret the DEB_MAINT_OPTIONS=hardening=+foo,-bar stuff in dh-golang but I certainly haven’t done that yet :slight_smile: