During spread testing we build a native package of snapd (for the current OS and architecture) and copy several files over from the native package to the core snap (the core snap is uncompressed and recompressed for this operation). This introduces several kinds of issues.
Native C code has assumptions about directory structure of the host that may be incompatible with the fixed structure inside the core snap
Native C code was compiled and linked with the host toolchain which may use incompatible set of libraries, including more recent libc, additional libraries (e.g. selinux) that are not present in the core snap. This happened in Fedora 29 with libc 2.27 which is not present in Ubuntu 16.04-based core snap.
Native package may have performed interpreter path mangling that is not compatible with the core snap. This specifically includes rewriting /bin/sh to /usr/bin/sh which affects the snap-device-helper shell script. While the script itself uses absolute path names compatible with the core snap for the commands it invokes (and resorts to shell builtins for everything else) the interpreter path itself is broken.
A simple but costly solution is to restore the correct build environment. Snapd must be built with the Ubuntu 16.04 toolchain and build configuration if it ends up in a repackaged core snap.
We could adjust the core snap to have paths compatible with what is used in Fedora. This would also require updating the C library as the world moves on to ever-more-recent version of libc.
To fix the issue we need to perform several changes that prevent us from using incompatible executables from the host distribution inside the guest namespace.
Stop repackaging core the way we do now unless we are testing against a ubuntu 16.04 which is ABI compatible with the core snap. Unless on Ubuntu 16.04 we must only replace snap-exec. Please see the list of executables below for detailed analysis.
When snap-confine is invoked with —base it bind mounts /usr/lib/snapd from the host. This must be changed to a bind mount from /snap/core/current/usr/lib/snapd or /snap/snapd/current/usr/lib/snapd instead. This prevents us from using incompatible binaries from the host distribution.
This also means that to run a snap with arbitrary base we need to either have snapd or core snap installed. This is a new prerequisite that needs to be changed in the snap manager.
Dynamically linked go but that’s fine. No need to repackage.
Until it radically changes. (Document this in snapctl/main.go)
Dynamically linked C can stay as is with caveat.
Dynamically linked C. Can stay as-is with caveat.
Shell script. Can stay as-is with caveat.
We don’t need to repackage it as the rest of the core snap is old.
The caveat above is that if any of those files changes significantly we won’t see the new versions inside the core and inside the execution environment until said changes are released to stable. To fix this we need the full solution that involves building snapd twice, once for the host and once for the core snap repackaging (or snapd snap repackaging).