System utilities mysteriously replaced by (broken) links to /snap/core

On August 29, I shut down my computer, which is a Thinkpad Yoga X1 running Xubuntu 16.04. The next time I tried to boot it, it was utterly broken.

The immediate problem was that a bunch of system services wouldn’t start. When I dug into it a bit, I found that 258 of the executable system utilities in /usr/bin (for example: find, top, who, wc, locale, du, debconf, ldd, nohup, among many others) had been replaced with symlinks to the equivalent binaries in /snap/core/4830/usr/bin. (Most of the symlinks were to /snap/core/4830, but a few were to /snap/core/5145 – I’m assuming these are version numbers, but I am unfamiliar with snap, so I’m not sure.) Most of the .py files in /usr/lib/python3.5 were also symlinked to files in /snap/core/4830/usr/lib/python3.5.

Unfortunately, on August 29, the /snap/core/4830 files were no longer present, so the symlinks were broken, and, lacking many basic system utilities, my computer was also broken. I was able to get my computer to boot by using chroot from a live CD and copying the live CD’s system utilities over the broken symlinks in /usr/bin. However, fixing Python has so far not been possible, and, after several days of fruitless googling and tinkering, I’m now simply wiping my computer and reinstalling Linux.

I have no idea why these symlinks were created in the first place, or whether the culprit was snapcraft itself or something else. It turns out that these symlinks were all created on August 7. One thing that happened on August 7 was that I ran apt dist-upgrade. (I don’t know if I did anything with snap that day – I suspect not, since I’ve never really used snap for anything, but it’s possible I did in the course of doing something else.) Here is the entry in /var/log/apt/history.log:

Start-Date: 2018-08-07 13:11:50
Commandline: apt dist-upgrade -y
Requested-By: jack (1000)
Upgrade: xserver-xorg-input-all:amd64 (1:7.7+13ubuntu3, 1:7.7+13ubuntu3.1), xserver-xorg:amd64 (1:7.7+13ubuntu3, 1:7.7+13ubuntu3.1), yarn:amd64 (1.9.2-1, 1.9.4-1), signal-desktop:amd64 (1.15.0, 1.15.2), libsystemd0:amd64 (229-4ubuntu21.2, 229-4ubuntu21.4), libsystemd0:i386 (229-4ubuntu21.2, 229-4ubuntu21.4), keybase:amd64 (2.5.0-20180803162902.dee7010d7a, 2.5$
End-Date: 2018-08-07 13:12:50

Also, here are the logs in /var/log/dpkg.log: https://gist.github.com/tinybike/d9262cbb9ca736133f33aa3dd638c98a

I’m posting all this here because I’m curious how this happened in the first place, because I want to avoid it happening again, and maybe most importantly because I wasn’t able to find anything about this particular issue by googling (mysterious, unwanted symlinks to /snap/core replacing core system utilities). Any insight into this issue would be much appreciated!

Edited to add: the timestamps on the symlinks don’t match the timestamps on the apt dist-upgrade (https://gist.github.com/tinybike/077539c9ee0af04891ae7637a588cb6e), so my guess is that this was caused by something besides that.

Did you ever run snapcraft with sudo or as root (never ever do this !!!) ?

if there was some snapcraft.yaml that tried to create links for all these binaries during build to link them into the snap (which would be an utterly broken thing to do but would explain the linking), the “running as root” of snapcraft would explain how they end up in your rootfs.

rule of thumb … use snapcraft cleanbuild when you can afford the extra time the containerized build takes and never ever run snapcraft as root or under sudo. if snapcraft needs extra privileges it will ask for your sudo password and the functions it uses then are well tested to not cause any harm to the host machine installation.