Daemon not seeing binary under $SNAP

Hi,

We have a MicroK8s user reporting a strange behavior that I cannot explain and I would like your insight on what might be happening. The issue reported is at https://github.com/ubuntu/microk8s/issues/595.

We have a daemon (called apiservice-kicker) that sleeps for a few seconds and when it wakes up it checks if the apiserver needs to be restarted. Our user reported that at some point the apiservice-kicker daemon could not find the sleep binary inside under $SNAP/bin/. I checked that in revision 743 of MicroK8s the sleep binary is in the right location. Why would this happen? Under what circumstances this may happen?

Thank you

could it be that the microk8s snap got updated underneath the sleeping shell ? (snap changes and comparing timestamps might be helpful if the entries arent gone yet)

Hi, I’m the issue reporter.

snap changes doesn’t list an entry for microk8s.

snap info microk8s shows

refresh-date: 2019-08-08T22:50:17-04:00

Timestamps from the current symlink and root and bin/sleep paths in the microk8s 743 fs:

drwxr-xr-x 24 root root  1961 2019-08-05 08:55 /snap/microk8s/743/
-rwxr-xr-x  1 root root 35504 2019-08-05 08:55 /snap/microk8s/743/bin/sleep
lrwxrwxrwx  1 root root     3 2019-08-08 22:50 /snap/microk8s/current -> 743

Snap fs mount:

/var/lib/snapd/snaps/microk8s_743.snap on /snap/microk8s/743 type squashfs (ro,nodev,relatime,x-gdu.hide)

Snap fs image:

-rw------- 1 root root 192184320 2019-08-08 22:49 /var/lib/snapd/snaps/microk8s_743.snap

without answering your question, why is microk8s shipping bin/sleep and not using the one from its base?

On the affected system, what’s the output of ldd /snap/microk8s/743/bin/sleep? What’s the output of the same thing when LD_LIBRARY_PATH is set up as it is at the top of apiservice-kicker?

That second one would be (at least on my system!),

LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/743/lib:/snap/microk8s/743/usr/lib:/snap/microk8s/743/lib/x86_64-linux-gnu:/snap/microk8s/743/usr/lib/x86_64-linux-gnu ldd /snap/microk8s/743/bin/sleep

or, more generally, running ‘inside’ the snap (via e.g. snap run --shell microk8s.config),

LD_LIBRARY_PATH="$SNAP_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu" ldd $SNAP/bin/sleep

these questions about dynamic libraries are because “no such file or directory” is also the error you get when trying to run a dynamically linked executable when the executable exists but the libraries it’s trying to load do not.

1 Like
: kai@dipper kai; LD_LIBRARY_PATH="$SNAP_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu" ldd $SNAP/bin/sleep
	linux-vdso.so.1 (0x00007fff17454000)
	libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007f7cb6413000)
	/snap/core/current/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f7cb67df000)

snap changes has no entry for core, but the symlink timestamp is about 2 seconds after the failure occurred.

lrwxrwxrwx 1 root root 4 2019-08-12 10:57:05.364969562 -0400 /snap/core/current -> 7396

First error:

2019-08-12T10:57:03.299625-0400 dipper microk8s.daemon-apiserver-kicker[1621]: /snap/microk8s/743/apiservice-kicker: line 21: /snap/microk8s/743/bin/sleep: No such file or directory

That would be nice! How would I put paths in core in the snap’s PATH?

There is also this: Git-ubuntu issue with core 2.35.2 saying classic snaps would better not use binaries from the core.

I think this kind of proves my theory, just that it was not /bin/sleep in your microk8s snap being updated but /bin/bash from the core snap while your script was sleeping for 3 seconds.

I assume the first error occured when the old link went away … a gross hack (but working around the issue) would be to ship bash in the application snap and mangle all hasbang lines in the scripts (perhaps a layout would also work) … but this is indeed rather awful.