Daemon not seeing binary under $SNAP


#1

Hi,

We have a MicroK8s user reporting a strange behavior that I cannot explain and I would like your insight on what might be happening. The issue reported is at https://github.com/ubuntu/microk8s/issues/595.

We have a daemon (called apiservice-kicker) that sleeps for a few seconds and when it wakes up it checks if the apiserver needs to be restarted. Our user reported that at some point the apiservice-kicker daemon could not find the sleep binary inside under $SNAP/bin/. I checked that in revision 743 of MicroK8s the sleep binary is in the right location. Why would this happen? Under what circumstances this may happen?

Thank you


#2

could it be that the microk8s snap got updated underneath the sleeping shell ? (snap changes and comparing timestamps might be helpful if the entries arent gone yet)


#3

Hi, I’m the issue reporter.

snap changes doesn’t list an entry for microk8s.

snap info microk8s shows

refresh-date: 2019-08-08T22:50:17-04:00

Timestamps from the current symlink and root and bin/sleep paths in the microk8s 743 fs:

drwxr-xr-x 24 root root  1961 2019-08-05 08:55 /snap/microk8s/743/
-rwxr-xr-x  1 root root 35504 2019-08-05 08:55 /snap/microk8s/743/bin/sleep
lrwxrwxrwx  1 root root     3 2019-08-08 22:50 /snap/microk8s/current -> 743

Snap fs mount:

/var/lib/snapd/snaps/microk8s_743.snap on /snap/microk8s/743 type squashfs (ro,nodev,relatime,x-gdu.hide)

Snap fs image:

-rw------- 1 root root 192184320 2019-08-08 22:49 /var/lib/snapd/snaps/microk8s_743.snap

#4

without answering your question, why is microk8s shipping bin/sleep and not using the one from its base?


#5

On the affected system, what’s the output of ldd /snap/microk8s/743/bin/sleep? What’s the output of the same thing when LD_LIBRARY_PATH is set up as it is at the top of apiservice-kicker?

That second one would be (at least on my system!),

LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/743/lib:/snap/microk8s/743/usr/lib:/snap/microk8s/743/lib/x86_64-linux-gnu:/snap/microk8s/743/usr/lib/x86_64-linux-gnu ldd /snap/microk8s/743/bin/sleep

or, more generally, running ‘inside’ the snap (via e.g. snap run --shell microk8s.config),

LD_LIBRARY_PATH="$SNAP_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu" ldd $SNAP/bin/sleep

these questions about dynamic libraries are because “no such file or directory” is also the error you get when trying to run a dynamically linked executable when the executable exists but the libraries it’s trying to load do not.


#6
: kai@dipper kai; LD_LIBRARY_PATH="$SNAP_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu" ldd $SNAP/bin/sleep
	linux-vdso.so.1 (0x00007fff17454000)
	libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007f7cb6413000)
	/snap/core/current/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f7cb67df000)

snap changes has no entry for core, but the symlink timestamp is about 2 seconds after the failure occurred.

lrwxrwxrwx 1 root root 4 2019-08-12 10:57:05.364969562 -0400 /snap/core/current -> 7396

First error:

2019-08-12T10:57:03.299625-0400 dipper microk8s.daemon-apiserver-kicker[1621]: /snap/microk8s/743/apiservice-kicker: line 21: /snap/microk8s/743/bin/sleep: No such file or directory

#7

That would be nice! How would I put paths in core in the snap’s PATH?


#8

There is also this: Git-ubuntu issue with core 2.35.2 saying classic snaps would better not use binaries from the core.


#9

I think this kind of proves my theory, just that it was not /bin/sleep in your microk8s snap being updated but /bin/bash from the core snap while your script was sleeping for 3 seconds.

I assume the first error occured when the old link went away … a gross hack (but working around the issue) would be to ship bash in the application snap and mangle all hasbang lines in the scripts (perhaps a layout would also work) … but this is indeed rather awful.