We have a MicroK8s user reporting a strange behavior that I cannot explain and I would like your insight on what might be happening. The issue reported is at https://github.com/ubuntu/microk8s/issues/595.
We have a daemon (called apiservice-kicker) that sleeps for a few seconds and when it wakes up it checks if the apiserver needs to be restarted. Our user reported that at some point the apiservice-kicker daemon could not find the sleep binary inside under $SNAP/bin/. I checked that in revision 743 of MicroK8s the sleep binary is in the right location. Why would this happen? Under what circumstances this may happen?
could it be that the microk8s snap got updated underneath the sleeping shell ? (snap changes and comparing timestamps might be helpful if the entries arent gone yet)
On the affected system, what’s the output of ldd /snap/microk8s/743/bin/sleep? What’s the output of the same thing when LD_LIBRARY_PATH is set up as it is at the top of apiservice-kicker?
That second one would be (at least on my system!),
these questions about dynamic libraries are because “no such file or directory” is also the error you get when trying to run a dynamically linked executable when the executable exists but the libraries it’s trying to load do not.
2019-08-12T10:57:03.299625-0400 dipper microk8s.daemon-apiserver-kicker[1621]: /snap/microk8s/743/apiservice-kicker: line 21: /snap/microk8s/743/bin/sleep: No such file or directory
I think this kind of proves my theory, just that it was not /bin/sleep in your microk8s snap being updated but /bin/bash from the core snap while your script was sleeping for 3 seconds.
I assume the first error occured when the old link went away … a gross hack (but working around the issue) would be to ship bash in the application snap and mangle all hasbang lines in the scripts (perhaps a layout would also work) … but this is indeed rather awful.