I have come to realize that consistently when a machine is rebooted, in order for etcd (the brains of kubernetes) to come up in a master control node, the snap must be manually plugged in to the removable-media interface.
This is quite unfortunate as etcd is very important kubernetes component as its key-value store.
First is this expected behavior or am i hitting some sort of edge case? Second if it is expected is there anyway to allow autoconnect to happen in the future?
If you connect the interface, it should stay connected after reboot. @zyga-snapd may want more information on this.
Out of curiosity, what are the denials you are seeing? I’m a bit surprised etcd needs removable-media at all (unless you are actually storing data on removable drives; then it makes perfect sense).
Jun 07 05:03:07 nuc2-2-ubuntu-2CPU-8GB-1 etcd[1267]: cannot access data directory: mkdir /var/snap/etcd/current: permission denied
and similarly
cannot access data directory: /var/snap/etcd/current/.touch: permission denied
As for removable-drives i dont have enough knowledge on the privs involved there and optionally the other types of interfaces. All i know is this allows me the permission to let etcd make changes to my ‘drive’. Please look at my linked document above if you are curious.
Did you symlink /var, /var/snap or /var/snap/etcd to something on an external drive? Can you give the output of ‘journalctl |grep audit’ at the time of the denial?
So, no to any link. Also the snap is run in an lxd container whose configured to use the zfs device as memory, so maybe thats the issue?. I will try to get the output tomorrow morning.
Unfortunately, this didn’t give useful information. Please make sure the interface is disconnected, then try to reproduce the issue, when you see the problem, then do inside and outside the container sudo journalctl |grep 'audit:' (I added the ‘:’ this time to get rid of the kube messages).