Hey folks. I’m working on a bug in canonical-kubernetes. For context, in order to make Kubernetes work in LXD containers, we have a special LXD profile that does a few things, most notably setting lxc.apparmor.profile=unconfined.
Initially everything works, but when the host machine is rebooted, all calls to snap executables within the containers fail:
$ /snap/bin/kube-controller-manager -h
cannot change profile for the next exec call: No such file or directory
snap-update-ns failed with code 1
This affects all snaps within the LXD containers, including snaps with confinement: strict and confinement: classic.
The output of sudo aa-status (link to full output) within the containers does not show any profiles for the snaps installed within them, e.g. kube-controller-manager.
I can work around it by manually reloading apparmor profiles:
It’s not clear to me why the profiles aren’t being applied on reboot. I have confirmed that this issue is not present when using a standard LXD profile, so it is definitely specific to our profile. Unfortunately, we’re pretty much stuck with this profile for now due to requirements in Kubernetes itself.
I think this issue is recent - we have been able to reboot LXD machines in the past without the snaps giving us too much trouble.
Any ideas why this might be happening, or how we might be able to fix it? Thanks.
$ uname -a
Linux ip-172-31-63-188 4.4.0-1060-aws #69-Ubuntu SMP Sun May 20 13:42:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
Guest machine:
$ uname -a
Linux test2 4.4.0-1060-aws #69-Ubuntu SMP Sun May 20 13:42:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
Correct me if I’m wrong but if you switch an LXD container to unconfined mode it disables apparmor stacking and then the profiles you see on the inside of the container as the same as those on the outside of the container. Long ago we said that this is an unsupported configuration simply because we cannot make this work in general.
Zyga, that’s correct - under this configuration, all of the containers and the host are using the same set of apparmor profiles.
It’s understandable that this is a configuration y’all can’t reasonably support. Unfortunately, we’re stuck with it for now to make kubelet work under LXD.
Do you at least have any quick pointers or advice for me while I’m looking into workarounds? How do the snapd apparmor profiles normally get loaded on boot?
Assuming that using something other than ‘unconfined’ causes the container to use stacked profiles, the only way you can reasonably pull this off is instead of using ‘unconfined’ you define another profile for your container that is wide open. Eg, on the host (untested):
Then launch with lxc.apparmor.profile=my-temporary-kubelet-container. Assuming simply not using ‘unconfined’ is enough, this should allow it to reboot and have the profiles loaded.
If that doesn’t work, then you would need to add to the guest a systemd unit or modify /etc/rc.local or something to run your sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/* command. The downside to this approach is that the policy in the container may interfere with the host’s processes if the same snaps are installed but have different interface connections, etc.
although, it has been a while since the last post in this forum, but worth a try. I followed the instructions from the website configuring microk8s inside LxD containers (microk8s.io/docs/lxd). The suggested profile was used in the containers. The steps worked fine inside the container following the first time launch.
The problem was seen once the host is turned off and not simply container restarted. Every time, I ran the commands after restarting the containers such as microk8s.inspect or microk8s.status the output was like below
snap-confine has elevated permissions and is not confined but should be
Then running the hack as below followed by the container restart solved the problem. The question is why the profile is not persistent?
