Certainly. let me start with package versions:
ii lxd 2.14-0ubuntu1~ubuntu16.0 amd64
snapd 2.21
charles@carbon:~$ snap list
Name Version Rev Developer Notes
charm 2.2 15 charms classic
conjure-up 2.1.5 352 canonical classic
core 16-2 1689 canonical -
petname 2.7 12 kirkland -
I started by bootstrapping lxd
juju bootstrap localhost localhost
Next step is to deploy kubernetes core bundle via conjure-up
conjure-up kubernetes-core localhost localhost $(petname)
During the installation phase, both kubernetes-worker and kubernetes-master will fail, upon further investigation it appears that the snap packages for kube-apiserver and kubelet are the culprits of the failure.
Manual installation returns > 0
All i see in the debug log is regarding cgroup denials.
[45167.170493] audit: type=1400 audit(1496685571.760:30238): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.core.hook.configure" pid=12801 comm="apparmor_parser"
[45183.016805] audit: type=1400 audit(1496685587.604:30239): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.kube-apiserver.daemon" pid=13517 comm="apparmor_parser"
[45183.097640] audit: type=1400 audit(1496685587.684:30240): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.kube-apiserver.hook.configure" pid=13519 comm="apparmor_parser"
[45183.187374] audit: type=1400 audit(1496685587.776:30241): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.kube-apiserver.kube-apiserver" pid=13523 comm="apparmor_parser"
[45183.789064] audit: type=1400 audit(1496685588.376:30242): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/run/snapd/lock/" pid=13593 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[45183.865330] audit: type=1400 audit(1496685588.452:30243): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/run/snapd/lock/" pid=13725 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[45184.859105] audit: type=1400 audit(1496685589.448:30244): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/snap/core/1689/usr/lib/snapd/snap-confine" pid=14027 comm="apparmor_parser"
[45184.874528] audit: type=1400 audit(1496685589.464:30245): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/snap/core/1689/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=14027 comm="apparmor_parser"
[45184.879056] audit: type=1400 audit(1496685589.468:30246): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.core.hook.configure" pid=14047 comm="apparmor_parser"
[45196.169401] audit: type=1400 audit(1496685600.756:30247): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/run/snapd/lock/" pid=14473 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[45196.249654] audit: type=1400 audit(1496685600.836:30248): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/run/snapd/lock/" pid=14492 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[45196.618100] audit: type=1400 audit(1496685601.204:30249): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/run/snapd/lock/" pid=14561 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Note: that when using conjure-up it does apply a special profile to the containers contained in this model.
source: https://github.com/conjure-up/spells/blob/master/kubernetes-core/steps/lxd-profile.yaml
which looks like the following when expanded
charles@carbon:~$ lxc profile show juju-frank-bat
config:
boot.autostart: "true"
linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
raw.lxc: |
lxc.aa_profile=unconfined
lxc.mount.auto=proc:rw sys:rw
lxc.cap.drop=
security.nesting: "true"
security.privileged: "true"
description: ""
devices:
aadisable:
path: /sys/module/nf_conntrack/parameters/hashsize
source: /dev/null
type: disk
aadisable1:
path: /sys/module/apparmor/parameters/enabled
source: /dev/null
type: disk
root:
path: /
pool: conjureup
type: disk
name: juju-frank-bat
used_by:
- /1.0/containers/juju-523d1d-0
- /1.0/containers/juju-523d1d-1
- /1.0/containers/juju-523d1d-2
- /1.0/containers/juju-523d1d-3
It’s worrth noting we are also leveraging the squashfuse
package since we’re running snapd in lxd in this setup. this is implied with layer-snap
.
Manual attempts to install the snap package return > 0 as mentioned above.
ubuntu@juju-523d1d-2:~$ sudo snap install kube-apiserver --channel=1.6/stable
2017-06-05T18:08:11Z INFO cannot auto connect kube-apiserver:network-bind to core:network-bind: (plug auto-connection), existing connection state "kube-apiserver:network-bind core:network-bind" in the way
error: cannot perform the following tasks:
- Run configure hook of "kube-apiserver" snap if present (run hook "configure": cannot create lock directory /run/snapd/lock: Permission denied)
ubuntu@juju-523d1d-2:~$ echo $?
1
and relevant entries from dmesg
[45524.016400] audit: type=1400 audit(1496685928.577:30290): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/snap/core/1689/usr/lib/snapd/snap-confine" pid=25977 comm="apparmor_parser"
[45524.042417] audit: type=1400 audit(1496685928.605:30291): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/snap/core/1689/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=25977 comm="apparmor_parser"
[45524.047111] audit: type=1400 audit(1496685928.609:30292): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.core.hook.configure" pid=25979 comm="apparmor_parser"
[45687.503631] audit: type=1400 audit(1496686092.055:30293): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.kube-apiserver.daemon" pid=29891 comm="apparmor_parser"
[45687.571116] audit: type=1400 audit(1496686092.123:30294): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.kube-apiserver.hook.configure" pid=29893 comm="apparmor_parser"
[45687.669554] audit: type=1400 audit(1496686092.219:30295): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.kube-apiserver.kube-apiserver" pid=29895 comm="apparmor_parser"
[45688.274890] audit: type=1400 audit(1496686092.823:30296): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/run/snapd/lock/" pid=29956 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[45688.362630] audit: type=1400 audit(1496686092.911:30297): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/run/snapd/lock/" pid=30067 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[45688.491876] audit: type=1400 audit(1496686093.043:30298): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/run/snapd/lock/" pid=30072 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[45689.426779] audit: type=1400 audit(1496686093.975:30299): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/snap/core/1689/usr/lib/snapd/snap-confine" pid=30278 comm="apparmor_parser"
[45689.443265] audit: type=1400 audit(1496686093.995:30300): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/snap/core/1689/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=30278 comm="apparmor_parser"
[45689.447997] audit: type=1400 audit(1496686093.999:30301): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.core.hook.configure" pid=30280 comm="apparmor_parser"