So let me bring everyone up to speed and explain how I debugged the process:
LXD is a bit of a special snap, with more permissions than what is available usually. Using those powers it constructs a different mount namespace than the one snapd assumes. Let me show you the details.
Using nsenter(1) we can move to a mount namespace associated with a given PID or a given preserved mount namespace file (snapd uses this heavily). I installed the lxd snap and since it contains a daemon an appropriate namespace was constructed so that the daemon can start. I also used a random “regular” snap (here I used gnome-calculator) to compare how the mount namespace layout looks like compared to that of LXD.
I used nsenter and two separate terminals to enter the namespaces of LXD and gnome-calculator.
In LXD I see this:
zyga@fyke:~$ sudo nsenter -m/run/snapd/ns/lxd.mnt
root@fyke:/# cat /proc/self/mountinfo | grep /run
1507 213 0:20 / /var/lib/snapd/hostfs/run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=391612k,mode=755
1508 1507 0:23 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
1509 1507 0:46 / /var/lib/snapd/hostfs/run/user/1000 rw,nosuid,nodev,relatime master:413 - tmpfs tmpfs rw,size=391612k,mode=700,uid=1000,gid=1000
1510 1509 0:47 / /var/lib/snapd/hostfs/run/user/1000/gvfs rw,nosuid,nodev,relatime master:495 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
1511 1507 0:20 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=391612k,mode=755
1512 1507 0:50 / /var/lib/snapd/hostfs/run/vmblock-fuse rw,nosuid,nodev,relatime master:507 - fuse.vmware-vmblock vmware-vmblock rw,user_id=0,group_id=0,default_permissions,allow_other
1633 1649 0:60 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
In gnome-calculator I see this:
zyga@fyke:~$ sudo nsenter -m/run/snapd/ns/gnome-calculator.mnt
root@fyke:/run/snapd/ns# cat /proc/self/mountinfo | grep /run
768 222 0:20 / /var/lib/snapd/hostfs/run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=391612k,mode=755
831 768 0:23 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
849 768 0:46 / /var/lib/snapd/hostfs/run/user/1000 rw,nosuid,nodev,relatime master:413 - tmpfs tmpfs rw,size=391612k,mode=700,uid=1000,gid=1000
935 849 0:47 / /var/lib/snapd/hostfs/run/user/1000/gvfs rw,nosuid,nodev,relatime master:495 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
1015 768 0:20 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=391612k,mode=755
1121 768 0:50 / /var/lib/snapd/hostfs/run/vmblock-fuse rw,nosuid,nodev,relatime master:507 - fuse.vmware-vmblock vmware-vmblock rw,user_id=0,group_id=0,default_permissions,allow_other
1440 1405 0:20 / /run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=391612k,mode=755
1441 1440 0:23 / /run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
1442 1440 0:46 / /run/user/1000 rw,nosuid,nodev,relatime master:413 - tmpfs tmpfs rw,size=391612k,mode=700,uid=1000,gid=1000
1443 1442 0:47 / /run/user/1000/gvfs rw,nosuid,nodev,relatime master:495 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
1444 1440 0:20 /snapd/ns /run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=391612k,mode=755
1445 1440 0:50 / /run/vmblock-fuse rw,nosuid,nodev,relatime master:507 - fuse.vmware-vmblock vmware-vmblock rw,user_id=0,group_id=0,default_permissions,allow_other
1450 1440 0:20 /netns /run/netns rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=391612k,mode=755
In contrast on my Ubuntu 16.04 host I see this:
zyga@fyke:~$ cat /proc/self/mountinfo | grep /run
23 25 0:20 / /run rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=391612k,mode=755
28 23 0:23 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k
425 23 0:46 / /run/user/1000 rw,nosuid,nodev,relatime shared:413 - tmpfs tmpfs rw,size=391612k,mode=700,uid=1000,gid=1000
510 425 0:47 / /run/user/1000/gvfs rw,nosuid,nodev,relatime shared:495 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
518 23 0:20 /snapd/ns /run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=391612k,mode=755
553 518 0:3 mnt:[4026532410] /run/snapd/ns/etcd.mnt rw - nsfs nsfs rw
608 23 0:50 / /run/vmblock-fuse rw,nosuid,nodev,relatime shared:507 - fuse.vmware-vmblock vmware-vmblock rw,user_id=0,group_id=0,default_permissions,allow_other
281 518 0:3 mnt:[4026532294] /run/snapd/ns/core.mnt rw - nsfs nsfs rw
794 518 0:3 mnt:[4026532295] /run/snapd/ns/corebird.mnt rw - nsfs nsfs rw
978 518 0:3 mnt:[4026532296] /run/snapd/ns/handbrake-jz.mnt rw - nsfs nsfs rw
1164 518 0:3 mnt:[4026532297] /run/snapd/ns/telegram-sergiusens.mnt rw - nsfs nsfs rw
1376 518 0:3 mnt:[4026532298] /run/snapd/ns/gnome-calculator.mnt rw - nsfs nsfs rw
1528 518 0:3 mnt:[4026532299] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
Reading those takes some getting used to and some perspective. One essential fact and difference is here though:
LXD 1633 1649 0:60 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
gnome-calculator 1440 1405 0:20 / /run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=391612k,mode=755
Host 23 25 0:20 / /run rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=391612k,mode=755
As we can see 0:60 != 0:20 so LXD does not see the same tmpfs as the host and as other snaps.
A quick peek at /run/ confirms this:
root@fyke:/# ls -l /run
total 0
lrwxrwxrwx 1 root root 40 Aug 17 15:52 NetworkManager -> /var/lib/snapd/hostfs/run/NetworkManager
drwx------ 4 root root 80 Aug 17 15:52 lxcfs
drwxr-xr-x 2 root root 60 Aug 17 15:52 mount
lrwxrwxrwx 1 root root 36 Aug 17 15:52 resolvconf -> /var/lib/snapd/hostfs/run/resolvconf
drwx------ 3 root root 60 Aug 17 16:02 snapd
lrwxrwxrwx 1 root root 43 Aug 17 15:52 snapd-snap.socket -> /var/lib/snapd/hostfs/run/snapd-snap.socket
lrwxrwxrwx 1 root root 38 Aug 17 15:52 snapd.socket -> /var/lib/snapd/hostfs/run/snapd.socket
lrwxrwxrwx 1 root root 33 Aug 17 15:52 systemd -> /var/lib/snapd/hostfs/run/systemd
The error that blocked the update was related to the assumption that snapd can join a given mount namespace and run snap-update-ns and see /run/snapd/ns.
I spoke with the LXD developers and updated version of the LXD snap should hit the edge/candidate soon.
