Failure to upgrade to 2.27.1 and 2.27.2 with LXD installed

Upgrading from 2.26.14 to 2.27.1 (currently in the beta channel) is failing for me, looks to be related to the lxd snap?

⟫ snap version
snap    2.26.14
snapd   2.26.14
series  16
ubuntu  17.04
kernel  4.10.0-32-generic
⟫ snap refresh --beta core
INFO cannot auto connect core:network-bind (slot auto-connection), candidates found: "canonical-livepatch:network-bind, redacted:network-bind"
error: cannot perform the following tasks:
- Setup snap "core" (2660) security profiles (cannot setup mount for snap "lxd": cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory)
- Setup snap "core" (2660) security profiles (cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory)
- Setup snap "core" (2660) security profiles (phase 2) (cannot setup mount for snap "lxd": cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory)
- Setup snap "core" (2660) security profiles (phase 2) (cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory)

⟫ snap info lxd
name:      lxd
summary:   "System container manager and API"
publisher: canonical
contact:   https://github.com/lxc/lxd/issues
description: |
  LXD is a container manager for system containers.

  It offers a REST API to remotely manage containers over the network, using an
  image based workflow and with support for live migration.

  Images are available for all Ubuntu releases and architectures as well as for
  a wide number of other Linux distributions.

  LXD containers are lightweight, secure by default and a great alternative to
  virtual machines.
commands:
  - lxd.benchmark
  - lxd.check-kernel
  - lxd.lxc
  - lxd
tracking:        stable
installed:       2.16 (3346) 36MB -
refreshed:       2017-08-12 01:30:43 +0100 BST
channels:
  stable:        2.16        (3346) 36MB -
  candidate:     2.16        (3346) 36MB -
  beta:          ↑
  edge:          git-9855c7b (3361) 37MB -

  2.0/stable:    2.0.10      (2943) 12MB -
  2.0/candidate: 2.0.10      (2943) 12MB -
  2.0/beta:      ↑
  2.0/edge:      git-c4b7c8d (2951) 12MB -

⟫ snap change 892
Status  Spawn                 Ready                 Summary
Undone  2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Download snap "core" (2660) from channel "beta"
Done    2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Fetch and check assertions for snap "core" (2660)
Undone  2017-08-16T12:24:03Z  2017-08-16T12:24:19Z  Mount snap "core" (2660)
Undone  2017-08-16T12:24:03Z  2017-08-16T12:24:18Z  Stop snap "core" services
Undone  2017-08-16T12:24:03Z  2017-08-16T12:24:18Z  Remove aliases for snap "core"
Undone  2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Make current revision for snap "core" unavailable
Undone  2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Copy snap "core" data
Error   2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Setup snap "core" (2660) security profiles
Undone  2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Make snap "core" (2660) available to the system
Error   2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Setup snap "core" (2660) security profiles (phase 2)
Hold    2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Set automatic aliases for snap "core"
Hold    2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Setup snap "core" aliases
Hold    2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Start snap "core" (2660) services
Hold    2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Remove data for snap "core" (2312)
Hold    2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Remove snap "core" (2312) from the system
Hold    2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Clean up "core" (2660) install
Hold    2017-08-16T12:24:03Z  2017-08-16T12:24:17Z  Run configure hook of "core" snap if present

......................................................................
Make current revision for snap "core" unavailable

2017-08-16T13:24:17+01:00 INFO Requested daemon restart.

......................................................................
Setup snap "core" (2660) security profiles

2017-08-16T13:24:14+01:00 INFO cannot auto connect core:home (slot auto-connection), candidates found: "lxd:home, redacted2:home"
2017-08-16T13:24:14+01:00 INFO cannot auto connect core:network (slot auto-connection), candidates found: "lxd:network, emoj:network, redacted:network, redacted2:network"
2017-08-16T13:24:14+01:00 INFO cannot auto connect core:network-bind (slot auto-connection), candidates found: "canonical-livepatch:network-bind, redacted:network-bind"
2017-08-16T13:24:17+01:00 ERROR cannot setup mount for snap "lxd": cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory
2017-08-16T13:24:17+01:00 ERROR cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory

......................................................................
Make snap "core" (2660) available to the system

2017-08-16T13:24:14+01:00 INFO Requested daemon restart.

......................................................................
Setup snap "core" (2660) security profiles (phase 2)

2017-08-16T13:24:17+01:00 ERROR cannot setup mount for snap "lxd": cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory
2017-08-16T13:24:17+01:00 ERROR cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory

Confirmed failing in 2.27.2 too

⟫ snap tasks 893
Status  Spawn                 Ready                 Summary
Undone  2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Download snap "core" (2677) from channel "beta"
Done    2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Fetch and check assertions for snap "core" (2677)
Undone  2017-08-16T14:24:42Z  2017-08-16T14:25:00Z  Mount snap "core" (2677)
Undone  2017-08-16T14:24:42Z  2017-08-16T14:25:00Z  Stop snap "core" services
Undone  2017-08-16T14:24:42Z  2017-08-16T14:25:00Z  Remove aliases for snap "core"
Undone  2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Make current revision for snap "core" unavailable
Undone  2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Copy snap "core" data
Error   2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Setup snap "core" (2677) security profiles
Undone  2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Make snap "core" (2677) available to the system
Error   2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Setup snap "core" (2677) security profiles (phase 2)
Hold    2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Set automatic aliases for snap "core"
Hold    2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Setup snap "core" aliases
Hold    2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Start snap "core" (2677) services
Hold    2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Remove data for snap "core" (2312)
Hold    2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Remove snap "core" (2312) from the system
Hold    2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Clean up "core" (2677) install
Hold    2017-08-16T14:24:42Z  2017-08-16T14:24:58Z  Run configure hook of "core" snap if present

......................................................................
Make current revision for snap "core" unavailable

2017-08-16T15:24:58+01:00 INFO Requested daemon restart.

......................................................................
Setup snap "core" (2677) security profiles

2017-08-16T15:24:56+01:00 INFO cannot auto connect core:home (slot auto-connection), candidates found: "lxd:home, redacted:home"
2017-08-16T15:24:56+01:00 INFO cannot auto connect core:network (slot auto-connection), candidates found: "lxd:network, redacted:network, emoj:network, redacted2:network"
2017-08-16T15:24:56+01:00 INFO cannot auto connect core:network-bind (slot auto-connection), candidates found: "redacted2:network-bind, canonical-livepatch:network-bind"
2017-08-16T15:24:58+01:00 ERROR cannot setup mount for snap "lxd": cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory
2017-08-16T15:24:58+01:00 ERROR cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory

......................................................................
Make snap "core" (2677) available to the system

2017-08-16T15:24:56+01:00 INFO Requested daemon restart.

......................................................................
Setup snap "core" (2677) security profiles (phase 2)

2017-08-16T15:24:58+01:00 ERROR cannot setup mount for snap "lxd": cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory
2017-08-16T15:24:58+01:00 ERROR cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory

I’m working on reproducing and analyzing this now.

I believe the issue is understood now and we are working on resolving it with LXD developers. I will post technical details soon.

So let me bring everyone up to speed and explain how I debugged the process:

LXD is a bit of a special snap, with more permissions than what is available usually. Using those powers it constructs a different mount namespace than the one snapd assumes. Let me show you the details.

Using nsenter(1) we can move to a mount namespace associated with a given PID or a given preserved mount namespace file (snapd uses this heavily). I installed the lxd snap and since it contains a daemon an appropriate namespace was constructed so that the daemon can start. I also used a random “regular” snap (here I used gnome-calculator) to compare how the mount namespace layout looks like compared to that of LXD.

I used nsenter and two separate terminals to enter the namespaces of LXD and gnome-calculator.

In LXD I see this:

zyga@fyke:~$ sudo nsenter -m/run/snapd/ns/lxd.mnt
root@fyke:/# cat /proc/self/mountinfo | grep /run
1507 213 0:20 / /var/lib/snapd/hostfs/run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=391612k,mode=755
1508 1507 0:23 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
1509 1507 0:46 / /var/lib/snapd/hostfs/run/user/1000 rw,nosuid,nodev,relatime master:413 - tmpfs tmpfs rw,size=391612k,mode=700,uid=1000,gid=1000
1510 1509 0:47 / /var/lib/snapd/hostfs/run/user/1000/gvfs rw,nosuid,nodev,relatime master:495 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
1511 1507 0:20 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=391612k,mode=755
1512 1507 0:50 / /var/lib/snapd/hostfs/run/vmblock-fuse rw,nosuid,nodev,relatime master:507 - fuse.vmware-vmblock vmware-vmblock rw,user_id=0,group_id=0,default_permissions,allow_other
1633 1649 0:60 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755

In gnome-calculator I see this:

zyga@fyke:~$ sudo nsenter -m/run/snapd/ns/gnome-calculator.mnt
root@fyke:/run/snapd/ns# cat /proc/self/mountinfo | grep /run
768 222 0:20 / /var/lib/snapd/hostfs/run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=391612k,mode=755
831 768 0:23 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
849 768 0:46 / /var/lib/snapd/hostfs/run/user/1000 rw,nosuid,nodev,relatime master:413 - tmpfs tmpfs rw,size=391612k,mode=700,uid=1000,gid=1000
935 849 0:47 / /var/lib/snapd/hostfs/run/user/1000/gvfs rw,nosuid,nodev,relatime master:495 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
1015 768 0:20 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=391612k,mode=755
1121 768 0:50 / /var/lib/snapd/hostfs/run/vmblock-fuse rw,nosuid,nodev,relatime master:507 - fuse.vmware-vmblock vmware-vmblock rw,user_id=0,group_id=0,default_permissions,allow_other
1440 1405 0:20 / /run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=391612k,mode=755
1441 1440 0:23 / /run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
1442 1440 0:46 / /run/user/1000 rw,nosuid,nodev,relatime master:413 - tmpfs tmpfs rw,size=391612k,mode=700,uid=1000,gid=1000
1443 1442 0:47 / /run/user/1000/gvfs rw,nosuid,nodev,relatime master:495 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
1444 1440 0:20 /snapd/ns /run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=391612k,mode=755
1445 1440 0:50 / /run/vmblock-fuse rw,nosuid,nodev,relatime master:507 - fuse.vmware-vmblock vmware-vmblock rw,user_id=0,group_id=0,default_permissions,allow_other
1450 1440 0:20 /netns /run/netns rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=391612k,mode=755

In contrast on my Ubuntu 16.04 host I see this:

zyga@fyke:~$ cat /proc/self/mountinfo   | grep /run
23 25 0:20 / /run rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=391612k,mode=755
28 23 0:23 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k
425 23 0:46 / /run/user/1000 rw,nosuid,nodev,relatime shared:413 - tmpfs tmpfs rw,size=391612k,mode=700,uid=1000,gid=1000
510 425 0:47 / /run/user/1000/gvfs rw,nosuid,nodev,relatime shared:495 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
518 23 0:20 /snapd/ns /run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=391612k,mode=755
553 518 0:3 mnt:[4026532410] /run/snapd/ns/etcd.mnt rw - nsfs nsfs rw
608 23 0:50 / /run/vmblock-fuse rw,nosuid,nodev,relatime shared:507 - fuse.vmware-vmblock vmware-vmblock rw,user_id=0,group_id=0,default_permissions,allow_other
281 518 0:3 mnt:[4026532294] /run/snapd/ns/core.mnt rw - nsfs nsfs rw
794 518 0:3 mnt:[4026532295] /run/snapd/ns/corebird.mnt rw - nsfs nsfs rw
978 518 0:3 mnt:[4026532296] /run/snapd/ns/handbrake-jz.mnt rw - nsfs nsfs rw
1164 518 0:3 mnt:[4026532297] /run/snapd/ns/telegram-sergiusens.mnt rw - nsfs nsfs rw
1376 518 0:3 mnt:[4026532298] /run/snapd/ns/gnome-calculator.mnt rw - nsfs nsfs rw
1528 518 0:3 mnt:[4026532299] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw

Reading those takes some getting used to and some perspective. One essential fact and difference is here though:

LXD 1633 1649 0:60 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
gnome-calculator 1440 1405 0:20 / /run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=391612k,mode=755
Host 23 25 0:20 / /run rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=391612k,mode=755

As we can see 0:60 != 0:20 so LXD does not see the same tmpfs as the host and as other snaps.
A quick peek at /run/ confirms this:

root@fyke:/# ls -l /run    
total 0
lrwxrwxrwx 1 root root 40 Aug 17 15:52 NetworkManager -> /var/lib/snapd/hostfs/run/NetworkManager
drwx------ 4 root root 80 Aug 17 15:52 lxcfs
drwxr-xr-x 2 root root 60 Aug 17 15:52 mount
lrwxrwxrwx 1 root root 36 Aug 17 15:52 resolvconf -> /var/lib/snapd/hostfs/run/resolvconf
drwx------ 3 root root 60 Aug 17 16:02 snapd
lrwxrwxrwx 1 root root 43 Aug 17 15:52 snapd-snap.socket -> /var/lib/snapd/hostfs/run/snapd-snap.socket
lrwxrwxrwx 1 root root 38 Aug 17 15:52 snapd.socket -> /var/lib/snapd/hostfs/run/snapd.socket
lrwxrwxrwx 1 root root 33 Aug 17 15:52 systemd -> /var/lib/snapd/hostfs/run/systemd

The error that blocked the update was related to the assumption that snapd can join a given mount namespace and run snap-update-ns and see /run/snapd/ns.

I spoke with the LXD developers and updated version of the LXD snap should hit the edge/candidate soon.

1 Like

This is fixed by updating to lxd from candidate or edge channels and then snapd successfuly updates to 2.27

hmm, would love to…

⟫ snap refresh lxd --candidate
error: cannot perform the following tasks:
- Setup snap "lxd" (3407) security profiles (cannot setup mount for snap "lxd": cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory)
- Setup snap "lxd" (3407) security profiles (cannot update mount namespace of snap "lxd": cannot update preserved namespace of snap "lxd": cannot update snap namespace: cannot save current mount profile of snap "lxd": open /run/snapd/ns: no such file or directory)

Update: Looks like I did manage to make it to 2.27.2? This confuses me:
⟫ snap version
snap 2.27.2+17.04
snapd 2.27.2+17.04
series 16
ubuntu 17.04
kernel 4.10.0-32-generic

This is interesting, I was on 16.04 and I was specifically still on core from stable where 2.26.14 reigned. Your version numbers indicate that you were on 2.27.2 from the Debian package. With the version you are on you probably need to stop snap services and only then refresh it.

Ah right, I’d pulled in snapd package from zesty-proposed in the background in between trying to update core and using the fixed LXD snap. I downgraded snapd to the version in zesty-updates (2.26); snap refresh --candidate lxd; snap refresh --beta core and am happily using new snapd and LXD