Parallel instances: can't remove, can't connect interfaces, can't use layouts, env vars are wrong

About a month ago, I was able to build, install, and run two instances of my snap. Now, I’m facing trouble.

Cannot connect interfaces

I started out with two instances: my-snap_1 and my-snap_2.

The trouble starts when I try to connect interfaces on the second instance.

root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:account-control
error: cannot perform the following tasks:
- Connect my-snap_2:account-control to core:account-control (cannot update mount namespace of snap "my-snap_2": cannot update preserved namespace of snap "my-snap_2": cannot update snap namespace: remove /snap/my-snap/x1/usr/bin/nodejs12.x: no such file or directory)
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:home
error: cannot perform the following tasks:
- Connect my-snap_2:home to core:home (cannot update mount namespace of snap "my-snap_2": cannot update preserved namespace of snap "my-snap_2": cannot update snap namespace: remove /snap/my-snap/x1/usr/bin/nodejs12.x: no such file or directory)
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:log-observe
error: cannot perform the following tasks:
- Connect my-snap_2:log-observe to core:log-observe (cannot update mount namespace of snap "my-snap_2": cannot update preserved namespace of snap "my-snap_2": cannot update snap namespace: remove /snap/my-snap/x1/usr/bin/nodejs12.x: no such file or directory)
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:mount-observe
error: cannot perform the following tasks:
- Connect my-snap_2:mount-observe to core:mount-observe (cannot update mount namespace of snap "my-snap_2": cannot update preserved namespace of snap "my-snap_2": cannot update snap namespace: remove /snap/my-snap/x1/usr/bin/nodejs12.x: no such file or directory)
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:network-bind
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:network-control
error: cannot perform the following tasks:
- Connect my-snap_2:network-control to core:network-control (cannot update mount namespace of snap "my-snap_2": cannot update preserved namespace of snap "my-snap_2": cannot update snap namespace: remove /snap/my-snap/x1/usr/bin/nodejs12.x: no such file or directory)
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:process-control
error: cannot perform the following tasks:
- Connect my-snap_2:process-control to core:process-control (cannot update mount namespace of snap "my-snap_2": cannot update preserved namespace of snap "my-snap_2": cannot update snap namespace: remove /snap/my-snap/x1/usr/bin/nodejs12.x: no such file or directory)
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:system-backup
error: cannot perform the following tasks:
- Connect my-snap_2:system-backup to core:system-backup (cannot update mount namespace of snap "my-snap_2": cannot update preserved namespace of snap "my-snap_2": cannot update snap namespace: remove /snap/my-snap/x1/usr/bin/nodejs12.x: no such file or directory)
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder# snap connect my-snap_2:system-observe
error: cannot perform the following tasks:
- Connect my-snap_2:system-observe to core:system-observe (cannot update mount namespace of snap "my-snap_2": cannot update preserved namespace of snap "my-snap_2": cannot update snap namespace: remove /snap/my-snap/x1/usr/bin/nodejs12.x: no such file or directory)
root@ip-172-31-59-113:/home/ubuntu/SnapBuilder#
ubuntu@ip-172-31-59-113:~$ snap connections my-snap_1
Interface        Plug                                  Slot              Notes
account-control  my-snap_1:account-control  :account-control  manual
home             my-snap_1:home             :home             manual
log-observe      my-snap_1:log-observe      :log-observe      manual
mount-observe    my-snap_1:mount-observe    :mount-observe    manual
network-bind     my-snap_1:network-bind     :network-bind     -
network-control  my-snap_1:network-control  :network-control  manual
process-control  my-snap_1:process-control  :process-control  manual
system-backup    my-snap_1:system-backup    :system-backup    manual
system-observe   my-snap_1:system-observe   :system-observe   manual
ubuntu@ip-172-31-59-113:~$ snap connections my-snap_2
Interface        Plug                                  Slot           Notes
account-control  my-snap_2:account-control  -              -
home             my-snap_2:home             -              -
log-observe      my-snap_2:log-observe      -              -
mount-observe    my-snap_2:mount-observe    -              -
network-bind     my-snap_2:network-bind     :network-bind  -
network-control  my-snap_2:network-control  -              -
process-control  my-snap_2:process-control  -              -
system-backup    my-snap_2:system-backup    -              -
system-observe   my-snap_2:system-observe   -              -
ubuntu@ip-172-31-59-113:~$

When I shell into the second snap, I’m unable to cd into $SNAP directory.

ubuntu@ip-172-31-59-113:/snap$ cd $SNAP
bash: cd: /snap/my-snap/x1: No such file or directory

From the outside of the snap I see that the /snap/my-snap/ dir is indeed empty. When I tried this a month ago, however, the /snap/my-snap/ dir was redirecting to /snap/my-snap_2/ inside of the _2 snap.

Cannot remove an instance

ubuntu@ip-172-31-59-113:~$ sudo snap remove --purge my-snap_2
error: cannot perform the following tasks:
- Disconnect my-snap_2:network-control from core:network-control (internal error: connection "my-snap_2:network-control core:network-control" not found in state)

Layouts are only showing up in the original instance

After having trouble with the my-snap_2 instance, I tried just dealing with the my-snap_1 and my-snap instances.

However, the layouts I assigned to the snap only show up in my-snap. The $SNAP_DATA directory of my-snap_1 is empty, unlike that of my-snap.

Environment Variables are not working

The $SNAP_DATA environment variable of my-snap_1 has the same value as that of my-snap. When I shell in, I’m able to go to the “real” Snap Data directory by cd'ing.

The fact that I’m able to see both Snap Data directories after shelling in is also odd.

/usr/bin layout not working

I also had trouble with my /usr/bin/python3.7 -> $SNAP/usr/bin/python3.7 symlink, which had been working for a long time until now. I was able to remove that without issue, but it might serve as some clue. I don’t remember the error message.

This looks like a mount namespace issue. Perhaps @zyga-snapd can take a look at this.

Can you also post snap changes output and also snap tasks <id> for any tasks that are not in Done state from the snap changes output?

An update on my end: installing snapd allows me to remove the parallel instances. However, I still have issues with connecting interfaces.

Edit: I spoke too soon. I still can’t remove the second instance after reinstalling it.

Edit: I accidentally deleted the snap changes output I posted earlier. The following is the output I just acquired.

ubuntu@ip-172-31-31-253:~$ sudo snap changes
ID   Status  Spawn                   Ready                   Summary
76   Done    yesterday at 11:08 UTC  yesterday at 11:08 UTC  Auto-refresh snap "multipass"
77   Done    yesterday at 15:18 UTC  yesterday at 15:18 UTC  Auto-refresh snap "snapd"
78   Done    yesterday at 18:03 UTC  yesterday at 18:04 UTC  Remove "aws-iot-greengrass-prime" snap
79   Done    yesterday at 18:05 UTC  yesterday at 18:05 UTC  Install "aws-iot-greengrass_1" snap from file "aws-iot-greengrass_1.11.0_amd64.snap"
80   Done    yesterday at 18:05 UTC  yesterday at 18:05 UTC  Running service command for snap "aws-iot-greengrass_1"
81   Done    yesterday at 18:05 UTC  yesterday at 18:05 UTC  Remove "aws-iot-greengrass_1" snap
82   Done    yesterday at 18:05 UTC  yesterday at 18:05 UTC  Remove "aws-iot-greengrass" snap
83   Done    yesterday at 21:01 UTC  yesterday at 21:01 UTC  Refresh "core" snap from "edge" channel
84   Done    yesterday at 21:01 UTC  yesterday at 21:02 UTC  Refresh "core18" snap from "edge" channel
85   Done    yesterday at 21:02 UTC  yesterday at 21:02 UTC  Refresh "snapd" snap from "edge" channel
86   Done    yesterday at 21:04 UTC  yesterday at 21:05 UTC  Install "aws-iot-greengrass_1" snap from file "aws-iot-greengrass_1.11.0_amd64.snap"
87   Done    yesterday at 21:04 UTC  yesterday at 21:05 UTC  Running service command for snap "aws-iot-greengrass_1"
88   Done    yesterday at 21:05 UTC  yesterday at 21:05 UTC  Install "aws-iot-greengrass_2" snap from file "aws-iot-greengrass_1.11.0_amd64.snap"
89   Done    yesterday at 21:05 UTC  yesterday at 21:05 UTC  Running service command for snap "aws-iot-greengrass_2"
90   Error   yesterday at 21:08 UTC  yesterday at 21:08 UTC  Running service command
91   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:account-control to snapd:account-control
92   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:home to snapd:home
93   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:log-observe to snapd:log-observe
94   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:mount-observe to snapd:mount-observe
95   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:network-bind to snapd:network-bind
96   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:network-control to snapd:network-control
97   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:process-control to snapd:process-control
98   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:system-backup to snapd:system-backup
99   Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_1:system-observe to snapd:system-observe
100  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:account-control to snapd:account-control
101  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:home to snapd:home
102  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:log-observe to snapd:log-observe
103  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:mount-observe to snapd:mount-observe
104  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:network-bind to snapd:network-bind
105  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:network-control to snapd:network-control
106  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:process-control to snapd:process-control
107  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:system-backup to snapd:system-backup
108  Done    yesterday at 21:12 UTC  yesterday at 21:12 UTC  Connect aws-iot-greengrass_2:system-observe to snapd:system-observe
109  Done    yesterday at 21:53 UTC  yesterday at 21:53 UTC  Remove snaps "aws-iot-greengrass_1", "aws-iot-greengrass_2"
110  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Install "aws-iot-greengrass_1" snap from file "aws-iot-greengrass_1.11.0_amd64.snap"
111  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Running service command for snap "aws-iot-greengrass_1"
112  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Install "aws-iot-greengrass_2" snap from file "aws-iot-greengrass_1.11.0_amd64.snap"
113  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Running service command for snap "aws-iot-greengrass_2"
114  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:account-control to snapd:account-control
115  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:home to snapd:home
116  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:log-observe to snapd:log-observe
117  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:mount-observe to snapd:mount-observe
118  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:network-bind to snapd:network-bind
119  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:network-control to snapd:network-control
120  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:process-control to snapd:process-control
121  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:system-backup to snapd:system-backup
122  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_1:system-observe to snapd:system-observe
123  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:account-control to snapd:account-control
124  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:home to snapd:home
125  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:log-observe to snapd:log-observe
126  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:mount-observe to snapd:mount-observe
127  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:network-bind to snapd:network-bind
128  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:network-control to snapd:network-control
129  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:process-control to snapd:process-control
130  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:system-backup to snapd:system-backup
131  Done    yesterday at 21:54 UTC  yesterday at 21:54 UTC  Connect aws-iot-greengrass_2:system-observe to snapd:system-observe
132  Error   yesterday at 22:07 UTC  yesterday at 22:07 UTC  Running service command
133  Error   yesterday at 22:15 UTC  yesterday at 22:15 UTC  Running service command
134  Done    yesterday at 22:19 UTC  yesterday at 22:19 UTC  Install "aws-iot-greengrass" snap from file "aws-iot-greengrass_1.11.0_amd64.snap"
135  Done    yesterday at 22:19 UTC  yesterday at 22:19 UTC  Running service command for snap "aws-iot-greengrass"
136  Error   yesterday at 22:22 UTC  yesterday at 22:22 UTC  Running service command
137  Error   yesterday at 22:24 UTC  yesterday at 22:24 UTC  Running service command
138  Error   yesterday at 22:24 UTC  yesterday at 22:24 UTC  Running service command
139  Done    yesterday at 22:31 UTC  yesterday at 22:31 UTC  Connect aws-iot-greengrass:account-control to snapd:account-control
140  Done    yesterday at 22:31 UTC  yesterday at 22:31 UTC  Connect aws-iot-greengrass:home to snapd:home
141  Done    yesterday at 22:31 UTC  yesterday at 22:31 UTC  Connect aws-iot-greengrass:log-observe to snapd:log-observe
142  Done    yesterday at 22:31 UTC  yesterday at 22:31 UTC  Connect aws-iot-greengrass:mount-observe to snapd:mount-observe
143  Done    yesterday at 22:31 UTC  yesterday at 22:31 UTC  Connect aws-iot-greengrass:network-bind to snapd:network-bind
144  Done    yesterday at 22:31 UTC  yesterday at 22:31 UTC  Connect aws-iot-greengrass:network-control to snapd:network-control
145  Done    yesterday at 22:31 UTC  yesterday at 22:32 UTC  Connect aws-iot-greengrass:process-control to snapd:process-control
146  Done    yesterday at 22:32 UTC  yesterday at 22:32 UTC  Connect aws-iot-greengrass:system-backup to snapd:system-backup
147  Done    yesterday at 22:32 UTC  yesterday at 22:32 UTC  Connect aws-iot-greengrass:system-observe to snapd:system-observe
148  Done    yesterday at 22:35 UTC  yesterday at 22:35 UTC  Running service command
149  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Install "aws-iot-greengrass-prime" snap from file "aws-iot-greengrass-prime_1.11.0_amd64.snap"
150  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Running service command for snap "aws-iot-greengrass-prime"
151  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:account-control to snapd:account-control
152  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:home to snapd:home
153  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:log-observe to snapd:log-observe
154  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:mount-observe to snapd:mount-observe
155  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:network-bind to snapd:network-bind
156  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:network-control to snapd:network-control
157  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:process-control to snapd:process-control
158  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:system-backup to snapd:system-backup
159  Done    yesterday at 22:36 UTC  yesterday at 22:36 UTC  Connect aws-iot-greengrass-prime:system-observe to snapd:system-observe
160  Error   yesterday at 22:40 UTC  yesterday at 22:40 UTC  Running service command
161  Error   yesterday at 22:42 UTC  yesterday at 22:42 UTC  Running service command
162  Done    yesterday at 22:42 UTC  yesterday at 22:42 UTC  Running service command
163  Done    yesterday at 22:47 UTC  yesterday at 22:47 UTC  Running service command
164  Done    yesterday at 22:48 UTC  yesterday at 22:48 UTC  Running service command
165  Done    today at 05:22 UTC      today at 05:22 UTC      Auto-refresh snaps "lxd", "snapd"
166  Done    today at 07:17 UTC      today at 07:17 UTC      Remove "aws-iot-greengrass" snap
167  Error   today at 07:18 UTC      today at 07:18 UTC      Running service command
168  Done    today at 07:21 UTC      today at 07:22 UTC      Remove snaps "aws-iot-greengrass-prime", "aws-iot-greengrass_1", "aws-iot-greengrass_2"
169  Done    today at 07:22 UTC      today at 07:22 UTC      Install "aws-iot-greengrass_1" snap from file "aws-iot-greengrass_1.11.0_amd64.snap"
170  Done    today at 07:22 UTC      today at 07:22 UTC      Running service command for snap "aws-iot-greengrass_1"
171  Done    today at 07:23 UTC      today at 07:23 UTC      Install "aws-iot-greengrass_2" snap from file "aws-iot-greengrass_1.11.0_amd64.snap"
172  Done    today at 07:23 UTC      today at 07:23 UTC      Running service command for snap "aws-iot-greengrass_2"
173  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:account-control to snapd:account-control
174  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:home to snapd:home
175  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:log-observe to snapd:log-observe
176  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:mount-observe to snapd:mount-observe
177  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:network-bind to snapd:network-bind
178  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:network-control to snapd:network-control
179  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:process-control to snapd:process-control
180  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:system-backup to snapd:system-backup
181  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_1:system-observe to snapd:system-observe
182  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:account-control to snapd:account-control
183  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:home to snapd:home
184  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:log-observe to snapd:log-observe
185  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:mount-observe to snapd:mount-observe
186  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:network-bind to snapd:network-bind
187  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:network-control to snapd:network-control
188  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:process-control to snapd:process-control
189  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:system-backup to snapd:system-backup
190  Done    today at 07:24 UTC      today at 07:24 UTC      Connect aws-iot-greengrass_2:system-observe to snapd:system-observe
191  Error   today at 07:31 UTC      today at 07:31 UTC      Running service command

First of all, can you please switch the core snap, and the snapd snap if you have them to edge to see if that fixes any of the issues you mentioned here?

1 Like

Also, please attach the output of:

$ grep -n '' /var/lib/snapd/mount/snap.my-snap_*.fstab
1 Like

Sorry for the late response. We might be in different time zones.

ubuntu@ip-172-31-31-253:~$ sudo grep -n '' /var/lib/snapd/mount/snap.my-snap_*.fstab
/var/lib/snapd/mount/snap.my-snap_1.fstab:1:/snap/my-snap_1 /snap/my-snap none rbind,x-snapd.origin=overname 0 0
/var/lib/snapd/mount/snap.my-snap_1.fstab:2:/var/snap/my-snap_1 /var/snap/my-snap none rbind,x-snapd.origin=overname 0 0
/var/lib/snapd/mount/snap.my-snap_1.fstab:3:/var/snap/my-snap/x1/my-writable-dir/certs /snap/my-snap/x1/my-writable-dir/certs none rbind,rw,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_1.fstab:4:/var/snap/my-snap/x1/my-writable-dir/config /snap/my-snap/x1/my-writable-dir/config none rbind,rw,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_1.fstab:5:none /usr/bin/java8 none x-snapd.kind=symlink,x-snapd.symlink=/snap/my-snap/x1/usr/lib/jvm/java-8-openjdk-arch-symlink/jre/bin/java,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_1.fstab:6:none /usr/bin/nodejs12.x none x-snapd.kind=symlink,x-snapd.symlink=/snap/my-snap/x1/node-v12.16.3-linux/bin/node,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_1.fstab:7:none /usr/bin/python2.7 none x-snapd.kind=symlink,x-snapd.symlink=/snap/my-snap/x1/usr/bin/python2.7,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_1.fstab:8:none /usr/bin/python3.7 none x-snapd.kind=symlink,x-snapd.symlink=/snap/my-snap/x1/usr/bin/python3.7,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_1.fstab:9:/var/lib/snapd/hostfs/var/lib/dhcp /var/lib/dhcp none bind,rw,x-snapd.ignore-missing 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:1:/snap/my-snap_2 /snap/my-snap none rbind,x-snapd.origin=overname 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:2:/var/snap/my-snap_2 /var/snap/my-snap none rbind,x-snapd.origin=overname 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:3:/var/snap/my-snap/x1/my-writable-dir/certs /snap/my-snap/x1/my-writable-dir/certs none rbind,rw,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:4:/var/snap/my-snap/x1/my-writable-dir/config /snap/my-snap/x1/my-writable-dir/config none rbind,rw,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:5:none /usr/bin/java8 none x-snapd.kind=symlink,x-snapd.symlink=/snap/my-snap/x1/usr/lib/jvm/java-8-openjdk-arch-symlink/jre/bin/java,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:6:none /usr/bin/nodejs12.x none x-snapd.kind=symlink,x-snapd.symlink=/snap/my-snap/x1/node-v12.16.3-linux/bin/node,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:7:none /usr/bin/python2.7 none x-snapd.kind=symlink,x-snapd.symlink=/snap/my-snap/x1/usr/bin/python2.7,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:8:none /usr/bin/python3.7 none x-snapd.kind=symlink,x-snapd.symlink=/snap/my-snap/x1/usr/bin/python3.7,x-snapd.origin=layout 0 0
/var/lib/snapd/mount/snap.my-snap_2.fstab:9:/var/lib/snapd/hostfs/var/lib/dhcp /var/lib/dhcp none bind,rw,x-snapd.ignore-missing 0 0

That helped with interfaces, but I’m still getting errors with the environment variables and layouts.

When I install my-snap, my-snap_1, and my-snap_2, there doesn’t seem to be a problem.

However, when I just install my-snap_1 and my-snap_2, I get issues.

ubuntu@ip-172-31-31-253:~$ sudo snap run --shell my-snap_1.my-daemon
root@ip-172-31-31-253:/home/ubuntu# env | grep SNAP
SNAP_USER_DATA=/root/snap/my-snap_1/x1
SNAP_REVISION=x1
SNAP_ARCH=amd64
SNAP_INSTANCE_KEY=1
SNAP_USER_COMMON=/root/snap/my-snap_1/common
SNAP=/snap/my-snap/x1
SNAP_COMMON=/var/snap/my-snap/common
SNAP_NAME=my-snap
SNAP_INSTANCE_NAME=my-snap_1
SNAP_DATA=/var/snap/my-snap/x1
SNAP_COOKIE=DT-RRWyAraxQ6Y3j1WjVcd47xZu4g_bXbUiVpJHAO4n55QZ-2U1f
SNAP_REEXEC=
SNAP_CONTEXT=DT-RRWyAraxQ6Y3j1WjVcd47xZu4g_bXbUiVpJHAO4n55QZ-2U1f
SNAP_VERSION=1.11.0
SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void
root@ip-172-31-31-253:/home/ubuntu# cd $SNAP
bash: cd: /snap/my-snap/x1: No such file or directory

/snap/my-snap_1/x1 has the contents I expect from $SNAP.

I can’t cd into $SNAP_DATA either. /snap/my-snap_1/current works though.

ubuntu@ip-172-31-31-253:~$ sudo snap run --shell my-snap_2.my-daemon
root@ip-172-31-31-253:/home/ubuntu# env | grep SNAP
SNAP_USER_DATA=/root/snap/my-snap_2/x1
SNAP_REVISION=x1
SNAP_ARCH=amd64
SNAP_INSTANCE_KEY=2
SNAP_USER_COMMON=/root/snap/my-snap_2/common
SNAP=/snap/my-snap/x1
SNAP_COMMON=/var/snap/my-snap/common
SNAP_NAME=my-snap
SNAP_INSTANCE_NAME=my-snap_2
SNAP_DATA=/var/snap/my-snap/x1
SNAP_COOKIE=x_rTqIuYzc4c5qb7pPILP8xQzLR-7nFotTedoJ4EdXl0qrMHmn4p
SNAP_REEXEC=
SNAP_CONTEXT=x_rTqIuYzc4c5qb7pPILP8xQzLR-7nFotTedoJ4EdXl0qrMHmn4p
SNAP_VERSION=1.11.0
SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void
root@ip-172-31-31-253:/home/ubuntu# cd $SNAP
root@ip-172-31-31-253:/snap/my-snap/x1#

Strangely, $SNAP and $SNAP_DATA work fine on the second instance.

However, /snap/my-snap/ and /snap/my-snap_2/ have slightly different contents. My layouts aren’t affecting /snap/my-snap_2. Isn’t /snap/my-snap/ supposed to redirect to /snap/my-snap_2/?

I think this is the key point. The first mounts are intended to make the paths for $SNAP, $SNAP_DATA, and $SNAP_COMMON fixed within the sandbox no matter what instance key is used:

/snap/my-snap_1 /snap/my-snap none rbind,x-snapd.origin=overname 0 0
/var/snap/my-snap_1 /var/snap/my-snap none rbind,x-snapd.origin=overname 0 0

When you have my-snap installed (i.e. a version without an instance key), the targets for these bind mounts exist. When you only have versions with instance keys installed, then the targets do not exist.

This would generally result in snap-update-ns trying to create the target directory, but perhaps the AppArmor profile it is running under prevents that from happening?

We do ensure that the target directory of the mount (i.e. the /var/snap/ and /snap/, without the instance key) get created during installation.

Certainly works here:

maciek@galeon:~ snap list |grep test-snapd-tools
test-snapd-tools_bar           1.0                         6      latest/stable     test-snaps-canonical  -
test-snapd-tools_foo           1.0                         6      latest/stable     test-snaps-canonical  -
maciek@galeon:~ test-snapd-tools_foo.cmd sh -c 'env |grep -E "^SNAP_(NAME|INSTANCE)" | sort; ls -la $SNAP $SNAP_DATA' 
SNAP_INSTANCE_KEY=foo
SNAP_INSTANCE_NAME=test-snapd-tools_foo
SNAP_NAME=test-snapd-tools
/snap/test-snapd-tools/6:
total 4
drwxr-xr-x 4 root root   38 Mar 14  2017 .
drwxr-xr-x 3 root root 4096 Jul 10 10:27 ..
drwxr-xr-x 2 root root  122 Mar 14  2017 bin
drwxr-xr-x 2 root root   48 Mar 14  2017 meta

/var/snap/test-snapd-tools/6:
total 8
drwxr-xr-x 2 root root 4096 Jul 10 10:27 .
drwxr-xr-x 4 root root 4096 Jul 10 10:27 ..
maciek@galeon:~ test-snapd-tools_bar.cmd sh -c 'env |grep -E "^SNAP_(NAME|INSTANCE)" | sort; ls -la $SNAP $SNAP_DATA' 
SNAP_INSTANCE_KEY=bar
SNAP_INSTANCE_NAME=test-snapd-tools_bar
SNAP_NAME=test-snapd-tools
/snap/test-snapd-tools/6:
total 4
drwxr-xr-x 4 root root   38 Mar 14  2017 .
drwxr-xr-x 3 root root 4096 Jul 10 10:31 ..
drwxr-xr-x 2 root root  122 Mar 14  2017 bin
drwxr-xr-x 2 root root   48 Mar 14  2017 meta

/var/snap/test-snapd-tools/6:
total 8
drwxr-xr-x 2 root root 4096 Jul 10 10:31 .
drwxr-xr-x 4 root root 4096 Jul 10 10:31 ..

For the record, when ther’s only my-snap_1 and my-snap_2 installed, can you do:

$ ls -l /snap/ | grep my-snap
$ ls -l /var/snap/ | grep my-snap

Edit: also please check dmesg | grep DENIED for any AppArmor denials.