Layout issues in aws-iot-greengrass with parallel instances

I previously had trouble with parallel instances under certain circumstances, but I could work with parallel instances as recently as mid-October (maybe sooner, but I didn’t record doing so). Now, I’m having trouble with simply connecting interfaces.

$ sudo snap install --channel=edge snapd; sudo snap refresh --channel=edge snapd
$ sudo snap set system experimental.parallel-instances=true
$ sudo snap install --channel=edge aws-iot-greengrass
aws-iot-greengrass (edge) 1.11.0 from Amazon Web Services (aws✓) installed
$ sudo snap install --channel=edge aws-iot-greengrass_prime
aws-iot-greengrass_prime (edge) 1.11.0 from Amazon Web Services (aws✓) installed
$ snap list
Name                      Version               Rev    Tracking         Publisher   Notes
amazon-ssm-agent          2.3.978.0             2012   latest/stable/…  aws✓        classic
aws-iot-greengrass        1.11.0                44     latest/edge      aws✓        -
aws-iot-greengrass_prime  1.11.0                44     latest/edge      aws✓        -
core                      16-2.47.1             10185  latest/stable    canonical✓  core
core18                    20200724              1885   latest/stable    canonical✓  base
lxd                       4.0.3                 16922  4.0/stable/…     canonical✓  -
snapd                     2.48+git481.g5387521  10450  latest/edge      canonical✓  snapd

I can’t connect with the first instance:

$ sudo snap connect aws-iot-greengrass:hardware-observe
error: cannot perform the following tasks:
- Connect aws-iot-greengrass:hardware-observe to snapd:hardware-observe (cannot update mount namespace of snap "aws-iot-greengrass": cannot update preserved namespace of snap "aws-iot-greengrass": cannot update snap namespace: remove /snap/aws-iot-greengrass/44/usr/bin/python3: read-only file system)
$ sudo snap connect aws-iot-greengrass:hardware-observe
error: cannot perform the following tasks:
- Connect aws-iot-greengrass:hardware-observe to snapd:hardware-observe (cannot update mount namespace of snap "aws-iot-greengrass": cannot update preserved namespace of snap "aws-iot-greengrass": cannot update snap namespace: remove /usr/bin/python3.8: read-only file system)

However, it’ll work with the second instance:

$ sudo snap connect aws-iot-greengrass_prime:hardware-observe
ubuntu@ip-172-31-29-24:~$ snap connections
Interface           Plug                                                      Slot                 Notes
greengrass-support  aws-iot-greengrass:greengrass-support-no-container        :greengrass-support  -
greengrass-support  aws-iot-greengrass_prime:greengrass-support-no-container  :greengrass-support  -
hardware-observe    aws-iot-greengrass_prime:hardware-observe                 :hardware-observe    manual
home                aws-iot-greengrass:home-for-greengrassd                   :home                -
home                aws-iot-greengrass:home-for-hooks                         :home                -
home                aws-iot-greengrass_prime:home-for-greengrassd             :home                -
home                aws-iot-greengrass_prime:home-for-hooks                   :home                -
lxd-support         lxd:lxd-support                                           :lxd-support         -
network             aws-iot-greengrass:network                                :network             -
network             aws-iot-greengrass_prime:network                          :network             -
network             lxd:network                                               :network             -
network-bind        aws-iot-greengrass:network-bind                           :network-bind        -
network-bind        aws-iot-greengrass_prime:network-bind                     :network-bind        -
network-bind        lxd:network-bind                                          :network-bind        -
network-control     aws-iot-greengrass:network-control                        :network-control     -
network-control     aws-iot-greengrass_prime:network-control                  :network-control     -
opengl              aws-iot-greengrass:opengl                                 :opengl              -
opengl              aws-iot-greengrass_prime:opengl                           :opengl              -
optical-drive       aws-iot-greengrass:optical-drive                          :optical-drive       -
optical-drive       aws-iot-greengrass_prime:optical-drive                    :optical-drive       -
process-control     aws-iot-greengrass:process-control                        :process-control     -
process-control     aws-iot-greengrass_prime:process-control                  :process-control     -
system-observe      aws-iot-greengrass:system-observe                         :system-observe      -
system-observe      aws-iot-greengrass_prime:system-observe                   :system-observe      -
system-observe      lxd:system-observe                                        :system-observe      -
ubuntu@ip-172-31-29-24:~$ snap connections aws-iot-greengrass
Interface                Plug                                                Slot                 Notes
camera                   aws-iot-greengrass:camera                           -                    -
dvb                      aws-iot-greengrass:dvb                              -                    -
gpio                     aws-iot-greengrass:gpio                             -                    -
gpio-memory-control      aws-iot-greengrass:gpio-memory-control              -                    -
greengrass-support       aws-iot-greengrass:greengrass-support-no-container  :greengrass-support  -
hardware-observe         aws-iot-greengrass:hardware-observe                 -                    -
hardware-random-control  aws-iot-greengrass:hardware-random-control          -                    -
home                     aws-iot-greengrass:home-for-greengrassd             :home                -
home                     aws-iot-greengrass:home-for-hooks                   :home                -
hugepages-control        aws-iot-greengrass:hugepages-control                -                    -
i2c                      aws-iot-greengrass:i2c                              -                    -
iio                      aws-iot-greengrass:iio                              -                    -
joystick                 aws-iot-greengrass:joystick                         -                    -
log-observe              aws-iot-greengrass:log-observe                      -                    -
mount-observe            aws-iot-greengrass:mount-observe                    -                    -
network                  aws-iot-greengrass:network                          :network             -
network-bind             aws-iot-greengrass:network-bind                     :network-bind        -
network-control          aws-iot-greengrass:network-control                  :network-control     -
opengl                   aws-iot-greengrass:opengl                           :opengl              -
optical-drive            aws-iot-greengrass:optical-drive                    :optical-drive       -
process-control          aws-iot-greengrass:process-control                  :process-control     -
raw-usb                  aws-iot-greengrass:raw-usb                          -                    -
removable-media          aws-iot-greengrass:removable-media                  -                    -
serial-port              aws-iot-greengrass:serial-port                      -                    -
spi                      aws-iot-greengrass:spi                              -                    -
system-observe           aws-iot-greengrass:system-observe                   :system-observe      -

Looks like it’s complaining about the layouts used to symlink to lambda runtimes.

Even when I remove the second instance , it complains

ubuntu@ip-172-31-29-24:~$ sudo snap remove --purge aws-iot-greengrass_prime
aws-iot-greengrass_prime removed
ubuntu@ip-172-31-29-24:~$ sudo snap connect aws-iot-greengrass:hardware-observe
error: cannot perform the following tasks:
- Connect aws-iot-greengrass:hardware-observe to snapd:hardware-observe (cannot update mount namespace of snap "aws-iot-greengrass": cannot update preserved namespace of snap "aws-iot-greengrass": cannot update snap namespace: remove /usr/bin/python3.8: read-only file system)

Here’s the layout section of the yaml:

layout:
  $SNAP/greengrass/config:
    bind: $SNAP_DATA/user-certs/config
  $SNAP/greengrass/certs:
    bind: $SNAP_DATA/user-certs/certs
  /usr/bin/python3.8:
    symlink: $SNAP/usr/bin/python3.8
  /usr/bin/python3.7:
    # python3.7 lambdas will be redirected to python3.8
    symlink: $SNAP/usr/bin/python3.8
  $SNAP/usr/bin/python3:
    symlink: $SNAP/usr/bin/python3.8
  /usr/bin/python2.7:
    symlink: $SNAP/usr/bin/python2.7
  $SNAP/usr/bin/python:
    symlink: $SNAP/usr/bin/python2.7
  /usr/bin/nodejs12.x:
    symlink: $SNAP/wrapper-scripts/exec-node.sh
  /usr/bin/node:
    symlink: $SNAP/node-v12.18.4-linux/bin/node
  /usr/bin/java8:
    symlink: $SNAP/usr/lib/jvm/java-8-openjdk-arch-symlink/jre/bin/java
  /usr/bin/java:
    symlink: $SNAP/usr/lib/jvm/java-8-openjdk-arch-symlink/jre/bin/java

Is there a workaround for this?

btw is there a roadmap for getting the parallel instances feature to stable? I know it’s experimental, but we’ve only experienced more and more problems.

Thanks for the report. I tried to reproduce the problem on a 20.04 instance.

google:ubuntu-20.04-64 .../mini/hello# sudo snap install --channel=edge snapd; sudo snap refresh --channel=edge snapd
snap "snapd" is already installed, see 'snap help refresh'
2020-12-03T12:13:16Z INFO Waiting for automatic snapd restart...
snapd (edge) 2.48+git490.g6b9cc09 from Canonical✓ refreshed
google:ubuntu-20.04-64 .../mini/hello# sudo snap set system experimental.parallel-instances=true
google:ubuntu-20.04-64 .../mini/hello# sudo snap install --channel=edge aws-iot-greengrass
aws-iot-greengrass (edge) 1.11.0 from Amazon Web Services (aws✓) installed
google:ubuntu-20.04-64 .../mini/hello# sudo snap install --channel=edge aws-iot-greengrass_prime
aws-iot-greengrass_prime (edge) 1.11.0 from Amazon Web Services (aws✓) installed
google:ubuntu-20.04-64 .../mini/hello# snap list
Name                      Version               Rev    Tracking         Publisher          Notes
aws-iot-greengrass        1.11.0                44     latest/edge      aws✓               -
aws-iot-greengrass_prime  1.11.0                44     latest/edge      aws✓               -
core18                    20200929              1932   latest/stable    canonical✓         base
google-cloud-sdk          319.0.0               160    latest/stable/…  google-cloud-sdk✓  classic
lxd                       4.0.4                 18150  4.0/stable/…     canonical✓         -
snapd                     2.48+git490.g6b9cc09  10477  latest/edge      canonical✓         snapd

However, it seems to connect properly:

google:ubuntu-20.04-64 .../mini/hello# sudo snap connect aws-iot-greengrass:hardware-observe
google:ubuntu-20.04-64 .../mini/hello# snap connections
Interface           Plug                                                      Slot                 Notes
greengrass-support  aws-iot-greengrass:greengrass-support-no-container        :greengrass-support  -
greengrass-support  aws-iot-greengrass_prime:greengrass-support-no-container  :greengrass-support  -
hardware-observe    aws-iot-greengrass:hardware-observe                       :hardware-observe    manual
home                aws-iot-greengrass:home-for-greengrassd                   :home                -
home                aws-iot-greengrass:home-for-hooks                         :home                -
home                aws-iot-greengrass_prime:home-for-greengrassd             :home                -
home                aws-iot-greengrass_prime:home-for-hooks                   :home                -
lxd-support         lxd:lxd-support                                           :lxd-support         -
network             aws-iot-greengrass:network                                :network             -
network             aws-iot-greengrass_prime:network                          :network             -
network             lxd:network                                               :network             -
network-bind        aws-iot-greengrass:network-bind                           :network-bind        -
network-bind        aws-iot-greengrass_prime:network-bind                     :network-bind        -
network-bind        lxd:network-bind                                          :network-bind        -
network-control     aws-iot-greengrass:network-control                        :network-control     -
network-control     aws-iot-greengrass_prime:network-control                  :network-control     -
opengl              aws-iot-greengrass:opengl                                 :opengl              -
opengl              aws-iot-greengrass_prime:opengl                           :opengl              -
optical-drive       aws-iot-greengrass:optical-drive                          :optical-drive       -
optical-drive       aws-iot-greengrass_prime:optical-drive                    :optical-drive       -
process-control     aws-iot-greengrass:process-control                        :process-control     -
process-control     aws-iot-greengrass_prime:process-control                  :process-control     -
system-observe      aws-iot-greengrass:system-observe                         :system-observe      -
system-observe      aws-iot-greengrass_prime:system-observe                   :system-observe      -
system-observe      lxd:system-observe                                        :system-observe      -

Are there more steps needed to reproduce? Does it happen when parallel instances is not enabled, or enabled but only with aws-iot-greengrass snap (without the _prime one)?

Did you try discarding the snap’s mount namespace? Use sudo /usr/lib/snapd/snap-discard-ns aws-iot-greengrass and sudo /usr/lib/snapd/snap-discard-ns aws-iot-greengrass_instance to discard them. I imagine it is indeed related to the layouts

@mborzecki probably couldn’t reproduce since the daemon is disabled by default and thus the mount namespace would not be created

I tried again. Good news and bad news:

The good news is that I can at least connect my interfaces. I didn’t get any trouble there. I have no idea why it failed that first time.

The bad news is that I’m once again running into some of the problems I faced earlier.

The prime instance has messed up layouts. The SNAP_DATA directory doesn’t have any of the bind mounts I specified in my snapcraft.yaml.

Is that to force snapd to redefine the mount namespace?

After running those commands, I still get the errors I described:

I don’t see my layouts.

ubuntu@ip-172-31-29-219:~$ sudo snap run --shell aws-iot-greengrass_prime.greengrassd
root@ip-172-31-29-219:/home/ubuntu# ls $SNAP_DATA
hook-logs
root@ip-172-31-29-219:/home/ubuntu# exit
exit
ubuntu@ip-172-31-29-219:~$ sudo snap run --shell aws-iot-greengrass.greengrassd
root@ip-172-31-29-219:/home/ubuntu# ls $SNAP_DATA
hook-logs  user-certs

Notice how user-certs is only available in the original instance.

In fact, it looks like the prime instance is referencing the original instance’s contents.

ubuntu@ip-172-31-29-219:~$ sudo snap run --shell aws-iot-greengrass.greengrassd
root@ip-172-31-29-219:/home/ubuntu# ls $SNAP/greengrass/certs/
a12e616ba8.cert.pem  a12e616ba8.private.key  a12e616ba8.public.key  root.ca.pem
root@ip-172-31-29-219:/home/ubuntu# ls $SNAP_DATA/user-certs/certs/
a12e616ba8.cert.pem  a12e616ba8.private.key  a12e616ba8.public.key  root.ca.pem
root@ip-172-31-29-219:/home/ubuntu# exit
exit
ubuntu@ip-172-31-29-219:~$ sudo snap run --shell aws-iot-greengrass_prime.greengrassd
root@ip-172-31-29-219:/home/ubuntu# ls $SNAP/greengrass/certs/
a12e616ba8.cert.pem  a12e616ba8.private.key  a12e616ba8.public.key  root.ca.pem
root@ip-172-31-29-219:/home/ubuntu# ls $SNAP_DATA
hook-logs

Note that $SNAP_DATA/user-certs/certs is supposed to be bind-mounted to $SNAP/greengrass/certs, but the prime instance is missing it.

Furthermore, the $SNAP/greengrass/certs for the prime instance has the original instance’s contents.

Can you post the output of:

sudo nsenter -m/run/snapd/ns/aws-iot-greengrass.mnt /bin/findmnt

and the same for /run/snapd/ns/aws-iot-greengrass_prime.mnt.

1 Like

One sec. The text is bigger than is allowed on a post.

You can use https://paste.ubuntu.com also please add -o+PROPAGATION to the command.

1 Like

nsenter-original.txt
nsenter-prime.txt
nsenter-diff.txt

I ran commands such as

sudo nsenter -m/run/snapd/ns/aws-iot-greengrass.mnt /bin/findmnt > nsenter-original.txt

How would I add -o+PROPAGATION?

just after the findmnt command, i.e.

sudo nsenter -m/run/snapd/ns/aws-iot-greengrass.mnt /bin/findmnt -o+PROPAGATION > file.txt
1 Like

nsenter-original-oprop.txt
nsenter-prime-oprop.txt
nsenter-oprop-diff.txt

btw if you shell into the snap with

sudo snap run --shell aws-iot-greengrass_prime.greengrassd

Do you see my issues with the layouts? If you ls $SNAP_DATA, do you see the user-certs directory?

Also, what did you mean by

Does it happen when parallel instances is not enabled, or enabled but only with aws-iot-greengrass snap (without the _prime one)?

I enabled parallel instances with sudo snap set system experimental.parallel-instances=true. Are you saying that it’s possible to target one instance?

Thanks for the logs. There seems to be an problem with order of layout operations indeed.

The simple case of $SNAP_DATA and $SNAP_COMMON looks correct:

$ touch /var/snap/aws-iot-greengrass/common/non-instance-common
$ touch /var/snap/aws-iot-greengrass/current/non-instance-current
$ touch /var/snap/aws-iot-greengrass_prime/
44/      common/  current/ 
$ touch /var/snap/aws-iot-greengrass_prime/common/prime-common
$ touch /var/snap/aws-iot-greengrass_prime/current/prime-current
$ snap run --shell aws-iot-greengrass_prime.greengrassd
root@dec040744-973833:/home/spread-mini/mini/hello# ls $SNAP_DATA
hook-logs  prime-current
root@dec040744-973833:/home/spread-mini/mini/hello# ls $SNAP_COMMON
prime-common
root@dec040744-973833:/home/spread-mini/mini/hello# exit
$ snap run --shell aws-iot-greengrass.greengrassd
root@dec040744-973833:/home/spread-mini/mini/hello# ls $SNAP_DATA
hook-logs  non-instance-current  user-certs
root@dec040744-973833:/home/spread-mini/mini/hello# ls $SNAP_COMMON
non-instance-common

However, looking at the contents, debug logs and findmnt output the layout operations referencing $SNAP_DATA are applied to early, before /var/snap/aws-iot-greengrass_prime -> /var/snap/aws-iot-greengrass mapping is established.

I’ll keep digging and will update this topic once I have a fix. We’ve just tagged 2.48.1, but I should have something ready for 2.48.2 (or edge as soon as it lands in master).

1 Like

I’ve filed a bug to track this: https://bugs.launchpad.net/snapd/+bug/1906821

1 Like

The PR is up https://github.com/snapcore/snapd/pull/9751

1 Like

Thank you! I’ll try again when the code change goes into edge.

The PR has landed in master. The fix should appear in edge with the next rebuild of the snapd snap.

That fixed the issue. Thank you very much!

1 Like

Hi, could this be related to Problem with content interface and parallel install ? Unfortunally the latest snapd die not fix it for me.