Aws-iot-greengrass 1.8.x doesn't start on Ubuntu Core 16

zjoseal · January 8, 2021, 7:16am

We’re having trouble using the 1.8.x snap on Ubuntu Core 16 for raspberry pi (I had no trouble on plain Ubuntu 20 on aws ec2).

After installation (sudo snap install --channel=1.8.x aws-iot-greengrass), attempting to set the certs will fail (because the daemon tries to start immediately). Attempting to start the daemon fails.

We get snappy-debug messages such as these:

= AppArmor =
Time: Jan  8 01:04:13
Log: apparmor="DENIED" operation="open" profile="/usr/lib/snapd/snap-confine" name="/sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd/" pid=4722 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd/ (read)
Suggestion:
* adjust program to not access '/sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd/'

The filepath /sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd/ shows up repeatedly, which is strange given that owner /old_rootfs/sys/fs/cgroup/{,**} r, would catch it.

In case the /old_rootfs prefix was preventing a match, we added a line in the AppArmor profile exactly for that line:

owner /sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd r,

We used apparmor_parser to reload it, but we still got the error message.

We didn’t find the Greengrass logs under $SNAP_DATA/ggc-writable even though populating that directory is one of the first things Greengrass does on startup. That and the fact that we couldn’t run --shell into the snap indicates that there’s something fundamentally broken with the way permissions are granted.

Admittedly, the 1.8.x channel no longer receives new snaps, but its latest snap is supposed to have basic functionality.

Countryy · January 8, 2021, 8:57am

AWS IoT Greengrass also provides containerized environments that run the AWS the AWS IoT Greengrass Core software on your core device, but it doesn’t 3.0.161.0 2996 latest/stable aws✓ classic core 16 -2.48 10444 latest/stable For examples, if you specify --channel= 1.8 . x

mborzecki · January 8, 2021, 9:13am

I built an image using this model: https://github.com/snapcore/models/blob/master/ubuntu-core-16-amd64.model but can’t reproduce the problem. I think we’ll need a bit more logs from the device. Specifically:

snap list
snap changes
snap change <id> of the install change that failed
journal contents

Also, can you double check that aa-status lists a profile for snap-confine? There could be more than one if the snapd snap is installed.

ijohnson · January 8, 2021, 11:30am

The odd thing here is that it’s snap-confine which is trying to read the cgroups, not Greengrass. We aren’t even getting to the point where greengrassd runs since snap-confine is stuck and dies. This is especially peculiar given that snap-confine should also be allowed to read these cgroups paths:

github.com

snapcore/snapd/blob/master/cmd/snap-confine/snap-confine.apparmor.in

# Author: Jamie Strandboge <jamie@canonical.com>
#include <tunables/global>

@LIBEXECDIR@/snap-confine (attach_disconnected) {
    # Include any additional files that snapd chose to generate.
    # - for $HOME on NFS
    # - for $HOME on encrypted media
    #
    # Those are discussed on https://forum.snapcraft.io/t/snapd-vs-upstream-kernel-vs-apparmor
    # and https://forum.snapcraft.io/t/snaps-and-nfs-home/
    #include "/var/lib/snapd/apparmor/snap-confine"

    # We run privileged, so be fanatical about what we include and don't use
    # any abstractions
    /etc/ld.so.cache r,
    /etc/ld.so.preload r,
    /{,usr/}lib{,32,64,x32}/{,@{multiarch}/{,atomics/}}ld-*.so mrix,
    # libc, you are funny
    /{,usr/}lib{,32,64,x32}/{,@{multiarch}/{,atomics/}}libc{,-[0-9]*}.so* mr,
    /{,usr/}lib{,32,64,x32}/{,@{multiarch}/{,atomics/}}libpthread{,-[0-9]*}.so* mr,

This file has been truncated. show original

Can you provide the output of

sudo SNAP_CONFINE_DEBUG=1 snap run aws-iot-greengrass.greengrassd

ijohnson · January 8, 2021, 11:27am

Also what is snap version?

zjoseal · January 12, 2021, 12:50am

Sorry for the delay. My coworker’s dog ate her raspberry pi last week.

We don’t see the snap-confine error anymore, but we’ve seen this error on multiple people’s Ubuntu 16 machines (armhf and x86):

= AppArmor =
Time: Jan 11 22:50:09
Log: apparmor="ALLOWED" operation="open" profile="snap.aws-iot-greengrass.greengrassd" name="/sys/block/" pid=2147 comm="mount" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /sys/block/ (read)
Suggestion:
* adjust program to not access '/sys/block/'

That log is with --devmode, which is why AppArmor allowed it. However, it prevents the daemon from starting in strict mode. Adding /sys/block/ r, to the AppArmor profile fixes the issue.

As a note, we also tested on Ubuntu Core 18 without seeing the error. It looks like this can only be experienced on Ubuntu Core 16.

ijohnson · January 13, 2021, 3:44pm

I can reproduce the /sys/block denial in an Ubuntu Core 16 VM with revision 4 on amd64 but the daemon still runs and my lambdas are running properly for me. I’m fairly certain it’s just noise given that it’s from the mount process.
I provided my certs via snap set aws-iot-greengrass gg-certs=$(pwd)/certs.tgz and the daemon started up properly, and after deploying my group with the hello-world python lambda I could see the messages sent to the Test subscription.

zjoseal · January 14, 2021, 6:21pm

We haven’t tested on a VM, but that error has consequences in strict mode on at least two devices: a raspberry pi and a dell edge gateway device.

On both devices, editing the AppArmor profile unblocks the daemon startup. This definitely looks like an interface issue.

ijohnson · January 14, 2021, 6:23pm

Can you provide any more logs from i.e. greengrassd’s private logs or the snap service logs when this happens?

zjoseal · January 14, 2021, 6:30pm

If I recall correctly, there were no Greengrass logs because the permission denial occurs before GG can create its logs.

~~However, I can double-check.~~ There were no Greengrass logs. I’ll have to see about the snap service logs.

zjoseal · January 14, 2021, 6:40pm

Are the “snap service logs” simply the output of snap services?

ijohnson · January 14, 2021, 6:54pm

You can use snap logs -n=all aws-iot-greengrass

zjoseal · January 19, 2021, 5:41pm

We’ve had some confusing experiments on our end.

The permission error doesn’t always occur. As far as we can tell, it might be avoided by creating ggc_user/ggc_group ahead of time, even if GG isn’t going to use it. It might also be avoided if the group has at least one successful deployment (even if it is on another machine).

For now, we’ve solved the issues being faced, but we highly recommend that people use the supported versions of the GG snap.

ijohnson · January 19, 2021, 6:53pm

Is there anything special about how the snap is installed, is it installed with --dangerous perhaps or with any interfaces disconnected ?