Aws-iot-greengrass 1.8.x doesn't start on Ubuntu Core 16

We’re having trouble using the 1.8.x snap on Ubuntu Core 16 for raspberry pi (I had no trouble on plain Ubuntu 20 on aws ec2).

After installation (sudo snap install --channel=1.8.x aws-iot-greengrass), attempting to set the certs will fail (because the daemon tries to start immediately). Attempting to start the daemon fails.

We get snappy-debug messages such as these:

= AppArmor =
Time: Jan  8 01:04:13
Log: apparmor="DENIED" operation="open" profile="/usr/lib/snapd/snap-confine" name="/sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd/" pid=4722 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd/ (read)
Suggestion:
* adjust program to not access '/sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd/'

The filepath /sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd/ shows up repeatedly, which is strange given that owner /old_rootfs/sys/fs/cgroup/{,**} r, would catch it.

In case the /old_rootfs prefix was preventing a match, we added a line in the AppArmor profile exactly for that line:

owner /sys/fs/cgroup/devices/snap.aws-iot-greengrass.greengrassd r,

We used apparmor_parser to reload it, but we still got the error message.

We didn’t find the Greengrass logs under $SNAP_DATA/ggc-writable even though populating that directory is one of the first things Greengrass does on startup. That and the fact that we couldn’t run --shell into the snap indicates that there’s something fundamentally broken with the way permissions are granted.

Admittedly, the 1.8.x channel no longer receives new snaps, but its latest snap is supposed to have basic functionality.

AWS IoT Greengrass also provides containerized environments that run the AWS the AWS IoT Greengrass Core software on your core device, but it doesn’t 3.0.161.0 2996 latest/stable aws✓ classic core 16 -2.48 10444 latest/stable For examples, if you specify --channel= 1.8 . x

I built an image using this model: https://github.com/snapcore/models/blob/master/ubuntu-core-16-amd64.model but can’t reproduce the problem. I think we’ll need a bit more logs from the device. Specifically:

  • snap list
  • snap changes
  • snap change <id> of the install change that failed
  • journal contents

Also, can you double check that aa-status lists a profile for snap-confine? There could be more than one if the snapd snap is installed.

The odd thing here is that it’s snap-confine which is trying to read the cgroups, not Greengrass. We aren’t even getting to the point where greengrassd runs since snap-confine is stuck and dies. This is especially peculiar given that snap-confine should also be allowed to read these cgroups paths:

Can you provide the output of

sudo SNAP_CONFINE_DEBUG=1 snap run aws-iot-greengrass.greengrassd

Also what is snap version?

Sorry for the delay. My coworker’s dog ate her raspberry pi last week.

We don’t see the snap-confine error anymore, but we’ve seen this error on multiple people’s Ubuntu 16 machines (armhf and x86):

= AppArmor =
Time: Jan 11 22:50:09
Log: apparmor="ALLOWED" operation="open" profile="snap.aws-iot-greengrass.greengrassd" name="/sys/block/" pid=2147 comm="mount" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /sys/block/ (read)
Suggestion:
* adjust program to not access '/sys/block/'

That log is with --devmode, which is why AppArmor allowed it. However, it prevents the daemon from starting in strict mode. Adding /sys/block/ r, to the AppArmor profile fixes the issue.

As a note, we also tested on Ubuntu Core 18 without seeing the error. It looks like this can only be experienced on Ubuntu Core 16.

I can reproduce the /sys/block denial in an Ubuntu Core 16 VM with revision 4 on amd64 but the daemon still runs and my lambdas are running properly for me. I’m fairly certain it’s just noise given that it’s from the mount process.
I provided my certs via snap set aws-iot-greengrass gg-certs=$(pwd)/certs.tgz and the daemon started up properly, and after deploying my group with the hello-world python lambda I could see the messages sent to the Test subscription.

We haven’t tested on a VM, but that error has consequences in strict mode on at least two devices: a raspberry pi and a dell edge gateway device.

On both devices, editing the AppArmor profile unblocks the daemon startup. This definitely looks like an interface issue.

Can you provide any more logs from i.e. greengrassd’s private logs or the snap service logs when this happens?

If I recall correctly, there were no Greengrass logs because the permission denial occurs before GG can create its logs.

However, I can double-check. There were no Greengrass logs. I’ll have to see about the snap service logs.

Are the “snap service logs” simply the output of snap services?

You can use snap logs -n=all aws-iot-greengrass

We’ve had some confusing experiments on our end.

The permission error doesn’t always occur. As far as we can tell, it might be avoided by creating ggc_user/ggc_group ahead of time, even if GG isn’t going to use it. It might also be avoided if the group has at least one successful deployment (even if it is on another machine).

For now, we’ve solved the issues being faced, but we highly recommend that people use the supported versions of the GG snap.

Is there anything special about how the snap is installed, is it installed with --dangerous perhaps or with any interfaces disconnected ?

No, we verified that the required interfaces were connected on the machines involved.