AppArmor denials for DBus.ObjectManager [network-manager]

My snap has a backend service which needs to interact with the network-manager snap through nmcli. I have specified the plug network-manager which is connected to network-manager:service. All is working well in --devmode. When I put my snap in strict confinement, I get the following AppArmor denials:

= AppArmor =
Time: Feb  6 15:34:45
Log: apparmor="DENIED" operation="dbus_method_call"  bus="system" path="/org/freedesktop" interface="org.freedesktop.DBus.ObjectManager" member="GetManagedObjects" mask="send" name=":1.14" pid=24554 label="snap.my-snap.servicename" peer_pid=1525 peer_label="unconfined"
DBus access

The backend service is a Rails app that calls network-manager's nmcli, which is also bundled in my snap as explained in Access nmcli command from network-manager via interface. I also confirmed that the AppArmor rules in /var/lib/snapd/apparmor/profiles/snap.network-manager.networkmanager allow DBus traffic for my snap’s applications that implement the network-manager interface.

apps:
  backend:
    command: sh $SNAP/bin/start-stack.sh # <-- starts the Rails app, nothing fancy
    daemon: simple
    plugs: [network, network-bind, network-control, network-manager]

# ...
backend:
  stage-packages:
      - libatm1
      - libxtables12
      - ruby
      - sqlite3
      - wireless-tools
      - network-manager
snap interfaces my-snap
Slot                     Plug
# ...
network-manager:service  my-snap:network-manager,network-manager:nmcli

I think the denials didn’t occur when I was building the snap with snapcraft v2.x and LXD, but can’t confirm right now since I switched to snapcraft v3 with base18. Maybe it’s due to a network-manager version mismatch? network-manager snap uses xenial/1.2.2 and my base18 build pulls in 1.10.6-2ubuntu1 from the bionic repos.

~$ snap version
snap    2.37.1
snapd   2.37.1
series  16
kernel  4.4.0-1101-raspi2
user@localhost: ~$ nmcli --version # calling /snap/bin/nmcli
nmcli tool, version 1.2.2

sudo snap run --shell my-snap.backend

root@localhost: ~$ which nmcli
/snap/my-snap/x1/usr/bin/nmcli

root@localhost: ~$ nmcli --version
nmcli tool, version 1.10.6

If you run snappy-debug.security scanlog it might hint at possible resolutions. The dbus denial seems unrelated to network-manager because the requested dbus interface is org.freedesktop.DBus.ObjectManager which I think is not provided by NM.

The error output actually comes from snappy-debug.security scanlog – but unfortunately, no suggestions are provided. I also thought it would be unrelated to NM at first, but the error pops up each time my backend application tries to interact with NM.

What’s more problematic is that running any nmcli command from within my strict confined snap (e. g. sudo my-snap.app) results in this error when called directly:

Error: Could not create NMClient object: GDBus.Error:org.freedesktop.DBus.Error.AccessDenied: An AppArmor policy prevents this sender from sending this message to this recipient; type="method_call", sender=":1.1910" (uid=0 pid=24265 comm="/snap/my-snap/x1/bin/nmcli -f NAME c show -" label="snap.my-snap.nmcli-cmd (enforce)") interface="org.freedesktop.DBus.ObjectManager" member="GetManagedObjects" error name="(unset)" requested_reply="0" destination=":1.13" (uid=0 pid=1556 comm="/usr/sbin/NetworkManager --no-daemon " label="unconfined").

As you can see, my nmcli-cmd app runs as root and calls the bundled nmcli.

The nmcli-cmd app is a shell script:

#!/bin/sh
$SNAP/bin/nmcli -f NAME c show --active

I’ve also dis- and reconnected the network-manager:service interface multiple times, to no avail.

Could it in fact be the nmcli version mismatch? Would sending DBus calls directly work around this problem? I’ve ran some quick tests with dbus-send and they do not seem to throw errors, but I’m completely new to DBus calls.

Answering my own question for reference:

TL;DR: Mismatching versions of nmcli (or the whole network-manager package) seem to cause DBus AppArmor denials that are somewhat hard to debug.

I’ve put together two minimal snaps to test if the version mismatch between network-manager's nmcli and my snap’s nmcli from the bionic repos might be the cause. Also installed is the network-manager snap version 1.2.2-22 (rev 379) from stable. Host system is Ubuntu 18.04.5 LTS.

One snap has a core16 base:

name: nmclitest16
base: core16
grade: devel
confinement: strict

apps:
  version:
    command: nmcli --version
    plugs: [network-manager]
  test:
    command: nmcli c
    plugs: [network-manager]

parts:
  network-manager:
    plugin: nil
    stage-packages: [network-manager, libatm1]

Output of the commands:

user@host:~$ nmclitest16.version 
nmcli tool, version 1.2.6

user@host:~$ nmclitest16.test
# prints the expected list of connections

The other snap with a core18 base:

name: nmclitest18
base: core18
grade: devel
confinement: strict

apps:
   # same as above

parts:
  network-manager:
    plugin: nil
    stage-packages: [network-manager, libatm1, libxtables12]
user@host:~$ nmclitest18.version 
nmcli tool, version 1.10.6

user@host:~$ nmclitest18.test
Error: Could not create NMClient object: GDBus.Error:org.freedesktop.DBus.Error.AccessDenied: An AppArmor policy prevents this sender from sending this message to this recipient; type="method_call", sender=":1.8054" (uid=1000 pid=13488 comm="nmcli c " label="snap.nmclitest18.test (enforce)") interface="org.freedesktop.DBus.ObjectManager" member="GetManagedObjects" error name="(unset)" requested_reply="0" destination=":1.13" (uid=0 pid=1556 comm="/usr/sbin/NetworkManager --no-daemon " label="unconfined").

The network-manager snap has a 1.10/{beta, edge} track, but only for amd64, so I can’t test it on my armhf target. Attempting to install this version on my main Ubuntu host err’s out in the post-refresh hook with This snap is only supported on Ubuntu Core 18.

@tobias certainly you should use the same nmcli version as the NM you are using, or maybe try to use directly the DBus interface for NM.

In any case we plan to release the NM snap 1.10 for armhf soon.

Yes, in hindsight it makes sense that different versions are bound to error :slight_smile: What threw me off was that the AppArmor denials are referencing the org.freedesktop.DBus.ObjectManager interface – don’t know much about DBus, but at first it looked like something completely unrelated.

I tested NM DBus calls from within my snap using dbus-send which worked fine, but replacing simple nmcli calls with their DBus equivalent would require quite some refactoring in my application (of which I am the sole developer at the moment). So I’ll wait for the 1.10 snap release for now :slight_smile: Thanks in advance for this, and thank you for your work!

@tobias fyi there are now armhf snaps for NM 1.10 available at 1.10/edge and 1.10/beta channels.

@tobias - do things work with the newer network-manager @abeato mentioned? If not, can you provide exact steps on how to reproduce?

@abeato any plans to release network-manager 1.10 to stable? I am waiting for this for a longer time already and it would be very helpful for us (compare Network-manager problems on RPI 3)

Thanks

@getty23 sorry, but I cannot give dates. However, if you have tested the 1.10 edge/beta channels, could you please comment on that? Did you have any issues?

It took me a lo(oooooo)ng time to finally test network-manager 1.10/stable with our gadget stack, so apologies for the belated reply. I’ve since migrated our application to use NetworkManager’s D-Bus API instead of nmcli, which works fine with both the default and 1.10 tracks of network-manager. So we don’t need to advise existing gadget users to switch to the 1.10 track before we roll out the core18-based update for our main snap.

I was hoping the switch to NM 1.10 might provide more insight for some WiFi issues we’re experiencing on the Pi 3B+ (fails to join 802.11ac networks reliably even though the 3B+ hardware is capable of this), but those seem to stem from bugs in the Broadcom drivers. Did I already mention that low-level WiFi debugging isn’t my field of expertise :sweat_smile:

But as far as I have tested it, the 1.10/stable channel works just fine. Thanks @abeato & Co!

How do i run network manager through dbus? Do you have any examples that i can refer to?

For starters, you can check out its D-Bus API reference to learn about the available objects and their properties/methods. The link points to the current stable release, please note that some features might not be available in the currently available snap revisions.

Inside a snap, you have access to dbus-send, and Ubuntu Core ships with busctl which provides a few more options to discover and experiment with the running services. If you have an application inside your snap which should utilize D-Bus, there are some wrapper libraries to facilitate D-Bus operations, and for Python, there are even dedicated network-manager D-Bus packages. See https://medium.com/@f.i.rabby/python-network-manager-tutorial-c504cdb8fc76