Raspberry Pi 3 Ubuntu Core 18 network-control interface issue

snapd
1

Hey there!

I’ve encountered some strange behavior when running my snap on a Raspberry Pi 3 with Ubuntu Core 18.

To demonstrate this behavior, I created the following snap declaration:

name: mytest
base: core18
version: '0.1'
summary: Single-line elevator pitch for your amazing snap
description: |
  This is my-snap's description. You have a paragraph or two to tell the
  most important story about your snap. Keep it under 100 words though,
  we live in tweetspace and your description wants to look good in the snap
  store.

grade: devel
confinement: strict

apps:
  add:
    command: bin/ip tuntap add mode tun tun10
    adapter: full
    plugs:
      - network-control
  delete:
    command: bin/ip link delete tun10
    adapter: full
    plugs:
      - network-control
    
parts:
  mytest:
    plugin: nil
    stage-packages: 
      - iproute2

The snap uses the network-control interface to add and remove a tun device.

Without attaching the network-control interface, mytest.add responds with

Cannot open netlink socket: Operation not permitted

When connecting the interface, the action is executed as intended on every machine I tested, except the Raspberry Pi 3 running Ubuntu Core 18. Here, another error is displayed

open: Operation not permitted

I tested the following configurations (clean images with refreshed snaps):

  • Ubuntu 19.10 amd64 desktop (working)
  • Ubuntu 19.10 server armhf Raspberry Pi 3 (working)
  • Ubuntu Core 18 amd64 (working)
  • Ubuntu Core 18 armhf Raspberry Pi 3 (NOT working)
  • Ubuntu Core 18 arm64 Raspberry Pi 3 (NOT working)

Is this a bug in snapd? Or is this somehow intended behavior?

2

I can at least say that this seems very much like a bug in snapd, or at least with something Ubuntu Core specific.

3

Thank you for your reply, Ian.
Is there anything I can do to get this resolved quickly? Should I report the problem anywhere else? I’m not sure if this is really a snapd problem or if the Raspberry Pi Core image has a bug, that is why I did not yet create an issue on https://bugs.launchpad.net/snappy/

4

FYI, I looked into this and I can confirm.

In strict mode with network-control connected, there are no sandbox denials but there is the Cannot open netlink socket: Operation not permitted with sudo mytest.add. In devmode, it works. Deleting the device with sudo mytest.delete and going back to strict mode, it works.

What is happening is that on boot, the tun module is not loaded and it isn’t autoloading from the strict mode snap:

$ sudo reboot
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu Core 18"

$ snap list|grep pi
pi            18-1                  17     18-pi3    canonical*  gadget
pi-kernel     4.15.0-1053.57        69     18-pi3    canonical*  kernel

$ lsmod|grep tun
$
$ sudo snap install --dangerous ./mytest_0.1_armhf.snap
$ sudo mytest.add
open: Operation not permitted
$ lsmod|grep tun

$ sudo modprobe tun  # load the module manually and it works
$ sudo mytest.add
$ sudo mytest.delete

$ sudo rmmod tun  # unload the module and it fails again
$ sudo mytest.add
open: Operation not permitted

On the pi3, ‘tun’ is compiled as a module (which isn’t autoloading under strict mode):

$ grep 'CONFIG_TUN=' /snap/*kernel/current/config*
CONFIG_TUN=m

Strace show us:

openat(AT_FDCWD, "/dev/net/tun", O_RDWR|O_LARGEFILE) = -1 EPERM (Operation not permitted)

On a pi2, core16, things work as expected:

$ snap list|grep pi
pi2-kernel    4.4.0-1127.136        105   stable     canonical*  kernel
pi3           16.04-0.7             32    stable     canonical*  gadget

$ lsmod|grep tun
$ sudo snap install --dangerous /tmp/mytest_0.1_armhf.snap
$ sudo snap connect mytest:network-control
$ sudo mytest.add
$ sudo mytest.delete
$ lsmod|grep tun
$

‘tun’ is compiled into the kernel. This is confirmed with:

$ grep 'CONFIG_TUN=' /snap/*kernel/current/config*
CONFIG_TUN=y

On amd64 core16, ‘tun’ is also compiled into the kernel:

$ grep 'CONFIG_TUN=' /snap/*kernel/current/config*
CONFIG_TUN=y

With the module unloaded, inspected the snap’s device cgroup and discovered that c 10:200 rwm was not listed in /sys/fs/cgroup/devices/snap.mytest.add (ls -l /dev/net/tun shows it is crw-rw-rw- 1 root root 10, 200 Jan 10 14:02 /dev/net/tun). If I modprobe tun then c 10:200 rwm appears.

The symptom of this bug is that we are seeing a device cgroup denial.

What is happening is that /dev/net/tun does not get udev tagged until the module is loaded. The snap is accessing /dev/net/tun directly, but because the udev tagging didn’t happen, this isn’t yet in the device cgroup and the open fails. Because the open failed, the module doesn’t autoload.

First off, we probably should build the tun module into the kernel (CONFIG_TUN=y), which is in alignment with our other kernels (verified on core16/amd64, core16/armhf, are16/arm64, classic 16.04/amd64, classic 18.04/amd64). This will side-step the issue with CONFIG_TUN=m.

To support CONFIG_TUN=m, snap-confine should be adjusted to unconditionally add /dev/net/tun to the device cgroup (if it exists). AppArmor mediates access to /dev/net/tun already, so this is safe. I’ll add this to my next batch of policy updates.

1 Like
5

FYI, this is https://bugs.launchpad.net/snappy/+bug/1859084

6

Thank you so much for your quick and thorough investigation. :smiley:

7

For those who are hitting this issue (like I am with the wireguard snap): until the upstream issues get fixed in pi-kernel and/or snapd, a workaround is to create a file /etc/modules-load.d/tun.conf with the contents tun.

1 Like
8

FYI, https://github.com/snapcore/snapd/pull/8233

1 Like
9

Thanks for the fix, jdstrand.

10

As of snapd 2.45, the /etc/modules-load.d/tun.conf workaround is no longer needed.