How to modify name resolution parameters from a snap?

I am trying to enable the snap of a VPN client (openfortivpn) to modify name resolution parameters (DNS servers, domain search list). Currently and for historical reasons openfortivpn attempts to:

  1. run resolveconf (not the one from systemd-resolved which is conceptually broken as far as I can see but the one from resolvconf or openresolv),
  2. modify /etc/resolv.conf directly.

I have two questions:

  1. I fully understand that the above method is not the best for recent Linux distributions. For example on systems that have systemd, openfortivpn should instead talk to systemd-resolved. Actually I would be very much interested in a portable (across Linux distributions) method to modify name resolution parameters, if you know of any.
  2. All my attempts to build a snap of openfortivpn in its current state have failed, as running /sbin/resolvconf or modifying /etc/resolv.conf directly both break confinement - unless I am missing an interface. Can you help me? See my attempts below.

Run built-in /sbin/resolveconf from core

I have tried with the core base and the /sbin/resolvconf from resolvconf bundled with core.

The sbin/resolvconf script fails with:

mkdir: cannot create directory '/run/resolvconf': Permission denied

and AppArmor complains:

Log: apparmor="DENIED" operation="mkdir" profile="snap.openfortivpn.openfortivpn" name="/run/resolvconf/" pid=32367 comm="mkdir" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
File: /run/resolvconf/ (write)
Suggestions:
* adjust program to use $SNAP_DATA
* adjust program to use /run/shm/snap.$SNAP_NAME.*
* adjust program to use /run/snap.$SNAP_NAME.*
* adjust snap to use snap layouts (https://forum.snapcraft.io/t/snap-layouts/7207)
* add 'network-control' to 'plugs'

Any way around that?

Run /sbin/resolvconf as a stage dependency in core18

I have tried with the core18 base and the /sbin/resolvconf from resolvconf as a stage dependency of openfortivpn.

Again the sbin/resolvconf script fails with:

mkdir: cannot create directory '/run/resolvconf': Permission denied

and AppArmor complains:

Log: apparmor="DENIED" operation="mkdir" profile="snap.openfortivpn.openfortivpn" name="/run/resolvconf/" pid=34887 comm="mkdir" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
File: /run/resolvconf/ (write)
Suggestions:
* adjust program to use $SNAP_DATA
* adjust program to use /run/shm/snap.$SNAP_NAME.*
* adjust program to use /run/snap.$SNAP_NAME.*
* adjust snap to use snap layouts (https://forum.snapcraft.io/t/snap-layouts/7207)
* add 'network-control' to 'plugs'

Again any way to work around the confinment?

Modify /etc/resolv.conf directly, with both core and core18

When the openfortivpn snap attempts to modify /etc/resolv.conf directly, it fails with :

WARN:   Could not open /etc/resolv.conf (Permission denied).

and AppArmor complains:

Log: apparmor="DENIED" operation="capable" profile="snap.openfortivpn.openfortivpn" pid=33520 comm="openfortivpn" capability=1  capname="dac_override"
Capability: dac_override
Suggestions:
* adjust program to not require 'CAP_DAC_OVERRIDE' (see 'man 7 capabilities')
* add one of 'log-observe' to 'plugs'
* do nothing if program otherwise works properly

I doubt log-observe can help to write into /etc/resolv.conf. Any clue how to allow a snap to modify /etc/resolv.conf?

It’s unclear to me if this is sufficient for you, but you could try the system-files interface for /etc/resolv.conf. See https://snapcraft.io/docs/system-files-interface

randomly modifying /etc/resolv.conf directly will likely clash with systemd-resolved … on core systems you should rather use network-setup-control and modify/generate/apply netplan config changes or install network-manager and use nmcli to adjust dns settings.

1 Like

The problem is that randomly modifying /etc/resolv.conf is or was the only portable solution across all Linux distributions. Until recently, clashes were a theoretical problem but in practice it would work.

However:

  • Recently clashes did start appearing, for example on Ubuntu 20.04 servers.
  • We need to support split DNS, more than 2 DNS servers, etc.

For those reasons we have been aware for quite some time that modifying /etc/resolv.conf can only be a last resort solution for old Linux distributions, and that resolvconf is not the solution that it appeared to be at some point, mostly because of systemd-resolvd which ships with a broken resolvconf that breaks the promise that it should work on any system with any back-end - a conscious choice and deliberate sabotage?

It looks like we’re still missing something portable across Linux distributions. Having something portable across core systems is progress, but we still need to be portable across Linux distributions. I’ll see what we can do, perhaps we’ll end up with specific code for snaps.

I’ll have a look at netplan - I wasn’t aware it might be a portable solution. Is it available on CentOS and Fedora machines?

And are you certain about NetworkManager? Ubuntu 20.04 servers rely on systemd-networkd while Ubuntu 20.04 workstation rely on NetworkManager. How can NetworkManager be a solution on servers?

Finally how do I decide whether to use netplan, NetworkManager of systemd-networkd? It looks like even core systems don’t offer a portable solution.

heh, definitely not if your distribution used any tool that manages this file.

if you had resolvconf installed like ubuntu did since 2012 by default, this file was managed by it … to make propely used persistent changes to the system you always had to edit files in /etc/resolvconf/resolv.conf.d/ …

resolvconf simply got replaced by another tool, the fact that you should not edit /etc/resolv.conf directly but modify the configuration of the tool that manages it has not changed though.

also:

$ grep ^# /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

(from a 16.04 system)

:slight_smile:

On the contrary, definitely yes: in combination with resolvconf that was the only portable and working solution - perhaps not in theory but in practice it did work despite the theoretical clashes with the tools that manage name resolution. Unfortunately our bad hack has started failing - I do acknowledge it is a bad hack and I do want to get rid of it.

We don’t need persistent changes, only temporary changes for as long as the VPN client is running. That’s why our first choice is resolvconf. There’s a caveat though:

if you had resolvconf installed […]

That’s the heart of our problem. We do not get to choose what is installed. End-users install openfortivpn on a variety of systems that vary wildly, for example:

  • on Fedora some users keep systemd-resolved disabled (but installed) by default and others enable systemd-resolved,
  • Ubuntu 20.04 servers use systemd-networkd while workstations use NetworkManager.

Exactly. The problem remains how to identify the tool that manages name resolution on a given Linux machine or even a given core system. For example on Fedora 31 systemd-resolved is installed but it is not active by default, and resolvconf is broken by default (it’s the one bundled with systemd-resolved).

Like OpenVPN, we may end up externalizing routing and name resolution changes into a script. We wouldn’t endorse any specific script and it would be up to packagers to write/modify/maintain a script that works with each release of their Linux distribution.

However the external script trick doesn’t work for snaps. Snaps still need to operate on and modify a variety of host systems. As s snap packager I’d have to write an external script to detect which tool is in charge and interact with that tool to modify name resolution parameters of the host system. Besides if I could write such as an external script, I could as well rewrite it in C and directly integrate it in openfortivpn (both solutions have their pros and cons which I won’t discuss here).

As long as there is no method to identify which tool is in charge and interact with any of the detected tools, It feels impossible to create a snap of VPN clients. I think it’s no coincidence Canonical provide an OpenVPN server snap but no OpenVPN client snap that I know of.

I’ve read and boldly disregarded that - as long as modifying /etc/resolv.conf was the only portable and working solution. I am aware this is a bad hack, am unhappy with it, and have precisely been trying to find a portable and correct solution.

The caveat is the above term “you”. Who should install NetworkManager? End-users on their host system or we snap packagers as a staging dependency?

  • Some end-users do not want to install NetworkManager. Ubuntu 20.04 servers use systemd-networkd instead by default so we cannot ask server administrators to install Networkmanager. Some Linux distributions just don’t use systemd anyway - but perhaps snapd has not been ported to these systems.
  • I have shown in my initial post that using the resolvconf bundled with core or adding resolvconf as staging dependency has not been working - perhaps the system-files interface can help though. I can try to add network-manager as a staging dependency, but I’m afraid it too will fail to modify the host system’s name resolution parameters. Unless I’m missing something?

sorry, I wasnt clear in my wording, with “on core systems” I meant: “on Ubuntu Core systems” … indeed for classic systems you should have a method for every common way out there, detect the way thats in use and manage it as needed …

@ogra By Ubuntu Core you mean the Ubuntu Core for IoT devices, don’t you? Yes, that would be one of the targets together with Ubuntu desktops and servers. I’m confident I’ll be able to handle these systems with resolvconf and the system-files interface in the short term. But snaps must also support Fedora, CentOS and other distributions. I’ll keep trying to find a portable solution but the fact that OpenVPN have failed to find one and left packagers or even end-users pick a script themselves is disturbing.

My intent is to try in that order:

  1. Determine whether systemd-resolvd is installed, enabled, and able to modify all back-ends.
  2. Failing to update back-ends using systemd-resolvd, try resolvconf from resolvconf or openresolv (avoid the one from systemd-resolvd which is hopelessly broken).
  3. If all else fails attempt to modify /etc/resolv.conf directly.

Especially item 1 is complex. For example on Fedora 31, systemd-resolvd is installed (which unfortunately puts its own broken resolvconf in the path), disabled by default, and once enabled changes to systemd-resolvd are not reflected in the C library.

Hopefully items 2 and 3 can handle SysVinit and OpenRC systems…

I think there are two main questions, that I haven’t been able to address yet:

  1. Can we talk directly to systemd-resolved when NetworkManager or systemd-networkd are in charge? Otherwise we will have to talk directly to NetworkManager or systemd-networkd which gets really complicated.
  2. How does sytemd-resolvd interact with other back-ends and how to detect the kind of interaction? For example in Fedora 33 sytemd-resolvd should be enabled by default and update other back-ends such as glibc.

snapd does currently require systemd being installed so i guess SysV and OpenRC are irrelevant for the moment … also i guess most systems using systemd will also have resolved installed so i’d rather check via the process-control plug if it is in the process list, checkng if it is installed wont be enough …

1 Like

I wasn’t aware that systemd is a dependency of snapd. This should perhaps be stressed in the documentation.

I can follow your advice and use the process-control interface to find if systemd-resolvd and/or systemd-networkd are running.

I read Enable/Disable systemd services and had a look at the dbus interface: it looks like snaps cannot initiate communication with the host system via systemd or D-BUS. I’m still trying to understand how I can get through confinement. I guess I have to read through all interfaces, but that’s clearly not enough. Understanding the details of confinment and how to interact with the host system is probably the toughest part of the learning curve.

I hit this problem while working on tailscale snap. tailscale tries to understand how the host system manages DNS and looks like it’s trying to read /etc/resolv.conf and doing some checks on it. (like searching for keywords, verifying stub resolver etc…)

I tried system-files interface with both read & write permissions but on my host system (Pop!_OS), it’s a symlink to /run/systemd/resolve/stub-resolv.conf – so I tried adding both:

plugs:
  hostfs-etc-resolv-conf:
    interface: system-files
    read:
      - /etc/resolv.conf
      - /run/systemd/resolve/stub-resolv.conf
    write:
      - /etc/resolv.conf
      - /run/systemd/resolve/stub-resolv.conf

It doesn’t matter if I only use top-level plugs keyword like this or add it under specific apps. I always get “permission denied” errors. :frowning:

Also, tailscale tries to ping the stub-resolver through DBus and it fails as well inside the strict confinement.

Hi there. I am fighting with the same problem for several days but without any luck :frowning:

Are there any updates? Is there any universal/portable solution for snapped VPN applications (with strict confinement) to control DNS settings?

Since the system-files interface (r/w access to /etc/resolv.conf) is usually not approved during the manual review, it looks like, there is no chance to publish a VPN application on the Snap Store.

For future reference, I gave an answer in the other thread.

The comment in this post that changing /etc/resolv.conf is a universal solution is not the case anymore as it doesn’t work on modern versions of Fedora anymore. (This thread is 2 years old, which is a lifetime in “Fedora years”)

1 Like