During recent work on some web based Core appliance images I discovered that none of our snaps can actually properly use mdns, i.e. firefox, vlc or chromium can not use http://foo.local urls at all.
even if you allow them to connect avahi-observe or avahi-control they will not be able to resolve anything using mdns. this seems to be due to the fact that /etc/nsswitch.conf comes from the core snap for any snap used.
the /etc/nsswitch.conf inside the core snap has:
hosts: files dns myhostname
while for proper mdns support the line would need to look like:
hosts: files mdns4_minimal [NOTFOUND=return] dns
adding this line by default and not having mdns/avahi/bonjour in use might cause DNS lookup delays though (additionally you might need to stage libavahi-client3 in your snap to have support for the mdns protocol at all).
we could indeed make all snaps use a layout here and re-map their own nsswitch.conf on top of the core snaps /etc/nsswitch.conf (sadly glibc has this path hardcoded, not overridable and former upstream discussions seem to suggest that an attempt to change this will not be approved) but that sounds pretty painful to do for every snap that could potentially use .local resolution.
We’re actually having more dire (and possibly related) issues with this in Fedora. I’ve had reports about the SELinux context and in some cases the contents of the file being clobbered, which winds up breaking everything. One example of this is RH#1596753. Another is RH#1612430.
I’m not sure what’s going on here, as this isn’t supposed to be happening at all. But it’s causing problems.
My best guess is that it’s an application that would normally be prevented from doing so with confinement, but because the confinement is very incomplete when Ubuntu AppArmor is unavailable, it was permitted anyway. But the current bug reports are about its SELinux context being wiped out, which breaks everything anyway.
I wondered, from my lack of experience with such matters (therefore, take this with a grain of salt), if selinux is not correctly partitioned via the mount namespaces that snapd sets up.
Consider that you’re seeing the selinux context of the file changing on disk the moment a snap is started. As @ogra pointed out the core snap is mounted such that it provides a file at /etc/nsswitch.conf for the snapped application’s use. This file will inevitably have a different selinux context, or none at all. If selinux isn’t correctly partitioned by the mount namespace which provides for the duplication of file locations (the host system has a file at what it thinks is /etc/nsswitch.conf and the snap also has a file at what it thinks is /etc/nsswitch.conf) then selinux will not know which of the files is the correct source of truth for the context metadata.
In general +1 on making /etc/nsswitch.conf writable, but let’s be careful…
Do we know how system would behave in cases where /etc/nsswitch.conf is modified but client snap has no avahi interface?
Modifying /etc/nsswitch.conf will enable mdns for entire system, but only snaps with connected avahi interface can perform search. In order to make mdns work in client snap, following has to be met:
modify /etc/nsswitch.conf
include avahi mdns libraries added to /etc/nsswitch.conf such as libnss_mdns.so
avahi snap installed, or deb on classic
client snap has to have connected interface to avahi snap (or system if avahi is installed through deb)
ideal situation would be having /etc/nsswitch.conf per snap, but at same predictable location, which start overlapping with layout feature though
To add extra complexity, we would also need writable /etc/mdns.allow, this config is needed if we want to enable mdns for other than .local domain
i was not proposing to make anything writable but to ship with a different default instead, so that snaps shipping libnss_mdns.so have a chance of working at all … /etc/nsswitch.conf should stay readonly but allow to use mdns. the question was if this has any impact on DNS resolution speed of all snaps or not.
Right, in which case we might even consider shipping libnss_mdns.so libraries in core. Those libraries are only 64Bk anyway.
They suppose to only resolve when you are working with .local (unless configured otherwise with /etc/mdns.allow). So what is faster, failing to load library, or having loaded library and failing to connect to the socket…
May be something to measure.
And which one are you going to add there, mdns4_minimal, or mdns_minimal to cover also ipv6?
I guess something generic enough, as we can always use overlay if somebody wants to customise…
You can’t modify the version in the core* snap, as it is read-only. If you want to modify it for just your snap, you need to change the remmina snap to use a layout on /etc/nsswitch.conf to somewhere in $SNAP or $SNAP_DATA.
This old nscd workaround on the host side was actually posted by me, more than a year ago, but I do not remember how I discovered that. I even did not remember today when I discovered this problem that I had already worked around it earlier until I found this thread here on the snapcraft.io forum by Googling.
But now I have found a way to implement the solution presented in this thread on the Snap side. One simply creates an nsswitch,conf file with the mDNS support added to its hosts: line and mirrors it into /etc/nsswitch,conf by a layout: entry. In addition one installs the libnss-mdns package simply as a stage package, the libraries only need to be present, they do not need to get mirrored into the Core Snap.
In snapcraft.yaml it looks like this:
[...]
# Make resolution of ".local" host names (Zero-Conf/mDNS/DNS-SD)
# working: Mirror nsswitch.conf with added mDNS look-up support into
# /etc/nsswitch.conf
layout:
/etc/nsswitch.conf:
bind-file: $SNAP/etc/nsswitch.conf
[...]
parts:
[...]
utils:
plugin: nil
stage-packages:
- perl-base
override-prime: ""
nsswitchconf:
# Make resolution of ".local" host names (Zero-Conf/mDNS/DNS-SD)
# working: Extract the original nsswitch.conf file from the libc-bin
# package and add "mdns4_minimal [NOTFOUND=return]" to its "hosts:"
# line
plugin: nil
stage-packages:
- libc-bin
organize:
usr/share/libc-bin/nsswitch.conf: etc/nsswitch.conf
override-prime: |
set -eux
perl -p -i -e 's/^\s*hosts:(\s+)files/hosts:\1files mdns4_minimal [NOTFOUND=return]/' $SNAPCRAFT_STAGE/etc/nsswitch.conf
snapcraftctl prime
prime:
- etc/nsswitch.conf
after: [utils]
libnss-mdns:
# Make resolution of ".local" host names (Zero-Conf/mDNS/DNS-SD)
# working: Install the libnss-mdns libraries needed for mDNS host name
# look-up
plugin: nil
stage-packages:
- libnss-mdns
prime:
- lib/x86_64-linux-gnu/libnss_mdns*
[...]
Note also that your apps: need to plug one of the avahi-observe or avahi-control interfaces.
With this workaround my Snaps work correctly and actually resolve the mDNS .local host names, but note that this is very awkward to have to be done in every Snap which needs host name look-up, so the fix of the Core Snaps is urgently needed.
As we need the Core Snaps fixed as soon as possible as complete host name look-up support is essentially important, I have extended the Launchpad bug by the Core Snaps.
If you are going to ship a modified nsswitch.conf file, I would suggest starting with the version of that file from your base snap as a starting point rather than the one from the libc-bin deb. This is what the snap would otherwise see, and differs from the libc-bin version (in particular it enables the extrausers plugin, needed for user/group lookups on Ubuntu Core systems).
You probably don’t need to plug either avahi-observe or avahi-control to do simple .local hostname lookups. The libnss-mdns module does not speak Avahi’s D-Bus API, but instead uses a simpler special purpose protocol on /run/avahi-daemon/socket. Access to this socket is granted by the <abstractions/nameservice> AppArmor fragment, which is included with the network plug.
As far as enabling all this by default for all snaps, I suspect it wouldn’t be particularly intrusive. The mdns4_minimal plugin is 18KB before compression, and only responds to lookups for domains ending in .local and 169.254.x.x IP addresses. If the Avahi socket is missing or not listening for connections, then it is going to return immediately. It’s likely that this could all work seamlessly without any AppArmor policy modifications on both Core and Classic systems.
@jamesh Great, then we could expect a fix in the base Snaps soon?
For the workaround for the time being in the application Snaps I have a question. You write:
I have taken the nsswitch.conf from the package as I assumed that it is the stage packages are the from the same repository/Ubuntu version as from the base Snap used (core20 in my case).
Is the build of the Snap happening under the same base Snap as the execution of the Snap will take place? And how can I grab /etc/nsswitch.conf from the base Snap under which I am building to include an edited version of it in my Snap?
The scripts used to build the base snaps from the Ubuntu archive modify the file:
Snapcraft should install your base snap in the build environment, so you can probably access this as /snap/core20/current/etc/nsswitch.conf.
As for actually implementing mdns hostname lookup by default, I’d like to check whether it’s something that is likely to be accepted before doing the work.
we discussed it in the core16 times and back then it would not have been denied, though we didn’t do any tests on UC devices wether it potentially delays DNS lookups …
but indeed things changed in UC20 and the core20 snap has removed a lot of stuff for size reasons that you still had around in 16 and 18. might be that you need additional dependencies added for libnss-mdns.
The dependency on avahi-daemon is there because it gets the resolution from avahi-daemon, the libraru errors quickly and gracefully if avahi-daemon is not there as @jamesh told. So somehow in the build process of the base Snap one needs to override the dependency on avahi-daemon or remove/not prime avahi-daemon afterwards (in case the external, classically installed avahi-daemon of the system is supposed to be used).
I think mdns4_minimal is unlikely to trigger problems. From it’s readme file:
libnss_mdns{4,6,}_minimal.so (new in version 0.8) is mostly identical to the versions without _minimal. However, they differ in one way. The minimal versions will always deny to resolve host names that don’t end in .local or addresses that aren’t in the range 169.254.x.x (the range used by IPV4LL/APIPA/RFC3927.) Combining the _minimal and the normal NSS modules allows us to make mDNS authoritative for Zeroconf host names and addresses (and thus creating no extra burden on DNS servers with always failing requests) and use it as fallback for everything else.
So the plugin is going to be inert for most hostname lookups that don’t end in .local.
The plugin only links to libresolv.so.2 and libc.so.6, so the only additional library is the NSS plugin itself. The plugin doesn’t require any extra libraries because it doesn’t use Avahi’s full D-Bus API. Instead it speaks a simple line oriented protocol over the /run/avahi-daemon/socket socket. For example:
You will of course need Avahi installed for this lookup to succeed (either a traditional package on classic distros, or the snap on Ubuntu Core). But if Avahi is missing, it should return as soon as it fails to connect to the unix socket.
2020-05-01 already and no solution…