On the first boot (only), DNS requests from our snap fails. After the reboot, everything works just fine.
We’re using NetworkManager as a renderer, here are its logs of NetworkManager startup
I checked whether /etc/resolv.conf has correct content. It looks like being right. This is extracted from inside the snap service:
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
DNS name resolution from inside the python service running in the snap looks fine:
[ 896.413701] python3[5950]: urllib3.connectionpool Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xb3b812f8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/4504479311986688/store/
I tried to make a DBus call to resolve1, and it fails like this:
[ 874.864453] python3[5950]: screenly.networkprobe Cannot resolve hostname with resolved: g-io-error-quark: GDBus.Error:org.freedesktop.resolve1.NoNameServers: No appropriate name servers or networks for name found (36)
My theory is that on the first boot systemd-networkd configures the network and when NetworkManager takes over the network configuration after netplan apply and DNS settings break.
As I expected, plugging the ethernet cable after NM takes the network doesn’t break DNS name resolution anymore. So, I expect that it’s some kind of interference between networkd and NetworkManager.
Hi @renat2017 I am a bit confused by the problem description, you say that it is working inside the snap, but it is not if you do it from a shell?
Then I’m not sure what you mean with plugging the ethernet cable, it is not plugged originally and you are using wifi in the beginning? And when you plug the ethernet cable the problme is gone?
Hi @abeato, that’s how it looks like. Perhaps, it doesn’t have DNS inside the shell either, but I cannot check it because there is no user in our device.
What I can tell though, that the device can connect to an NTP server, which means that it actually can resolve domain names.
I can set up a user once device is provisioned, but then the problem goes away, because it happens just after the first boot, once device is provisioned.
I think that there is a conflict between networkd and network manager during the first boot. To check my theory I will try to systemd.mask systemd-networkd.service from the kernel command line.
I also will give you the output of snap list, but a little bit later.
As I expected - disabling systemd-networkd with the kernel command line fixes everything. Anyhow, I don’t like that solution in case that at some point in the future NetworkManager might be used as a frontend for networkd.
Ok, here are the facts I have now:
Device provisioning with Ethernet cable plugged in causes DNS failure.
After the reboot DNS works fine even if reboot was done with Ethernet.
Device provisioning without Ethernet cable, but plugging it in after the provisioning finishes doesn’t break DNS.
Disabling systemd-networkd.service with cmdline.txt (in our own gadget snap), and then provisioning with Etherned cable pluggied in doesn’t break DNS.
So I suggest that it’s some systemd-networkd vs NetworkManager conflict happening during the first boot.
Hi @renat2017, thanks for the detailed analysys. Something that bring my attention is that I see core20 and core22 bases installed, is this a UC20 or UC22 system? snap model --verbose | grep base: would give us the answer.
Note that if this is a UC22 system it is recommended that you install network-manager from the 22/stable channel, as it is needed to have it in sync with the base as both snaps need to interact for the networking configuration.
