Snapd 2.66.1 update broke DNS

ryan.rubley · December 5, 2024, 10:53pm

We have hundreds of thousands of devices globally, all running Ubuntu, with tens of thousands running Ubuntu 22.04. Last week, these devices seemed to have auto-updated to snapd 2.66.1 and on roughly 30% of them, DNS was broken as a result, which is still around 10k broken devices people began to notice earlier this week.

For note, we use ifupdown, not netplan, and we do not really use any snap packages. It seems core20, lxd, and snapd are all installed and probably all auto-refreshed, which is not actually something we intended to allow. We have our own aptly mirrors for .deb packages and have all automatic updates disabled for that. Any update is validated internally and scheduled for updates on our own cadence.

Our DNS is configured in /etc/network/interfaces and the systemd-resolved will write these same values to /run/systemd/resolve/resolv.conf - and this file was changed at the exact same second as the snapd 2.66.1 update on every system we checked. Instead of having “nameserver ” lines, it simply got written with “# No DNS servers known.”

$ ls -l --time-style=full-iso /run/systemd/resolve/resolv.conf
-rw-r--r-- 1 systemd-resolve systemd-resolve 793 2024-11-26 16:23:24.044080817 -0500 /run/systemd/resolve/resolv.conf

From syslog:

Nov 26 16:23:22 HOST snapd[2120229]: AppArmor status: apparmor is enabled and all features are available
Nov 26 16:23:23 HOST snapd[2120229]: overlord.go:274: Acquiring state lock file
Nov 26 16:23:23 HOST snapd[2120229]: overlord.go:279: Acquired state lock file
Nov 26 16:23:23 HOST snapd[2120229]: patch.go:64: Patching system state level 6 to sublevel 1...
Nov 26 16:23:23 HOST snapd[2120229]: patch.go:64: Patching system state level 6 to sublevel 2...
Nov 26 16:23:23 HOST snapd[2120229]: patch.go:64: Patching system state level 6 to sublevel 3...
Nov 26 16:23:23 HOST snapd[2120229]: daemon.go:250: started snapd/2.66.1 (series 16; classic) ubuntu/22.04 (amd64) linux/6.0.0-1006-oem.
Nov 26 16:23:23 HOST kernel: [2865664.502867] loop6: detected capacity change from 0 to 8
Nov 26 16:23:23 HOST snapd[2120229]: daemon.go:353: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Nov 26 16:23:23 HOST systemd[1]: tmp-syscheck\x2dmountpoint\x2d4033360333.mount: Deactivated successfully.
Nov 26 16:23:23 HOST snapd[2120229]: backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
Nov 26 16:23:23 HOST snapd[2120229]: helpers.go:160: error trying to compare the snap system key: system-key versions not comparable
Nov 26 16:23:24 HOST systemd-udevd[2120407]: Using default interface naming scheme 'v249'.
Nov 26 16:23:24 HOST systemd-udevd[2120415]: Using default interface naming scheme 'v249'.
Nov 26 16:23:24 HOST systemd-udevd[2120399]: Using default interface naming scheme 'v249'.

In some cases, it seems simply restarting the systemd-resolved service would cause the file to be rewritten with the correct nameservers. In other cases, we had to ifdown && ifup to fix it. Of course, rebooting devices would also fix them.

Management is, of course, furious with snapd, and we will likely issue a global command to all devices to mask the snapd service entirely as a result, but I wanted to bring this issue some attention in case it impacted anybody else.

mborzecki1 · December 6, 2024, 7:00am

Hi,

I’m sorry you hit a problem. Can you answer a couple of questions to help me better understand how the problem could be related to snapd? Are those Ubuntu Core or classic Ubuntu installations? What snaps are installed in the system? How do you prevent snap updates? Are you aware of https://snapcraft.io/docs/managing-updates#p-32248-pause-or-stop-automatic-updates? Can you attach the output of /snap/snapd/current/usr/lib/snapd/snap-debug-info.sh ?

ryan.rubley · December 6, 2024, 7:36am

Our systems are based on the jammy-live-server iso, and are a somewhat stock Ubuntu 22.04 server LTS setup. We do not actually use snap or pay much attention to it until now. The snapd service just happens to be running by default and was never disabled. We will probably have to disable it now by masking the snapd service.

==================== SNAP VERSION ====================
snap    2.66.1
snapd   2.66.1
series  16
ubuntu  22.04
kernel  6.0.0-1006-oem

==================== SNAP WHOAMI ====================
email: -

==================== SNAP MODEL ====================
brand-id:   generic
model:      generic-classic
serial:     b57ac84d-3dfd-496d-978e-888294a69631
classic:    true
timestamp:  2017-07-27

==================== SNAP MODEL SERIAL ====================
brand-id:   generic
model:      generic-classic
serial:     b57ac84d-3dfd-496d-978e-888294a69631
timestamp:  2023-03-23
device-key-sha3-384: uqZ9ywFKtI7VH7B_WEvUh-ovt8uNScoP2yLKLJim_0BWzRk1JLV7qxT5fZN_qtS3
device-key: |
  AcbBTQRWhcGAARAAqw6Ek9PcCCu3y0kswfchx4hbylXK/PEmPxbrRIV3zzrKLxWtNG8aJwmStx/QlexkY0NnQ7xnaGIxxalSUi
  x0EIx7cfHS2X8lS1yGt7ko3DdF/IDx9lDXSO2C89WapsMqtG1IAmmhfYzm4jaJei0sUxFihDQ3kvDU3hSsk5ADAPNkvZX5qmua
  l+MV0iOWwSyv0zHfJE/rDv2ikOdNaC2p4QP4aa5jrJscf9fuJhLJSJJ3deLzoIueEa0kRfaCpEBO/A4d4go7/zA7jjg5t7/sSV
  u9xYreQJz8uJubu7PXHdcnnqMP1vgKYo8bgpLTIBlPoFWfQfVWrKyp16y4Y+OK/BbODqY6ZUMDibTOLHttnRxgS8K4w3UDjdXW
  C0A6ZdRYv5VPnJSJbL1MwJz1D9S6yB8zMnHVPM9chHSvSNUzEvaV/SF/ZtYdM5cpwlMYIKL/+2z22L+Klwoz3qxHy5b+v5O3r1
  r9Y8YpEA5my7JOIHiqzDibUhfO1P8SNUNxaL8jYHWv7XEvg9nDF8mpm1/gX+qEedsQ3Cw+nQIg9UWiSIaFV3Q652JH61tTkME3
  +WnwR2dv8fHQWsv7i3Y09778BpaWkLBP/u2ZdhmB8+nS6JXNDF0dKceTArz2nE/+xFkH4D1zxhCAirPJaK0AU/0nn/JoJtC7re
  D1IE7N9ENg0H0AEQEAAQ==

==================== SNAP LIST ====================
Name    Version        Rev    Tracking       Publisher   Notes
core20  20240705       2379   latest/stable  canonical✓  base,disabled
core20  20240911       2434   latest/stable  canonical✓  base
lxd     5.0.3-80aeff7  29351  5.0/stable/…   canonical✓  disabled
lxd     5.0.4-497fe1e  31333  5.0/stable/…   canonical✓  -
snapd   2.63           21759  latest/stable  canonical✓  snapd,disabled
snapd   2.66.1         23258  latest/stable  canonical✓  snapd

==================== SNAP SERVICES ====================
Service          Startup   Current   Notes
lxd.activate     disabled  inactive  -
lxd.daemon       disabled  inactive  socket-activated
lxd.user-daemon  disabled  inactive  socket-activated

==================== SNAP CONNECTIONS ====================
Interface       Plug                Slot             Notes
lxd-support     lxd:lxd-support     :lxd-support     -
network         lxd:network         :network         -
network-bind    lxd:network-bind    :network-bind    -
system-observe  lxd:system-observe  :system-observe  -

==================== PER-SNAP CONNECTIONS ====================

========== PER-SNAP core20 CONNECTIONS ==========

========== PER-SNAP lxd CONNECTIONS ==========
Interface       Plug                Slot             Notes
lxd             -                   lxd:lxd          -
lxd-support     lxd:lxd-support     :lxd-support     -
network         lxd:network         :network         -
network-bind    lxd:network-bind    :network-bind    -
system-observe  lxd:system-observe  :system-observe  -

========== PER-SNAP snapd CONNECTIONS ==========
Interface                         Plug                Slot                               Notes
account-control                   -                   :account-control                   -
accounts-service                  -                   :accounts-service                  -
acrn-support                      -                   :acrn-support                      -
adb-support                       -                   :adb-support                       -
allegro-vcu                       -                   :allegro-vcu                       -
alsa                              -                   :alsa                              -
appstream-metadata                -                   :appstream-metadata                -
audio-playback                    -                   :audio-playback                    -
audio-record                      -                   :audio-record                      -
autopilot-introspection           -                   :autopilot-introspection           -
avahi-control                     -                   :avahi-control                     -
avahi-observe                     -                   :avahi-observe                     -
block-devices                     -                   :block-devices                     -
bluetooth-control                 -                   :bluetooth-control                 -
bluez                             -                   :bluez                             -
broadcom-asic-control             -                   :broadcom-asic-control             -
browser-support                   -                   :browser-support                   -
calendar-service                  -                   :calendar-service                  -
camera                            -                   :camera                            -
can-bus                           -                   :can-bus                           -
cifs-mount                        -                   :cifs-mount                        -
classic-support                   -                   :classic-support                   -
contacts-service                  -                   :contacts-service                  -
core-support                      -                   :core-support                      -
cpu-control                       -                   :cpu-control                       -
cups-control                      -                   :cups-control                      -
daemon-notify                     -                   :daemon-notify                     -
dcdbas-control                    -                   :dcdbas-control                    -
desktop                           -                   :desktop                           -
desktop-launch                    -                   :desktop-launch                    -
desktop-legacy                    -                   :desktop-legacy                    -
device-buttons                    -                   :device-buttons                    -
display-control                   -                   :display-control                   -
dm-crypt                          -                   :dm-crypt                          -
docker-support                    -                   :docker-support                    -
dvb                               -                   :dvb                               -
firewall-control                  -                   :firewall-control                  -
fpga                              -                   :fpga                              -
framebuffer                       -                   :framebuffer                       -
fuse-support                      -                   :fuse-support                      -
fwupd                             -                   :fwupd                             -
gconf                             -                   :gconf                             -
gpg-keys                          -                   :gpg-keys                          -
gpg-public-keys                   -                   :gpg-public-keys                   -
gpio-control                      -                   :gpio-control                      -
gpio-memory-control               -                   :gpio-memory-control               -
greengrass-support                -                   :greengrass-support                -
gsettings                         -                   :gsettings                         -
hardware-observe                  -                   :hardware-observe                  -
hardware-random-control           -                   :hardware-random-control           -
hardware-random-observe           -                   :hardware-random-observe           -
home                              -                   :home                              -
hostname-control                  -                   :hostname-control                  -
hugepages-control                 -                   :hugepages-control                 -
intel-mei                         -                   :intel-mei                         -
intel-qat                         -                   :intel-qat                         -
io-ports-control                  -                   :io-ports-control                  -
ion-memory-control                -                   :ion-memory-control                -
jack1                             -                   :jack1                             -
joystick                          -                   :joystick                          -
juju-client-observe               -                   :juju-client-observe               -
kernel-crypto-api                 -                   :kernel-crypto-api                 -
kernel-firmware-control           -                   :kernel-firmware-control           -
kernel-module-control             -                   :kernel-module-control             -
kernel-module-load                -                   :kernel-module-load                -
kernel-module-observe             -                   :kernel-module-observe             -
kubernetes-support                -                   :kubernetes-support                -
kvm                               -                   :kvm                               -
libvirt                           -                   :libvirt                           -
locale-control                    -                   :locale-control                    -
log-observe                       -                   :log-observe                       -
login-session-control             -                   :login-session-control             -
login-session-observe             -                   :login-session-observe             -
lxd-support                       lxd:lxd-support     :lxd-support                       -
media-control                     -                   :media-control                     -
microceph-support                 -                   :microceph-support                 -
microstack-support                -                   :microstack-support                -
modem-manager                     -                   :modem-manager                     -
mount-control                     -                   :mount-control                     -
mount-observe                     -                   :mount-observe                     -
multipass-support                 -                   :multipass-support                 -
netlink-audit                     -                   :netlink-audit                     -
netlink-connector                 -                   :netlink-connector                 -
network                           lxd:network         :network                           -
network-bind                      lxd:network-bind    :network-bind                      -
network-control                   -                   :network-control                   -
network-manager                   -                   :network-manager                   -
network-manager-observe           -                   :network-manager-observe           -
network-observe                   -                   :network-observe                   -
network-setup-control             -                   :network-setup-control             -
network-setup-observe             -                   :network-setup-observe             -
network-status                    -                   :network-status                    -
nfs-mount                         -                   :nfs-mount                         -
nomad-support                     -                   :nomad-support                     -
nvidia-drivers-support            -                   :nvidia-drivers-support            -
ofono                             -                   :ofono                             -
opengl                            -                   :opengl                            -
openvswitch                       -                   :openvswitch                       -
openvswitch-support               -                   :openvswitch-support               -
optical-drive                     -                   :optical-drive                     -
packagekit-control                -                   :packagekit-control                -
password-manager-service          -                   :password-manager-service          -
pcscd                             -                   :pcscd                             -
personal-files                    -                   :personal-files                    -
physical-memory-control           -                   :physical-memory-control           -
physical-memory-observe           -                   :physical-memory-observe           -
polkit                            -                   :polkit                            -
power-control                     -                   :power-control                     -
ppp                               -                   :ppp                               -
process-control                   -                   :process-control                   -
ptp                               -                   :ptp                               -
pulseaudio                        -                   :pulseaudio                        -
qualcomm-ipc-router               -                   :qualcomm-ipc-router               -
raw-input                         -                   :raw-input                         -
raw-usb                           -                   :raw-usb                           -
registry                          -                   :registry                          -
remoteproc                        -                   :remoteproc                        -
removable-media                   -                   :removable-media                   -
ros-opt-data                      -                   :ros-opt-data                      -
screen-inhibit-control            -                   :screen-inhibit-control            -
screencast-legacy                 -                   :screencast-legacy                 -
scsi-generic                      -                   :scsi-generic                      -
sd-control                        -                   :sd-control                        -
shared-memory                     -                   :shared-memory                     -
shutdown                          -                   :shutdown                          -
snap-interfaces-requests-control  -                   :snap-interfaces-requests-control  -
snap-refresh-control              -                   :snap-refresh-control              -
snap-refresh-observe              -                   :snap-refresh-observe              -
snap-themes-control               -                   :snap-themes-control               -
snapd-control                     -                   :snapd-control                     -
ssh-keys                          -                   :ssh-keys                          -
ssh-public-keys                   -                   :ssh-public-keys                   -
steam-support                     -                   :steam-support                     -
system-backup                     -                   :system-backup                     -
system-files                      -                   :system-files                      -
system-observe                    lxd:system-observe  :system-observe                    -
system-packages-doc               -                   :system-packages-doc               -
system-source-code                -                   :system-source-code                -
system-trace                      -                   :system-trace                      -
tee                               -                   :tee                               -
time-control                      -                   :time-control                      -
timeserver-control                -                   :timeserver-control                -
timezone-control                  -                   :timezone-control                  -
tpm                               -                   :tpm                               -
u2f-devices                       -                   :u2f-devices                       -
udisks2                           -                   :udisks2                           -
uhid                              -                   :uhid                              -
uinput                            -                   :uinput                            -
unity7                            -                   :unity7                            -
upower-observe                    -                   :upower-observe                    -
userns                            -                   :userns                            -
vcio                              -                   :vcio                              -
wayland                           -                   :wayland                           -
x11                               -                   :x11                               -
xilinx-dma                        -                   :xilinx-dma                        -

==================== SNAP CHANGES ====================
no changes found

==================== NO GADGET SNAP DETECTED ====================

==================== SNAP CHANGES (in Doing) ====================
no changes found

==================== SNAP CHANGES (in Error) ====================
no changes found

==================== VALIDATION SET ASSERTIONS ====================

==================== VALIDATION SETS ====================
No validations are available

==================== OFFLINE SNAP CHANGES ====================
ID   Status  Spawn  Ready  Label  Summary

==================== SNAPD STACKTRACE ====================
goroutine 758 [running]:
github.com/snapcore/snapd/daemon.getStacktraces()
	/build/snapd/parts/snapd/build/daemon/api_debug_stacktrace.go:29 +0x45
github.com/snapcore/snapd/daemon.postDebug(0x5627298e4d40, 0xc0000a9e00, 0x0?)
	/build/snapd/parts/snapd/build/daemon/api_debug.go:409 +0x66b
github.com/snapcore/snapd/daemon.(*Command).ServeHTTP(0x5627298e4d40, {0x5627292cdaf0, 0xc00065f920}, 0xc0000a9d00)
	/build/snapd/parts/snapd/build/daemon/daemon.go:164 +0x557
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000639440, {0x5627292cdaf0, 0xc00065f920}, 0xc0000a9b00)
	/build/snapd/parts/snapd/build/vendor/github.com/gorilla/mux/mux.go:212 +0x202
github.com/snapcore/snapd/daemon.logit.func1({0x5627292ce3c0?, 0xc0002fc540}, 0xc0000a9b00)
	/build/snapd/parts/snapd/build/daemon/daemon.go:216 +0xdd
net/http.HandlerFunc.ServeHTTP(0x0?, {0x5627292ce3c0?, 0xc0002fc540?}, 0x56272827dfc5?)
	/snap/go/10029/src/net/http/server.go:2084 +0x2f
net/http.serverHandler.ServeHTTP({0x5627292cc010?}, {0x5627292ce3c0, 0xc0002fc540}, 0xc0000a9b00)
	/snap/go/10029/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc00055a500, {0x5627292cf230, 0xc00057cab0})
	/snap/go/10029/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
	/snap/go/10029/src/net/http/server.go:3071 +0x4db

goroutine 1 [select, 13020 minutes]:
main.run(0xc000593920)
	/build/snapd/parts/snapd/build/cmd/snapd/main.go:166 +0x465
main.main()
	/build/snapd/parts/snapd/build/cmd/snapd/main.go:68 +0x13b

goroutine 24 [syscall, 13020 minutes]:
os/signal.signal_recv()
	/snap/go/10029/src/runtime/sigqueue.go:151 +0x2f
os/signal.loop()
	/snap/go/10029/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
	/snap/go/10029/src/os/signal/signal.go:151 +0x2a

goroutine 41 [syscall, 588 minutes]:
syscall.Syscall6(0x17, 0xc, 0xc0005afe48, 0x0, 0x0, 0x0, 0x0)
	/snap/go/10029/src/syscall/asm_linux_amd64.s:43 +0x5
syscall.Select(0x5627282f295b?, 0x5627282ef134?, 0x48?, 0xb?, 0x4?)
	/snap/go/10029/src/syscall/zsyscall_linux_amd64.go:1280 +0x6a
github.com/snapcore/snapd/osutil/udev/netlink.stopperSelectReadable(0x6?, 0xb)
	/build/snapd/parts/snapd/build/osutil/udev/netlink/rawsockstop.go:57 +0x168
github.com/snapcore/snapd/osutil/udev/netlink.RawSockStopper.func1()
	/build/snapd/parts/snapd/build/osutil/udev/netlink/rawsockstop.go:31 +0x67
github.com/snapcore/snapd/osutil/udev/netlink.(*UEventConn).Monitor.func4()
	/build/snapd/parts/snapd/build/osutil/udev/netlink/conn.go:139 +0x82
created by github.com/snapcore/snapd/osutil/udev/netlink.(*UEventConn).Monitor
	/build/snapd/parts/snapd/build/osutil/udev/netlink/conn.go:136 +0x2d1

goroutine 759 [IO wait]:
internal/poll.runtime_pollWait(0x7f00e4038868, 0x72)
	/snap/go/10029/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0001ae600?, 0xc00052cb51?, 0x0)
	/snap/go/10029/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/snap/go/10029/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0001ae600, {0xc00052cb51, 0x1, 0x1})
	/snap/go/10029/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc0001ae600, {0xc00052cb51?, 0x5627291c29c0?, 0xc0000bd748?})
	/snap/go/10029/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0005690a0, {0xc00052cb51?, 0xc0006e9850?, 0xc000296cd0?})
	/snap/go/10029/src/net/net.go:183 +0x45
net/http.(*connReader).backgroundRead(0xc00052cb40)
	/snap/go/10029/src/net/http/server.go:672 +0x3f
created by net/http.(*connReader).startBackgroundRead
	/snap/go/10029/src/net/http/server.go:668 +0xca

goroutine 86 [IO wait, 13020 minutes]:
internal/poll.runtime_pollWait(0x7f00e4038b38, 0x72)
	/snap/go/10029/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0006bc100?, 0xc0002160d0?, 0x0)
	/snap/go/10029/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/snap/go/10029/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).ReadMsg(0xc0006bc100, {0xc0002160d0, 0x10, 0x10}, {0xc000020020, 0x1000, 0x1000}, 0x1?)
	/snap/go/10029/src/internal/poll/fd_unix.go:304 +0x31c
net.(*netFD).readMsg(0xc0006bc100, {0xc0002160d0?, 0x5627292d22b0?, 0x1?}, {0xc000020020?, 0x0?, 0x8?}, 0x5627298b4b70?)
	/snap/go/10029/src/net/fd_posix.go:78 +0x37
net.(*UnixConn).readMsg(0xc000010120, {0xc0002160d0?, 0xc0006aa690?, 0x5627292c5e38?}, {0xc000020020?, 0x5627292c5e38?, 0xc00052cab0?})
	/snap/go/10029/src/net/unixsock_posix.go:115 +0x4f
net.(*UnixConn).ReadMsgUnix(0xc000010120, {0xc0002160d0?, 0xc0005e9df8?, 0x56272827dfc5?}, {0xc000020020?, 0xc0005e9da0?, 0x5627282a91e0?})
	/snap/go/10029/src/net/unixsock.go:143 +0x3c
github.com/godbus/dbus.(*oobReader).Read(0xc000020000, {0xc0002160d0?, 0xc0005e9e20?, 0x56272827e327?})
	/build/snapd/parts/snapd/build/vendor/github.com/godbus/dbus/transport_unix.go:21 +0x45
io.ReadAtLeast({0x5627292c6558, 0xc000020000}, {0xc0002160d0, 0x10, 0x10}, 0x10)
	/snap/go/10029/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/snap/go/10029/src/io/io.go:350
github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc0006aa480)
	/build/snapd/parts/snapd/build/vendor/github.com/godbus/dbus/transport_unix.go:91 +0x11e
github.com/godbus/dbus.(*Conn).inWorker(0xc0006a8b40)
	/build/snapd/parts/snapd/build/vendor/github.com/godbus/dbus/conn.go:294 +0x3b
created by github.com/godbus/dbus.(*Conn).Auth
	/build/snapd/parts/snapd/build/vendor/github.com/godbus/dbus/auth.go:118 +0x9ed

goroutine 106 [select]:
github.com/snapcore/snapd/overlord/standby.(*StandbyOpinions).Start.func1()
	/build/snapd/parts/snapd/build/overlord/standby/standby.go:105 +0x11b
created by github.com/snapcore/snapd/overlord/standby.(*StandbyOpinions).Start
	/build/snapd/parts/snapd/build/overlord/standby/standby.go:96 +0x9b

goroutine 112 [select, 13020 minutes]:
github.com/snapcore/snapd/overlord/ifacestate/udevmonitor.(*Monitor).Run.func1()
	/build/snapd/parts/snapd/build/overlord/ifacestate/udevmonitor/udevmon.go:147 +0x205
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0003ea7e0, 0xc0003ea090?)
	/build/snapd/parts/snapd/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x36
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/build/snapd/parts/snapd/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xee

goroutine 107 [select]:
github.com/snapcore/snapd/overlord.(*Overlord).Loop.func1()
	/build/snapd/parts/snapd/build/overlord/overlord.go:479 +0x185
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0004f5130, 0x0?)
	/build/snapd/parts/snapd/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x36
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/build/snapd/parts/snapd/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xee

goroutine 108 [IO wait]:
internal/poll.runtime_pollWait(0x7f00e4038e08, 0x72)
	/snap/go/10029/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0003af780?, 0x5627282f29ca?, 0x0)
	/snap/go/10029/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/snap/go/10029/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc0003af780)
	/snap/go/10029/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc0003af780)
	/snap/go/10029/src/net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0xc000065d50?)
	/snap/go/10029/src/net/unixsock_posix.go:166 +0x1c
net.(*UnixListener).Accept(0xc000098600)
	/snap/go/10029/src/net/unixsock.go:260 +0x3d
github.com/snapcore/snapd/daemon.(*ucrednetListener).Accept(0x56272853a49b?)
	/build/snapd/parts/snapd/build/daemon/ucrednet.go:150 +0x25
net/http.(*Server).Serve(0xc00031c540, {0x5627292cdac0, 0xc000098690})
	/snap/go/10029/src/net/http/server.go:3039 +0x385
github.com/snapcore/snapd/daemon.(*Daemon).Start.func3()
	/build/snapd/parts/snapd/build/daemon/daemon.go:402 +0x8a
gopkg.in/tomb%2ev2.(*Tomb).run(0xc00039e050, 0x0?)
	/build/snapd/parts/snapd/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x36
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/build/snapd/parts/snapd/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xee

goroutine 109 [select]:
main.runWatchdog.func1()
	/build/snapd/parts/snapd/build/cmd/snapd/main.go:104 +0x9c
created by main.runWatchdog
	/build/snapd/parts/snapd/build/cmd/snapd/main.go:102 +0x10e

goroutine 110 [IO wait, 13020 minutes]:
internal/poll.runtime_pollWait(0x7f00e4038d18, 0x72)
	/snap/go/10029/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0003af880?, 0x0?, 0x0)
	/snap/go/10029/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/snap/go/10029/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc0003af880)
	/snap/go/10029/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc0003af880)
	/snap/go/10029/src/net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0x0?)
	/snap/go/10029/src/net/unixsock_posix.go:166 +0x1c
net.(*UnixListener).Accept(0xc000098630)
	/snap/go/10029/src/net/unixsock.go:260 +0x3d
github.com/snapcore/snapd/daemon.(*ucrednetListener).Accept(0x5627298d9520?)
	/build/snapd/parts/snapd/build/daemon/ucrednet.go:150 +0x25
net/http.(*Server).Serve(0xc00031c540, {0x5627292cdac0, 0xc000098720})
	/snap/go/10029/src/net/http/server.go:3039 +0x385
github.com/snapcore/snapd/daemon.(*Daemon).Start.func3.1()
	/build/snapd/parts/snapd/build/daemon/daemon.go:394 +0x32
gopkg.in/tomb%2ev2.(*Tomb).run(0xc00039e050, 0x0?)
	/build/snapd/parts/snapd/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x36
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/build/snapd/parts/snapd/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xee

==================== SNAP SYSTEM CONFIG ====================
{
	"refresh": {},
	"seed": {
		"loaded": true
	},
	"system": {
		"hostname": "HOST",
		"network": {},
		"timezone": "UTC"
	}
}

==================== SNAPD JOURNAL ====================
Nov 25 10:50:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 25 14:45:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 25 19:25:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "snapd"
Nov 25 19:25:49 HOST snapd[879]: services.go:1043: RemoveSnapServices - socket snap.lxd.user-daemon.unix.socket
Nov 25 19:25:49 HOST snapd[879]: services.go:1067: RemoveSnapServices - disabling snap.lxd.user-daemon.service
Nov 25 19:25:49 HOST snapd[879]: services.go:1067: RemoveSnapServices - disabling snap.lxd.activate.service
Nov 25 19:25:49 HOST snapd[879]: services.go:1043: RemoveSnapServices - socket snap.lxd.daemon.unix.socket
Nov 25 19:25:49 HOST snapd[879]: services.go:1067: RemoveSnapServices - disabling snap.lxd.daemon.service
Nov 25 19:25:55 HOST snapd[879]: storehelpers.go:923: cannot refresh snap "lxd": snap has no updates available
Nov 26 00:45:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 26 08:35:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 26 15:05:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 26 18:40:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 27 02:45:40 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 27 08:15:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 27 15:20:39 HOST snapd[879]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd"
Nov 27 15:23:54 HOST snapd[879]: daemon.go:519: gracefully waiting for running hooks
Nov 27 15:23:54 HOST snapd[879]: daemon.go:521: done waiting for running hooks
Nov 27 15:23:57 HOST snapd[879]: overlord.go:515: Released state lock file
Nov 27 15:23:57 HOST systemd[1]: snapd.service: Deactivated successfully.
Nov 27 15:23:57 HOST systemd[1]: snapd.service: Consumed 9min 22.881s CPU time.
Nov 27 15:23:57 HOST systemd[1]: snapd.service: Scheduled restart job, restart counter is at 1.
Nov 27 15:23:57 HOST systemd[1]: Stopped Snap Daemon.
Nov 27 15:23:57 HOST systemd[1]: snapd.service: Consumed 9min 22.881s CPU time.
Nov 27 15:23:57 HOST systemd[1]: Starting Snap Daemon...
Nov 27 15:23:57 HOST snapd[1264174]: AppArmor status: apparmor is enabled and all features are available
Nov 27 15:23:58 HOST snapd[1264174]: overlord.go:274: Acquiring state lock file
Nov 27 15:23:58 HOST snapd[1264174]: overlord.go:279: Acquired state lock file
Nov 27 15:23:58 HOST snapd[1264174]: patch.go:64: Patching system state level 6 to sublevel 1...
Nov 27 15:23:58 HOST snapd[1264174]: patch.go:64: Patching system state level 6 to sublevel 2...
Nov 27 15:23:58 HOST snapd[1264174]: patch.go:64: Patching system state level 6 to sublevel 3...
Nov 27 15:23:58 HOST snapd[1264174]: daemon.go:250: started snapd/2.66.1 (series 16; classic) ubuntu/22.04 (amd64) linux/6.0.0-1006-oem.
Nov 27 15:23:58 HOST snapd[1264174]: daemon.go:353: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Nov 27 15:23:58 HOST snapd[1264174]: backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
Nov 27 15:23:58 HOST snapd[1264174]: helpers.go:160: error trying to compare the snap system key: system-key versions not comparable
Nov 27 15:24:00 HOST systemd[1]: Started Snap Daemon.
Nov 27 15:24:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh snap "snapd": snap has no updates available
Nov 27 22:34:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 28 00:39:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 28 11:19:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 28 13:59:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 28 20:34:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 29 03:29:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 29 11:24:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 29 17:14:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 29 21:04:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 30 05:49:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 30 07:54:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 30 16:39:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 30 18:19:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 01 01:49:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 01 10:54:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 01 16:04:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 01 21:34:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 02 03:44:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 02 09:29:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 02 12:14:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 02 22:19:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 03 01:54:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 03 06:34:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 03 14:09:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 03 19:04:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 04 05:24:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 04 10:54:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 04 14:59:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 04 21:19:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 05 02:24:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 05 09:54:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 05 15:09:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 05 21:29:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 06 01:04:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 06 09:34:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 06 15:29:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"

==================== SNAPD.SERVICE STATUS ====================
● snapd.service - Snap Daemon
     Loaded: loaded (/lib/systemd/system/snapd.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-11-27 15:24:00 JST; 1 week 2 days ago
TriggeredBy: ● snapd.socket
   Main PID: 1264174 (snapd)
      Tasks: 12 (limit: 4145)
     Memory: 31.9M
        CPU: 1min 1.492s
     CGroup: /system.slice/snapd.service
             └─1264174 /usr/lib/snapd/snapd

Dec 04 10:54:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 04 14:59:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 04 21:19:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 05 02:24:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 05 09:54:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 05 15:09:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 05 21:29:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 06 01:04:02 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 06 09:34:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Dec 06 15:29:03 HOST snapd[1264174]: storehelpers.go:1044: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"

==================== UPTIME ====================
 16:24:24 up 99 days, 14:28,  2 users,  load average: 7.17, 7.23, 7.38

==================== DATE (IN UTC) ====================
Fri Dec  6 07:24:24 AM UTC 2024

==================== DISK SPACE ====================
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              358M  1.8M  357M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  233G   23G  198G  11% /
tmpfs                              1.8G  194M  1.6G  11% /dev/shm
tmpfs                              5.0M  4.0K  5.0M   1% /run/lock
/dev/nvme0n1p2                     974M  269M  638M  30% /boot
/dev/nvme0n1p1                     511M  5.3M  506M   2% /boot/efi
tmpfs                              358M   48K  358M   1% /run/user/1000

==================== DENIED MESSAGES ====================

mborzecki1 · December 6, 2024, 7:54am

Thanks for the log. It’s clearly a system we call ‘classic’ and you clearly do not have network-manager snap. Unfortunately the changes logs has already been garbage collected, so I cannot say when snapd was updated, but it must have been more than 5 days ago.

I’m not sure how you concluded that snapd broke your setup, because it does not manipulate network nor DNS configuration in any way, and especially does not overwrite any files there.

However, there may be a different angle to it. When snapd snap is updated, systemctl daemon-reload gets called after a mount unit file is written. Perhaps this triggered some preexisting problem with configuration. I guess it’s a long shot, but if I were to look for clues, I would start with taking a look at the journal from around the time snapd may have refreshed to see if there are any errors from systemd or other services. In any case, it’s still worth looking at the journal to investigate when systemd-resolved may have been restarted/reloaded.

ernestl · December 6, 2024, 9:40am

Hi ryan.rubley,

Could you please run the following commands and share their output? This might help us gain a better understanding of the situation: data-gather-commands

ogra · December 6, 2024, 9:50am

On a side note you should take into account that Ubuntu abandoned ifupdown 8 years ago and switched to netplan as default (which had been introduced and tested since 2014 before)…

You should be aware that you are on an unsupported setup here…

I.e. I don’t mean to talk you out of it, it is your setup after all, but you should be aware that this setup gets zero testing by the manufacturer of your OS, nor does any integration work happen.

If it breaks you got to keep the pieces and are pretty much on your own (I’d personally not build my whole business with thousands of machines on top of this).

That said, Debian still uses a new variant ifupdown AFAIK and we have not had any reports about any DNS issues from there yet…

ryan.rubley · December 6, 2024, 11:53pm

I’m not sure how you concluded that snapd broke your setup

We correlated it to the second on hundreds of devices based on the syslog info. We have several thousands of devices which were impacted by this over the course of last week as they apparently did the snapd auto-refresh on various time spreads. Most of these machines have uptimes ranging from months to several years and do not just suddenly lose DNS. It might not be snapd itself which rewrote the system DNS file, but it was something which normally does not happen on the machines, that was triggered by this process that started from the snapd update for sure.

Also, I assume snapd has been updating numerous other times over the past couple years, without impacting any of these devices in any way. It was definitely something specific to the 2.66.1 update that did this.

ryan.rubley · December 7, 2024, 12:08am

netplan:
00-installer-config-wifi.yaml  00-installer-config.yaml
/etc/netplan
├── 00-installer-config-wifi.yaml
└── 00-installer-config.yaml

0 directories, 2 files
# This is the network config written by 'subiquity'
network:
  version: 2
  wifis: {}
# This is the network config written by 'subiquity'
network:
  ethernets: {}
  version: 2
resolvectl:
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: foreign
      DNS Domain: DOMAINS

Link 2 (eno1)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: REDACTED
       DNS Servers: REDACTED REDACTED
        DNS Domain: DOMAINS

Link 3 (wlp1s0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (tun0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
systemd-resolved:
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-12-05 13:52:57 JST; 1 day 19h ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 3579545 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 4423)
     Memory: 1.8M
        CPU: 1.179s
     CGroup: /system.slice/systemd-resolved.service
             └─3579545 /lib/systemd/systemd-resolved

Dec 05 13:52:57 REDACTED systemd[1]: Starting Network Name Resolution...
Dec 05 13:52:57 REDACTED systemd-resolved[3579545]: Positive Trust Anchors:
Dec 05 13:52:57 REDACTED systemd-resolved[3579545]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 05 13:52:57 REDACTED systemd-resolved[3579545]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.…
Dec 05 13:52:57 REDACTED systemd-resolved[3579545]: Using system hostname 'REDACTED'.
Dec 05 13:52:57 REDACTED systemd[1]: Started Network Name Resolution.
Dec 05 13:53:26 REDACTED systemd-resolved[3579545]: eno1: Bus client set search domain list to: DOMAINS
Dec 05 13:53:26 REDACTED systemd-resolved[3579545]: eno1: Bus client set DNS server list to: REDACTED, REDACTED
Hint: Some lines were ellipsized, use -l to show in full.
/etc/resolv.conf symlink:
lrwxrwxrwx 1 root root 29 Dec 12 2022 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
/run/systemd/resolve/resolv.conf:
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver REDACTED
nameserver REDACTED
search DOMAINS
/run/systemd/resolve/stub-resolv.conf:
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search DOMAINS
/run/resolvconf/resolv.conf:
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
search DOMAINS
/etc/systemd/resolved.conf:
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
#Cache=no-negative
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
systemd-networkd:
● systemd-networkd.service - Network Configuration
     Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-10-18 10:57:25 JST; 1 month 19 days ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
   Main PID: 753 (systemd-network)
     Status: "Processing requests..."
      Tasks: 1 (limit: 4423)
     Memory: 1.2M
        CPU: 48.031s
     CGroup: /system.slice/systemd-networkd.service
             └─753 /lib/systemd/systemd-networkd

Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Link DOWN
Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Lost carrier
Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Link UP
Dec 05 13:53:29 REDACTED systemd-networkd[753]: eno1: Gained carrier
Dec 05 13:53:31 REDACTED systemd-networkd[753]: eno1: Gained IPv6LL
Notice: journal has been rotated since unit was started, output may be incomplete.
NetworkManager:
Unit NetworkManager.service could not be found.
nmcli dev show:
Command 'nmcli' not found, but can be installed with:
sudo snap install network-manager  # version 1.2.2-31, or
sudo apt  install network-manager  # version 1.36.6-0ubuntu2
See 'snap info network-manager' for additional versions.
/etc/network/if-up.d:
total 24
-rwxr-xr-x 1 root root 1053 Aug 20  2020 000resolvconf
-rwxr-xr-x 1 root root  145 Feb  8  2022 chrony
-rwxr-xr-x 1 root root 1685 Jan 19  2022 ethtool
-rwxr-xr-x 1 root root  385 Mar 23  2022 openvpn
-rwxr-xr-x 1 root root 4626 Dec 12  2022 resolved
#!/bin/sh
#
# ifup hook script for resolvconf
#
# This file is part of the resolvconf package.
#

[ -x /sbin/resolvconf ] || exit 0

case "$ADDRFAM" in
  inet|inet6) : ;;
  *) exit 0 ;;
esac

# ifupdown hook pushes into systemd-resolved for non-lo interfaces
# thus skip pulling info into resolvconf
if systemctl --quiet is-enabled systemd-resolved ; then
    case "$IFACE" in
        lo) : ;;
        *) exit 0 ;;
    esac
fi

R=""
if [ "$IF_DNS_DOMAIN" ] ; then
	R="${R}domain $IF_DNS_DOMAIN
"
fi
if [ "$IF_DNS_SEARCH" ] ; then
	R="${R}search $IF_DNS_SEARCH
"
fi
if [ "$IF_DNS_SORTLIST" ] ; then
	R="${R}sortlist $IF_DNS_SORTLIST
"
fi
for NS in $IF_DNS_NAMESERVERS ; do
	R="${R}nameserver $NS
"
done

# Note: arguments of multiple instances of options are separated by newlines
set_NS_to_first_arg() { NS="$1" ; }
STANDARD_IFS="$IFS"
IFS='
'
for OPT in $IF_DNS_NAMESERVER ; do
	IFS="$STANDARD_IFS"
	set_NS_to_first_arg $OPT
	[ "$NS" ] && R="${R}nameserver $NS
"
done
IFS="$STANDARD_IFS"

echo -n "$R" | /sbin/resolvconf -a "${IFACE}.${ADDRFAM}" || :

#!/bin/sh

set -e

[ -x /usr/sbin/chronyd ] || exit 0

if [ -e /run/chrony/chronyd.pid ]; then
    chronyc onoffline > /dev/null 2>&1
fi

exit 0
#!/bin/sh

ETHTOOL=/sbin/ethtool

test -x $ETHTOOL || exit 0

[ "$IFACE" != "lo" ] || exit 0

# Find settings with a given prefix and print them as they appeared in
# /etc/network/interfaces, only with the prefix removed.
# This actually prints each name and value on a separate line, but that
# doesn't matter to the shell.
gather_settings () {
    set | sed -n "
/^IF_$1[A-Za-z0-9_]*=/ {
    h;                             # hold line
    s/^IF_$1//; s/=.*//; s/_/-/g;  # get name without prefix
    y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/;  # lower-case
    p;
    g;                             # restore line
    s/^[^=]*=//; s/^'\(.*\)'/\1/;  # get value
    p;
}"
}

# Gather together the mixed bag of settings applied with -s/--change
SETTINGS="\
${IF_LINK_SPEED:+ speed $IF_LINK_SPEED}\
${IF_LINK_DUPLEX:+ duplex $IF_LINK_DUPLEX}\
"

# WOL has an optional pass-key
set -- $IF_ETHERNET_WOL
SETTINGS="$SETTINGS${1:+ wol $1}${2:+ sopass $2}"

# Autonegotiation can be on|off or an advertising mask
case "$IF_ETHERNET_AUTONEG" in
'')	;;
on|off)	SETTINGS="$SETTINGS autoneg $IF_ETHERNET_AUTONEG" ;;
*)	SETTINGS="$SETTINGS autoneg on advertise $IF_ETHERNET_AUTONEG" ;;
esac

[ -z "$SETTINGS" ] || $ETHTOOL --change "$IFACE" $SETTINGS

SETTINGS="$(gather_settings ETHERNET_PAUSE_)"
[ -z "$SETTINGS" ] || $ETHTOOL --pause "$IFACE" $SETTINGS

SETTINGS="$(gather_settings HARDWARE_IRQ_COALESCE_)"
[ -z "$SETTINGS" ] || $ETHTOOL --coalesce "$IFACE" $SETTINGS

SETTINGS="$(gather_settings HARDWARE_DMA_RING_)"
[ -z "$SETTINGS" ] || $ETHTOOL --set-ring "$IFACE" $SETTINGS

SETTINGS="$(gather_settings OFFLOAD_)"
[ -z "$SETTINGS" ] || $ETHTOOL --offload "$IFACE" $SETTINGS
#!/bin/sh

OPENVPN=/usr/sbin/openvpn
OPENVPN_INIT=/etc/init.d/openvpn
SYSTEMCTL=/bin/systemctl
SYSTEMD=/run/systemd/system

if [ ! -x $OPENVPN ]; then
  exit 0
fi

if [ -n "$IF_OPENVPN" ]; then
  for vpn in $IF_OPENVPN; do
    ## check systemd present
    if [ -d $SYSTEMD ]; then
      $SYSTEMCTL --no-block start openvpn@$vpn
    else
      $OPENVPN_INIT start $vpn
    fi
  done
fi
#!/bin/sh
#
# Script fragment to make ifupdown supply DNS infromation to resolved
#

case "$ADDRFAM" in
    inet|inet6) : ;;
    *) exit 0 ;;
esac            

if systemctl is-enabled systemd-resolved > /dev/null 2>&1; then


    interface=$IFACE
    if [ ! "$interface" ]; then
        return
    fi
    # TODO handle lo interface settings
    if [ "$interface" = "lo" ]; then
        return
    fi
    ifindex=$(cat "/sys/class/net/$interface/ifindex")
    if [ ! "$ifindex" ]; then
        return
    fi
    mystatedir=/run/network
    mkdir -p $mystatedir

    statedir=/run/systemd/resolve/netif
    mkdir -p $statedir
    chown systemd-resolve:systemd-resolve $statedir

    oldstate="$(mktemp)"
    # ignore errors due to nonexistent file
    md5sum "$mystatedir/isc-dhcp-v4-$interface" "$mystatedir/isc-dhcp-v6-$interface" "$mystatedir/ifupdown-inet-$interface" "$mystatedir/ifupdown-inet6-$interface" > "$oldstate" 2> /dev/null || true

    NEW_DEFAULT_ROUTE=$IF_DNS_DEFAULT_ROUTE
    NEW_DNS=$(echo $IF_DNS_NAMESERVERS $IF_DNS_NAMESERVER)
    NEW_DOMAINS=$(echo $IF_DNS_DOMAIN $IF_DNS_SEARCH)
    DNS=DNS
    DOMAINS=DOMAINS
    if [ "$ADDRFAM" = "inet6" ]; then
        DNS=DNS6
        DOMAINS=DOMAINS6
    fi
    if  [ -n "$NEW_DNS" ]; then
        cat <<EOF >"$mystatedir/ifupdown-${ADDRFAM}-$interface"
$DNS="$NEW_DNS"
EOF
        if  [ -n "$NEW_DOMAINS" ]; then
            cat <<EOF >>"$mystatedir/ifupdown-${ADDRFAM}-$interface"
$DOMAINS="$NEW_DOMAINS"
EOF
        fi
    fi
    case "$NEW_DEFAULT_ROUTE" in
        1|yes|true|on) NEW_DEFAULT_ROUTE=yes ;;
        0|no|false|off) NEW_DEFAULT_ROUTE=no ;;
        *) NEW_DEFAULT_ROUTE= ;;
    esac
    if [ -n "$NEW_DEFAULT_ROUTE" ]; then
        cat <<EOF >>"$mystatedir/ifupdown-${ADDRFAM}-$interface"
DEFAULT_ROUTE="$NEW_DEFAULT_ROUTE"
EOF
    fi

    newstate="$(mktemp)"
    # ignore errors due to nonexistent file
    md5sum "$mystatedir/isc-dhcp-v4-$interface" "$mystatedir/isc-dhcp-v6-$interface" "$mystatedir/ifupdown-inet-$interface" "$mystatedir/ifupdown-inet6-$interface" > "$newstate" 2> /dev/null || true
    if ! cmp --silent "$oldstate" "$newstate" 2>/dev/null; then

        # v4 first
        if [ -e "$mystatedir/isc-dhcp-v4-$interface" ]; then
            . "$mystatedir/isc-dhcp-v4-$interface"
        fi
        # v4 manual config overrides
        if [ -e "$mystatedir/ifupdown-inet-$interface" ]; then
            . "$mystatedir/ifupdown-inet-$interface"
        fi
        # v6 preffered
        if [ -e "$mystatedir/isc-dhcp-v6-$interface" ]; then
            . "$mystatedir/isc-dhcp-v6-$interface"
        fi
        # v6 manual config overrides
        if [ -e "$mystatedir/ifupdown-inet6-$interface" ]; then
            . "$mystatedir/ifupdown-inet6-$interface"
        fi
        resolvectl_failed=
        if [ "$DNS" ] || [ "$DNS6" ] ; then
            cat <<EOF >"$statedir/$ifindex"
# This is private data. Do not parse.
LLMNR=yes
MDNS=no
SERVERS=$(echo $DNS6 $DNS)
DOMAINS=$(echo $DOMAINS6 $DOMAINS)
EOF
            if [ -n "$DEFAULT_ROUTE" ]; then
                cat <<EOF >>"$statedir/$ifindex"
DEFAULT_ROUTE=$DEFAULT_ROUTE
EOF
            fi
            chown systemd-resolve:systemd-resolve "$statedir/$ifindex"
            # In addition to creating the state file (needed if we run before
            # resolved is started), also feed the information directly to
            # resolved.
            if systemctl --quiet is-active systemd-resolved; then
                resolvectl llmnr "$ifindex" yes || resolvectl_failed=$?
                resolvectl mdns "$ifindex" no || resolvectl_failed=$?
                if [ "$DOMAINS6" ] || [ "$DOMAINS" ]; then
                    resolvectl domain "$ifindex" $DOMAINS6 $DOMAINS || resolvectl_failed=$?
                else
                    resolvectl domain "$ifindex" "" || resolvectl_failed=$?
                fi
                resolvectl dns "$ifindex" $DNS6 $DNS || resolvectl_failed=$?
                if [ "$DEFAULT_ROUTE" ]; then
                    resolvectl default-route "$ifindex" $DEFAULT_ROUTE || resolvectl_failed=$?
                fi
            fi
        else
            rm -f "$statedir/$ifindex"
            if systemctl --quiet is-active systemd-resolved; then
                resolvectl revert "$ifindex" || resolvectl_failed=$?
            fi
        fi

        # resolved was running, but without dbus, it means state files
        # will not be read & resolvectl commands failed, restart it
        if [ "$resolvectl_failed" ]; then
                systemctl try-restart systemd-resolved
        fi
    fi
    rm -f "$oldstate" "$newstate"
fi
resolvconf package:
resolvconf:
  Installed: 1.84ubuntu1
  Candidate: 1.84ubuntu1
  Version table:
 *** 1.84ubuntu1 500
        500 REDACTED jammy/main amd64 Packages
        100 /var/lib/dpkg/status
/etc/nsswitch.conf:
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files systemd
group:          files systemd
shadow:         files
gshadow:        files

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis
/etc/hosts:
127.0.0.1   localhost
127.0.1.1   REDACTED

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
systemd-networkd journal:
Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Link DOWN
Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Lost carrier
Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Link UP
Dec 05 13:53:29 REDACTED systemd-networkd[753]: eno1: Gained carrier
Dec 05 13:53:31 REDACTED systemd-networkd[753]: eno1: Gained IPv6LL
systemd-resolved journal:
Dec 05 13:11:19 REDACTED systemd-resolved[807]: eno1: Bus client set default route setting: yes
Dec 05 13:11:33 REDACTED systemd-resolved[807]: eno1: Bus client set default route setting: no
Dec 05 13:49:13 REDACTED systemd[1]: Stopping Network Name Resolution...
Dec 05 13:49:14 REDACTED systemd[1]: systemd-resolved.service: Deactivated successfully.
Dec 05 13:49:14 REDACTED systemd[1]: Stopped Network Name Resolution.
Dec 05 13:49:14 REDACTED systemd[1]: systemd-resolved.service: Consumed 34.544s CPU time.
Dec 05 13:49:14 REDACTED systemd[1]: Starting Network Name Resolution...
Dec 05 13:49:14 REDACTED systemd-resolved[3576871]: Positive Trust Anchors:
Dec 05 13:49:14 REDACTED systemd-resolved[3576871]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 05 13:49:14 REDACTED systemd-resolved[3576871]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dec 05 13:49:14 REDACTED systemd-resolved[3576871]: Using system hostname 'REDACTED'.
Dec 05 13:49:14 REDACTED systemd[1]: Started Network Name Resolution.
Dec 05 13:50:01 REDACTED systemd[1]: Stopping Network Name Resolution...
Dec 05 13:50:01 REDACTED systemd[1]: systemd-resolved.service: Deactivated successfully.
Dec 05 13:50:01 REDACTED systemd[1]: Stopped Network Name Resolution.
Dec 05 13:50:01 REDACTED systemd[1]: Starting Network Name Resolution...
Dec 05 13:50:01 REDACTED systemd-resolved[3577071]: Positive Trust Anchors:
Dec 05 13:50:01 REDACTED systemd-resolved[3577071]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 05 13:50:01 REDACTED systemd-resolved[3577071]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dec 05 13:50:01 REDACTED systemd-resolved[3577071]: Using system hostname 'REDACTED'.
Dec 05 13:50:01 REDACTED systemd[1]: Started Network Name Resolution.
Dec 05 13:50:17 REDACTED systemd[1]: Stopping Network Name Resolution...
Dec 05 13:50:17 REDACTED systemd[1]: systemd-resolved.service: Deactivated successfully.
Dec 05 13:50:17 REDACTED systemd[1]: Stopped Network Name Resolution.
Dec 05 13:50:17 REDACTED systemd[1]: Starting Network Name Resolution...
Dec 05 13:50:17 REDACTED systemd-resolved[3577793]: Positive Trust Anchors:
Dec 05 13:50:17 REDACTED systemd-resolved[3577793]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 05 13:50:17 REDACTED systemd-resolved[3577793]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dec 05 13:50:17 REDACTED systemd-resolved[3577793]: Using system hostname 'REDACTED'.
Dec 05 13:50:17 REDACTED systemd[1]: Started Network Name Resolution.
Dec 05 13:50:38 REDACTED systemd[1]: Stopping Network Name Resolution...
Dec 05 13:50:38 REDACTED systemd[1]: systemd-resolved.service: Deactivated successfully.
Dec 05 13:50:38 REDACTED systemd[1]: Stopped Network Name Resolution.
Dec 05 13:50:38 REDACTED systemd[1]: Starting Network Name Resolution...
Dec 05 13:50:38 REDACTED systemd-resolved[3577825]: Positive Trust Anchors:
Dec 05 13:50:38 REDACTED systemd-resolved[3577825]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 05 13:50:38 REDACTED systemd-resolved[3577825]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dec 05 13:50:38 REDACTED systemd-resolved[3577825]: Using system hostname 'REDACTED'.
Dec 05 13:50:38 REDACTED systemd[1]: Started Network Name Resolution.
Dec 05 13:50:43 REDACTED systemd[1]: Stopping Network Name Resolution...
Dec 05 13:50:43 REDACTED systemd[1]: systemd-resolved.service: Deactivated successfully.
Dec 05 13:50:43 REDACTED systemd[1]: Stopped Network Name Resolution.
Dec 05 13:50:43 REDACTED systemd[1]: Starting Network Name Resolution...
Dec 05 13:50:43 REDACTED systemd-resolved[3577844]: Positive Trust Anchors:
Dec 05 13:50:43 REDACTED systemd-resolved[3577844]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 05 13:50:43 REDACTED systemd-resolved[3577844]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dec 05 13:50:43 REDACTED systemd-resolved[3577844]: Using system hostname 'REDACTED'.
Dec 05 13:50:43 REDACTED systemd[1]: Started Network Name Resolution.
Dec 05 13:51:06 REDACTED systemd[1]: Stopping Network Name Resolution...
Dec 05 13:51:06 REDACTED systemd[1]: systemd-resolved.service: Deactivated successfully.
Dec 05 13:51:06 REDACTED systemd[1]: Stopped Network Name Resolution.
Dec 05 13:51:06 REDACTED systemd[1]: Starting Network Name Resolution...
Dec 05 13:51:06 REDACTED systemd-resolved[3578483]: Positive Trust Anchors:
Dec 05 13:51:06 REDACTED systemd-resolved[3578483]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 05 13:51:06 REDACTED systemd-resolved[3578483]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dec 05 13:51:06 REDACTED systemd-resolved[3578483]: Using system hostname 'REDACTED'.
Dec 05 13:51:06 REDACTED systemd[1]: Started Network Name Resolution.
Dec 05 13:52:57 REDACTED systemd[1]: Stopping Network Name Resolution...
Dec 05 13:52:57 REDACTED systemd[1]: systemd-resolved.service: Deactivated successfully.
Dec 05 13:52:57 REDACTED systemd[1]: Stopped Network Name Resolution.
Dec 05 13:52:57 REDACTED systemd[1]: Starting Network Name Resolution...
Dec 05 13:52:57 REDACTED systemd-resolved[3579545]: Positive Trust Anchors:
Dec 05 13:52:57 REDACTED systemd-resolved[3579545]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 05 13:52:57 REDACTED systemd-resolved[3579545]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dec 05 13:52:57 REDACTED systemd-resolved[3579545]: Using system hostname 'REDACTED'.
Dec 05 13:52:57 REDACTED systemd[1]: Started Network Name Resolution.
Dec 05 13:53:26 REDACTED systemd-resolved[3579545]: eno1: Bus client set search domain list to: DOMAINS
Dec 05 13:53:26 REDACTED systemd-resolved[3579545]: eno1: Bus client set DNS server list to: REDACTED, REDACTED
snapd journal:
Nov 26 03:27:31 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "snapd"
Nov 26 03:27:48 REDACTED snapd[891]: services.go:1067: RemoveSnapServices - disabling snap.lxd.activate.service
Nov 26 03:27:48 REDACTED snapd[891]: services.go:1043: RemoveSnapServices - socket snap.lxd.daemon.unix.socket
Nov 26 03:27:48 REDACTED snapd[891]: services.go:1067: RemoveSnapServices - disabling snap.lxd.daemon.service
Nov 26 03:27:48 REDACTED snapd[891]: services.go:1043: RemoveSnapServices - socket snap.lxd.user-daemon.unix.socket
Nov 26 03:27:48 REDACTED snapd[891]: services.go:1067: RemoveSnapServices - disabling snap.lxd.user-daemon.service
Nov 26 03:27:57 REDACTED snapd[891]: storehelpers.go:923: cannot refresh snap "lxd": snap has no updates available
Nov 26 07:17:30 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 26 15:32:31 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 26 23:07:30 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 27 01:52:30 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 27 11:42:31 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 27 14:37:31 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 27 20:42:30 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd", "snapd"
Nov 28 01:52:30 REDACTED snapd[891]: storehelpers.go:923: cannot refresh: snap has no updates available: "core20", "lxd"
Nov 28 01:52:42 REDACTED snapd[891]: daemon.go:519: gracefully waiting for running hooks
Nov 28 01:52:42 REDACTED snapd[891]: daemon.go:521: done waiting for running hooks
Nov 28 01:52:45 REDACTED snapd[891]: overlord.go:515: Released state lock file
Nov 28 01:52:45 REDACTED systemd[1]: snapd.service: Deactivated successfully.
Nov 28 01:52:45 REDACTED systemd[1]: snapd.service: Consumed 13min 46.529s CPU time.
Nov 28 01:52:45 REDACTED systemd[1]: snapd.service: Scheduled restart job, restart counter is at 1.
Nov 28 01:52:45 REDACTED systemd[1]: Stopped Snap Daemon.
Nov 28 01:52:45 REDACTED systemd[1]: snapd.service: Consumed 13min 46.529s CPU time.
Nov 28 01:52:45 REDACTED systemd[1]: Starting Snap Daemon...
Nov 28 01:52:46 REDACTED snapd[2406290]: AppArmor status: apparmor is enabled and all features are available
Nov 28 01:52:47 REDACTED snapd[2406290]: overlord.go:274: Acquiring state lock file
Nov 28 01:52:47 REDACTED snapd[2406290]: overlord.go:279: Acquired state lock file
Nov 28 01:52:47 REDACTED snapd[2406290]: patch.go:64: Patching system state level 6 to sublevel 1...
Nov 28 01:52:47 REDACTED snapd[2406290]: patch.go:64: Patching system state level 6 to sublevel 2...
Nov 28 01:52:47 REDACTED snapd[2406290]: patch.go:64: Patching system state level 6 to sublevel 3...
Nov 28 01:52:47 REDACTED snapd[2406290]: daemon.go:250: started snapd/2.66.1 (series 16; classic) ubuntu/22.04 (amd64) linux/6.0.0-1006-oem.
Nov 28 01:52:47 REDACTED snapd[2406290]: daemon.go:353: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Nov 28 01:52:47 REDACTED snapd[2406290]: backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
Nov 28 01:52:47 REDACTED snapd[2406290]: helpers.go:160: error trying to compare the snap system key: system-key versions not comparable
Nov 28 01:52:49 REDACTED systemd[1]: Started Snap Daemon.
Nov 28 01:53:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 28 01:53:55 REDACTED snapd[2406290]: taskrunner.go:299: Change 32 task (Monitoring snap "snapd" to determine whether extra refresh steps are required) failed: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 28 10:33:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 28 10:33:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 28 14:08:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 28 14:08:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 28 20:28:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 28 20:28:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 01:58:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 05:48:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 05:48:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 07:33:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 07:33:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 14:03:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 14:03:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 23:53:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 29 23:53:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 01:48:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 01:48:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 02:03:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 08:53:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 08:53:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 12:03:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 12:03:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 19:18:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Nov 30 19:18:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 02:08:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 04:38:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 04:38:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 11:08:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 11:08:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 15:23:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 15:23:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 21:53:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 01 21:53:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 02:08:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 03:58:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 03:58:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 07:18:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 07:18:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 17:08:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 17:08:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 22:38:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 02 22:38:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 02:13:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 03:23:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 03:23:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 06:18:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 06:18:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 12:48:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 12:48:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 23:28:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 03 23:28:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 01:03:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 01:03:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 02:18:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 06:43:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 06:43:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 13:28:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 13:28:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 18:03:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 04 18:03:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 01:53:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 01:53:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 02:18:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 06:53:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 06:53:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 13:23:22 REDACTED snapd[2406290]: autorefresh.go:628: Cannot prepare auto-refresh change: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 13:23:22 REDACTED snapd[2406290]: stateengine.go:161: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 13:30:22 REDACTED systemd[1]: snapd.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Dec 05 13:30:22 REDACTED systemd[1]: Stopping snapd.service...
Dec 05 13:30:22 REDACTED snapd[2406290]: main.go:168: Exiting on terminated signal.
Dec 05 13:30:25 REDACTED systemd[1]: snapd.service: Got notification message from PID 2406290, but reception is disabled.
Dec 05 13:30:25 REDACTED snapd[2406290]: overlord.go:518: Released state lock file
Dec 05 13:30:25 REDACTED systemd[1]: snapd.service: Deactivated successfully.
Dec 05 13:30:25 REDACTED systemd[1]: Stopped snapd.service.
Dec 05 13:30:25 REDACTED systemd[1]: snapd.service: Consumed 2min 31.066s CPU time.
Dec 05 13:46:50 REDACTED systemd[1]: Starting Snap Daemon...
Dec 05 13:46:50 REDACTED snapd[3573933]: AppArmor status: apparmor is enabled and all features are available
Dec 05 13:46:51 REDACTED snapd[3573933]: overlord.go:274: Acquiring state lock file
Dec 05 13:46:51 REDACTED snapd[3573933]: overlord.go:279: Acquired state lock file
Dec 05 13:46:52 REDACTED snapd[3573933]: daemon.go:250: started snapd/2.66.1 (series 16; classic) ubuntu/22.04 (amd64) linux/6.0.0-1006-oem.
Dec 05 13:46:52 REDACTED snapd[3573933]: daemon.go:353: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Dec 05 13:46:52 REDACTED snapd[3573933]: backends.go:58: AppArmor status: apparmor is enabled and all features are available (using snapd provided apparmor_parser)
Dec 05 13:46:52 REDACTED systemd[1]: Started Snap Daemon.
Dec 05 13:47:24 REDACTED snapd[3573933]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/sections": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
Dec 05 14:12:02 REDACTED systemd[1]: snapd.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Dec 05 14:12:02 REDACTED systemd[1]: Stopping snapd.service...
Dec 05 14:12:03 REDACTED snapd[3573933]: main.go:168: Exiting on terminated signal.
Dec 05 14:12:06 REDACTED systemd[1]: snapd.service: Got notification message from PID 3573933, but reception is disabled.
Dec 05 14:12:06 REDACTED snapd[3573933]: overlord.go:518: Released state lock file
Dec 05 14:12:06 REDACTED systemd[1]: snapd.service: Deactivated successfully.
Dec 05 14:12:06 REDACTED systemd[1]: Stopped snapd.service.
Dec 05 14:12:06 REDACTED systemd[1]: snapd.service: Consumed 4.673s CPU time.
/var/log/dpkg.log:
/var/log/apt/history.log:

ryan.rubley · December 7, 2024, 12:12am

You should be aware that you are on an unsupported setup here…

Not the default does not mean totally unsupported. The ifupdown package is officially available from Ubuntu.

Debian still uses a new variant ifupdown

We may very well be forced to move from Ubuntu to Debian in the future due to the direction things are heading. But for now, we have a fleet of devices running various Ubuntu LTS back to 14.04 that we need to operate as similar as possible to newer hardware. Having totally different network management or package management makes certain things total non-starters.

ogra · December 8, 2024, 10:39pm

The package is in universe which means it gets synced from Debian during the development cycle a few times but that is all the maintenance it gets. There is probably still an old patch set on top of it to support upgrades from old installs but to get any additional maintenance for it you’d need some community maintainer to step up doing it…

Since it isn’t on any install media it will also not get any field testing during a development cycle …

I agree that having to manage multiple setups is indeed a pain, but keeping the whole setup on an unsupported path because you have one release in use (that will soon even be out of paid support) that does no yet support the new (and way more scalable) way seems a bit odd…

ryan.rubley · December 8, 2024, 11:04pm

Well, this is not the place to be combative about other company’s choices. I’m simply trying to help provide information about what we see as a clear problem triggered by snapd. Trying to shift blame or focus on how supported various packages are, is honestly, totally off-base and missing the point here.

ogra · December 8, 2024, 11:29pm

I’m not trying to put up blame or anything and I didn’t mean to de-rail the topic (though I can’t imagine how snapd could in any way have an impact on your DNS settings).

Sorry if it came across like that, as I said in the beginning I don’t want to talk you out of your setup or anything. It just seems pretty risky to build a production setup on something abandoned/unsupported, that’s all I wanted to point out, I’ll keep quiet about that stuff now

yhodge · December 9, 2024, 5:14am

Out of sheer curiosity, I dug into this a little bit.

Ultimately, I don’t believe a change in snapd itself has caused this to occur, however the snap refresh of snapd is likely what sets this off judging on the timestamps in the original post.

Looking at the journal logs for systemd-resolved, it appears to be constantly restarting, which might be from the /etc/network/if-up.d/resolved script hitting this part in a loop:

        # resolved was running, but without dbus, it means state files
        # will not be read & resolvectl commands failed, restart it
        if [ "$resolvectl_failed" ]; then
                systemctl try-restart systemd-resolved
        fi

at various stages of that script, it calls resolvectl and conditionally sets the exit code to a variable if not exit code 0:

resolvectl dns "$ifindex" $DNS6 $DNS || resolvectl_failed=$?

back in the journal, it seems that we did not get this line:

Dec 05 13:53:26 REDACTED systemd-resolved[3579545]: eno1: Bus client set search domain list to: DOMAINS
Dec 05 13:53:26 REDACTED systemd-resolved[3579545]: eno1: Bus client set DNS server list to: REDACTED, REDACTED

Until the cable was pulled and plugged back in:

Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Link DOWN
Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Lost carrier
Dec 05 13:53:26 REDACTED systemd-networkd[753]: eno1: Link UP
Dec 05 13:53:29 REDACTED systemd-networkd[753]: eno1: Gained carrier
Dec 05 13:53:31 REDACTED systemd-networkd[753]: eno1: Gained IPv6LL

My suspicion is that ifupdown has got itself into a loop trying to configure systemd-resolved, but fails and then tries again to the same result after restarting the service. (Script seems to be reliant on dbus?)

I would investigate the journal around the time of the snapd refresh but also look at what other services are doing, maybe the restart of snapd has an odd systemd dependency hook that’s putting the system into this state?

I’d focus on dbus first, but perhaps also systemd-udevd which had some activity in the journal in your first post after the snapd refresh.

Hope this gets you in the right direction!

ryan.rubley · December 9, 2024, 6:21am

The event that lost the DNS took place at “Nov 28 01:52:47” on the device in question here. The Dec 5 was likely a fix or debug attempts where on this (lab device not in production) we manually restarted various services and may have run ifdown/ifup on Dec 5. We tried unsuccessfully to reproduce the issue as well, through various service restarts and such, but were never able to get it to happen again. It only occurred on the first start-up of snapd 2.66.1 after it upgraded itself.

mborzecki1 · December 9, 2024, 7:19am

One other possible vector I forgot about earlier is when snapd regenerates sandbox profiles. Among various sandbox backends, there’s a udev one. Snapd writes out rules files that tag devices for access from respective snaps. This needs to be followed by udev reload and trigger and happens only when the content of rules files is different than previously. This operates only on char and block devices, and should not affect network interfaces.

Try switching snapd to previous version and back. Run snap list --all snap, make sure that inactive revision is 2.63, then snap revert snapd and snap refresh --stable snapd.

jocado · December 9, 2024, 8:47am

If you’re going to try reverting snapd to the previous version on a classic system, don’t you need to revert the deb package for snapd as well ?

Cheers, Just

ryan.rubley · December 9, 2024, 11:16pm

Strange enough, when I did the revert on a lab devices, the issue re-occured. The /run/systemd/resolve/resolv.conf file was written with “# No DNS servers known.” I restarted systemd-resolved and it fixed that file. I then refreshed snapd and the issue did not occur.

mborzecki1 · December 11, 2024, 4:05pm

Were you able to find anything interesting in the logs?

Without doing further changes to the snapd snap, can you try running systemctl daemon-reload a couple of times to see if it triggers a problem? If that command does not trigger any problems with resolved, I would try udevadm trigger --settle a couple of times next.

ryan.rubley · December 11, 2024, 11:08pm

I tried reverting and refreshing snapd a dozen more times after that, and the issue did not reoccur.

I tried systemctl daemon-reload 10 times just now, and it did not trigger the issue.

Then I tried udevadm trigger --settle and managed to reproduce the issue. Running systemctl restart systemd-resolved.service would fix the issue. I repeated this process about 20 times and noticed it only causes the issue about 20-30% of the time. Most of the times, running this udevadm command does not break the DNS settings.

mborzecki1 · January 8, 2025, 9:27am

Were you able to dig more into systemd-resolved?

In any case, I think at this point I would suggest to file a bug for systemd in LP https://bugs.launchpad.net/ubuntu/+source/systemd