Snapd gets stuck, doesn't process changes

On Kubuntu 24.04, I’ve now twice had a situation where snap gets stuck, and just stops processing any changes (installing, removing, refreshing snaps).

Installed snaps still work, I can do snap list / changes / etc., but any changes to snaps just hangs. ‘snap changes’ then shows the stuck changes as status ‘Hold’, and all tasks under the changes as ‘Hold’ too.

I tried following Comment #16 : Bug #2130315 : Bugs : snapd package : Ubuntu to just remove the entries, but didn’t change anything. When trying to do any operation after removing the changes, the changes would just get stuck again.

Restart of snapd didn’t do anything, neither did a reboot. In both cases, I had to apt purge snapd, then reinstall together with all my snaps to it unstuck.

I collected the following logs from ‘journalctl -r -u snapd.service’. June 18 is the boot where things apparently started failing (error: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available). June 19 I then realized that snap was stuck. Every subsequent boot looks the same, with no further output no matter what snap install/remove operations I try, and all new changes just ending up getting stuck.

-- Boot 22b1a49afcc74a9c824ba48817acbc69 --
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: snapd.service: Consumed 27.978s CPU time, 2.5G memory peak, 0B memory swap peak.
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: snapd.service: Failed to enqueue OnFailure= job, ignoring: Transaction for snapd.failure.service/start is destructive (local-fs-pre.target has 'stop' job queued, but 'start' is included in tran>
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: snapd.service: Triggering OnFailure= dependencies.
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: Stopped snapd.service - Snap Daemon.
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: snapd.service: Unit process 74222 (udevadm) remains running after unit stopped.
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: snapd.service: Failed with result 'timeout'.
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: snapd.service: Killing process 68204 (snapd) with signal SIGKILL.
Jun 19 16:25:54 ZRHLTPF5MR365 systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.
Jun 19 16:24:27 ZRHLTPF5MR365 snapd[68204]: standby.go:121: standby monitoring stop requested
Jun 19 16:24:24 ZRHLTPF5MR365 snapd[68204]: main.go:168: Exiting on terminated signal.
Jun 19 16:24:24 ZRHLTPF5MR365 systemd[1]: Stopping snapd.service - Snap Daemon...
Jun 19 07:51:40 ZRHLTPF5MR365 snapd[68204]: cache.go:243: removed 0 entries/    0 from downloads cache
Jun 19 07:51:40 ZRHLTPF5MR365 snapd[68204]: snapmgr.go:1661: performing periodic snap downloads cache cleanup
Jun 19 07:51:07 ZRHLTPF5MR365 systemd[1]: Started snapd.service - Snap Daemon.
Jun 19 07:51:07 ZRHLTPF5MR365 snapd[68204]: standby.go:95: will consider standby after: 5s
Jun 19 07:51:07 ZRHLTPF5MR365 snapd[68204]: backend.go:145: reloading profiles for snap-confine
Jun 19 07:51:07 ZRHLTPF5MR365 snapd[68204]: backends.go:70: AppArmor status: apparmor is enabled and all features are available
Jun 19 07:51:07 ZRHLTPF5MR365 snapd[68204]: daemon.go:370: adjusting startup timeout by 2m10s (pessimistic estimate of 30s plus 5s per snap)
Jun 19 07:51:07 ZRHLTPF5MR365 snapd[68204]: daemon.go:276: started snapd/2.75.2+ubuntu24.04 (series 16; classic) ubuntu/24.04 (amd64) linux/6.8.0-124-generic.
Jun 19 07:51:07 ZRHLTPF5MR365 snapd[68204]: overlord.go:314: Acquired state lock file
Jun 19 07:51:07 ZRHLTPF5MR365 snapd[68204]: overlord.go:309: Acquiring state lock file
Jun 19 07:51:06 ZRHLTPF5MR365 systemd[1]: Starting snapd.service - Snap Daemon...
-- Boot a1629ffea2f44e7a86f340c55a818c6d --
Jun 18 17:02:50 ZRHLTPF5MR365 systemd[1]: snapd.service: Consumed 15min 37.063s CPU time, 2.8G memory peak, 0B memory swap peak.
Jun 18 17:02:50 ZRHLTPF5MR365 systemd[1]: Stopped snapd.service - Snap Daemon.
Jun 18 17:02:50 ZRHLTPF5MR365 systemd[1]: snapd.service: Deactivated successfully.
Jun 18 17:02:50 ZRHLTPF5MR365 snapd[54328]: overlord.go:558: Released state lock file
Jun 18 17:02:50 ZRHLTPF5MR365 snapd[54328]: standby.go:121: standby monitoring stop requested
Jun 18 17:02:47 ZRHLTPF5MR365 systemd[1]: Stopping snapd.service - Snap Daemon...
Jun 18 17:02:47 ZRHLTPF5MR365 snapd[54328]: main.go:168: Exiting on terminated signal.
Jun 18 14:28:47 ZRHLTPF5MR365 snapd[54328]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Jun 18 12:12:19 ZRHLTPF5MR365 snapd[54328]: storehelpers.go:914: cannot refresh: snap has no updates available: "bare", "chromium-ffmpeg", "clion", "code", "core20", "core22", "core24", "gimp", "gnome-46-2404", "gtk-common-themes", "lx>
Jun 18 09:52:43 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:52:42 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:29:58 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:07:43 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:07:41 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:07:40 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:07:40 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:07:40 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:07:37 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:07:37 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 09:07:34 ZRHLTPF5MR365 snapd[54328]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Jun 18 09:07:24 ZRHLTPF5MR365 snapd[54328]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Jun 18 09:07:13 ZRHLTPF5MR365 snapd[54328]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Jun 18 09:07:03 ZRHLTPF5MR365 snapd[54328]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Jun 18 09:06:52 ZRHLTPF5MR365 snapd[54328]: backend.go:285: reloading profiles of snap-confine provided by the system snap
Jun 18 09:06:49 ZRHLTPF5MR365 snapd[54328]: udevmon.go:149: udev event error: Unable to parse uevent, err: no buffer space available
Jun 18 08:25:25 ZRHLTPF5MR365 snapd[54328]: storehelpers.go:914: cannot refresh: snap has no updates available: "bare", "chromium-ffmpeg", "clion", "code", "core20", "core22", "core24", "gimp", "gnome-46-2404", "gtk-common-themes", "lx>
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: cache.go:243: removed 1 entries/44.9M from downloads cache
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: snapmgr.go:1661: performing periodic snap downloads cache cleanup
Jun 18 08:20:23 ZRHLTPF5MR365 systemd[1]: Started snapd.service - Snap Daemon.
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: standby.go:95: will consider standby after: 5s
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: backend.go:145: reloading profiles for snap-confine
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: backends.go:70: AppArmor status: apparmor is enabled and all features are available
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: daemon.go:370: adjusting startup timeout by 2m5s (pessimistic estimate of 30s plus 5s per snap)
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: daemon.go:276: started snapd/2.75.2+ubuntu24.04 (series 16; classic) ubuntu/24.04 (amd64) linux/6.8.0-124-generic.
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: overlord.go:314: Acquired state lock file
Jun 18 08:20:23 ZRHLTPF5MR365 snapd[54328]: overlord.go:309: Acquiring state lock file
Jun 18 08:20:23 ZRHLTPF5MR365 systemd[1]: Starting snapd.service - Snap Daemon...

Any ideas about what could be wrong or how to further debug this?

Please attach the output of the following commands:

  • snap changes
  • snap change <id>, where <id> is the ID of the change that appears to be stuck
  • snap version --verbose

I had the following (software-nightly is an internal snap that I’m working on - might very well be related to that, but I guess it in any case shouldn’t be able to corrupt the full snapd installation):

> snap changes
ID   Status  Spawn                      Ready                      Summary
75   Done    2 days ago, at 17:41 CEST  2 days ago, at 19:58 CEST  Auto-refresh snap "vivaldi"
76   Done    yesterday at 09:06 CEST    yesterday at 09:07 CEST    Install "software-nightly" snap from file "/tmp/tmp.omP0fjjzKE/software-nightly_702026061800_amd64.snap"
77   Done    yesterday at 09:07 CEST    yesterday at 09:07 CEST    Connect software-nightly:dot-config to snapd:personal-files
78   Done    yesterday at 09:07 CEST    yesterday at 09:07 CEST    Connect software-nightly:aws-credentials to snapd:personal-files
79   Done    yesterday at 09:07 CEST    yesterday at 09:07 CEST    Connect software-nightly:cache to snapd:personal-files
80   Done    yesterday at 09:07 CEST    yesterday at 09:07 CEST    Connect software-nightly:raw-usb to snapd:raw-usb
81   Done    yesterday at 09:07 CEST    yesterday at 09:07 CEST    Connect software-nightly:shared-memory to snapd:shared-memory
82   Done    yesterday at 14:28 CEST    yesterday at 14:29 CEST    Install "software-nightly" snap from file "/tmp/tmp.EYpDIezRML/software-nightly_702026061812_amd64.snap"
83   Done    yesterday at 14:29 CEST    yesterday at 14:29 CEST    Connect software-nightly:dot-config to snapd:personal-files
84   Done    yesterday at 14:29 CEST    yesterday at 14:29 CEST    Connect software-nightly:aws-credentials to snapd:personal-files
85   Done    yesterday at 14:29 CEST    yesterday at 14:29 CEST    Connect software-nightly:cache to snapd:personal-files
86   Done    yesterday at 14:29 CEST    yesterday at 14:29 CEST    Connect software-nightly:raw-usb to snapd:raw-usb
87   Done    yesterday at 14:29 CEST    yesterday at 14:29 CEST    Connect software-nightly:shared-memory to snapd:shared-memory
88   Hold    today at 14:37 CEST        today at 15:31 CEST        Install "software-nightly" snap from file "/tmp/tmp.a9QphU7RMw/software-nightly_702026061912_amd64.snap"

And:

> snap change 88
Status  Spawn                Ready                Summary
Hold    today at 14:37 CEST  today at 15:31 CEST  Ensure prerequisites for "software-nightly" are available
Hold    today at 14:37 CEST  today at 15:31 CEST  Prepare snap "/var/lib/snapd/snaps/.local-install-765975256" (unset)
Hold    today at 14:37 CEST  today at 15:31 CEST  Mount snap "software-nightly" (unset) 
Hold    today at 14:37 CEST  today at 15:31 CEST  Run pre-refresh hook of "software-nightly" snap if present
Hold    today at 14:37 CEST  today at 15:31 CEST  Stop snap "software-nightly" services 
Hold    today at 14:37 CEST  today at 15:31 CEST  Remove aliases for snap "software-nightly"
Hold    today at 14:37 CEST  today at 15:31 CEST  Make current revision for snap "software-nightly" unavailable
Hold    today at 14:37 CEST  today at 15:31 CEST  Copy snap "software-nightly" data
Hold    today at 14:37 CEST  today at 15:31 CEST  Setup snap "software-nightly" (unset) security profiles
Hold    today at 14:37 CEST  today at 15:31 CEST  Make snap "software-nightly" (unset) available to the system
Hold    today at 14:37 CEST  today at 15:31 CEST  Automatically connect eligible plugs and slots of snap "software-nightly"
Hold    today at 14:37 CEST  today at 15:31 CEST  Set automatic aliases for snap "software-nightly"
Hold    today at 14:37 CEST  today at 15:31 CEST  Setup snap "software-nightly" aliases 
Hold    today at 14:37 CEST  today at 15:31 CEST  Run post-refresh hook of "software-nightly" snap if present
Hold    today at 14:37 CEST  today at 15:31 CEST  Start snap "software-nightly" (unset) services
Hold    today at 14:37 CEST  today at 15:31 CEST  Remove data for snap "software-nightly" (x1)
Hold    today at 14:37 CEST  today at 15:31 CEST  Remove snap "software-nightly" (x1) from the system
Hold    today at 14:37 CEST  today at 15:31 CEST  Clean up "software-nightly" (unset) install
Hold    today at 14:37 CEST  today at 15:31 CEST  Run configure hook of "software-nightly" snap if present
Hold    today at 14:37 CEST  today at 15:31 CEST  Run health check of "software-nightly" snap
Hold    today at 14:37 CEST  today at 15:31 CEST  Process delayed security backend side effects for affected snaps

And:

> snap version --verbose 
snap            2.75.2+ubuntu24.04
snapd           2.75.2+ubuntu24.04
series          16
ubuntu          24.04
kernel          6.8.0-124-generic
architecture    amd64
snapd-bin-from  native-package
snap-bin-from   native-package

Note that I don’t really have any further logs from the broken state, as I purged and reinstalled snapd to get it back to working.

I’m confused. It looks like a change was created but not started at all. I need to look at the code to find out if that’s even possible.

do you mean you tried installing/manipulating other snaps and that didn’t work at all?

Correct, any change just stays hanging once in this state. Also meant that other snaps didn’t update.