I have a snap that incorporates a few systemd services. In a new release of the snap, I’ve added a new service that depends on one of the existing ones to run. For reference, the snapcraft.yaml file is here, with the new ae-pub-meta service added here. ae-pub-meta depends on the mosquitto service to run, and waits for the Mosquitto broker to be up in order to run successfully.
I’m seeing very odd behavior when trying to do snap refresh from the previous release of the snap - which doesn’t have the ae-pub-meta service. Namely, snapd will disable all existing services for this snap and only try to start ae-pub-meta. Also not the mosquitto service, which ae-pub-meta depends on in order to run.
Hereby snap refresh simply hangs on the Start snap "ammp-edge" (809) services step, until I manually find and kill the process. That’s obviously not a viable situation for refreshing in practice.
The system log at the time of the refresh contains the following:
Sep 23 11:10:49 hostname systemd[1]: Mounting Mount unit for ammp-edge, revision 809...
Sep 23 11:10:49 hostname systemd[1]: Mounted Mount unit for ammp-edge, revision 809.
Sep 23 11:10:50 hostname systemd[1]: snap.ammp-edge.env-scan.timer: Succeeded.
Sep 23 11:10:50 hostname systemd[1]: Stopped Timer env-scan for snap application ammp-edge.env-scan.
Sep 23 11:10:50 hostname snapd[1159]: services.go:1039: RemoveSnapServices - disabling snap.ammp-edge.mosquitto.service
Sep 23 11:10:50 hostname snapd[1159]: services.go:1039: RemoveSnapServices - disabling snap.ammp-edge.ae-init.service
Sep 23 11:10:50 hostname snapd[1159]: services.go:1039: RemoveSnapServices - disabling snap.ammp-edge.ammp-edge.service
Sep 23 11:10:50 hostname snapd[1159]: services.go:1039: RemoveSnapServices - disabling snap.ammp-edge.web-ui.service
Sep 23 11:10:50 hostname snapd[1159]: services.go:1039: RemoveSnapServices - disabling snap.ammp-edge.wifi-ap-control.service
Sep 23 11:10:50 hostname snapd[1159]: services.go:1029: RemoveSnapServices - timer snap.ammp-edge.env-scan.timer
Sep 23 11:10:50 hostname snapd[1159]: services.go:1039: RemoveSnapServices - disabling snap.ammp-edge.env-scan.service
Sep 23 11:10:51 hostname systemd[1]: Reloading.
Sep 23 11:10:54 hostname systemd[1]: Reloading.
Sep 23 11:10:56 hostname systemd[1]: Reloading.
Sep 23 11:10:57 hostname systemd[1]: Started Timer env-scan for snap application ammp-edge.env-scan.
Sep 23 11:10:57 hostname systemd[1]: Starting Service for snap application ammp-edge.ae-pub-meta...
Sep 23 11:10:57 hostname ammp-edge.ae-pub-meta[103581]: [INFO ae::interfaces::mqtt] Establishing MQTT connection to localhost:1883 as local-pub-meta-18d8fe
Sep 23 11:10:57 hostname ammp-edge.ae-pub-meta[103581]: [ERROR ae::command::mqtt_pub] MQTT publish error after 0.4s: I/O: Connection refused (os error 111)
Sep 23 11:10:58 hostname ammp-edge.ae-pub-meta[103581]: [INFO ae::interfaces::mqtt] Establishing MQTT connection to localhost:1883 as local-pub-meta-8e3e45
Sep 23 11:10:58 hostname ammp-edge.ae-pub-meta[103581]: [ERROR ae::command::mqtt_pub] MQTT publish error after 0.7s: I/O: Connection refused (os error 111)
Sep 23 11:10:58 hostname ammp-edge.ae-pub-meta[103581]: [INFO ae::interfaces::mqtt] Establishing MQTT connection to localhost:1883 as local-pub-meta-b9b7f8
...ad infinitum
Based on this, it looks like snapd disables all existing services for that snap before trying to start ae-pub-meta by itself.
Removing the snap and installing it afresh works just fine. It’s only the refresh that fails in this way.
I believe I’m running the latest versions:
$ snap version
snap 2.57.1
snapd 2.57.1
series 16
debian 11
kernel 5.10.103-v8+