Refreshing a snap does not restart running services

snapd
1

Seems to be a bug introduced with 2.67(.1 ?) as it was working before.

I have a snap with services running in daemon: simple. It is in install-mode: disable and refresh-mode is default value (so restart)

I perform a refresh to a new revision (from rev 127 to rev 128), with no big changes.

Then service ends up in state inactive, while snap changes says the services were restarted.

Status  Spawn               Ready               Summary
Done    today at 11:40 UTC  today at 11:40 UTC  Ensure prerequisites for "charmed-mongodb" are available
Done    today at 11:40 UTC  today at 11:40 UTC  Prepare snap "/var/lib/snapd/snaps/charmed-mongodb_128.snap" (128)
Done    today at 11:40 UTC  today at 11:40 UTC  Run pre-refresh hook of "charmed-mongodb" snap if present
Done    today at 11:40 UTC  today at 11:40 UTC  Stop snap "charmed-mongodb" services
Done    today at 11:40 UTC  today at 11:40 UTC  Remove aliases for snap "charmed-mongodb"
Done    today at 11:40 UTC  today at 11:40 UTC  Make current revision for snap "charmed-mongodb" unavailable
Done    today at 11:40 UTC  today at 11:40 UTC  Copy snap "charmed-mongodb" data
Done    today at 11:40 UTC  today at 11:40 UTC  Setup snap "charmed-mongodb" (128) security profiles
Done    today at 11:40 UTC  today at 11:40 UTC  Make snap "charmed-mongodb" (128) available to the system
Done    today at 11:40 UTC  today at 11:40 UTC  Automatically connect eligible plugs and slots of snap "charmed-mongodb"
Done    today at 11:40 UTC  today at 11:40 UTC  Set automatic aliases for snap "charmed-mongodb"
Done    today at 11:40 UTC  today at 11:40 UTC  Setup snap "charmed-mongodb" aliases
Done    today at 11:40 UTC  today at 11:40 UTC  Run post-refresh hook of "charmed-mongodb" snap if present
Done    today at 11:40 UTC  today at 11:40 UTC  Start snap "charmed-mongodb" (128) services
Done    today at 11:40 UTC  today at 11:40 UTC  Clean up "charmed-mongodb" (128) install
Done    today at 11:40 UTC  today at 11:40 UTC  Run configure hook of "charmed-mongodb" snap if present
Done    today at 11:40 UTC  today at 11:40 UTC  Run health check of "charmed-mongodb" snap
Done    today at 11:40 UTC  today at 11:40 UTC  Monitoring snap "charmed-mongodb" to determine whether extra refresh steps are required

Running journalctl -u snapd I see:

Apr 02 11:40:36 juju-4af4d5-2 snapd[13103]: services.go:1152: RemoveSnapServices - disabling snap.charmed-mongodb.mongodb-exporter.service
Apr 02 11:40:36 juju-4af4d5-2 snapd[13103]: services.go:1152: RemoveSnapServices - disabling snap.charmed-mongodb.pbm-agent.service
Apr 02 11:40:36 juju-4af4d5-2 snapd[13103]: services.go:1152: RemoveSnapServices - disabling snap.charmed-mongodb.mongos.service
Apr 02 11:40:36 juju-4af4d5-2 snapd[13103]: services.go:1152: RemoveSnapServices - disabling snap.charmed-mongodb.mongod.service

I have confirmed this with another snap (charmed-etcd) where this is also the case.

This used to work as we were testing our charm upgrades and it always restarted correctly, so I’m sending a message for help here in case anybody has an idea or if it’s a possible bug?

Some more information: Base is 22.04, but it also fails on 24.04. This is running in an LXD container. I can’t find a way to install former versions of snapd because I can’t find a way to find the revision associated with snapd 2.66.x.

2 Likes
2

Have you tried using systemctl show snap.<name>.<svcname>.service to see when it was started? There’ll be ExecMainThreadTimestamp… or something along those lines in the output.

3

Before the refresh, the service is running:

snap.charmed-mongodb.mongod.service - Service for snap application charmed-mongodb.mongod
   Loaded: loaded (/etc/systemd/system/snap.charmed-mongodb.mongod.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2025-04-02 13:14:18 UTC; 1min 33s ago
 Main PID: 15773 (mongod)
    Tasks: 90 (limit: 37869)
   Memory: 162.3M
      CPU: 4.696s
   CGroup: /system.slice/snap.charmed-mongodb.mongod.service
           └─15773 /snap/charmed-mongodb/127/usr/bin/mongod --config /var/snap/charmed-mongodb/127/etc/mongod/mongod.conf

Then I refresh and it ends up in

snap.charmed-mongodb.mongod.service - Service for snap application charmed-mongodb.mongod
   Loaded: loaded (/etc/systemd/system/snap.charmed-mongodb.mongod.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

For the two systemctl show, we have before and after: ExecMainStartTimestamp=Wed 2025-04-02 13:14:18 UTC vs ExecMainStartTimestamp=n/a

It seems that the service is disabled, then removed and recreated in a disabled state and not started at all, which is the weird part.

4

Thank you for opening a topic on this. This was actually quite interesting to dive into. My original thought would be that the service was indeed restarted, however some investigating and confirming my original thoughts are that this is actually the currently intended behaviour.

If you start a disabled service (and not enable it), then the service won’t be restarted on refreshes (succesful and failed), unless the service has refresh-mode set to endure. If this is set, the service state will persist through refreshes.

This may seem a bit counterintuitive, but the act of starting a disabled service is seen kinda as a one-off operation, meaning this only persists until something disruptive as a refresh or reboot.

If you need it to persist, you must enable the service first or configure the refresh behavior for the service.

The behavior you saw prior to 2.67 may have been the bug, I can just confirm that currently it’s working like expected.

2 Likes
5

Thanks a lot for the answer!

I can confirm that if I enable the service, it works and the service is restarted as expected. Glad that we could have this chat and I’ll fix our charms then so that we enable the services on start.