This is one of the issues we haven’t yet implemented a solution for (part of the reason user-daemons hasn’t been enabled by default yet).
Getting this all working is non-trivial, but doable. To understand, it helps to understand how that snapctl command works. In essence, it does the following:
a process from snap myapp executes the snapctl stop daemon command.
the snapctl executable makes a corresponding API call to snapd.
snapd identifies the caller as running in the context of the myapp snap.
snapd checks the SO_PEERCRED credentials to make sure the caller is root.
snapd asks the system instance of systemd to stop snap.myapp.daemon.service.
The error you’re seeing is from step (4). But supporting user daemons is not as simple as removing the check. User daemons are not managed by the system instance of systemd: instead they’re managed by a separate user instance (one per user logged in on the system). The user instances also refuse to talk to any process owned by a different user ID, so snapd running as root cannot directly talk to it.
We have put together a system to allow snapd to communicate with running user sessions in the form of the snap session agent, and are already using it to e.g. stop user daemons before uninstalling or upgrading a snap. We could use the same to handle the snapctl start/stop commands.
Of course, there’s still the question of what snapctl start daemon should actually do for a user daemon. Should it just affect the instance of the daemon for the calling user, or should it try to control all running user sessions? The first is the case is the one you’re asking for. The second obviously requires root, but might be useful for hooks.
For the “control current user’s daemon” case, there’s also the question of whether going via snapd is desirable, since it should be possible without escalating to root privileges. It might be difficult to add a fast path given the snapctl architecture where all the smarts are on the snapd side.
Thank you for the detailed explanation! Quite complicated indeed, I appreciate your work. I’ve been thinking more about my user daemon “restart on command” use case. The main reason for it was to reload config. Instead I’ll just have the two apps use dbus to communicate, since I also want the GUI app to monitor and report the daemon’s health anyway, and make sure the daemon is resilient enough to not exit unnecessarily.
In any case, I’m sure other use cases exist to require working snapctl start/stop commands. I hope my case helps you and others decide priorities. For instance, I would love to see the user daemons feature become available (almost) as it exists now, despite the missing functionality. In other words, would it make sense to add guards around or disable the snapctl commands that don’t currently work for user daemons, and release a lean version of the feature earlier, vs. implementing everything that you described?
@jamesh I ran into a problem where my user daemon doesn’t start upon first install and doesn’t stop after removal, but only when running on Manjaro with xorg. I think refreshes are a problem too. I see some discussion in https://github.com/snapcore/snapd/pull/5822 about how snapd can’t stop user services and some systemd versions don’t work well with the user-daemons experimental feature. I was wondering if you could confirm this is the issue I am facing.
I tested my (still unreleased) snap on other linux variants (see First snap - testing across different linux variants) and the user daemon worked really well on all except manjaro with org (e.g. xfce). It seems that there is something in this particular setup that’s different from the others with respect to how snapd works with systemd.
This is great work. Can you tell me if user daemons is moving forward and when it is reasonable to think it might make it to stable? This thread seemed not to make it into 2022. Just for clarity, I am referring to the “experimental.user-daemons”.
It hasn’t been abandoned: in fact, I’m relying on it for one of the internal projects we’re working on.
Getting it turned on by default has stalled a bit though, as there were some features available for system daemons that are not implemented for user daemons: namely service control through snapctl. At the moment those commands return errors for user daemons, so perhaps it is worth reconsidering whether that’s a blocker for enabling the feature by default.
Thanks jamesh. So glad to hear it is moving forward. Our project is operating un-manned and headless so we do need it turned on by default. Do you know where in the priority list this might be?
