Enabling user daemons and D-Bus activation

Currently snapd includes support for user daemons and dbus activation that is currently gated behind experimental feature flags. I would like to see those flags turned on by default. For reference, these flags enable the following:

  1. experimental.user-daemons adds a new app property daemon-scope. When set to user, this causes the daemon to be installed to run under user session instances of systemd rather than the pid 1 system instance.

  2. experimental.dbus-activation adds a new app property activates-on, which contains a list of slot names referencing dbus interface slots a daemon should activate on. If set, the daemon’s start up will be delayed until any of the given service names are used on the relevant bus.

Both flags can be confined to support D-Bus session bus activatable services. But each flag can be useful on its own. The user-daemons flag on its own enables user session timers and services. The dbus-activation flag on its own enables activatable system bus services.

Below are the things I believe need to be done before these features can be enabled. If anyone has anything needed to complete it, that would be great.

user-daemons

The user-daemons feature flag enables snap.yaml syntax like the following:

apps:
  myservice:
    command: bin/myservice
    daemon: simple
    daemon-scope: user

The daemon-scope property can take the values system or user, with system as the default providing the current behaviour. An app that sets daemon-scope: user will instead be registered with the user session instance(s) of systemd. All properties used to configure system daemons can also be used to configure user session daemons too. This means things like socket activation and timers will also work for user session daemons.

When snapd installs, upgrades, or removes a snap containing a user daemon, it contacts the snap session agent for each running user session to ask that they start or stop the associated daemons. At present, that is the extent to how much snapd controls user daemons at present. Snapd does provide other ways of interacting with system daemons, which have not been updated though:

  1. At the moment, snapd records the enabled state of all system daemons before an upgrade, and restores that state after the upgrade. It should do the same for user daemons.

  2. snap services lists the status of system daemons. We can easily enumerate user daemons too and determine their global enabled/disabled state. We should be able to determine the active state of user daemons via the session agent, but as there could be 0…n instances of the daemon it might not be cleanly “active” or “inactive”.

  3. snap start and snap stop can be used to start and stop system daemons, and optionally enable or disable the service. This could be extended to cover user daemons, but it opens the question of whether they should affect all instances of the daemon or just that for the current user. Maybe we want to support both, but via an option for the command?

    If we do allow starting/stopping an individual user’s instance of a user daemon, should unprivileged users be able to call these commands? We can certainly detect the calling process’s user ID within the REST API, but does exposing those endpoints to unprivileged users have any unwanted side effects?

    If we allow unprivileged users to control their own instance of the daemon, does that mean we need to track per-user enabled state of user daemons? (currently we only care about the global enabled state).

  4. There are snapctl equivalents of all of the above that can be called by code running in the context of a snap, and limited to daemons belonging to the snap. I believe the same concerns are relevant here.

I think that covers all the missing features. It’s possible that we decide not to implement some of these features for user daemons, or decide that they shouldn’t block us from turning on the feature. I think (1) probably needs doing though.

dbus-activation

I think dbus-activation is the less risky one to enable by default. Here is an example of the snap.yaml syntax for a D-Bus activated service:

slots:
  dbus-slot:
    interface: dbus
    bus: system
    name: org.example.Foo
apps:
  myservice:
    command: bin/myservice
    daemon: simple
    activates-on: [dbus-slot]

Compared with a similar snap.yaml without the activates-on property, the behaviour is changed as follows:

  1. The systemd service generated for myservice is no longer wanted by multi-user.target, so will not be started on boot.
  2. A /var/lib/snapd/dbus-1/system-services/org.example.Foo.service file will be written out allow dbus-daemon to start the service. The service activation file includes a SystemdService key so that dbus-daemon will start the corresponding systemd service. The AssumedAppArmorLabel key is also set so that communication that depends on the label (e.g. mediated method calls between two snaps) can activate the daemon.

The snap has no more access to the system, since without activates-on, it could have the daemon start on boot and request the given D-Bus name. The main change is that the snap can now delay initialisation until (or even if) the service is used. If the service implements some kind of “exit on idle” policy, it may even be able to remain stopped most of the time.

And since the activated daemon is still run under systemd, snapd can control it the same as any other daemon provided by a snap.

Developer Testing

I’d welcome feedback from developers who would like to experiment with these features. You can enable them on your system by executing the following commands:

snap set system experimental.user-daemons=true
snap set system experimental.dbus-activation=true

The new syntax is not supported by Snapcraft yet (there are PR #3129 and PR #3425, but they probably won’t be merged until the features are enabled), but they can still be used via the passthrough: feature described here:

I’d welcome any feedback, including where things do not behave as you’d expect or additional issues you believe should be addressed before the features are enabled.

2 Likes

I illustrated some of my thinking about this in https://bugs.launchpad.net/snapd/+bug/1912120 but to summarize, I think that we should exclude user daemons from the default output of snap services unless there is a root session and an instance of the user service running in that root session, and instead relegate getting user session service status’s to a dedicated command like snap services --user for the current user possibly also allowing (with sufficient authorization i.e. root) something like snap services --user=1002 or something. In the snap services --user case, probably the snap command could just talk directly to the user session agent, but in the --user=... case probably talking to snapd first is necessary which would then signal to the individual user session agent for that user.

IMHO, all of 1-4 are blockers (and also the referenced/linked bug) before user session daemons can be non-experimental.

Of course this is all just my thinking and really we need architect input here.

I think we need one more thing for dbus-activation before that can be unexperimental and that is integration with snap services and/or snap info where you get some information about the DBus activation about a service from those outputs. snap info is quite busy so I think we decided not to add anything more there until we can clean it up, but definitely something in snap services is appropriate IMHO

1 Like

Thanks for the detailed feedback!

I agree that snap services and friends should not malfunction in the presence of user daemons, and I’ll see if I can work on fixes for the problems you’ve identified. My thought was more that extending these utilities to be able to usefully manipulate user daemons (as opposed to returning an error message) might not be a pre-req for turning on the feature flag.

As for dbus-activation, adding something to the notes column of the snap services output sounds like a great idea.

I’ve put together the following PR to add the dbus-activated note to snap services output. It also does the bare minimum to stop that command from exploding in the presence of user scope daemons.

There’s more to address, but this is a start.

For enabling user daemons support by default we need at least to:

  • make sure no command as-is misbehaves in their presence
  • we need a design and implementation for how snapctl can at least for the most obvious use cases interact with them (4. in the list)
  • what we decide/implement about that will influence 1 which also needs to be fixed, because depending on what we allow or not, the enabled state can get complicated
  • some initial visibility via snap services would be good but is not completely a blocker, it should be noted that snapctl services also exists and needs to be designed for as part of the snapctl question
  • snap commands are not entirely blockers but would be driven also by what we decide for snapctl

Hello @jamesh @ijohnson,

I follow up this approach from my post (The dbus approach for single client/server snap), and here is my snapcraft.yaml to use user daemons and DBus activtation:

slots:
  dbus-server-slot:
    interface: dbus
    bus: session
    name: in.softprayog.add_server

plugs:
  dbus-client-plug:
    interface: dbus
    bus: session                                                                                                                                                                                           
    name: in.softprayog.add_server

apps:
  server:
    command: bin/server
    daemon: simple
    passthrough:
      daemon-scope: user
      activates-on:
        - dbus-server-slot
  client:
    command: bin/client
    plugs:
      - dbus-client-plug

However, server app was not in the snap services, and syslog had the following error:

snapd[136290]: snap.go:182: cannot get full app info for snap "dbus-test-snap": cannot get status of services of app "server": cannot get unit status: empty field "Type" in ‘systemctl show’ output

And, above experimental options should be set on 20.04 host:

$ sudo snap get system experimental.user-daemons
true
$ sudo snap get system experimental.dbus-activation
true

Can you point me if something missing?

Thanks,
Woodrow

This one should be partially fixed by PR #9872, mentioned above: the command will no longer return an error, but won’t include status of the user daemon. There will need to be a follow-up to get it to display reasonable information about user daemons.

Despite that, you should be able to see the status of the user service with a command like systemctl --user status snap.$snap_name.$app_name.service. If you’re having trouble getting the D-Bus communication working, double check that the plug and slot are connected.

Thanks for fast reply!

Yes, I can see the service file:

$ systemctl --user status snap.dbus-test-snap.server.service 
● snap.dbus-test-snap.server.service - Service for snap application dbus-test-snap.server
     Loaded: loaded (/etc/xdg/systemd/user/snap.dbus-test-snap.server.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

It can be started normally:

$ systemctl --user status snap.dbus-test-snap.server.service 
● snap.dbus-test-snap.server.service - Service for snap application dbus-test-snap.server
     Loaded: loaded (/etc/xdg/systemd/user/snap.dbus-test-snap.server.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-02-03 15:44:06 CST; 1s ago
   Main PID: 398367 (server)
     CGroup: /user.slice/user-1000.slice/user@1000.service/snap.dbus-test-snap.server.service
             └─398367 /snap/dbus-test-snap/x2/bin/server

Feb 03 15:44:06 woodrow-Inspiron-7520 systemd[3149]: Started Service for snap application dbus-test-snap.server.

Thank you!

Yep. Services using the activates-on stanza are not started by default. Instead, you should see a /var/lib/snapd/dbus-1/services/in.softprayog.add_server.service file that the session bus will use to start the corresponding service when it is first used by a client.

If you are finding that you need to manually start the systemd service for things to work then it indicates something is broken: either within your snap, or with the way snapd is configuring dbus-daemon and systemd. I’d need to see any error messages your client generates to give any further advice.

I can see the content of /var/lib/snapd/dbus-1/services/in.softprayog.add_server.service:

[D-BUS Service]                                                                                                                                                                                            
Name=in.softprayog.add_server
Comment=Bus name for snap application dbus-test-snap.server
SystemdService=snap.dbus-test-snap.server.service
Exec=/usr/bin/snap run dbus-test-snap.server
AssumedAppArmorLabel=snap.dbus-test-snap.server
X-Snap=dbus-test-snap

Right now it’s always inactive state when snap is just installed, so not sure which part is the root cause why it can’t be auto start. I’m please to provide the log if any specific.

Here is full syslog:

What I meant was what errors does your dbus-test-snap.client program get when the server is inactive?

OK, let me try this:

When server is inactive:

$ systemctl --user status snap.dbus-test-snap.server.service 
● snap.dbus-test-snap.server.service - Service for snap application dbus-test-snap.server
     Loaded: loaded (/etc/xdg/systemd/user/snap.dbus-test-snap.server.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

Feb 03 15:50:28 woodrow-Inspiron-7520 systemd[3149]: snap.dbus-test-snap.server.service: Succeeded.
Feb 03 15:50:28 woodrow-Inspiron-7520 systemd[3149]: Stopped Service for snap application dbus-test-snap.server.
Feb 03 15:50:39 woodrow-Inspiron-7520 systemd[3149]: Started Service for snap application dbus-test-snap.server.
Feb 03 17:03:40 woodrow-Inspiron-7520 systemd[3149]: Stopping Service for snap application dbus-test-snap.server...
Feb 03 17:03:40 woodrow-Inspiron-7520 systemd[3149]: snap.dbus-test-snap.server.service: Succeeded.
Feb 03 17:03:40 woodrow-Inspiron-7520 systemd[3149]: Stopped Service for snap application dbus-test-snap.server.
Feb 03 17:25:44 woodrow-Inspiron-7520 systemd[3149]: Started Service for snap application dbus-test-snap.server.
Feb 03 21:28:49 woodrow-Inspiron-7520 systemd[3149]: Stopping Service for snap application dbus-test-snap.server...
Feb 03 21:28:49 woodrow-Inspiron-7520 systemd[3149]: snap.dbus-test-snap.server.service: Succeeded.
Feb 03 21:28:49 woodrow-Inspiron-7520 systemd[3149]: Stopped Service for snap application dbus-test-snap.server.

To trigger the client:

$ dbus-test-snap.client 
Please type two numbers: 2 2 
Method "add_numbers" with signature "s" on interface "in.softprayog.dbus_example" doesn't exist

Is it what you expect to see the error?

First of all, does your client and server program work correctly when run without snap confinement?

Next, does the method call match the restrictions imposed by snapd’s dbus interface? In particular, for a snapd interface for name foo.bar.baz, either (a) the D-Bus interface must start withfoo.bar.baz, or (b) the object path must start with /foo/bar/baz.

yes, client/server can work with --devmode. But I think you’re right way to guide me now.
Because I’ve tried to use the following for connection name in C code and yaml:

  const char *const INTERFACE_NAME = "org.foo.bar.baz";
  const char *const SERVER_BUS_NAME = "org.foo.bar";
  const char *const SERVER_OBJECT_PATH_NAME = "/org/foo/bar";                                                                                                                                              
  const char *const METHOD_NAME = "add_numbers";
slots:
  dbus-server-slot:
    interface: dbus
    bus: session
    #name: in.softprayog.add_server
    name: org.foo.bar
  dbus-client-slot:
    interface: dbus
    bus: session
    #name: in.softprayog.add_client
    name: org.foo.car

plugs:
  dbus-client-plug:
    interface: dbus
    bus: session
    #name: in.softprayog.add_server
    name: org.foo.bar

In that case, the result is changing:

$ systemctl --user status snap.dbus-test-snap.server.service 
● snap.dbus-test-snap.server.service - Service for snap application dbus-test-snap.server
     Loaded: loaded (/etc/xdg/systemd/user/snap.dbus-test-snap.server.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
...

$ dbus-test-snap.client 
Please type two numbers: 3 3
Method "add_numbers" with signature "s" on interface "org.foo.bar.baz" doesn't exist

Please type two numbers: 3 3
Sum is 6

$ systemctl --user status snap.dbus-test-snap.server.service 
● snap.dbus-test-snap.server.service - Service for snap application dbus-test-snap.server
     Loaded: loaded (/etc/xdg/systemd/user/snap.dbus-test-snap.server.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-02-03 22:41:52 CST; 5s ago
   Main PID: 419110 (server)
     CGroup: /user.slice/user-1000.slice/user@1000.service/snap.dbus-test-snap.server.service
             └─419110 /snap/dbus-test-snap/x5/bin/server

Feb 03 22:41:52 woodrow-Inspiron-7520 systemd[3149]: Started Service for snap application dbus-test-snap.server.
Feb 03 22:41:52 woodrow-Inspiron-7520 dbus-test-snap.server[419110]: Did not get message

So client will get back to work at second time when server is from inactive to active? Interesting!

I think we’ve cleared all the blockers for the dbus-activation feature flag now. I had a chat with @alexmurray this morning, and he seemed satisfied that the feature from a security stand point (doesn’t increase the privilege of snaps, doesn’t allow them to run code at a time they couldn’t without it). So we could probably look at enabling it soon.

There is still some more work to be done on user-daemons. There is still some work to make sure the feature doesn’t cause commands like snap start to malfunction in the presence of user daemons, but there’s also the question of how those commands should act when run on user daemons if we don’t just return an error.