Expose a more consistent subset of systemd's service directives

[chipaca]

In reading Expose systemd’s RestartSec?, we looked at the systemd service directives subset that we expose, and realise that perhaps the policy so far of adding things on request has resulted in a bit of a mess.

So, let’s fix that. How about, when something is a daemon, you can specify [and the systemd directive to which it roughly maps]:

• command: this is the daemon itself, as now [ExecStart]
• pre-command: a command to run before the daemon is started [ExecStartPre]
• post-command: a command to run after the daemon is started [ExecStartPost]
• reload-command: a command to run to reload the daemon’s configuration (if not specified, the daemon can only restart, not reload) [ExecReload]
• stop-command: a command to run to stop the daemon (if not specified, the daemon is killed) [ExecStop]
• post-stop-command: a command to run after stopping the daemon [ExecStopPost]
• restart-delay: how long to wait after a daemon dies before trying to start it (default 100ms) [RestartSec]
• start-timeout: how long after starting the daemon has to notify startup success [TimeoutStartSec]
• stop-timeout: how long stop-command has before the daemon is just killed outright [TimeoutStopSec]
• restart-condition: in what situations should the daemon terminating cause it to be restarted [Restart]
• before, after: order this daemon before/after the other daemons (if the other daemon is run at all). Can only refer to daemons in the same snap; this is a list of names. [Before, After]

We would continue to not support multiple commands in each of the command options (so you could only have at most one pre-command per app).

Anything else you’d like? Speak now, or … or speak later, but now would be nicer.

[zyga]

Message from Capt. Obvious, the various entries that can happen more than once can be, you know, a list? This would look like this:

apps:
  foo:
    command:
      - one
      - two
      - three

[chipaca]

Yes, they’re considered a list. Do we really want to / need to / are there benefits to exposing that?
There’s also the whole “prefix it with a - for it to be able to fail” which seems also silly (given how trivial writing a wrapper to do that for you is)

[kyrofa]

Would service ordering be included here?

[chipaca]

Would service ordering be included here?

Might as well! I don’t really see why not, it’s touching all the same code.

[kyrofa]

Another use-case that falls under the systemd domain is grouping services. For example, take a robot that can operate in multiple modes, perhaps “normal” operation as well as a “mapping” mode. The way such grouping is solved outside of snaps is with systemd targets and e.g. PartOf. Is this something we can support? I suppose another aspect of this is having the ability to not fire up all services automatically.

[jdstrand]

Thus far, this looks fine. I was a bit confused by the introduction to the topic. You aren’t actually proposing to change existing items, instead this is a call for comments on new things to add all at the same time to our existing list?

[chipaca]

You aren’t actually proposing to change existing items, instead this is a call for comments on new things to add all at the same time to our existing list?

pre-command, post-command, start-timeout, restart-delay (and now before and after) are all new

[cratliff]

Another use-case that falls under the systemd domain is grouping services

+1 to grouping services and being able to turn them on and off. For battery operated devices being able to have the ability to reduce the power load on the system is great. Having groups to only keep core services running would make this much easier.

[chipaca]

Would that be something like, say

apps:
  foo:
    command: powerhungryd
    daemon: simple
    group: powered
    procrastinate: true
  bar:
    command: moarpowerhungryd
    daemon: simple
    group: powered
    procrastinate: false
  baz:
    command: statusd
    daemon: simple

and then somehow arange for

snap services start mysnap.powered

to run when on power?

I can see the value of procrastinate (ok so I couldn’t find a suitable antonym for start, i’m really bad at names, but this needs to be something that means “don’t start”, as the default needs to be false). The grouping thing seems less useful, given snap services start can already take a list, but I do see the convenience. Not sure it’s worth the complexity it introduces (but maybe I’m overthinking it)

[chipaca]

or maybe I misunderstood, and by grouping you mean to have predefined targets, and the ability to flag services to depend on those targets? Like, say, we create a ac.target that’s only reached if the system is on AC¹, and then

apps:
  foo:
    command: powerhungryd
    daemon: simple
    group: ac

means that powerhungryd would only run when on AC?

---

1. much handwaving here as I’m not sure that would properly handle going off the group, but it’'s doable behind the scenes one way or another

[morphis]

Another interesting option would be WatchdogSec= in order to utilize the watchdog feature from systemd to automatically restart a service when it doesn’t respond to systemd anymore. See here for details.

[cratliff]

Yeah, targets would be the ideal implementation of solving grouping and being able to launch or kill a group of services depending on system state. I don’t know how you would implement it, but keeping the keywords to be able to start and stop targets like individual services can be in their related grouping would be great, eg. Before=, After=, WantedBy=, Wants=, RequiredBy=, Requires=, PartOf=, BindsTo=:

[kyrofa]

This ties into service control with snapctl as well, I suppose.

[jamesh]

It’d also be nice to support user vs. system services. That might be orthogonal to adding these other options though.

[chipaca]

Yes @morphis, systemd watchdog support is on my list but a little bit later (as we need to look into whether we’ll need to tweak apparmor because of sd_notify for the watchdog).

[chipaca]

@cratliff I’m trying to understand if grouping would necessarily involve grouping with things outside of the snap, or if it’s just (or primarily) within the same snap. Could you clarify that?

Or maybe we can meet and talk about that; are you still around?

[cratliff]

I am primarily thinking of things internal to the snap. There may need to be something external, such as multi-user.target if the snap doesn’t run anything similar internal, but that can probably be handled internally somehow.

[jdstrand]

as we need to look into whether we’ll need to tweak apparmor because of sd_notify for the watchdog

FYI, this is on my list and not forgotten, but behind a few things.

[hcochran]

Hello. We are currently trying to adapt our ROS-based robot to use Snappy Core. Since ROS is highly componentized, we publish around 20 different ROS services from our snap, which have ordering and dependency requirements. We also use .target files to group services.

New Directives that we need include:

Before=, After= For ordering, as mentioned in the OP.

WantedBy=, Wants=, RequiredBy=, Requires=, PartOf=, BindsTo=
These specify dependencies between service and, especially between targets and services. PartOf= enables the services that are started by a target to also be stopped if that target is stopped. BindsTo= allows us to start & stop the ROS service that interfaces with a hardware device whenever the hardware appears or disappears.

ConditionPathExists=
We use this to enter to start a different set of services if we have not yet been configured. For example, don’t start our main application node yet but do start a service to download configuration information from the cloud. Although this is the only Condition… directive that we currently use, I can imagine that most of the others are equally useful.

Environment=, EnvironmentFile=
Some behavior in ROS launch files can only be configured using environment variables. Note that the current ability to specify an environment variable in snapcraft.yaml only accepts hard-coded values as far as I know. We need to load them from a config file.

Conflicts=
This is helpful for systemd target-based mode switching, in cases where modes are mutually exclusive. For example, higher-energy ROS services would Conflicts= with our low-power.target, and our “normal.target” will Conflict= with our low-power.target

Alias=
We use this for debugging, by specifying all dependencies via an aliased service name. Thus we can disable the normal service and enable a different one with the same alias to make a substitute. The substitute may compiled with debug flags and no optimization, or run under gdb, strace, etc.

As you can imagine, lack of support for these made porting to Snap a very big challenge. So far, we have to run a configure hook that rewrites the .service files, which is very messy and totally breaks confinement. Supporting these directives in snapcraft + snapd will be a HUGE plus!

Thanks