Cross snap service ordering

Since 2.31, we support service ordering within a single snap using after and before keywords, as described here The snap format. However, we still lack support for cross-snap service ordering.

In short, the idea is that a snap foo, defining a service app (foo.app), can request the service to be ordered after or before service other from snap bar (bar.other).

The systemd After/Before= stanzas generated based on the declaration define a loose coupling between the services involved. Meaning, the service mentioned in the right side of the declaration does not need to be defined or started.

I propose to extend the current snap.yaml syntax with support for external services like so:

name: my-snap
apps:
 foo:
  command: bin/start-foo
  daemon: simple
  # declare to be started before service 'baz' from snap 'bar'
  before: [snap.bar.baz]
 bar:
  command: bin/start-bar
  daemon: simple
  # declare to be started after service 'daz' from snap 'zed'
  after: [snap.zed.daz]
  # referring to own apps is illegal
  before: [snap.my-snap.app]

Upsides:

  • the syntax feels familiar, external dependencies are clearly visible
  • matches our way of naming the units
  • we can validate dependency cycles within a snap to a limited extent (will to add ‘mocked’ external dependencies just for validation)
  • syntax can be extended to support system services (eg. with system.<srv> prefix)

Caveats:

  • depending on a service from other snap, while the service got renamed
  • ordering after/before services from other snap does not make that snap a dependncy
  • during installation, only the services from the current snap will be started in the right order
1 Like

That feels a little awkward. Can you explain a use-case where one specifies before or after but doesn’t require it?

This was confusing to me for a while too, but the best example I have found is one where you have say a web server with two possible database backends, say DB1 and DB2. The user installs the snap and disables whichever database backend they aren’t using, for example DB2 (because they want to use DB1 and the web server will only use one of the backends). Then you have the following situation:

  • web-server service -> enabled, needs to start after the currently configured database
  • DB1 service -> enabled
  • DB2 service -> disabled

Without dynamically modifying the service dependencies the only way to ensure you have a startup ordering of

  1. the currently enabled database backend
  2. web-server

is to make the web-server have “soft” dependencies on both of the database backends, where the dependency only affects the order in which services are started when they are both enabled. You can’t make the dependency “hard” and make the web-server fail to startup if the database backend isn’t available because you don’t know which backend the user will choose, and as such would need to have them both always enabled, even though you’re only using on of them.

I personally think that systemd service files are overly complicated and snapd relying on systemd imports too much confusing concepts and terminology from systemd’s (i.e. the difference between After and Requires, and a much better solution would be to instead make the dependencies dynamic by regenerating the systemd unit files with some command the user could run (or also could be run automatically). Continuing the web-server example, this would then enable a user to choose which backend they want and only run that one, but the snap author could also make the dependency “hard” so that if the chosen database backend fails or isn’t available at all snapd doesn’t attempt to start the web-server at all.

The other problem I see with the proposal that would be solved with dynamically handling service dependencies is the ability to declare a dependency on services the snap author doesn’t know about, i.e. what if the web-server snap doesn’t know about some better version of DB2 called DB3 (and DB3 is in a new snap that is different from DB2), and the user goes and installs this new DB3 snap that the upstream web-server can work with, but the snap packaging hasn’t been updated to handle yet, the user could decide to order the web-server snap services after the DB3 service.

After/before conveys ordering, not dependency.

Given services A and B, when B has after: [A] this literally means when both A and B are started together, start B after A has started. In systemd’s notion, when both are getting activated, means B will be started once A becomes active. However, the after/before does not mean A requires B to function (in oher words, when B goes down, A must go down too), this is what systemd Requires does, but we do not support it yet.

These are some ideas that were sketched discussing with @niemeyer and @mvo recently.

First some context:

  • So far snaps almost never refer to other snaps directly, there is always some mediation through an interface etc, here the issue is not only consistency/aesthetic, or avoiding to expose systemd details. The transparency is useful because on different systems (classic vs not, different core systems) the same services could be hosted and split among different snaps.
  • Usually two snaps never interact unless there is an interface connection between the two.

Service order interface

The first sketched idea is to create synthetic slots on each snap for each service they contain for a special interface (name to be defined). Then one could set up service order relationships by plugging manually created service plugs on a snap to service slot on another, plus likely an attribute to control the desired order after: true|before: true.

  • we would need to be careful with naming these plugs and slots also because of potential clashes
  • we would need to extend our tools to set up auto-connections via declarations to help connecting these as needed (as is, all pairs of these would be candidates creating ambiguity), we need to make the connection as early as possible to know the order as early as possible
  • related to the latter we probably want the services in the snap to have a way to specify some kind tag (in the spirit of content interface content or dbus names) that would go as an attribute on the plug/slot

Implicit ordering

A simpler/blunter approach is to make it so that if two snaps have services and an interface connection between them, all the services in the slot snap should be before the services in the plug snap.

  • we need to be careful whether we have the necessary information at the right moments in for example a multi-snap install setting up auto-connections to write the correct service units, start things in the right order
  • there could be circular situations which we would need to somehow handle

Open questions

  • Do we need a way to setup order relationships with services that come from the system as well? how we control/name which in that case?

  • Is the use case really about after/before, or also require relationships?