Service ordering

snapd
1

(I thought I’d created this topic ages ago, but can’t find it; apologies for either the duplication or the tardiness).

We want to have an (explicit) way of ordering services within a snap. I think just supporting an “after” keyword in services would be enough. This would look like

apps:
 foo:
  command: bin/start-foo
  daemon: simple
 bar:
  command: bin/start-bar
  daemon: simple
  after: foo
 baz:
  command: bin/start-baz
  daemon: simple
  after: foo

Also perhaps a way (behind an interface?) of ordering services of a snap to services “outside” (note this is ordering and not dependencies, so for example you can have a snap that depends on a database service, and you ship with a sqlite backend but also support connecting to the system’s postgres if there is one, you could do it with this). Ideally we’d use after: <some magic> to not complicate the yaml more with this, but atm I don’t know what the magic would be (it would have to be clear and obvious to the reader and writer, and at the same time not lead to things like a snap shipping a postgres app and having that after get confused with the postgres service from “outside”. This needs more discussion obviously and would probably come in a second phase, but I thought it worth mentioning here.

NOTE in-snap service ordering is implemented from 2.31 on, and covered in the snap format.

3 Likes
2

I like the explicitness of this. I suspect most people won’t use this but those that do will appreciate the clarity.

3

Proposal looks nice indeed.

For the more complex case, we may be able to introduce ordering support somewhat implicitly via interfaces themselves, instead of expecting snaps to magically know about each other. In other words, if we have a slot on provider.app and a plug on consumer.app, it seems reasonable to order the startup of the daemon in the former before the daemon in the latter.

There are surely details to figure out, but this sounds like a reasonable direction to explore at least.

4

I like this as well. @niemeyer to clarify for phase 2, you’re saying all the daemons in the provider should be running before we start any daemons in the consumer? Not the most efficient, but definitely the simplest and pretty much sure to work.

5

Yeah, something along those lines. It’s not so simple because there may be loops, so not every interface will be treated the same, and we may need to introduce more explicit hints to solve non-trivial cases. But it may solve most of the easy cases transparently, and does feel like a direction worth exploring.

6

note that the original request and discussion was Snap service start ordering

1 Like
7

… and this might be continued in the Expose a more consistent subset of systemd’s service directives topic.

8

So how would the snap.yaml look if we wanted to be able to express a hard dependency for services, i.e. using both After= and Requires= in the systemd unit?

9

As per the linked topic, and AIUI, that would be

apps:
  foo:
    starts-with: [bar]
    before: [bar]
    ... etc
  bar:
    ...

(but it’s not implemented yet)

10

Hi guys!

First of all, thank you for your great work!

We’re making snap consisting of several daemons, and some of that daemons depends from anothers. I’ve tried to use “after:” sections in snap.yaml - and I see that sections appeared in generated systemd .service files, but I still see some kind of randomness in daemons starting order in logs.
Plus, we have one daemon that works with hardware - and it may start only after some attempts - so we should start daemons that depends on it only after it started successfully - but, with just “after:” I don’t observe this…
I’ve experimented a little, and find that we can solve this if we will use not just “after”, but “after” + “wants” - in this case, startup order is correct, and, more important, if daemon fails and systemd restarts it, dependent daemons will be restarted also.
So, to make it works on my development unit, I’ve just altered services.go to make list of what we have in “after” goes in “wants” section also.
I’ve seen discussion about “more consistent subset of systemd’s service directives” - and “Wants” was mentioned there about year ago - but it is not realized yet…

So, because we really wants this directive to be available in snapd - maybe, we can make PR for that?
Is there some reason why “wants” isn’t realized yet - or it is just because developers are busy with more priority things?

Thanks a lot!

11

This is from Bug #1796125 “services in a snap with dependencies aren't starte...” : Bugs : snapd and should be fixed in edge and beta channels of the core snap. The way to fix it is to start all of the services one at a time in a topological sort order.

I think this is partly because of developer time not being readily available and because there’s a discussion currently going on regarding how to order snap services in a more robust and generic way. See Cross snap service ordering

12

Thanks for reply!
I’ve tried latest (master) snapd - and now service start-up order looks good, when I executing snapd start .
But, if I do snapd restart - there is random order of stopping-starting in logs, and restart function fails. But if I do "snapd stop && snapd start " - all works great.
Is there any solution to fix ordering in “restart” command?

Thanks again!

13

Yes, that appears to be a bug. I actually independently ran into the same problem yesterday and filed a bug: https://bugs.launchpad.net/snapd/+bug/1803212

EDIT: actually this bug is different, your problem is that snap restart doesn’t obey the service ordering, while the bug I filed is about snap restart starting disabled services.