How to set the boot order for snaps after reboot?

pangda · October 23, 2019, 9:27am

hi all, i want to know how to set the boot order for snaps after reboot? such as i have install three snap(init, ros, pangda-hello) in ubuntu core system. how to configure the start sequence？（example: init snap start first then start ros snap, finally start pangda-hello snap）

thanks!

jamesh · October 24, 2019, 6:17am

There is no way for a snap developer to specify start up order of daemons packaged in different snaps.

If the three daemons are always intended to be used together, then perhaps packaging them as a single snap is the best option. You can then use the before: and after: keys on the daemons to impose ordering.

If you want looser coupling, another option is to keep the daemons in separate packages and use socket activation. This basically means that the systemd will create the listening sockets for your network services before trying to start any daemons, and pass those sockets in when it spawns them. This allows daemons to start in parallel but block when they try to talk to another service that isn’t yet ready. This removes the need for many explicit ordering constraints.

pangda · October 24, 2019, 6:42am

Thanks reply.

Could you give me a example for how to use before: and after: keys ?

And if i want use socket activation, What needs to be done for systemd and my app snaps to ordering?

Could you tell me some detail for using activation?

jamesh · October 24, 2019, 6:57am

It seems to be missing from the docs, but it should look something like this:

github.com

snapcore/snapd/blob/master/tests/lib/snaps/test-snapd-after-before-service/meta/snap.yaml

name: test-snapd-after-before-service
version: 1.0
apps:
  before-middle:
    command: bin/start
    daemon: notify
    before:
      - middle
    plugs: [daemon-notify]
  middle:
    command: bin/start
    daemon: notify
    plugs: [daemon-notify]
  after-middle:
    command: bin/start
    daemon: notify
    after:
      - middle
    plugs: [daemon-notify]

(it isn’t restricted to notify-type daemons: that’s just what this particular test snap is using)

On the snapcraft.yaml side, you add a sockets section to your app and specify the socket name via the listen-stream key as described in the docs:

On the application side, rather than creating the listening socket via connect/bind/listen system calls, use sd_listen_fds() (or whatever equivalent API is available in the language the daemon is written in) to adopt the socket systemd has opened for you.

Here’s a trivial socket activated web app I wrote for a talk I gave earlier in the year as a full example:

It’s written in Go, but it should give you an idea of how everything hooks together.

ogra · October 24, 2019, 7:56am

note that in ubuntu core, while you can not influence the start order of different snaps after first boot, you can define the order in which the snaps get initialized during first boot … snapd processes the snap installation in the order the snaps are listed in the required-snaps field in your model assertion … so if you only require them to be in order in a first boot scenario, this is actually possible …

later on the snaps start in whatever order systemd finds is appropriate though.

pangda · October 24, 2019, 10:38am

thanks @jamesh
as you say for using the socket activation, i don’t see the two app snaps startup ordering, i only see between the webapp with systemd. not the webapp with another app snap.how to control the webapp snap is startup before another app snap.

jamesh · October 24, 2019, 11:08am

I think you might have some misconceptions about how systemd starts daemons: absent any other ordering constraints, systemd will try to start multiple services simultaneously. It won’t necessarily wait for one service to finish starting before starting a second service.

Let’s take a concrete example of a database daemon, and a web daemon that stores its configuration in a database. In a traditional init system, you’d want to impose an ordering constraint so the database daemon is always started before the web daemon. If we instead use systemd and the database server supports socket activation, then it doesn’t matter what order the two daemons are started. There two possibilities:

The database daemon starts first, and web daemon starts second. The web can talk to the database as normal.
The web daemon starts first and blocks when it tries to connect to the database. Systemd notices the attempt to connect to the socket and starts the database daemon. The web daemon continues running once the database daemon has finished initialising and accepts the connection.

As you can see from case (2), systemd could even try to start both daemons simultaneously. It’s the act of communication that synchronises the start up of the two daemons rather than metadata in the .service files.

jocado · November 11, 2020, 9:08am

@ogra Sorry to dig up an old post

Is this initialisation order something that is documented, or at least guaranteed to remain ?

I believe I have hit a problem around this, specifically with the content interface. if snap A has a content interface to snap B, then snap B must be installed first.

So I guess system set-up ordering is a good way to solve that. Or is there any other way you know if ?

Cheers, Just

ogra · November 11, 2020, 9:34am

if your app comes as a service, you can craft something via the connect and install hooks that keeps the service stopped until the interface is connected …

start with an install hook that by default disables the service:

github.com

ogra1/pi-fancontrol-snap/blob/master/snap/hooks/install

#! /bin/sh

# make sure all required defaults are set
[ -n "$(snapctl get gpio)" ] || snapctl set gpio=14
[ -n "$(snapctl get threshold)" ] || snapctl set threshold=50
[ -n "$(snapctl get interval)" ] || snapctl set interval=120

# we install with the service disabled, the connect hooks for
# hardware-observe and gpio will start it once both interfaces
# are connected
snapctl stop --disable ${SNAP_NAME}.${SNAP_NAME} 2>&1 || true

then create a “connect-plug-your-content-interface” hook:

github.com

ogra1/pi-fancontrol-snap/blob/master/snap/hooks/connect-plug-hardware-observe

#! /bin/sh

# we need both interfaces connected, so drop out if the other one is not
# attached to the slot yet
if ! snapctl is-connected gpio; then
  echo "please run 'snap connect ${SNAP_NAME}:gpio <gadget>:bcm-gpio-14"
  echo
  echo "keeping service disabled for the moment"
  exit 0
fi

# now we can start the service
if snapctl services ${SNAP_NAME}.${SNAP_NAME} | grep -q inactive; then
  snapctl start --enable ${SNAP_NAME}.${SNAP_NAME} 2>&1 || true
fi

(if you have multiple interfaces you can also cross check with “snapctl is-connected …” if the other interfaces are connected and have one hook per interface. in that design the last connected hook then starts the service)

jocado · November 11, 2020, 10:34am

Thanks @ogra that is extremely useful.

For some reason I haven’t fully consumed the fact that interface connection hooks existed, but now I remember reading that previously.

I will incorporate that to protect dependencies, but also list the snaps in the model assertion in the required order too.

Cheers,
Just

snowsky · March 3, 2022, 11:42pm

A quick question, how can I set up to make a snapd service run after all cloud-init scripts finish running? Thanks

ijohnson · March 4, 2022, 12:28am

There is not currently any predefined way, you could potentially use log-observe to watch the system journal and see when a given cloud-init target is reached and then run the actual code you want in the service but that could be fragile

snowsky · March 4, 2022, 12:45am

also thinking if this can work, use a snap set command in cloud-init and then start the service in configure hook?

ijohnson · March 4, 2022, 5:36pm

You would need to make sure that the snap you are calling snap set on is fully seeded before doing that, but you could probably do snap wait system seed.loaded before calling snap set to ensure that