Finding reason for Stopped snap service at bootup

snapd
1

Device - x86_64 (AMD64)
Image - Ubuntu Image built with ubuntu-image snap <args> (with snap built locally in build VM)
Issue - Device fails to boot after fresh first boot when service of a snap stops immediately after starting and subsequent dependent snap fails to start.

Log snippet (look for os-platform whose service stopped after starting. Log marked with ------->ISSUE):

....
....
    Mount snap "os-1000kernel" (unset)                                          |         Mounting Mount unit for os-1000kernel, revision x1...
    Mount snap "os-1000kernel" (unset)                                          /[  OK  ] Mounted Mount unit for os-1000kernel, revision x1.
    Mount snap "core18" (1880)                                                     -         Mounting Mount unit for core18, revision 1880...
[  OK  ] Mounted Mount unit for core18, revision 1880.
    Mount snap "os-1000gadget" (unset)                                          |         Mounting Mount unit for os-1000gadget, revision x1...
    Mount snap "os-1000gadget" (unset)                                          /[  OK  ] Mounted Mount unit for os-1000gadget, revision x1.
    Mount snap "os-hw-wtdog" (unset)                                            -         Mounting Mount unit for os-hw-wtdog, revision x1...
    Mount snap "os-hw-wtdog" (unset)                                            \[  OK  ] Mounted Mount unit for os-hw-wtdog, revision x1.
    Mount snap "os-redis" (unset)                                               /[  OK  ] Mounted Mount unit for os-redis, revision x1.
    Setup snap "os-redis" (x1) security profiles for auto-connections           |[  OK  ] Started Service for snap application os-redis.redis-server.
    Mount snap "os-mosquitto" (unset)                                           -         Mounting Mount unit for os-mosquitto, revision x1...
    Mount snap "os-mosquitto" (unset)                                           \[  OK  ] Mounted Mount unit for os-mosquitto, revision x1.
    Setup snap "os-mosquitto" aliases                                           -[  OK  ] Started Service for snap application os-mosquitto.mosquitto.
    Mount snap "os-core" (unset)                                                -         Mounting Mount unit for os-core, revision x1...
    [  OK  ] Mounted Mount unit for os-core, revision x1.
    Start snap "os-core" (unset) services                                       -[  OK  ] Started Service for snap application os-core.launch.
    Mount snap "os-platform" (unset)                                            \         Mounting Mount unit for os-platform, revision x1...
    [  OK  ] Mounted Mount unit for os-platform, revision x1.
    Set automatic aliases for snap "os-platform"                                |[  OK  ] Started Service for snap application os-platform.launch.
    Mount snap "os-dp" (unset)                                                  /[  OK  ] Stopped Service for snap application os-platform.launch.-------------> ISSUE
    [  OK  ] Started Service for snap application os-platform.launch.
    Mount snap "os-dp" (unset)                                                  |         Mounting Mount unit for os-dp, revision x1...
    Mount snap "os-dp" (unset)                                                  /[  OK  ] Mounted Mount unit for os-dp, revision x1.
    [  OK  ] Stopped Service for snap application os-platform.launch.
    [  OK  ] Started Service for snap application os-platform.launch.
    Setup snap "os-dp" (unset) security profiles                                |[  OK  ] Stopped Service for snap application os-platform.launch.
    [  OK  ] Started Service for snap application os-platform.launch.
    Setup snap "os-dp" (unset) security profiles                                -[  OK  ] Stopped Service for snap application os-platform.launch.
....

os-dp depends on os-platform to provide some data. But since os-platform service did not start os-dp fails to come up and boot never completes.

Is there a way to check why os-platform service started and stopped immediately?

2

Are you sure that it isn’t the hooks requesting configuration changes or service restarts?

Since it’s first boot, you can inspect the output of $ snap change 1 (where 1 is the ID).

3

Yes, because when the image is built with snaps from store (private) there is no such issue. Also, suspecting an issue of interface not getting connected resulting in stopping of service, I edited my gadget.yaml and connected the interface manually. But got same result

Can’t do. As I mentioned in the original post, boot never completes as dependent snap os-dp goes into a loop of starting and stopping again. Hence I never get a console.

4

Can you try adding systemd.journald.forward_to_console=1. this should make all the logs, including snapd, go to console.

5

If you don’t mind can you tell me where to add this config / setting?

6

You are using grub? If the prompt has not been disabled in your gadget, then then go to edit mode when the default entry is highlighted (IIRC by pressing ‘e’), and append systemd.journald.forward_to_console=1 to the command line after $cmdline

1 Like
7

I am seeing the errors now :+1: Will work with that. Thanks @mborzecki

8

There are two reasons why the platform service is stopping on boot:

[   64.656570] os-platform.launch[849]: /snap/os-platform/x1/dev.sh: 6: /snap/os-platform/x1/dev.sh: ip: Permission denied
[   76.660380] os-platform.launch[849]: /snap/os-platform/x1/usr/bin/platform-info: error while loading shared libraries: libutil.so: cannot open shared object file: No such file or directory

Not sure why this is happening as:

  1. for first error, I have required plugs (network and network-bind) in my snapcraft.yaml

  2. for the second error, libutil.so is part of a content interface provided by os-core slot which is mounted before os-platform and has come up fine. Also the require plug is present in os-platform snapcraft.yaml

Any idea, why despite having the required connections in respective snapcraft.yaml’s and connecting them in gadget.yaml an image built from local source code of snaps fails like above, whereas for an image built with snaps from store it works fine.

9

Local snap is not signed. It doesn’t have snap declarations that the signed store snap does have.

2 Likes