Diagnosing failure to automatically reboot after pi2-kernel update

Last night the pi2-kernel snap on a Raspberri Pi 3 with Ubuntu Core was auto-refreshed from the beta channel, which it’s set to track. I had an SSH session open and could see the sequence of The system is going down for reboot messages, followed by the expected eventual discussion.

However when I checked this morning, the device had not actually booted up after the refresh. Specifically, the heartbeat light was off. I power-cycled it and it came back up seemingly normally, but I’d really love to understand what went wrong. We’re going to be deploying a fleet of these in remote locations where there won’t be anybody around to conduct a manual power-cycle - so behavior like this would be very problematic.

Unfortunately I didn’t think to connect a screen when I encountered the dead device. Is there any way to diagnose what happened, now that I’ve manually rebooted it? So far all the logs I’ve looked at indicate a successful shut-down last night, and a normal boot-up this morning (when I power-cycled). Specifically:

  • systemd logs (for which I have persistent storage) indicate a successful scheduled shutdown at 23:00 UTC last night. Then the (manual) boot-up at 06:25 this morning.
  • In dmesg I can see that the new kernel booted up fine after my manual reboot.
  • Snap changes look normal:
user@localhost:~$ snap changes
ID   Status  Spawn                 Ready                 Summary
[...]
141  Done    2018-05-08T22:48:51Z  2018-05-08T22:50:30Z  Auto-refresh snap "pi2-kernel"
142  Done    2018-05-09T06:25:06Z  2018-05-09T06:25:09Z  Connect stromm:hardware-observe to core:hardware-observe
[Other normal boot-up activities]
  • The actual kernel upgrade looks normal too:
user@localhost:~$ snap tasks 141
Status  Spawn                 Ready                 Summary
Done    2018-05-08T22:48:51Z  2018-05-08T22:48:51Z  Ensure prerequisites for "pi2-kernel" are available
Done    2018-05-08T22:48:51Z  2018-05-08T22:49:55Z  Download snap "pi2-kernel" (53) from channel "beta"
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:24Z  Fetch and check assertions for snap "pi2-kernel" (53)
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:28Z  Mount snap "pi2-kernel" (53)
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:28Z  Run pre-refresh hook of "pi2-kernel" snap if present
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:28Z  Stop snap "pi2-kernel" services
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:28Z  Remove aliases for snap "pi2-kernel"
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:28Z  Make current revision for snap "pi2-kernel" unavailable
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:29Z  Copy snap "pi2-kernel" data
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:29Z  Setup snap "pi2-kernel" (53) security profiles
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:29Z  Make snap "pi2-kernel" (53) available to the system
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:30Z  Automatically connect eligible plugs and slots of snap "pi2-kernel"
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:30Z  Set automatic aliases for snap "pi2-kernel"
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:30Z  Setup snap "pi2-kernel" aliases
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:30Z  Run post-refresh hook of "pi2-kernel" snap if present
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:30Z  Start snap "pi2-kernel" (53) services
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:30Z  Clean up "pi2-kernel" (53) install
Done    2018-05-08T22:48:51Z  2018-05-08T22:50:30Z  Run configure hook of "pi2-kernel" snap if present

......................................................................
Make snap "pi2-kernel" (53) available to the system

2018-05-08T22:50:29Z INFO Requested system restart.
  • Kernel snap info:
use@localhost:~$ snap info pi2-kernel
name:      pi2-kernel
summary:   The canonical raspi2 armhf kernel
publisher: canonical
contact:   snappy-canonical-storeaccount@canonical.com
license:   unknown
description: |
  The Ubuntu linux-raspi2 kernel package as a snap
type:      kernel
snap-id:   m0rvvnmRdDgexonz1XSP9a9U4K7vUbiy
tracking:  beta
refreshed: 2018-05-08T22:32:59Z
installed:   4.4.0-1089.97 (53) 124MB kernel
channels:                       
  stable:    4.4.0-1089.97 (53) 124MB -
  candidate: 4.4.0-1089.97 (53) 124MB -
  beta:      4.4.0-1089.97 (53) 124MB -
  edge:      4.4.0-1089.97 (53) 124MB -

Any way to try to get to the bottom of this, or should I just accept that it’s gremlins?