Raspberry Pi 4 - Watchdog not working - bug or configuration issue?

dang · May 5, 2021, 8:36pm

I’m using core 20 on Raspberry Pi 4. Using the latest published snaps.

As soon as I set the watchdog runtime-timeout, systems starts rebooting every minute - regardless of the timeout. It appears the watchdog is being enabled, but never serviced.

Configuration before - as expected, runtime timeout not set, no 10-snapd-watchdog.conf file:

$ snap list
 Name       Version        Rev    Tracking       Publisher   Notes
 core20     20210319       974    latest/stable  canonical?  base
 pi         20-1           89     20/stable      canonical?  gadget
 pi-kernel  5.4.0-1025.28  231    20/stable      canonical?  kernel
 snapd      2.49.2         11584  latest/stable  canonical?  snapd
$ cat /etc/systemd/system.conf.d/10-snapd-watchdog.conf
cat: /etc/systemd/system.conf.d/10-snapd-watchdog.conf: No such file or directory
$ systemctl show | grep -i watchdog
RuntimeWatchdogUSec=0
RebootWatchdogUSec=10min
KExecWatchdogUSec=0
ServiceWatchdogs=yes

Next, I set runtime-timeout to 15 min. Resulting configuration looks good

$ snap set core watchdog.runtime-timeout=15m
$ cat /etc/systemd/system.conf.d/10-snapd-watchdog.conf
[Manager]
RuntimeWatchdogSec=900
$ systemctl show | grep -i watchdog
RuntimeWatchdogUSec=15min
RebootWatchdogUSec=10min
KExecWatchdogUSec=0
ServiceWatchdogs=yes

Approximately 60 seconds later, board will reset. It will boot successfully, run for about 60 secs, then reboot again. I can break it out of loop by clearing runtime-timeout.

Thoughts?
Dan

mborzecki · May 6, 2021, 6:59am

I would suggest to enable persistent journal (snap set system journal.persistent=true), then enable watchdog, wait for a reboot, disable it and inspect the journal.

dang · May 7, 2021, 2:46pm

Thanks for the suggestion. In the logs I see the following when I set runtime-timeout=10m

Hardware watchdog ' Broadcom BCM2835 Watchdog timer', version 0
Failed to set timeout to 600s: Invalid argument

The failure matches this line in systemd version 245

github.com

systemd/systemd/blob/v245/src/shared/watchdog.c#L42


        r = ioctl(watchdog_fd, WDIOC_SETOPTIONS, &flags);
        if (r < 0)
                return log_warning_errno(errno, "Failed to disable hardware watchdog: %m");
} else {
        int sec, flags;
        char buf[FORMAT_TIMESPAN_MAX];
        sec = (int) DIV_ROUND_UP(watchdog_timeout, USEC_PER_SEC);
        r = ioctl(watchdog_fd, WDIOC_SETTIMEOUT, &sec);
        if (r < 0)
                return log_warning_errno(errno, "Failed to set timeout to %is: %m", sec);
        watchdog_timeout = (usec_t) sec * USEC_PER_SEC;
        log_info("Set hardware watchdog to %s.", format_timespan(buf, sizeof(buf), watchdog_timeout, 0));
        flags = WDIOS_ENABLECARD;
        r = ioctl(watchdog_fd, WDIOC_SETOPTIONS, &flags);
        if (r < 0) {
                /* ENOTTY means the watchdog is always enabled so we're fine */
                log_full(errno == ENOTTY ? LOG_DEBUG : LOG_WARNING,
                         "Failed to enable hardware watchdog: %m");

Not sure why ioctl is failing, or watchdog remains enabled after failure to set timeout, but something to go on.

dang · May 7, 2021, 3:31pm

Followup - appears BCM2835 only support timeout of up to 15 secs. Setting timeout to 15 secs worked

https://raspberrypi.stackexchange.com/questions/71052/changing-watchdog-timeout-on-rpi3