Unexpected system time changes on device with no RTC

snapd
1

We have an Ubuntu Core device running on a Raspberry Pi CM4 that does not have an RTC. I noticed some uexpected behaviour when going through a few reboots without a network cable plugged in to trigger network time synchronization. Here is what I see:

  1. After the first boot where snaps are seeded and set up, I get a system time of October 13, 2021 (this corresponds to the latest timestamp of model, snap-version, and snap-declaration assertions of seeded snaps on the device as per some PR I saw on this from back in March-ish timeframe).
  2. Reboot the system. The system time is now showing September 24, 2021 which is before the original time.

This seems odd that it would go back. Is this expected or did I find a bug? If so, I can create a bug for it in Launchpad.

2

@mvo Is this something you can look at/advise on? This one of our projects. Thanks.

3

Hey kyleN,

On the firstboot we read the model and set the time according to that. But after the firstboot AFAICT we don’t do that anymore and rely on /var/lib/sytemd/timesync/clock which is maintained by systemd-timesyncd(1). It looks like we don’t sync this file over during the install which might explain this behavior.

I suspect something like:

diff --git a/overlord/devicestate/handlers_install.go b/overlord/devicestate/handlers_install.go
index 9b6f2e8766..8d6f50acf3 100644
--- a/overlord/devicestate/handlers_install.go
+++ b/overlord/devicestate/handlers_install.go
@@ -25,11 +25,13 @@ import (
        "os"
        "os/exec"
        "path/filepath"
+       "time"
 
        "gopkg.in/tomb.v2"
 
        "github.com/snapcore/snapd/asserts"
        "github.com/snapcore/snapd/boot"
+       "github.com/snapcore/snapd/dirs"
        "github.com/snapcore/snapd/gadget"
        "github.com/snapcore/snapd/gadget/install"
        "github.com/snapcore/snapd/logger"
@@ -318,6 +320,19 @@ func (m *DeviceManager) doSetupRunSystem(t *state.Task, _ *tomb.Tomb) error {
                return fmt.Errorf("cannot store the model: %v", err)
        }
 
+       // keep track of the time
+       clockSrc := filepath.Join(dirs.GlobalRootDir, "/var/lib/systemd/timesync/")
+       clockDst := filepath.Join(boot.InstallHostWritableDir, "/var/lib/systemd/timesync/clock")
+       if err = os.MkdirAll(filepath.Dir(clockDst), 0755); err != nil {
+               return fmt.Errorf("cannot store the clock: %v", err)
+       }
+       if err = osutil.CopyFile(clockSrc, clockDst, osutil.CopyFlagPreserveAll); err != nil {
+               return fmt.Errorf("cannot store the clock: %v", err)
+       }
+       if err := os.Chtimes(clockDst, time.Now(), time.Now()); err != nil {
+               return fmt.Errorf("cannot set clock: %v", err)
+       }
+
        // configure the run system
        opts := &sysconfig.Options{TargetRootDir: boot.InstallHostWritableDir, GadgetDir: gadgetDir}
        // configure cloud init

will fix it (this diff obviously needs tests/cleanup etc it’s just a quick draft).

4

Based on what @mvo said below, can you please file a bug here. Thanks!

5

Ah it seems we only implemented copying that file from run mode -> tmpfs for recovery mode in the initramfs, when we also should do what @mvo suggests too

6

I’ve opened a PR that preserves timesyncd timestamp: https://github.com/snapcore/snapd/pull/10980

1 Like
7

Thanks @mborzecki! I see that the PR is merged and will watch for it on the next release of snapd.

8

The branch was merged and the change should available in snapd edge channel now. If possible, please build an image with snapd from edge and verify that it indeed fixes the problem on your device.