Refresh Timer doesn't seem to honor timezone change

Nikolaj · July 6, 2023, 8:32pm

I’m currently looking for a way to update the timezone (and/or offset) used by snap refresh.timer.

Long story short. I have an Ubuntu 22.04 (Test) Server which was set to use UTC timezone during the installation. Today, I want to change the timezone of the system to America/Los_Angeles however snap refresh schedule doesn’t seem to play along correctly - or (perhaps) I have to calculate the offset myself. I have tried rebooting the (test) server but Snap still isn’t playing along.

In reality, it’s actually the the other way around. I have a couple of other Ubuntu 22.04 Server units which all have America/Los_Angeles timezone defined. Now I want to streamline across the fleet and update their timezone to UTC. Wondering if I’ll run into the same issue.

Is there a configuration file or alike somewhere on the system that I can modify? How do I go about address this challenge? Any input is highly appreciated.

snap version:

snap    2.59.5
snapd   2.59.5
series  16
ubuntu  22.04
kernel  5.15.0-71-generic

Test unit, initially UTC timezone (UTC +00:00):

timedatectl:
  Local time: Thu 2023-07-06 19:31:22 UTC
  Universal time: Thu 2023-07-06 19:31:22 UTC
  RTC time: Thu 2023-07-06 19:31:22
  Time zone: Etc/UTC (UTC, +0000)
  System clock synchronized: yes
  NTP service: active
  RTC in local TZ: no

ls -l /etc/localtime:
  lrwxrwxrwx 1 root root 27 Jul  6 19:31 /etc/localtime -> /usr/share/zoneinfo/Etc/UTC

cat /etc/timezone:
  Etc/UTC

snap get system system.timezone:
  UTC

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
  timer: sat5,23:00-01:00
  last: 12 days ago, at 23:02 UTC
  next: in 23 days, at 23:00 UTC

Test unit, after timezone changed from UTC (+00:00) to America/Los_Angeles (-08:00 / -07:00):

timedatectl:
  Local time: Thu 2023-07-06 12:17:46 PDT
  Universal time: Thu 2023-07-06 19:17:46 UTC
  RTC time: Thu 2023-07-06 19:17:46
  Time zone: America/Los_Angeles (PDT, -0700)
  System clock synchronized: yes
  NTP service: active
  RTC in local TZ: no

ls -l /etc/localtime:
  lrwxrwxrwx 1 root root 39 Jul  6 12:02 /etc/localtime -> /usr/share/zoneinfo/America/Los_Angeles

cat /etc/timezone:
  America/Los_Angeles

snap get system system.timezone:
  UTC

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
  timer: sat5,23:00-01:00
  last: 12 days ago, at 16:02 PDT
  next: in 23 days, at 16:00 PDT

debug log:
  snapd[552559]: logger.go:92: DEBUG: Next refresh scheduled for 2023-07-29T16:00:00-07:00.

Here’s one of my other units, set to America/Los_Angeles timezone during the installation: Unfortunately, can’t change the timezone to UTC just yet as this is one of my live production units.

timedatectl:
  Local time: Thu 2023-07-06 10:25:09 PDT
  Universal time: Thu 2023-07-06 17:25:09 UTC
  RTC time: Thu 2023-07-06 17:25:09
  Time zone: America/Los_Angeles (PDT, -0700)
  System clock synchronized: yes
  NTP service: active
  RTC in local TZ: no

ls -l /etc/localtime:
  lrwxrwxrwx 1 root root 39 May 31 09:01 /etc/localtime -> /usr/share/zoneinfo/America/Los_Angeles

cat /etc/timezone:
  America/Los_Angeles

snap get system system.timezone:
  UTC

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
  timer: sat5,23:00-01:00
  last: 12 days ago, at 23:03 PDT
  next: in 23 days, at 23:00 PDT

Nikolaj · July 8, 2023, 4:05pm

Still haven’t found a solution to this. Any input is highly appreciated

Tried restarting the snapd service. Tried rebooting the entire unit. Tried setting a (Snap) timezone via set/get but that feature seems to be broken as well.

1) snap get system system.timezone:
     UTC

2) snap set system system.timezone="America/Los_Angeles" (tried with and without quotes).

3) snap get system system.timezone:
     UTC

Is there a snapd configuration file somewhere that I can edit? The /var/lib/snapd/state.json file appears to hold the updated timezone “America/Los_Angeles” but yet the refresh.timer insist on calculating the offset for last/next update. So, in short; This is my current challenge.

Test unit, initially UTC timezone (UTC +00:00):

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
  timer: sat5,23:00-01:00            <<<------------------------|
  last: 12 days ago, at 23:02 UTC                               |
  next: in 23 days, at 23:00 UTC     <<<----- Following the "refresh.timer", OK.

Test unit, after timezone changed from UTC (+00:00) to America/Los_Angeles (-08:00 / -07:00):

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
  timer: sat5,23:00-01:00            <<<-------------------------------|
  last: 12 days ago, at 16:02 PDT                                      |
  next: in 23 days, at 16:00 PDT     <<<----- Not following the "refresh.timer", not OK.

debug log:
  snapd[552559]: logger.go:92: DEBUG: Next refresh scheduled for 2023-07-29T16:00:00-07:00.

Is the only option really to purge and re-install Snap?

Again, any input is highly appreciated.

ogra · July 8, 2023, 5:27pm

system.timezone is most likely only relevant on UbuntuCore installs (where you actually need to set it from that option) and will likely not have any effect on classic installations…

have you tried to simply delete the timer and re-add it after the timezone change ?

… and you should also file a bug indeed…

Nikolaj · July 10, 2023, 6:25pm

Hey @ogra,

Thanks for your reply. Make sense, yeah.

Actually, yes, I did try to “snap unset system refresh.timer” before setting it again. At one time, it actually looked correct however when retesting the steps, it went back to being incorrect. A reboot of the unit also put it back to an incorrect state after timezone change.

mborzecki1 · July 11, 2023, 1:44pm

I don’t think you can set a different timezone for this very specific operation, but you could override the timezone of the snapd process (note this will likely be applied to all processes it spawns too). For this you can overlay you own tweaks to the snapd.service file by running systemctl edit snapd.service and then add something like:

[Service]
Environment=TZ=UTC

Nikolaj · July 11, 2023, 7:40pm

Hey @mborzecki1,

Thank you for your reply.

I’ll definitely look into your suggestion, RE: override the timezone of the snapd process.

No, like you, I don’t think one can set a different timezone for the refresh.timer either. I’m just curious about the logic behind it and whether there’s a config file or alike I can modify/adjust to instruct Snap not to calculate the offset or at least update it’s initial timezone settings?

It appears to me (I may be wrong) that refresh.timer is following the initial timezone of the system (whenever snap was installed) instead of honoring the timezone configured on the system itself.

Initially everything looks correct (as can be seen below) but once one changes the timezone of the system, Snap insist on calculating an offset.

Test unit, initially UTC timezone (UTC +00:00):

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
  timer: sat5,23:00-01:00            <<<------------------------|
  last: 12 days ago, at 23:02 UTC                               |
  next: in 23 days, at 23:00 UTC     <<<----- Following the "refresh.timer", OK.

Test unit, after timezone changed from UTC (+00:00) to America/Los_Angeles (-08:00 / -07:00):

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
  timer: sat5,23:00-01:00            <<<-------------------------------|
  last: 12 days ago, at 16:02 PDT                                      |
  next: in 23 days, at 16:00 PDT     <<<----- Not following the "refresh.timer", not OK.

debug log:
  snapd[552559]: logger.go:92: DEBUG: Next refresh scheduled for 2023-07-29T16:00:00-07:00.

Another Test unit, initially America/Los_Angeles timezone (UTC -08:00 / -07:00):

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
  timer: sat5,23:00-01:00            <<<------------------------|
  last: yesterday at 15:15 PDT                                  |
  next: in 18 days, at 23:00 PDT     <<<----- Following the "refresh.timer", OK.

Another Test unit, after timezone changed from America/Los_Angeles (-08:00 / -07:00) UTC (+00:00):

snap get system refresh.timer:
  sat5,23:00-01:00

snap refresh --time:
timer: sat5,23:00-01:00            <<<----------------------------|
last: yesterday at 22:15 UTC                                      |
next: in 19 days, at 06:00 UTC     <<<----- Not following the "refresh.timer", not OK.

Nikolaj · July 18, 2023, 12:11am

Update: So, I finally figured out what was up 'n down here. Thank you all for chiming in.

The offset is based on the values stored within the /var/lib/snapd/state.json file (Ubuntu 18.04/22.04). One has to modify, at least, the value of last-refresh to match the new timezone of the system. In my case, I simply just modified all timestamps from being -07:00 or -08:00 (American/Los_Angeles) to be Z (Etc/UTC). This made the snap refresh --time look and function as expected after a snapd service restart.

Note in my example below, I did also modify the value of refresh.timer (not a mistake and not needed as part of the workaround).

Immediately, after changing timezone from America/Los_Angeles (-08:00 / -07:00) UTC (+00:00):

snap refresh --time:
  timer: sat5,23:00-01:00            <<<----------------------------|
  last: today at 23:39 UTC                                          |
  next: in 13 days, at 06:00 UTC     <<<----- Not following the "refresh.timer", not OK.

After modifying all timestamps within the /var/lib/snapd/state.json file:

snap refresh --time:
  timer: tue5,23:45-01:45            <<<------------------------|
  last: today at 16:39 UTC                                      |
  next: in 8 days, at 23:45 UTC     <<<----- Following the "refresh.timer", OK.

How to:

Change system timezone.
Update all timestamps within /var/lib/snapd/state.json to follow current system timezone.
Update refresh.timer within /var/lib/snapd/state.json if needed. (Not needed for the workaround to work). - Do not use the snap refresh.timer=xxyyzz command as this will overwrite any manual changes made to the /var/lib/snapd/state.json file.
Restart the snapd service.
Validate the snap refresh schedule, snap refresh --time.
Done.

ogra · July 18, 2023, 9:09am

While it is great you found a workaround, hacking the state.json is indeed not expected …

so please make sure to file this bug and link here from it, this needs to be fixed on a snapd level …