Snapd 2.59.2 in-place updates not working (firefox beta): PROBABLY FIXED

There is a documentation page about deferred updates but it hasn’t been updated to reflect the new behaviour. We’ll try to update that soon.

About the manual triggering of a background download, I think there are plans for work on a two-part refresh (first a download, then the update of the snap in separate steps) which I think would fit that use case. One would be able to download the snap first, then close the snap and manually refresh the snap with less downtime. As far as I know, that work is planned for later in the year.

3 Likes

thanks for that claryfication. What is still not clear to me though: does “retrying refresh” after pre-downloading the files imply that restarting a snap will always start with the newer version or is there still some ~cron-based refresh time involved?

that sounds good

The refresh retry should start almost immediately after you close the snap, although it’s technically possible to restart it so fast that you prevent the refresh from going through. During the refresh process, the snap becomes unavailable to run for a bit and then becomes available again once its refreshed. Since version 2.59.2, snapd also creates a notification stating that the snap was refreshed so at that point it’s definitely the new version that is running.

1 Like

I have intermittently seen this problem reoccur, although it works at other times. When I have the problem, there is no cgroup or process hanging around. However, I always notice that snap changes does not report a pre-download of the update whenever the auto update does not happen. Whenever the update works, snap changes shows that snap has pre-downloaded the update.

snap changes does not report a pre-download of the update whenever the auto update does not happen

Can you expand on this? Does that mean that sometimes you get a notification to close a snap but, when you do, the refresh isn’t triggered and there’s no pre-download entry in output of snap changes?

that seems unlikely when you put it like that. I will watch and report here. It would be a lot easier if I knew a good way of triggering the update process.

Right, i confirm this. I am using kubuntu 23.04 at the moment. Firefox snap (stable) is 113. I got the notification about a snap update. But after I closed firefox, nothing happens. There is no cg or processes open.

tim@ElecGear:~$ snap changes
ID   Status  Spawn                    Ready                    Summary
13   Done    yesterday at 19:24 AEST  yesterday at 19:25 AEST  Install "authy" snap
14   Done    yesterday at 19:24 AEST  yesterday at 19:24 AEST  Auto-refresh snap "core22"
15   Done    yesterday at 19:25 AEST  yesterday at 19:26 AEST  Install "bitwarden" snap
16   Done    today at 17:36 AEST      today at 17:37 AEST      Pre-download tasks for auto-refresh

tim@ElecGear:~$ snap refresh --list Name Version Rev Size Publisher Notes firefox 114.0-3 2751 256MB mozilla✓ - tim@ElecGear:~$

[I then exited firefox again … nothing happens. sudo snap refresh updated firefox without errors]

Hi. Was this reply to my message? If so, then my firefox is still on version 113.0.1-1 despite me waiting for it to DTRT for two weeks since last post. As far as I know the snap is not on hold, because I never issued such command.

> $ snap info firefox
> name:      firefox
> summary:   Mozilla Firefox web browser
> publisher: Mozilla✓
> store-url: https://snapcraft.io/firefox
> contact:   https://support.mozilla.org/kb/file-bug-report-or-feature-request-mozilla
> license:   unset
> description: |
>   Firefox is a powerful, extensible web browser with support for modern web application
>   technologies.
> commands:
>   - firefox
>   - firefox.geckodriver
> snap-id:      3wdHCAVyZEmYsCMFDE9qt92UV8rC8Wdk
> tracking:     latest/stable
> refresh-date: 24 days ago, at 11:55 CEST
> channels:
>   latest/stable:    114.0-3       2023-06-06 (2748) 256MB -
>   latest/candidate: 114.0-3       2023-06-02 (2744) 256MB -
>   latest/beta:      114.0b9-1     2023-05-26 (2724) 255MB -
>   latest/edge:      116.0a1       2023-06-07 (2754) 262MB -
>   esr/stable:       102.12.0esr-1 2023-06-06 (2735) 186MB -
>   esr/candidate:    102.12.0esr-1 2023-05-29 (2735) 186MB -
>   esr/beta:         ↑                                     
>   esr/edge:         ↑                                     
> installed:          113.0.1-1                (2667) 253MB -


$ snap refresh --list
Name           Version        Rev   Size   Publisher   Notes
firefox        114.0-3        2748  256MB  mozilla✓    -

Just wondering whats going on. In those two weeks I closed firefox multiple times. Did not receive any popup messages about pending firefox update as far as I remember.

ps. other snaps are updating ok

Just adding another datapoint here that the issue with firefox in-place updates still persists. When a new update is available a notification will pop up indicating to close firefox to start the update. After closing firefox, the ‘snap changes’ command will show something analogous to this:

2916  Done    today at 13:57 EDT      today at 13:57 EDT      Pre-download tasks for auto-refresh

Even after waiting a few minutes the message remains the same and the refresh is never started. In contrast, for other snaps the status will show a change in summary to Refresh snap x with a status of doing. Once done, another notification appears to indicate that ‘snap x’ is ready for relaunch. That process typically takes about 20-30 sec.

There do not appear to be any stray c-group entries for firefox that are blocking the update (based on the command in one of the above comments). In fact, when firefox is closed, running snap refresh firefox will download and refresh firefox as expected. The fact that the manual command starts with a download suggests to me that there may be an issue/delay with the autodownload component, given that the new snap version doesn’t appear to be queued up when the message appears to restart firefox. I’ve tested the stable, candidate and core22 channels and all have the same issue.

Perhaps there is some relation to the phased update issue as indicated by ogra, where the metadata is out of sync with the download. In that case, maybe a solution would be to ensure that no notification is shown until the autodownload is fully completed?

When it goes wrong, it stays wrong. It never does the pre-download

Hi timattrn,

When you encounter the issue again, would you please also confirm that you see “Pre-download tasks for auto-refresh” when running snap changes.

Then would you please also run and share output of snap refresh --time

Thank you for the report. I thought this could be caused by the interplay between the pre-download and the phased updates but I don’t think that’s the case. The pre-download (and the notification) is only triggered if the snap started to auto-refresh and, in that case, a subsequent auto-refresh would also get the new firefox snap. I spoke to the store team and they confirmed that the phased updates are based on client id so, if firefox could be refreshed once, it should be refreshed in the auto-refresh that’s triggered by closing it.

I think this might be happening because the auto-refresh of the “core22” snap causes snapd to restart and, in 2.59, the post-predownload routines that wait for snaps to close weren’t restored after startup. Having this waiting routines survive restarts has been implemented for a while but I think the 2.59 release branch was cut just before that code merged. If that’s correct, this should be fixed by using the edge version or by waiting for version 2.60. However, if you could provide me with your snapd logs when you observe this behaviour, it would help me confirm or disprove this. In particular, the output of snap changes, snap info firefox and journalctl -u snapd would be helpful. Thanks for your help

what would make this much, much easier to test would be a way of triggering the update rather than haphazardly waiting for a refresh of the snap. I am using firefox beta so there is an update every few days, but to replicate this i have to have it open. If I revert to a prior version, the autoupdate does not seem to be triggered. I can see why. However, for debugging this, can we force it to look for updates and so trigger the update at will? I have asked a few times but I don’t have an authoritative answer. It is very inefficient to debug like this, the iteration time is so long. Also, is your hypothesis restricted to FIrefox, or some characteristic of Firefox, such as it being a large download? If not, is there a testing snap that is updates every hour, or something like that?

Actually, on desktop only snapd refreshing should cause it to restart so either something causes a restart or that’s not the issue. The logs will show whether snapd restarted or not. There is a way to force an auto-refresh but it involves changing snapd’s state manually which is dangerous and not recommended since a mistake might brick snapd. The procedure is (as root):

$ snap refresh --channel=latest/stable firefox
# open firefox
$ snap switch --channel=latest/beta firefox
$ systemctl stop snapd.{service,socket}
$ cp /var/lib/snapd/state.json /var/lib/snapd/state.json.bk
$ jq ".data[\"last-refresh\"] = \"2007-08-22T09:30:44.449455783+01:00\"" /var/lib/snapd/state.json > /var/lib/snapd/state.json.new
$ mv /var/lib/snapd/state.json.new /var/lib/snapd/state.json
$ systemctl start snapd.{socket,service}

This will trigger an auto-refresh. The state.json.bk file in the snapd directory can also be removed at the end. It’s just a backup so you can manually roll back, if anything goes wrong.

I used the procedure above to trigger auto-refreshes but I can’t reproduce with any snap (including Firefox, although I’m not included in the phased update so, to get the refresh, I have to modify things a bit which might be throwing this off)

I am still on 2.59.

I just received a notification for Firefox Beta. This is before I exit Firefox Beta:

[~] [cultiver@eks] $ snap changes
ID   Status  Spawn                Ready                Summary
429  Done    today at 08:58 AEST  today at 08:58 AEST  Auto-refresh snap "snap-store"
430  Done    today at 18:43 AEST  today at 18:43 AEST  Pre-download tasks for auto-refresh

snap info firefox name: firefox summary: Mozilla Firefox web browser publisher: Mozilla\u2713 store-url: https://snapcraft.io/firefox contact: https://support.mozilla.org/kb/file-bug-report-or-feature-request-mozilla license: unset description: | Firefox is a powerful, extensible web browser with support for modern web application technologies. commands:

  • firefox
  • firefox.geckodriver snap-id: 3wdHCAVyZEmYsCMFDE9qt92UV8rC8Wdk tracking: latest/beta refresh-date: yestesnap infrday at 08:07 AEST channels: latest/stable: 114.0.1-1 2023-06-09 (2760) 256MB - latest/candidate: 114.0.1-1 2023-06-09 (2760) 256MB - latest/beta: 115.0b5-1 2023-06-14 (2779) 256MB - latest/edge: 116.0a1 2023-06-14 (2780) 262MB - esr/stable: 102.12.0esr-1 2023-06-06 (2735) 186MB - esr/candidate: 102.12.0esr-1 2023-05-29 (2735) 186MB - esr/beta: \u2191
    esr/edge: \u2191
    installed: 115.0b4-1 (2773) 257MB -

I exit firefox, and wait. After a 40 seconds or so, I got a desktop notification about an update to core18 snap had completed. There was no notification about Firefox. Firefox has not updated.

After exiting Firefox and that one update …

[~] [cultiver@eks] $ snap changes
ID   Status  Spawn                Ready                Summary
429  Done    today at 08:58 AEST  today at 08:58 AEST  Auto-refresh snap "snap-store"
430  Done    today at 18:43 AEST  today at 18:43 AEST  Pre-download tasks for auto-refresh
431  Done    today at 18:48 AEST  today at 18:48 AEST  Auto-refresh snap "core18"



snap info firefox
name:      firefox
summary:   Mozilla Firefox web browser
publisher: Mozilla\u2713
store-url: https://snapcraft.io/firefox
contact:   https://support.mozilla.org/kb/file-bug-report-or-feature-request-mozilla
license:   unset
description: |
  Firefox is a powerful, extensible web browser with support for modern web application
  technologies.
commands:
  - firefox
  - firefox.geckodriver
snap-id:      3wdHCAVyZEmYsCMFDE9qt92UV8rC8Wdk
tracking:     latest/beta
refresh-date: yesterday at 08:07 AEST
channels:
  latest/stable:    114.0.1-1     2023-06-09 (2760) 256MB -
  latest/candidate: 114.0.1-1     2023-06-09 (2760) 256MB -
  latest/beta:      115.0b5-1     2023-06-14 (2779) 256MB -
  latest/edge:      116.0a1       2023-06-14 (2780) 262MB -
  esr/stable:       102.12.0esr-1 2023-06-06 (2735) 186MB -
  esr/candidate:    102.12.0esr-1 2023-05-29 (2735) 186MB -
  esr/beta:         \u2191                                     
  esr/edge:         \u2191                                     
installed:          115.0b4-1                (2773) 257MB -


snap refresh --time
timer: 00:00~24:00/4
last: today at 18:48 AEST
next: tomorrow at 05:17 AEST

I did the journalctl but I don;t know how to attach it

So far that all seems consistent with the behaviour of phased updates and it doesn’t seem related to pre-downloads. No pre-download or notification occurred because there was no auto-refresh in the first place. snap refresh --list shows a newer revision but that doesn’t mean there’s a pending auto-refresh. It only means there’s a newer revision available (and you can refresh to it manually). But, since phased updates are used for Firefox, the auto-refresh might still only pick it up at some later date. The refresh schedule is controlled by Mozilla so I can’t say when that will be.

You can run journalctl -b0 -u snapd > snapd_logs when this happens to write the logs to a file and then attach it to the original bug report (https://bugs.launchpad.net/snapd/+bug/2019158). Same thing with systemctl status > status_logs.

However, in Firefox I received the Gnome desktop notification that there was an update pending. The update did not occur. This can not be correct behaviour.

(is the bug that the notification is sent because the snap refresh detected a new version, but the pre-download failed because of a phased updates delay? So that is why it works sometimes … when randomly you are in the early part of the phased rollout, it works? That is pretty funny, and not a very good user experience!)

EDIT: I see from above that you already ruled out such a sync error (where the notification is based on one trigger and the pre-download on another). That’s too bad, because it fits what happens. However, just to be crystal clear, I received a desktop notification that there was a snap firefox update, so that definitely happened. The snap changes before I exited firefox is not specific about which updates it downloaded, it just says " ``` “Pre-download tasks for auto-refresh”. Based on your comments above, you rule out that it has pre-downloaded firefox because the firefox update did not happen. But for sure, I got the pending update notification referring to firefox.

This is very frustrating. My firefox beta is stuck on 115.0b4 (64-bit)

but

tim@yellow:~$ snap refresh --list
Name     Version    Rev   Size   Publisher  Notes
firefox  115.0b6-1  2786  257MB  mozilla✓   -

so I am now two version behind. I got the desktop notification for 115.0b5 but as I said above, the update did not actually happen. And now I get no more notifications. Why not? So what happens now? I have set the snap refresh time to every hour but still no notifications.

snap changes actually did things today, but it just ignored firefox, why?

snap changes
ID   Status  Spawn                    Ready                    Summary
248  Done    yesterday at 19:43 AEST  yesterday at 19:43 AEST  Auto-refresh snap "firmware-updater"
249  Done    today at 10:22 AEST      today at 10:22 AEST      Change configuration of "core" snap
250  Done    today at 10:22 AEST      today at 10:23 AEST      Auto-refresh snaps "code", "snapd"
1 Like

This issue was addressed and the fix is available on snapd latest/edge. Test feedback will be appreciated.