[WIP] Refresh App Awareness

Hey @zyga-snapd, has there been any progress on this? The chromium snap continues to receive new bug reports of it being refreshed while running, and losing profile data as a consequence.

Can we expect to have the feature on by default in an upcoming release of snapd?

And what is the status of the work on notifying applications when a refresh is available/has been installed? I’m willing to have the chromium snap react to such notifications, and would gladly test the feature in the early stages of its implementation if it could help get it available to everyone.

1 Like

There’s slow progress. We are landing parts of the feature and shrinking working branches.

1 Like

This is quite painful and I have snap set core experimental.refresh-app-awareness=true. I run Chromium constantly. I keep it open for weeks. It’s incredibly stable on Ubuntu, except for these snap refreshes bringing it down. Every time it has crashed I go look and the snap was updated.

One thing that was very interesting was I happened to check and see there was snap refresh before it crashed. I checked the about dialog in Chromium and it showed there was a new version pending. I didn’t expect that unless you’ve patched it to be snap aware. But it didn’t show the usual update pending icon over the menu elipses in the main UI. If that were visible it would be fantastic because I could easily notice that and manually restart it when I’m ready before a crash.

If helpful: Today I noticed (disk light flashing) during my snap Chromium [edit: No experimental flags anywhere] session that it was auto-refreshing. Soon there was a KDE(?) notification (paraphrased) “Chromium has updated and needs to restart”.

I had made a noteworthy bookmark in this session prior to this update.

I closed and restarted. :white_check_mark:The noteworthy bookmark was still there (this has failed in the past) and :white_check_mark:Chromium did not issue a “did not close properly” warning upon restart (which had been typical in this scenario).

It’s possible I am misreporting the facts but I did pay attention because I was concerned I was going to lose my new bookmark.

I’m glad the notification served its purpose :slight_smile: . Typically, the bookmark would have been lost if you had created it after the refresh to the new version.

Hello everyone. I’d like to post a small update to aid discussing another chunk of work related to this feature.

Context

Currently snapd performs two checks, dubbed the soft and hard checks, as it performs a refresh of a particular snap. Both checks look at the processes associated with the snap being refreshed but differ in how they decide if the check succeeded or not.

The soft check fails if there are any applications or hooks running. Services are ignored as they are normally stopped later in the refresh process. This check catches the simple instance of an application being open.

The hard check is more strict as it runs after services have been stopped. At this time presence of any application process is a problem as snapd is about to create a copy of the snap data directory and the running application can affect the process.

That’s the current state, now for the new component.

Run inhibition lock

I’d like to introduce a new mechanism, called the run inhibition lock. The lock is scoped to a particular snap and is persistent across reboots. Once held the lock causes snap run to pause and wait for the lock to be released. While waiting the user is notified that a snap is being refreshed with either a graphical information dialog or directly in the console (if a tty exists).

The lock file is /var/lib/snapd/inhibit/$SNAP_NAME.lock. Unlike the snap lock, which is stored in /run/snapd/lock/$SNAP_NAME.lock, the file may exist for arbitrary amount of time The snap lock is ephemeral and has a hard limit of being locked for several seconds only.

The lock files are created and removed by snapd and read by snap run.

Hint

The inhibition lock may be empty or may contain a hint. The hint exist to communicate, without having to talk to snapd, why the lock exists. Currently the only value that is defined is the word refresh, which indicates that snapd is performing a refresh operation that affects this snap.

Empty hint indicates that snap execution is uninhibited.

Non-empty hint indicates that snap execution is inhibited.

Expected usage

I wrote a prototype that grabs the run inhibition lock during the snap-unlink task and removes it during the snap-link task. This is technically correct, as the snap cannot be used during that window. I’d like to expand the lock region to start immediately during the soft check mentioned above. While the snap lock is still held. This would allow snapd to detect that a refresh is possible and that it will be possible by the time the hard check is performed.

Inhibition lock operations

The lock files are stored in /var/lib/snapd/inhibit/$SNAP_NAME.lock.

LockAndSetHint

  • open the lock file as 0644
  • obtain an exclusive lock (LOCK_EX)
  • write the hint content
  • close the file.

Unlock

  • open the lock file
  • obtain an exclusive lock
  • truncate the file
  • close the file

IsLocked

  • open the lock file - if missing return informing the lock is not held
  • obtain a shared lock (LOCK_SH)
  • stat the file - if empty then close and return informing the lock is not held
  • read the contents and return informing the lock is held and the hint is the read value
  • close the file
3 Likes

I’ve implemented this as https://github.com/snapcore/snapd/pull/8566

I’ve updated the simple UI branch to use the mechanism outlined above: https://github.com/snapcore/snapd/pull/7700

I’ve pushed many extra branches towards making this feature complete.

For a better overview please track https://github.com/snapcore/snapd/projects/3

only if the check succeed? will it removed later if the hard check fails?

Yes, only if the check succeeds. I will need to think about all kinds of exit paths and ensure that just like with unlink snap, we return the system to an operational state

2 Likes

I just noticed that telegram was failing to download attachments in the usual way that indicated it was refreshed while running:

[1921659.491083] audit: type=1400 audit(1591279662.027:1005541): apparmor="DENIED" operation="mkdir" profile="snap.telegram-desktop.telegram-desktop" name="/home/laney/snap/telegram-desktop/1616/" pid=81570 comm="telegram-deskto" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000

but I have refresh-app-awareness on:

laney@raleigh> sudo snap get core experimental.refresh-app-awareness
true

yet snapd did indeed refresh telegram:

laney@raleigh> snap info telegram-desktop
...
tracking:     latest/stable
refresh-date: 4 days ago, at 16:31 BST

the process accounting seems to be working though

laney@raleigh> snap refresh telegram-desktop --edge
error: cannot refresh "telegram-desktop": snap "telegram-desktop" has running apps

here’s my versions of stuff

snap    2.45
snapd   2.45
series  16
ubuntu  20.04
kernel  5.4.0-29-generic

anything I can usefully do to see what happened? Or maybe the above referenced project means that this is still under active development and it’s better to try a new version.

We do refresh after a week though.

There’s a snap warning in 2.45 (or 2.46, I forgot which), that tells you about force-refresh if it happens.

Ah right, so I guess the most this feature can do for long running processes is put the annoyance off for a week.

2 Likes

We are going to discuss changing the duration period. I was thinking it should be much larger, similar to the length of the snap refresh hold period, aka ~60 days. Details will be available later today.

We’re also looking at enabling this by default in the next release, perhaps.

4 Likes

Apologies for infrequent updates. Over the past few weeks we’ve discussed that the next steps in making this feature available involve:

  • Displaying desktop notifications to the user, making them aware of the pending refresh
  • Increasing the grace period for which an app can block the refresh process for up to two weeks

We are lining up pieces to complete this for the next release. We will again gather feedback and would hope to enable this new behavior by default for the release after that.

There are still several technical changes needed to enable this option by default, those are being worked on but are unlikely to hit this milestone. From the top of my head, changes to SElinux policy relevant to the tracking mechanism, changes to the snapd test suite to accommodate the new behavior and some additional logic related to run inhibition locks that is under review and development.

2 Likes

I’m sure this was discussed, but please keep in mind that an unprivileged user (eg, without snap login, sudo, etc) is in a position to prevent refresh via the refresh app awareness mechanism. I mention this only so this aspect of auto-refresh awareness is kept in mind (I’m not saying that two weeks is too long, but it feels like we might be getting close to too long).

I’ll also mention that of course simply informative desktop notifications are fine, but as soon as we want to do something more interactive, please incorporate policykit/etc to mediate any controlling of the refresh process so that unprivileged users can’t do more than they should.

Lastly, thanks for all your work on this!

1 Like

Two weeks was specifically picked to be longer than one week, so that users are not surprised by something that may have happened over weekend.

I acknowledge the need for policykit. The current plan is to not do any interaction yet. The next plan will be to implement remaining snapctl design, to allow application to be notified the same way the user is, and to allow them to act on it by requesting an update after the process quits.

2 Likes

Refresh app awareness has landed in master and should be available in edge soon. Please try it by enabling the experimental feature:

sudo snap set system experimental.refresh-app-awareness true

Please share your thoughts about the experience.

2 Likes

Thanks! Should we unset the experimental variable documented in the original post before trying this?

(And good luck in your future endeavours when you leave the team!)

1 Like