Notifications for out of date stage-packages

For some time we’ve wanted a mechanism to alert snap publishers who use ‘stage-packages’ in their snapcraft.yaml when USNs (Ubuntu Security Notices) have been issued for their staged packages so that the publisher can rebuild the snap with the security updates. All the pieces have come together (snapcraft, store, review-tools, internal hosting, etc) and we are now sending alerts via email.

Phase 1 of the service has limited store integration so the configurability of the service is also limited. Phase 2 of the service will have fuller store integration.

The service works as follows:

  • snaps built with SNAPCRAFT_BUILD_INFO=1 will have a snap/manifest.yaml inserted into the snap before the ‘snapcraft pack’ step
  • Any snaps built on Launchpad will have SNAPCRAFT_BUILD_INFO=1 set automatically
  • non-LP builds opt-in to the service by setting this environment variable
  • periodically the service will examine snaps that have manifest.yaml files for their currently published channels/tracks and checks if USNs have been issued for the versions of the staged packages in the snap. If any revisions are affected, the tool will generate a report to send via email
  • the tool records the emails that have already been sent for USNs by revisions of the snap and will only send the email for a particular report once, unless something changes in the report (eg, a new USN affects the snap or a new affected revision of the snap is published)
  • the report will be emailed to the publisher email and the uploader email by default. The tool has support for sending to additional addresses. For example, we are using this for various Canonical-owned snaps. In phase 2 I would expect this to be configurable via web UI

If you discover any issues, please report it either here in the forum using the ‘store’ tag or at https://bugs.launchpad.net/review-tools/+filebug.

Thanks!

4 Likes

@jdstrand this is awesome, thanks!

1 Like

For those reading this, there’s an interesting thread where we’re discussing how you manage the re-building of stable snaps when you get these notifications.

That’s great, thanks @jdstrand and everyone who made this possible!

@jdstrand Do we have any plans to do something if developers don’t update their snaps? If a developer goes AWOL, runs out of time/money or just doesn’t care, they might not update the applications they uploaded some months back. This could be common among community run projects where someone responsible left the project and the snap becomes orphaned. In the worst case, the developer may actively refuse to update the snap for whatever reason.

When that happens, what’s our plan? Do we send repeated mails to tell them it’s outdated? Will we de-list it from the store? Could we notify users of those snaps that they’re potentially insecure? Would we go as far as reaching into computers and disabling or removing the snap?

I’m asking because this subject has come up a bunch of times, and now we can tell (for some snaps) what they’re made of, we should consider what responsible steps we take next.

Please note that this is phase 1 of the feature, and I’m not driving the overall design of the feature and may not have all the context or the most recent information (@niemeyer, @sergiusens, @roadmr, @noise and/or @ratliff may have more information, and any of them may correct me :slight_smile: ).

What I can say is that the service by design (for phase 1) does not nag if a snap is out of date since just because a staged package is out of date, it does not mean that the snap is using the code in the staged binary that is vulnerable. It is up to the snap publisher to decide how to maintain the snap.

For similar reasons the store (in phase 1) will not give a public report of out-of-date snaps because the automated tool that would generate the report has no context on the snap. Eg, a snap might stage ‘openssl’ to have access to a hashing algorithm. This would pull in libssl. openssl the source package gets routine USNs, but the hashing algorithms rarely have vulnerabilities. Reporting to users that a snap is out of date with libssl would present an inaccurate view of the snap and its publisher.

Since we can’t accurately report on if the snap is vulnerable to the code that a USN addressed, we can’t consider disabling the snap for new users or removing the snap from existing machines. Even if we could accurately report on if vulnerable code is used, we don’t know how users are using the snap. Eg, a snap could be riddled with vulnerabilities but if it is used on a lawnmower with no network reachable services, it may not matter.

What we could do in a future phase of the service which has full store integration is generate a report in the web ui of the snap for the publisher, then the publisher could check off the USNs that don’t affect the snap. We could then nag the publisher to update the status in the web form, which opens the possibility to have a reporting mechanism on the health of the snap and the responsiveness of its publisher (the publisher could obviously game this though). Just because we could do this doesn’t mean we should though; it would have to be very carefully and tastefully designed.

Several people have approached me saying that while the service is very useful, it would be nice if you could run a command on a snap to report everything that was open as opposed to simply having to track the emails. While this was always possible with the review-tools.updates-available --snap=<your snap> --usn-db=<usndb> command, you had to know to fetch the usn database and keep it up to date. I recently added a small wrapper command to the review-tools (review-tools.check-notices) that you can use today to query your snaps more easily. Eg:

$ sudo snap install review-tools --beta
# copy some snap somewhere that check-notices can read it
$ cp /var/lib/snapd/snaps/chromium_512.snap ~/snap/review-tools/common/
$ review-tools.check-notices ~/snap/review-tools/common/chromium_512.snap 
{
  "chromium_512": 
    {"libpam-systemd": ["3806-1", "3816-1", "3816-2", "3816-3"]}
}

Hope this helps. Feedback welcome and as always, feel free to file bugs at https://bugs.launchpad.net/review-tools/+filebug.

2 Likes

Here is a little blog entry that connects the dots on review-tools.check-notices: https://blog.strandboge.com/2019/02/01/monitoring-your-snaps-for-security-updates/

There is now preliminary support for kernel snaps. os and base snaps continue to be supported so long as they use snapcraft and stage-packages (just like app snaps). I also added support for the os snaps (ie, core*) from Canonical that are historically built differently without stage-packages but only when running review-tools.check-notices or review-tools.updates-available --snap <snap> (ie, not with the snap USN notification service). Please report any bugs you may find at https://bugs.launchpad.net/review-tools/+filebug.

Is opt-in by setting SNAPCRAFT_BUILD_INFO=1 still needed? I notice the manifest.yaml is now created without explicitly setting that.

Does the notification mechanism apply to snaps published in brand stores?

Is notification therefore always ON: snap publishers need do nothing as long as the snap was built with a manifest?

Does the mechanism support stage-snaps, or will it?

I’ll let @sergiusens answer this.

AFAIK, yes. @roadmr, the swift dump is for all stores, not just the public store, correct?

It is always on, yes.

Not at this time, but I’ve taken a TODO to look into this.

Yes, this is correct, the dump covers snaps in all stores.

  • Daniel