"Snap/Snapcraft test" autopkgtest equivalent

Have we considered adding standardised, lightweight smoke testing to Snap/Snapcraft, similar in some ways to autopkgtests in debs?


  1. In one of the projects I contribute to, we had a release successfully pass our automated testing, but an issue in the snapcraft packaging process meant that, even though snapcraft apparently gave no errors, the end snap did not work at all.
  2. There are issues running (and therefore testing) snaps in Docker-based CI/CD pipelines. My snapped project already has extensive CI/CD testing running on Gitlab, but it is awkward to add snap testing into this existing workflow. While I could set up separate spread testing for the snap, any infrastructure I have to create for this is testing infrastructure I need to do only for snap and so I want it to be as simple as possible.
  3. If there was a standardised way to represent some lightweight smoke testing in snaps, the store could automatically run these tests as part of the snap upload and listing process, raising the overall quality of snaps in the store. This would be particularly valuable because of how comparatively easy it is to push a version of an application to the entire user base without adequate testing (unlike the months of betas etc for debs that are part of a release; obviously snap Risk-levels etc are great if you use them properly and have users subscribed to them).

I do not have strong views on how this should be implemented. We could use the autopkgtest format or integrate parts of spread (though without the complexity/flexibility of different Backends or System).

Nor do I have strong views on whether this should be part of snap or snapcraft, though I lean towards snapcraft:

  1. It keeps the complexity/size out of snapd.
  2. It is more aligned to the building process than the using process.
  3. We could then reuse the virtualised environment approach of snapcraft to run the tests in a VM/LXD container by default with a --destructive-mode option.

Note that I am aware of snap Health Checks, but it sounds more like a check of ongoing health (e.g. one application within your Microk8s snap breaks at some point, so health-checks let you surface that) rather than something designed to run in CI as a gate. The deliberately idempotent design and the snap needing something within it to set the status (with timers then being required to check in on the status) make me think it is less appropriate for this use case than simply running test commands and spitting out an exit status and message.