Discussion: snapcraft test

snapcraft
1

The Starcraft team is in the design phase for a new command, snapcraft test. The UX will be very similar to the recently introduced charmcraft command charmcraft test.

In short, the changes will be:

  1. snapcraft init will create a tests directory with a smoketest and some helper functions.
  2. snapcraft test will leverage spread to run user-defined tests on different backends.

Spread supports many backends but we are considering Multipass, LXD VMs, and a GitHub CI backend.

The LXD VM backend would allow using desktop images from https://images.lxd.canonical.com/, which includes Ubuntu, Arch, and OpenSUSE VMs. This would allow testing on Gnome and KDE.

We plan on adding helper utilities to install and remove the snap-under-test and to test the snap with different snapd releases. We’d also like to add a helper function similar to the snapcrafter’s get-screenshot action.

Please post a comment if you have any feedback or ideas. I’m interested in complex testing scenarios so we don’t design a system that precludes a test scenario.

5 Likes
2

To backends, I would also think about supporting incus in general, to be as broadly compatible as possible with regards to backends.

Also additional testing step which checks if all libraries are resolved using ldd or similar utility would be beneficial

And, when we talk about steps, additional step which checks interfaces permissions would be also beneficial to be tested

1 Like
3

Hey @mr_cal

Spread supports many backends but we are considering Multipass, LXD VMs, and a GitHub CI backend.

+1 for LXD. Would it be required that the container be constructed with --vm? Those of us with low resource workstations would benefit from not having to allocate N GB of RAM and M CPU cores.

The LXD VM backend would allow using desktop images from https://images.lxd.canonical.com/, which includes Ubuntu, Arch, and OpenSUSE VMs. This would allow testing on Gnome and KDE.

This is a good idea.

Sounds like an exciting feature is underway.

Best regards

Daniel

1 Like
4

Yeah, I’m planning on using lxd launch --vm because we need some sort of virtualization to test GUI apps. There will be a configuration file called spread.yaml where you can define the RAM, CPU, and number of concurrent vms to launch.

1 Like
5

Good ideas. It may be worthwhile adding a utility to run snapcraft lint and review-tools on the snap.

I wonder if there is a programmatic way to check connected interfaces and their permissions.

3 Likes
6

I might be wrong but I think the question was asking if it were possible with LXD containers rather than LXD VM’s, so that there’s no need to define resources since the containers share nicer than VM’s.

As for my own opinions, I think this all sounds nifty but ultimately it depends on execution. Being able to have an automated built in framework for something as basic as open gui, take screenshot, prove the GUI works kind of thing though would be massive gains even not considering more specific functionality testing.

(Which is to say, if I break something I usually break it in such a way the app doesn’t launch at all, rarely do individual features break unexpectedly though in the snaps I maintain).

7

Yes - using containers instead of VMs would be helpful for computer with limited resources.

8

I see. I’ll experiment and see if I can get an X server running inside a LXD container and gauge how complex it will be.

9

Technically it should be possible, simply because you can pass GPU to container. And we have already existing profile over ubuntu discourse.

1 Like