Now that we are fully focused on the snapcraft snap, we need to test it of course. We currently rely on travis and the autopkgtest runners. Travis has a trusty kernel that doesn’t support snaps. And the autopkgtest thing is to test the deb. So, we will reuse the tools from the snapd team and run the tests for the snapcraft snap using spread.
We have an integration suite that calls the snapcraft command to run the different steps of the snap-ing process. This suite doesn’t fake anything, and it can just call the snapcraft installed in the system, so it’s the right way to check that the snap works as good as running snapcraft from source or from the deb. Here’s the plan:
Set up spread to run the integration suite locally
Get resources to run the spread tests remotely
Trigger the spread tests once a day to test the snapcraft snap in edge
We also have other suites that take a lot more time to run, and check real world scenarios, like making snaps for our demos projects, install them and execute the binaries. Or making snaps for the most popular applications in the store. They are all currently being run using the deb, and they are out of the scope of this task. We will get back to the other suites once we finish with integration.
We already used this suite to check that the snap in edge works correctly and release it to beta.
There’s only one problem I foresee. Our integration suite takes around an hour to run in travis. We can’t currently parallelize it properly because with two threads trying to run snapcraft, they will conflict trying to get the apt lock. As a quick solution we have split the suite in three, which is enough for now but we will hit the same problem soon. I expect that by the time we hit problems again, we can run the snapcraft commands of each test in a container, which gives us full isolation for free.
In order to run the tests on demand to check the edge snap every day we need resources in some cloud.
@niemeyer do you have enough resources on linode to add snapcraft tests in there? Usually, we will just need at most two hours a day of executions. It could be a little more if we need to debug issues that can’t be reproduced locally.
@JamieBennett and @evan mentioned the option to get resources in Amazon too.
@elopio Yes, it should be fine, and if we need to add more resources it’s better to add those in a shared pool. Let me know if you need more Linode API keys.
@elopio We haven’t dropped any keys, so it should still work. Please just keep it for personal use. If you need keys to use in automated infrastructure I can emit more.
About the parallelism and time limit issue described above, note that spread can automatically split tasks into multiple workers. Just add workers: 2, for example, under the respective system. The documentation has more details on it.
To make full use of that kind of parallelization, it would require to have one spread task per snapcraft integration test.
We do a heavy use of python test modules to set up our tests and to check for the results, and that’s good, I want to keep it that way. So, what I have in mind is a different type of parallelization.
First, the suite is a good mix of CPU-bound and IO-bound tasks. When we have the container work ready, it will give us a big boost to run 3 or 4 tests in parallel in the same machine. Then, the suite has some natural partitions that we can easily send to different machines. Currently it’s manually partitioned in three, but if needed it can be partitioned in 20 or something like that. And also there’s a pending item in our backlog to use caches during the tests. This must happen soon because we need better tests for our caches, and it might reduce the execution time considerably (but I still need to check if that’s true).
If that’s not enough, then we can use the methods to partition python suites and have full parallelization with multiple tests running on multiple machines. That would require a closer integration with the spread runners, and I think it won’t be necessary, but it will be an option if we grow to need it.
For now, we don’t need any of it so it’s also out of scope on this task.
Set up spread to run the integration suite locally
Get resources to run the spread tests remotely