Running spread tests for SRU VALIDATION on spread-cron

There are some jobs such as snapd-from-zesty-updates-edge or snapd-from-zesty-updates-edge on https://travis-ci.org/snapcore/spread-cron/branches which are running periodically all the spread tests by installing snapd from apt and then updating from proposed.

It is failing in most of the cases due to the tests on master (the branch used to run the tests) often is not aligned with the build downloaded from proposed. An example of that is seen on https://travis-ci.org/snapcore/spread-cron/builds/262177661#L1534 where the command “snap interface” is not still in the snapd used for these tests.

A possible solution for this could be tagging the snap used in proposed and then use this tag to run the tests.
Perhaps those tests should not be executed periodically, but manually when SRU needs to be validated.

I would like to hear other opinions about this.

Why are you testing proposed?

Isn’t the idea to start from the deb in -updates, then install a deb generated from master, and run the tests on the new deb?

I might be misunderstanding, of course.

@elopio, well, for SRU validation we need to make sure that having installed snapd and update from proposed all the functionalities still work for that update.

The script which makes that is the following:

apt install -y snapd
cp /etc/apt/sources.list sources.list.back
echo "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -c -s)-proposed restricted main multiverse universe" | tee /etc/apt/sources.list -a
apt update
apt install -y --only-upgrade snapd
mv sources.list.back /etc/apt/sources.list
apt update

The problem is that when we test it we use the master branch, which often has new tests for new features that are not included in the update. So perheps a solution is to tag each update on the repository and use that tag to run the tests.

I’m sorry if I’m still simplifying the problem, as I said, I might be missing details. Bug when you put a new version in -proposed, that version must correspond to a git tag in the master branch. Instead of using master, you should use the tests from the corresponding tag.

All of this is solved when you use autopkgtests, because the deb includes the tests. And in order to get the new version approved in -proposed, the tests have to be green. So the tests and the code inside the deb are always in sync.