Core snap validation process at beta channel

snapd
1

The snapd core team is responsible for validating the core snap once it reaches beta for all the architectures, as described in Core snap delivery process. This post describes in detailt the actions taken to perform that validation.

There are five test scenarios to cover. The log output of each test execution should be saved in paste.ubuntu.com, and the complete results should be kept together as in In progress: snapd 2.26.9. These are the test scenarios:

Execution of snapd’s spread suite with an image built from the beta channel

Dragonboard:

The dragonboard tests are executed in the lab using testflinger. First of all, install testflinger-cli snap:

sudo snap install testflinger-cli --devmode

Then install the tpr tool, it helps to create testflinger jobs:

go get github.com/fgimenez/validator/cmd/tpr

Then change to snapd’s project directory and checkout the release branch

git fetch upstream
git checkout release/<release_number>
git rebase upstream/release/<release_number>

Now execute tpr to get the testflinger jobs:

tpr -channel beta -release release/<release_number>

The output will look like:

2017/07/26 16:27:48 [/tmp/941344167 /tmp/779053786 /tmp/019702897 /tmp/816795420]

Each of the tmp files is a testflinger job, the full suite is split in for of them so that they can be executed in parallel in the lab. In order for them to be accessible after first boot, the images used in the lab are created with a model assertion different from canonical’s. Because of that the tests/main/auto-refresh test will fail and can make the whole test execution, fail, you should remove it from the file that haves it before running the tests.

To actually run it you need the canonical VPN to be up, for each job file:

JOB_ID=$(testflinger-cli submit -q <job_file>)
echo $JOB_ID
testflinger-cli poll $JOB_ID

Rest of architectures: create beta images for each architecture.

We are currently using the create.sh script from https://github.com/fgimenez/validator, the command to issue from the root of the project is:

./create.sh beta

This creates images for the five reference architectures in the ./images directory. Then, start one instance from that images:

For amd64 and i386:

First boot an instance from the image:

kvm -snapshot -smp 2 -m 1500 -redir tcp:8022::22 -nographic -serial mon:stdio <path_to_validator_project>/images/pc-<arch>-edge/pc.img

being amd64 or i386, and run the initial console-conf configuration.

The next steps are performed from the snapd project directory, first, change to the release branch:

git fetch upstream
git checkout release/<release_number>

Then execute the suite with:

export SPREAD_EXTERNAL_ADDRESS=localhost:8022
. ./tests/lib/external/prepare-ssh.sh localhost 8022
spread -v external:ubuntu-core-16-64

For pi2 and pi3:

First flash the beta image to a sd card, if it is accessible at /dev/sdc, for pi3 this command would work:

sudo dd if=<path_to_validator_project>/images/pi3-beta/pi3.img of=/dev/sdc bs=4M oflag=sync status=noxfer

Then boot the board and run the initial console-conf setup, after it is finished you get the system shows the testbed ip, save it for later.

The next steps are performed from the snapd project directory, first, change to the release branch:

git fetch upstream
git checkout release/<release_number>
git rebase upstream/release/<release_number>

Then execute the suite with:

export SPREAD_EXTERNAL_ADDRESS=<testbed_ip>:22
. ./tests/lib/external/prepare-ssh.sh <testbed_ip> 22
spread -v external:ubuntu-core-16-arm-32

Refresh core from stable image to beta channel and execution of snapd’s spread suite

Dragonboard

Given that the dragonboard test are going to be executed on the lab the procedure differs for this architecture:

First of all, install testflinger-cli snap:

sudo snap install testflinger-cli --devmode

Then install the tpr tool, it helps to create testflinger jobs:

go get github.com/fgimenez/validator/cmd/tpr

Then change to snapd’s project directory and checkout the release branch

git fetch upstream
git checkout release/<release_number>
git rebase upstream/release/<release_number>

Now execute tpr to get the testflinger jobs:

tpr -channel beta -from stable -release release/<release_number>

The output will look like:

2017/07/26 16:27:48 [/tmp/941344167 /tmp/779053786 /tmp/019702897 /tmp/816795420]

Each of the tmp files is a testflinger job, the full suite is split in for of them so that they can be executed in parallel in the lab. In order for them to be accessible after first boot, the images used in the lab are created with a model assertion different from canonical’s. Because of that the tests/main/auto-refresh test will fail and can make the whole test execution, fail, you should remove it from the file that haves it before running the tests.

To actually run it you need the canonical VPN to be up, for each job file:

JOB_ID=$(testflinger-cli submit -q <job_file>)
echo $JOB_ID
testflinger-cli poll $JOB_ID

For the rest of architectures (amd64, i386, pi2, pi3):

kvm -snapshot -smp 2 -m 1500 -redir tcp:8022::22 -nographic -serial mon:stdio <path_stable_image>
  • For pi2 and pi3:
    First flash the image to a sd card, if it is accessible at /dev/sdc, for pi3 this command would work:
sudo dd if=<path_to_stable_image> of=/dev/sdc bs=4M oflag=sync status=noxfer

Then boot the board and run the initial console-conf setup, after it is finished you get the system shows the testbed ip, save it for later.

  • Refresh
    First ssh into the image, for amd64 and i386:
ssh -p 8022 localhost

and for pi2 and pi3:

ssh <testbed_ip>

Then refresh all the snaps:

sudo snap refresh

The core snap refresh currently gets stuck refreshing profiles (phase 2), you can send the process to background with CTRL+z and then reboot manually (if not it takes 10 min to make the programmed reboot). Before rebooting manually make sure that the only ongoing upgrade is core (first running snap changes and the snap change <refresh_change_id>)

  • Refresh core to beta
    After rebooting ssh again into the instance, check that the refresh went fine (snap changes) and refresh core to beta:
sudo snap refresh --beta core

Same as above for dealing with the after-refresh reboot.

  • Run tests, from the snapd project directory, first change to the release branch:
git fetch upstream
git checkout release/<release_number>
git rebase upstream/release/<release_number>
  • For amd64 and i386:
export SPREAD_EXTERNAL_ADDRESS=localhost:8022
. ./tests/lib/external/prepare-ssh.sh localhost 8022
spread -v external:ubuntu-core-16-64
  • For pi2 and pi3:
export SPREAD_EXTERNAL_ADDRESS=<testbed_ip>:22
. ./tests/lib/external/prepare-ssh.sh <testbed_ip> 22
spread -v external:ubuntu-core-16-arm-32

console-conf automated tests

Starting from a beta image as described in Execution of snapd's spread suite with an image built from the beta channel

The test reside in https://github.com/sergiocazzolato/console-conf-tests, first clone the repo. Then, depending on the architecture:

  • For amd64, i386:
    First boot an instance from the beta image:
kvm -snapshot -smp 2 -m 1500 -redir tcp:8022::22 -nographic -serial mon:stdio <path_to_validator_project>/images/pc-<arch>-edge/pc.img

being amd64 or i386, and run the initial console-conf configuration.

Then execute the suite with:

export SPREAD_EXTERNAL_ADDRESS=localhost:8022
. ./external/prepare-ssh localhost 8022
spread -v external:ubuntu-core-16-64:tests/
  • For pi3 and dragonboard:
    First flash the beta image to a sd card, if it is accessible at /dev/sdc, for pi3 this command would work:
sudo dd if=<path_to_validator_project>/images/pi3-beta/pi3.img of=/dev/sdc bs=4M oflag=sync status=noxfer

Then boot the board and run the initial console-conf setup, after it is finished you get the system shows the testbed ip, save it for later.
Finally execute the suite with:

export WIFI_SSID=<WIFI_SSID>
export WIFI_PASSWORD=<WIFI_PASSWORD>
export SPREAD_EXTERNAL_ADDRESS=<testbed_ip>:22
. ./external/prepare-ssh <testbed_ip> 22
spread -v external:ubuntu-core-16-arm-32:tests/
  • For pi2: same as for pi3 but without setting WIFI_SSID nor WIFI_PASSWORD

Core revert test

From the release branch, execute:

$ export SPREAD_CORE_CHANNEL=stable
$ export SPREAD_CORE_REFRESH_CHANNEL=beta
$ spread -v qemu:ubuntu-16.04-64:tests/nested/core-revert

snapd deb from proposed

From the snapd project directory, first change to the release branch:

git fetch upstream
git checkout release/<release_number>
git rebase upstream/release/<release_number>

Setup environment variables:

export SPREAD_MODIFY_CORE_SNAP_FOR_REEXEC=0
export SPREAD_TRUST_TEST_KEYS=false
export SPREAD_SRU_VALIDATION=1
export SPREAD_SNAP_REEXEC=0
export SPREAD_CORE_CHANNEL=beta

And then, for release in (14.04, 16.04, 16.10, 17.04) run the full suite:

spread -v qemu:ubuntu-<release>-64:tests/main/

Once those scenarios passed successfully these are the next steps:

  • Notify CE team about the promotion.

  • Ask Jamie Bennet and Michael Vogt for their approval of the promotion.

With the approval from the last point, do the promotion from beta to candidate. This script can be useful for that https://github.com/fgimenez/validator/blob/master/promote.sh, running:

$ ./promote.sh beta candidate

(snapcraft login with credentials allowed to access the core snap is required for this to work).

After the promotion to candidate is done, these steps are required:

2

Thanks Federico, this documentation is awesome. I have worked on automating part of this process and pushed the code in the project https://github.com/sergiocazzolato/snappy-qa-jobs