Building and testing the core snap on edge channel

[cachio]

Intro

The idea of this topic is to give a brief introduction about how the core snap is gonna be built as part of the spread-cron daily execution. Any suggestion is welcome.

There are 2 spread-cron branches involved in the process:

• b1) snapd-vendor-sync
• b2) build-core-lp

There are two repositories involved as well:

• r1) github.com/snapcore/snapd
• r2) git.launchpad.net/snapd-vendor

Branches description

The branch b1 is in charged of copying all the code from the repository r1 to r2, including the vendor dependencies.

The branch b2 is in charged of interact with launchpad in order to build the core snap in the different supported architectures (‘amd64’, ‘i386’, ‘armhf’, ‘arm64’, ‘ppc64el’, ‘s390x’), and track the relation between the commit used from repo r2 to build the core snap and the different architectures. This tracking is needed then to be able to test the core snap on edge.

Branches execution

The branch b1 is executed when we have a green in travis for the master branch for the repo r1 and b2 is executed by default once a day (nightly). Both branches could be triggered manually if needed.

Tracking information

The tracking information is a file that just leaves in repo r2 with the following format:

amd64 0e2947f169a253661245bbcd64609f21d1a4925a
i386 0e2947f169a253661245bbcd64609f21d1a4925a
armhf 0e2947f169a253661245bbcd64609f21d1a4925a
arm64 0e2947f169a253661245bbcd64609f21d1a4925a
ppc64el 0e2947f169a253661245bbcd64609f21d1a4925a
s390x 0e2947f169a253661245bbcd64609f21d1a4925a

This file is updated with the new commits when the core snap is built as part of the branch b2. Just the sucessfully architectures are updated.

[niemeyer]

Thanks for those notes. Moving a few questions originally made in the PR here, and expanding a bit on it.

How are the credentials controlled? The core snap is obviously a very sensitive one, and we don’t want a CI machine in the wild being able to publish those, at least not in any channels other than edge.

The process seems somewhat complex, with many moving parts. In general that usually means we’ll see it breaking in many different ways. How can we simplify it? Can we use a more traditional building mechanism? I mean, we use git, Launchpad supports git, Launchpad supports synchronization of repositories, and Launchpad supports building of snaps. That seems quite fitting. What are we missing to have Launchpad driving this all by itself? Perhaps this might also help solving the credentials issue. I’ll ignore that open point for the next questions.

Why do we need two different spread-cron branches, one for syncing, and one for building? Can’t we have a single one that does what’s needed, and just once a day instead of doing repeated work unnecessarily (and taking resources)?

Why does it have a file with a list of architectures and the same digest every time? When would it diverge? Also, I believe we already bundle git digest into the snapd version… is that not working? If so, it’d be nicer to fix it than to have more of it.

[cachio]

Thanks @niemeyer for reading this.

How are the credentials controlled? The core snap is obviously a very sensitive one, and we don’t want a CI machine in the wild being able to publish those, at least not in any channels other than edge.

Well, the idea is to use the same machine we use for spread-cron branches. It is a vm in canonistar with restricted access, so, few many people is able to access to there. I need to see if it is possible to restrict that machine to just build the core snap in edge.

Why do we need two different spread-cron branches, one for syncing, and one for building? Can’t we have a single one that does what’s needed, and just once a day instead of doing repeated work unnecessarily (and taking resources)?

Currently we have 2 branches because we are synching on each green that we get on travis for the master branch, and we plan to build the core snap on edge once a day, but also makes sense to synch and build once a day and by doing that we will have just one branch. Do you agree with that?

Why does it have a file with a list of architectures and the same digest every time? When would it diverge?

Well we have different architectures in that file because if we build and it fails for a particular arch, then the diges is not updated for that one, but the others are updated. So the digest could diverge. Always when I arch fails we will tag that buidl as failed, so we can review the reasons.

Also, I believe we already bundle git digest into the snapd version… is that not working? If so, it’d be nicer to fix it than to have more of it.

These commits are used for running tests against edge, and for that we use the launchpad branch, so it is much easier to sabe the commit hash of that repo to then use it to run the tests, otherwise I need either to download the github snapd repo or to parse the commit messages to match the hash.

[niemeyer]

Well, the idea is to use the same machine we use for spread-cron branches. It is a vm in canonistar with restricted access, so, few many people is able to access to there. I need to see if it is possible to restrict that machine to just build the core snap in edge.

Let’s please have a call to discuss this, including @mvo and @jdstrand. But only after we’ve nailed down the other issues, please.

Currently we have 2 branches because we are synching on each green that we get on travis for the master branch, and we plan to build the core snap on edge once a day, but also makes sense to synch and build once a day and by doing that we will have just one branch. Do you agree with that?

The question is why can’t we sync once, and build once, and do that on exactly the digest we want to?

Well we have different architectures in that file because if we build and it fails for a particular arch, then the diges is not updated for that one, but the others are updated. So the digest could diverge. Always when I arch fails we will tag that buidl as failed, so we can review the reasons.

Not sure we want that. Having N different versions of snapd being released on edge, one for each architecture, sounds a bit like a nightmare.

These commits are used for running tests against edge, and for that we use the launchpad branch, so it is much easier to sabe the commit hash of that repo to then use it to run the tests, otherwise I need either to download the github snapd repo or to parse the commit messages to match the hash.

That’s another red flag. We do want to be able to associate a build with its source digest, as we do today for git releases. And that digest must be the real one on our real branch, not a digest on a Launchpad branch that most of us will not see.

Then, this paragraph seems to have been ignored:

The process seems somewhat complex, with many moving parts. In general that usually means we’ll see it breaking in many different ways. How can we simplify it? Can we use a more traditional building mechanism? I mean, we use git, Launchpad supports git, Launchpad supports synchronization of repositories, and Launchpad supports building of snaps. That seems quite fitting. What are we missing to have Launchpad driving this all by itself? Perhaps this might also help solving the credentials issue. I’ll ignore that open point for the next questions.

[cachio]

Let’s please have a call to discuss this, including @mvo and @jdstrand. But only after we’ve nailed down the other issues, please.

I’ll schedule something for tomorrow to discuss about that.

The question is why can’t we sync once, and build once, and do that on exactly the digest we want to?

We can do it. I agree with that idea.

Not sure we want that. Having N different versions of snapd being released on edge, one for each architecture, sounds a bit like a nightmare.

The only thing to consider is what happen if the build for some arch fails. Currently if some build fails, we have different versions of the core for the different archs. That’s the reason why we have the commits by architecture.

That’s another red flag. We do want to be able to associate a build with its source digest, as we do today for git releases. And that digest must be the real one on our real branch, not a digest on a Launchpad branch that most of us will not see.

Agree, but the digests are just in the launchpad repo, so it is also a mess to save cross digests, isn’t it?

The process seems somewhat complex, with many moving parts. In general that usually means we’ll see it breaking in many different ways. How can we simplify it? Can we use a more traditional building mechanism? I mean, we use git, Launchpad supports git, Launchpad supports synchronization of repositories, and Launchpad supports building of snaps. That seems quite fitting. What are we missing to have Launchpad driving this all by itself? Perhaps this might also help solving the credentials issue. I’ll ignore that open point for the next questions.

I am researching if we can do this, that should be the optimal approach.

[ogra]

The process seems somewhat complex, with many moving parts. In general that usually means we’ll see it breaking in many different ways. How can we simplify it? Can we use a more traditional building mechanism? I mean, we use git, Launchpad supports git, Launchpad supports synchronization of repositories, and Launchpad supports building of snaps. That seems quite fitting. What are we missing to have Launchpad driving this all by itself?

this is exactly how we build the dailies, please see the “Launchpad mirror and automatic builds” section at the bottom of

snapcore/core/blob/master/README.md

It mirrors to:

https://code.launchpad.net/~snappy-dev/core-snap/+git/github-import/+ref/master

which has a snap set up at

https://code.launchpad.net/~snappy-dev/+snap/core

the lp-build-core python script in my home on people.canonical.com:~/core-builds/ then logs in to LP via launchpadlib and does the equivalent of clicking the “Request builds” button on that page … which means handing it over the list of architectures, the list of archives (xenial-updates and the image PPA) and actually triggering the build.

Note that we already have per-commit builds of the snapd deb running in the edge PPA at

https://launchpad.net/~snappy-dev/+archive/ubuntu/edge?field.series_filter=xenial

All you need to do for getting per-commit builds of the core snap is to set up your own copy of the lp-build-core on people.canonical.com (to keep your credentials safe), configure your lp-build-core to additionally use the edge PPA and have a script that regulary checks the edge PPA for new snapd versions … then have that simply run lp-build-core as soon as a new snapd lands in that PPA …

i.e. the only actual thing you need over my setup is to take cron out of the loop and replace it with a PPA watcher to trigger the builds and you have per-commit core builds in edge…

Regarding the git commits, the deb in the edge PPA should already use the commit hash in its deb version string as well as inside the snapd version string from /usr/lib/snapd/info which will be used to craft the core snap version string during the snap build (i am not 100% sure if this hash in the edge PPA is from master or the vendor tree though, this would be a question to @mvo)