The automatic build and pubish process of snaps owned by the Snapcrafters

doc
1

One of the tasks in the Snapcrafters template claims:

  • Ask a Snapcrafters admin to fork your repo into github.com/snapcrafters, and configure the repo for automatic publishing into edge on commit

I wonder how it is achieved and what’s the purpose of the .build-trigger.txt file at each repo’s root dir?

2

Great question. This isn’t written down anywhere, and should be.

Here’s what happens next when that request is made.

  • Someone from the snapcrafters admin team (@popey @Wimpress @evan @Igor) will do the import to the snapcrafters org on github
  • We then login to build.snapcraft.io under the snaprafters account and add the repo
    • This requires us to authenticate to the store with the snapcrafters account to prove it owns the registered name
    • Once done, build.snapcraft.io will immediately trigger a build for all supported (or specified with architectures: - build-on: stanza, e.g. obs-studio) architectures and publish to the store edge channel.
  • We then test the build in edge and potentially promote to another channel such as stable.

We recommend that repos in snapcrafters use override-pull scripts to determine the upstream release automatically, rather than hard-wire it in the version: stanza. For example in obs-studio we do this. Note that we compare the latest tagged version in the upstream repo with the version in the beta channel. We’re abusing the beta channel a bit here. If the beta channel version is older than the latest upstream tagged release (or non-existent) then we build the latest stable build into edge, if not then we build the latest master.

Ongoing, builds need to be re-triggered for various reasons. If the source part is a github repo, then in theory build.snapcraft.io will automatically trigger a build each day. However many of the repos in snapcrafters are actually built from debs or other artifacts. Build can’t detect upstream updates of those unfortunately.

However, if we want to build a new stable release of a package, we can just snapcraft close the beta channel, and somehow trigger a build. Due to the work explained above, that will trigger a new build of the latest stable release. the .build-trigger.txt file is simply a file in the repo which will trigger build.snapcraft.io to detect a change in the repo and thus build the product.

Here’s some workflows.

  • We notice the upstream developers have released a new version, or someone files an issue to ping us if we don’t notice.

    • We either login to build.snapcraft.io and press the ‘build now’ button on that project or we touch a file like .build-trigger.txt to make it automatically build

  • Like any snap store publisher, we get an email regarding a snapcrafters snap having a dependency which contains security notices.

    • We touch the .build-trigger.txt file in every affected repo to trigger a rebuild of them all. Under normal circumstances (see above) that would result in an edge build. However, we also snapcraft close [snap] beta to close the beta channel and thus effectively re-build what’s in stable (or the next release if we missed that)

After the build lands in edge:-

  • We do some testing to make sure the build in edge works
  • We promote the snap to the stable (and beta) channels

To assist with these we have a few tools. A bot that @evan wrote which announces in our internal team Slack channel whenever apps in snapcrafters are updated upstream. So we know when snapcrafters repos are in need of an update.

We also have a script that I wrote, which we use to trigger builds-en-masse. It’s simply run with a list of snaps and will then iterate over each one, appending the date to .build-trigger.txt in every repo specified. This is why you might see a flood of updates to repos over a short period.

Finally we use travis in all the repos to help gate changes and test builds before they end up in in the launchpad build system

I think that covers it. Let me know if any of it doesn’t make sense, or if you can think of any improvements we can make. :slight_smile:

2 Likes
3

One immediate improvement that we could make is document why build-trigger.txt exists at the top of the file, like a comment. It should only need 3 lines or so as a pull request for each repo, describing the file.

4

wouldn’t it be nice if the bot that @evan wrote did the update of the build-trigger.txt itself, and thus completely automates the triggering??!

1 Like
5

Yup! Good idea. They were developed separately and not hooked up, but they do run on the same box. However, the bsi-trigger requires signing into github to make the changes. So it’s manually run and then I type in the github password for the snapcrafters account. I made it as a simple shell script, and I guess it needs re-writing to use the github api “properly”

1 Like
6

I’ve made the topic wiki, feel free to make it proper.

7

I noticed that the pull scriptlet trick also useful for the upstream as they have no easy control on the build of the stable release packages via build.snapcraft.io (which, only build the revision at push, but not at new tagged revisions).

Reference: Proposal: Allow overriding the source-tag property for an one-time build in the build infrastructure