Selective-checkout: Check out the tagged release revision if it isn't promoted to the stable channel

The problem

Currently, due to the limitation of the Snap Store build service and Launchpad, it’s not possible to ensure that a snap will be built on certain tagged release as a build is only triggered when:

  • Someone pushes to the associated repository(which development tip (a.k.a. HEAD) not necessarily a tagged release)
  • A Launchpad import occurs if the repo is set to be importing periodically from an external repository

The workaround

The selective-checkout scriptlet enhances the pull step to only build development snapshots snaps if the latest tagged release has already been promoted to the stable channel. This ensures that there’s always a revision of the stable release snap available in the edge channel for the publisher to promote to the stable channel.

Case: Tagged release published to the stable channel

selective-checkout: Last tagged release(1.8.1) is in the stable channel, building development snapshot
selective-checkout: Snap version determined to be "1.8.1-26-gd6ddb74".

Case: Tagged release hasn’t been published to the stable channel

selective-checkout: Last tagged release(1.8.2) hasn't promoted to the stable channel(1.8.1) on the Snap Store, building tagged release instead.
Note: checking out 'v1.8.2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at dcd1bd3 release version 1.8.2
selective-checkout: Snap version determined to be "1.8.2".

This scriptlet is inspired by the snapcrafters’ adopted snaps workflow, documented in the following forum topic:

Refer to the following forum topic for a possible solution proposal to this problem:

Prerequisites

  • Your snap recipe is using the base keyword (i.e. not the legacy snapcraft syntax)
  • Your main component part must be using one of the following version control systems:
    • Git
    • Mercurial
    • Subversion
  • The upstream release tags must be version sorted and be either dot-separated or underscore-separated, optionally prefixed with a fixed string, e.g. 1.2, 1.2.3, v1.2.3, or FOOBAR_3_0_0.
  • The build host must have internet access (for querying snap status on the snap store)
  • Your snap must not be private on the snap store (unlisted is fine)

How to use

  1. In the top-level snap metadata, DO NOT SET the version key, instead, set the adopt-info key with the main component part’s name as the value.

    #version: 
    adopt-info: _part_name_
    
  2. Merge the following part definition:

    parts:
      # Check out the tagged release revision if it isn't promoted to the stable channel
      # https://forum.snapcraft.io/t/selective-checkout-check-out-the-tagged-release-revision-if-it-isnt-promoted-to-the-stable-channel/10617
      selective-checkout:
        source: https://github.com/brlin-tw/selective-checkout.git
        source-tag: v3.0.0
        plugin: dump
        build-packages:
          # Scriptlet dependencies
          - curl
          - jq
          - sed
    
          # Uncomment the VCS your main part is using
          #- git
          #- mercurial
          #- subversion
        stage:
          - scriptlets/selective-checkout
        prime:
          - -*
    
  3. Merge the following keys to the main component part:

    parts:
      _part_name_:
        after:
          - selective-checkout
        # Either the root source tree with a .git repository
        #source: .
        # or an external repository
        #source: https://example.com/project/.git
        # If you specify the `source-depth` property should be deep enough to include a tagged release
        #source-depth: 500
        override-pull: |
          snapcraftctl pull
    
          "${SNAPCRAFT_STAGE}"/scriptlets/selective-checkout
    
          # Do your additional regular stuff
    

The logic

  • The scriptlet will checkout the latest tagged stable release and build tagged release snap if the latest tagged release hasn’t been promoted to the stable channel (check is based on version-sort)
  • The scriptlet will checkout the latest tagged beta/candidate releases if they were found and it is version-sortwise more recent than the version shipped in the more stabler channels (for releasing the snap to beta/candidate channels)
  • The scriptlet will not touch the source tree and build as-is if
    • A VCS repository is not found (will set the snap version to unknown)
    • Working tree is dirty(building test snaps)

Supported command-line options

--append-packaging-revision

Appending the packaging revision string after the snap version string, in the format: _upstream_version_+pkg-_packaging_revision_. Useful for out-of-upstream-source-tree snap build.

--beta-tag-pattern=_pattern_

Customize the extended regular expression pattern of the beta release tags.

Default: -beta[[:digit:]]+$
Example: b[[:digit:]]+$

--debug (DEPRECATED)

Enable debugging messages. This command option is only supported for compatibility reasons, see the SELECTIVE_CHECKOUT_DEBUG environment variable for the recommended method.

--debug-tracing

Enable execution tracing, just in case you’ve encountered a bug.

--dry-run

Don’t run snapcraftctl so that you can run it out of snap build(after changing the working directory to SNAPCRAFT_PART_SRC and exporting the SNAPCRAFT_PROJECT_NAME environment variable, of course!)

--force-snapshot

Force building development snapshot regardless of the status of the snap release, for whatever reason.

--force-stable

Force building stable release regardless of the status of the snap’s releases, for whatever reason.

--packaging-dirty-marker-postfix=_postfix_

The postfix string to be appended to the packaging revision as the dirty marker.

Default: -d

--packaging-revision-minimal-length=_string_length_

The minimal length of the packaging revision string.

Default: 4

--release-candidate-tag-pattern=_pattern_

Set the extended regular expression pattern to match the beta release tags.

Default: -rc[[:digit:]]+$

--release-tag-pattern=_pattern_

Set the extended regular expression pattern of all to release tags to filter from all tags, useful when non-release tags are used.

Default: .*[._].* (any string that contain a dot(.) or an underscore(_)
Example: foobar_[[:digit:]]+(\.[[:digit:]]+){2}

--release-tag-prefix=_prefix_

Set the prefix for all release tags, the prefix will be stripped from the resulting snap’s version string.

Default: v

--snap-postfix-seperator=_seperator_

Set the separator for the postfixed string in the snap version string, the postfixed string will be stripped from the snap version string before comparing with the stripped upstream release version.

Default: +

--stable-tag-pattern=_pattern_

Customize the extended regular expression pattern of the stable release tags.

Default: (unset) Derived from the exclusion of all the release candidates and beta release tags.

--upstream-dirty-marker-postfix=_postfix_

The postfix string to be appended to the upstream version string as the dirty marker.

Default: -dirty

--upstream-revision-minimal-length=_string_length_

The minimal length of the upstream revision string.

Default: 7

Supported environment variables

SELECTIVE_CHECKOUT_DEBUG

Enable debugging messages, supported value: true / false

Default: false

The implementation

https://github.com/brlin-tw/selective-checkout

Snaps that are using this stage snap

TODO

Patches welcome!

  • Support other SCMs(Bazaar, etc.)
  • Don’t require user exporting SNAPCRAFT_PROJECT_NAME before running --dry-run

Happy snapcrafting!

3 Likes

@popey Just FYI, you might find this one useful.

selective-checkout now supports Subversion!

selective-checkout now supports automatic building release candidate and beta tagged releases, the default patterns for matching them are -beta[[:digit:]]+$ and -rc[[:digit:]]+$.

Note that the next update will require new dependencies that need to be added to the build-packages, builds without updating the recipe will fail due to the change.

You might want to enable this post’s notification to hear from these changes and adjust the recipe accordingly.

Hi @Lin-Buo-Ren! The picard snap is failing to build with:

/root/stage/scriptlets/selective-checkout: line 509: last_release_candidate_version: unbound variable
Failed to run 'override-pull': Exit code was 1.
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.
An error occurred when trying to execute 'sudo -i env SNAPCRAFT_HAS_TTY=True snapcraft snap' with 'multipass': returned exit code 2.

So it looks that it’s related to this…didn’t make any change on it since your PR.

1 Like

Apologies for the delay, I’ve identified the issue and is working on fixing it.

It should be fixed since https://github.com/Lin-Buo-Ren/selective-checkout-stage-snap/commit/bc7bd6d34ec0afe14c9ca886466e2d6d217885c2 .

1 Like

This is working now, thanks a lot!

1 Like

Just one question: This is what I’m getting now:

$ snap info picard
name:      picard
summary:   Picard is the official MusicBrainz tagger.
publisher: –
license:   unset
description: |
  Picard supports the majority of audio file formats, is capable of using
  audio fingerprints (AcoustIDs), performing CD lookups and disc ID submissions,
  and it has excellent Unicode support. Additionally, there are several plugins
  available that extend Picard's features.
commands:
  - picard
tracking:     edge
refresh-date: yesterday at 23:23 CEST
channels:
  stable:    2.1.3              2019-05-06 (18) 129MB -
  candidate: 2.1.3-396-gf4d27d1 2019-07-28 (20) 128MB -
  beta:      2.1.3-396-gf4d27d1 2019-07-28 (20) 128MB -
  edge:      2.1.3-396-gf4d27d1 2019-07-28 (20) 128MB -

What does the 396 means? And the gf4d27d1? I guess that the latest is a commit, but I cannot find it in: https://github.com/metabrainz/picard/commits/master

And what do I have to do if I want a snap with just the version number 2.1.3?

It means that the checked out upstream source code is the 396th commit past the last Git tag and its commit hash is f4d27d1 (without the g prefix, which denotes the hash is a Git commit hash)

The selective-checkout scriptlet’s logic is that it will automatically start building the development snapshot snaps (for the edge channel) if the lastest upstream release is already published to the stable channel, if you prefer only building the stable releases you may specify the --force-stable command-line option to the selective-checkout command.

1 Like

You might want to close these channels if those are unused.

OK, so that I can fully understand its logic: should I have two snapcraft.yaml files then? One to get that latest development snapshots on edge and another one with --force-stable for when the snap needs to be rebuild because some Ubuntu packages were updated with security fixes?

Thanks for all your work and support!

Apologies for the ignorance.

I would refrain from doing so, this workaround is built so that we won’t have to toggle snapcraft.yaml much in the first place. As snaps are confined already I would simply wait for the next upstream release that triggers a new snap build and supersede the last one.

The solution might be:

  1. Temporarily close the snap’s stable channel
  2. Trigger snap rebuild, due to last_stable_version does not match the version in the stable channel(none) the last stable version should be checked out and built again
  3. Promote the newly built stable release snaps from edge to stable channel
  4. Rollback to the previous snap revisions previously in the edge channel
1 Like

Due to recent snap info command output changes selective-checkout no longer can match the stable channel version, which, making it always building stable releases even when it is being promoted to the stable channel.

This issue is now fixed by version 2.0.0, we don’t ship the new version through the stage snap to ensure the new dependencies won’t break existing snaps. Now we require using the new part definition which allows the consumer to pin the part to a tagged version.

We would like to thank @tstack for reporting the issue, and @ijohnson for the helpful input regarding the API inquiry.

Cheers! :clinking_glasses:

1 Like