Call for testing: configurable apt repositories in snapcraft.yaml

NOTE: This is an experimental feature and is subject to change!

What’s this about?

To improve the snapcraft workflow for snaps requiring non-standard repositories (such as PPAs, additional repository components, etc.), apt repository configuration via snapcraft.yaml is now available experimentally! :smile:

The syntax developed here tried to balance:

Other interesting syntax includes:

How do I test it?

It’s currently available in snapcraft’s edge/pr-2911 channel!

To update snapcraft to this experimental build, use:

sudo snap refresh snapcraft --channel=edge/pr-2911

GitHub PR: https://github.com/snapcore/snapcraft/pull/2911

How do I configure it?

package-management

This feature adds a high-level key package-management to snapcraft.yaml that enables users to configure additional repositories & components. Specifically, the scope of package-management is for anything affecting the behavior and availability of:

  • build-packages
  • stage-packages
  • build-snaps
  • stage-snaps
  • python-packages (related note: python-packages should probably be decoupled into build & stage)

package-management.repositories

Under package-management is the repositories property, a list of repository configs that you would like snapcraft to install and configure. Once configured, packages provided by these repositories will become available via stage-packages and build-packages.

Repositories have the following properties:

  • source
  • gpg-public-key
  • gpg-public-key-id
  • gpg-key-server

repository.source (required)

String specifying source repository URI components. Supports the following syntax supported by add-apt-repository:

  1. short-form PPAs, e.g.:

    • source: ppa:cjp256/test-ppa
    • source: ppa:mozillateam/ppa
  2. traditional deb repository line for sources.list, with format deb <url> <suite> <components...>, e.g.:

  3. repository URI, with suite assumed to match host (such as “bionic” for core18 / 18.04 build environment) and optional component defaulting to “main”, e.g.:

repository.gpg-public-key

String specifying full ASCII-armored public key. Must include the PGP header/footer, e.g.:

gpg-public-key: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        <snipped>
        -----END PGP PUBLIC KEY BLOCK-----

repository.gpg-public-key-id

String specifying the GPG public key identifier/fingerpint to fetch from a key server, e.g.:

gpg-public-key-id: 590CA3D8E4826565BE3200526A634116E00F4C82

repository.gpg-key-server

String specifying the key server to use for fetching GPG key (specified by gpg-public-key-id). Defaults to keyserver.ubuntu.com. E.g.:

gpg-public-key-id: 590CA3D8E4826565BE3200526A634116E00F4C82
gpg-key-server: keyserver.ubuntu.com

TLDR; Examples

Here are some examples to help get you started!

package-management:

  repositories:

    # Option 1: PPA shortcut (automatically imports PPA key from LP):
    - source: ppa:cjp256/test-ppa

    # Option 2: Enable apt repository via "deb" sources.list format using OS-installed keys:
    - source: deb http://archive.canonical.com/ubuntu bionic partner

    # Option 3: Enable repository with specified component using OS-installed keys:
    - source: http://archive.canonical.com/ubuntu partner

    # Option 4: Use repository with GPG key fetched from keyserver:
    - source: http://ppa.launchpad.net/cjp256/test-ppa/ubuntu
      gpg-public-key-id: 6A634116E00F4C82
      gpg-key-server: keyserver.ubuntu.com

    # Option 5: Use repository with provided GPG key:
    - source: http://ppa.launchpad.net/cjp256/test-ppa/ubuntu
      gpg-public-key: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        mQINBF4y3MIBEAC6rYxrDtleK/OPftHKySm5nN8OIc+tBFnGr9pz4eu2CRukFxXc
        Ap07pj5EkRbzI60BeGVK/spLFtrU7PPxBXJ3DiaxEaWmOUW/7MdukSWTVk9EgEAZ
        dwKjbwNrrxmCif8RJzK9JFz1IqGqY/uzR7X90J2Wn4/+FvWNofE2kCQeGEpDQtmF
        bpcBdHZmcru8g/t3KQy11i4GJ5gLXqFdDF3T3YcVgLyoOx910VK0Y41Rsux7PwlR
        PmF3doZ+fjLBYVBo0l5ek56dc7p1Z6RL30IHgbwdSbDcYJ8d8dXR+gFUuB4xSYTk
        XRD2Uzc+jDcwXEaTGFvxQNxIXW4ELwwB5ceqbt/l1w4Yo+pQrSKDjOeG80kwRWsw
        bGnaAEa8ZKMHvQdr6wIGwiD6LfNAKGtr5xw4O7VzalF6KuzmrRt9RQc/bkso3DQV
        tCC/nkI55gXEM6zH3NiGQiz47SrjENXWOwJvGrBT36eHCsLZ984SwRHCOqubEAwE
        G7mkRcew/YC2xe6ic7ZMxVQ3DkmHKojO3qoS73KbuRgp6xwcBXxtPN2k/rAEmVTz
        aXIPp46mXbwAfcuo5gFRknGzgyOP8cG5KMbl/AGJjRRfiWlIC5RmCcguGsgC5p0n
        4cPX7+u0J9+pjyPdOntSm5WS1+2HYkd7vS0Qmio5/e7Sw1blCai1mxc7rQARAQAB
        tCFMYXVuY2hwYWQgUFBBIGZvciBDaHJpcyBQYXR0ZXJzb26JAjgEEwECACIFAl4y
        3MICGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEGpjQRbgD0yCanYP/001
        tx9sgjfGCsA2WNbHAcoqN5Di1dIo4js5inFHWhFFDte4qY1uAx3lKxpq5iyqzwtB
        rN/fbp3URd+ZqsVQ8IkfPf+lzMubf9YnKxbuBeXU+y5Kj8pmoL6sCpieobk9+3Go
        rtAkrlmFw7GmpPeCzaHhr5YzOO82T75EEb2a6FYduPv+ONGJqk8P2f20CzlK84ze
        G6ApSD7BdZ70iUvC0oy6AJoU8uSgoOfMdTIgT7NWXLfqzOIo+Fy5dlnmGS09y132
        hpjqvx3VCFdGH+NSfcmEVCTcLnqoPqD6InXoROqi30c2D4PmX2h1Endac6BF9uCB
        xdFBb8SKZ7mJ/dydAEV9KbUr+mQCAG5JO0+GnxgtKanHjMmDcF9BtmHMZsKDjILs
        7iyiWEsrk+QsUpV5gxOdpOFvL2FfSVTTYhmKgCI9h+KqesAPnJwIhCI5Gz94MaYo
        V8T+4ZjX36DNh5f7gzWLvajvKznOXGJkX6TgbqFo/wrijK75DPmuuR7m2Xa4AlAT
        J0JZFdHWXuJnYI3vSZ2OfgHJKsLePheLFtJvpFJTPC0nOFwWgg2JH//UDbEIk5md
        nLQl+agC5z32g+Qet+D/5y+xamJP/5HKgix8RhggPSIsaMi8ezNrF08jyollwDFk
        c24ABjaDjM0uWKllldFtmpnW7V0tOFxm+RIA6dCM
        =ulGo
        -----END PGP PUBLIC KEY BLOCK-----

As always, feedback and suggestions are appreciated! If you test this out and discover any bugs, please let us know. If you test this out and it works great, that’s good to know too :grin:

In-progress Discussion Points / Open Questions

These considerations are not reflected in the PR

  1. Rename repositories to apt.repositories based on @niemeyer’s feedback that sources are likely to be apt-specific. It is agreed the safest option would be to just make it apt-specific decouple the schema for future bases. Though it does require an additional layer of nesting to maintain a list without preventing future expansion of other configuration keys.

    Previously it was assumed that the repositories would be parsed in the context of the base in which they are building.

    For reference, the original EPEL example in a hypothetical future where base: centos exists:

    - source: http://download.fedoraproject.org/pub/epel/testing/8/$basearch
      gpg-public-key: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        <snipped>
        -----END PGP PUBLIC KEY BLOCK-----
  1. Rename source to ?? based on @niemeyer’s feedback that source may be confused with other context of “source”.

    Data points to consider:

    • source is used by cloud-init
    • repo is used by ansible
    • source is used in snapcraft’s part configuration to point to a source repository
  2. Remove package-mangement and push keys to higher level? @niemeyer

    Some hypothetical keys to consider to support future configuration:

package-mangement:
  apt:
    repositories:
      - ...
      - ...
  pip:
    index-url: <index-url>
    extra-index-urls:
    - <extra-index-url>
    - <extra-index-url2>
  snap:
    store-id: <UBUNTU_STORE_ID>

Being able to configure the snap store ID is part of a story to improve the snapcraft builds for brand store users. Bug #1833220 “stage-snaps not usable with a brand store” : Bugs : Snapcraft

7 Likes

Rename repositories to apt.repositories based on @niemeyer’s feedback that sources are likely to be apt-specific.

It’s not clear to me how the latter implies the former. Most likely every snapcraft file will end up defining a single one of those, which means having a deeply nested package-management.apt.repositories setting, in which both top-levels will likely be alone. Seems a bit unfriendly to write and to read.

I had in mind something somewhat simpler, such as:

package-repositories:
    - type: apt
      ...

but maybe I’m missing some requirement of the problem space.

Thank you for the feedback @niemeyer

I can certainly do this, but I am reluctant to for two reasons:

  1. Schema validation. Overloading types can be done, but it does have the effect of making the json schema more complex. In my experience, this complexity often results in validation errors becoming confusing and snapcraft must compensate by moving certain checks into code. This also tends to affect editors with auto-completion & linting, which tend to fare best with simpler schema.

    As a data point, one of my first tasks on snapcraft was to convert the checks currently being done in python code for source-<properties> to json schema. In general, it seems fairly straight forward at the face of it, but due to the limitations of json schema I ultimately had to give it up. The only way to do was to auto-generate the schema to handle a large number of permutations, which wasn’t going work for multiple reasons:

    • performance due to incredibly large schema
    • validation errors quickly became useless due to a complex set of (sometimes nested) oneOf configurations

    How we get around this today:

    • source do their checks in python code for late-validation when pulling.
    • plugins work by composing their delta schema for late-validation, allowing for additionalProperties: true on the initial validation pass for part object.

    If these objects were decoupled, the schema could be implemented in a more readable, self-documenting manner with most of the validation happening up-front.

  2. Extensibility. Today we may only care about adding repositories, but I think it wise to consider potential future options, even if we haven’t committed to, or even imagined, them yet. Examples include allowing for configuration of package-manager specific options, or modifying the default source repositories (as opposed to strictly supplementing), etc. I think having package-management as the high-level key allows us to group these options together. I’d rather we not paint us into that corner, and would recommend at least one high-level key for grouping.

However, these are personal preferences and I agree we can check the box with your proposed schema. If you are still unconvinced by my reasoning, I will be happy to adapt the PR and documentation according to your suggested approach. :smile: Just let me know!

For point 1, I believe we covered that during our conversation in the sprint: the file format of snapcraft.yaml is already not context free, because parts have a plugin which define which attributes are supported. The same logic used to validate these can be used to validate repositories considering their type.

I’m also curious about where that requirement comes from. I would focus on having great error reporting, rather than having JSON schema as a goal.

For the second point, the suggested sketch seems to present no lack of extensibility in comparison to your original proposal. Can you clarify?

Finally, let me raise a meta-concern here: the key point I raised was oriented towards the user experience someone would have while typing and reading those options. The replies to that point didn’t seem to address that, which was having something nice for the user to read and write. It’s naturally okay that the end result is not exactly what I’m proposing, but we do need to keep that kind of issue in mind while designing this and other features in snapcraft.

Agreed.

We certainly agree with this, I am sorry if I gave you the impression otherwise. If we don’t get the desired error reporting by using JSON schema, we do it other ways that are viable. In most cases, the validation via schema is perfectly fine. I would personally always steer towards defining a schema that is readable and best supports error checking, though I wouldn’t say it’s a requirement. But I won’t hammer on that point any futher :wink:

I assume you are referring to:

I interpret package-repositories as a top-level key, defined as a sequence of <Repository> objects…

A couple of concrete scenarios:

  • Scenario #1: add a toggles for apt configuration options (e.g. Apt::Install-Recommends, Acquire::AllowInsecureRepositories, etc.) that cannot apply to an individual repository.

    Maybe we could add a type: apt-config-option ?

package-repositories:
  - type: apt-config-option
    apt-option: Apt::Install-Recommends=1
  - type: apt-config-option
    apt-option: Acquire::AllowInsecureRepositories=1
  - type: apt
    source: deb http://my-insecure-repository
  • Scenario #2: remove default component (e.g. multiverse, restricted)

    Maybe add a type: apt-remove-component ?

package-repositories:
  - type: apt-remove-component
    component: multiverse

If we do some upfront organization with a couple of nested keys, we could have a strongly typed schema without overloading objects to implement the above scenarios, e.g.:

package-management:
  apt:
    repositories:
      - source: ppa:cjp256/projectrepo
      - source: ppa:certbot/cerbot
    override-sources-list: |
      deb http://archive.ubuntu.com/ubuntu bionic main restricted universe
      deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe
      deb http://security.ubuntu.com/ubuntu bionic-security main restricted universe
   options:
      - Apt::Install-Recommends=1
  pip:
    index-url: <index-url>
    extra-index-urls:
    - <extra-index-url>
    - <extra-index-url2>
  snap:
    store-id: <UBUNTU_STORE_ID>

We could reduce nesting in the previous example by promoting apt, pip, and snap to top-level keys:

apt-configuration:
  repositories:
    - source: ppa:cjp256/projectrepo
    - source: ppa:certbot/cerbot
  override-default-sources: |
    deb http://archive.ubuntu.com/ubuntu bionic main restricted universe
    deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe
    deb http://security.ubuntu.com/ubuntu bionic-security main restricted universe
  options:
    - Apt::Install-Recommends=1
pip-configuration:
  index-url: <index-url>
  extra-index-urls:
    - <extra-index-url>
    - <extra-index-url2>
snap-configuration:
  store-id: <UBUNTU_STORE_ID>

Would you have any concerns with this approach?

@wimpress pointed out it may be necessary to incorporate advanced-grammar for unusual cases.

An example of that could look something like:

apt-configuration:
  repositories:
    - on amd64:
      - source: deb http://my.org/project-repo/ubu-amd64 bionic main
    - on arm64:
      - source: deb http://my.org/project-repo/ubu-arm64 bionic main

Generally adding both repositories shouldn’t hurt due to natural architecture selection, but there may be a conflict in all arch packages.

I do, and still the same concerns mentioned earlier: it’s deeply nested, hard to read at a glance, which also means it’s hard to remember how to write.

Also, it’s probably an example, but we don’t want this either:

Yes, these are not package repositories and that syntax is not great. If we start building a procedural language onto the syntax we may as well drop the idea altogether and tell people to go hack on the system. I’d rather have a nice declarative way to have repositories, though.

As for the specific examples, they’re not great and although I know they are just examples I’ll raise it to avoid future issues: it’s strange to ask for recommendations to be installed. If there’s an actual dependency in the list of recommendations that is required, it should be listed explicitly instead of installing every recommendation and hoping that the requirement is (and remains) in that list. Also, if you ask for every signature to be ignored when building a snap, please let me know which snap that is so I can avoid it. :slight_smile:

This indeed doesn’t feel polished. Sometimes it’s raw text, sometimes it’s a map of lists of maps of text. Sometimes it’s a list of APT-familiar strings with non-YAML equal assignments… it’s all over the place.

We need something that looks simple and solid. Please try another pass on it and if you can’t come up with something you’re happy with, let’s reserve some time with more bandwidth and brainstorm on it together.