Proposal: additional package sources

ok, here’s an example of rpm:

repositories:
  my-ppa:
    keys: [ 'abcd1234' ] # these are still pgp keys in rpm so there is no need to alter the behaviour
    # keyserver: optional, defaults to hkp://keyserver.ubuntu.com:80
    definition: 'http://rpm-repo.example.com/'

The above should then create a file file at /etc/yum.repos.d/my-ppa.repo with the following:

[myPpa]
name=myPpa
baseurl=http://rpm-repo.example.com/
enabled=1
gpgcheck=1

I’m unsure whether or where hyphens are allowed in the repo names so I’ve camelCased the name at the hyphen.

The base idea sounds sane. We need to confirm whether the syntax would work for the other major systems we care about, and need to make sure that we’re not changing the host system’s repositories without user consent, as this might create significant trouble there and will often be unintended.

After looking at other major systems, let’s please review the syntax before it’s final. Would likely have public-keys instead of keys, as a first note.

1 Like

+1 on changing keys to public-keys

Here’s a few example entries of RPM .repo files:

[fedora-cisco-openh264]
name=Fedora $releasever openh264 (From Cisco) - $basearch
baseurl=https://codecs.fedoraproject.org/openh264/$releasever/$basearch/
type=rpm-md
enabled=0
enabled_metadata=1
metadata_expire=14d
repo_gpgcheck=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False
[mageia-x86_64]
name=Mageia $releasever - x86_64
mirrorlist=https://www.mageia.org/mirrorlist/?release=$releasever&arch=x86_64&section=core&repo=release
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Mageia
failovermethod=priority
enabled=1
[home:Pharaoh_Atem:DNF_SUSE]
name=DNF for SUSE/openSUSE (openSUSE_Leap_42.2)
enabled=1
baseurl=http://download.opensuse.org/repositories/home:/Pharaoh_Atem:/DNF_SUSE/openSUSE_Leap_42.2/
type=rpm-md
pkg_gpgcheck=1
repo_gpgcheck=1
gpgkey=http://download.opensuse.org/repositories/home:/Pharaoh_Atem:/DNF_SUSE/openSUSE_Leap_42.2//repodata/repomd.xml.key
[runner_gitlab-ci-multi-runner]
name=runner_gitlab-ci-multi-runner
baseurl=https://packages.gitlab.com/runner/gitlab-ci-multi-runner/el/7/$basearch
repo_gpgcheck=1
pkg_gpgcheck=0
enabled=1
gpgkey=https://packages.gitlab.com/runner/gitlab-ci-multi-runner/gpgkey
type=rpm-md

For us, there are some things we will always want set:

  • type=rpm-md. No other types are valid anymore, anyway, so this should always be passed in. DNF assumes this type if omitted, so we don’t have to set it if we don’t want to.
  • enabled=1. If you’re adding a repository, you probably want it enabled. :slight_smile:
  • skip_if_unavailable=False should probably always be set :slight_smile:

Things that user will want to set:

  • section header and name
  • gpgcheck (and the pkg_gpgcheck and repo_gpgcheck sub options) and gpgkey
  • baseurl, mirrorlist, or metalink. At least one of the three needs to be set.
    • baseurl can be a list of URLs, too (ex. baseurl=[ 'http://mirror.hmc.edu/centos/7/os/x86_64/', 'http://mirrors.liquidweb.com/CentOS/7/os/x86_64/ ]).

The variable substitution is done by DNF itself, so $releasever, $basearch, $arch, and others should work as-is.

Can these two be set based on the yaml as I have shown in my example above where my-ppa is used as the section header and name? (I camelCased it becasue I wasn’t sure that hyphens were supported, but I see now that they are fine in both locations)

We probably need to include something specific to rpm for pkg_gpgcheck and repo_gpgcheck. gpgkey can be repurposed from the public-keys stanza (renamed from keys) where we list the URL or path instead of key-id for RPM systems.

I envisioned that definition would handle this. If we can set the schema to allow a single string which then gets placed into baseurl AND allow subkeys such that we can do:

repositories:
  my-repo:
    definition: 'http://baseurl.example.com/' # compatible with the usage for apt repo line in previous post

OR

repositories:
  my-repo:
    definition:
      baseurl:
        - 'http://baseurl.example.com/'
        - ...
      mirrorlist:
        - mirror1
        - ...
      metalink: 'http://metalink.example.com' # does this also need to accept a list?

Yes. You could set them to the same value. They just need to be set.

We could probably also support retrieving GPG keys from a keyserver and just writing it to a temporary file that is used in the repository definition. Note that you can have more than one GPG key (it’s a list type).

The name definition is a bit weird for this. I’d probably call this source-url or something a bit more directly descriptive.

Also, only baseurl is a list type, metalink and mirrorlist are single strings.

That’s confusing. The mirrorlist attribute is not a list despite it’s name?!

The mirrorlist is a resource that contains literally a list of mirrors. For example, the Mageia 6 core/release x86_64 mirrorlist returns the following for me:

http://mirrors.kernel.org/mageia/distrib/6/x86_64/media/core/release/
http://distro.ibiblio.org/pub/linux/distributions/mageia/distrib/6/x86_64/media/core/release/
ftp://mageia.jameswhitby.net/mageia/distrib/6/x86_64/media/core/release/
http://mirrors-usa.go-parts.com/mageia/distrib/6/x86_64/media/core/release/
http://mirror.dacentec.com/mageia/distrib/6/x86_64/media/core/release/
http://mageia.jameswhitby.net/mageia/distrib/6/x86_64/media/core/release/
http://mirror.nexcess.net/mageia/distrib/6/x86_64/media/core/release/
http://mirror.math.princeton.edu/pub/mageia/distrib/6/x86_64/media/core/release/
http://ftp.acc.umu.se/mirror/mageia/distrib/6/x86_64/media/core/release/
ftp://mageia.c3sl.ufpr.br/mageia/distrib/6/x86_64/media/core/release/

So it’s literally a list of mirrors! It’s a shortcut when you don’t want to write out the baseurl list in the repo file or want to retrieve the remote list from elsewhere (as we’re doing now).

In addition to supporting pulling keys from a keyserver by ID, we should support keyfiles.

So is this a thing we can use now or just an idea?

Just an idea at the moment, but it’s on the roadmap.

4 Likes

Is there any update on this? Is it still WIP?

Is there a workaround in the meantime? I am trying to add armhf sources but I can’t get it to work.

Incorporating this thread and cloud-init as reference material, I extended the snapcraft.json schema in a manner that I think should work for a variety of potential cases. I put up a GitHub PR for discussion purposes that have some tests to help illustrate examples:

The PR adds a high-level key ‘package-management’, configured with
two properties:

  • apt for apt-specific configuration.

  • repositories for a generic approach to allowing additional
    source repositories with a configurable/optional GPG key.

While I can’t guarantee repositories will work for every scenario
for every possible OS, it appears that a common pattern is to require
a source URL and GPG key. With the provided structure to support
OS-specific configuration (e.g. apt), this schema remains extendable
to support OS-specific requirements.

Example:

package-management:

  apt:

    # Reconfigure primary mirror (default: http://archive.ubuntu.com/ubuntu)
    primary_mirror: http://us.archive.ubuntu.com/ubuntu/

    # Reconfigure security mirror (default: http://security.ubuntu.com)
    security_mirror: http://us.archive.ubuntu.com/ubuntu/

  repositories:

    # Option 1: PPA shortcut (automatically imports PPA key from LP):
    - source: "ppa:mozillateam/ppa

    # Option 2: Use repository with OS-installed keys:
    - source: "deb http://ppa.launchpad.net/mozillateam/ppa/ubuntu bionic main"

    # Option 3: Use repository with GPG key fetched from keyserver:
    - source: "deb http://ppa.launchpad.net/mozillateam/ppa/ubuntu bionic main"
      gpg-key-server: keyserver.ubuntu.com
      gpg-public-key-id: 0ab215679c571d1c8325275b9bdb3d89ce49ec21

    # Option 4: Use repository with provided GPG key:
    - source: "deb http://ppa.launchpad.net/mozillateam/ppa/ubuntu bionic main"
      gpg-public-key: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        <snipped>
        -----END PGP PUBLIC KEY BLOCK-----

    # EPEL example:
    - source: http://download.fedoraproject.org/pub/epel/testing/8/$basearch
      gpg-public-key: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        <snipped>
        -----END PGP PUBLIC KEY BLOCK-----

Thoughts? :smiley:

1 Like

Selecting an Ubuntu archive mirror seems out of scope for the snapcraft.yaml file. While I might want to use an Australian mirror when testing a project build locally, I wouldn’t want to force e.g. build.snapcraft.io to waste time downloading everything from Australia when it builds the project.

Repository mirrors seems like more of a “per build host” configuration rather than “per project”. What you’ve got under package-management.repositories looks like a decent addition though.

2 Likes

@jamesh I agree that primary_mirror would rarely used to override the mirror for the recipe as a whole. But there are users that probably would use it if they are always using their own build infrastructure and mirrors. And while not strictly in the scope of “adding repositories”, I wanted to include these relatively simple ones to help demonstrate OS-specific configuration (and I personally achieve a boost of productivity with the ability to modify the mirror :wink:).

While outside the scope of this, to properly address the “per build host” case, I do intend to add something like a --override-yaml option to snapcraft which would let you modify the YAML for that run. It could be leveraged for local configuration that may be duplicate, or overlap with the snapcraft.yaml configuration. For example, a few that come to mind:
snapcraft --override-yaml package-management.apt.primary_mirror= "http://us.archive.ubuntu.com/ubuntu"
snapcraft --override-yaml version="$SOME_CI_VARIABLE"
snapcraft --override-yaml parts.project.source="git://github.com/cjp256/some-fork" --override-yaml parts.project.source-branch="experimental-feature"

I’m sure there would be other interesting cases for something like --override-yaml too… :slight_smile:

That said, I’d be happy to descope the primary/secondary mirror configuration if it’s a problem.

I agree with @jamesh, mirrors seem to be closer to a per host solution and done through configuration.

I am a bit worried about overriding anything with “security” in its name, @jdstrand what are your thoughts?

I guess I don’t see any reason why mirror selection should be tied to project description. If Snapcraft had the ability to select a local mirror, I’d want it to apply to all projects I build locally. If I had to edit the yaml or remember to always pass long command line arguments to snapcraft, I’d probably ignore the feature even though it would probably save me time.

The repositories hierarchy you’ve proposed is quite different though. It’s not just an optimisation, and really is part of a project’s configuration: the project will likely fail to build if you omit the PPA, and other projects might misbuild if you injected the PPA and they weren’t expecting it.

I agree with your point and I will be removing the mirrors :slight_smile: Thank you for the feedback!