Proposal: additional package sources

[daniel]

When a user wishes to include packages from a PPA or alternative repository it is currently not easy, and certainly not obvious how to add the repository. Some developers are likely to expect that they can add a PPA or alternative repository using the prepare step. My proposal is to formalise adding of alternative repositories into a declarative form, rather than relying on scriptlets. I suggest we add a top-level stanza as follows:

apps:
  command: blah

repositories:
  my-ppa:
    keys:
      - 'ABCD1234'
      - '1234ABCD' # these are pgp key-ids
    keyserver: hkp://keyserver.ubuntu.com:80 # optional - this should be the default - only accepts servers running on port 80 due to build server proxy
    definition: 'deb http://my-ppa.example.com/ xenial main'

parts:
  blah:
    plugin: nil
    build-packages:
      - a-package-from-ppa

Specifically this should allow for non-debian repositories for when snapcraft supports base snaps other than the ubuntu core.

[sergiusens]

Hi, thanks for the suggestion. Before going any further a few questions:

• How would it look and work for rpm or eopkg (if possible)?
• Is this just for build-packages?
• If this is for stage-packages too shouldn’t it be a property of the part? Is it worth making a difference?

[daniel]

How would it look and work for rpm or eopkg (if possible)?

I’m hoping it would look almost identical for rpm and eopkg, though not being familiar with those packaging systems I can’t be certain.

Is this just for build-packages?
If this is for stage-packages too shouldn’t it be a property of the part? Is it worth making a difference?

It is for both. All parts should depend on the same set of package-repositories or else you’ll potentially be building each part against different, incompatible, shared libraries and then only having one of those installed at the end (presuming that they’re using the same filename for example).

[daniel]

ok, here’s an example of rpm:

repositories:
  my-ppa:
    keys: [ 'abcd1234' ] # these are still pgp keys in rpm so there is no need to alter the behaviour
    # keyserver: optional, defaults to hkp://keyserver.ubuntu.com:80
    definition: 'http://rpm-repo.example.com/'

The above should then create a file file at /etc/yum.repos.d/my-ppa.repo with the following:

[myPpa]
name=myPpa
baseurl=http://rpm-repo.example.com/
enabled=1
gpgcheck=1

I’m unsure whether or where hyphens are allowed in the repo names so I’ve camelCased the name at the hyphen.

[niemeyer]

The base idea sounds sane. We need to confirm whether the syntax would work for the other major systems we care about, and need to make sure that we’re not changing the host system’s repositories without user consent, as this might create significant trouble there and will often be unintended.

After looking at other major systems, let’s please review the syntax before it’s final. Would likely have public-keys instead of keys, as a first note.

[daniel]

+1 on changing keys to public-keys

[Conan_Kudo]

Here’s a few example entries of RPM .repo files:

[fedora-cisco-openh264]
name=Fedora $releasever openh264 (From Cisco) - $basearch
baseurl=https://codecs.fedoraproject.org/openh264/$releasever/$basearch/
type=rpm-md
enabled=0
enabled_metadata=1
metadata_expire=14d
repo_gpgcheck=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

[mageia-x86_64]
name=Mageia $releasever - x86_64
mirrorlist=https://www.mageia.org/mirrorlist/?release=$releasever&arch=x86_64&section=core&repo=release
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Mageia
failovermethod=priority
enabled=1

[home:Pharaoh_Atem:DNF_SUSE]
name=DNF for SUSE/openSUSE (openSUSE_Leap_42.2)
enabled=1
baseurl=http://download.opensuse.org/repositories/home:/Pharaoh_Atem:/DNF_SUSE/openSUSE_Leap_42.2/
type=rpm-md
pkg_gpgcheck=1
repo_gpgcheck=1
gpgkey=http://download.opensuse.org/repositories/home:/Pharaoh_Atem:/DNF_SUSE/openSUSE_Leap_42.2//repodata/repomd.xml.key

[runner_gitlab-ci-multi-runner]
name=runner_gitlab-ci-multi-runner
baseurl=https://packages.gitlab.com/runner/gitlab-ci-multi-runner/el/7/$basearch
repo_gpgcheck=1
pkg_gpgcheck=0
enabled=1
gpgkey=https://packages.gitlab.com/runner/gitlab-ci-multi-runner/gpgkey
type=rpm-md

[Conan_Kudo]

For us, there are some things we will always want set:

• type=rpm-md. No other types are valid anymore, anyway, so this should always be passed in. DNF assumes this type if omitted, so we don’t have to set it if we don’t want to.
• enabled=1. If you’re adding a repository, you probably want it enabled.
• skip_if_unavailable=False should probably always be set

Things that user will want to set:

• section header and name
• gpgcheck (and the pkg_gpgcheck and repo_gpgcheck sub options) and gpgkey
• baseurl, mirrorlist, or metalink. At least one of the three needs to be set.
  • baseurl can be a list of URLs, too (ex. baseurl=[ 'http://mirror.hmc.edu/centos/7/os/x86_64/', 'http://mirrors.liquidweb.com/CentOS/7/os/x86_64/ ]).

The variable substitution is done by DNF itself, so $releasever, $basearch, $arch, and others should work as-is.

[daniel]

section header and name

Can these two be set based on the yaml as I have shown in my example above where my-ppa is used as the section header and name? (I camelCased it becasue I wasn’t sure that hyphens were supported, but I see now that they are fine in both locations)

gpgcheck (and the pkg_gpgcheck and repo_gpgcheck sub options) and gpgkey

We probably need to include something specific to rpm for pkg_gpgcheck and repo_gpgcheck. gpgkey can be repurposed from the public-keys stanza (renamed from keys) where we list the URL or path instead of key-id for RPM systems.

baseurl, mirrorlist, or metalink. At least one of the three needs to be set.

baseurl can be a list of URLs, too (ex. baseurl=[ ‘http://mirror.hmc.edu/centos/7/os/x86_64/’, 'http://mirrors.liquidweb.com/CentOS/7/os/x86_64/ ]).

I envisioned that definition would handle this. If we can set the schema to allow a single string which then gets placed into baseurl AND allow subkeys such that we can do:

repositories:
  my-repo:
    definition: 'http://baseurl.example.com/' # compatible with the usage for apt repo line in previous post

OR

repositories:
  my-repo:
    definition:
      baseurl:
        - 'http://baseurl.example.com/'
        - ...
      mirrorlist:
        - mirror1
        - ...
      metalink: 'http://metalink.example.com' # does this also need to accept a list?

[Conan_Kudo]

Can these two be set based on the yaml as I have shown in my example above where my-ppa is used as the section header and name? (I camelCased it becasue I wasn’t sure that hyphens were supported, but I see now that they are fine in both locations)

Yes. You could set them to the same value. They just need to be set.

We probably need to include something specific to rpm for pkg_gpgcheck and repo_gpgcheck. gpgkey can be repurposed from the public-keys stanza (renamed from keys) where we list the URL or path instead of key-id for RPM systems.

We could probably also support retrieving GPG keys from a keyserver and just writing it to a temporary file that is used in the repository definition. Note that you can have more than one GPG key (it’s a list type).

I envisioned that definition would handle this. If we can set the schema to allow a single string which then gets placed into baseurl AND allow subkeys such that we can do:

The name definition is a bit weird for this. I’d probably call this source-url or something a bit more directly descriptive.

Also, only baseurl is a list type, metalink and mirrorlist are single strings.

[daniel]

Also, only baseurl is a list type, metalink and mirrorlist are single strings.

That’s confusing. The mirrorlist attribute is not a list despite it’s name?!

[Conan_Kudo]

That’s confusing. The mirrorlist attribute is not a list despite it’s name?!

The mirrorlist is a resource that contains literally a list of mirrors. For example, the Mageia 6 core/release x86_64 mirrorlist returns the following for me:

http://mirrors.kernel.org/mageia/distrib/6/x86_64/media/core/release/
http://distro.ibiblio.org/pub/linux/distributions/mageia/distrib/6/x86_64/media/core/release/
ftp://mageia.jameswhitby.net/mageia/distrib/6/x86_64/media/core/release/
http://mirrors-usa.go-parts.com/mageia/distrib/6/x86_64/media/core/release/
http://mirror.dacentec.com/mageia/distrib/6/x86_64/media/core/release/
http://mageia.jameswhitby.net/mageia/distrib/6/x86_64/media/core/release/
http://mirror.nexcess.net/mageia/distrib/6/x86_64/media/core/release/
http://mirror.math.princeton.edu/pub/mageia/distrib/6/x86_64/media/core/release/
http://ftp.acc.umu.se/mirror/mageia/distrib/6/x86_64/media/core/release/
ftp://mageia.c3sl.ufpr.br/mageia/distrib/6/x86_64/media/core/release/

So it’s literally a list of mirrors! It’s a shortcut when you don’t want to write out the baseurl list in the repo file or want to retrieve the remote list from elsewhere (as we’re doing now).

[kyrofa]

In addition to supporting pulling keys from a keyserver by ID, we should support keyfiles.