When a user wishes to include packages from a PPA or alternative repository it is currently not easy, and certainly not obvious how to add the repository. Some developers are likely to expect that they can add a PPA or alternative repository using the prepare step. My proposal is to formalise adding of alternative repositories into a declarative form, rather than relying on scriptlets. I suggest we add a top-level stanza as follows:
apps:
command: blah
repositories:
my-ppa:
keys:
- 'ABCD1234'
- '1234ABCD' # these are pgp key-ids
keyserver: hkp://keyserver.ubuntu.com:80 # optional - this should be the default - only accepts servers running on port 80 due to build server proxy
definition: 'deb http://my-ppa.example.com/ xenial main'
parts:
blah:
plugin: nil
build-packages:
- a-package-from-ppa
Specifically this should allow for non-debian repositories for when snapcraft supports base snaps other than the ubuntu core.
Iâm hoping it would look almost identical for rpm and eopkg, though not being familiar with those packaging systems I canât be certain.
It is for both. All parts should depend on the same set of package-repositories or else youâll potentially be building each part against different, incompatible, shared libraries and then only having one of those installed at the end (presuming that theyâre using the same filename for example).
repositories:
my-ppa:
keys: [ 'abcd1234' ] # these are still pgp keys in rpm so there is no need to alter the behaviour
# keyserver: optional, defaults to hkp://keyserver.ubuntu.com:80
definition: 'http://rpm-repo.example.com/'
The above should then create a file file at /etc/yum.repos.d/my-ppa.repo with the following:
The base idea sounds sane. We need to confirm whether the syntax would work for the other major systems we care about, and need to make sure that weâre not changing the host systemâs repositories without user consent, as this might create significant trouble there and will often be unintended.
After looking at other major systems, letâs please review the syntax before itâs final. Would likely have public-keys instead of keys, as a first note.
For us, there are some things we will always want set:
type=rpm-md. No other types are valid anymore, anyway, so this should always be passed in. DNF assumes this type if omitted, so we donât have to set it if we donât want to.
enabled=1. If youâre adding a repository, you probably want it enabled.
skip_if_unavailable=False should probably always be set
Things that user will want to set:
section header and name
gpgcheck (and the pkg_gpgcheck and repo_gpgcheck sub options) and gpgkey
baseurl, mirrorlist, or metalink. At least one of the three needs to be set.
baseurl can be a list of URLs, too (ex. baseurl=[ 'http://mirror.hmc.edu/centos/7/os/x86_64/', 'http://mirrors.liquidweb.com/CentOS/7/os/x86_64/ ]).
The variable substitution is done by DNF itself, so $releasever, $basearch, $arch, and others should work as-is.
Can these two be set based on the yaml as I have shown in my example above where my-ppa is used as the section header and name? (I camelCased it becasue I wasnât sure that hyphens were supported, but I see now that they are fine in both locations)
We probably need to include something specific to rpm for pkg_gpgcheck and repo_gpgcheck. gpgkey can be repurposed from the public-keys stanza (renamed from keys) where we list the URL or path instead of key-id for RPM systems.
I envisioned that definition would handle this. If we can set the schema to allow a single string which then gets placed into baseurl AND allow subkeys such that we can do:
repositories:
my-repo:
definition: 'http://baseurl.example.com/' # compatible with the usage for apt repo line in previous post
OR
repositories:
my-repo:
definition:
baseurl:
- 'http://baseurl.example.com/'
- ...
mirrorlist:
- mirror1
- ...
metalink: 'http://metalink.example.com' # does this also need to accept a list?
Yes. You could set them to the same value. They just need to be set.
We could probably also support retrieving GPG keys from a keyserver and just writing it to a temporary file that is used in the repository definition. Note that you can have more than one GPG key (itâs a list type).
The name definition is a bit weird for this. Iâd probably call this source-url or something a bit more directly descriptive.
Also, only baseurl is a list type, metalink and mirrorlist are single strings.
The mirrorlist is a resource that contains literally a list of mirrors. For example, the Mageia 6 core/release x86_64 mirrorlist returns the following for me:
So itâs literally a list of mirrors! Itâs a shortcut when you donât want to write out the baseurl list in the repo file or want to retrieve the remote list from elsewhere (as weâre doing now).
Incorporating this thread and cloud-init as reference material, I extended the snapcraft.json schema in a manner that I think should work for a variety of potential cases. I put up a GitHub PR for discussion purposes that have some tests to help illustrate examples:
https://github.com/snapcore/snapcraft/pull/2880
The PR adds a high-level key âpackage-managementâ, configured with
two properties:
apt for apt-specific configuration.
repositories for a generic approach to allowing additional
source repositories with a configurable/optional GPG key.
While I canât guarantee repositories will work for every scenario
for every possible OS, it appears that a common pattern is to require
a source URL and GPG key. With the provided structure to support
OS-specific configuration (e.g. apt), this schema remains extendable
to support OS-specific requirements.
Example:
package-management:
apt:
# Reconfigure primary mirror (default: http://archive.ubuntu.com/ubuntu)
primary_mirror: http://us.archive.ubuntu.com/ubuntu/
# Reconfigure security mirror (default: http://security.ubuntu.com)
security_mirror: http://us.archive.ubuntu.com/ubuntu/
repositories:
# Option 1: PPA shortcut (automatically imports PPA key from LP):
- source: "ppa:mozillateam/ppa
# Option 2: Use repository with OS-installed keys:
- source: "deb http://ppa.launchpad.net/mozillateam/ppa/ubuntu bionic main"
# Option 3: Use repository with GPG key fetched from keyserver:
- source: "deb http://ppa.launchpad.net/mozillateam/ppa/ubuntu bionic main"
gpg-key-server: keyserver.ubuntu.com
gpg-public-key-id: 0ab215679c571d1c8325275b9bdb3d89ce49ec21
# Option 4: Use repository with provided GPG key:
- source: "deb http://ppa.launchpad.net/mozillateam/ppa/ubuntu bionic main"
gpg-public-key: |
-----BEGIN PGP PUBLIC KEY BLOCK-----
<snipped>
-----END PGP PUBLIC KEY BLOCK-----
# EPEL example:
- source: http://download.fedoraproject.org/pub/epel/testing/8/$basearch
gpg-public-key: |
-----BEGIN PGP PUBLIC KEY BLOCK-----
<snipped>
-----END PGP PUBLIC KEY BLOCK-----
Selecting an Ubuntu archive mirror seems out of scope for the snapcraft.yaml file. While I might want to use an Australian mirror when testing a project build locally, I wouldnât want to force e.g. build.snapcraft.io to waste time downloading everything from Australia when it builds the project.
Repository mirrors seems like more of a âper build hostâ configuration rather than âper projectâ. What youâve got under package-management.repositories looks like a decent addition though.
@jamesh I agree that primary_mirror would rarely used to override the mirror for the recipe as a whole. But there are users that probably would use it if they are always using their own build infrastructure and mirrors. And while not strictly in the scope of âadding repositoriesâ, I wanted to include these relatively simple ones to help demonstrate OS-specific configuration (and I personally achieve a boost of productivity with the ability to modify the mirror ).
While outside the scope of this, to properly address the âper build hostâ case, I do intend to add something like a --override-yaml option to snapcraft which would let you modify the YAML for that run. It could be leveraged for local configuration that may be duplicate, or overlap with the snapcraft.yaml configuration. For example, a few that come to mind:
snapcraft --override-yaml package-management.apt.primary_mirror= "http://us.archive.ubuntu.com/ubuntu"snapcraft --override-yaml version="$SOME_CI_VARIABLE"snapcraft --override-yaml parts.project.source="git://github.com/cjp256/some-fork" --override-yaml parts.project.source-branch="experimental-feature"
Iâm sure there would be other interesting cases for something like --override-yaml tooâŚ
That said, Iâd be happy to descope the primary/secondary mirror configuration if itâs a problem.
).