Preview: Snapcraft remote build

An often requested feature for Snapcraft is the ability to build snaps for foreign architectures directly from the command line. This is because the currently available web-based tools aren’t appropriate to every situation, and not everyone has the hardware needed to build native ARM binaries (for example).

This preview of remote building provides this functionality, allowing you to effortlessly target multiple architectures available in the Snap Store from any system capable of running Snapcraft. You can test it now from the beta/remote-build branch of Snapcraft.

This is what it looks like in action:

$ snapcraft remote-build --arch=arm64,armhf
snapcraft remote-build is offered as a preview. Authentication and transport mechanisms will change in future releases. Use with caution in scripts.
All data sent to remote builders is public. Are you sure you want to continue? [y/N]: y
Sending data to remote builder...
Building package for arm64 and armhf. This may take some time to finish.
If interrupted, resume with: 'snapcraft remote-build --recover 49111050'
Snapped figlet_2.2.5+git.b61ba79_arm64.snap
Snapped figlet_2.2.5+git.b61ba79_armhf.snap
Build complete.

As this release is a preview, a few extra steps are currently required to set up your environment, and there’s one important caveat: the project code sent to remote build will be public on Launchpad. Your code can be removed afterwards, but for now, remote building should only be used on projects where public availability won’t cause an issue.

Setting up the system

You will need a Launchpad account with a registered SSH public key to use remote build with this preview release.

If you don’t have an account, head over to Launchpad and click ‘Register’. You will then be able to login and add a pre-existing SSH public key, or follow the instructions on Launchpad’s Creating an SSH key page to create one as needed.

When you have a Launchpad account and a properly associated SSH public key, you can use snapcraft remote-build to build a project.

Using remote build

Snapcraft will first need to authorize access to the Launchpad account when remote build is run for the first time. It does this by opening a web page when running the snapcraft command:

$ snapcraft remote-build --user <your launchpad id>
snapcraft remote-build is offered as a preview. Authentication and transport mechanisms will change in future releases. Use with caution in scripts.
All data sent to remote builders is public. Are you sure you want to continue? [y/N]: y
Sending data to remote builder...
The authorization page: (https://launchpad.net/+authorize-token?oauth_token=...&allow_permission=DESKTOP_INTEGRATION)
should be opening in your browser. Use your browser to authorize
this program to access Launchpad on your behalf.
Waiting to hear from Launchpad about your decision...

With Snapcraft properly authorized, the project is dispatched for remote build. Snapcraft will wait for the build to complete before retrieving the resultant snaps. The build time depends on the target architecture, the package size, and the availability of builder back-ends.

Building package. This may take some time to finish.
If interrupted, resume with: 'snapcraft remote-build --recover 49112457'
Snapped my-snap-name_0.1_amd64.snap
Snapped my-snap-name_0.1_i386.snap
Build complete.

If an architecture is not specified, snapcraft remote-build will default to amd64 and i386. Specific architectures can be listed in the snapcraft.yaml file, or passed as an argument to the --arch command line option. To build for all available architectures, use --arch=all.

When the code is on a local git repository

If the prospective build code is in a git repository, the --git option can be used to send the current branch HEAD instead of your working tree. This allows Snapcraft to correctly use git hashes when specifying version: 'git' in the snapcraft.yaml. Any uncommitted changes will be ignored.

Launchpad credentials

Integration with other systems, such as continuous integration, may require setting Launchpad credentials. These are stored in $HOME/.local/share/snapcraft/launchpad/credentials.

Future improvements

As mentioned in the command output, authentication and transport mechanisms will be changed for the final release and won’t require SSH keys to be registered on Launchpad.

9 Likes

Worth noting you can keep an eye on the build in a browser by visiting launchpad.net -> login -> click your name in the top right (or visit launchpad . net / ~your_lp_id ) -> scroll down and click “View snap packages”. The build will be at the top of the list. Click through to find the build logs.

e.g.

Click through to see the status…

3 Likes

https://launchpad.net/~/+snaps

Or just click there, which will Do The Right Thing :tm:

3 Likes

I encounter several errors that aren’t caused by build failure:

Priming desktop-qt5 
Priming launchers 
Priming patching 
Priming selective-checkout 
Priming main 
sed --file /build/snapcraft-fe18a192c056412982c7e9eba40d9ca0/stage/patching/patch-desktop-entries.sed --in-place share/applications/featherpad.desktop 
Priming scriptlets 
Snapping 'featherpad' ...

Snapped featherpad_0.10.0+pkg-8c14_ppc64el.snap
Revoking proxy token...
Failed to gather results: execv() arg 2 must contain only stringsRUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=bionic --arch=ppc64el SNAPBUILD-580422
Scanning for processes to kill in build SNAPBUILD-580422

Log: https://launchpad.net/~buo-ren-lin/+snap/snapcraft-fe18a192c056412982c7e9eba40d9ca0/+build/580422/+files/buildlog_snap_ubuntu_bionic_ppc64el_snapcraft-fe18a192c056412982c7e9eba40d9ca0_BUILDING.txt.gz

Can anyone check it out? /cc @cjwatson

Please file this as a launchpad-buildd bug, and we’ll take it from there.

2 Likes

Issue filed at https://bugs.launchpad.net/launchpad-buildd/+bug/1832072, apologies for any nuisance I might have caused.

1 Like

This works quite well, thanks @cmatsuoka !
An improvement that would make this nicer to use is to allow --arch=arm64 for example even when the architectures key is specified. In this case I would expect snapcraft to ensure that the specified architectures are listed in architectures as build-on, and then only run that build. As it is, I only need to build one specific architecture, arm64, remotely, but I currently need to launch unnecessary architectures (i.e. amd64) on launchpad.

The architecture constraints are due to the way launchpad buildd behaves when it is defined in snapcraft.yaml
We can certainly work it out, but it requires work on both ends.

Is this feature still available in the beta/remote-build channel? Currently got 3.7 installed but can’t seem to call it:

$ snapcraft remote-build --arch=amd64
Usage: snapcraft [OPTIONS] COMMAND [ARGS]...

Error: No such command "remote-build".

Still a beta feature

2 Likes

I don’t see this option in 3.8+git15.g88b13cb (from beta/remote-build)?

This functionality has been removed for one that just does the right thing.

The overall idea looks interesting, is there a way to keep the source code private ? We have a close-source project that we would like to publish for all arches but can’t because build.snapcraft.io does not support that.

At this time, we do not yet have support for private projects, sorry!

It’s on our radar, but I don’t expect it to be available in the near future. Though if someone knows differently, feel free to chime in :slight_smile:

Is this option still available? It isn’t listed under snapcraft remote-build --help

If so, that’s useful. But if I want to build my working tree rather than HEAD (with un-staged or uncommited changes), is there a way to tell it to exclude some files?

I’m thinking particularly about node_modules which is very large and gets packaged and uploaded unnecessarily. I note that .snap files don’t get uploaded (clever), so I wondered if it’s possible to exclude other files, like with .gitignore?

It existed in the feature preview, but the interface was changed for the stable release. @cjp256, is there a way to do what @benfrancis is describing with the current release?

1 Like

snapcraft remote-build will tarball up your sources, including the .git directory, if present. --git is no longer required.

Unfortunately, there is no filtering - if you don’t want to include certain files, I’d suggest (re)moving them prior to the build. This matches the behavior for non-remote snapcraft operation when pointing to local sources.

@benfrancis If you would like, consider filing a feature request (https://bugs.launchpad.net/snapcraft/+filebug). Maybe we can support a --apply-gitignore or similar flag in the future.

1 Like

It appears that remote-build currently clones HEAD, which isn’t what I thought it used to do. How do I get it to build my working tree?

EDIT: It doesn’t even clone the HEAD of the current branch, it appears to clone the master branch. Even when I specify --package-all-sources

Blockquote[quote=“benfrancis, post:18, topic:11541”]
It appears that remote-build currently clones HEAD
[/quote]

Is your project source-type: git but referring to a local git repo (e.g. source: .)? If that’s the case, it will tar up your working directory. If it’s a remote git repo, it should honor source-branch|tag|commit

Ah OK, thank you. It was set to a remote GitHub URL.

Is it my imagination or did this behaviour change? I don’t remember having to modify snapcraft.yaml to get it to package my local working tree before. If so, when did it change?