Cross compile snap on amd64 to arm

Hello,
I’ve a snap for arm and building it with snapcraft (classic ubuntu) on a rpi (arm) is working absolutely fine. Though building is taking a while on rpi and I’d also like to automate my build. That’s the reason why I’d like to build my arm snap on amd64.
I’ve studied the different post in the forum and the documentation. It sounds easy by just defining the architecture and running snapcraft. Though when I do this, I get a snap which is marked as arm (…_armhf.snap) but it is not running and contains x86_64 packages. Also during the build process I just see deb package updates againsta amd64 and not arm.

I am not really sure what the right approach is over here. Did I miss something? Do I need to mark all my stage-packages, like described in here: Stage-Packages with --target-arch? Shouldn’t snapcraft pick this up automatically.

Here is a bit of my snapcraft.yaml:

...
base: core
architectures:
  - build-on: amd64
    run-on: armhf
grade: stable
confinement: strict
...
parts:
  some-part:
    plugin: python
    python-version: python3
    source: .
    stage-packages:
      - libdbus-1-dev
      - libglib2.0-dev
      - libc6
    build-packages:
      - gcc
      - gfortran
      - python-dev
      - libblas3
      - liblapack3
      - libopenblas-dev
      - liblapack-dev
      - cython
...
another-part:
    source: ./bla
    plugin: autotools
    override-build: |
      snapcraftctl build
      sudo ldconfig
    configflags:
      - --disable-root-check
    build-packages:
      - autoconf
      - automake
      - libtool
      - libcppunit-dev
      - libcppunit-1.13-0v5
      - uuid-dev
      - pkg-config

I have run snapcraft (3.3) on Ubuntu 18.04 and I want to build for arm Ubuntu Core 16 (for rpi).

Would be great if you could advice me on what’s the best approach here.

Thanks a lot and best regards,
Florian

1 Like

I have a similar issue.

I’m using the go plugin to build a simple Go executable.

I’m building the snap locally on an Ubuntu 16.04 server, with the latest versions of snapd (2.38) and snapcraft (3.4, installed as snap), so the build process is launching a VM with this “multipass” snap. I transfer the resulting example_0.1.0_all.snap to an ARM server, install it, and get:

/snap/example/x1/command-example.wrapper: 2: exec: example: Exec format error

It’s the same with different snapcraft.yaml configurations:

architectures:
  - all

And:

architectures:
  - build-on: amd64
  - run-on: all

Looking at the go plugin: https://github.com/snapcore/snapcraft/blob/master/snapcraft/plugins/go.py
There’s this:

      go_archs = {"armhf": "arm", "i386": "386", "ppc64el": "ppc64le"}
      env["GOARCH"] = go_archs.get(self.project.deb_arch, self.project.deb_arch)
      if self.project.deb_arch == "armhf":
          env["GOARM"] = "7"
return env

Which doesn’t look like it’s iterating through the architectures. But maybe the plugin isn’t supposed to do that, but snapcraft?

The plugin need to support multi arch build. Also check out the --target-arch command option in the snapcraft build --help command’s output.

That didn’t work. I used snapcraft build --target-arch armhf. Logs:

Pulling example
Cloning into '/root/parts/example/src'...
remote: Enumerating objects: 43, done.
remote: Counting objects: 100% (43/43), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 43 (delta 2), reused 27 (delta 1), pack-reused 0
Unpacking objects: 100% (43/43), done.
Please consider setting `go-importpath` for the 'example' part
go get -t -d ./example/...
Building example
Please consider setting `go-importpath` for the 'example' part
go build -o /root/parts/example/go/bin/example example

It stopped right there and didn’t create any *.snap file at all.

Also: Is there a list of valid architectures? Neither https://docs.snapcraft.io/snapcraft-top-level-metadata/8334 nor https://docs.snapcraft.io/architectures/4972 contains one.

Also: Let’s say we get this to work, how do I get the snapcraft build service to cross compile with “–target-architecture”?

Ok I think using the additional build command instead of plain snapcraft lead to the above output and it’s not erroneous. But then using snapcraft --target-arch armhf lead to an amd64 snap being built instead of an arm one:

Setting target machine to 'armhf'
Using 'snap/snapcraft.yaml': Project assets will be searched for from the 'snap' directory.
Launching a VM.
Using 'snap/snapcraft.yaml': Project assets will be searched for from the 'snap' directory.
Skipping pull example (already ran)
Skipping build example (already ran)
Staging example
Priming example
Snapping 'example' /
Snapped example_0.1.0_amd64.snap

Is this also the error of the go plugin?

My other questions also still remain: Which architectures are valid? And how to cross compile on the snapcraft build service?

Also, snapcraft clean seems to have issues now. Previously it launched the VM managed by multipass and cleaned the project so that running snapcraft would lead to a new build (without the clean running that command only prints that the snap was already built, even when changing the snapcraft.yaml). But now snapcraft clean doesn’t do anything (not even printing a log) and snapcraft clean example (the name of the “part”) leads to an error each time:

Sorry, an error occurred in Snapcraft.
[… reporting traceback …]
An error occurred when trying to execute ‘sudo -i env SNAPCRAFT_HAS_TTY=True snapcraft clean example’ with ‘multipass’: returned exit code 1.

So now I have to snap remove multipass && snapcraft

I don’t believe the Python plugin supports cross compilation. You shouldn’t have any issues with the Go plugin however.

you want to use run-on: armhf instead of “all” in your architectures: field to make the snap actually say “armhf” in the filename …

you can combine multiple arches in the run-on: statement to run on others too (in that case the value from --target-arch actually defines the final filename) …

2 Likes

I used the following at one point:

architectures:
  - build-on: amd64
  - build-on: i386
  - build-on: armhf

And because the docs say “The default value for run-on is the value of build-on” I thought that should work.

But now I did exactly what you said (but added a build-on value because otherwise snapcraft complains about the yaml schema:

architectures:
  - build-on: all
    run-on: [amd64, i386, armhf]

I didn’t use --target-arch when building.

The result was a example_0.1.0_multi.snap. The “multi” sounded promising, so I transferred the file to an ARM machine, but got:

error: cannot install snap file: snap “example” supported architectures (amd64, i386, armhf) are incompatible with this system (arm64)

My bad, armhf != arm64.

Ok changed the snapcraft.yaml to include arm64. => The file got built successfully! So I transferred it again. This time no error occurred during installation, nice.

But when running:

/snap/example/x1/command-example.wrapper: 2: exec: example: Exec format error

Why? The previous “multi” snap seemed to be built for three architectures and now I additionally included arm64.

Is it because of the go plugin this time? I.e.:

go_archs = {"armhf": "arm", "i386": "386", "ppc64el": "ppc64le"}
env["GOARCH"] = go_archs.get(self.project.deb_arch, self.project.deb_arch)

Ok, one last try for now: With --target-arch…:

$ snapcraft clean example
$ snapcraft build example --target-arch arm64
$ snapcraft --target-arch arm64
...
Snapped example_0.1.0_multi.snap

Why “multi” this time? You said:

in that case the value from --target-arch actually defines the final filename

Well, but maybe it still works on the ARM machine this time…

/snap/example/x1/command-example.wrapper: 2: exec: example: Exec format error

:frowning_face:

Can anyone please provide an example of a “hello world” in Go with a cross compiling snapcraft.yaml and the appropriate snapcraft commands?

Really, I’ve tried so many combinations, none of them are 100% clear from the documentation, and none seem to work. I might be doing something wrong here, but I’m probably not the only one having difficulties, and maybe others trying out snapcraft aren’t as persistent and asking in the forum, but moving to a different solution with a “not production-ready yet” in mind. That’s really bad because I see a lot of potential here and I would love to see this working.

PS:

Not even building ON my ARM machine works:

$ snapcraft
Using 'snap/snapcraft.yaml': Project assets will be searched for from the 'snap' directory.
Support for 'multipass' needs to be set up. Would you like to do that it now? [y/N]: y
snapd is not logged in, snap install commands will use sudo
sudo: unable to resolve host <REDACTED>
multipass (beta) 0.6.1 from Canonical✓ installed
Waiting for multipass...
Launching a VM.
Starting snapcraft-example -[2019-04-29T19:10:01.163] [error] [snapcraft-example] process error occurred FailedToStart
launch failed: failed to start qemu instance
An error occurred with the instance when trying to launch with 'multipass': returned exit code 2.
Ensure that 'multipass' is setup correctly and try again.
1 Like

I think there’s 2 compounding issues here you’re running into.

  1. the new snapcraft 3.X doesn’t support cross-compiling the way that the old snapcraft 2.X did.
  2. the new architectures spec doesn’t support both a cross-compiling specification for armhf and a native compiling specification for armhf

for 1, the issue is that even when you specify snapcraft --target-arch=armhf the env var SNAPCRAFT_ARCH_TRIPLET when snapcraft actually runs the build is still the host var. This is a bug with snapcraft and if there’s not already a LP bug I will file one

For 2, the issue is with the design of the architectures key in that there’s no way to specify that a given snapcraft.yaml when run on a particular host architecture could produce either a armhf architecture snap or an amd64 architecture snap. It only allows specifying that a given snapcraft.yaml when built on a particular host architecture results in a specific single architecture (i.e. run-on: armhf) or a single snap which can run on multiple architectures (i.e. architecture generic code like a shell script or python script) which is known as a multiarch snap and specified with run-on: [armhf, amd64].

For now you have two options:

  1. Don’t use base: core18 and don’t use architectures -> then you can cross-compile a go part with snapcraft cleanbuild --target-arch=armhf successfully, this will require you to install lxd on your build machine
  2. Use base: core18 and specify a single architectures stanza for cross-compiling (note that this is mutually exclusive with specifying a native compiling snapcraft.yaml architectures stanza) and hard-code the architecture you want to cross-compile to inside the part’s override-build step, i.e. do something like:
architectures:
  - build-on: amd64
    run-on: armhf

parts:
  my-part:
    source: .
    plugin: go
    override-build:
      mkdir -p $SNAPCRAFT_PART_INSTALL/bin
      GOOS=linux GOARCH=armhf go build -o $SNAPCRAFT_PART_INSTALL/bin/my-app ./cmd/my-app

Then if you wanted to build your app natively for amd64, you would need to change the architectures to be:

architectures:
  - build-on: amd64
    run-on: amd64

Thanks for the hint. I’ll follow this up once python is supported and I’ve asked for an update here: Cross compile support for Python plugin

Well I don’t really need a base, but:

bare: Empty base snap, useful for testing and fully statically linked snaps (e.g. busybox-static) (not yet supported)

(Emphasis mine). Source: https://docs.snapcraft.io/snapcraft-top-level-metadata/8334

and:

NOTE: Without specifying the base property in the snapcraft.yaml Snapcraft will operate in legacy mode, which is essentially a 2.4X old version of Snapcraft and will lose the functionalities and bug fixes introduced in the new version.

Source: https://docs.snapcraft.io/snapcraft-overview/8940

I don’t really want to rely on some old version which has known bugs. Maybe even bugs in the packaged snap.

I still went with option one, because I didn’t understand this part of option two:

specify a single architectures stanza for cross-compiling (note that this is mutually exclusive with specifying a native compiling snapcraft.yaml architectures stanza)

So with option one:

$ apt install lxd lxd-client
$ lxd init
...
$ snapcraft cleanbuild --target-arch=armhf

Setting target machine to 'armhf'
Creating snapcraft-manipulatedly-clammy-milagro
Starting snapcraft-manipulatedly-clammy-milagro
error: Error calling 'lxd forkstart snapcraft-manipulatedly-clammy-milagro /var/lib/lxd/containers /var/log/lxd/snapcraft-manipulatedly-clammy-milagro/lxc.conf': err='Failed to run: /usr/bin/lxd forkstart snapcraft-manipulatedly-clammy-milagro /var/lib/lxd/containers /var/log/lxd/snapcraft-manipulatedly-clammy-milagro/lxc.conf: '
  lxc 20190429200938.583 ERROR lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:222 - If you really want to start this container, set
  lxc 20190429200938.583 ERROR lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:223 - lxc.aa_allow_incomplete = 1
  lxc 20190429200938.583 ERROR lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:224 - in your container configuration file
  lxc 20190429200938.584 ERROR lxc_sync - sync.c:__sync_wait:59 - An error occurred in another process (expected sequence number 5)
  lxc 20190429200938.743 ERROR lxc_container - lxccontainer.c:wait_on_daemonized_start:804 - Received container state "ABORTING" instead of "RUNNING"
  lxc 20190429200938.743 ERROR lxc_start - start.c:__lxc_start:1802 - Failed to spawn container "snapcraft-manipulatedly-clammy-milagro".

Try `lxc info --show-log local:snapcraft-manipulatedly-clammy-milagro` for more info
Failed to create container: a new LXD container could not be created.
Refer to the documentation at https://linuxcontainers.org/lxd/getting-started-cli.

This sounds like (another) snapcraft error, because although this error log suggests to edit the container config, the container isn’t reused when running the snapcraft build again. Instead another container is started with the same old config, leading to the same error.

What OS are you installing lxd on?

Ubuntu 16.04 on AMD64.

But I realized that only the cleanbuild leads to the creation of another container. So adding lxc.aa_allow_incomplete = 1 in /var/log/lxd/snapcraft-manipulatedly-clammy-milagro/lxc.conf and then running snapcraft build example --target-arch=armhf worked a bit better.

“A bit better” because of two issues:

  1. I had to remove the title, license and worst of all go-channel from the snapcraft.yaml. The go-channel set to an empty string previously was a great way to NOT use some unofficial Go snap, but with the multipass snap snapcraft installed the Go package from the Ubuntu 18.04 package repo, which was Go 1.10. Now with the LXC container being Ubuntu 16.04 I have to use the ancient Go version 1.6.

  2. The log now ends with go build -o /root/go/src/github.com/myname/example/parts/example/go/bin/example example, without any snap being built.

So since you’re now building with snapcraft 2.X, you’ll need to build go from source using the go remote part since build-snaps aren’t supported AFAIK. You can do so like this:

parts:
  go:
    source-branch: go1.11.9
    source-depth: 1
  my-part:
    after: [go]
...

I actually had that in my snapcraft.yaml for about one year, but now I implemented some updates to my app and wanted to check out what’s new with Snapcraft as preparation to a new release of the app. This is what lead me to seeing the suggestion to use base18 in the docs and the possibility to use architecture and trying that out.

Having to build go from source each time is also very time consuming and shouldn’t be necessary. Why is there no (easy?) way to just use the official Go binaries?

Anyway, having to go back to what I had already, not being able to update the snapcraft.yaml, worrying about “legacy mode, which is essentially a 2.4X old version of Snapcraft and will lose the functionalities and bug fixes introduced in the new version”, doesn’t feel good.

And the other option you mentioned includes an override-build, which probably makes it impossible to properly use your build.snapcraft.io CI server, which I used so far.

You can of course continue to use build.snapcraft.io with an override-build spec. If you run into issues with building your snap on build.snapcraft.io with override-build, you should report back on the forum as it’s surely a bug (or possibly missing feature). build.snapcraft.io should be able to build just about any go based snap that you can build locally.

I agree that using the legacy branch of snapcraft 2.X isn’t great and that ideally we would all use snapcraft 3.X with bases, but I build go-based snaps using the legacy branch of snapcraft 2.X regularly and don’t encounter many issues. It is slower to have to build go from source, however the “best” solution to that is to use the go snap with build-snaps, (which is what is used internally for a part when you specify go-channel in the part definition) however using build-snaps requires using snapcraft 3.X with bases.

Also, I have found LP bug https://bugs.launchpad.net/snapcraft/+bug/1813634 which seems to be about this issue. Hopefully the snapcraft developers will be able to fix that bug soon and you will be able to cross-compile with snapcraft 3.X easily.

Sorry for the misunderstanding, I didn’t mean it’s not possible to use build.snapcraft.io generally when using override-build. I was referring to your specific suggestion, which included GOOS=linux GOARCH=armhf, in which case I guess no AMD64 snap would be produced by the build system.

I’m not sure about that. That’s what I was referring to in my previous comment:

The Go snap is by Michael Hudson-Doyle and he seems to be a Canonical employee, but it’s not an official package, neither by the Go team, nor by Canonical. If someone could hack his account the attacker could inject malicious code into potentially hundres of daily built snaps that are written in Go. That’s a huge incentive for an attacker, so the risk is real. That’s why I would prefer to set that to an empty string, using the official Go 1.10 package from the Ubuntu package repositories, while at the same time not having to build Go from source.

Ah sorry I misunderstood your concern regarding the version of go. If that’s your concern, you can always build go from the official upstream source by copying the full go remote-part definition (if you copy it you ensure that you control all of the part definition, as using remote parts could open you up to someone maliciously modifying the wiki, secured as it is) available here which will ensure that you build go from source from the trusted upstream sources.

Alternatively, you could implement your own part providing the go binaries which downloads and verifies the official releases as provided by the official go team.