Snapcraft fails using LXD on NVIDIA Jetson TX2

snapcraft
1

Hi forum – I’m trying to use my NVIDIA Jetson TX2 to build a snap for ARM64. Virtualization/KVM is not available (and so neither is Multipass) on the TX2, so I’m trying to use LXD as my build environment.

As an FYI, the snap in question builds fine for amd64, also using LXD, but my problems occur before the build step of my snap begins.

LXD is installed as a snap from latest/stable (v4.0.1).

First try: I installed snapcraft from the latest/stable track (v3.11) and snapcraft --use-lxd returns the error “An error occured with the instance when trying to launch ‘LXD’: Create instance: Requested architecture isn’t supported by this host.”

Decided to try freshing snapcraft to the channel latest/candidate (v4.0) and I get much further – an LXD container is created. It’s assigned an IP and apt-get update appears to execute just fine. However the next challenge is:

An error occured when trying to execute 'apt-get dist-upgrade --yes' with 'LXD': returned exit code 100.

Immediately preceding this error, there’s lots of messages involving ‘Temporary failure resolving ports.ubuntu.com’.

I also appear to be unable to drop in to a shell with ‘–debug’ at this phase. Any help greatly appreciated.

2

the --use-lxd switch is for using snapcraft outside of the container … i’d rather recommend:

snap install lxd
sudo lxd init # hit enter until done
sudo lxc launch ubuntu:18.04 bionic
sudo lxc exec bionic bash ....

inside the container:

apt update
apt upgrade
apt install git
snap install snapcraft --classic
git clone $your_desired_repo
cd $to_what_you_cloned
snapcraft --destructive-mode
3

From 4.0 we now support --use-lxd on all supported architectures supported by LXD

1 Like
4

That doesnt help if you are on UbuntuCore where you can’t install --classic snaps like snapcraft :wink: …

–destructive-mode inside an lxd container is essential if you want to build on core …

5

The bootstrapping failed due to network errors so not at a point to get debug going just yet. Can you please try edge and see if the issue persists? Are you behind a proxy? Can you try and use SNAPCRAFT_BUILD_ENVIRONMENT_NAMESERVER with the IP of your DNS server? Snapcraft defaults to the one on the host.

6

And yet, there is no mention of Ubuntu Core on this post nor an issue with installing Snapcraft on the system.

1 Like
7

Working on trying these suggestions; will get back to you shortly.

In the meantime – correct, I’m not building on core at the moment. HOWEVER I will need to build on Core in the near future so this is all great information :wink:

8

oh … one pro-tip i always forget … lxd ships this extremely useful test tool:

lxd.check-kernel

try it out :wink:

9

@sergiusens – this does still reproduce on edge. I’ve tried to set the variable via export SNAPCRAFT_BUILD_ENVIRONMENT_NAMESERVER=8.8.8.8 and this does not resolve the issue. I should also note I’ve had some issues with LXD on Linux for Tegra being able to get an IP address – not sure if there’s some strange networking configuration on this image, and frankly I’m not experienced enough at Linux Networking to be able to tell.

@ogra – I was optimistic but when I run snapcraft --destructive-mode inside the container, it appears to hang. That, or the build output I’ve expected isn’t redirected to stdout. Thoughts?

10

On the outside, does lxc list return a list of containers with IPs assigned?
what if you lxc launch ubuntu:18.04? What is the host OS?

11

The host is Linux for Tegra which is derived from Ubuntu 18.04.

On IP addresses: yes and no. If I run lxc launch ubuntu:18.04 there is not an IP address assigned. Further more if I run ping from within the newly created container I get ping: socket: Operation not permitted. If I set the container to be privileged via lxc config set <name> security.privileged true && lxc restart <name> then I do see IPv4 and IPv6 addresses assigned to the container. Pings are also successful from within the container.

On the other hand, running sudo snapcraft --use-lxd generates a container which does get an IPv4 address, but not an IPv6 address. Based on the output, apt-get update appears to hit the targets successfully and pull down updates. It later fails on the apt-get dist-upgrade step as described in my original post.

12

Some background the L4T kernel is version 4.9 I am curious what the lxd.check-kernel command @ogra pointed out will return. Maybe another option could be running the destructive mode inside a Docker container https://snapcraft.io/docs/build-on-docker As far as I know Nvidia is supporting Docker on the L4T distribution.

I skipped the L4T stuff pulled out a Raspberry PI 3 and installed Ubuntu Core there. Maybe this can be used as a build platform for the snap.

13

As @ogra said up thread, you’ll have trouble building snaps on an Ubuntu Core system. One alternative would be to install classic Ubuntu Server on the Pi instead:

14

From my understanding running LXD in the Core running on a PI and inside the LXD container you shall be able to build snaps or am I wrong?

15

The usual way to build a snap with LXD is to install Snapcraft on the host system and run snapcraft --use-lxd. Snapcraft will then spin up an ephemeral container with a clean environment to build your snap, and discard it when done.

The problem is that the snapcraft snap uses classic confinement, which are not installable Ubuntu Core systems. The Ubuntu Server systems should work fine though.

16

i definitely did not say that :slight_smile:

i’m personally owning 50+ different arm boards, none of them has seen ubuntu classic installs for the last 4 years, the powerful ones are all build machines using lxd in the way i described above …

what i’d really like to know is the output of lxd.check-kernel from @theseankelly. i suspect the kernel simply misses config options lxd requires …

1 Like
17 
$ lxd.check-kernel
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/memory
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/devices
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/blkio
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/freezer
/sys/fs/cgroup/pids
/sys/fs/cgroup/debug

Cgroup v2 mount points:
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: enabled, loaded
CONFIG_NF_NAT_IPV6: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, loaded

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: missing
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /snap/lxd/14958/bin/lxc-checkconfig
18

there you go … that will definitely cause networking issues …

19

@theseankelly it looks like those are config parameters which are available in the kernel since 3.10 so a recompile of the kernel shall do the trick.

20

i think @abeato is already looking into adjusting the kernel for his jetson images …