Hey @ogra I’ve rebuilt the kernel with those features enabled and I still hit the same error (unble to resolve ports.ubuntu.com) using snapcraft --use-lxd outside the container.
Here’s my latest lxd.check-kernel output:
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled
--- Control groups ---
Cgroups: enabled
Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/blkio
/sys/fs/cgroup/devices
/sys/fs/cgroup/memory
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/freezer
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/pids
/sys/fs/cgroup/debug
/sys/fs/cgroup/perf_event
Cgroup v2 mount points:
/sys/fs/cgroup/unified
Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled
--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: enabled, loaded
CONFIG_NF_NAT_IPV6: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, loaded
--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:
Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /snap/lxd/14958/bin/lxc-checkconfig
Oh one more thing – I blew away all my lxd installation(s) in order to re-try the building inside a container via snapcraft --use-destructive. I noticed that the first time I run snap install snapcraft --classic I get:
If I run snap install snapcraft --classic again, it appears to succeed! But the snapcraft --destructive-mode command again appears to hang. It doesn’t look like snapd is running (per the error in the first installation attempt).
Though it dawned on me that since I’m already down the path of recompiling my kernel, I might as well enable KVM and give Multipass a shot…but it would still be ideal if I could get builds within LXD containers working.
I don’t think multipass would work considering the user space logic it has.
I think that the LXD discourse would be a good place to bounce into to solve the LXD specifics.
But as soon as you get LXD working fine, there should be no issues with snapcraft --use-lxd or lxc launch ubuntu:18.04 foo && lxc exec foo -- snap install --classic snapcraft && lxc exec foo -- sh -c "cd my-source && snapcraft --destructive-mode" (this last command is illustrative, I did not try or expect that to be a one liner copy paste )
Hey @ogra – while I’m working through the LXD issues on the LXD forum (linked above) I was hoping to get your take on the problems with building inside an LXD container – this is actually our ideal usage scenario for internal development as well as for our customers.
I’ve tried this out on L4T on my TX2 and Nano devkits, and also on my Nano using @abeato’s Jetson Ubuntu Core images (I built yesterday so as to have his latest LXD fixes included).
Regardless of the operating system, I struggle with the same things:
LXD containers don’t get IP addresses. This can be solved by running dhclient eth0 inside the container (but things like ping still don’t work, permission denied). Realistically I need to lxc config set <name> security.privileged true on the container to fix this.
The following workflow doesn’t work (for a container named ‘snapbuilder’):
Looks like simply re-running the install tends to unblock, I now have snapcraft installed. But I’m back to the issue I mentioned a few posts above – the snapcraft command inside the LXD container seems to hang.
journalctl -u snapd shows:
May 22 13:56:00 test systemd[1]: snapd.service: Failed to reset devices.list: Operation not permitted
May 22 13:56:00 test systemd[1]: Starting Snappy daemon...
May 22 13:56:01 test snapd[207]: AppArmor status: apparmor not enabled
May 22 13:56:01 test snapd[207]: daemon.go:346: started snapd/2.42.1+18.04 (series 16; classic; devmode) ubuntu/18.04 (arm64) linux/4.9.
May 22 13:56:01 test snapd[207]: daemon.go:439: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
May 22 13:56:01 test systemd[1]: Started Snappy daemon.
May 22 13:56:01 test snapd[207]: api.go:952: Installing snap "snapcraft" revision unset
May 22 13:56:06 test snapd[207]: daemon.go:540: gracefully waiting for running hooks
May 22 13:56:06 test snapd[207]: daemon.go:542: done waiting for running hooks
May 22 13:56:06 test snapd[207]: daemon stop requested to wait for socket activation
May 22 13:56:07 test systemd[1]: snapd.service: Failed to reset devices.list: Operation not permitted
May 22 13:56:07 test systemd[1]: Starting Snappy daemon...
May 22 13:56:07 test snapd[338]: AppArmor status: apparmor not enabled
May 22 13:56:07 test snapd[338]: daemon.go:346: started snapd/2.42.1+18.04 (series 16; classic; devmode) ubuntu/18.04 (arm64) linux/4.9.
May 22 13:56:07 test snapd[338]: daemon.go:439: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
May 22 13:56:07 test systemd[1]: Started Snappy daemon.
May 22 13:56:07 test snapd[338]: api.go:952: Installing snap "snapcraft" revision unset
May 22 13:56:14 test systemd[1]: snapd.service: Failed to reset devices.list: Operation not permitted
May 22 13:56:15 test systemd[1]: snapd.service: Failed to reset devices.list: Operation not permitted
May 22 13:56:17 test snapd[338]: handlers.go:460: Reported install problem for "snapd" as 00116b7a-9c34-11ea-aa41-fa163ee63de6 OOPSID
May 22 14:01:22 test snapd[338]: daemon.go:540: gracefully waiting for running hooks
May 22 14:01:22 test snapd[338]: daemon.go:542: done waiting for running hooks
May 22 14:01:22 test snapd[338]: daemon stop requested to wait for socket activation
Googling that brought me here: https://github.com/lxc/lxd/issues/2004 – with some comments about ‘installing snaps inside lxd containers won’t work’ but that’s quite old.
Have been unsuccessful following all guidance involving manipulating the raw.lxc object in the configuration for my container (I guess trying to set to run unconfined).
And with enough persistence, it appears I have prevailed (EDIT: I haven’t…intermittent success at least)
I used the recommendations to set the raw.lxc parameters from this issue I referenced above. However lxc config edit test complained about formatting issues when adding in the raw.lxc section. Then I found this issue which describes how many of the raw.lxc parameters were either renamed or removed.
I came up with the following modifications to my config:
I’m going to keep updating my progress here in case anyone else stumbles on this issue later.
Not smooth sailing yet – it works sometimes, and other times when I run ‘snapcraft’ I get the hang (and it seems like the whole LXD container hangs?). I’ve gotten better results (so far at least) installing ‘snapcraft’ from apt inside the LXD container, instead of as a snap.
Haha well okay. My build ended up failing anyway using it…it seems like the common denominator here is: ‘have success → post on forum → stop having success’
I’ve done so much cable wiggling over the last couple weeks.
I’ll pick this topic (building snaps inside LXD itself) up in a day or two. Currently have the machine set up for remote access for the LXD team to debug core LXD issues.
FTR, I get the same hang when running snapcraft in lxd. Interestingly, when I run the command with snap run --strace ..., it does not block anymore - which also means I cannot find where it is blocking.
Yeah, it’s the Linux for Tegra (L4T) standard image. Networking problems seemingly can be worked around by running privileged (lxc config set security.privileged true) but I still hit the hang.
)
the apt packaged snapcraft is ancient and unmantained and will give you different binary results …
I’ve done so much cable wiggling over the last couple weeks.