Snap commands panicking

I am running snapd on debian 9 based custom kernel on a Beagle bone black. The snaps work as expected for some time but starts panicking with out of memory error. Even the regular snap commands fails with similar errors. Here is one such error for snap list. Any idea what could be going wrong here?

Snap info

snap version
snap    2.49.2
snapd   2.49.2
series  16
debian  9
kernel  4.9.33
# snap list
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x7f9f6d46, 0x16)
	/usr/lib/go-1.7/src/runtime/panic.go:566 +0x80 fp=0xbe9da82c sp=0xbe9da820
runtime.sysMap(0x86f00000, 0x9000000, 0x8ff00001, 0x7fc9bb68)
	/usr/lib/go-1.7/src/runtime/mem_linux.go:230 +0x174 fp=0xbe9da854 sp=0xbe9da82c
runtime.(*mheap).mapBits(0x7fc8da48, 0x90000000)
	/usr/lib/go-1.7/src/runtime/mbitmap.go:159 +0xe4 fp=0xbe9da86c sp=0xbe9da854
runtime.(*mheap).sysAlloc(0x7fc8da48, 0x100000, 0x1)
	/usr/lib/go-1.7/src/runtime/malloc.go:408 +0x164 fp=0xbe9da8c0 sp=0xbe9da86c
runtime.(*mheap).grow(0x7fc8da48, 0x8, 0x0)
	/usr/lib/go-1.7/src/runtime/mheap.go:726 +0x68 fp=0xbe9da8f4 sp=0xbe9da8c0
runtime.(*mheap).allocSpanLocked(0x7fc8da48, 0x1, 0x7f64a948)
	/usr/lib/go-1.7/src/runtime/mheap.go:630 +0x694 fp=0xbe9da924 sp=0xbe9da8f4
runtime.(*mheap).alloc_m(0x7fc8da48, 0x1, 0x10, 0x0, 0x1)
	/usr/lib/go-1.7/src/runtime/mheap.go:515 +0x15c fp=0xbe9da960 sp=0xbe9da924
runtime.(*mheap).alloc.func1()
	/usr/lib/go-1.7/src/runtime/mheap.go:579 +0x40 fp=0xbe9da97c sp=0xbe9da960
runtime.systemstack(0xbe9da990)
	/usr/lib/go-1.7/src/runtime/asm_arm.s:261 +0xb4 fp=0xbe9da980 sp=0xbe9da97c
runtime.(*mheap).alloc(0x7fc8da48, 0x1, 0x10, 0x100, 0x7f634dd0)
	/usr/lib/go-1.7/src/runtime/mheap.go:580 +0x5c fp=0xbe9da9a8 sp=0xbe9da980
runtime.(*mcentral).grow(0x7fc8e858, 0x0)
	/usr/lib/go-1.7/src/runtime/mcentral.go:210 +0xbc fp=0xbe9da9d4 sp=0xbe9da9a8
runtime.(*mcentral).cacheSpan(0x7fc8e858, 0x7f62ea70)
	/usr/lib/go-1.7/src/runtime/mcentral.go:91 +0x670 fp=0xbe9daa18 sp=0xbe9da9d4
runtime.(*mcache).refill(0xb6e7d000, 0x10, 0x7fc9bb70)
	/usr/lib/go-1.7/src/runtime/mcache.go:121 +0xe0 fp=0xbe9daa2c sp=0xbe9daa18
runtime.(*mcache).nextFree.func1()
	/usr/lib/go-1.7/src/runtime/malloc.go:505 +0x28 fp=0xbe9daa3c sp=0xbe9daa2c
runtime.systemstack(0xbe9daa58)
	/usr/lib/go-1.7/src/runtime/asm_arm.s:261 +0xb4 fp=0xbe9daa40 sp=0xbe9daa3c
runtime.(*mcache).nextFree(0xb6e7d000, 0x7f682e10, 0x4000, 0x7fc8a9f0, 0x7fc9bb00)
	/usr/lib/go-1.7/src/runtime/malloc.go:506 +0x164 fp=0xbe9daa64 sp=0xbe9daa40
runtime.mallocgc(0xf0, 0x7fb3c120, 0x7f64f101, 0x0)
	/usr/lib/go-1.7/src/runtime/malloc.go:658 +0xd0c fp=0xbe9dab04 sp=0xbe9daa64
runtime.newobject(0x7fb3c120, 0x1)
	/usr/lib/go-1.7/src/runtime/malloc.go:785 +0x30 fp=0xbe9dab18 sp=0xbe9dab04
runtime.malg(0x8000, 0x7fc8acc8)
	/usr/lib/go-1.7/src/runtime/proc.go:2688 +0x24 fp=0xbe9dab34 sp=0xbe9dab18
runtime.mpreinit(0x7fc8b238)
	/usr/lib/go-1.7/src/runtime/os_linux.go:261 +0x1c fp=0xbe9dab40 sp=0xbe9dab34
runtime.mcommoninit(0x7fc8b238)
	/usr/lib/go-1.7/src/runtime/proc.go:507 +0x118 fp=0xbe9dab64 sp=0xbe9dab40
runtime.schedinit()
	/usr/lib/go-1.7/src/runtime/proc.go:441 +0x44 fp=0xbe9dab88 sp=0xbe9dab64
runtime.rt0_go(0xbe9dad24, 0xb6fa7000, 0xbe9dad24, 0x2, 0x7f685f44, 0xb6ff7000, 0xaaaaaaab, 0x7e47a518, 0x76373ab5, 0x130, ...)
	/usr/lib/go-1.7/src/runtime/asm_arm.s:61 +0x8c fp=0xbe9dabc8 sp=0xbe9dab88

NOTE: this is a machine with single cpu core with 512MB RAM

Does snapd run on this device? What is systemctl status snapd ? If there’s not enough memory for snapd to even run then I guess I can’t say I’m surprised the snap command fails too.

If snapd is running however you could try using curl or httpie on this device to try and talk to snapd over the HTTP API.

No the service is also killed, But I see there is enough memory when I run free command (around 200+ MB). Restarting the snapd service also fails with the same error.

Also, snapd daemon is running fine, communication over REST API also works fine. Snap commands failing with out of memory error.

Sorry, your messages are somewhat contradicting, is the snapd daemon running out of memory or not? You said the service is killed but also that the daemon is running fine ?

1 Like

Apologies for the confusion, There are two scenarios I am observing , in the first case snapd daemon is running but the commands fail with out of memory error. In the second, snapd daemon is also killed.

Here is the snapd logs when snapd daemin fails

root@27C001223357:/home/smc# journalctl -f -u snapd
-- Logs begin at Thu 2021-04-22 18:05:46 UTC. --
Apr 23 12:58:33 27C001223357 snapd[28407]: AppArmor status: apparmor is enabled but some kernel features are missing: dbus, mount, namespaces, network, ptrace, signal
Apr 23 12:59:39 27C001223357 systemd[1]: snapd.service: Start operation timed out. Terminating.
Apr 23 12:59:40 27C001223357 systemd[1]: snapd.service: Main process exited, code=killed, status=15/TERM
Apr 23 12:59:40 27C001223357 systemd[1]: snapd.service: Failed with result 'timeout'.
Apr 23 12:59:40 27C001223357 systemd[1]: Failed to start Snappy daemon.
Apr 23 12:59:41 27C001223357 systemd[1]: snapd.service: Service RestartSec=100ms expired, scheduling restart.
Apr 23 12:59:41 27C001223357 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 773.
Apr 23 12:59:41 27C001223357 systemd[1]: Stopped Snappy daemon.
Apr 23 12:59:42 27C001223357 systemd[1]: Starting Snappy daemon...
Apr 23 13:00:26 27C001223357 snapd[28414]: AppArmor status: apparmor is enabled but some kernel features are missing: dbus, mount, namespaces, network, ptrace, signal
Apr 23 13:01:11 27C001223357 systemd[1]: snapd.service: Start operation timed out. Terminating.
Apr 23 13:01:13 27C001223357 systemd[1]: snapd.service: Main process exited, code=killed, status=15/TERM
Apr 23 13:01:13 27C001223357 systemd[1]: snapd.service: Failed with result 'timeout'.
Apr 23 13:01:14 27C001223357 systemd[1]: Failed to start Snappy daemon.
Apr 23 13:01:14 27C001223357 systemd[1]: snapd.service: Service RestartSec=100ms expired, scheduling restart.
Apr 23 13:01:14 27C001223357 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 774.
Apr 23 13:01:15 27C001223357 systemd[1]: Stopped Snappy daemon.
Apr 23 13:01:15 27C001223357 systemd[1]: Starting Snappy daemon...

It could be because of the custom kernel. Still trying to figure out