I was able to update the recipe to snapd 2.41.1 and it seems to work in qemu, however the raspberry pi image that we produce is still faced by similar issues.
Here is a log that I just gathered. Do note that the last two start attempts failed with different errors.
Do these logs hint to anything fundamentally wrong that we are doing ? (cc @zyga-snapd)
-- Logs begin at Thu 1970-01-01 00:00:05 UTC, end at Thu 2020-10-22 06:52:05 UTC. --
Oct 22 06:49:36 raspberrypi3 systemd[1]: Starting Snap Daemon...
Oct 22 06:49:39 raspberrypi3 snapd[210]: AppArmor status: apparmor not enabled
Oct 22 06:49:39 raspberrypi3 snapd[210]: daemon.go:343: started snapd/2.47.1 (series 16; classic; devmode) poky/3.1.3 (armhf) linux/5.4.59-v7.
Oct 22 06:49:39 raspberrypi3 snapd[210]: daemon.go:436: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
Oct 22 06:49:40 raspberrypi3 snapd[210]: panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 06:49:40 raspberrypi3 snapd[210]: [signal SIGSEGV: segmentation violation code=0x1 addr=0xc pc=0x765a14a0]
Oct 22 06:49:40 raspberrypi3 snapd[210]: goroutine 1 [running]:
Oct 22 06:49:40 raspberrypi3 snapd[210]: regexp.(*machine).match(0x456c6000, 0x76b3f370, 0x456c6054, 0x0, 0x1)
Oct 22 06:49:40 raspberrypi3 snapd[210]: /usr/lib/go/src/regexp/exec.go:222 +0x27c
Oct 22 06:49:40 raspberrypi3 snapd[210]: regexp.(*Regexp).doExecute(0x457771a0, 0x0, 0x0, 0x45734000, 0x7a, 0x600, 0x0, 0x0, 0x0, 0x0, ...)
Oct 22 06:49:40 raspberrypi3 snapd[210]: /usr/lib/go/src/regexp/exec.go:542 +0xf8
Oct 22 06:49:40 raspberrypi3 snapd[210]: regexp.(*Regexp).doMatch(...)
Oct 22 06:49:40 raspberrypi3 snapd[210]: go/src/regexp/exec.go:514
Oct 22 06:49:40 raspberrypi3 snapd[210]: regexp.(*Regexp).Match(...)
Oct 22 06:49:40 raspberrypi3 snapd[210]: go/src/regexp/regexp.go:513
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/sandbox/seccomp.(*Compiler).VersionInfo(0x45780f60, 0x0, 0x0, 0x1b, 0x457ce920)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/sandbox/seccomp/compiler.go:74 +0x100
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/interfaces/seccomp.snapSeccompVersionInfoImpl(0xdcb7b8, 0x45780f60, 0x0, 0x0, 0x1, 0x456bbc48)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/interfaces/seccomp/backend.go:68 +0x24
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/interfaces/seccomp.(*Backend).Initialize(0x4540f340, 0x457578f0, 0x457d3c18, 0x0)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/interfaces/seccomp/backend.go:144 +0x40c
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/overlord/ifacestate.(*InterfaceManager).addBackends(0x45777500, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/overlord/ifacestate/helpers.go:72 +0xb0
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/overlord/ifacestate.(*InterfaceManager).StartUp(0x45777500, 0x0, 0x0)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/overlord/ifacestate/ifacemgr.go:156 +0x1ac
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/overlord.(*StateEngine).StartUp(0x45772040, 0x0, 0x0)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/overlord/stateengine.go:109 +0x138
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/overlord.(*Overlord).StartUp(0x45668c00, 0x1c, 0x0)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/overlord/overlord.go:339 +0x94
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/daemon.(*Daemon).Start(0x45668b80, 0x0, 0x0)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/daemon/daemon.go:440 +0x94
Oct 22 06:49:40 raspberrypi3 snapd[210]: main.run(0x4590cb40, 0x0, 0x0)
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/cmd/snapd/main.go:137 +0x2b4
Oct 22 06:49:40 raspberrypi3 snapd[210]: main.main()
Oct 22 06:49:40 raspberrypi3 snapd[210]: github.com/snapcore/snapd/cmd/snapd/main.go:64 +0x100
Oct 22 06:49:40 raspberrypi3 systemd[1]: snapd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 06:49:40 raspberrypi3 systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 22 06:49:40 raspberrypi3 systemd[1]: Failed to start Snap Daemon.
Oct 22 06:49:40 raspberrypi3 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 1.
Oct 22 06:49:40 raspberrypi3 systemd[1]: Stopped Snap Daemon.
Oct 22 06:49:40 raspberrypi3 systemd[1]: Starting Snap Daemon...
Oct 22 06:49:40 raspberrypi3 snapd[320]: AppArmor status: apparmor not enabled
Oct 22 06:49:41 raspberrypi3 snapd[320]: daemon.go:343: started snapd/2.47.1 (series 16; classic; devmode) poky/3.1.3 (armhf) linux/5.4.59-v7.
Oct 22 06:49:41 raspberrypi3 snapd[320]: daemon.go:436: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
Oct 22 06:49:41 raspberrypi3 snapd[320]: panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 06:49:41 raspberrypi3 snapd[320]: [signal SIGSEGV: segmentation violation code=0x1 addr=0xe pc=0x7653a710]
Oct 22 06:49:41 raspberrypi3 snapd[320]: goroutine 1 [running]:
Oct 22 06:49:41 raspberrypi3 snapd[320]: regexp.(*Regexp).doOnePass(0x45740c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0xbc7c67, 0xf, 0x0, 0x0, ...)
Oct 22 06:49:41 raspberrypi3 snapd[320]: /usr/lib/go/src/regexp/exec.go:429 +0x168
Oct 22 06:49:41 raspberrypi3 snapd[320]: regexp.(*Regexp).doExecute(0x45740c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0xbc7c67, 0xf, 0x0, 0x0, ...)
Oct 22 06:49:41 raspberrypi3 snapd[320]: /usr/lib/go/src/regexp/exec.go:532 +0x37c
Oct 22 06:49:41 raspberrypi3 snapd[320]: regexp.(*Regexp).doMatch(...)
Oct 22 06:49:41 raspberrypi3 snapd[320]: go/src/regexp/exec.go:514
Oct 22 06:49:41 raspberrypi3 snapd[320]: regexp.(*Regexp).MatchString(...)
Oct 22 06:49:41 raspberrypi3 snapd[320]: go/src/regexp/regexp.go:507
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/snap/naming.ValidateInterface(0xbc7c67, 0xf, 0xf, 0x0)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/snap/naming/validate.go:116 +0x74
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/snap.ValidateInterfaceName(...)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/snap/validate.go:90
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/interfaces.(*Repository).AddInterface(0x45867560, 0xde00b0, 0x457581e0, 0x0, 0x0)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/interfaces/repo.go:78 +0x94
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/overlord/ifacestate.(*InterfaceManager).addInterfaces(0x458f99e0, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/overlord/ifacestate/helpers.go:57 +0x6c
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/overlord/ifacestate.(*InterfaceManager).StartUp(0x458f99e0, 0x0, 0x0)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/overlord/ifacestate/ifacemgr.go:153 +0x178
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/overlord.(*StateEngine).StartUp(0x458f7500, 0x0, 0x0)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/overlord/stateengine.go:109 +0x138
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/overlord.(*Overlord).StartUp(0x4574ac00, 0x1c, 0x0)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/overlord/overlord.go:339 +0x94
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/daemon.(*Daemon).Start(0x4574ab80, 0x0, 0x0)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/daemon/daemon.go:440 +0x94
Oct 22 06:49:41 raspberrypi3 snapd[320]: main.run(0x458d3980, 0x0, 0x0)
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/cmd/snapd/main.go:137 +0x2b4
Oct 22 06:49:41 raspberrypi3 snapd[320]: main.main()
Oct 22 06:49:41 raspberrypi3 snapd[320]: github.com/snapcore/snapd/cmd/snapd/main.go:64 +0x100
Oct 22 06:49:41 raspberrypi3 systemd[1]: snapd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 06:49:41 raspberrypi3 systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 22 06:49:41 raspberrypi3 systemd[1]: Failed to start Snap Daemon.
Oct 22 06:49:41 raspberrypi3 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 2.
Oct 22 06:49:41 raspberrypi3 systemd[1]: Stopped Snap Daemon.
Oct 22 06:49:41 raspberrypi3 systemd[1]: Starting Snap Daemon...
Oct 22 06:49:42 raspberrypi3 snapd[363]: AppArmor status: apparmor not enabled
Oct 22 06:49:42 raspberrypi3 snapd[363]: daemon.go:343: started snapd/2.47.1 (series 16; classic; devmode) poky/3.1.3 (armhf) linux/5.4.59-v7.
Oct 22 06:49:42 raspberrypi3 snapd[363]: daemon.go:436: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
Oct 22 06:49:42 raspberrypi3 snapd[363]: panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 06:49:42 raspberrypi3 snapd[363]: [signal SIGSEGV: segmentation violation code=0x1 addr=0xc pc=0x765394a0]
Oct 22 06:49:42 raspberrypi3 snapd[363]: goroutine 1 [running]:
Oct 22 06:49:42 raspberrypi3 snapd[363]: regexp.(*machine).match(0x45918180, 0x76ad73c0, 0x459181e0, 0x0, 0x4)
Oct 22 06:49:42 raspberrypi3 snapd[363]: /usr/lib/go/src/regexp/exec.go:222 +0x27c
Oct 22 06:49:42 raspberrypi3 snapd[363]: regexp.(*Regexp).doExecute(0x45660c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb87c67, 0xf, 0x0, 0x0, ...)
Oct 22 06:49:42 raspberrypi3 snapd[363]: /usr/lib/go/src/regexp/exec.go:542 +0xf8
Oct 22 06:49:42 raspberrypi3 snapd[363]: regexp.(*Regexp).doMatch(...)
Oct 22 06:49:42 raspberrypi3 snapd[363]: go/src/regexp/exec.go:514
Oct 22 06:49:42 raspberrypi3 snapd[363]: regexp.(*Regexp).MatchString(...)
Oct 22 06:49:42 raspberrypi3 snapd[363]: go/src/regexp/regexp.go:507
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/snap/naming.ValidateInterface(0xb87c67, 0xf, 0xf, 0x0)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/snap/naming/validate.go:116 +0x74
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/snap.ValidateInterfaceName(...)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/snap/validate.go:90
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/interfaces.(*Repository).AddInterface(0x45642570, 0xda00b0, 0x456761e0, 0x0, 0x0)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/interfaces/repo.go:78 +0x94
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/overlord/ifacestate.(*InterfaceManager).addInterfaces(0x45661080, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/overlord/ifacestate/helpers.go:57 +0x6c
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/overlord/ifacestate.(*InterfaceManager).StartUp(0x45661080, 0x0, 0x0)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/overlord/ifacestate/ifacemgr.go:153 +0x178
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/overlord.(*StateEngine).StartUp(0x4540c020, 0x0, 0x0)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/overlord/stateengine.go:109 +0x138
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/overlord.(*Overlord).StartUp(0x45668c00, 0x1c, 0x0)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/overlord/overlord.go:339 +0x94
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/daemon.(*Daemon).Start(0x45668b80, 0x0, 0x0)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/daemon/daemon.go:440 +0x94
Oct 22 06:49:42 raspberrypi3 snapd[363]: main.run(0x45876e40, 0x0, 0x0)
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/cmd/snapd/main.go:137 +0x2b4
Oct 22 06:49:42 raspberrypi3 snapd[363]: main.main()
Oct 22 06:49:42 raspberrypi3 snapd[363]: github.com/snapcore/snapd/cmd/snapd/main.go:64 +0x100
Oct 22 06:49:42 raspberrypi3 systemd[1]: snapd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 06:49:42 raspberrypi3 systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 22 06:49:42 raspberrypi3 systemd[1]: Failed to start Snap Daemon.
Oct 22 06:49:43 raspberrypi3 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 3.
Oct 22 06:49:43 raspberrypi3 systemd[1]: Stopped Snap Daemon.
Oct 22 06:49:43 raspberrypi3 systemd[1]: Starting Snap Daemon...
Oct 22 06:49:43 raspberrypi3 snapd[377]: AppArmor status: apparmor not enabled
Oct 22 06:49:43 raspberrypi3 snapd[377]: cannot run daemon: cannot export experimental config flags: mkdir -
Oct 22 06:49:43 raspberrypi3 snapd[377]: -
Oct 22 06:49:43 raspberrypi3 snapd[377]: 0
Oct 22 06:49:43 raspberrypi3 snapd[377]: 9
Oct 22 06:49:43 raspberrypi3 snapd[377]: a
Oct 22 06:49:43 raspberrypi3 snapd[377]: z
Oct 22 06:49:43 raspberrypi3 snapd[377]: : invalid argument
Oct 22 06:49:43 raspberrypi3 systemd[1]: snapd.service: Main process exited, code=exited, status=1/FAILURE
Oct 22 06:49:43 raspberrypi3 systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 22 06:49:43 raspberrypi3 systemd[1]: Failed to start Snap Daemon.
Oct 22 06:49:44 raspberrypi3 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 4.
Oct 22 06:49:44 raspberrypi3 systemd[1]: Stopped Snap Daemon.
Oct 22 06:49:44 raspberrypi3 systemd[1]: Starting Snap Daemon...
Oct 22 06:49:44 raspberrypi3 snapd[386]: AppArmor status: apparmor not enabled
Oct 22 06:49:44 raspberrypi3 snapd[386]: panic: cannot initialize the builtin base-declaration: cannot assemble the builtin base-declaration: slot-snap-type in allow-installation in slot rule for interface "modem-manager" contains an invalid element: "core"
Oct 22 06:49:44 raspberrypi3 snapd[386]: goroutine 1 [running]:
Oct 22 06:49:44 raspberrypi3 snapd[386]: github.com/snapcore/snapd/interfaces/policy.init.0()
Oct 22 06:49:44 raspberrypi3 snapd[386]: github.com/snapcore/snapd/interfaces/policy/basedeclaration.go:197 +0xe4
Oct 22 06:49:44 raspberrypi3 systemd[1]: snapd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 06:49:44 raspberrypi3 systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 22 06:49:44 raspberrypi3 systemd[1]: Failed to start Snap Daemon.
Oct 22 06:49:45 raspberrypi3 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 5.
Oct 22 06:49:45 raspberrypi3 systemd[1]: Stopped Snap Daemon.
Oct 22 06:49:45 raspberrypi3 systemd[1]: snapd.service: Start request repeated too quickly.
Oct 22 06:49:45 raspberrypi3 systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 22 06:49:45 raspberrypi3 systemd[1]: Failed to start Snap Daemon.
Oct 22 06:49:45 raspberrypi3 systemd[1]: snapd.service: Triggering OnFailure= dependencies.