cross post from ask-ubuntu https://askubuntu.com/q/1196406/152287
I’m trying 5.4 and 5.5 kernels to test a bug fix for the increase in power use while idle for intel integrated video on machines of gen 6 or later (it’s in the official 5.5 kernel rc1, the patch applies cleanly to 5.4.3).
These are not official Ubuntu kernels, although we are invited to do just what I’ve done when testing bug fixes. Other users on the bug list say there are stability problems seen in extended use, so I thought I’d try to use the 5.5 kernel in real life for a while, to see what happens.
But: snap apps don’t launch under either of these kernels. They fail silently. If I can find relevant errors I will add them here.
This manjaro post https://forum.manjaro.org/t/kernel-5-4-snaps-and-flatpaks-not-opening-for-the-most-part/106535 sasys “That is a config issue of the kernel. It is fixed in 5.3 but not yet in 5.4 series.”
Does anyone here know how to fix?
EDIT: By the way, I am copying the kernel config of my most recent ubuntu kernel (5.3) before building the mainline kernel, so therfore I expected to catch any important configuration options.
Is there anything in the journal for snapd? Try journalctl -e --no-pager -u snapd
Also, are there any denials in the system log? Try journalctl -e --no-pager | grep DENIED
After trying to launch libreoffice:
journalctl -e --no-pager -u snapd
has many lines.
This seems to recur:
Dec 17 22:45:14 raffles snapd[27225]: AppArmor status: apparmor is enabled but some kernel features are missing: dbus, network
Dec 17 22:45:14 raffles snapd[27225]: AppArmor status: apparmor is enabled but some kernel features are missing: dbus, network
Dec 17 22:45:15 raffles snapd[27225]: patch.go:64: Patching system state level 6 to sublevel 1…
Dec 17 22:45:15 raffles snapd[27225]: patch.go:64: Patching system state level 6 to sublevel 2…
Dec 17 22:45:15 raffles snapd[27225]: patch.go:64: Patching system state level 6 to sublevel 3…
Dec 17 22:45:15 raffles snapd[27225]: daemon.go:346: started snapd/2.43~pre1+git1601.9d911ca~ubuntu16.04.1 (series 16; classic; devmode) ubuntu/
Dec 17 22:45:15 raffles snapd[27225]: daemon.go:439: adjusting startup timeout by 1m30s (pessimistic estimate of 30s plus 5s per snap)
Dec 17 22:45:17 raffles systemd[1]: Started Snappy daemon.
there are DENIEDs
journalctl -e --no-pager | grep DENIED
Dec 17 22:45:10 raffles audit[27067]: AVC apparmor="DENIED" operation="file_mmap" profile="/snap/core/8323/usr/lib/snapd/snap-confine" name="/us
r/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0" pid=27067 comm="snap-confine" requested_mask="m" denied_mask="m" fsuid=0 ouid=0
Dec 17 22:45:10 raffles kernel: audit: type=1400 audit(1576583110.447:336): apparmor="DENIED" operation="file_mmap" profile="/snap/core/8323/usr
/lib/snapd/snap-confine" name="/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0" pid=27067 comm="snap-confine" requested_mask="m" denied_mask="m" fs
uid=0 ouid=0
Dec 17 22:46:19 raffles audit[27735]: AVC apparmor="DENIED" operation="file_mmap" profile="/snap/core/8329/usr/lib/snapd/snap-confine" name="/us
r/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0" pid=27735 comm="snap-confine" requested_mask="m" denied_mask="m" fsuid=0 ouid=0
Dec 17 22:46:19 raffles kernel: audit: type=1400 audit(1576583179.503:404): apparmor="DENIED" operation="file_mmap" profile="/snap/core/8329/usr
/lib/snapd/snap-confine" name="/usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0" pid=27735 comm="snap-confine" requested_mask="m" denied_mask="m" fs
uid=0 ouid=0
Can you also run snap debug confinement and snap debug sandbox-features and snap version?
BTW, this is definitely the source of the issue, but what’s a bind confusing is why this denial happens. I’m not sure why snap-confine would be trying to access /usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0, this seems to be some kind of unity library?
It’'s kubuntu 19.10 if that helps, with the backports PPA (latest KDE). Unity has never been installed on the computer.
tim@raffles ((v5.5-rc2)):~/development/linux$ snap debug confinement
partial
tim@raffles:~$ snap debug sandbox-features
apparmor: kernel:caps kernel:domain kernel:file kernel:mount kernel:namespaces kernel:networ
k_v8 kernel:policy kernel:ptrace kernel:query kernel:rlimit kernel:signal parser:unsafe policy:downgrade
d support-level:partial
confinement-options: classic devmode
dbus: mediated-bus-access
kmod: mediated-modprobe
mount: freezer-cgroup-v1 layouts mount-namespace per-snap-persistency per-snap-profiles p
er-snap-updates per-snap-user-profiles stale-base-invalidation
seccomp: bpf-actlog bpf-argument-filtering kernel:allow kernel:errno kernel:kill_process ke
rnel:kill_thread kernel:log kernel:trace kernel:trap kernel:user_notif
udev: device-cgroup-v1 device-filtering tagging
tim@raffles:~$
tim@raffles:~$ snap version
snap 2.43~pre1+git1601.9d911ca~ubuntu16.04.1
snapd 2.43~pre1+git1601.9d911ca~ubuntu16.04.1
series 16
ubuntu 19.10
kernel 5.5.0-rc2-custom-a
host amd64
tim@raffles:~$
So this library is not a unity library, it is actually a library used with GTK to disable some things that shouldn’t affect snapd. Can you see if you have this package installed libgtk3-nocsd0 ? If you do, can you try:
$ unset LD_PRELOAD
$ snap install hello-world
$ SNAPD_DEBUG=1 SNAP_CONFINE_DEBUG=yes snap run hello-world
(also changed to #snapd since this is about running snapd, not snapcraft)
I do have it installed, but I don’t know why. I don’t think it’s needed on modern KDE. But I have not recently installed it, so the problem is exposed only with the current kernel.
tim@raffles:~$ snap install hello-world
hello-world 6.4 from Canonical✓ installed
tim@raffles:~$ SNAPD_DEBUG=1 SNAP_CONFINE_DEBUG=yes snap run hello-world
2019/12/19 07:14:02.515817 cmd_linux.go:224: DEBUG: restarting into "/snap/core/current/usr/bin/snap"
2019/12/19 07:14:02.531495 cmd_run.go:364: DEBUG: SELinux not enabled
DEBUG: umask reset, old umask was 02
DEBUG: security tag: snap.hello-world.hello-world
DEBUG: executable: /usr/lib/snapd/snap-exec
DEBUG: confinement: non-classic
DEBUG: base snap: core
DEBUG: ruid: 1000, euid: 0, suid: 0
DEBUG: rgid: 1000, egid: 0, sgid: 0
DEBUG: apparmor label on snap-confine is: /snap/core/8329/usr/lib/snapd/snap-confine
DEBUG: apparmor mode is: enforce
DEBUG: creating lock directory /run/snapd/lock (if missing)
DEBUG: opening lock directory /run/snapd/lock
DEBUG: opening lock file: /run/snapd/lock/.lock
DEBUG: sanity timeout initialized and set for 30 seconds
DEBUG: acquiring exclusive lock (scope (global), uid 0)
DEBUG: sanity timeout reset and disabled
DEBUG: ensuring that snap mount directory is shared
DEBUG: unsharing snap namespace directory
cannot self-bind mount /run/snapd/ns: Cannot allocate memory
My rough guess is is trying to preload this library (AFAIK it’s something to disable Gtk3 client side decorations), therefore it sets LD_PRELOAD. When s-c is executed, it placed under the snap-confine specific apparmor profile automatically by the kernel. Next, when ld.so attempts to load the libraries and fails due to AppArmor blocking the access.
I’m slightly confused at this point, IIRC setuid binaries get special treatment so that LD_PRELOAD is automatically dropped on exec (cc @jdstrand).
LD_PRELOAD can be used with setuid binaries so long as a few conditions are met. From man ld.so:
LD_PRELOAD
...
In secure-execution mode, preload pathnames containing slashes are
ignored. Furthermore, shared objects are preloaded only from the
standard search directories and only if they have set-user-ID mode
bit enabled (which is not typical).
Note, in addition to the LD_PRELOAD env var, /etc/ld.so.preload can also be used to specify additional libraries to load (again, man ld.so). It sounds like the snap-confine profile is doing precisely what it is supposed to be doing.
@timattrn - if you have a library that meets the above conditions but don’t know why it is present, I strongly suggest investigating further.
Installed libgtk3-nocsd0, there is something off with the library:
guest@ubuntu:~$ ls -la /usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0
-rwSr--r-- 1 root root 26616 Mar 3 2018 /usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0
This would explain why ld.so tries to load it.
@timattrn - now that the cause is known, you can work around this by adding to /var/lib/snapd/apparmor/snap-confine/gtk3-nocsd:
/usr/lib/@{multiarch}/libgtk3-nocsd.so* mr,
then run: sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap-confine*.
We’ll discuss the next steps for the package in the bug. IMO, a package should not be putting its library in the address space of (and potentially overriding functionality in) setuid executables without explicit admin consent.
Thanks for your investigation. I have been looking into why this package is installed… it seems that it is not maintained, and the workaround it provides to help KDE is not needed any more. So can I remove.
Is it a coincidence that I see this with more up to date kernel?
I removed the library. Still problems, but different ones. This time the denies are with cups-browsed which is a core package for Ubuntu, I think.
tim@raffles:~$ SNAPD_DEBUG=1 snap run hello-world
2019/12/21 10:33:14.590030 cmd_linux.go:224: DEBUG: restarting into "/snap/core/current/usr/bin/snap"
2019/12/21 10:33:14.607107 cmd_run.go:364: DEBUG: SELinux not enabled
DEBUG: umask reset, old umask was 02
DEBUG: security tag: snap.hello-world.hello-world
DEBUG: executable: /usr/lib/snapd/snap-exec
DEBUG: confinement: non-classic
DEBUG: base snap: core
DEBUG: ruid: 1000, euid: 0, suid: 0
DEBUG: rgid: 1000, egid: 0, sgid: 0
DEBUG: apparmor label on snap-confine is: /snap/core/8335/usr/lib/snapd/snap-confine
DEBUG: apparmor mode is: enforce
DEBUG: creating lock directory /run/snapd/lock (if missing)
DEBUG: opening lock directory /run/snapd/lock
DEBUG: opening lock file: /run/snapd/lock/.lock
DEBUG: sanity timeout initialized and set for 30 seconds
DEBUG: acquiring exclusive lock (scope (global), uid 0)
DEBUG: sanity timeout reset and disabled
DEBUG: ensuring that snap mount directory is shared
DEBUG: unsharing snap namespace directory
cannot self-bind mount /run/snapd/ns: Cannot allocate memory
tim@raffles:~$ journalctl -e --no-pager | grep DENIED
Dec 20 11:06:34 raffles audit[30229]: AVC apparmor=“DENIED” operation=“open” profile
="/usr/sbin/cups-browsed" name="/usr/share/cups/locale/" pid=30229 comm=“cups-browse
d” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0
Dec 20 11:06:34 raffles kernel: audit: type=1400 audit(1576800394.135:150): apparmor
=“DENIED” operation=“open” profile="/usr/sbin/cups-browsed" name="/usr/share/cups/lo
cale/" pid=30229 comm=“cups-browsed” requested_mask=“r” denied_mask=“r” fsuid=0 ouid
=0
Dec 20 11:06:34 raffles kernel: audit: type=1400 audit(1576800394.135:151): apparmor
=“DENIED” operation=“open” profile="/usr/sbin/cups-browsed" name="/usr/share/locale/
" pid=30229 comm=“cups-browsed” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0
Dec 20 11:06:34 raffles audit[30229]: AVC apparmor=“DENIED” operation=“open” profile
="/usr/sbin/cups-browsed" name="/usr/share/locale/" pid=30229 comm=“cups-browsed” re
quested_mask=“r” denied_mask=“r” fsuid=0 ouid=0
Dec 21 10:28:12 raffles audit[3134]: AVC apparmor=“DENIED” operation=“open” profile=
“/usr/sbin/cups-browsed” name="/usr/share/cups/locale/" pid=3134 comm=“cups-browsed”
requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0
Dec 21 10:28:12 raffles audit[3134]: AVC apparmor=“DENIED” operation=“open” profile=
“/usr/sbin/cups-browsed” name="/usr/share/locale/" pid=3134 comm=“cups-browsed” requ
ested_mask=“r” denied_mask=“r” fsuid=0 ouid=0
Dec 21 10:28:12 raffles kernel: audit: type=1400 audit(1576884492.336:152): apparmor
=“DENIED” operation=“open” profile="/usr/sbin/cups-browsed" name="/usr/share/cups/lo
cale/" pid=3134 comm=“cups-browsed” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=
0
Dec 21 10:28:12 raffles kernel: audit: type=1400 audit(1576884492.336:153): apparmor
=“DENIED” operation=“open” profile="/usr/sbin/cups-browsed" name="/usr/share/locale/
" pid=3134 comm=“cups-browsed” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0
tim@raffles:~$
This is unrelated to snaps or snapd. I suggest you file a bug with your distro.
This one is interesting though. Maybe @zyga-snapd has some ideas. From the manpage ENOMEM is when The kernel could not allocate a free page to copy filenames or data into., but I haven’t looked at the kernel side to check other possibilities. Does the problem persist after a reboot?
The problem happens after reboots, on both 5.5RC1 and 5.5RC2
This issue is occurring for me on Manjaro, however snaps DO work on 5.4, it’s only any version of 5.5 where they do not work, however last I checked they work on 5.5 in vanilla Arch. journalctl -e --no-pager | grep DENIED returns 0 results, I’ve also tried it just in case with grep -i DENIED, also no results. Therefore, the workaround suggested here doesn’t really help. snap debug confinement returns strict. The message I get when trying to run any snap is cannot self-bind mount /run/snapd/ns: Cannot allocate memory, same as @timattrn. That is the only message I can get, and it happens with all of the snaps I’ve tried (electronplayer, snap-store and a couple others).
SNAPD_DEBUG=1 snap run snap-store
2019/12/23 15:29:28.758633 cmd_linux.go:70: DEBUG: re-exec not supported on distro "manjaro" yet
2019/12/23 15:29:28.763249 cmd_run.go:364: DEBUG: SELinux not enabled
DEBUG: umask reset, old umask was 022
DEBUG: security tag: snap.snap-store.snap-store
DEBUG: executable: /usr/lib/snapd/snap-exec
DEBUG: confinement: non-classic
DEBUG: base snap: core18
DEBUG: ruid: 1000, euid: 0, suid: 0
DEBUG: rgid: 1000, egid: 0, sgid: 0
DEBUG: apparmor label on snap-confine is: /usr/lib/snapd/snap-confine
DEBUG: apparmor mode is: enforce
DEBUG: creating lock directory /run/snapd/lock (if missing)
DEBUG: opening lock directory /run/snapd/lock
DEBUG: opening lock file: /run/snapd/lock/.lock
DEBUG: sanity timeout initialized and set for 30 seconds
DEBUG: acquiring exclusive lock (scope (global), uid 0)
DEBUG: sanity timeout reset and disabled
DEBUG: ensuring that snap mount directory is shared
DEBUG: unsharing snap namespace directory
cannot self-bind mount /run/snapd/ns: Cannot allocate memory
