A few months ago, I installed snap on a small server with Red Hat Enterprise Linux. It worked fine, but when I now try to run the snap Certbot nothing happens at all. I do not even get an error message.
However, when I do run sudo snap version I get (after a few minutes!) the message snapd unavailable.
Does somebody know what could be wrong, and how I can restore the installation?
Distribution: Red Hat Enterprise Linux 8.5
Snap version: 2.53.4-1.el8
I already tried the commands below, but they didn’t have any effect (?):
I run the command sudo sealert -a /var/log/audit/audit.log but didn’t find an issue that could be related.
Tried restarting snapd with sudo systemctl restart snapd snapd.socket
but it didn’t have any effect.
Reload systemd with sudo systemctl daemon-reload but also didn’t have any effect.
Output of sudo journalctl --no-pager -u snapd is:
-- Logs begin at Mon 2021-12-20 23:44:07 CET, end at Tue 2021-12-21 17:20:25 CET. --
Dec 20 23:44:10 IntelNUC systemd[1]: Starting Snap Daemon...
Dec 20 23:44:10 IntelNUC snapd[975]: AppArmor status: apparmor not enabled
Dec 20 23:44:10 IntelNUC snapd[975]: daemon.go:242: started snapd/2.53.4-1.el8 (series 16; classic; devmode) rhel/8.5 (amd64) linux/4.18.0-348.2.1.el8_5.x86_.
Dec 20 23:44:10 IntelNUC snapd[975]: daemon.go:335: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Dec 20 23:44:10 IntelNUC systemd[1]: Started Snap Daemon.
Dec 21 17:09:49 IntelNUC systemd[1]: Stopping Snap Daemon...
Dec 21 17:09:49 IntelNUC snapd[975]: main.go:155: Exiting on terminated signal.
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: Killing process 975 (snapd) with signal SIGKILL.
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: Failed with result 'timeout'.
Dec 21 17:11:19 IntelNUC systemd[1]: Stopped Snap Daemon.
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: Triggering OnFailure= dependencies.
Dec 21 17:11:19 IntelNUC systemd[1]: Starting Snap Daemon...
Dec 21 17:11:19 IntelNUC snapd[38937]: AppArmor status: apparmor not enabled
Dec 21 17:11:19 IntelNUC snapd[38937]: daemon.go:242: started snapd/2.53.4-1.el8 (series 16; classic; devmode) rhel/8.5 (amd64) linux/4.18.0-348.2.1.el8_5.x86_.
Dec 21 17:11:20 IntelNUC snapd[38937]: daemon.go:335: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Dec 21 17:11:20 IntelNUC systemd[1]: Started Snap Daemon
AppArmor is of course not enabled because snap(-d) is run on RHEL which uses SELinux.
Output of sudo snap version is:
snap 2.53.4-1.el8
snapd unavailable
series -
Output of sudo snap changes is:
error: cannot communicate with server: timeout exceeded while waiting for response
Okay, so there is a timeout, but why?
Output of sudo snap debug state /var/lib/snapd/state.json is:
ID Status Spawn Ready Label Summary
16 Done yesterday at 23:42 CET yesterday at 23:42 CET refresh-snap Refresh all snaps: no updates
17 Done yesterday at 23:42 CET yesterday at 23:42 CET refresh-snap Refresh all snaps: no updates
You did not really turned SELinux of with ‘setenfoce 0’; you just set it to permissive. The advantage here is that events are still logged.
So no what?
Setenforce to 0 again, run some stuff like restarting snapd snapd.socket and maybe even try installing something or fetching updates. Then turn SELinux back on
This will print output for a .te file of everything that happened in the last 10 minutes. It might include booleans you could enable, but if you want you could just create a policy for that output by replacing -m with -M. The output will show you instructions in how to install your just created SELinux module
If you prefer you can also restart snapd when it’s set to permissive (and turn it back on again) and paste the output here so we can discuss it first
I see the same issue on Centos8 and Rocky.
When It appeared ~ few days ago snapd re-installation helped until next reboot.
Now snapd re-installation does not help. Beneath are the log immediately after re-installation.
First “snapd version” see snapd, the next does not.
I did nothing between the commands:
[root@]# snap version
snap 2.53.4-1.el8
snapd 2.53.4-1.el8
series 16
rocky 8.5
kernel 4.18.0-348.2.1.el8_5.x86_64
[root@]# snap version
snap 2.53.4-1.el8
snapd unavailable
series -
[root@]# snap version
snap 2.53.4-1.el8
snapd unavailable
series -
[root@]# setenforce 0
[root@]# snap version
snap 2.53.4-1.el8
snapd unavailable
series -
[root@]# systemctl status snapd.socket
Loaded: loaded (/usr/lib/systemd/system/snapd.socket; enabled; vendor preset: disabled)
Active: active (running) ...
[root@]# systemctl status snapd
Loaded: loaded (/usr/lib/systemd/system/snapd.service; disabled; vendor preset: disabled)
Active: active (running) ...
Dec 21 Starting Snap Daemon...
Dec 21 AppArmor status: apparmor not enabled
Dec 21 daemon.go:242: started snapd/2.53.4-1.el8 (series 16; classic; devmode) rocky/8.5 (amd64) linux/4.18.0-3>
Dec 21 daemon.go:335: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
Dec 21 helpers.go:105: error trying to compare the snap system key: system-key missing on disk
Dec 21 Started Snap Daemon.
---> Package snap-confine.x86_64 0:2.53.2-2.el7 will be updated
---> Package snap-confine.x86_64 0:2.53.4-1.el7 will be an update
---> Package snapd.x86_64 0:2.53.2-2.el7 will be updated
---> Package snapd.x86_64 0:2.53.4-1.el7 will be an update
---> Package snapd-selinux.noarch 0:2.53.2-2.el7 will be updated
---> Package snapd-selinux.noarch 0:2.53.4-1.el7 will be an update
and it stopped working. In permissive mode, it seems fine although I didn’t test much. After reboot, snap refresh, list and some more, it looks like it’s related to timedated:
The sssd_selinux_manager_t is “normal” and proc_xen_t:dir is /proc/xen because it’s running on a Xen VM. I guess it’s scanning the whole /proc directory? (Why?) Either way, those two were also there before the update…
I’ve updated the Fedora and EPEL packages to 2.54.1 which already includes the necessary update of the policy. As always the packages go through the testing repository first:
snapd seems to call systemd-detect-virt to detect virtualization and that is denied by selinux as it’s running as snappy_t. snapd is running but I don’t know how important it is to correctly know if this is running in a virtual machine or not…
I belive this PR should fix your problem: https://github.com/snapcore/snapd/pull/11194 However, as I mentioned before, I don’t use Xen, so it’d be great if you gave it a try. You can download the snapd source tree a from that PR, then dnf build-dep snapd -y, enter data/selinux directory, run make, install the module (semodule -i snappy.pp.bz2 shall install the module with a higher priority), restart snapd and check whether the denial is still getting logged.
Thanks. I have tested it. Running CentOS 7 I have pulled the fc, if, te files from the branch, compiled and installed it manually:
# make -f /usr/share/selinux/devel/Makefile snappy.pp DISTRO=rhel7
Compiling targeted snappy module
/usr/bin/checkmodule: loading policy configuration from tmp/snappy.tmp
/usr/bin/checkmodule: policy configuration loaded
/usr/bin/checkmodule: writing binary representation (version 19) to tmp/snappy.mod
Creating targeted snappy.pp policy package
rm tmp/snappy.mod tmp/snappy.mod.fc
# semodule -i snappy.pp
libsemanage.semanage_direct_install_info: Overriding snappy module at lower priority 200 with module at priority 400.
Restarting snapd doesn’t show any avc denials anymore. So it’s working. I also find a fitting rule with sesearch. So it looks good to me.
However, what kind of confused me now is the permissive attribute on snapd_t (which already was there before as you can see in the audit log 5 days ago).