Snapd unavailable - Red Hat Enterprise Linux

snapd
1

Hello Snap developers and administrators!

A few months ago, I installed snap on a small server with Red Hat Enterprise Linux. It worked fine, but when I now try to run the snap Certbot nothing happens at all. I do not even get an error message.

However, when I do run sudo snap version I get (after a few minutes!) the message snapd unavailable.

Does somebody know what could be wrong, and how I can restore the installation?

Distribution: Red Hat Enterprise Linux 8.5
Snap version: 2.53.4-1.el8

I already tried the commands below, but they didn’t have any effect (?):

sudo systemctl enable snapd
sudo systemctl start snapd

Yours sincerely,

Scott Trakker

2

I’m having this same issue on multiple servers. I managed to fixed one, but I’m unable to get that on others:

$ SNAPD_DEBUG=1 snap version
2021/12/21 07:10:40.154300 tool_linux.go:68: DEBUG: re-exec not supported on distro "almalinux" yet
snap    2.53.4-1.el8
snapd   unavailable
series  -


$ systemctl status snapd
● snapd.service - Snap Daemon
  Loaded: loaded (/usr/lib/systemd/system/snapd.service; disabled; vendor preset: disabled)
  Active: active (running) since Tue 2021-12-21 07:06:54 CET; 4min 53s ago
Main PID: 87577 (snapd)
    Tasks: 8 (limit: 4936)
  Memory: 24.6M
  CGroup: /system.slice/snapd.service
          └─87577 /usr/libexec/snapd/snapd

Dec 21 07:06:54 - systemd[1]: Starting Snap Daemon...
Dec 21 07:06:54 - snapd[87577]: AppArmor status: apparmor not enabled
Dec 21 07:06:54 - snapd[87577]: daemon.go:242: started snapd/2.53.4-1.el8 (series 16; classic; devmode) almalinux/8.5 (amd64) linux/4.18.0-348.2.1.el8_5.x86_.
Dec 21 07:06:54 - snapd[87577]: daemon.go:335: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Dec 21 07:06:54 - systemd[1]: Started Snap Daemon.


$ systemctl status snapd.socket
● snapd.socket - Socket activation for snappy daemon
  Loaded: loaded (/usr/lib/systemd/system/snapd.socket; enabled; vendor preset: disabled)
  Active: active (running) since Tue 2021-12-21 07:06:54 CET; 4min 55s ago
  Listen: /run/snapd.socket (Stream)
          /run/snapd-snap.socket (Stream)
    Tasks: 0 (limit: 4936)
  Memory: 4.0K
  CGroup: /system.slice/snapd.socket

Dec 21 07:06:54 - systemd[1]: Starting Socket activation for snappy daemon.
Dec 21 07:06:54 - systemd[1]: Listening on Socket activation for snappy daemon.

Restarting or reinstalling snapd does not change this

(note: this is AlmaLinux, but that’s similar to RHEL)

1 Like
3

Found an SELinux related issue. See Snap.d error: cannot communicate with server connection refused for a fix for that

1 Like
4

I did some debugging but still without success:

  • I run the command sudo sealert -a /var/log/audit/audit.log but didn’t find an issue that could be related.

  • Tried restarting snapd with sudo systemctl restart snapd snapd.socket but it didn’t have any effect.

  • Reload systemd with sudo systemctl daemon-reload but also didn’t have any effect.


Output of sudo journalctl --no-pager -u snapd is:

-- Logs begin at Mon 2021-12-20 23:44:07 CET, end at Tue 2021-12-21 17:20:25 CET. --
Dec 20 23:44:10 IntelNUC systemd[1]: Starting Snap Daemon...
Dec 20 23:44:10 IntelNUC snapd[975]: AppArmor status: apparmor not enabled
Dec 20 23:44:10 IntelNUC snapd[975]: daemon.go:242: started snapd/2.53.4-1.el8 (series 16; classic; devmode) rhel/8.5 (amd64) linux/4.18.0-348.2.1.el8_5.x86_.
Dec 20 23:44:10 IntelNUC snapd[975]: daemon.go:335: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Dec 20 23:44:10 IntelNUC systemd[1]: Started Snap Daemon.
Dec 21 17:09:49 IntelNUC systemd[1]: Stopping Snap Daemon...
Dec 21 17:09:49 IntelNUC snapd[975]: main.go:155: Exiting on terminated signal.
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: Killing process 975 (snapd) with signal SIGKILL.
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: Main process exited, code=killed, status=9/KILL
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: Failed with result 'timeout'.
Dec 21 17:11:19 IntelNUC systemd[1]: Stopped Snap Daemon.
Dec 21 17:11:19 IntelNUC systemd[1]: snapd.service: Triggering OnFailure= dependencies.
Dec 21 17:11:19 IntelNUC systemd[1]: Starting Snap Daemon...
Dec 21 17:11:19 IntelNUC snapd[38937]: AppArmor status: apparmor not enabled
Dec 21 17:11:19 IntelNUC snapd[38937]: daemon.go:242: started snapd/2.53.4-1.el8 (series 16; classic; devmode) rhel/8.5 (amd64) linux/4.18.0-348.2.1.el8_5.x86_.
Dec 21 17:11:20 IntelNUC snapd[38937]: daemon.go:335: adjusting startup timeout by 45s (pessimistic estimate of 30s plus 5s per snap)
Dec 21 17:11:20 IntelNUC systemd[1]: Started Snap Daemon

AppArmor is of course not enabled because snap(-d) is run on RHEL which uses SELinux.



Output of sudo snap version is:

snap    2.53.4-1.el8
snapd   unavailable
series  -


Output of sudo snap changes is:

error: cannot communicate with server: timeout exceeded while waiting for response

Okay, so there is a timeout, but why?


Output of sudo snap debug state /var/lib/snapd/state.json is:

ID   Status  Spawn                   Ready                   Label         Summary
16   Done    yesterday at 23:42 CET  yesterday at 23:42 CET  refresh-snap  Refresh all snaps: no updates
17   Done    yesterday at 23:42 CET  yesterday at 23:42 CET  refresh-snap  Refresh all snaps: no updates

I’m out of ideas at the moment.

5

Where did you find the solution you mention in that thread?
I wonder what the problem is, and if there isn’t an easier solution.

6

I did not find this anywhere, but disabled SELinux and looked at the issues afterwards

What if you setenforce 0, restart snapd and snapd.socket?

If that works enable SELinux again and have a look at the audit log

1 Like
7

What if you setenforce 0, restart snapd and snapd.socket?

Great idea!!


When I turn off SELinux with setenforce 0, snap seems to work fine because when I run snap version I get:

snap    2.53.4-1.el8
snapd   2.53.4-1.el8
series  16
rhel    8.5
kernel  4.18.0-348.2.1.el8_5.x86_64

So it is a SELinux issue!


When I turn it on again (with setenforce 1) and run sealert, I only get one alert:

SELinux is preventing /usr/libexec/platform-python3.6 from read access on the directory kpatch.

This doesn’t seem to be related so something else must be happening in the background.


Thanks for your suggestions and help!

Any more ideas :slight_smile: ?


Edit: I fixed the SELinux alert (by running the suggested commands) but this didn’t have any effect. When I now run sealert, it founds 0 alerts.

8

You did not really turned SELinux of with ‘setenfoce 0’; you just set it to permissive. The advantage here is that events are still logged.

So no what?

Setenforce to 0 again, run some stuff like restarting snapd snapd.socket and maybe even try installing something or fetching updates. Then turn SELinux back on

THEN we’re going to see what just happened

Run: ausearch -ts recent --raw | audit2allow -m snapd-selinux

This will print output for a .te file of everything that happened in the last 10 minutes. It might include booleans you could enable, but if you want you could just create a policy for that output by replacing -m with -M. The output will show you instructions in how to install your just created SELinux module

If you prefer you can also restart snapd when it’s set to permissive (and turn it back on again) and paste the output here so we can discuss it first

1 Like
9

I see the same issue on Centos8 and Rocky. When It appeared ~ few days ago snapd re-installation helped until next reboot. Now snapd re-installation does not help. Beneath are the log immediately after re-installation. First “snapd version” see snapd, the next does not. I did nothing between the commands:

[root@]# snap version
snap    2.53.4-1.el8
snapd   2.53.4-1.el8
series  16
rocky   8.5
kernel  4.18.0-348.2.1.el8_5.x86_64
[root@]# snap version
snap    2.53.4-1.el8
snapd   unavailable
series  -
[root@]# snap version
snap    2.53.4-1.el8
snapd   unavailable
series  -
[root@]# setenforce 0
[root@]# snap version
snap    2.53.4-1.el8
snapd   unavailable
series  -
[root@]# systemctl status snapd.socket
Loaded: loaded (/usr/lib/systemd/system/snapd.socket; enabled; vendor preset: disabled)
Active: active (running) ...
[root@]# systemctl status snapd
  Loaded: loaded (/usr/lib/systemd/system/snapd.service; disabled; vendor preset: disabled)
  Active: active (running) ...
    Dec 21  Starting Snap Daemon...
    Dec 21 AppArmor status: apparmor not enabled
    Dec 21 daemon.go:242: started snapd/2.53.4-1.el8 (series 16; classic; devmode) rocky/8.5 (amd64) linux/4.18.0-3>
     Dec 21 daemon.go:335: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
     Dec 21 helpers.go:105: error trying to compare the snap system key: system-key missing on disk
     Dec 21 Started Snap Daemon.
10

I have the same issue. Updated this on CentOS 7

---> Package snap-confine.x86_64 0:2.53.2-2.el7 will be updated
---> Package snap-confine.x86_64 0:2.53.4-1.el7 will be an update
---> Package snapd.x86_64 0:2.53.2-2.el7 will be updated
---> Package snapd.x86_64 0:2.53.4-1.el7 will be an update
---> Package snapd-selinux.noarch 0:2.53.2-2.el7 will be updated
---> Package snapd-selinux.noarch 0:2.53.4-1.el7 will be an update

and it stopped working. In permissive mode, it seems fine although I didn’t test much. After reboot, snap refresh, list and some more, it looks like it’s related to timedated:

# audit2allow < /var/log/audit/audit.log


#============= snappy_t ==============
allow snappy_t proc_xen_t:dir search;
allow snappy_t systemd_timedated_t:dbus send_msg;

#============= sssd_selinux_manager_t ==============
allow sssd_selinux_manager_t init_t:unix_stream_socket append;

#============= systemd_timedated_t ==============
allow systemd_timedated_t snappy_t:dbus send_msg;

The sssd_selinux_manager_t is “normal” and proc_xen_t:dir is /proc/xen because it’s running on a Xen VM. I guess it’s scanning the whole /proc directory? (Why?) Either way, those two were also there before the update…

I don’t know about EL 8.

11

Thank you very much: I succeeded!


I did what you described:

[NNNNNN@IntelNUC ~]$ sudo setenforce 0
[sudo] password for NNNNNN: 
[NNNNNN@IntelNUC ~]$ getenforce
Permissive
[NNNNNN@IntelNUC ~]$ sudo systemctl restart snapd snapd.socket
[NNNNNN@IntelNUC ~]$ sudo snap refresh
All snaps up to date.
[NNNNNN@IntelNUC ~]$ sudo snap version
snap    2.53.4-1.el8
snapd   2.53.4-1.el8
series  16
rhel    8.5
kernel  4.18.0-348.2.1.el8_5.x86_64
[NNNNNN@IntelNUC ~]$ sudo setenforce 1
[NNNNNN@IntelNUC ~]$ sudo ausearch -ts recent --raw | audit2allow -m snapd-selinux

module snapd-selinux 1.0;

require {
	type systemd_timedated_t;
	type snappy_t;
	class dbus send_msg;
}

#============= snappy_t ==============
allow snappy_t systemd_timedated_t:dbus send_msg;

#============= systemd_timedated_t ==============
allow systemd_timedated_t snappy_t:dbus send_msg;
[NNNNNN@IntelNUC ~]$ sudo ausearch -ts recent --raw | audit2allow -M snapd-selinux
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i snapd-selinux.pp

[NNNNNN@IntelNUC ~]$ sudo semodule -i snapd-selinux.pp
[NNNNNN@IntelNUC ~]$ getenforce
Enforcing
[NNNNNN@IntelNUC ~]$ sudo snap version
snap    2.53.4-1.el8
snapd   2.53.4-1.el8
series  16
rhel    8.5
kernel  4.18.0-348.2.1.el8_5.x86_64

After this, I was able to run certbot succesfully!


A million thanks!! :slight_smile:

12

Opened issue SeLinux issue with 2.53.4

1 Like
14

I’ve updated the Fedora and EPEL packages to 2.54.1 which already includes the necessary update of the policy. As always the packages go through the testing repository first:

15

Thanks. I have just installed it from epel-testing and it’s starting again in enforcing mode.

There is still the xen avc deny as my server is running on a xen host:

type=PROCTITLE msg=audit(1640771959.147:236972): proctitle="systemd-detect-virt"
type=SYSCALL msg=audit(1640771959.147:236972): arch=c000003e syscall=2 success=no exit=-2 a0=56171189acc3 a1=80000 a2=1b6 a3=24 items=0 ppid=21108 pid=21113 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-detect-" exe="/usr/bin/systemd-detect-virt" subj=system_u:system_r:snappy_t:s0 key=(null)
type=AVC msg=audit(1640771959.147:236972): avc:  denied  { search } for  pid=21113 comm="systemd-detect-" name="xen" dev="proc" ino=4026532003 scontext=system_u:system_r:snappy_t:s0 tcontext=system_u:object_r:proc_xen_t:s0 tclass=dir permissive=1

snapd seems to call systemd-detect-virt to detect virtualization and that is denied by selinux as it’s running as snappy_t. snapd is running but I don’t know how important it is to correctly know if this is running in a virtual machine or not…

16

Thanks, I’ll take a look. We do not use Xen in testing, so this never comes up.

17

I belive this PR should fix your problem: https://github.com/snapcore/snapd/pull/11194 However, as I mentioned before, I don’t use Xen, so it’d be great if you gave it a try. You can download the snapd source tree a from that PR, then dnf build-dep snapd -y, enter data/selinux directory, run make, install the module (semodule -i snappy.pp.bz2 shall install the module with a higher priority), restart snapd and check whether the denial is still getting logged.

18

Thanks. I have tested it. Running CentOS 7 I have pulled the fc, if, te files from the branch, compiled and installed it manually:

# make -f /usr/share/selinux/devel/Makefile snappy.pp DISTRO=rhel7
Compiling targeted snappy module
/usr/bin/checkmodule:  loading policy configuration from tmp/snappy.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 19) to tmp/snappy.mod
Creating targeted snappy.pp policy package
rm tmp/snappy.mod tmp/snappy.mod.fc
# semodule -i snappy.pp
libsemanage.semanage_direct_install_info: Overriding snappy module at lower priority 200 with module at priority 400.

Restarting snapd doesn’t show any avc denials anymore. So it’s working. I also find a fitting rule with sesearch. So it looks good to me.

However, what kind of confused me now is the permissive attribute on snapd_t (which already was there before as you can see in the audit log 5 days ago).

# semanage permissive -l 

Customized Permissive Types


Builtin Permissive Types 

snappy_confine_t
snappy_t
snappy_unconfined_snap_t
snappy_cli_t
snappy_mount_t

Is that a good idea?

# For development purposes, snappy_t domain is to be marked permissive
permissive snappy_t;