Getrlimit blocked by seccomp on focal/arm64

robertliu · May 6, 2020, 1:49pm

Hi,

I tried to run my snap on focal/arm64, but the getrlimit calls are blocked by seccomp. The snap works well on focal/amd64.

May 04 18:17:12 ubuntu audit[5830]: SECCOMP auid=4294967295 uid=0 gid=0 ses=4294967295 pid=5830 comm="lighttpd" exe="/snap/adsb-box/259/usr/sbin/lighttpd" sig=0 arch=c00000b7 syscall=163 compat=0 ip=0xffffabc48d08 code=0x50000
May 04 18:17:12 ubuntu audit[5830]: SECCOMP auid=4294967295 uid=0 gid=0 ses=4294967295 pid=5830 comm="lighttpd" exe="/snap/adsb-box/259/usr/sbin/lighttpd" sig=0 arch=c00000b7 syscall=163 compat=0 ip=0xffffabc48d08 code=0x50000
May 04 18:17:12 ubuntu adsb-box.lighttpd[5830]: 2020-05-04 18:17:12: (server.c.738) couldn't get 'max filedescriptors' Operation not permitted
May 04 18:17:12 ubuntu systemd[1]: snap.adsb-box.lighttpd.service: Main process exited, code=exited, status=255/EXCEPTION

$ snappy-debug.scmp-sys-resolver -a aarch64 163
getrlimit

I did a quick search and found it seems only docker-support allows getrlimit calls. I wonder if there is any other interface or solution?

jdstrand · May 6, 2020, 9:01pm

That is quite curious since getrlimit is allowed by the default template: https://github.com/snapcore/snapd/blob/master/interfaces/seccomp/template.go#L170

You can verify that the policy has the access with grep getrlimit /var/lib/snapd/seccomp/snap.adsb-box.lighttpd.src. Perhaps lighttpd implements its own syscall filter and left out getrlimit?

robertliu · May 8, 2020, 4:27am

It does have getrlimit in the seccomp src file.
According to my test results, this issue doesn’t happen on focal/amd64 and focal/armhf.

I also did a test with hello snap and observed this issue.

ubuntu@ubuntu:~$ sudo snap install hello
hello 2.10 from Canonical✓ installed
ubuntu@ubuntu:~$ sudo snap run --shell hello.hello
root@ubuntu:/home/ubuntu# ulimit -a
bash: ulimit: core file size: cannot get limit: Operation not permitted
bash: ulimit: data seg size: cannot get limit: Operation not permitted
bash: ulimit: scheduling priority: cannot get limit: Operation not permitted
bash: ulimit: file size: cannot get limit: Operation not permitted
bash: ulimit: pending signals: cannot get limit: Operation not permitted
bash: ulimit: max locked memory: cannot get limit: Operation not permitted
bash: ulimit: max memory size: cannot get limit: Operation not permitted
bash: ulimit: open files: cannot get limit: Operation not permitted
pipe size            (512 bytes, -p) 8
bash: ulimit: POSIX message queues: cannot get limit: Operation not permitted
bash: ulimit: real-time priority: cannot get limit: Operation not permitted
bash: ulimit: stack size: cannot get limit: Operation not permitted
bash: ulimit: cpu time: cannot get limit: Operation not permitted
bash: ulimit: max user processes: cannot get limit: Operation not permitted
bash: ulimit: virtual memory: cannot get limit: Operation not permitted
bash: ulimit: file locks: cannot get limit: Operation not permitted
root@ubuntu:/home/ubuntu# exit

In the journal log, it also has SECCOMP error of syscall=163

May 08 04:12:47 ubuntu sudo[4476]:   ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/snap run --shell hello.hello
May 08 04:12:47 ubuntu sudo[4476]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
May 08 04:12:48 ubuntu audit[4477]: SECCOMP auid=1000 uid=0 gid=0 ses=1 pid=4477 comm="bash" exe="/bin/bash" sig=0 arch=c00000b7 syscall=163 compat=0 ip=0xffffb399ad08 code=0x50000
May 08 04:12:48 ubuntu kernel: kauditd_printk_skb: 5 callbacks suppressed

Does this mean there’s something wrong with kernel or system settings?

jamesh · May 8, 2020, 5:50am

Looking at libseccomp, it seems that it might not believe getrlimit exists on armhf or arm64:

github.com

seccomp/libseccomp/blob/main/src/syscalls.csv#L131


getpmsg,188,181,181,PNR,PNR,208,174,174,PNR,PNR,187,187,PNR,188,188,PNR
getppid,64,110,110,64,173,64,108,108,64,64,64,64,173,64,64,64
getpriority,96,140,140,96,141,96,137,137,96,96,96,96,141,96,96,96
getrandom,355,318,318,384,278,353,313,317,339,339,359,359,278,349,349,373
getresgid,171,120,120,171,150,191,118,118,171,171,170,170,150,171,211,171
getresgid32,211,PNR,PNR,211,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,211,PNR,211
getresuid,165,118,118,165,148,186,116,116,165,165,165,165,148,165,209,165
getresuid32,209,PNR,PNR,209,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,209,PNR,209
getrlimit,76,97,97,PNR,163,76,95,95,76,76,76,76,163,76,191,76
get_robust_list,312,274,531,339,100,310,269,273,290,290,299,299,100,305,305,312
getrusage,77,98,98,77,165,77,96,96,77,77,77,77,165,77,77,77
getsid,147,124,124,147,156,151,122,122,147,147,147,147,156,147,147,147
getsockname,367,51,51,286,204,172,50,50,44,44,331,331,204,367,367,345
getsockopt,365,55,542,295,209,173,54,54,182,182,340,340,209,365,365,354
get_thread_area,244,211,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR
gettid,224,186,186,224,178,222,178,178,206,206,207,207,178,236,236,224
gettimeofday,78,96,96,78,169,78,94,94,78,78,78,78,169,78,78,78
get_tls,PNR,PNR,PNR,983046,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR
getuid,24,102,102,24,174,24,100,100,24,24,24,24,174,24,199,24
getuid32,199,PNR,PNR,199,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,PNR,199,PNR,199
getxattr,229,191,191,229,8,227,183,183,241,241,212,212,8,227,227,229

The 5th and 6th columns in the file are armhf and arm64 respectively. Presumably this means it gets left out of the compiled seccomp program on those architectures.

mborzecki · May 8, 2020, 7:24am

Looks like getrlimit is listed here for arm64 but not form armhf: https://fedora.juszkiewicz.com.pl/syscalls.html is it a bug in libseccomp?

jamesh · May 8, 2020, 8:08am

The setrlimit man page mentions glibc implementing the function in terms of the Linux specific prlimit64 syscall due to the older syscalls using smaller integer types on 32-bit systems.

As a new architecture, maybe the syscalls weren’t present at some point on ARM EABI and ARM64?

jamesh · May 8, 2020, 8:47am

It looks like getrlimit/setrlimit definitely should exist for ARM64, so this looks like a bug in libseccomp:

github.com

torvalds/linux/blob/master/arch/arm64/include/uapi/asm/unistd.h

/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
/*
 * Copyright (C) 2012 ARM Ltd.
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License version 2 as
 * published by the Free Software Foundation.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */

#define __ARCH_WANT_RENAMEAT
#define __ARCH_WANT_NEW_STAT
#define __ARCH_WANT_SET_GET_RLIMIT

This file has been truncated. show original

getrlimit was explicitly excluded from ARM EABI due to not being able to represent 64-bit limits. It still has setrlimit, since there is no chance of overflow in extending smaller types from user space.

jdstrand · May 8, 2020, 4:58pm

I observed that on my 20.04 amd64, I was getting UNKNOWN when calling scmp_sys_resolver -a aarch64 163 and I recalled that the snappy-debug snap uses base: core18 and stages scmp_sys_resolver from 18.04, and it resolves. Here is what I found:

xenial amd64, libseccomp 2.4.1-0ubuntu0.16.04.2 (this is from xenial-security)

$ lsb_release -d
Description:	Ubuntu 16.04.6 LTS
$ scmp_sys_resolver -a aarch64 163
getrlimit
$ scmp_sys_resolver -a aarch64 getrlimit
163

bionic amd64, libseccomp 2.4.1-0ubuntu0.18.04.2 (this is from bionic-security)

$ lsb_release -d
Description:	Ubuntu 18.04.4 LTS
$ scmp_sys_resolver -a aarch64 163
getrlimit
$ scmp_sys_resolver -a aarch64 getrlimit
163

focal amd64, libseccomp 2.4.3-1ubuntu1 – BROKEN

$ lsb_release -d
Description:	Ubuntu 20.04 LTS
$ scmp_sys_resolver -a aarch64 163
UNKNOWN
$ scmp_sys_resolver -a aarch64 getrlimit
-10180

UC16 armhf with core from candidate, libseccomp 2.4.1-0ubuntu0.16.04.3 (as seen in /usr/share/snapd/dpkg.list. This comes from https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+packages, which is patched for 5.4 syscalls from 2.4.2) – BROKEN
```
$ grep DESC /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu Core 16"
$ scmp_sys_resolver -a aarch64 163
UNKNOWN
$ scmp_sys_resolver -a aarch64 getrlimit
-10180
```
UC16 arm64 with core from candidate, libseccomp 2.4.1-0ubuntu0.16.04.3 (as seen in /usr/share/snapd/dpkg.list. This comes from https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+packages, which is patched for 5.4 syscalls from 2.4.2) – BROKEN
```
$ grep DESC /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu Core 16"
$ scmp_sys_resolver -a aarch64 163
UNKNOWN
$ scmp_sys_resolver -a aarch64 getrlimit
-10180
```

The ‘core’ snap (which corresponds to the above UC16 devices) was recently updated in the build ppa to address https://github.com/snapcore/core20/issues/48 with an update to libseccomp to adjust it to add 5.4 syscalls from the 2.4.2 upstream commits (and a corresponding update was made to the snapd snap). Ubuntu 20.04 uses libseccomp 2.4.3 and has the same issue.

Then on the arm64 UC16 system that is broken, I generated the policy and the pfc file with:

$ cp /var/lib/snapd/seccomp/bpf/snap.hello-world.hello-world.src /tmp/snap.hello-world.hello-world.src
$ SNAP_SECCOMP_DEBUG=1 /snap/core/current/usr/lib/snapd/snap-seccomp compile /tmp/snap.hello-world.hello-world.src /tmp/snap.hello-world.hello-world.bin > /tmp/snap.hello-world.hello-world.pfc

I looked for getrlimit in the .src file, and it was there. I looked in the .pfc file and found:

# filter for arch aarch64 (3221225655)
if ($arch == 3221225655)
...
  # filter for syscall "getrlimit" (4294957116) [priority: 65535]
  if ($syscall == 4294957116)
    action ALLOW;
...
# filter for arch arm (1073741864)
if ($arch == 1073741864)
...
  # filter for syscall "getrlimit" (4294957116) [priority: 65535]
  if ($syscall == 4294957116)
    action ALLOW;

(snap-seccomp on arm64 correctly generates a syscall filter for both the native arch, aarch64/arm64 and its compat arch, arm/armhf).

Note: 4294957116−2^32 = -10180 where we were expecting 163 for aarch64 (-10180 is what was returned on the broken systems when running scmp_sys_resolver -a aarch64 getrlimit). The kernel understands getrlimit to be 163 and therefore since the filter is not allowing 163, the kernel is blocking it.

I then downgraded the core snap to what is in stable now (built with 2.4.1-0ubuntu0.16.04.2 as seen in /usr/share/snapd/dpkg.list), then generated a new .pfc file and see:

# filter for arch aarch64 (3221225655)
if ($arch == 3221225655)
...
  # filter for syscall "getrlimit" (163) [priority: 65535]
  if ($syscall == 163)
    action ALLOW;
...
# filter for arch arm (1073741864)
if ($arch == 1073741864)
...
  # filter for syscall "getrlimit" (4294957116) [priority: 65535]
  if ($syscall == 4294957116)
    action ALLOW;

Altogether, I believe this indicates a bug was introduced in libseccomp 2.4.2 that is unaddressed in 2.4.3 where getrlimit on arm64 (at least) is not properly resolving which is affecting the generated policy.

@alexmurray is currently preparing an SRU of 2.4.3 to Ubuntu 16.04, 18.04 and 19.10 so we have the same libseccomp in the core snap, snapd snap, stage-packages and throughout Ubuntu, but we’ll need to make sure this issue is addressed before pushing that update. I filed https://bugs.launchpad.net/ubuntu/+source/libseccomp/+bug/1877633 and assigned it to @alexmurray.

jdstrand · May 8, 2020, 5:05pm

For people experiencing this bug, please refresh to use core and/or snapd from stable instead of candidate. I’ve alerted @mvo to not promote what is in candidate to stable until we have a libseccomp that doesn’t have this regression.

alexmurray · May 11, 2020, 5:08am

I have submitted a PR upstream to libseccomp which should fix this - https://github.com/seccomp/libseccomp/pull/235 - once that (or some other solution) is merged we can look at SRUing that fix as part of the 2.4.3 SRU which is already in progress.