Custom kernel error on readlinkat() in mount namespace

Ummmmh… guess I was too optimistic about apparmor workarounds… :smiley:

Probably looking in the wrong place again.

Tried changing the aa profile for e.g. snap.gimp.gimp from enforce to complain, and tried to disable apparmor completely, but still getting the same error when trying to start gimp.

IS there actually a quick and dirty workaround to take apparmor out of the equation temporarily or convince it that these snaps are perfectly fine? :wink:

using an ubuntu kernel is the easiest “workaround” here i guess

the confinement is a semi-complex communication between snapd and the involved parts, killing off one side of that communication will not help, both sides need adjustment (which is what zyga is working on) to make it work.

1 Like

dang.

ok, thanks, guess I need to work on a different machine then for now.

Going back to an earlier kernel unfortunately is not an option, it results in frequent freezes thanks to still unreliable support for the Raven Ridge processors with integrated Vega graphics…

I’m sure we can get it to work on mainline.

1 Like

dmesg | grep DENIED
[ 36.097077] audit: type=1400 audit(1530124482.591:45): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4830/usr/lib/snapd/snap-confine" pid=2492 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
[ 60.609228] audit: type=1400 audit(1530124507.079:46): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4830/usr/lib/snapd/snap-confine" pid=2570 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”

uname -r
4.18.0-041800rc1-generic

Note, this probably isn’t an apparmor change but a capabilities change in the upstream kernel that now triggers this denial. The snap-confine profile seems to need: ptrace (read) peer=unconfined, to read this file.

Hmm, curiously, this is in /etc/apparmor.d/snap.core.4938.usr.lib.snapd.snap-confine:

    # support for the mount namespace sharing
    capability sys_ptrace,
    # allow snap-confine to read /proc/1/ns/mnt
    ptrace trace peer=unconfined,

It seems 4.18 changed the check from ‘trace’ to ‘read’. @zyga-snapd, we could adjust to have:

ptrace trace peer=unconfined, # 4.17 and earlier
ptrace read peer=unconfined, # 4.18 and later

though @zyga-snapd, a much better fix would be to have only the ‘ptrace read’ rule in the profile and conditionally add the ‘ptrace trace’ if on kernel <=4.17. The trace rule is really powerful and it would be good to avoid it if possible. We should be able to do this based on the #include mechanisms we use for nfs and overlay.

@MartinTheWanderer, as a workaround, you can add to the /etc/apparmor.d/*snap-confine* files (on a line before the final ‘}’ at the end of the file):

ptrace read peer=unconfined,

then run sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*. You will need to redo this when the core snap refreshes until the proper fix is included.

Actually, I discussed this with John and this was an AppArmor change (well, more precisely, an unrelated change turned the read into a trace and AppArmor needed an adjustment to turn it back into a read, but this adjustment didn’t happen until 4.18). The ‘trace’ was undesirable and should’ve always been ‘read’. He is in the process of rolling that out to all Ubuntu kernels. @zyga-snapd, it would be great to default to read when possible. Eg, if <4.18 and isNotUbuntu, add ptrace trace rule (we’d have to wait for those kernels to rollout of course.

I’ve asked John to add a flag for the backports tree so we can interrogate the kernel for this. So, we can check for the flag or 4.18. Anyone who picks up the patch will just get it.

1 Like

@zyga-snapd, I mentioned this to you, but I’ll make the PR for snapd since I’ll work with John on testing the patch.

Thank you for the in-depth analysis and the patch :slight_smile:

So is the patch currently available somewhere? I would be keen to give it a go as kernel 4.18 seems to work much better on my system.

The patch is not proposed yet. Once it is in snapd master you can test it simply by switching to the edge channel with “snap refresh —edge core”

@jzimm - you can use this workaround in the meantime.

Yay!

Huge thanks to @jdstrand and all who helped!

Workaround works like a charm, so I can use my snaps and patiently wait until the official patch is released.

1 Like

Thanks for the tip, will try.

I’ve upgraded the kernel 4.18.0rc4 to see if the boot problem with my ryzen 2200g was still present (surprisingly it seems to boot reliably now, or at least it seems so) but I couldn’t run snaps anymore.
I just wanted to confirm that the workaround works.

ls /etc/apparmor.d/snap-confine
/etc/apparmor.d/snap.core.4917.usr.lib.snapd.snap-confine
/etc/apparmor.d/usr.lib.snapd.snap-confine.real

There were 2 files, the first one had at line 366:
ptrace trace peer=unconfined,
Same thing second file line 354.

It may not be able to run snaps for a different reason. Are there any security denials in the journalctl logs?

It works with the workaround but I have to re-run apparmor_parser after every boot otherwise I see “readlinkat()…” and every snap doesn’t work.

This is the log before the apparmor_parser command

lug 09 19:02:35 ghv kernel: audit: type=1400 audit(1531155755.217:78): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4917/usr/lib/snapd/snap-confine" pid=3919 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
lug 09 19:02:35 ghv kernel: audit: type=1400 audit(1531155755.836:79): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4917/usr/lib/snapd/snap-confine" pid=3933 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
lug 09 19:04:05 ghv kernel: audit: type=1400 audit(1531155845.207:80): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4917/usr/lib/snapd/snap-confine" pid=4259 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
lug 09 19:04:12 ghv kernel: audit: type=1400 audit(1531155852.133:81): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name="/snap/core/4917/usr/lib/snapd/snap-confine" pid=4326 comm=“apparmor_parser”
lug 09 19:04:12 ghv kernel: audit: type=1400 audit(1531155852.153:82): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name="/snap/core/4917/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=4326 comm=“apparmor_parser”
lug 09 19:04:12 ghv kernel: audit: type=1400 audit(1531155852.153:83): apparmor=“STATUS” operation=“profile_replace” info=“same as current profile, skipping” profile=“unconfined” name="/usr/lib/snapd/snap-confine" pid=4327 comm=“apparmor_parser”
lug 09 19:04:12 ghv kernel: audit: type=1400 audit(1531155852.153:84): apparmor=“STATUS” operation=“profile_replace” info=“same as current profile, skipping” profile=“unconfined” name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=4327 comm=“apparmor_parser”

Hi,

I’m seeing this problem on Ubuntu 18.04 with kernel 4.18 and the workaround doesn’t seem to do anything for me.

> ls /etc/apparmor.d/*snap-confine*
/etc/apparmor.d/usr.lib.snapd.snap-confine.real

> tail /etc/apparmor.d/usr.lib.snapd.snap-confine.real
    # from the core snap but we are already inside the constructed mount
    # namespace. Here the apparmor kernel module re-constructs the path to
    # snap-update-ns using the "hostfs" mount entry rather than the more
    # "natural" /snap mount entry but we have no control over that.  This is
    # reported as (LP: #1716339). The variants here represent different
    # locations of snap mount directory across distributions.
    /var/lib/snapd/hostfs/{,var/lib/snapd/}snap/core/*/usr/lib/snapd/snap-update-ns r,

    ptrace read peer=unconfined,
}

> sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*

> gnome-calculator
cannot perform readlinkat() on the mount namespace file descriptor of the init process: Permission denied

> dmesg | grep DENIED         
[   28.836643] audit: type=1400 audit(1534247507.507:50): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=1759 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   29.030691] audit: type=1400 audit(1534247507.699:51): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=1892 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   29.230245] audit: type=1400 audit(1534247507.899:53): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2040 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   29.404266] audit: type=1400 audit(1534247508.075:54): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2165 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   29.665339] audit: type=1400 audit(1534247508.335:55): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2228 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   33.802955] audit: type=1400 audit(1534247512.471:56): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2670 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   39.115107] audit: type=1400 audit(1534247517.783:57): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2958 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   41.458075] audit: type=1400 audit(1534247520.127:58): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=3020 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   94.297198] audit: type=1400 audit(1534247572.951:59): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=3238 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 9039.010150] audit: type=1400 audit(1534256517.733:64): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=19462 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 9058.478617] audit: type=1400 audit(1534256537.201:65): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=19816 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 9083.475983] audit: type=1400 audit(1534256562.197:66): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=19924 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 9113.703202] audit: type=1400 audit(1534256592.425:67): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=19968 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[10614.319740] audit: type=1400 audit(1534258093.050:70): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=21750 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[10637.672492] audit: type=1400 audit(1534258116.406:71): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=21872 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[10862.835626] audit: type=1400 audit(1534258341.566:74): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=22062 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[11041.500133] audit: type=1400 audit(1534258520.235:75): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=22154 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[11752.936363] audit: type=1400 audit(1534259231.670:80): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=22689 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[70917.166364] audit: type=1400 audit(1534318396.649:81): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=27267 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[78768.839106] audit: type=1400 audit(1534326247.974:84): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=30089 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"

Any ideas on why it still doesn’t work?