Custom kernel error on readlinkat() in mount namespace

upcoming
jdstrand

#17

dmesg | grep DENIED
[ 36.097077] audit: type=1400 audit(1530124482.591:45): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4830/usr/lib/snapd/snap-confine" pid=2492 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
[ 60.609228] audit: type=1400 audit(1530124507.079:46): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4830/usr/lib/snapd/snap-confine" pid=2570 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”

uname -r
4.18.0-041800rc1-generic


#18

Note, this probably isn’t an apparmor change but a capabilities change in the upstream kernel that now triggers this denial. The snap-confine profile seems to need: ptrace (read) peer=unconfined, to read this file.


#19

Hmm, curiously, this is in /etc/apparmor.d/snap.core.4938.usr.lib.snapd.snap-confine:

    # support for the mount namespace sharing
    capability sys_ptrace,
    # allow snap-confine to read /proc/1/ns/mnt
    ptrace trace peer=unconfined,

It seems 4.18 changed the check from ‘trace’ to ‘read’. @zyga, we could adjust to have:

ptrace trace peer=unconfined, # 4.17 and earlier
ptrace read peer=unconfined, # 4.18 and later

though @zyga, a much better fix would be to have only the ‘ptrace read’ rule in the profile and conditionally add the ‘ptrace trace’ if on kernel <=4.17. The trace rule is really powerful and it would be good to avoid it if possible. We should be able to do this based on the #include mechanisms we use for nfs and overlay.

@MartinTheWanderer, as a workaround, you can add to the /etc/apparmor.d/*snap-confine* files (on a line before the final ‘}’ at the end of the file):

ptrace read peer=unconfined,

then run sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*. You will need to redo this when the core snap refreshes until the proper fix is included.


#20

Actually, I discussed this with John and this was an AppArmor change (well, more precisely, an unrelated change turned the read into a trace and AppArmor needed an adjustment to turn it back into a read, but this adjustment didn’t happen until 4.18). The ‘trace’ was undesirable and should’ve always been ‘read’. He is in the process of rolling that out to all Ubuntu kernels. @zyga, it would be great to default to read when possible. Eg, if <4.18 and isNotUbuntu, add ptrace trace rule (we’d have to wait for those kernels to rollout of course.


#21

I’ve asked John to add a flag for the backports tree so we can interrogate the kernel for this. So, we can check for the flag or 4.18. Anyone who picks up the patch will just get it.


Snaps are not working with Linux Kernel v4.18-rc
#22

@zyga, I mentioned this to you, but I’ll make the PR for snapd since I’ll work with John on testing the patch.


#23

Thank you for the in-depth analysis and the patch :slight_smile:


#24

So is the patch currently available somewhere? I would be keen to give it a go as kernel 4.18 seems to work much better on my system.


#25

The patch is not proposed yet. Once it is in snapd master you can test it simply by switching to the edge channel with “snap refresh —edge core”


#26

@jzimm - you can use this workaround in the meantime.


#27

Yay!

Huge thanks to @jdstrand and all who helped!

Workaround works like a charm, so I can use my snaps and patiently wait until the official patch is released.


#28

Thanks for the tip, will try.


#29

I’ve upgraded the kernel 4.18.0rc4 to see if the boot problem with my ryzen 2200g was still present (surprisingly it seems to boot reliably now, or at least it seems so) but I couldn’t run snaps anymore.
I just wanted to confirm that the workaround works.

ls /etc/apparmor.d/snap-confine
/etc/apparmor.d/snap.core.4917.usr.lib.snapd.snap-confine
/etc/apparmor.d/usr.lib.snapd.snap-confine.real

There were 2 files, the first one had at line 366:
ptrace trace peer=unconfined,
Same thing second file line 354.


#30

It may not be able to run snaps for a different reason. Are there any security denials in the journalctl logs?


#31

It works with the workaround but I have to re-run apparmor_parser after every boot otherwise I see “readlinkat()…” and every snap doesn’t work.

This is the log before the apparmor_parser command

lug 09 19:02:35 ghv kernel: audit: type=1400 audit(1531155755.217:78): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4917/usr/lib/snapd/snap-confine" pid=3919 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
lug 09 19:02:35 ghv kernel: audit: type=1400 audit(1531155755.836:79): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4917/usr/lib/snapd/snap-confine" pid=3933 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
lug 09 19:04:05 ghv kernel: audit: type=1400 audit(1531155845.207:80): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4917/usr/lib/snapd/snap-confine" pid=4259 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
lug 09 19:04:12 ghv kernel: audit: type=1400 audit(1531155852.133:81): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name="/snap/core/4917/usr/lib/snapd/snap-confine" pid=4326 comm=“apparmor_parser”
lug 09 19:04:12 ghv kernel: audit: type=1400 audit(1531155852.153:82): apparmor=“STATUS” operation=“profile_replace” profile=“unconfined” name="/snap/core/4917/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=4326 comm=“apparmor_parser”
lug 09 19:04:12 ghv kernel: audit: type=1400 audit(1531155852.153:83): apparmor=“STATUS” operation=“profile_replace” info=“same as current profile, skipping” profile=“unconfined” name="/usr/lib/snapd/snap-confine" pid=4327 comm=“apparmor_parser”
lug 09 19:04:12 ghv kernel: audit: type=1400 audit(1531155852.153:84): apparmor=“STATUS” operation=“profile_replace” info=“same as current profile, skipping” profile=“unconfined” name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=4327 comm=“apparmor_parser”


#32

Hi,

I’m seeing this problem on Ubuntu 18.04 with kernel 4.18 and the workaround doesn’t seem to do anything for me.

> ls /etc/apparmor.d/*snap-confine*
/etc/apparmor.d/usr.lib.snapd.snap-confine.real

> tail /etc/apparmor.d/usr.lib.snapd.snap-confine.real
    # from the core snap but we are already inside the constructed mount
    # namespace. Here the apparmor kernel module re-constructs the path to
    # snap-update-ns using the "hostfs" mount entry rather than the more
    # "natural" /snap mount entry but we have no control over that.  This is
    # reported as (LP: #1716339). The variants here represent different
    # locations of snap mount directory across distributions.
    /var/lib/snapd/hostfs/{,var/lib/snapd/}snap/core/*/usr/lib/snapd/snap-update-ns r,

    ptrace read peer=unconfined,
}

> sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*

> gnome-calculator
cannot perform readlinkat() on the mount namespace file descriptor of the init process: Permission denied

> dmesg | grep DENIED         
[   28.836643] audit: type=1400 audit(1534247507.507:50): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=1759 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   29.030691] audit: type=1400 audit(1534247507.699:51): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=1892 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   29.230245] audit: type=1400 audit(1534247507.899:53): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2040 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   29.404266] audit: type=1400 audit(1534247508.075:54): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2165 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   29.665339] audit: type=1400 audit(1534247508.335:55): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2228 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   33.802955] audit: type=1400 audit(1534247512.471:56): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2670 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   39.115107] audit: type=1400 audit(1534247517.783:57): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=2958 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   41.458075] audit: type=1400 audit(1534247520.127:58): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=3020 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[   94.297198] audit: type=1400 audit(1534247572.951:59): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=3238 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 9039.010150] audit: type=1400 audit(1534256517.733:64): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=19462 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 9058.478617] audit: type=1400 audit(1534256537.201:65): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=19816 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 9083.475983] audit: type=1400 audit(1534256562.197:66): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=19924 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 9113.703202] audit: type=1400 audit(1534256592.425:67): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=19968 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[10614.319740] audit: type=1400 audit(1534258093.050:70): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=21750 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[10637.672492] audit: type=1400 audit(1534258116.406:71): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=21872 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[10862.835626] audit: type=1400 audit(1534258341.566:74): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=22062 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[11041.500133] audit: type=1400 audit(1534258520.235:75): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=22154 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[11752.936363] audit: type=1400 audit(1534259231.670:80): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=22689 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[70917.166364] audit: type=1400 audit(1534318396.649:81): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=27267 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[78768.839106] audit: type=1400 audit(1534326247.974:84): apparmor="DENIED" operation="ptrace" profile="/snap/core/5145/usr/lib/snapd/snap-confine" pid=30089 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"

Any ideas on why it still doesn’t work?


#33

You may have a core snap that is newer than the deb, so the policy you actually need to modify is /etc/apparmor.d/snap.core.REVISION.usr.lib.snapd.snap-confine, where REVISION is the revision of the core snap you have installed.

You can also just sudo snap refresh core --beta.


#34

Using the beta channel solved it for me, thanks!


#35

I am running snap on a non-Ubuntu system as follows:

$ snap version                                                                                                                                   
snap    2.30-5+b1
snapd   2.30-5+b1
series  16
pureos  
kernel  4.18.0-1-amd64

I have tried switching to both beta and edge channels and the problem persists. I have tried modifying /etc/apparmor.d/snap.core.REVISION.usr.lib.snapd.snap-confine then running apparmor_parser and the problem persists. See https://unix.stackexchange.com/questions/472068 for details.

What other information can I provide to help find an alternate workaround?


#36

Following the steps to modify /etc/apparmor.d/usr.lib.snapd.snap-confine.real (which didn’t work for yhrn) did in fact work for me.


some Custom kernel error on readlinkat () in mount namespace