Custom kernel error on readlinkat() in mount namespace

jdstrand
upcoming

#2

I tried the 4.18-rc2 kernel form http://kernel.ubuntu.com/~kernel-ppa/mainline/ on Ubuntu 18.04 today and encountered the same problem.


#3

Can you please provide some additional debugging information:

dmesg | grep DENIED

Can you share what UKUU is? I’m not familiar with that.


#4

UKUU is a hack that installs mainline kernels from PPAs to your system (which are likely missing the right config options and possibly also patches to make snapd work propery):

https://www.omgubuntu.co.uk/2017/02/ukuu-easy-way-to-install-mainline-kernel-ubuntu


#5

I “loved” your response though I’m not sure that qualifies about how I feel about UKUU :wink:

Still, I believe we should be able to get this to work. Alternatively we can patch snapd to detect this and fail more gracefully.


#6

@MartinTheWanderer As hinted in the conversation above, the issue is that snapd detects that it’s running inside Ubuntu, and expects certain things to work correctly, but they don’t because the kernel is not in fact an Ubuntu kernel. The two alternatives here are to either convince the kernel to work more like an Ubuntu kernel, or convince snapd that this is not in fact a normal Ubuntu system. Hacking /etc/os-release might help, but I agree with @zyga that we should try to introspect and disable the particular features that are breaking for you automatically if that’s a reasonable amount of work given the benefits.

@zyga Who’s our go-to person for mount namespace issues in the kernel at the moment? Can we at least tell what specific patch is missing?


#7

I strongly suspect this is related to apparmor more than to anything else. We’ve seen some changing behaviour in this area that results in exactly this issue, readlink /proc/1/ns/mnt being denied. To be certain we need to see the denial run question and check how the upstream kernel treats that operation. Once we have this information we can discuss with the security team where (probably) jj might pinpoint the missing patch.


Zyga's work queue
#8

Guys, thanks for all your feedback and ideas!

Here are some of the lines from DMESG, seems like an apparmor issue indeed.

[ 6897.664101] audit: type=1400 audit(1529974423.014:835): apparmor="DENIED" operation="ptrace" profile="/usr/lib/snapd/snap-confine" pid=3618 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[ 6933.336394] audit: type=1400 audit(1529974458.686:836): apparmor="DENIED" operation="ptrace" profile="/usr/lib/snapd/snap-confine" pid=4058 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"
[51331.607353] audit: type=1400 audit(1530018856.521:837): apparmor="DENIED" operation="ptrace" profile="/usr/lib/snapd/snap-confine" pid=9709 comm="snap-confine" requested_mask="read" denied_mask="read" peer="unconfined"

Interesting and not really logical for me (as someone with - ahem - very limited insight in the inner workings of Snap things…) that some snaps, namely Atom, do not have that problem and run just fine even with 4.18RC2

BTW I am not doing this to keep you guys busy, lol, damn AMD support in 4.17 is just too buggy, keep getting freezes every couple minutes, hours max, with 4.18 it seems to be fine.


#9

classic snaps do not route their disk access though apparmor while strict confined ones (and devmode ones) do …

if you check with snap list check the last column. i assume all snaps listed as classic there will work.


#10

indeed. Actually of those I kept Atom is the only classic one, and the only one that runs, lol


#11

I will fix this issue asap


#12

Thanks @zyga!

Don’t break your back, I remember seeing some quick&dirty workarounds for apparmor issues in other threads and will try those today.

Thanks all for helping to pinpoint the problem, I was unsure where to start looking… :smiley:


#13

Ummmmh… guess I was too optimistic about apparmor workarounds… :smiley:

Probably looking in the wrong place again.

Tried changing the aa profile for e.g. snap.gimp.gimp from enforce to complain, and tried to disable apparmor completely, but still getting the same error when trying to start gimp.

IS there actually a quick and dirty workaround to take apparmor out of the equation temporarily or convince it that these snaps are perfectly fine? :wink:


#14

using an ubuntu kernel is the easiest “workaround” here i guess

the confinement is a semi-complex communication between snapd and the involved parts, killing off one side of that communication will not help, both sides need adjustment (which is what zyga is working on) to make it work.


#15

dang.

ok, thanks, guess I need to work on a different machine then for now.

Going back to an earlier kernel unfortunately is not an option, it results in frequent freezes thanks to still unreliable support for the Raven Ridge processors with integrated Vega graphics…


#16

I’m sure we can get it to work on mainline.


#17

dmesg | grep DENIED
[ 36.097077] audit: type=1400 audit(1530124482.591:45): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4830/usr/lib/snapd/snap-confine" pid=2492 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”
[ 60.609228] audit: type=1400 audit(1530124507.079:46): apparmor=“DENIED” operation=“ptrace” profile="/snap/core/4830/usr/lib/snapd/snap-confine" pid=2570 comm=“snap-confine” requested_mask=“read” denied_mask=“read” peer=“unconfined”

uname -r
4.18.0-041800rc1-generic


#18

Note, this probably isn’t an apparmor change but a capabilities change in the upstream kernel that now triggers this denial. The snap-confine profile seems to need: ptrace (read) peer=unconfined, to read this file.


#19

Hmm, curiously, this is in /etc/apparmor.d/snap.core.4938.usr.lib.snapd.snap-confine:

    # support for the mount namespace sharing
    capability sys_ptrace,
    # allow snap-confine to read /proc/1/ns/mnt
    ptrace trace peer=unconfined,

It seems 4.18 changed the check from ‘trace’ to ‘read’. @zyga, we could adjust to have:

ptrace trace peer=unconfined, # 4.17 and earlier
ptrace read peer=unconfined, # 4.18 and later

though @zyga, a much better fix would be to have only the ‘ptrace read’ rule in the profile and conditionally add the ‘ptrace trace’ if on kernel <=4.17. The trace rule is really powerful and it would be good to avoid it if possible. We should be able to do this based on the #include mechanisms we use for nfs and overlay.

@MartinTheWanderer, as a workaround, you can add to the /etc/apparmor.d/*snap-confine* files (on a line before the final ‘}’ at the end of the file):

ptrace read peer=unconfined,

then run sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*. You will need to redo this when the core snap refreshes until the proper fix is included.


#20

Actually, I discussed this with John and this was an AppArmor change (well, more precisely, an unrelated change turned the read into a trace and AppArmor needed an adjustment to turn it back into a read, but this adjustment didn’t happen until 4.18). The ‘trace’ was undesirable and should’ve always been ‘read’. He is in the process of rolling that out to all Ubuntu kernels. @zyga, it would be great to default to read when possible. Eg, if <4.18 and isNotUbuntu, add ptrace trace rule (we’d have to wait for those kernels to rollout of course.


#21

I’ve asked John to add a flag for the backports tree so we can interrogate the kernel for this. So, we can check for the flag or 4.18. Anyone who picks up the patch will just get it.


Snaps are not working with Linux Kernel v4.18-rc