Requesting autoconnect for interfaces in Pigmeat (process-control & home)

I did not do so and it appears to be running just fine with process-control regardless. Such a funny little quirk. Glad you caught it!

1 Like

Bumping. To be clear, I am now requesting autoconnects instead of ‘classic’ confinement. Thank you.

Hmm the only denials listed in the logs are for /proc/$PID/mountinfo - this can be resolved by adding the mount-observe interface to plugs.

However I would like to understand why process-control is needed in this case - this interface allows a snap to control all processes on the system and so is not something that would normally be granted auto-connect for (especially for a static site generator app). I am hoping there is some other less privileged interface which can be used instead - so I need to understand what pigmeat is using from this interface - @lucyllewy since you seem to be more familiar with this snap are you able to help identify this more specifically?

I gave the snap a quick run and used snappy-debug to help determine what was being denied:

$ sudo snap install pigmeat

Then in another terminal:

$ sudo snap install snappy-debug
$ sudo journalctl --output=short --follow --all | sudo snappy-debug

And finally run pigmeat itself back in the first terminal:

$ pigmeat
Failed to create CoreCLR, HRESULT: 0x80070008

And we see back in the snappy-debug terminal the following output:

= AppArmor =
Time: Jun 15 16:21:14
Log: apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/1882/mountinfo" pid=1882 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
File: /proc/1882/mountinfo (read)
Suggestions:
* adjust program to not access '@{PROC}/@{pid}/mountinfo'
* add 'mount-observe' to 'plugs'

= Seccomp =
Time: Jun 15 16:21:14
Log: auid=1000 uid=1000 gid=1000 ses=3 pid=1882 comm="pigmeat" exe="/snap/pigmeat/77/pigmeat" sig=0 arch=c000003e 203(sched_setaffinity) compat=0 ip=0x7fdc2a33def7 code=0x50000
Syscall: sched_setaffinity
Suggestion:
* add 'process-control' to 'plugs'

So this suggests the issue is with sched_setaffinity being denied by seccomp - we can confirm this further by running the snap via strace:

$ snap run --strace pigmeat > /tmp/strace.log 2>&1
$ grep sched_setaffinity /tmp/strace.log 
[pid  2218] sched_setaffinity(2226, 128, [0]) = -1 EPERM (Operation not permitted)

The man page for sched_setaffinity(2) says regarding EPERM:

       EPERM  (sched_setaffinity()) The calling thread does not have appropri‐
              ate  privileges.  The caller needs an effective user ID equal to
              the real user ID or effective user ID of the  thread  identified
              by  pid,  or  it must possess the CAP_SYS_NICE capability in the
              user namespace of the thread pid.

And we can see from the strace log that the thread making the call to sched_setaffinity is the one which created this child process (ie it is calling sched_setaffinity with a pid of 2226):

$ grep 2226 /tmp/strace.log 
[pid  2218] clone(child_stack=0x7f3475b69fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f3475b6a9d0, tls=0x7f3475b6a700, child_tidptr=0x7f3475b6a9d0) = 2226
[pid  2218] sched_setaffinity(2226, 128, [0]) = -1 EPERM (Operation not permitted)
[pid  2218] tgkill(2218, 2226, SIGRTMIN) = 0

So this should succeed (except in this case it doesn’t since sched_setaffinity is blocked by the default seccomp policy) - @jdstrand this feels like something which we should allow in the base policy to me - and not have to require something like process-control to use - thoughts?

1 Like

Agreed. process-control shouldn’t be required to set the affinity of your own processes. As far as I’m aware, correct me if I’m wrong, the affinity syscall is still mediated by capabilities of the user running the process, too. This would mean that only root or otherwise elevated-privilege (granted specific CAPs?) processes will be able to use it to actually effect a change.

1 Like

The default policy already allows:

# enforce pid_t is 0 so the app may only change its own scheduler and affinity.
# Use process-control interface for controlling other pids.
sched_setaffinity 0 - -
sched_setparam 0 -

which is correct for the default policy.

@alexmurray’s analysis shows that the snap is trying to sched_setaffinity on a different process (pid 2226), which is what process-control is for. Note, that the syscall filter does not kill processes and so the application is free to proceed with sched_setaffinity failing if that makes sense for the application. To avoid process-control, the application may be able to be adjusted to use sched_setaffinity(0, ...).

This appears to be a bug in dotnet: https://github.com/dotnet/runtime/issues/1634 so I’ve added a comment to that bug for this topic.

The mount-observe denials are often just noise, so may be non-fatal.

2 Likes

@emilsayahi - can you participate in the upstream dotnet bug? With some small changes the application should be able to run without process-control. I’d like to hear from upstream dotnet before casting my vote.

Apologies if I sound a little stupid when I say this, I’m not entirely sure what to contribute to the issue? C# is a high-level, managed language—I’m not sure if there’s anything I could do on my end to resolve this problem. If it’s a bug with dotnet, and it’s not going to be resolved on their end anytime soon, then I would say connecting the interface would be the only option.
You certainly know a lot more than I do; I have very little experience in software development or the Linux ecosystem. I’m not seeing what ‘small changes’ could be made as a result.
I really appreciate your help so far. Sorry if I’m being a little tricky to work with, haha.

You are right that there isn’t anything you can do, but I’d like to hear what the dotnet folks have to say before we vote (there might be alternatives to explore, they might just fix dotnet, etc).

1 Like

Not to be a bother, it’s just that it’s been a while … since very little activity is occurring on the side of .NET, can this be revisited? I’ve had two people go so far as to leave negative reviews on the Snap Store as a result of this bug, and I’m growing a bit concerned that this autoconnect situation is harming users’ experiences and damaging the little reputation my app has:
Screenshot%20from%202020-07-03%2015-39-11

I’ve done a bit of digging, and think I found their offending code. I commented on the issue explaining my theory…

https://github.com/dotnet/runtime/blob/master/src/coreclr/src/pal/src/thread/thread.cpp#L760

1 Like

You’re the best, man. Seriously, the internet needs more people like you. Thanks for getting the ball rolling again.

Thanks @lucyllewy! I responded in the upstream bug and also suggested as a possible workaround to use LD_PRELOAD for sched_setaffinity until the issue is sorted out in dotnet proper. @lucyllewy - I can’t recall, do you have an example of this (perhaps it should be added to snapcraft-preload)?

For your future snaps, some snap publishers wait to promote their snaps to stable until auto-connections are established. Also, I believe you may be able to use an LD_PRELOAD to fix your snap now that @lucyllewy has triaged the issue.

@emilsayahi - it looks like upstream targeted fixing dotnet for this for 6.0.0. Were you able to do anything with LD_PRELOAD to work around this?

Nope, unfortunately. Thank you for your continued help, seriously. It’s unfortunate that this will take so long to fix upstream, but I’m also glad that we finally caught their attention. Hopefully, .NET Core will become more useable with Snapcraft. Currently, with .NET Core 3.1, I’ve been using a modified dotnet plugin that someone wrote in a Launchpad issue a while back, to build with snap, as the official dotnet plugin has been known to be broken (unless this has changed quite recently). I’m afraid that by the time 5.0, or even 6.0 come out, this modified plugin will itself become incompatible, creating yet another issue. Maybe I should learn Python to fix it myself, if this happens :woozy_face:?

EDIT: Here’s where I found the custom plugin: https://bugs.launchpad.net/snapcraft/+bug/1857705. The bug with the official plugin was marked as triaged and of high importance, yet was not resolved back in June when I first looked into publishing on Snapcraft. If this has been fixed it hasn’t been marked as such. If it has not, I suggest accepting ed10vi’s patch, which was written toward the end of May, ASAP and then looking into ensuring the dotnet plugin continues to work with future releases. .NET 5 releases rather soon, November 2020 (release previews are out now), with 6.0 being targeted for November 2021 (source).

+1 for auto-connection of process-control for now. When dotnet is updated, we can consider migrating your snap away from it.

@reviewers - can others please vote?

1 Like

@advocacy, @leecow - perhaps dotnet could consider backporting the fix for https://github.com/dotnet/runtime/issues/1634 (currently milestoned for 6) to .NET 5 once it is applied to 6? We could probably handle the developer friction until 5 is released (ie, dotnet runtime consumers having to plugs and ask for auto-connect of process-control) until November, but waiting more than a year for the fix would be suboptimal.

Also see:

1 Like

+1 for auto-connect of process-control to pigmeat as a stop-gap.

+2 votes for, 0 votes against, this is now live.

1 Like

FYI, progress is being made on https://github.com/dotnet/runtime/issues/1634 and it looks like they are considering backporting as far back as dotnet runtime 3.1.