Requesting autoconnect for interfaces in Pigmeat (process-control & home)

emilsayahi · June 4, 2020, 10:32am

EDIT (2020-06-10): Request changed from ‘classic’ confinement to autoconnecting interfaces. Disregard the initial request, now seeking approval for autoconnects. Original post is below:

Hello! I wrote a CLI static site generator in C# called Pigmeat, and, due to the issues Snapcraft has with .NET Core programs, I am forced to use ‘classic’ confinement.

The first technical issue I encountered was with the ‘dotnet’ plugin, which only supported up to .NET Core 2.2. I mitigated this issue using a custom plugin, with the code available on my respository: dotnet3.py.

However, the one issue which I cannot solve otherwise is the following error when attempting to run Pigmeat under ‘strict’ confinement:
Failed to create CoreCLR, HRESULT: 0x80070008

Besides the fact that this is a static site generator, which would already want traditional filesystem access to do its job, this needs to be under ‘classic’ confinement to run on a technical level. This appears to be related to snap’s issues with handling .NET Core 3.1 in general—something about the way the program is confined is making it impossible for the .NET runtime to work.

Thank you for taking the time to read. I would appreciate any and all further suggestions, though I would suggest cloning the git repository yourselves to see if the issue is merely on my side (though I do believe my testing was thorough).

EDIT: Wanted to mention that, while the technical issue remains, I do understand one should be using the ‘home’ and ‘removable-devices’ interfaces for a static site generator. Unfortunately, ‘classic’ confinement is still required to run the program, so this does not appear to be useful at the time being.

lucyllewy · June 4, 2020, 10:45am

This is caused by the ulimit being too small for the dotnet runtime. The solution is simply to add a wrapper script that sets the open files limit to 4096 (or potentially higher):

#!/bin/sh
ulimit -n 4096
exec "$@"

You can run this by adding the script to your snap with the dump plugin:

parts:
  ulimit-patch:
    plugin: dump
    source: path/to/dir/containing/script
    organize:
      scriptname.sh: bin/scriptname.sh # move it to the bin folder

apps:
  pigmeat:
    command: path/to/pigmeat
    command-chain:
      - bin/scriptname.sh # the ulimit script goes here

emilsayahi · June 4, 2020, 11:18am

It appears this did not resolve the error. I’m currently experimenting with ulimit to try and resolve this, however. My ulimit -n is 1024 on my machine, and I run the non-snap version just fine, so I suspect something else is at play. I’ll update this reply with the results of my fiddling around.

UPDATE (1/2):
It appears setting ulimit -n to unlimited did not work, and neither did setting ulimit -v to unlimited. Currently trying ulimit -u unlimited.

UPDATE (2/2):
Did not work. While I do understand where you’re coming from, considering that’s what pops up when you Google-search the error, it does not appear to be the solution. I have verified the script is indeed running as well. I’m a little lost as to why running it under ‘classic’ has any affect.

lucyllewy · June 4, 2020, 11:55am

Oh, I’m sorry that was fruitless it certainly helped one other snap that I’ve been working with, so I surmised it would have helped you. Ho hum…

emilsayahi · June 4, 2020, 11:57am

Well, Mr. Bowl Hat, I do appreciate the help, regardless. Best of luck on your own projects (and I must say I’m really digging that CSS on your website!)

emilsayahi · June 5, 2020, 10:55am

I have gone through and manually rejected several builds marked for review. I do not intend to give the message to @reviewers that I am no longer seeking ‘classic’ confinement, however. I’d just linked the snap with my GitHub repository, so new builds keep piling on with each new commit.

jdstrand · June 8, 2020, 9:18pm

Is this still an issue? It seems like everything is currently rejected so future uploads should be automatically reviewed.

emilsayahi · June 9, 2020, 10:03am

This is still an issue. I manually rejected the builds so you guys wouldn’t be prompted to manually review several almost-identical uploads. Should I upload a new build, and wait for it to get reviewed, as part of the process to get classic confinement?

UPDATE:
I have made a new build, #40, and I have requested manual review on it. Apologies for the confusion.

alexmurray · June 9, 2020, 10:35am

Thanks - so if you are still requesting classic confinement then you will need to be more explicit as to why this is required - what specific things does Pigmeat need to do which are not able to be done under strict confinement? What denials do you see in dmesg / journalctl when running Pigmeat when it is under strict confinement?

emilsayahi · June 9, 2020, 11:14am

Beyond the vague Failed to create CoreCLR, HRESULT: 0x80070008 error which I initially described and have been unable to troubleshoot, the logs do not seem to mean much to me. Perhaps they will to you. journalctl outputs the following when I run Pigmeat:

Jun 09 07:05:16 emil-G11CD audit[3716]: AVC apparmor="DENIED" operation="capable" profile="/snap/core/9289/usr/lib/snapd/snap-confine" pid=3716 comm="snap-confine" capability=4  capname="fsetid"
Jun 09 07:05:16 emil-G11CD kernel: audit: type=1400 audit(1591700716.581:66): apparmor="DENIED" operation="capable" profile="/snap/core/9289/usr/lib/snapd/snap-confine" pid=3716 comm="snap-confine" capability=4  capname="fsetid"
Jun 09 07:05:17 emil-G11CD audit[3716]: AVC apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3716/mountinfo" pid=3716 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Jun 09 07:05:17 emil-G11CD audit[3716]: AVC apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3716/mountinfo" pid=3716 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Jun 09 07:05:17 emil-G11CD kernel: audit: type=1400 audit(1591700717.469:67): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3716/mountinfo" pid=3716 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=10
Jun 09 07:05:17 emil-G11CD kernel: audit: type=1400 audit(1591700717.469:68): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3716/mountinfo" pid=3716 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=10
Jun 09 07:05:17 emil-G11CD audit[3716]: SECCOMP auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=3716 comm="pigmeat" exe="/snap/pigmeat/x1/pigmeat" sig=0 arch=c000003e syscall=203 compat=0 ip=0x7f2e4a5b2ef7 code=0x50000
Jun 09 07:05:17 emil-G11CD kernel: audit: type=1326 audit(1591700717.473:69): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=3716 comm="pigmeat" exe="/snap/pigmeat/x1/pigmeat" sig=0 arch=c000003e syscall=203 compat=0 ip=0x7f2e4a5b2ef
Jun 09 07:05:20 emil-G11CD clamd[1281]: Tue Jun  9 07:05:20 2020 -> SelfCheck: Database status OK.
Jun 09 07:06:29 emil-G11CD audit[3833]: AVC apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3833/mountinfo" pid=3833 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Jun 09 07:06:29 emil-G11CD kernel: audit: type=1400 audit(1591700789.265:70): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3833/mountinfo" pid=3833 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=10
Jun 09 07:06:29 emil-G11CD kernel: audit: type=1400 audit(1591700789.265:71): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3833/mountinfo" pid=3833 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=10
Jun 09 07:06:29 emil-G11CD kernel: audit: type=1326 audit(1591700789.265:72): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=3833 comm="pigmeat" exe="/snap/pigmeat/x1/pigmeat" sig=0 arch=c000003e syscall=203 compat=0 ip=0x7fc163322ef
Jun 09 07:06:29 emil-G11CD audit[3833]: AVC apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3833/mountinfo" pid=3833 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Jun 09 07:06:29 emil-G11CD audit[3833]: SECCOMP auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=3833 comm="pigmeat" exe="/snap/pigmeat/x1/pigmeat" sig=0 arch=c000003e syscall=203 compat=0 ip=0x7fc163322ef7 code=0x50000
Jun 09 07:07:55 emil-G11CD plank.desktop[4702]: [5060:5060:0609/070755.781458:ERROR:shared_image_manager.cc(212)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
Jun 09 07:07:55 emil-G11CD plank.desktop[4702]: [5060:5060:0609/070755.861679:ERROR:shared_image_manager.cc(212)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
Jun 09 07:08:45 emil-G11CD PackageKit[3289]: daemon quit
Jun 09 07:08:58 emil-G11CD plank.desktop[4702]: Fontconfig error: Cannot load default config file: No such file: (null)
Jun 09 07:10:25 emil-G11CD audit[4189]: AVC apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/4189/mountinfo" pid=4189 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Jun 09 07:10:25 emil-G11CD kernel: audit: type=1400 audit(1591701025.766:73): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/4189/mountinfo" pid=4189 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=10
Jun 09 07:10:25 emil-G11CD kernel: audit: type=1400 audit(1591701025.766:74): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/4189/mountinfo" pid=4189 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=10
Jun 09 07:10:25 emil-G11CD kernel: audit: type=1326 audit(1591701025.766:75): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=4189 comm="pigmeat" exe="/snap/pigmeat/x1/pigmeat" sig=0 arch=c000003e syscall=203 compat=0 ip=0x7f29ab7ccef
Jun 09 07:10:25 emil-G11CD audit[4189]: AVC apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/4189/mountinfo" pid=4189 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Jun 09 07:10:25 emil-G11CD audit[4189]: SECCOMP auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=4189 comm="pigmeat" exe="/snap/pigmeat/x1/pigmeat" sig=0 arch=c000003e syscall=203 compat=0 ip=0x7f29ab7ccef7 code=0x50000

Running dmesg | tail -20 shows that Pigmeat, while running under strict confinement, creates the following output:

[176054.350016] audit: type=1400 audit(1591700301.391:58): apparmor="STATUS" operation="profile_load" profile="snap.multipass.multipassd" name="multipass.snapcraft-pigmeat.qemu-system-x86_64" pid=1193 comm="apparmor_parser"
[176054.719504] IPv6: ADDRCONF(NETDEV_CHANGE): tap-7c9c1793201: link becomes ready
[176054.719591] mpqemubr0: port 3(tap-7c9c1793201) entered blocking state
[176054.719593] mpqemubr0: port 3(tap-7c9c1793201) entered forwarding state
[176175.707256] audit: type=1400 audit(1591700422.748:59): apparmor="STATUS" operation="profile_load" profile="snap.multipass.multipassd" name="multipass.snapcraft-pigmeat.a99b7eb6.sshfs_server" pid=2874 comm="apparmor_parser"
[176175.923645] audit: type=1400 audit(1591700422.964:60): apparmor="DENIED" operation="open" profile="multipass.snapcraft-pigmeat.a99b7eb6.sshfs_server" name="/etc/ssh/ssh_config" pid=2875 comm="sshfs_server" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[176176.176899] audit: type=1400 audit(1591700423.220:61): apparmor="STATUS" operation="profile_remove" profile="snap.multipass.multipassd" name="multipass.snapcraft-pigmeat.a99b7eb6.sshfs_server" pid=2877 comm="apparmor_parser"
[176193.885111] audit: type=1400 audit(1591700440.928:62): apparmor="STATUS" operation="profile_load" profile="snap.multipass.multipassd" name="multipass.snapcraft-pigmeat.a99b7eb6.sshfs_server" pid=2889 comm="apparmor_parser"
[176193.889845] audit: type=1400 audit(1591700440.932:63): apparmor="DENIED" operation="open" profile="multipass.snapcraft-pigmeat.a99b7eb6.sshfs_server" name="/etc/ssh/ssh_config" pid=2890 comm="sshfs_server" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[176449.625991] audit: type=1400 audit(1591700696.669:64): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap-update-ns.pigmeat" pid=3694 comm="apparmor_parser"
[176449.693386] audit: type=1400 audit(1591700696.737:65): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.pigmeat.pigmeat" pid=3695 comm="apparmor_parser"
[176469.537815] audit: type=1400 audit(1591700716.581:66): apparmor="DENIED" operation="capable" profile="/snap/core/9289/usr/lib/snapd/snap-confine" pid=3716 comm="snap-confine" capability=4  capname="fsetid"
[176470.426764] audit: type=1400 audit(1591700717.469:67): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3716/mountinfo" pid=3716 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[176470.426766] audit: type=1400 audit(1591700717.469:68): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3716/mountinfo" pid=3716 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[176470.429290] audit: type=1326 audit(1591700717.473:69): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=3716 comm="pigmeat" exe="/snap/pigmeat/x1/pigmeat" sig=0 arch=c000003e syscall=203 compat=0 ip=0x7f2e4a5b2ef7 code=0x50000
[176542.223144] audit: type=1400 audit(1591700789.265:70): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3833/mountinfo" pid=3833 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[176542.223146] audit: type=1400 audit(1591700789.265:71): apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/3833/mountinfo" pid=3833 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[176542.223237] audit: type=1326 audit(1591700789.265:72): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=3833 comm="pigmeat" exe="/snap/pigmeat/x1/pigmeat" sig=0 arch=c000003e syscall=203 compat=0 ip=0x7fc163322ef7 code=0x50000

Again, I have no idea how to interpret this. I’m not even that familiar with Linux to be honest. I’m just some seventeen-year-old kid that wanted to make a static site-gen that’s only working under ‘classic’ and I haven’t a clue why.

lucyllewy · June 9, 2020, 2:11pm

@jdstrand, this might be a confinement issue? https://github.com/dotnet/runtime/issues/36632 suggests that this error might be caused by mlock() calls being denied or otherwise failing. I haven’t checked to see if the confinement rules specifically block or allow this access.

lucyllewy · June 9, 2020, 2:15pm

OK!

The solution is to add process-control plug and ensure it is connected. This request can therefore probably be transformed into a request for automatic connection of process-control, but as it’s not my Snap I can’t make that decision

🎳🎩  Pigmeat (master *%) $ snap install pigmeat_2.0.3_amd64.snap --dangerous
pigmeat 2.0.3 installed

🎳🎩  Pigmeat (master *%) $ snap connect pigmeat:process-control

🎳🎩  Pigmeat (master *%) $ pigmeat
Pigmeat supports the following commands:
	pigmeat new - Creates an empty Pigmeat project.
	pigmeat build - Outputs a publishable Pigmeat project.
	pigmeat b - Outputs a publishable Pigmeat project. Same as above.
	pigmeat clean - Deletes all generated data that results from the build process.
	pigmeat help - Shows this message.
	pigmeat help <string> - Displays a message outlining the usage of a given parameter (e.g. 'pigmeat help serve').

emilsayahi · June 9, 2020, 2:22pm

Why, that’s brilliant, then! Your help is much appreciated, @lucyllewy! I’ll be renaming the thread and changing my request to the following:

Autoconnect for interfaces process-control & home.
Reasoning being that it’s a static site generator, so access to the filesystem is required.

EDIT: Apologies for all the trouble this has been so far! I know you guys deal with stuff like this on the regular, but it’s always someone’s first time with snap, eh?

lucyllewy · June 9, 2020, 3:05pm

One thing to note, I used a wrapper-script to set the ulimit on open files as discussed further up. This is the snapcraft.yaml I used:

name: pigmeat
version: 2.0.3
summary: A static content publishing tool for the modern web.
description: |
  Pigmeat is a static content publishing tool for the modern web. It takes data in the form of JSON or YAML, plugs it into a Liquid context, wraps it up in some HTML & CSS, and spits out a website that can be hosted on your favorite server. If you don't know what any of that means, it's OK. Point is, it's easy to learn and use. Pigmeat is for corporations, organizations, friends, peers, bloggers, hobbyists, and everyone under the sun.
confinement: strict
base: core18
grade: stable
license: GPL-3.0
icon: branding/Yellow/android-chrome-256x256.png

parts:
  set-ulimit:
    plugin: nil
    override-pull: |
      cat <<'EOF' > set-ulimit.sh
      #!/bin/sh
      ulimit -n 4096
      exec "$@"
      EOF
    override-build: |
      install -m755 -D -t $SNAPCRAFT_PART_INSTALL/bin set-ulimit.sh

  pigmeat:
    plugin: dotnet3
    source: .

apps:
  pigmeat:
    command: pigmeat
    command-chain:
      - bin/set-ulimit.sh
    plugs:
      - home
      - removable-media
      - process-control

emilsayahi · June 9, 2020, 3:07pm

I did not do so and it appears to be running just fine with process-control regardless. Such a funny little quirk. Glad you caught it!

emilsayahi · June 12, 2020, 9:59am

Bumping. To be clear, I am now requesting autoconnects instead of ‘classic’ confinement. Thank you.

alexmurray · June 15, 2020, 6:48am

Hmm the only denials listed in the logs are for /proc/$PID/mountinfo - this can be resolved by adding the mount-observe interface to plugs.

However I would like to understand why process-control is needed in this case - this interface allows a snap to control all processes on the system and so is not something that would normally be granted auto-connect for (especially for a static site generator app). I am hoping there is some other less privileged interface which can be used instead - so I need to understand what pigmeat is using from this interface - @lucyllewy since you seem to be more familiar with this snap are you able to help identify this more specifically?

alexmurray · June 15, 2020, 7:07am

I gave the snap a quick run and used snappy-debug to help determine what was being denied:

$ sudo snap install pigmeat

Then in another terminal:

$ sudo snap install snappy-debug
$ sudo journalctl --output=short --follow --all | sudo snappy-debug

And finally run pigmeat itself back in the first terminal:

$ pigmeat
Failed to create CoreCLR, HRESULT: 0x80070008

And we see back in the snappy-debug terminal the following output:

= AppArmor =
Time: Jun 15 16:21:14
Log: apparmor="DENIED" operation="open" profile="snap.pigmeat.pigmeat" name="/proc/1882/mountinfo" pid=1882 comm="pigmeat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
File: /proc/1882/mountinfo (read)
Suggestions:
* adjust program to not access '@{PROC}/@{pid}/mountinfo'
* add 'mount-observe' to 'plugs'

= Seccomp =
Time: Jun 15 16:21:14
Log: auid=1000 uid=1000 gid=1000 ses=3 pid=1882 comm="pigmeat" exe="/snap/pigmeat/77/pigmeat" sig=0 arch=c000003e 203(sched_setaffinity) compat=0 ip=0x7fdc2a33def7 code=0x50000
Syscall: sched_setaffinity
Suggestion:
* add 'process-control' to 'plugs'

So this suggests the issue is with sched_setaffinity being denied by seccomp - we can confirm this further by running the snap via strace:

$ snap run --strace pigmeat > /tmp/strace.log 2>&1
$ grep sched_setaffinity /tmp/strace.log 
[pid  2218] sched_setaffinity(2226, 128, [0]) = -1 EPERM (Operation not permitted)

The man page for sched_setaffinity(2) says regarding EPERM:

       EPERM  (sched_setaffinity()) The calling thread does not have appropri‐
              ate  privileges.  The caller needs an effective user ID equal to
              the real user ID or effective user ID of the  thread  identified
              by  pid,  or  it must possess the CAP_SYS_NICE capability in the
              user namespace of the thread pid.

And we can see from the strace log that the thread making the call to sched_setaffinity is the one which created this child process (ie it is calling sched_setaffinity with a pid of 2226):

$ grep 2226 /tmp/strace.log 
[pid  2218] clone(child_stack=0x7f3475b69fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f3475b6a9d0, tls=0x7f3475b6a700, child_tidptr=0x7f3475b6a9d0) = 2226
[pid  2218] sched_setaffinity(2226, 128, [0]) = -1 EPERM (Operation not permitted)
[pid  2218] tgkill(2218, 2226, SIGRTMIN) = 0

So this should succeed (except in this case it doesn’t since sched_setaffinity is blocked by the default seccomp policy) - @jdstrand this feels like something which we should allow in the base policy to me - and not have to require something like process-control to use - thoughts?

lucyllewy · June 15, 2020, 9:39am

Agreed. process-control shouldn’t be required to set the affinity of your own processes. As far as I’m aware, correct me if I’m wrong, the affinity syscall is still mediated by capabilities of the user running the process, too. This would mean that only root or otherwise elevated-privilege (granted specific CAPs?) processes will be able to use it to actually effect a change.

jdstrand · June 15, 2020, 4:55pm

The default policy already allows:

# enforce pid_t is 0 so the app may only change its own scheduler and affinity.
# Use process-control interface for controlling other pids.
sched_setaffinity 0 - -
sched_setparam 0 -

which is correct for the default policy.

@alexmurray’s analysis shows that the snap is trying to sched_setaffinity on a different process (pid 2226), which is what process-control is for. Note, that the syscall filter does not kill processes and so the application is free to proceed with sched_setaffinity failing if that makes sense for the application. To avoid process-control, the application may be able to be adjusted to use sched_setaffinity(0, ...).

This appears to be a bug in dotnet: https://github.com/dotnet/runtime/issues/1634 so I’ve added a comment to that bug for this topic.

The mount-observe denials are often just noise, so may be non-fatal.