Classic confinement for `parca-agent`?

jnsgruk · October 5, 2022, 1:52pm

Hi all

I’ve been working some more on the parca-agent snap. Given that it’s inspecting the system at such a low level (using eBPF for CPU process tracing, etc.) I’m beginning to wonder if classic confinement might be a better way to go?

While the eBPF functionality is there, and broadly the agent seems to work, I get a lot of messages in the journal complaining:

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/snappy-debug/598/usr/bin/python3.6" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/snappy-debug/598/usr/bin/python3.6 (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/parca/297/parca" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/parca/297/parca (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/microk8s/4055/kubelite" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/microk8s/4055/kubelite (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/parca/297/parca" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/parca/297/parca (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/microk8s/4055/kubelite" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/microk8s/4055/kubelite (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/snappy-debug/598/usr/bin/python3.6" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/snappy-debug/598/usr/bin/python3.6 (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/parca/297/parca" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/parca/297/parca (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/microk8s/4055/kubelite" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/microk8s/4055/kubelite (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/parca/297/parca" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/parca/297/parca (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

= AppArmor =
Time: Oct 05 14:47:16
Log: apparmor="DENIED" operation="open" profile="snap.parca-agent.parca-agent-svc" name="/snap/microk8s/4055/bin/k8s-dqlite" pid=1796859 comm="parca-agent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /snap/microk8s/4055/bin/k8s-dqlite (read)
Suggestion:
* adjust program to read necessary files from $SNAP, $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON

This is where the agent is trying to access the binaries/libs that it’s profiled, and I don’t know I could ever come up with a sufficiently broad interface to capture that, as it’ll change from system to system? I note that other snaps with a similar purpose (like telegraf) are classic, and wonder if perhaps that’s the way to go?

The agent also tries to execute systemctl on startup to get a list of active systemd units to make it’s reporting more accurate, which can be seen in the logs for the agent:

2022-10-05T14:45:07+01:00 parca-agent.parca-agent-svc[1796859]: level=warn name=parca-agent ts=2022-10-05T13:45:07.223014524Z caller=discovery_manager.go:221 msg="unable to start provider" provider=systemd/0 error="failed to list units: fork/exec /usr/bin/systemctl: permission denied"

Interested in any thoughts on this?

Many thanks!

Jon

jnsgruk · October 17, 2022, 1:35pm

I’ve gone ahead and made this a classic snap, could someone take a look please? @alexmurray/@emitorino

Latest review here: https://dashboard.snapcraft.io/snaps/parca-agent/revisions/24/

alexmurray · October 20, 2022, 5:09am

As per the Process for reviewing classic confinement snaps, for a snap to be granted classic confinement, it must first have a requirement for classic confinement (ie. requires access to arbitrary files / requires to be able to execute arbitrary, user defined executables from the host etc), and it must fit within one of the supported categories for classic confinement.

Can you please provide some more details to help establish if it meets these two requirements? Since it wants to try and profile everything that would seem to indicate that it needs arbitrary access to files from the host - but if it just needs to read them, the it could use the system-backup interface and then look at everything via /var/lib/snapd/hostfs/ - can you provide more info?

jnsgruk · October 20, 2022, 6:44am

Hey,

Thanks So I think this probably fits into the “debug tools” category.

I’m open to the idea of using system-backup, but presumably using that interface would require the app itself to have branches in the code to know its in a snap, and prepend /var/lib/snapd/hostfs to all paths when it’s looking at files? Or is there some magic I’m missing?

Cheers, Jon

N.B. the tool does also seek to execute applications from the host on occasion (such as systemctl for listing/interrogating units - I’m not sure this would work using system-backup?)

alexmurray · October 20, 2022, 7:18am

Yes this is correct - there is no magic here.

No system-backup only provides read access - not execute. If it is required to execute arbitrary binaries from the host then that can only be done with classic confinement. Although I wonder if the snap could ship systemctl itself and then whether there may be an appopriate interace which would allow it to see the required information? Perhaps snappy-debug may be able to shed some light.

jnsgruk · October 20, 2022, 7:29am

I think given the low level, and thus super privileged nature of the application, that might introduce quite a lot of overhead on the upstream to keep the confinement working. The agent is focused right now on CPU profiling, but is heading toward low-level, system-wide profiling of every single process on the system for CPU, memory and IO usage. My sense is that maintaining the confinement could become difficult to impossible as they expand!

As part of this, the process needs to gather information about systemd units, cgroups, fetch debug symbols, and run system utilities in the host system context.

I think this likely sits somewhere between “debug tools” and “public cloud agents” and definitely requires access to files on the host outside of the snap runtime.

jnsgruk · October 26, 2022, 11:54am

Hi

Do I need to provide any more info here?

Jon

alexmurray · October 31, 2022, 4:02pm

Thanks for the update @jnsgruk - no I don’t think anything more is required here from you. The requirements for classic confinement for parca-agent are understood.

@Igor could you please perform publisher vetting?

Igor · October 31, 2022, 4:14pm

@alexmurray From what I see, this is already a vetted acc.

alexmurray · October 31, 2022, 8:03pm

@Igor oh you are right - thanks. Classic confinement override granted for parca-agent. This is now live. Note existing users of the strictly confined snap will not be auto-upgraded to the classicly confined version - they will need to explicitly snap refresh --classic parca-agent.

jnsgruk · November 1, 2022, 11:10am

Excellent, thanks both!