Hostfs mapping /mnt

wililupy · 2018-06-11 23:14:22 UTC

Hello,

I am currently writing an interface for hugepages. I have the following in my interface for hugepages-control:

const hugepagesControlConnectedPlugAppArmor = `
# Description: Allow control to hugepages.
/sys/kernel/mm/hugepages/{,*} rw,
/var/lib/snapd/hostfs/{,*} rw,
/dev/hugepages wk,

and after compiling and install snap/snapd on my test system I get the following error from my snap:

Jun 11 22:52:36 dpdk-test kernel: [ 1408.757203] audit: type=1400 audit(1528757556.659:90): apparmor="DENIED" operation="open" profile="snap.dpdk-wililupy.testpmd" name="/var/lib/snapd/hostfs/mnt/huge/" pid=2546 comm="testpmd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

I’m using the User guide from DPDK about mounting the hugepages fs in /mnt/huge and when I run this in devmode or classic, this works, but running in confined (currently testing in jailmode) I get the error running the command.

I’m at a lost as to how to get /mnt loaded in the /var/lib/snapd/hostfs.

Any pointers?

Thanks,
Luke

ogra · 2018-06-12 09:27:44 UTC

if you really need /mnt i guess you have to wait for:

https://forum.snapcraft.io/t/mnt-mounts-should-be-supported

(and perhaps not use hostfs but access /mnt directly then)

zyga · 2018-06-12 09:29:38 UTC

As mentioned on IRC I think there are several topics here:

Does dpdk strictly require /mnt/huge as the mount point? If no then we could expose access to /dev/hugepages and ignore the extra mount point. As another possibility we could make a bind mount from /dev/hugepages to /mnt/huge when the interface is connected. Alternatively if the filesystem can be mounted multiple times we could mount a private instance of hugetblfs somewhere inside the mount namespace.

The rule /var/lib/snapd/hostfs/{,*} rw, grants full access to all the files in the root directory and even if corrected to use ** (which grants recursive access) would be a wide-open hole to all of the host. I don’t think we need that.

Ideally, to move forward, we could use a snap that can be used as a test vehicle and we would develop the most natural variant of the possibilities I mentioned.

jdstrand · 2018-06-12 13:29:10 UTC

I agree with @zyga’s points.

I will mention for completeness that your are seeing the denial because this rule should be: /var/lib/snapd/hostfs/{,**} rw,. But as @zyga said, this rule isn’t appropriate for the interface and we would want to come up with something better.

wililupy · 2018-06-12 16:25:50 UTC

Thank you everyone for the help.

Now that I understand how /var/lib/snaps/hostfs works I agree I need a better option.

Hugepage support is required for the large memory pool allocation used for packet buffers. By using hugepage allocations, performance is increased since fewer pages are needed, and therefore less Translation Lookaside Buffers (TLBs, high speed translation caches), which reduce the time it takes to translate a virtual page address to a physical page address. Without hugepages, high TLB miss rates would occur with the standard 4k page size, slowing performance.
To allow DPDK to use hugepages, after you set the parameter at boot with a kernel parameter (hugepages=1024)

In the deb version of DPDK, we use a script that is executed by systemd at startup that allocates the hugepages and mount point. I haven’t test if it is hardcoded to /mnt/huge, I will test that today, but if I am remember correct, all dpdk is looking for is a mount point with the flags of filesystem hugetlbfs and a nodev flag. I am using /mnt/huge since the DPDK user guide uses it as an example (section 2.3.2.2) and force of habit of using DPDK.

I will try the mount-observe interface and see what happens with that and let you know.

Thanks!

ogra · 2018-06-12 16:30:14 UTC

that makes everything easier, just have it mounted under /media then and (ab)use the removable-media interface

wililupy · 2018-06-12 16:43:36 UTC

I’ll give that a shot too. Thanks @ogra

wililupy · 2018-06-12 19:10:54 UTC

Ok. So I tried the mount-observe interface, but go the same error. I then tried mounting /huge under /media and using removable-media interface, but that resulted in the same error, but it did verify that DPDK is just looking for the hugetlbfs filesystem and the nodev flag since it did move the error to /var/lib/snapd/hostfs/media/huge.
I then modified my hugepages-observe interface that I am working on to just do the read permission for /var/lib/snapd/hostfs/ and that seemed to work, but now I get the following error when DPDK tries to write a lockfile to /var/lib/snapd/hostfs/media/huge:

Jun 12 19:01:46 dpdk-test kernel: [ 1305.404168] audit: type=1400 audit(1528830106.699:114): apparmor="DENIED" operation="file_lock" profile="snap.dpdk-wililupy.testpmd" name="/var/lib/snapd/hostfs/media/huge/" pid=2854 comm="testpmd" requested_mask="wk" denied_mask="wk" fsuid=0 ouid=0

Not quite sure how to proceed with this. I can test the hugepages-control interface with:
/var/lib/snapd/hostfs/{,**} wk,

But I fear that I’ll be in the same boat as I was before…

Thanks,
Luke

ogra · 2018-06-12 19:39:49 UTC

yeah, i actually meant /media, not /var/lib/snapd/hostfs/media you somehow need to teach the app to access it directly, not via the hostfs mount

wililupy · 2018-06-12 19:50:23 UTC

I wonder if layouts might fit here?

zyga · 2018-06-12 21:00:05 UTC

No, layouts are about taking your snap content and spreading it around the filesystem.

wililupy · 2018-06-12 22:04:04 UTC

So I changed my rule for hugepages-observe to read /var/lib/snapd/hostfs and that fixed me to the next step where I then got a deny for wk on /var/lib/snapd/hostfs/mnt/huge, so I created a rule in hugepages-control interface (temporary so I could continue working on confinement while I get guidence on the hostfs). I then was getting a mknod error for /run, but fixed that by adding /run/{,**} c, and I get the following error now:

Jun 12 21:46:37 dpdk-test kernel: [ 107.609886] audit: type=1400 audit(1528839997.736:31): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="/usr/lib/snapd/snap-confine" name="snap.dpdk-wililupy.testpmd" pid=1541 comm="snap-confine"

Not sure what this error means.

Thanks,
Luke

zyga · 2018-06-13 05:15:40 UTC

This means that the profile “snap.dpkd-wiliupy.testpmd” does not exist. Not sure why that happened.

wililupy · 2018-06-13 16:56:03 UTC

I have the profile in /var/lib/snapd/apparmor/profiles.

I’m still getting the error. Not quite sure how to proceed.

wililupy · 2018-06-13 17:25:35 UTC

Now I’m getting somewhere. I removed my snap, recompiled it with strict and stable confinement and grade, and when I was connecting the interfaces, when I went to connect the hugepages-control interface, I got this error:

Jun 13 17:08:21 dpdk-test kernel: [ 1221.310557] audit: type=1400 audit(1528909701.980:63): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/snap/core/4650/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=3364 comm="apparmor_parser"
Jun 13 17:08:21 dpdk-test kernel: [ 1221.323305] audit: type=1400 audit(1528909701.992:64): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="snap-update-ns.core" pid=3366 comm="apparmor_parser"
Jun 13 17:08:22 dpdk-test kernel: [ 1221.331349] audit: type=1400 audit(1528909702.000:65): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="snap.core.hook.configure" pid=3368 comm="apparmor_parser"
Jun 13 17:08:22 dpdk-test kernel: [ 1221.413606] audit: type=1400 audit(1528909702.080:66): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="snap-update-ns.dpdk-wililupy" pid=3375 comm="apparmor_parser"

And this was from the terminal:

error: cannot perform the following tasks:
- Connect dpdk-wililupy:hugepages-control to core:hugepages-control (cannot setup apparmor for snap "dpdk-wililupy": cannot load apparmor profile "snap.dpdk-wililupy.testpmd": cannot load apparmor profile: exit status 1
apparmor_parser output: 
AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap.dpdk-wililupy.testpmd in /var/lib/snapd/apparmor/profiles/snap.dpdk-wililupy.testpmd at line 449: syntax error, unexpected TOK_ID, expecting TOK_MODE
)
- Connect dpdk-wililupy:hugepages-control to core:hugepages-control (cannot load apparmor profile "snap.dpdk-wililupy.testpmd": cannot load apparmor profile: exit status 1
apparmor_parser output:
AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap.dpdk-wililupy.testpmd in /var/lib/snapd/apparmor/profiles/snap.dpdk-wililupy.testpmd at line 449: syntax error, unexpected TOK_ID, expecting TOK_MODE
)

I pushed my snapd changes to my git repo (https://github.com/wililupy/snapd) and so I think I have an issue with my hugepages-control interface. I’ll look there and see what I dig up.

Thanks,
Luke

wililupy · 2018-06-14 00:00:48 UTC

Success!
I ended up fixing the issue with /run and I had to add plugs for system-observe and process-control but now it executes properly.

I just have to figure out the hostfs mapping of /mnt since the setting I have in hugepages-control is wk.

I’m thinking that this might be ok since you have to be root to execute this application anyways (or at least sudo)
What do you think?

zyga · 2018-06-14 08:00:13 UTC

I am working on the /mnt aspect and I will update this thread when that is available. Right now it is not something that can be done with interfaces alone.