Python multiprocessing sem_open blocked in strict mode

snapcraft
1

I’m trying to snap an app that uses Python’s multiprocessing module but it’s getting blocked by AppArmour when accessing /dev/shm/<randomName>.

I traced the problem to the SemLock implementation in multiprocessing, specifically sem_open call, and changed the name format so that it creates a semaphore at /dev/shm/sem.snap.<snapName>.<randomName> since that path is allowed in the AppArmour profile. But that didn’t work. I then ran strace on the app and saw that sem_open() first issues an open() at /dev/shm/<randName> and then link() to /dev/shm/sem.snap... So open() call will always get blocked by AppArmour regardless of which name is specified as sem_open() argument.

Apparently, that’s the way sem_open is implemented, as is mentioned in a similar bug report here: https://github.com/sergiusens/snapcraft-preload/issues/2

So does anyone have an idea as to how this may be circumvented?

1 Like
2

Today you would need to adjust the call to sem_open(). As mentioned in the snapcraft-preload issue, it was always planned that snapcraft-preload would intercept the sem_open() call so that you don’t have to do anything when using snapcraft-preload. It looks like that work wasn’t completed. If you are able, perhaps you would be interested in using snapcraft-preload in your snap and submitting a patch for the aforementioned issue?

1 Like
3

Thanks. I’m going to have a look at snapcraft-preload and try to create a patch. Any pointers that you may have would be very useful.

4

@Trevinho should be able to help you in the snapcraft-preload issue (@3v1n0 in github; he has done quite a bit of work lately on it). As for using it at all, see https://github.com/sergiusens/snapcraft-preload. While I haven’t tried it myself, it seems you should be able to point ‘source’ at your remote branch, make changes and pull it in. @Trevinho may be able to comment on a better workflow for hacking on snapcraft-preload (it would be nice if had a HACKING file).

1 Like
5

@Trevinho - fyi this thread (I mistakenly referred to you by your github id so not sure you would be emailed my corrected comment, above).

6

Thanks for the info, I looked at the source and I understand what’s required. Will post updates here.

1 Like
7

Oh, I can look at this indeed… I’d just would love to get my C++ rewrite of snapcraft-preload to be merged before doing further modifications though. I’m waiting for @kyrofa last word and (once he’s back) @sergiusens merge.

1 Like
8

Hi, I have written the implementation of sem_open and sem_unlink and it seems to be working but still needs testing. I will push this to my repo and when the C++ changes get in, I can rewrite this bit and hopefully get it merged.

1 Like
9

@pop I hit the same problem today with one of our project involving python multiprocessing + strict confinement, I tried the snapcraft-preload suggested by the snappy-debug but It doesn’t seem to be solving it. Do you have any updates on how to solve this?

= AppArmor =
Time: Feb 15 09:50:07
Log: apparmor="DENIED" operation="mknod" profile="snap.brightedge.be" name="/dev/shm/qqSBOG" pid=28056 comm="python2" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
File: /dev/shm/qqSBOG (write)
Suggestions:
* adjust program to create files and directories in /dev/shm/snap.$SNAP_NAME.*
* try the snapcraft preload plugin: https://github.com/sergiusens/snapcraft-preload
10

Apologies for reviving the topic, however we recently hit the same issue and plan to use the workaround documented in https://bugs.python.org/issue19478 to configure the prefix for the semaphore such that it matches the apparmor rules.

Will report back if that’s successful!

1 Like
11

Unfortunately this didn’t work, I failed to get Python’s multiprocessing to generate a semaphore with a name that would match the apparmor rules. :frowning_face:

1 Like
12

Yes, this is bad and sad. My package is also dependant of Python multiprocessing. :confounded:

13

I took some time to take a look at this more closely and created a POC LD_PRELOAD library that seems to work[1]. I don’t consider this production code but hopefully some will find it helpful. Anyone is free to pick this up and submit it to the snapcraft-preload project.

Example:

$ sudo snap install test-sem-open --edge
test-sem-open 0 installed

$ test-sem-open.<tab>
test-sem-open.anon          test-sem-open.py2           test-sem-open.py3-broken
test-sem-open.named         test-sem-open.py2-broken    
test-sem-open.named-broken  test-sem-open.py3           

$ test-sem-open.anon 1 2 # unnamed semaphores always and continue to work
main() about to call sem_timedwait()
sem_post() from handler
sem_timedwait() succeeded

$ test-sem-open.named 1 2 # now named ones work
main() about to call sem_timedwait()
sem_post() from handler
sem_timedwait() succeeded

$ test-sem-open.py2 # even in python2
1 tasks are running
2 tasks are running
3 tasks are running
<Process(Process-3, started)> has finished
<Process(Process-2, started)> has finished
<Process(Process-1, started)> has finished

$ test-sem-open.py3 # and python3
1 tasks are running
2 tasks are running
3 tasks are running
<Process(Process-2, started)> has finished
<Process(Process-3, started)> has finished
<Process(Process-1, started)> has finished

$ test-sem-open.named-broken 1 2 # broken without LD_PRELOAD
sem_open: Permission denied
[1]
$ grep snap.test-sem-open.named-broken /var/log/syslog
audit: type=1400 audit(1544131584.937:2210): apparmor="DENIED" operation="mknod" profile="snap.test-sem-open.named-broken" name="/dev/shm/ueW6dZ" pid=5674 comm="sem-named-test" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000

$ test-sem-open.py2-broken 
Traceback (most recent call last):
  File "/snap/test-sem-open/x1/bin/mp-test.py", line 47, in <module>
    test_semaphore()
  File "/snap/test-sem-open/x1/bin/mp-test.py", line 28, in test_semaphore
    sema = multiprocessing.Semaphore(3)
  File "/snap/test-sem-open/x1/usr/lib/python2.7/multiprocessing/__init__.py", line 197, in Semaphore
    return Semaphore(value)
  File "/snap/test-sem-open/x1/usr/lib/python2.7/multiprocessing/synchronize.py", line 111, in __init__
    SemLock.__init__(self, SEMAPHORE, value, SEM_VALUE_MAX)
  File "/snap/test-sem-open/x1/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 13] Permission denied
[1]
$ grep 'snap.test-sem-open.py2-broken' /var/log/syslog
audit: type=1400 audit(1544131603.329:2211): apparmor="DENIED" operation="mknod" profile="snap.test-sem-open.py2-broken" name="/dev/shm/4ULMJ2" pid=5699 comm="python2" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000

$ test-sem-open.py3-broken 
Traceback (most recent call last):
  File "/snap/test-sem-open/x1/bin/mp-test3.py", line 47, in <module>
    test_semaphore()
  File "/snap/test-sem-open/x1/bin/mp-test3.py", line 28, in test_semaphore
    sema = multiprocessing.Semaphore(3)
  File "/snap/test-sem-open/x1/usr/lib/python3.5/multiprocessing/context.py", line 81, in Semaphore
    return Semaphore(value, ctx=self.get_context())
  File "/snap/test-sem-open/x1/usr/lib/python3.5/multiprocessing/synchronize.py", line 127, in __init__
    SemLock.__init__(self, SEMAPHORE, value, SEM_VALUE_MAX, ctx=ctx)
  File "/snap/test-sem-open/x1/usr/lib/python3.5/multiprocessing/synchronize.py", line 60, in __init__
    unlink_now)
PermissionError: [Errno 13] Permission denied
[1]
$ grep 'snap.test-sem-open.py3-broken' /var/log/syslog
audit: type=1400 audit(1544131645.601:2212): apparmor="DENIED" operation="mknod" profile="snap.test-sem-open.py3-broken" name="/dev/shm/8NJVRZ" pid=5733 comm="python3" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000

I confirmed this to work on amd64, armhf and arm64.

[1]https://git.launchpad.net/~jdstrand/+git/test-sem-open/tree/lib.c

1 Like
14

OK, I’m picking this up and massaging it into snapcraft-preload. If I fail then I’ll report back here, but otherwise expect a PR on @sergiusens’ github repo.

15

Great! Note that the code before was considered PoC and there was an important TODO item in it. I implemented this now, but in doing so noticed that the default policy does not allow hardlinking to the semaphore file, so I’m filing a PR to snapd and added a fallback mechanism to the library if the hardlink fails with permission denied. As such, until the PR is committed to snapd, the above pasted output will be different and there will be a “l” denial, but the operation on the whole continues to be successful. Example successful output until the PR lands:

$ test-sem-open.named 1 2
sem_open() wrapper: hard linking tempfile denied. Falling back to rename()
main() sem_getvalue() init: 0
main() about to call sem_timedwait()
sem_post() from handler
sem_getvalue() from hander pre: 0
sem_getvalue() from hander post: 1
sem_timedwait() succeeded
main() sem_getvalue() after sem_timedwait: 0

$ grep test-sem-open.named /var/log/syslog
audit: type=1400 audit(1544198965.450:2332): apparmor="DENIED" operation="link" profile="snap.test-sem-open.named" name="/dev/shm/sem.snap.test-sem-open.sem-named-test" pid=13742 comm="sem-named-test" requested_mask="l" denied_mask="l" fsuid=1000 ouid=1000 target="/dev/shm/snap.test-sem-open.sem-named-test.lTZ0rH"

Notice the new sem_open() wrapper: hard linking tempfile denied. Falling back to rename() message and a new, but harmless hardlink denial for the rewritten path.

The broken behavior remains the same:

$ test-sem-open.named-broken 1 2
sem_open: Permission denied
[1]

$ grep test-sem-open.named /var/log/syslog
audit: type=1400 audit(1544199149.495:2334): apparmor="DENIED" operation="mknod" profile="snap.test-sem-open.named-broken" name="/dev/shm/toNu6V" pid=13872 comm="sem-named-test" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000

Notice the broken, non-rewritten behavior is a “c” denial with an un-rewritten path.

2 Likes
16

FYI: the PRs for snapd are:

2 Likes
17

and here’s my PR on snapcraft-preload:

https://github.com/sergiusens/snapcraft-preload/pull/29

1 Like
18

My install hook fails because it tries to do things in parallel, using all CPU cores https://github.com/deskconn/deskconnd/blob/9a3d86d25904c0e537fd03e7ecc4f5f131de62aa/snap/hooks/install#L9 – Is there an update on the situation or do we still have to use snapcraft-preload thing ?

Here is the failure I see

error: cannot perform the following tasks:
- Run post-refresh hook of "deskconnd" snap if present (run hook "post-refresh": 
-----
Traceback (most recent call last):
  File "/snap/deskconnd/142/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/snap/deskconnd/142/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/snap/deskconnd/142/usr/lib/python3.8/compileall.py", line 327, in <module>
    exit_status = int(not main())
  File "/snap/deskconnd/142/usr/lib/python3.8/compileall.py", line 309, in main
    if not compile_dir(dest, maxlevels, args.ddir,
  File "/snap/deskconnd/142/usr/lib/python3.8/compileall.py", line 85, in compile_dir
    with ProcessPoolExecutor(max_workers=workers) as executor:
  File "/snap/deskconnd/142/usr/lib/python3.8/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/snap/deskconnd/142/usr/lib/python3.8/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/snap/deskconnd/142/usr/lib/python3.8/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/snap/deskconnd/142/usr/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/snap/deskconnd/142/usr/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/snap/deskconnd/142/usr/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
PermissionError: [Errno 13] Permission denied
-----)
19

snapcraft-preload does not yet have the support added yet (though there is an open PR). You can preload using Python multiprocessing sem_open blocked in strict mode

1 Like
20

snapcraft-preload seems to work with base: core20 however does not work with base: core22. It seems that the snap FS is corrupt/unreadable. Anyone have any luck?