Problems packaging app that uses gunicorn

iconloop · June 11, 2019, 12:27pm

loopchain need to access /tmp directory because of gunicorn access to /tmp.

loopchain github repository is :
https://github.com/icon-project/loopchain

snapcraft.yaml:

plugs:
  temporary-data:
    interface: system-files
    read: [ /tmp ]
    write: [ /tmp ]

apps:
  loopchain:
    command: bin/snapcraft-preload $SNAP/bin/loopchain
    plugs:
      - home
      - network
      - network-bind
      - temporary-data
  iconrpcserver:
    command: bin/snapcraft-preload $SNAP/bin/iconrpcserver
    plugs:
      - home
      - network
      - network-bind
      - temporary-data
  iconservice:
    command: bin/snapcraft-preload $SNAP/bin/iconservice
    plugs:
      - home
      - network
      - network-bind
      - temporary-data

gunicorn shows following errors when loopchain started.

Traceback (most recent call last):
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/arbiter.py", line 203, in run
    self.manage_workers()
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
    self.spawn_workers()
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
    self.spawn_worker()
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/arbiter.py", line 565, in spawn_worker
    self.cfg, self.log)
  File "/snap/loopchain/6/lib/python3.6/site-packages/sanic/worker.py", line 30, in __init__
    super().__init__(*args, **kw)
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/workers/base.py", line 58, in __init__
    self.tmp = WorkerTmp(cfg)
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/workers/workertmp.py", line 26, in __init__
    util.chown(name, cfg.uid, cfg.gid)
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/util.py", line 173, in chown
    os.chown(path, uid, gid)
PermissionError: [Errno 1] Operation not permitted: '/tmp/wgunicorn-7phm2h5f'

Thank you.

chipaca · June 11, 2019, 12:46pm

the problem is not access to /tmp ,the problem is that you’re trying to chown things, and that is not going to work.

iconloop · June 11, 2019, 1:31pm

thank you for your reply. @chipaca
Then, Is I need other plug to change owner?
or change tmp directory to somewhere to work chown?

please give me hint. thanks.

chipaca · June 11, 2019, 1:33pm

you can’t chown/setuid/etc inside a snap. Tell gunicorn not to do it (it has options for that, I’d wager).

roadmr · June 11, 2019, 2:48pm

https://docs.gunicorn.org/en/stable/faq.html#blocking-os-fchmod might be of interest, as might https://docs.gunicorn.org/en/stable/settings.html#worker-tmp-dir, though there doesn’t appear to be a way to tell gunicorn via confi to not chown things.

Daniel

iconloop · June 12, 2019, 11:12am

I try to change worker_tmp_dir to /mem in gunicorn.

create tmpfs on /mem as following :

sudo mkdir -p /mem
sudo mount -t tmpfs -o size=32M tmpfs /mem

Then looks like this…

$ df /mem
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs              32768     0     32768   0% /mem

But, gunicorn can’t find /mem directory.

Traceback (most recent call last):
  File "/snap/loopchain/10/lib/python3.6/site-packages/gunicorn/arbiter.py", line 203, in run
    self.manage_workers()
  File "/snap/loopchain/10/lib/python3.6/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
    self.spawn_workers()
  File "/snap/loopchain/10/lib/python3.6/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
    self.spawn_worker()
  File "/snap/loopchain/10/lib/python3.6/site-packages/gunicorn/arbiter.py", line 565, in spawn_worker
    self.cfg, self.log)
  File "/snap/loopchain/10/lib/python3.6/site-packages/sanic/worker.py", line 30, in __init__
    super().__init__(*args, **kw)
  File "/snap/loopchain/10/lib/python3.6/site-packages/gunicorn/workers/base.py", line 58, in __init__
    self.tmp = WorkerTmp(cfg)
  File "/snap/loopchain/10/lib/python3.6/site-packages/gunicorn/workers/workertmp.py", line 22, in __init__
    raise RuntimeError("%s doesn't exist. Can't create workertmp." % fdir)
RuntimeError: /mem/ doesn't exist. Can't create workertmp.

Is there any alternative to do?

bloodearnest · June 12, 2019, 12:31pm

Hi

The good news is, we have a snaps that runs multiple gunicorns without any chown issues, so its definitely possible.

I suspect that this is something to do with your use for system-files for /tmp - why do you need this?

By default, every snap gets its own isolated /tmp tmpfs mount, so your gunicorn has access to that. That’s what we use in our snap, and it works just fine with gunicorn. I would suggest removing the temporary-data plug altogether and seeing if that works.

HTH

iconloop · June 12, 2019, 2:40pm

hi @bloodearnest

I already removed temporary-data plug.

I try first time, I added [network, netwotk-bind] plugs.

I wonder where can I find isolated /tmp directory for my snap app?

default /tmp is not work.

zyga-snapd · June 12, 2019, 5:47pm

This is not visible from the mount namespace used by snap applications.

I would recommend that you inspect gunicorn to understand why it wants to chown and chmod things it creates and come back with this information. As stated elsewhere the sandbox is not going to allow that.

iconloop · June 13, 2019, 2:30pm

hi. @zyga-snapd

loopchain use sanic + gunicorn.

sanic is web server and running via gunicorn with GunicornWorker.

This is gunicorn workers/base.py parent of GunicornWorker.

github.com

benoitc/gunicorn/blob/19.9.0/gunicorn/workers/base.py#L63


    self.nr = 0
    jitter = randint(0, cfg.max_requests_jitter)
    self.max_requests = cfg.max_requests + jitter or MAXSIZE
    self.alive = True
    self.log = log
    self.tmp = WorkerTmp(cfg)
def __str__(self):
    return "<Worker %s>" % self.pid
def notify(self):
    """\
    Your worker subclass must arrange to have this method called
    once every ``self.timeout`` seconds. If you fail in accomplishing
    this task, the master process will murder your workers.
    """
    self.tmp.notify()
def run(self):
    """\
    This is the mainloop of a worker process. You should override

gunicorn create worker process and worker check alive by “worker temp file”.

github.com

benoitc/gunicorn/blob/19.9.0/gunicorn/workers/workertmp.py#L26


class WorkerTmp(object):
    def __init__(self, cfg):
        old_umask = os.umask(cfg.umask)
        fdir = cfg.worker_tmp_dir
        if fdir and not os.path.isdir(fdir):
            raise RuntimeError("%s doesn't exist. Can't create workertmp." % fdir)
        fd, name = tempfile.mkstemp(prefix="wgunicorn-", dir=fdir)
        # allows the process to write to the file
        util.chown(name, cfg.uid, cfg.gid)
        os.umask(old_umask)
        # unlink the file so we don't leak tempory files
        try:
            if not IS_CYGWIN:
                util.unlink(name)
            self._tmp = os.fdopen(fd, 'w+b', 1)
        except:
            os.close(fd)
            raise

do chown to allow worker process write to temp file when initialize worker process.
GunicornWorker call notify every seconds and temp file updated by os.chmod.

If failed check alive in timeout, process killed by gunicorn master worker.

I think loopchain can’t use strict confinement unless fix or replace gunicorn or allow permission system-files.

thanks.

zyga-snapd · June 13, 2019, 2:59pm

You can configure uid and gid to -1, this is a no-op then AFAIK.

iconloop · June 14, 2019, 11:37am

hi @zyga-snapd

I’ve found your reply this forum.

So, I have monkey patched to avoid to call os.chown in gunicorn.

here is Pull request of our project.

It works now in strict confinement.

thanks a lot.

jdstrand · June 17, 2019, 9:34pm

It depends on if it hits the syscall. This would be denied by seccomp if it did. Perhaps it is better to use an LD_PRELOAD or patch the source to not chown.

nottrobin · February 11, 2020, 4:26pm

I have this same problem and I don’t relish forking Gunicorn to solve it. I think it’s probably going to simply lead to me giving up on making my snap strict. Anyone got any other ideas?

iconloop · February 12, 2020, 2:21am

@nottrobin

This issue resolved in gunicorn 20.0.0 or later.

There is no need to ask the gunicorn. Just upgrade your gunicorn to 20.0.x.
If you want to use the current version of gunicorn, you can apply the “monkey patch” above.

nottrobin · February 13, 2020, 9:43am

Ah great! Yes, it works fine with Gunicorn 20.0.4. Thanks for pointing that out @iconloop.

@bloodearnest has updated Talisker to support Gunicorn >= 20 too!