Snapcraft-preload and /dev/shm access

hunterk · January 7, 2022, 12:59am

I have a few precompiled dynamic lib plugins that fail to load in my snap due to failure to shm_open due to the well-documented sandboxing of /dev/shm. Most of the threads about this topic have ended with browsers getting an exception or adding the snapcraft-preload part to the snapcraft.yaml and putting it after desktop-launch in the app command.

I’ve tried adding snapcraft-preload part (just copy/pasted from its github repo’s README) and the resulting snap always fails to launch with “snapcraft-preload: no such file or directory”, which I think is a result of that part failing to install properly for some reason.

Is this still the preferred way to handle this in core20? Is there something obvious I’m missing, like needing to specify a different prefix? (I recall reading something about default prefix changing from /usr to /usr/local with the transition to core20?)

I also read that layout binding could be a solution, but I read elsewhere that /dev/shm is off-limits for that.

So, I’m just a little lost on how I should handle this, and it seems like most of the action for this problem was back in 2017

hunterk · January 11, 2022, 4:58am

Replying to myself because I’m past the editing window:

It looks like this recent thread has all of the info I needed (I had read through it but it didn’t really sink in, apparently): Help creating a strict snap that uses shared memory

The important thing is that core20 does indeed install everything in /usr/local instead of /usr or /, so if you use the snapcraft-preload part, you need to add another line to set the prefix to match the older standard:

    plugin: cmake
    cmake-parameters:
      - -DCMAKE_INSTALL_PREFIX=/usr

hunterk · January 14, 2022, 4:57am

well, it seems I spoke too soon. My application still can’t shm_open. When I run it, I get:

ERROR: ld.so: object '/snap/retroarch/1140/usr/lib/snapcraft-preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

And then my plugins that use shared memory (namely, dolphin, swanstation, and citra libretro cores) still fail with (example from dolphin):

[libretro ERROR] 38:54:336 Common/MemArena.cpp:88 E[MI]: shm_open failed: Permission denied

I’ve tried adding export LD_PRELOAD=$SNAP/usr/lib/libsnapcraft-preload.so to my snapcraft.yaml recipe’s environment block, I’ve tried adding it to my launcher script. Nothing seems to help. Does anyone have any suggestions? Here’s my snapcraft.yaml if that helps for reference.

ogra · January 14, 2022, 12:13pm

there seems to be a slight name discrepancy, did you check $SNAP/usr/lib which is the correct name for the .so ?

hunterk · January 15, 2022, 12:34am

oh, sorry, it actually tries both:

ERROR: ld.so: object '/snap/retroarch/1140/lib/libsnapcraft-preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/snap/retroarch/1140/usr/lib/snapcraft-preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

but when I run snap run --shell retroarch and look in $SNAP/usr/lib, I see:

$ ls $SNAP/usr/lib | grep preload
libsnapcraft-preload.so
libsnapcraft-preload32.so

so, it looks like at least one of them should be found

ogra · January 15, 2022, 11:12am

it does not look for libsnapcraft-preload.so in $SNAP/usr/lib, only in $SNAP/lib …

hunterk · January 15, 2022, 2:16pm

ah, yeah, you’re right. clearly my brain has melted lol

So, I guess my real question is: what is snapcraft-preload doing? Exporting the proper path doesn’t seem to stop the errors–unless I’m just not doing it in the right place; is there a specific place where this should work?–and in any event, I still can’t shm_open anything. Should I be able to at this point? Is there something else I should have done to make this work with core20 aside from just setting the prefix to /usr instead of /usr/local?

ogra · January 15, 2022, 2:27pm

i’d assume you want to fix the path in $SNAP/bin/snapcraft-preload to contain usr/ …

hunterk · January 16, 2022, 4:31am

eyy, success

I needed to export the (correct) path in my launcher script and set a compile flag in snapcraft-preload.

Here’s the full snapcraft-preload part if anyone needs:

  snapcraft-preload:
    source: https://github.com/sergiusens/snapcraft-preload.git
    plugin: cmake
    cmake-parameters:
      - -DCMAKE_INSTALL_PREFIX=/usr -DLIBPATH=/lib
    build-packages:
      - on amd64:
        - gcc-multilib
        - g++-multilib
    stage-packages:
      - on amd64:
        - lib32stdc++6

It still coughs up an error, but shm_open actually works, so problem solved. Thanks for the help.

nemwiz · January 31, 2022, 12:04am

I’m running into same issue but the solution provided here or in the other threads is not working for me. I’m using Python multiprocessing module and this module tries to write to /dev/shm.

So even with the above solution, I still get the following error:

ERROR: ld.so: object '/snap/cursorly/x1/lib/libsnapcraft-preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared obCan anyone heject file)

And the rewrite doesn’t work because another error is thrown:

PermissionError: [Errno 13] Permission denied: '/dev/shm/pym-8525-uxcku8lm'

Here is a portion of my snapcraft.yml

apps:
  cursorly:
    command: bin/desktop-launch $SNAP/usr/bin/snapcraft-preload $SNAP/electron-app/cursorly --no-sandbox
    extensions:
      - gnome-3-38
    environment:
      TMPDIR: $XDG_RUNTIME_DIR
    plugs:
      - browser-support
      - camera
      - desktop
      - desktop-legacy
      - network
      - network-bind
      - opengl
      - wayland
      - x11

snapcraft-preload:
      source: https://github.com/sergiusens/snapcraft-preload.git
      plugin: cmake
      cmake-parameters:
        - -DCMAKE_INSTALL_PREFIX=/usr -DLIBPATH=/lib
      build-packages:
        - on amd64:
            - gcc-multilib
            - g++-multilib
      stage-packages:
        - lib32stdc++6

Here are a few things I’ve tried without success.

I’ve tried setting LD_PRELOAD env variable (still getting an error that it can’t find the libsnapcraft-preload.so
I’ve unsquashed the snap and manually copied the .so file in the lib folder and received a Traceback core dumped error
Played around with different paths

Am I missing something obvious? I’ve been stuck with this for a few days now.

nemwiz · February 6, 2022, 2:05am

Can some kind soul please help on this? Before I lose my sanity? I don’t know what to try anymore.

The libsnapcraft files are in this location…

ls squashfs-root/usr/lib/ | grep libsnapcraft
libsnapcraft-preload32.so
libsnapcraft-preload.so

And the preload script is pointing to that location

cat bin squashfs-root/usr/bin/snapcraft-preload 
cat: bin: No such file or directory
#!/bin/sh

export SNAPCRAFT_PRELOAD=$SNAP
export LD_PRELOAD="$SNAP/usr/lib/libsnapcraft-preload.so"

exec "$@"

My snap runs but I get a very cryptic No protocol specified error. I’ve run the gdb and I can’t make sense of this output.

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /snap/snapd/14549/usr/lib/snapd/snap-confine...
(No debugging symbols found in /snap/snapd/14549/usr/lib/snapd/snap-confine)
Starting program: /snap/snapd/14549/usr/lib/snapd/snap-confine --base core20 snap.cursorly.cursorly /usr/lib/snapd/snap-exec --command=gdb cursorly
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 13779]
[Detaching after fork from child process 13780]
process 13775 is executing new program: /usr/lib/snapd/snap-exec
[New LWP 13782]
[New LWP 13783]
[New LWP 13784]
[New LWP 13785]
[LWP 13785 exited]
[LWP 13784 exited]
[LWP 13783 exited]
[LWP 13782 exited]
process 13775 is executing new program: /usr/bin/dash
process 13775 is executing new program: /usr/bin/bash
[Detaching after fork from child process 13786]
[Detaching after fork from child process 13787]
[Detaching after fork from child process 13790]
[Detaching after fork from child process 13791]
[Detaching after fork from child process 13792]
[Detaching after fork from child process 13797]
mkdir: cannot create directory '/run/user/0': Permission denied
[Detaching after fork from child process 13798]
[Detaching after fork from child process 13799]
[Detaching after fork from child process 13801]
[Detaching after fork from child process 13803]
[Detaching after fork from child process 13805]
[Detaching after fork from child process 13807]
[Detaching after fork from child process 13809]
[Detaching after fork from child process 13811]
[Detaching after fork from child process 13813]
[Detaching after fork from child process 13815]
process 13775 is executing new program: /usr/lib/snapd/snap-gdb-shim


DEPRECATED: Please consider using --gdbserver instead.

Welcome to `snap run --gdb`.
You are right before your application is execed():
- set any options you may need
- use 'cont' to start



Thread 1 "snap-gdb-shim" received signal SIGTRAP, Trace/breakpoint trap.
0x000000000040e728 in ?? ()
Catchpoint 1 (exec)
Continuing.
process 13775 is executing new program: /usr/bin/bash

Thread 1 "desktop-launch" hit Catchpoint 1 (exec'd /usr/bin/bash), 0x00007f81e41be100 in ?? () from target:/lib64/ld-linux-x86-64.so.2
(gdb) cont
Continuing.
[Detaching after fork from child process 13816]
[Detaching after fork from child process 13817]
[Detaching after fork from child process 13820]
[Detaching after fork from child process 13823]
[Detaching after fork from child process 13824]
mkdir: cannot create directory ‘/run/user/0’: Permission denied
[Detaching after fork from child process 13825]
[Detaching after fork from child process 13827]
[Detaching after fork from child process 13829]
[Detaching after fork from child process 13831]
[Detaching after fork from child process 13833]
[Detaching after fork from child process 13835]
[Detaching after fork from child process 13837]
[Detaching after fork from child process 13839]
[Detaching after fork from child process 13841]
[Detaching after fork from child process 13843]
[Detaching after fork from child process 13845]
[Detaching after fork from child process 13847]
[Detaching after fork from child process 13849]
[Detaching after fork from child process 13851]
[Detaching after fork from child process 13853]
[Detaching after fork from child process 13855]
[Detaching after fork from child process 13857]
[Detaching after fork from child process 13859]
[Detaching after fork from child process 13860]
[Detaching after fork from child process 13862]
[Detaching after fork from child process 13864]
[Detaching after fork from child process 13866]
[Detaching after fork from child process 13868]
[Detaching after fork from child process 13870]
[Detaching after fork from child process 13872]
[Detaching after fork from child process 13874]
[Detaching after fork from child process 13876]
[Detaching after fork from child process 13877]
[Detaching after fork from child process 13878]
[Detaching after fork from child process 13879]
[Detaching after fork from child process 13880]
[Detaching after fork from child process 13881]
[Detaching after fork from child process 13882]
[Detaching after fork from child process 13883]
[Detaching after fork from child process 13884]
[Detaching after fork from child process 13885]
[Detaching after fork from child process 13886]
[Detaching after fork from child process 13887]
[Detaching after fork from child process 13889]
[Detaching after fork from child process 13890]
[Detaching after fork from child process 13891]
[Detaching after fork from child process 13892]
[Detaching after fork from child process 13893]
[Detaching after fork from child process 13894]
[Detaching after fork from child process 13895]
[Detaching after fork from child process 13896]
[Detaching after fork from child process 13897]
[Detaching after fork from child process 13898]
process 13775 is executing new program: /usr/bin/dash

Thread 1 "snapcraft-prelo" hit Catchpoint 1 (exec'd /usr/bin/dash), 0x00007f54b6de9100 in ?? () from target:/lib64/ld-linux-x86-64.so.2
(gdb) cont
Continuing.
process 13775 is executing new program: /snap/cursorly/x1/electron-app/cursorly

Thread 1 "cursorly" hit Catchpoint 1 (exec'd /snap/cursorly/x1/electron-app/cursorly), 0x00007f69adff2100 in ?? ()
   from target:/lib64/ld-linux-x86-64.so.2
(gdb) cont
Continuing.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7f69aa099700 (LWP 13899)]
[Detaching after fork from child process 13900]
[Detaching after fork from child process 13901]
[New Thread 0x7f69a9898700 (LWP 13902)]
[New Thread 0x7f69a9097700 (LWP 13903)]
[New Thread 0x7f69a8095700 (LWP 13905)]
[New Thread 0x7f69a8896700 (LWP 13904)]
[New Thread 0x7f69a7894700 (LWP 13906)]
[New Thread 0x7f69a7093700 (LWP 13907)]
[New Thread 0x7f69a688e700 (LWP 13908)]
[New Thread 0x7f69a608d700 (LWP 13909)]
[New Thread 0x7f69a588c700 (LWP 13910)]
[New Thread 0x7f69a508b700 (LWP 13911)]
[New Thread 0x7f69a488a700 (LWP 13912)]
09:45:46.654 › Server executable file path /snap/cursorly/x1/bin/server/main.py
[Detaching after fork from child process 13913]
09:45:46.665 › Environment configuration file path /snap/cursorly/x1/electron-app/resources/app/.env
09:45:46.665 › Gui file path /snap/cursorly/x1/gui/dist/index.html
09:45:46.666 › Cursorly app version 1.2.0
No protocol specified

Thread 1 "cursorly" received signal SIGTRAP, Trace/breakpoint trap.
0x000055cabdb6f4c6 in ?? ()

c-kr · February 22, 2022, 11:20am

I was able to fix this by adding a symlink:

override-build: |
  [...]
  ln -sf ../usr/lib/libsnapcraft-preload.so $SNAPCRAFT_PART_INSTALL/lib/libsnapcraft-preload.so

Daedalus28288 · April 29, 2022, 4:53am

I am hanging on exactly this problem.

I have tried using the cmake parameters mentioned above. This does not fix my problem. Using the symlink suggested by @c-kr does work BUT using override-build stops snapcraft from pulling in pip packages stated in python-packages:

This makes it currently impossible for me to get my snap working. 1.) vanilla snap package without preload -> apparmor problem 2.) snap with preload -> libsnapcraft-preload.so not found BUT pip working 3.) snap with preload AND override-build -> preload works BUT pip not working.

Unfortunately I do not know how to fix this… is there a suggested way to get snapcraft preload to install the libraries in the correct location on core20 without using override-build?

I certainly may just be doing something wrong…

ogra · April 29, 2022, 9:05am

you need to explicitly run snapcraftctl build when running override build, else only your ln command is executed …

Daedalus28288 · April 29, 2022, 6:03pm

that fixed it! Thank you. I looked it up and don’t know how I missed that part in the documentation. Thank you for pointing me in the right direction.