Snapping Folding@Home, distributed computing for research on diseases

fcole90 · March 27, 2020, 11:42am

This is weird

$ snap run --shell folding-at-home-fcole90.FAHClient
$ ls $SNAP/usr/lib/x86_64-linux-gnu/ | grep libOpenCL
libOpenCL.so
libOpenCL.so.1
libOpenCL.so.1.0.0
$ cat $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so
cat: /snap/folding-at-home-fcole90/11/usr/lib/x86_64-linux-gnu/libOpenCL.so: No such file or directory
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so; then echo "Exists"; else echo "Not found"; fi
Not found
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so.1; then echo "Exists"; else echo "Not found"; fi
Exists
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0; then echo "Exists"; else echo "Not found"; fi
Exists

The file is listed there but it can’t read it?

Edit: it actually can’t read the symlinked file:

ls -la  $SNAP/usr/lib/x86_64-linux-gnu/ | grep libOpenCL
lrwxrwxrwx  1 root root       39 Mar 27 11:57 libOpenCL.so -> usr/lib/x86_64-linux-gnu/libOpenCL.so.1
lrwxrwxrwx  1 root root       18 Apr  5  2017 libOpenCL.so.1 -> libOpenCL.so.1.0.0
-rw-r--r--  1 root root    43072 Apr  5  2017 libOpenCL.so.1.0.0

fcole90 · March 27, 2020, 12:28pm

Using ln -rs ... did the trick

Now I have a new error on OpenCL:

OpenCL: Not detected: clGetPlatformIDs() returned -1001

clGetPlatformIDs()

Edit: I think I had messed up with the library path

ogra · March 27, 2020, 2:48pm

any denials in the journal when this message shows up ?

fcole90 · March 27, 2020, 3:03pm

I discovered that this last issue was actually caused by the library path being wrong. I actually changed it in revision 13 but for some reason I still get the wrong one:

echo $LD_LIBRARY_PATH
/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/folding-at-home-fcole90/13/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu/:/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/:/snap/folding-at-home-fcole90/13/lib:/snap/folding-at-home-fcole90/13/usr/lib:/snap/folding-at-home-fcole90/13/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu

If I change it to

export LD_LIBRARY_PATH=$SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/:/var/lib/snapd/hostfs/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/

It detects the GPU correctly and says this:

ERROR:No compute devices matched GPU #0 NVIDIA:4 GM107 [GeForce GTX 850M].  You may need to update your graphics drivers.

However it seems I can actually use the GPU after setting this from within the snap shell. I just need to figure out why the environment is not being set

fcole90 · March 27, 2020, 3:20pm

Is there any way to have hardware-observe be autoconnected?

lucyllewy · March 27, 2020, 3:21pm

Yes, post a new topic in the store-requests category with your justification and the reviewers will vote on whether to grant the request or not. They might ask further questions, too.

see Process for aliases, auto-connections and tracks for details

fcole90 · March 27, 2020, 3:22pm

Thank you very much I’ll do it as soon as the GPU starts working correctly

fcole90 · March 27, 2020, 10:09pm

I got it working sometimes by running FAHClient from within the snap shell, but I couldn’t yet understand what is making the difference. It doesn’t seem consistent…

I also found these messages in the audit logs:

audit: type=1326 audit(1585344705.550:55918): auid=1000 uid=1000 gid=1000 ses=3 pid=120272 comm="FAHClient" exe="/snap/folding-at-home-fcole90/16/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7f335f2fa86d code=0x50000

audit: type=1400 audit(1585344705.550:55917): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.FAHClient" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=120272 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=120272 comm="FAHClient" exe="/snap/folding-at-home-fcole90/16/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7f335f2fa86d code=0x50000

AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.FAHClient" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=120272 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

fcole90 · March 30, 2020, 11:48am

I have a strange issue. I added a duplicate command to the snap, with the only difference that one (folding-at-home-fcole90) is run as a command, while the other (FAHClient) is run as a service daemon: simple. The command is the same for both and has an argument --chdir=$SNAP_USER_COMMON.

What happens is that starting snap start folding-at-home-fcole90.FAHClient uses a different configuration file than directly calling folding-at-home-fcole90.

I also found an init file in $SNAP/etc/init.d called FAHClient, which is pulled from the deb and has some hardcoded paths and values.

So, my concern is that snap may be calling the wrong command. How does snap run actually work? I didn’t find any documentation about that

The good news If I have hardware-observe enabled, the GPU is finally detected. However, I still get the OpenCL error Not detected: clGetPlatformIDs() returned -1001 and the GPU slot says Paused: waiting for idle.

Starting the client (from command line, not as service) produces the following:

AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=344930 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
audit: type=1400 audit(1585569750.645:61389): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=344930 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.678:61391): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.698:61392): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.702:61395): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.706:61397): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000

I have no clue on how to debug that however

fcole90 · March 30, 2020, 12:32pm

I tried changing name to the client, so to be called just client but I get still the same issue. I think there is something in the client which is recognising when it’s launched as a service and when as a command. That’s so weird…

I double checked and the service is being started with the command I wrote, so I have no clue on how to avoid this…

ogra · March 30, 2020, 1:03pm

$ snappy-debug.scmp-sys-resolver 133
mknod

digging through the interfaces in snapd:

github.com

snapcore/snapd/blob/master/interfaces/builtin/browser_support.go#L103


# webbrowser-app/webapp-container tries to read this file to determine if it is
# confined or not, so explicitly deny to avoid noise in the logs.
deny @{PROC}/@{pid}/attr/{,apparmor/}current r,
# This is an information leak but disallowing it leads to developer confusion
# when using the chromium content api file chooser due to a (harmless) glib
# warning and the noisy AppArmor denial.
owner @{PROC}/@{pid}/mounts r,
owner @{PROC}/@{pid}/mountinfo r,
# Since snapd still uses SECCOMP_RET_KILL, we have added a workaround rule to
# allow mknod on character devices since chromium unconditionally performs
# a mknod() to create the /dev/nvidiactl device, regardless of if it exists or
# not or if the process has CAP_MKNOD or not. Since we don't want to actually
# grant the ability to create character devices, explicitly deny the
# capability. When snapd uses SECCOMP_RET_ERRNO, we can remove this rule.
# https://forum.snapcraft.io/t/call-for-testing-chromium-62-0-3202-62/2569/46
deny capability mknod,
`
const browserSupportConnectedPlugAppArmorWithSandbox = `

might be that openCL wants to do something similar to the description above (creating and writing to /dev/nvidiactl) … try adding the browser-support interface for a test …

not weird at all … services always run as root … (with the respective permissions and the environment pointing to folders in /root) so it can indeed be possible that running as a user behaves different to running as service (as root).

fcole90 · March 30, 2020, 1:17pm

Thanks! Indeed the cause was that the directory I was passing was $SNAP_USER_COMMON, which was then using two different configuration, according to the user. I changed it to use $SNAP_COMMON

ogra · March 30, 2020, 1:20pm

note that normal users usually can not write to $SNAP_COMMON (it lives under /var/snap which is root writable only)

fcole90 · March 30, 2020, 1:23pm

I’m trying it now

fcole90 · March 30, 2020, 1:28pm

I know, that’s the reason why I had previously moved to the user version. However the client is meant to be run as a service, which then causes it to be run in a root owned folder

Even the deb version is configured with some hardcoded root-owned paths

Edit: These seem to be the default paths FAHClient expects: https://foldingathome.org/faqs/installation-guides/linux/file-locations/

ogra · March 30, 2020, 1:44pm

sounds like you want two layouts for these two dirs then …

fcole90 · March 30, 2020, 1:56pm

You mean perhaps using SNAP_COMMON for the service and SNAP_USER_COMMON for the command?

ogra · March 30, 2020, 2:11pm

well, if the app actually uses these paths hardcoded (without any way to override them) you need actual layouts to re-map these dirs to locations inside the snap environment …

fcole90 · March 30, 2020, 3:50pm

These are just defaults, which can be overridden using command line arguments (which is what they did)

fcole90 · April 3, 2020, 12:45pm

Hi, I was finally able to try @ogra 's suggestion and change a little the way the snap works.

It took me some time because I was experimenting using the lxd container, to try to reduce system resources usage, but it didn’t work for me, so I’m using Multipass again. Also, each test takes me a long time, to rebuild the snap (I usually make snapcraft clean mypart), I’m not sure there’s a faster procedure.

While I’m currently focusing on enablying the OpenGL support, next I will try to make the fahclient be a user service using an autostart desktop file. I’m still investigating how to do it properly. Also, I’d like to enable CUDA as well, but that’s for later too.

OpenGL I released revision 35 to all channels. Using the hardware-observe interface the GPU is detected, though there are still some errors. Adding the browser-support interface didn’t seem to help actually, but I’m not sure if I had to set something related to sandboxing and where to eventually do so.

Client errors Running the client, on first run, it produces the following error: 12:27:00:ERROR:No compute devices matched GPU #0 NVIDIA:4 GM107 [GeForce GTX 850M]. You may need to update your graphics drivers.

In the control app, the GPU slot appears but it is always yellow with the writing Paused: waiting for idle. For OpernGL it reports:

GPUs: 0
CUDA: Not detected: cuInit() returned 100
OpenCL: Not detected: clGetPlatformIDs() returned -1001

Logs:

15:34:05 kernel: audit: type=1400 audit(1585917245.233:67775): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=794779 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:34:05 FAHClient: AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=794779 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:43 kernel: audit: type=1400 audit(1585917223.173:67774): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=794779 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:43 FAHClient: AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=794779 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:09 kernel: audit: type=1400 audit(1585917189.721:67773): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=794687 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:09 FAHClient: AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=794687 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:09 kernel: audit: type=1400 audit(1585917189.709:67772): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=794687 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:09 FAHClient: AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=794687 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

I’m running the client as user, with the command folding-at-home-fcole90 which has the following plugs:

plugs:
      - network
      - network-bind
      - opengl
      - x11
      - unity7
      - home
      - serial-port
      - raw-usb
      - desktop
      - desktop-legacy
      - hardware-observe
      - browser-support

Any suggestion on how to proceed?