Snapping Folding@Home, distributed computing for research on diseases

ogra · March 26, 2020, 9:07pm

hmm, during the prime step you are actually in the prime/ folder so that is probably why the variable is not set … just make it a relative path:

ln -s usr/lib/$SNAPCRAFT_ARCH_TRIPLET/libOpenCL.so.1 usr/lib/$SNAPCRAFT_ARCH_TRIPLET/libOpenCL.so

fcole90 · March 27, 2020, 9:15am

I’m sorry, it’s still not working. I went into the snap shell and I run FAHClient --lspci, which returns

09:04:44:ERROR:Exception: Failed to open '/sys/class/power_supply/AC0/online': Failed to open '/sys/class/power_supply/AC0/online': Permission denied: iostream error: Permission denied
09:04:44:WARNING:Exception: Failed to open '/proc/bus/pci/devices': Failed to open '/proc/bus/pci/devices': Permission denied: iostream error: Permission denied
VendorID:DeviceID:PCI Bus:PCI Slot:PCI function:Vendor Name:Description

I’m now thinking that it may not even be looking for the libOpenCL object, but rather for the GPU first, which it can’t detect

Edit: I had to manually connect hardware-observe. Now the I get the following:

VendorID:DeviceID:PCI Bus:PCI Slot:PCI function:Vendor Name:Description
0x8086:0x0c04:0:0:0:Intel Corporation:
0x8086:0x0c01:0:1:0:Intel Corporation:
0x8086:0x0416:0:2:0:Intel Corporation:
0x8086:0x0c0c:0:3:0:Intel Corporation:
0x8086:0x8c31:0:20:0:Intel Corporation:
0x8086:0x8c3a:0:22:0:Intel Corporation:
0x8086:0x8c20:0:27:0:Intel Corporation:
0x8086:0x8c10:0:28:0:Intel Corporation:
0x8086:0x8c14:0:28:2:Intel Corporation:
0x8086:0x8c16:0:28:3:Intel Corporation:
0x8086:0x8c49:0:31:0:Intel Corporation:
0x8086:0x8c03:0:31:2:Intel Corporation:
0x8086:0x8c22:0:31:3:Intel Corporation:
0x10de:0x1391:1:0:0:NVIDIA Corporation:
0x168c:0x0036:3:0:0:Qualcomm Atheros:
0x10ec:0x5287:4:0:0:Realtek Semiconductor Co., Ltd.:
0x10ec:0x8168:4:0:1:Realtek Semiconductor Co., Ltd.:

Still it complains of missing libOpenCL…

fcole90 · March 27, 2020, 10:30am

i’d try hardware-observe though i dont see /sys/class in there (it has /run/udev/data/+power_supply:* r,), this might be a bug`

When I tried before, I didn’t notice that I had to manually connect the plug. It actually worked! Thanks!

fcole90 · March 27, 2020, 11:42am

This is weird

$ snap run --shell folding-at-home-fcole90.FAHClient
$ ls $SNAP/usr/lib/x86_64-linux-gnu/ | grep libOpenCL
libOpenCL.so
libOpenCL.so.1
libOpenCL.so.1.0.0
$ cat $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so
cat: /snap/folding-at-home-fcole90/11/usr/lib/x86_64-linux-gnu/libOpenCL.so: No such file or directory
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so; then echo "Exists"; else echo "Not found"; fi
Not found
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so.1; then echo "Exists"; else echo "Not found"; fi
Exists
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0; then echo "Exists"; else echo "Not found"; fi
Exists

The file is listed there but it can’t read it?

Edit: it actually can’t read the symlinked file:

ls -la  $SNAP/usr/lib/x86_64-linux-gnu/ | grep libOpenCL
lrwxrwxrwx  1 root root       39 Mar 27 11:57 libOpenCL.so -> usr/lib/x86_64-linux-gnu/libOpenCL.so.1
lrwxrwxrwx  1 root root       18 Apr  5  2017 libOpenCL.so.1 -> libOpenCL.so.1.0.0
-rw-r--r--  1 root root    43072 Apr  5  2017 libOpenCL.so.1.0.0

fcole90 · March 27, 2020, 12:28pm

Using ln -rs ... did the trick

Now I have a new error on OpenCL:

OpenCL: Not detected: clGetPlatformIDs() returned -1001

clGetPlatformIDs()

Edit: I think I had messed up with the library path

ogra · March 27, 2020, 2:48pm

any denials in the journal when this message shows up ?

fcole90 · March 27, 2020, 3:03pm

I discovered that this last issue was actually caused by the library path being wrong. I actually changed it in revision 13 but for some reason I still get the wrong one:

echo $LD_LIBRARY_PATH
/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/folding-at-home-fcole90/13/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu/:/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/:/snap/folding-at-home-fcole90/13/lib:/snap/folding-at-home-fcole90/13/usr/lib:/snap/folding-at-home-fcole90/13/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu

If I change it to

export LD_LIBRARY_PATH=$SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/:/var/lib/snapd/hostfs/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/

It detects the GPU correctly and says this:

ERROR:No compute devices matched GPU #0 NVIDIA:4 GM107 [GeForce GTX 850M].  You may need to update your graphics drivers.

However it seems I can actually use the GPU after setting this from within the snap shell. I just need to figure out why the environment is not being set

fcole90 · March 27, 2020, 3:20pm

Is there any way to have hardware-observe be autoconnected?

lucyllewy · March 27, 2020, 3:21pm

Yes, post a new topic in the store-requests category with your justification and the reviewers will vote on whether to grant the request or not. They might ask further questions, too.

see Process for aliases, auto-connections and tracks for details

fcole90 · March 27, 2020, 3:22pm

Thank you very much I’ll do it as soon as the GPU starts working correctly

fcole90 · March 27, 2020, 10:09pm

I got it working sometimes by running FAHClient from within the snap shell, but I couldn’t yet understand what is making the difference. It doesn’t seem consistent…

I also found these messages in the audit logs:

audit: type=1326 audit(1585344705.550:55918): auid=1000 uid=1000 gid=1000 ses=3 pid=120272 comm="FAHClient" exe="/snap/folding-at-home-fcole90/16/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7f335f2fa86d code=0x50000

audit: type=1400 audit(1585344705.550:55917): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.FAHClient" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=120272 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=120272 comm="FAHClient" exe="/snap/folding-at-home-fcole90/16/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7f335f2fa86d code=0x50000

AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.FAHClient" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=120272 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

fcole90 · March 30, 2020, 11:48am

I have a strange issue. I added a duplicate command to the snap, with the only difference that one (folding-at-home-fcole90) is run as a command, while the other (FAHClient) is run as a service daemon: simple. The command is the same for both and has an argument --chdir=$SNAP_USER_COMMON.

What happens is that starting snap start folding-at-home-fcole90.FAHClient uses a different configuration file than directly calling folding-at-home-fcole90.

I also found an init file in $SNAP/etc/init.d called FAHClient, which is pulled from the deb and has some hardcoded paths and values.

So, my concern is that snap may be calling the wrong command. How does snap run actually work? I didn’t find any documentation about that

The good news If I have hardware-observe enabled, the GPU is finally detected. However, I still get the OpenCL error Not detected: clGetPlatformIDs() returned -1001 and the GPU slot says Paused: waiting for idle.

Starting the client (from command line, not as service) produces the following:

AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=344930 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
audit: type=1400 audit(1585569750.645:61389): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=344930 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.678:61391): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.698:61392): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.702:61395): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.706:61397): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000

I have no clue on how to debug that however

fcole90 · March 30, 2020, 12:32pm

I tried changing name to the client, so to be called just client but I get still the same issue. I think there is something in the client which is recognising when it’s launched as a service and when as a command. That’s so weird…

I double checked and the service is being started with the command I wrote, so I have no clue on how to avoid this…

ogra · March 30, 2020, 1:03pm

$ snappy-debug.scmp-sys-resolver 133
mknod

digging through the interfaces in snapd:

github.com

snapcore/snapd/blob/master/interfaces/builtin/browser_support.go#L103


# webbrowser-app/webapp-container tries to read this file to determine if it is
# confined or not, so explicitly deny to avoid noise in the logs.
deny @{PROC}/@{pid}/attr/{,apparmor/}current r,
# This is an information leak but disallowing it leads to developer confusion
# when using the chromium content api file chooser due to a (harmless) glib
# warning and the noisy AppArmor denial.
owner @{PROC}/@{pid}/mounts r,
owner @{PROC}/@{pid}/mountinfo r,
# Since snapd still uses SECCOMP_RET_KILL, we have added a workaround rule to
# allow mknod on character devices since chromium unconditionally performs
# a mknod() to create the /dev/nvidiactl device, regardless of if it exists or
# not or if the process has CAP_MKNOD or not. Since we don't want to actually
# grant the ability to create character devices, explicitly deny the
# capability. When snapd uses SECCOMP_RET_ERRNO, we can remove this rule.
# https://forum.snapcraft.io/t/call-for-testing-chromium-62-0-3202-62/2569/46
deny capability mknod,
`
const browserSupportConnectedPlugAppArmorWithSandbox = `

might be that openCL wants to do something similar to the description above (creating and writing to /dev/nvidiactl) … try adding the browser-support interface for a test …

not weird at all … services always run as root … (with the respective permissions and the environment pointing to folders in /root) so it can indeed be possible that running as a user behaves different to running as service (as root).

fcole90 · March 30, 2020, 1:17pm

Thanks! Indeed the cause was that the directory I was passing was $SNAP_USER_COMMON, which was then using two different configuration, according to the user. I changed it to use $SNAP_COMMON

ogra · March 30, 2020, 1:20pm

note that normal users usually can not write to $SNAP_COMMON (it lives under /var/snap which is root writable only)

fcole90 · March 30, 2020, 1:23pm

I’m trying it now

fcole90 · March 30, 2020, 1:28pm

I know, that’s the reason why I had previously moved to the user version. However the client is meant to be run as a service, which then causes it to be run in a root owned folder

Even the deb version is configured with some hardcoded root-owned paths

Edit: These seem to be the default paths FAHClient expects: https://foldingathome.org/faqs/installation-guides/linux/file-locations/

ogra · March 30, 2020, 1:44pm

sounds like you want two layouts for these two dirs then …

fcole90 · March 30, 2020, 1:56pm

You mean perhaps using SNAP_COMMON for the service and SNAP_USER_COMMON for the command?