Snapping Folding@Home, distributed computing for research on diseases

This is weird

$ snap run --shell folding-at-home-fcole90.FAHClient
$ ls $SNAP/usr/lib/x86_64-linux-gnu/ | grep libOpenCL
libOpenCL.so
libOpenCL.so.1
libOpenCL.so.1.0.0
$ cat $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so
cat: /snap/folding-at-home-fcole90/11/usr/lib/x86_64-linux-gnu/libOpenCL.so: No such file or directory
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so; then echo "Exists"; else echo "Not found"; fi
Not found
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so.1; then echo "Exists"; else echo "Not found"; fi
Exists
$ if test -f $SNAP/usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0; then echo "Exists"; else echo "Not found"; fi
Exists

The file is listed there but it can’t read it?:thinking:

Edit:
it actually can’t read the symlinked file:

ls -la  $SNAP/usr/lib/x86_64-linux-gnu/ | grep libOpenCL
lrwxrwxrwx  1 root root       39 Mar 27 11:57 libOpenCL.so -> usr/lib/x86_64-linux-gnu/libOpenCL.so.1
lrwxrwxrwx  1 root root       18 Apr  5  2017 libOpenCL.so.1 -> libOpenCL.so.1.0.0
-rw-r--r--  1 root root    43072 Apr  5  2017 libOpenCL.so.1.0.0

Using ln -rs ... did the trick :smiley:

Now I have a new error on OpenCL:

OpenCL: Not detected: clGetPlatformIDs() returned -1001

clGetPlatformIDs()

Edit: I think I had messed up with the library path

any denials in the journal when this message shows up ?

I discovered that this last issue was actually caused by the library path being wrong. I actually changed it in revision 13 but for some reason I still get the wrong one:

echo $LD_LIBRARY_PATH
/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/folding-at-home-fcole90/13/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu/:/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/:/snap/folding-at-home-fcole90/13/lib:/snap/folding-at-home-fcole90/13/usr/lib:/snap/folding-at-home-fcole90/13/lib/x86_64-linux-gnu:/snap/folding-at-home-fcole90/13/usr/lib/x86_64-linux-gnu

If I change it to

export LD_LIBRARY_PATH=$SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/:/var/lib/snapd/hostfs/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/

It detects the GPU correctly and says this:

ERROR:No compute devices matched GPU #0 NVIDIA:4 GM107 [GeForce GTX 850M].  You may need to update your graphics drivers.

However it seems I can actually use the GPU after setting this from within the snap shell. I just need to figure out why the environment is not being set :thinking:

Is there any way to have hardware-observe be autoconnected? :blush:

Yes, post a new topic in the store-requests category with your justification and the reviewers will vote on whether to grant the request or not. They might ask further questions, too.

see Process for aliases, auto-connections and tracks for details

1 Like

Thank you very much :blush: I’ll do it as soon as the GPU starts working correctly :smile:

I got it working sometimes by running FAHClient from within the snap shell, but I couldn’t yet understand what is making the difference. It doesn’t seem consistent…

I also found these messages in the audit logs:

audit: type=1326 audit(1585344705.550:55918): auid=1000 uid=1000 gid=1000 ses=3 pid=120272 comm="FAHClient" exe="/snap/folding-at-home-fcole90/16/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7f335f2fa86d code=0x50000

audit: type=1400 audit(1585344705.550:55917): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.FAHClient" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=120272 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=120272 comm="FAHClient" exe="/snap/folding-at-home-fcole90/16/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7f335f2fa86d code=0x50000

AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.FAHClient" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=120272 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

I have a strange issue. I added a duplicate command to the snap, with the only difference that one (folding-at-home-fcole90) is run as a command, while the other (FAHClient) is run as a service daemon: simple. The command is the same for both and has an argument --chdir=$SNAP_USER_COMMON.

What happens is that starting snap start folding-at-home-fcole90.FAHClient uses a different configuration file than directly calling folding-at-home-fcole90.

I also found an init file in $SNAP/etc/init.d called FAHClient, which is pulled from the deb and has some hardcoded paths and values.

So, my concern is that snap may be calling the wrong command. How does snap run actually work? I didn’t find any documentation about that :confused:

The good news
If I have hardware-observe enabled, the GPU is finally detected. However, I still get the OpenCL error Not detected: clGetPlatformIDs() returned -1001 and the GPU slot says Paused: waiting for idle.

Starting the client (from command line, not as service) produces the following:

AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=344930 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
audit: type=1400 audit(1585569750.645:61389): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=344930 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.678:61391): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.698:61392): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.702:61395): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
SECCOMP auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000
audit: type=1326 audit(1585569750.706:61397): auid=1000 uid=1000 gid=1000 ses=3 pid=344930 comm="FAHClient" exe="/snap/folding-at-home-fcole90/27/usr/bin/FAHClient" sig=0 arch=c000003e syscall=133 compat=0 ip=0x7fcae850286d code=0x50000

I have no clue on how to debug that however :confused:

I tried changing name to the client, so to be called just client but I get still the same issue. I think there is something in the client which is recognising when it’s launched as a service and when as a command. That’s so weird…

I double checked and the service is being started with the command I wrote, so I have no clue on how to avoid this…

$ snappy-debug.scmp-sys-resolver 133
mknod

digging through the interfaces in snapd:

might be that openCL wants to do something similar to the description above (creating and writing to /dev/nvidiactl) … try adding the browser-support interface for a test …

not weird at all … services always run as root … (with the respective permissions and the environment pointing to folders in /root) so it can indeed be possible that running as a user behaves different to running as service (as root).

Thanks! Indeed the cause was that the directory I was passing was $SNAP_USER_COMMON, which was then using two different configuration, according to the user. I changed it to use $SNAP_COMMON

note that normal users usually can not write to $SNAP_COMMON (it lives under /var/snap which is root writable only)

I’m trying it now :blush:

I know, that’s the reason why I had previously moved to the user version. However the client is meant to be run as a service, which then causes it to be run in a root owned folder :thinking:

Even the deb version is configured with some hardcoded root-owned paths :thinking:

Edit:
These seem to be the default paths FAHClient expects: https://foldingathome.org/faqs/installation-guides/linux/file-locations/

sounds like you want two layouts for these two dirs then …

1 Like

You mean perhaps using SNAP_COMMON for the service and SNAP_USER_COMMON for the command? :blush:

well, if the app actually uses these paths hardcoded (without any way to override them) you need actual layouts to re-map these dirs to locations inside the snap environment …

These are just defaults, which can be overridden using command line arguments (which is what they did)

1 Like

Hi, I was finally able to try @ogra 's suggestion and change a little the way the snap works.

It took me some time because I was experimenting using the lxd container, to try to reduce system resources usage, but it didn’t work for me, so I’m using Multipass again. Also, each test takes me a long time, to rebuild the snap (I usually make snapcraft clean mypart), I’m not sure there’s a faster procedure.

While I’m currently focusing on enablying the OpenGL support, next I will try to make the fahclient be a user service using an autostart desktop file. I’m still investigating how to do it properly. Also, I’d like to enable CUDA as well, but that’s for later too.

OpenGL
I released revision 35 to all channels. Using the hardware-observe interface the GPU is detected, though there are still some errors. Adding the browser-support interface didn’t seem to help actually, but I’m not sure if I had to set something related to sandboxing and where to eventually do so.

Client errors
Running the client, on first run, it produces the following error:
12:27:00:ERROR:No compute devices matched GPU #0 NVIDIA:4 GM107 [GeForce GTX 850M]. You may need to update your graphics drivers.

In the control app, the GPU slot appears but it is always yellow with the writing Paused: waiting for idle. For OpernGL it reports:

  • GPUs: 0
  • CUDA: Not detected: cuInit() returned 100
  • OpenCL: Not detected: clGetPlatformIDs() returned -1001

Logs:

15:34:05 kernel: audit: type=1400 audit(1585917245.233:67775): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=794779 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:34:05 FAHClient: AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=794779 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:43 kernel: audit: type=1400 audit(1585917223.173:67774): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=794779 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:43 FAHClient: AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=794779 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:09 kernel: audit: type=1400 audit(1585917189.721:67773): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=794687 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:09 FAHClient: AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libX11.so.6.3.0" pid=794687 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:09 kernel: audit: type=1400 audit(1585917189.709:67772): apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=794687 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
15:33:09 FAHClient: AVC apparmor="DENIED" operation="open" profile="snap.folding-at-home-fcole90.folding-at-home-fcole90" name="/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28" pid=794687 comm="FAHClient" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

I’m running the client as user, with the command folding-at-home-fcole90 which has the following plugs:

plugs:
      - network
      - network-bind
      - opengl
      - x11
      - unity7
      - home
      - serial-port
      - raw-usb
      - desktop
      - desktop-legacy
      - hardware-observe
      - browser-support

Any suggestion on how to proceed? :pray: