Librt.so.1: undefined symbol: __clock_nanosleep

Hi,

I am trying to build MicroK8s snap from the snapcraft.yaml [1]. The Kubernetes binaries are build through the build process. When deploying on 20.04 one of the daemons (kube-apiserver) is failing with: /snap/microk8s/x1/kube-apiserver: symbol lookup error: /snap/core18/current/lib/x86_64-linux-gnu/librt.so.1: undefined symbol: __clock_nanosleep, version GLIBC_PRIVATE

Below you can see the LD_LIBRARY_PATH used.

Mar 27 13:51:23 aurora microk8s.daemon-apiserver[145588]: + ARCH=x86_64
Mar 27 13:51:23 aurora microk8s.daemon-apiserver[145588]: + export LD_LIBRARY_PATH=:/snap/microk8s/x1/lib:/snap/microk8s/x1/usr/lib:/snap/microk8s/x1/lib/x86_64-linux-gnu:/snap/microk8s/x1/usr/lib/x86_64-linux-gnu
Mar 27 13:51:23 aurora microk8s.daemon-apiserver[145588]: + LD_LIBRARY_PATH=:/snap/microk8s/x1/lib:/snap/microk8s/x1/usr/lib:/snap/microk8s/x1/lib/x86_64-linux-gnu:/snap/microk8s/x1/usr/lib/x86_64-linux-gnu
Mar 27 13:51:23 aurora microk8s.daemon-apiserver[145588]: + export LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/x1/lib:/snap/microk8s/x1/usr/lib:/snap/microk8s/x1/lib/x86_64-linux-gnu:/snap/microk8s/x1/usr/lib/x86_64-linux-gnu
Mar 27 13:51:23 aurora microk8s.daemon-apiserver[145588]: + LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/x1/lib:/snap/microk8s/x1/usr/lib:/snap/microk8s/x1/lib/x86_64-linux-gnu:/snap/microk8s/x1/usr/lib/x86_64-linux-gnu
......
Mar 27 13:51:23 aurora microk8s.daemon-apiserver[145588]: + exec /snap/microk8s/x1/kube-apiserver --cert-dir=/var/snap/microk8s/x1/certs --service-cluster-ip-range=10.152.183.0/24 --authorization-mode=AlwaysAllow --basic-auth-file=/var/snap/microk8s/x1/credentials/basic_auth.csv --service-account-key-file=/var/snap/microk8s/x1/certs/serviceaccount.key --client-ca-file=/var/snap/microk8s/x1/certs/ca.crt --tls-cert-file=/var/snap/microk8s/x1/certs/server.crt --tls-private-key-file=/var/snap/microk8s/x1/certs/server.key --kubelet-client-certificate=/var/snap/microk8s/x1/certs/server.crt --kubelet-client-key=/var/snap/microk8s/x1/certs/server.key --secure-port=16443 --token-auth-file=/var/snap/microk8s/x1/credentials/known_tokens.csv --allow-privileged=true --insecure-port=8080 --storage-dir /var/snap/microk8s/x1/var/kubernetes/backend/ --storage-backend=dqlite --requestheader-client-ca-file=/var/snap/microk8s/x1/certs/front-proxy-ca.crt --requestheader-allowed-names=front-proxy-client --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/var/snap/microk8s/x1/certs/front-proxy-client.crt --proxy-client-key-file=/var/snap/microk8s/x1/certs/front-proxy-client.key
Mar 27 13:51:24 aurora microk8s.daemon-apiserver[145588]: /snap/microk8s/x1/kube-apiserver: symbol lookup error: /snap/core18/current/lib/x86_64-linux-gnu/librt.so.1: undefined symbol: __clock_nanosleep, version GLIBC_PRIVATE

How can I address this?

Thank you

[1] https://github.com/ubuntu/microk8s/blob/feature/ha-dqlite/snapcraft.yaml

Did anyone find out how to address this?

I ask because I’m having a similar problem with the ldc2 D compiler snap that I am responsible for. On Ubuntu 18.04 and 16.04 it works fine, but on 20.04 one of its features (link-time optimization) fails:

$ ldc2 -flto=full hello.d
/usr/bin/ld.gold: error: /snap/ldc2/158/lib/LLVMgold-ldc.so: could not load plugin library: /snap/core/current/lib/x86_64-linux-gnu/librt.so.1: undefined symbol: __clock_nanosleep, version GLIBC_PRIVATE
/usr/bin/ld.gold: error: hello.o:1:3: invalid character
/usr/bin/ld.gold: error: hello.o:1:3: syntax error, unexpected $end
/usr/bin/ld.gold: error: hello.o: not an object or archive
/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/Scrt1.o:function _start: error: undefined reference to 'main'
collect2: error: ld returned 1 exit status
Error: /usr/bin/cc failed with status: 1

The LLVMgold-ldc.so plugin that is used for link-time optimization links against librt.so.1:

$ ldd /snap/ldc2/current/lib/LLVMgold-ldc.so 
	linux-vdso.so.1 (0x00007ffd391d0000)
	libz.so.1 => /snap/core/current/lib/x86_64-linux-gnu/libz.so.1 (0x00007ff2ead96000)
	librt.so.1 => /snap/core/current/lib/x86_64-linux-gnu/librt.so.1 (0x00007ff2eab8e000)
	libdl.so.2 => /snap/core/current/lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff2ea98a000)
	libpthread.so.0 => /snap/core/current/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff2ea76d000)
	libm.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libm.so.6 (0x00007ff2ea464000)
	libgcc_s.so.1 => /snap/core/current/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff2ea24e000)
	libc.so.6 => /snap/core/current/lib/x86_64-linux-gnu/libc.so.6 (0x00007ff2e9e82000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff2edd9d000)

However, the ld.gold that is being invoked is on the host 20.04 system (this is intended/unavoidable). I presume that I’m therefore running into a clash between the libc6 of the host system and that of the core snap.

Does anyone have any thoughts/ideas on what can be needed to fix this? I see that the PowerShell snap also ran into this, and fixed it, but I’m not clear on how transferable their solution is to these other snaps.

Upstream bug report for ldc2 package here: https://github.com/ldc-developers/ldc2.snap/issues/121

As your snap is using classic confinement, this is probably due to Snapcraft patching an RPATH into your plugin so it preferentially chooses libraries under /snap/core/current/ over those found on the host system. This is usually something you want to make sure your app runs against the expected libraries rather than a mix of host and snap libraries.

If you explicitly want to run an executable that mixes host system and snap libraries together, this could cause a problem if you end up loading a library from a snap that shadows a newer version from the host: if you then try to load some host library that depends on features in the new version of the shadowed library you could get a link error like you’ve seen.

You can turn off Snapcraft’s patching by adding build-attributes: [no-patchelf] to the part that builds the library in question. With this option, it is up to you to set the RPATH if needed on your executables and libraries. I’ll note that you’ll almost always find some system where your classic snap malfunctions if you’re mixing host system and snap libraries together.

1 Like

Sorry to reawaken an old topic, but a question that came up recently thinking about this, which is specific (I think) to packaging compilers.

How might one ensure that all the binaries in a snap have the RPATH patched, but not the shared libraries? The line of thought here is that the compiler binaries should link only internally within the snap package, but the standard-library shared libs should link against the host system (because they will be used by the executables built on that system by the compiler).