Several libraries from stage-packages missing despite proper LD_LIBRARY_PATH

prex · March 16, 2020, 2:54pm

I am missing several libraries in my snap. It builds just fine, but when I run it, it outputs the following:

error while loading shared libraries: liblapack.so.3: cannot open shared object file: No such file or directory

I have both liblapack-dev and liblapack3 in my stage-packages. I checked the folder /usr/lib/x86_64-linux-gnu/ in my VM, which includes liblapack.a, liblapack.so as well as liblapack.so.3. I assume that $SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET` is added automatically to the LD_LIBRARY_PATH, right? Therefore, I’m wondering why these can’t be found. The error message unfortunately doesn’t tell me where it is looking for this library. I’m using the catkin-tools plugin and ROS, if this is relevant.

The second suspicious warnings arise after the build:

This part is missing libraries that cannot be satisfied with any available stage-packages known to snapcraft:
- usr/lib/x86_64-linux-gnu/libpsm_infinipath.so.1
These dependencies can be satisfied via additional parts or content sharing. Consider validating configured filesets if this dependency was built.

I checked the folder /usr/lib/x86_64-linux-gnu/ as well and it contains libpsm_infinipath.so.1. It obviously works for all other stage-packages except these 2.

ogra · March 16, 2020, 3:05pm

do you have libpsm-infinipath1 in your stage-packages ?

prex · March 16, 2020, 3:27pm

Yes, both libpsm-infinipath1 and libpsm-infinipath1-dev.

ogra · March 16, 2020, 3:44pm

also in the correct part (if you have multiple) ?

prex · March 16, 2020, 9:16pm

Not sure anymore, if I use them correctly. But this worked so far. I defined different parts with all the build-packages and stage-packages and then added them with after: [part1, part2] in my workspace part.

I’m mostly concerned about liblapack, since I can’t even run my application without this library.

My snapcraft.yaml looks like this (relevant parts relatively at the end):

name: test
base: core18 
version: 'w1.0' 
summary:
description: |

grade: devel
confinement: strict 

apps:
  daemon:
    command: run-daemon wayland-launch roslaunch nav_launch nav_gui.launch
    daemon: simple
    restart-condition: always

plugs:
  opengl:
  wayland:
  network:
  network-bind:
  joystick:

layout:
  /etc/glvnd:
    bind: $SNAP/etc/glvnd
  /etc/fonts:
    bind: $SNAP/etc/fonts
  /etc/xdg:
    bind: $SNAP/etc/xdg
  /usr/share/X11/xkb:
    bind: $SNAP/usr/share/X11/xkb
  /usr/share/fonts:
    bind: $SNAP/usr/share/fonts
  /usr/share/glvnd:
    bind: $SNAP/usr/share/glvnd
  /usr/lib/${SNAPCRAFT_ARCH_TRIPLET}/qt5:
    bind: $SNAP/usr/lib/${SNAPCRAFT_ARCH_TRIPLET}/qt5
  /usr/lib/${SNAPCRAFT_ARCH_TRIPLET}/dri:
    bind: $SNAP/usr/lib/${SNAPCRAFT_ARCH_TRIPLET}/dri

environment:
  # Qt Platform to Wayland
  QT_QPA_PLATFORM: wayland
  QTCHOOSER_NO_GLOBAL_DIR: 1
  QT_SELECT: snappy-qt5
  
parts: 

  qtwayland:
    plugin: nil
    stage-packages:
      - qtwayland5

  qt5:
    plugin: nil
    build-packages:
      - build-essential
      - qtbase5-dev
      ...
      - libpsm-infinipath1
      - libpsm-infinipath1-dev
      - dpkg-dev
    stage-packages:
      - libqt5qml5
      ...
      - libgdk-pixbuf2.0-0
      - libpsm-infinipath1
      - libpsm-infinipath1-dev
      - libgtk2.0-0
      - locales-all
      - xdg-user-dirs
      - fcitx-frontend-qt5
 
  core-dep:
    plugin: nil
    build-packages:
      - libyaml-cpp-dev
      - swig
      - liblapack-dev
      - liblapack3
      - autoconf
      - libtool
      - libgtest-dev
    stage-packages:
      - liblapack-dev
      - liblapack3
      - freeglut3
      - freeglut3-dev
      - libslang2
      - libglu1-mesa
      - libglu1-mesa-dev

  qtchooser-config:
    plugin: dump
    source: snap/local/qtconfig
    organize:
      snappy-qt5.conf: etc/xdg/qtchooser/snappy-qt5.conf

  mir-kiosk-snap-launch:
    plugin: dump
    source: https://github.com/MirServer/mir-kiosk-snap-launch.git
    override-build: $SNAPCRAFT_PART_BUILD/build-with-plugs.sh opengl wayland network network-bind

  workspace:
    plugin: catkin-tools
    source: .
    catkin-packages: [catkin_simple, ads_catkin, eigen_catkin, mag_msgs, nav_controller,...]
    after: [qt5, core-dep]

prex · March 19, 2020, 10:38am

Any more hints or ideas? I’m really clueless on how to proceed here with these very sparse error messages.

cjp256 · March 19, 2020, 1:17pm

Looks like libpsm-infinipath1 ships these libs:

/usr/lib/libpsm1/libpsm_infinipath.so.1.16
/usr/lib/x86_64-linux-gnu/libinfinipath.so.4.0
/usr/lib/x86_64-linux-gnu/libinfinipath.so.4

It appears that it relies on a postinst action to link (using update-alternatives)
/usr/lib/x86_64-linux-gnu/libpsm_infinipath.so.1 to /usr/lib/libpsm1/libpsm_infinipath.so.1.16

Snapcraft does not run postinst actions for staged packages unfortunately, so this may require some manual fixups.

You have a couple options, but I can’t really test it without a good example:

symlink /usr/lib/libpsm1/libpsm_infinipath.so.1 yourself (using relative pathing)
add libpsm1 to the library path, but you may still need that .so.1 link

From what I’m told, liblapack also needs to be added to the library path to function as well.

So this may work for you for the runtime issue:
LIBRARY_PATH: $LD_LIBRARY_PATH:$SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/lapack:$SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/blas:$SNAP/usr/lib/libpsm1

The symlink for the major api versions may be required as well (libpsm_infinipath.so.1 -> libpsm_infinipath.so.1.16 and liblapack.so.3 -> liblapack.so.3.7.1).
You could do that with an override-prime, override-build, or override-stage.

prex · March 24, 2020, 12:19pm

Sorry, took me a while to test this with moving everything to home office ;-).

I added

environment:
  "LD_LIBRARY_PATH": "$LD_LIBRARY_PATH:$SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/lapack:$SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/blas:$SNAP/usr/lib/libpsm1"

to the apps. I didn’t realize that there’s also a lapack folder. The runtime error with the missing library is gone. But I still don’t understand why I need to add these folders although the same libraries/ symlinks are already in the ARCH_TRIPLET folder.

I assume I need to add the library path for every app now, right?

I still have the error with libpsm:

I have these files and symlinks:

/usr/lib/libpsm1/libpsm_infinipath.so.1.16
/usr/lib/x86_64-linux-gnu/libinfinipath.so
/usr/lib/x86_64-linux-gnu/libinfinipath.so.4
/usr/lib/x86_64-linux-gnu/libinfinipath.so.4.0
/usr/lib/x86_64-linux-gnu/libpsm_infinipath.so
/usr/lib/x86_64-linux-gnu/libpsm_infinipath.so.1

There’s already a libpsm_infinipath.so.1. Checking the symlink:

./libpsm_infinipath.so.1 -> /etc/alternatives/libpsm_infinipath.so.1

So should I replace this one and link to /usr/lib/libpsm1/libpsm_infinipath.so.1.16?

The first one is what we’re talking about anyway? For liblapack I found these symlinks in x86_64-linux-gnu:

./liblapack.so -> /etc/alternatives/liblapack.so-x86_64-linux-gnu
./liblapack.so.3 -> /etc/alternatives/liblapack.so.3-x86_64-linux-gnu

Is there a way to access the VM even after the build returned successfully?

sergiusens · March 24, 2020, 12:32pm

Just run

snapcraft --shell-after

cjp256 · March 24, 2020, 12:58pm

Yes, you will want to fixup the symlinks that point to /etc/alternatives, they won’t exist in the snap.

prex · March 24, 2020, 6:36pm

I tried the symlink and bind-file approach, but the error remains. This is what I tried:

  /usr/lib/${SNAPCRAFT_ARCH_TRIPLET}/libpsm_infinipath.so.1:
    symlink: $SNAP/usr/lib/libpsm1/libpsm_infinipath.so.1.16

When I check the symlink of /usr/lib/x86_64-linux-gnu/libpsm_infinipath.so.1 in the VM, it points to /usr/lib/libpsm1/libpsm_infinipath.so.1.16, which is obviously where the file is. I don’t even understand the error message, since the libpsm_infinipath.so.1 clearly exists.

prex · March 25, 2020, 7:36pm

I’m reading and reading, but the information is very sparse. I’m trying to make some assumptions to find out how this really works and finally solve this:

The stage-packages for my particular snap are put in the folder stage in my VM and the build-packages are just in their normal places in the VM (?).
How does snapcraft know missing stage-packages if they are only used during runtime (otherwise the snap would probably not succeed). I got an error before from a missing library without snapcraft giving me that hint during the build. Why is this different with libpsm?
Where are these stage-packages missing? In $SNAPCRAFT_STAGE or $SNAPCRAFT_PRIME? And how is $SNAP related to these?
What do I need to symlink? Files inside $SNAPCRAFT_STAGE, $SNAPCRAFT_PRIME or $SNAP (which I don’t know where to find)?
Why is the LD_LIBRARY_PATH necessary when I need to symlink anyway? The LD_LIBRARY_PATH obviously doesn’t save me from the error message during the build.

In my snapcraft.yaml above, I had both libpsm_infinipath1 and libpsm_infinipath1-dev also in the build packages. I removed them from the build-packages and have it only in the stage-packages now. The error I get now changed from usr/lib/x86_64-linux-gnu/libpsm_infinipath.so.1 to just libpsm_infinipath.so.1.

And I found this hint at the very bottom of this page:

Layouts cannot replace an existing but incompatible filesystem object. This means, for example, that files cannot replace directories or symbolic links, files cannot replace a directory, and existing symbolic links cannot be redirected to a new target.

cjp256 · March 26, 2020, 1:37pm

Yes.

Snapcraft will do its own dependency resolution. It’s not perfect, but that’s why they are only presented as warnings. The difference in behavior you are seeing is because snapcraft will resort to scanning the entire snap for libraries, assuming the snap author setup the correct LD_LIBRARY_PATH for runtime.

$SNAP is a runtime variable set in the process of the running snap. $SNAPCRAFT_STAGE,PRIME are build-time variables.

Maybe try something like this, using a relative symlink at build-time:

parts:
  <part>:
    override-build: |
      snapcraftctl build
      ln -sf ../libpsm1/libpsm_infinipath.so.1.16  $SNAPCRAFT_PART_INSTALL/usr/lib/x86_64-linux-gnu/libpsm_infinipath.so.1

If the source symlink is correct, and it resides in your library path, you won’t need to modify LD_LIBRARY_PATH.

For example, if you added $SNAP/usr/lib/libpsm1 to your LD_LIBRARY_PATH, the symlink example above would be:
ln -sf libpsm_infinipath.so.1.16 $SNAPCRAFT_PART_INSTALL/usr/lib/libpsm1/libpsm_infinipath.so.1

Hope that helps!

prex · March 26, 2020, 8:25pm

cjp256:

Maybe try something like this, using a relative symlink at build-time:
parts:
  <part>:
    override-build: |
      snapcraftctl build
      ln -sf ../libpsm1/libpsm_infinipath.so.1.16  $SNAPCRAFT_PART_INSTALL/

Thanks! This did the job.

It’s already pretty clear now. Still some questions for the learning effect:

Snapcraft will search for the libraries given in the snapcraft.yaml or the libraries used in my code? So why couldn’t it find the libpsm then, but it could find lapack. Both of them are not at the “default” location.

So this means the warning is not even so important if I know that I have set the correct LD_LIBRARY_PATH for runtime?

I don’t have the feeling for this structure yet. Are the directories just copied from one to another and they keep the paths? How can you otherwise know where to link to in $SNAP if you can’t look it up in the VM?

Makes me wonder why the layout-approach didn’t work. Is it only for runtime?

cjp256 · March 30, 2020, 4:41pm

Because it is looking for libpsm_infinipath.so.1, not libpsm_infinipath.so.1.16. Once that symlink is there, I imagine the warning should be going away?

For symlinking purposes at build time, you cannot rely on $SNAP, and must use a relative links, which should work wherever they end up (assuming it’s correct).

Yes

prex · March 30, 2020, 5:19pm

Great, thanks. Will mark this as solved.