Confusion about bin and usr/bin in snap filesytem

On my ArchLinux, I have:

$ ls -la /
lrwxrwxrwx   1 root root     7 Jan 18 22:32 bin -> usr/bin
lrwxrwxrwx   1 root root     7 Jan 18 22:32 lib -> usr/lib
lrwxrwxrwx   1 root root     7 Jan 18 22:32 lib64 -> usr/lib
lrwxrwxrwx   1 root root     7 Jan 18 22:32 sbin -> usr/bin

(I removed some lines for brevity)

But in a snap filesystem, those symlinks might not be there:

$ ls -la /snap/core20/current/
lrwxrwxrwx  1 root root    7 Dec 10 01:35 bin -> usr/bin
lrwxrwxrwx  1 root root    7 Dec 10 01:35 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Dec 10 01:35 lib32 -> usr/lib32
lrwxrwxrwx  1 root root    9 Dec 10 01:35 lib64 -> usr/lib64
lrwxrwxrwx  1 root root   10 Dec 10 01:35 libx32 -> usr/libx32
lrwxrwxrwx  1 root root    8 Dec 10 01:35 sbin -> usr/sbin

$ ls -la /snap/core18/current/
drwxr-xr-x  2 root root 1678 Jan 28 12:49 bin
drwxr-xr-x 14 root root  276 Jan 28 12:49 lib
drwxr-xr-x  2 root root   43 Jan 28 12:49 lib64
drwxr-xr-x  2 root root 1877 Jan 28 12:49 sbin
$ ls -la /snap/core18/current/usr/
drwxr-xr-x  2 root root 4943 Jan 28 12:49 bin
drwxr-xr-x 33 root root  631 Jan 28 12:49 lib
drwxr-xr-x  2 root root   45 Jan 28 12:49 lib32
drwxr-xr-x  2 root root 1083 Jan 28 12:49 sbin

$ ls -la /snap/lxd/current/
drwxr-xr-x  2 root root 2127 Jan 29 21:10 bin
drwxr-xr-x  5 root root 1380 Jan 29 21:10 lib

And this is very confusing to me. What are the guidelines for setting $SNAP/bin, $SNAP/sbin, and $SNAP/lib as symlinks? How do I do that in my snap?

1 Like

You don’t symlink those folders, you add them to $PATH and $LD_LIBRARY_PATH environment variables respectively, whether through a wrapper script or using the environment: functionality in the snapcraft.yaml (My memory might be incorrect, but I’m under the impression this is default behaviour anyway? If it isn’t, consider using extensions or Snapcraft Desktop Helpers to help set up the environment)

Strict snaps make use of mount namespaces, which means the filesystem they see isn’t the same as the filesystem the host sees, this is why they don’t match up. You can influence the folders outside of $SNAP to some degree by using layouts, but by design, the host doesn’t have influence here.

Effectively, every strict snap is guaranteed to have the same root filesystem on every system they’re installed on.

Thanks for the answer, @James-Carroll!

My question is more from the point of view of Snap creation. What filesystem hierarchy should I expect? I’m building a snap with core20, but I have $SNAP/bin and $SNAP/usr/bin as different directories (no symlinks). I was expecting the same structure as core20, but it is not the case.

whatever your source (or your stage packages) defines … snapcraft will not explicitly create any symlinks on its own inside your snap, if you want/need them you need to do that yourself … this is all a bit esoterical though since both $SNAP/bin and $SNAP/usr/bin will both be automatically in your PATH anyway (unless you override PATH in an environment stanza) so it doesnt really matter where in these two paths your binaries end up …

could you give one example on how to do that?

I went to the core18 and core20 sources but did not find out why their filesystems differ. Or those symlinks come from Ubuntu?

the filesystems differ because the setup as a whole changed between the two releases when ubuntu moved to the /usr merge (i think arch did that earlier already) …

see the bottom of:
https://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/

in 18.04 (which core18 is based on) /bin, /lib and /sbin are actual dirs like in all releases before, from 20.04 on ubuntu did the switch to the new systemd induced setup …

you just use the ln command in your snapcraft.yaml from an ```override-build: |````scriptlet …

here is an ugly hack where i use some links to cheat around the fact that my libs are newer than the binary wants:

This is really weird to me.

I made a dummy snap based on core20 and installed a libcurl package, just to have something with lib/ in it:

$ cat snap/snapcraft.yaml
name: my-snap-name
base: core20
version: '0.1'
summary: Single-line elevator pitch for your amazing snap
description:   This is my-snap's description. 
grade: devel
confinement: devmode

parts:
  my-part:
    plugin: nil
    stage-packages:
      - libcurl4-gnutls-dev

The final snap filesystem is then:

$ ls -la /snap/my-snap-name/current/
drwxr-xr-x 7 root root   85 Mar  3 12:55 ./
drwxr-xr-x 3 root root 4.0K Mar  3 12:55 ../
drwxr-xr-x 5 root root   58 Mar  3 12:55 etc/
drwxr-xr-x 3 root root   39 Mar  3 12:55 lib/
drwxr-xr-x 2 root root   43 Mar  3 12:55 lib64/
drwxr-xr-x 2 root root   32 Mar  3 12:52 meta/
drwxr-xr-x 6 root root   77 Mar  3 12:55 usr/

$ ls -la /snap/my-snap-name/current/usr/
drwxr-xr-x 6 root root 77 Mar  3 12:55 ./
drwxr-xr-x 7 root root 85 Mar  3 12:55 ../
drwxr-xr-x 2 root root 34 Mar  3 12:55 bin/
drwxr-xr-x 3 root root 39 Mar  3 12:55 include/
drwxr-xr-x 4 root root 64 Mar  3 12:55 lib/
drwxr-xr-x 6 root root 67 Mar  3 12:55 share/

I was expecting lib/ to be a symlink to usr/lib, because it is in Ubuntu 20.04, but that is not the case here. I know those paths are in PATH and LD_LIBRARY_PATH, but when we need to compile a package to snap it, it becomes a burden to find if a library is in bin or usr/bin when we ./configure the source.

Is it safe to create links in / within override-build like:

parts:
  my-part:
    plugin: dump
    ovrride-build: |
      ln -s $SNAPCRAFT_PART_INSTALL/usr/lib/foo /usr/lib/foo
      snapcraftctl build

this links to a place outside of your build environment and essentially mangles the host … i wouldnt recommend doing that … i also dont really get the issue with the whole linking you seem to have … if you use configure the pkg-config files of the dependencies should properly resolve to the places the build-packages: were unpacked (including the $SNAP_STAGE bits that snapcraft might add to them for staged parts) … all these symlinks dont really matter, they are just a convenience so the old paths from before the systemd induced change still work but they have absolutely no meaning when building snaps

My bad, i tried to hijack this post to try to solve another problem I described in How to use custom libraries with the autoconf plugin? .

I see. So whatever I do in during the override-build that are outside the stage/prime area will be ignored.

Things are getting clear to me! Thanks for the help, @ogra and @James-Carroll.

1 Like