Snap cross-compilation, expectations and potential issues?

[alissy]

Hello,

I’m trying to get the firefox snap (core22) cross-compiled from amd64 to aarch64 (and hopefully also armhf).

My first question would be: where can I find docs about the current status of core22 cross-compilation, outside of https://github.com/snapcore/snapcraft/commit/c8e9b2e423cb0702d240814b21224843541c9d2b ? Something like what is working, what would be known as not (yet) working, what is not supported at all …

First, using mozilla-provided sysroot and adjusting paths correct, I could get an arm64 build to finish and run.

on amd64 to arm64 build packages includes only libstdc++-11-dev-arm64-cross

On my road to be able to cross-compile, there were a few bumps, and some might be issues to fix, and some might not:

• is it expected that in this setup, only the x86-64 snap for GNOME SDK is being installed ? For the moment, I’m hacking by manually downloading and extracting it
• as a side effect, the snap contains pkg-config definitions, but they hardcode paths like /snap/gnome-42-2204-sdk/current/ which is problematic because we still need /snap/gnome-42-2204-sdk/current/ to run amd64 code for the build itself (pkg-config, gmake, …). I understand that the revision is a snap-store-level thing, so it might not be known at build time, but we know it at install time (for now, I am relying on a not super nice sed to update those paths)
• when I dont use --sysroot= during our build process, I end up on lld failing to locate /lib/aarch64-linux-gnu/libc.so.6 and I dont see any libc.so.6 provided by the GNOME SDK snap, I’m wondering if that is on purpose or a bug to only provide the ld script usr/lib/aarch64-linux-gnu/libc.so and no libc.so ; I believe this could be an actual issue to fix
• when I do use --sysroot=/snap/gnome-42-2204-sdk/xxx then some libraries are missing, so far I’m blocked on alsa.pc not being part of the snap (while libasound2 is) ; I believe this could be an actual issue to fix
• Once I hacked by copying host’s alsa.pc the above limitation, I am hitting another one because we dont have libgcc_s.so.1 in the snap.

Here is the relevant error:

#  /root/stage/usr/bin/clang --sysroot /snap/gnome-42-2204-sdk/195 -std=gnu99 --target=aarch64-linux-gnu -o conftest -isystem /root/parts/firefox/install/usr/include -isystem /root/stage/usr/include -isystem /root/parts/firefox/install/usr/include -isystem /root/stage/usr/include -Wl,-rpath-link=/root/parts/firefox/build/obj-aarch64-unknown-linux-gnu/dist/bin -Wl,-rpath-link=/snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu -Wl,-rpath-link=/snap/gnome-42-2204-sdk/195/usr/lib -Wl,-rpath-link=/usr/aarch64-linux-gnu/lib/ -fuse-ld=lld -Wl,-rpath-link,/snap/gnome-42-2204-sdk/195/lib/aarch64-linux-gnu -Wl,-rpath-link,/snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu -Wl,-rpath-link,/usr/aarch64-linux-gnu/lib conftest.c
ld.lld: error: /snap/gnome-42-2204-sdk/195/usr/lib/gcc/aarch64-linux-gnu/11/libgcc_s.so:4: unable to find libgcc_s.so.1
>>> GROUP ( libgcc_s.so.1 -lgcc )
>>>         ^
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
snapcraft-firefox-on-amd64-for-arm64-30556937 ../project# cat /snap/gnome-42-2204-sdk/195/usr/lib/gcc/aarch64-linux-gnu/11/libgcc_s.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library.  */
GROUP ( libgcc_s.so.1 -lgcc )

Now if I use the cross-arch libstdc++-11-dev-arm64-cross and remove the --sysroot= it is picked up from the host:

snapcraft-firefox-on-amd64-for-arm64-30556937 ../project#  /root/stage/usr/bin/clang -std=gnu99 --target=aarch64-linux-gnu -o conftest -isystem /root/parts/firefox/install/usr/include -isystem /root/stage/usr/include -isystem /root/parts/firefox/install/usr/include -isystem /root/stage/usr/include -Wl,-rpath-link=/root/parts/firefox/build/obj-aarch64-unknown-linux-gnu/dist/bin -Wl,-rpath-link=/snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu -Wl,-rpath-link=/snap/gnome-42-2204-sdk/195/usr/lib -Wl,-rpath-link=/usr/aarch64-linux-gnu/lib/ -fuse-ld=lld -Wl,-rpath-link,/snap/gnome-42-2204-sdk/195/lib/aarch64-linux-gnu -Wl,-rpath-link,/snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu -Wl,-rpath-link,/usr/aarch64-linux-gnu/lib conftest.c
snapcraft-firefox-on-amd64-for-arm64-30556937 ../project#

There might be other missing libs/pkg-config definitions, but for the moment being I am blocked on the alsa.pc one.

[alissy]

Update:

I have managed to make --sysroot= a bit more, via:

        # HACKS
        # Fix for using as --sysroot=
        sed -ri \
          "s|/snap/${SDK_SNAP_NAME}/current||g" \
          /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/lib/*/pkgconfig/*.pc \
          /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/lib/pkgconfig/*.pc \
          /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/share/pkgconfig/*.pc \
          /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/bin/itstool \
          /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/bin/libtoolize
        # Fix for missing libasound2-dev as staged package
        cp -a /usr/include/alsa /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/include/
        cp /usr/include/asoundlib.h /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/include/
        mkdir -p /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/include/sys/
        cp /usr/include/sys/asoundlib.h /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/include/sys/
        cp /usr/lib/x86_64-linux-gnu/pkgconfig/alsa.pc /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/lib/*/pkgconfig/
        if [ ! -L /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/lib/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/libasound.so ]; then
          ln -s /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/lib/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/libasound.so.2.0.0 /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/lib/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/libasound.so
        fi
        # Fix for missing libc6 and libc6-dev as staged package
        mkdir -p /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/
        cp /usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/libc.so* /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/
        cp /usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/libc_nonshared.a /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/
        if [ ! -L /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/lib/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/libc.so.6 ]; then
          ln -s /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/libc.so.6 /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/lib/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/libc.so.6
        fi
        cp /usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/ld-linux-*.so.1 /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/
        cp /usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/ld-linux-*.so.1 /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/lib/
        # Fix for missing libgcc-s1 as stage package
        cp /usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/libgcc_s.so.1 /snap/${SDK_SNAP_NAME}/${SDK_SNAP_REVISION}/usr/${CRAFT_ARCH_TRIPLET_BUILD_FOR}/lib/
        # END OF HACKS

From what I can see, it can all be solved by staging a few more packages on gnome-sdk, including libc6, libc6-dev, libasound2-dev, libgcc-s1

I’m now facing a different issue: libm.a references a few symbols but does not define them, but libc.a provides them:

47331 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.277 ::  9:23.19 ld.lld: error: undefined symbol: __frexpf
47332 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.277 ::  9:23.19 >>> referenced by s_cbrtf.o:(__cbrtf) in archive /snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu/libm.a
47333 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 ld.lld: error: undefined symbol: __scalbnf
47334 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 >>> referenced by s_csqrtf.o:(__csqrtf) in archive /snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu/libm.a
47335 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 >>> referenced by s_csqrtf.o:(__csqrtf) in archive /snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu/libm.a
47336 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 >>> referenced by s_csqrtf.o:(__csqrtf) in archive /snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu/libm.a
47337 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 >>> referenced 7 more times
47338 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 ld.lld: error: undefined symbol: __ldexpf
47339 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 >>> referenced by s_cbrtf.o:(__cbrtf) in archive /snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu/libm.a
47340 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 ld.lld: error: undefined symbol: __frexp
47341 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 >>> referenced by s_cbrt.o:(__cbrt) in archive /snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu/libm.a
47342 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 ld.lld: error: undefined symbol: __ldexp
47343 2023-09-22 12:24:22.717 :: 2023-09-22 11:58:25.448 ::  9:23.36 >>> referenced by s_cbrt.o:(__cbrt) in archive /snap/gnome-42-2204-sdk/195/usr/lib/aarch64-linux-gnu/libm.a

I can see that behavior on both aarch64 and x86_64 platforms (I have not checked on armhf):

$ objdump -tT /lib/aarch64-linux-gnu/libm.a 2>/dev/null |grep -Ew "__frexpf|__scalbnf|__ldexpf|__frexp|__ldexp"
0000000000000000         *UND*	0000000000000000 __frexp
0000000000000000         *UND*	0000000000000000 __frexp
0000000000000000         *UND*	0000000000000000 __ldexp
0000000000000000         *UND*	0000000000000000 __frexpf
0000000000000000         *UND*	0000000000000000 __scalbnf
0000000000000000         *UND*	0000000000000000 __frexpf
0000000000000000         *UND*	0000000000000000 __ldexpf
0000000000000000         *UND*	0000000000000000 __scalbnf
0000000000000000         *UND*	0000000000000000 __scalbnf
0000000000000000         *UND*	0000000000000000 __scalbnf
0000000000000000         *UND*	0000000000000000 __scalbnf
[...]
$ objdump -tT /usr/lib/x86_64-linux-gnu/libm-2.37.a 2>/dev/null |grep -Ew "__frexpf|__scalbnf|__ldexpf|__frexp|__ldexp"
0000000000000000         *UND*  0000000000000000 __frexp
0000000000000000         *UND*  0000000000000000 __frexp
0000000000000000         *UND*  0000000000000000 __ldexp
0000000000000000         *UND*  0000000000000000 __frexpf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __frexpf
0000000000000000         *UND*  0000000000000000 __ldexpf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf

On a sysroot packaged by mozilla, it is defined in libm.a:

$ objdump -tT /home/alexandre/.mozbuild/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/libm.a 2>/dev/null |grep -Ew "__frexpf|__scalbnf|__ldexpf|__frexp|__ldexp"
0000000000000000         *UND*  0000000000000000 __frexp
0000000000000000         *UND*  0000000000000000 __frexp
0000000000000000         *UND*  0000000000000000 __ldexp
0000000000000000 g     F .text  0000000000000082 __frexp
0000000000000000 g     F .text  0000000000000089 __ldexp
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __frexpf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __frexpf
0000000000000000         *UND*  0000000000000000 __ldexpf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000 g     F .text  00000000000000d5 __scalbnf
0000000000000000 g     F .text  0000000000000058 __frexpf
0000000000000000 g     F .text  0000000000000065 __ldexpf
0000000000000000         *UND*  0000000000000000 __scalbnf
$ objdump -tT /home/alexandre/.mozbuild/sysroot-aarch64-linux-gnu/usr/lib/aarch64-linux-gnu/libm.a 2>/dev/null |grep -Ew "__frexpf|__scalbnf|__ldexpf|__frexp|__ldexp"
0000000000000000         *UND*  0000000000000000 __frexp
0000000000000000         *UND*  0000000000000000 __frexp
0000000000000000         *UND*  0000000000000000 __ldexp
0000000000000000 g     F .text  0000000000000060 __frexp
0000000000000000 g     F .text  0000000000000078 __ldexp
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __frexpf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __frexpf
0000000000000000         *UND*  0000000000000000 __ldexpf
0000000000000000 g     F .text  00000000000000e0 __scalbnf
0000000000000000 g     F .text  0000000000000060 __frexpf
0000000000000000 g     F .text  0000000000000078 __ldexpf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf
0000000000000000         *UND*  0000000000000000 __scalbnf

[alissy]

Ok, so the changes to gnome-sdk that seems to be enough are:

diff --git a/snapcraft.yaml b/snapcraft.yaml
index 910d9de..e2c0f02 100644
--- a/snapcraft.yaml
+++ b/snapcraft.yaml
@@ -1521,9 +1521,12 @@ parts:
       - gcc
       - gettext
       - itstool
+      - libasound2-dev
       - libblkid1
       - libbrotli1
       - libbrotli-dev
+      - libc6
+      - libc6-dev
       - libcairo2-dev
       - libcanberra-gtk3-dev
       - libcurl4
@@ -1542,6 +1545,7 @@ parts:
       - libfreetype6
       - libgbm-dev
       - libgraphene-1.0-dev
+      - libgcc-s1
       - libgcr-3-dev
       - libgcrypt20
       - libgcrypt20-dev
@@ -1595,6 +1599,7 @@ parts:
       - libsqlite3-0
       - libsqlite3-dev
       - libstdc++6
+      - libstdc++-12-dev
       - libtasn1-6
       - libtdb1
       - libthai-dev

Then just rewriting the path of pkg-config files, and fixing the libm.so symlink, I get to link libxul.so now …

[alissy]

[alissy]

[alissy]

[ogra]

Your firefox seems to have that french encryption module switched on

… congrats !!!

[alissy]

Ok, nailed it. A few more changes to properly set the target triplets and/or features during the build of firefox, and we get armhf snap cross-compiled (I had to build a GNOME SDK armhf of course for the sysroot aspect).

[alissy]

[alissy]