Getting debug symbols for libs in core22 snap

renat2017 · August 3, 2023, 7:41am

I’m trying to debug an application and for majority of libraries I have debug symbols, but I’m struggling to get debug symbols to libc in core22 snap (core22_819.snap).

To get the debug symbols I use Debuginfod, but it seems like it doesn’t offer the symbols for libc

export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"
gdb -ex "set debuginfod enabled on" ....

For some libraries it finds the symbols and for others (including libc), it doesn’t

Part of GDB output (libraries provided by core snap which have debug symbols downloaded)

0xb0351eb8  0xb0361a48  Yes         /home/rgalimov/sysroot/squashfs-root/lib/arm-linux-gnueabihf/liblzma.so.5
0xb02615f8  0xb0263a5c  Yes         /home/rgalimov/sysroot/squashfs-root/lib/arm-linux-gnueabihf/libcap.so.2
0

Part of GDB output (libraries provided by core snap which don’t have symbols)

0xb0e1d208  0xb0e27134  Yes (*)     /home/rgalimov/sysroot/squashfs-root/lib/arm-linux-gnueabihf/libgcc_s.so.1
0xb0cfd540  0xb0dc892c  Yes (*)     /home/rgalimov/sysroot/squashfs-root/lib/arm-linux-gnueabihf/libc.so.6

Notice the (*) as said by GDB: (*): Shared library is missing debugging information.

I also extracted the build ID from the file if it helps.

$ readelf -n /home/rgalimov/sysroot/squashfs-root/lib/arm-linux-gnueabihf/libc.so.6

Displaying notes found in: .note.gnu.build-id
  Owner                Data size 	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 4fe6c4942f57f6440ab05a7d898720c95a049ce1

Where can I get debug symbols from?

sergiusens · August 3, 2023, 3:22pm

This base derives from Ubuntu 22.04 LTS, which should have its debug symbols served through debuginfod, so following https://ubuntu.com/blog/debuginfod-is-now-available-in-ubuntu should get you on the right track.

renat2017 · August 3, 2023, 3:31pm

Hi @sergiusens, thank you for you response. As I mentioned - it’s exactly how I get symbols now.

export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com" and then set debuginfod enabled on in GDB. It works great with many other libraries installed, but doesn’t for with libc for some reason.

I will update my post to highlight that I use debuginfod.

sergiusens · August 3, 2023, 4:02pm

@sergiodj would you know what’s going on?

sergiodj · August 3, 2023, 6:17pm

Yes. Unfortunately glibc is one of those special packages that don’t generate a ddeb artifact; instead, it puts its debug symbols inside a regular deb package whose suffix is -dbg. Our debuginfod service should be able to index them, though. Let me take a look at what’s going on.

sergiodj · August 3, 2023, 6:25pm

I think I found what’s going on.

@renat2017, could you give it a couple of hours and try again, please?

renat2017 · August 4, 2023, 9:26am

I tried to read debug symbols with gdb, but no results. GDB reports no debug symbols for libc.
I extracted the build-id from the file and tried to get it with debuginfo-find which also didn’t work:

rgalimov@rgalimov-Apple-Virtualization-Generic-Platform:/etc/apt$ debuginfod-find debuginfo 1aa33098b532f6b01ab5ef9b6de8db1489059e7c # liblzma (finds)
/home/rgalimov/.cache/debuginfod_client/1aa33098b532f6b01ab5ef9b6de8db1489059e7c/debuginfo
rgalimov@rgalimov-Apple-Virtualization-Generic-Platform:/etc/apt$ debuginfod-find debuginfo 4fe6c4942f57f6440ab05a7d898720c95a049ce1 # libc (cannot find)
Server query failed: No such file or directory

sergiodj · August 7, 2023, 9:26pm

Could you provide the glibc version you’re using? I did a quick test here inside a Mantic container and debuginfod-find was able to locate the debuginfo.

renat2017 · August 9, 2023, 1:13pm

I don’t know where I find the version of the lib, but what I know is that it’s

libc6
Snap: core22
Arch: armhf
Build ID: 4fe6c4942f57f6440ab05a7d898720c95a049ce1

@sergiodj, here is how I get the build ID from the file in the snap:

Debugging machine:

$ unsquashfs -no-xattrs core22_859.snap 
Parallel unsquashfs: Using 4 processors
12503 inodes (13099 blocks) to write


create_inode: could not create character device squashfs-root/dev/null, because you're not superuser!

create_inode: could not create character device squashfs-root/dev/random, because you're not superuser!

create_inode: could not create character device squashfs-root/dev/urandom, because you're not superuser!

create_inode: could not create character device squashfs-root/dev/zero, because you're not superuser!
[================================================================================================================================================================================================/ ] 13095/13099  99%

created 10326 files
created 1629 directories
created 2053 symlinks
created 0 devices
created 0 fifos
 $ cat squashfs-root/meta/snap.yaml | grep version
version: '20230725'     
 $ readelf -n squashfs-root/usr/lib/arm-linux-gnueabihf/libc.so.6  | head

Displaying notes found in: .note.gnu.build-id
  Owner                Data size 	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 4fe6c4942f57f6440ab05a7d898720c95a049ce1

Desktop, where debuginfod installed:

rgalimov@rgalimov-Apple-Virtualization-Generic-Platform:~$ export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"
rgalimov@rgalimov-Apple-Virtualization-Generic-Platform:~$ debuginfod-find debuginfo 4fe6c4942f57f6440ab05a7d898720c95a049ce1
Server query failed: No such file or directory

Another file from the same snap has the symbols:

$ readelf -n squashfs-root/usr/lib/arm-linux-gnueabihf/liblzma.so.5 | head

Displaying notes found in: .note.gnu.build-id
  Owner                Data size 	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 1aa33098b532f6b01ab5ef9b6de8db1489059e7c

rgalimov@rgalimov-Apple-Virtualization-Generic-Platform:~$ debuginfod-find debuginfo 1aa33098b532f6b01ab5ef9b6de8db1489059e7c
/home/rgalimov/.cache/debuginfod_client/1aa33098b532f6b01ab5ef9b6de8db1489059e7c/debuginfo