Request for enhanced gdb functionality

snapd
1

I had an offline discussion with @jdstrand and @mvo earlier.

I would like to request enhanced debug capability with gdb for snaps.

In the current form, it is very difficult to analyze application runtime, for several reasons:

  • There is no automatic core dump when one is created.
  • If one manually sets a core_pattern under /proc/sys (even inside $SNAP writable area), there will be an apparmor denial.
  • Gdb runs as root, so the reproduction of errors inside the debugger is not identical to what the user may experience when running the snap.

I would request:

  • Ability to dump cores into an area that is accessible by the snap (but possibly not other snaps) in the context of the current user. Perhaps this would not be a default, but enabled with a separate flag, e.g.: --core-dump.
  • Ability to run gdb as the user.
  • Ability to load previously dumped cores in the context of the snap and the user.

@jdstrand @mvo @niemeyer Any comments?

1 Like
2

Thanks for these suggestions!

I created a PR to run gdb as user in https://github.com/snapcore/snapd/pull/6641

We will look at the other ideas too.

1 Like
3

The loss of core dumps has bitten me a couple of times. Great idea! Difficult to know how best to manage symbol files for snaps, though. Any ideas?

4

Typically, for stripped binaries, you can do this:

objcopy --add-gnu-debuglink=foo.dbg foo

If the developer has symbols available, then could then link them during debug time, perhaps a combination of --gdb and --symbols option, which would then specify the location of symbols, and the actual linking would be done in the background (objcopy could be part of a symbols interface or similar).

5

This needs some design since we don’t want snaps modifying core_pattern, etc. I suspect we would want a crash handler ala apport/whoopsie that could process dumps from snaps and put them in the proper location. This should probably be accompanied with a ‘snap set’ command for turning off/on, defaulting to off. If that is in place, it shouldn’t be too difficult to add a companion command to ‘snap run --gdb’ to read the core file from the snap. I’m not sure that the core file should be in an existing SNAP_* location or in /var/cache/snapd/snap.<name>.* or similar.

6

A while back, I was working on a PR to get Snapcraft to do this:

This detached the debug symbols for binaries built by snapcraft itself (i.e. it didn’t handle stuff pulled in by stage-packages), and arranged them in a /usr/lib/debug style hierarchy indexed by build ID. Pointing GDB at these symbols via its debug-file-directory setting was enough to have it find symbols for the snapped binaries.

It’s obviously not a complete solution: collecting the symbols is just the first step towards being able to analyse crashes generated out in the field. It’s a necessary first step though.