Stracing snap commands

Using strace with snaps is sometimes frustrating because of how snap commands are launched and the resulting flow of execution. Here is a high-level overview:

  • /snap/bin/hello-world is a symlink to /usr/bin/snap
  • when /snap/bin/hello-world is run, the snap command looks at argv[0] to determine what snap the command belongs to, it then sets up the snap’s environment based on that and then calls snap-confine with appropriate arguments for running the command under confinement
  • snap-confine examines its arguments and launches the snap command under whatever confinement is available using execv() via snap-exec

As such, something along the lines of the following is what most people try:

$ strace -f -D -vv -o ./hello-world.trace /snap/bin/hello-world.env
need to run as root or suid

Because snap-confine is setuid root, strace is saying that you must run it as root[1]. You may of course give the -u option to have strace drop privileges to the user. Eg:

$ sudo strace -u <your username> -f -D -vv -o ./hello-world.trace /snap/bin/hello-world

What is interesting and annoying is that most of the time, the above command will hang and this has to do with stracing a setuid executable. One day I decided to strace the command out of /snap/bin and running the command directly out of /snap/<snap>/current/bin and compared the output and I found that certain syscalls hang such that if I remove them using the -e option, I could get reliable straces. Now the invocation is:

$ sudo strace -u <your username> -e '!select,pselect6,_newselect,clock_gettime,sigaltstack,gettid,gettimeofday,nanosleep' -f -D -vv -o ./hello-world.trace /snap/bin/hello-world
Hello World!

The above is known to work on amd64 and armhf, so it’s possible other calls should be removed (please comment in this post if you find others that should be omitted). Of course, you can also use -e to specify only the syscalls you want and so long as you don’t include the problematic syscalls, it should work fine. Eg:

$ sudo strace -u <your username> -e 'open,mmap,mmap2' -f -D -vv -o ./hello-world.trace  /snap/bin/hello-world

Hopefully this will help you debug your snaps. Happy stracing!

[1] You might also be interested how the kernel might limit who can ptrace what. Many systems these days limit the use of ptrace to processes of the calling user. This can be temporarily adjusted using the kernel.yama.ptrace_scope sysctl. Eg:

$ sysctl kernel.yama.ptrace_scope  # show current value
kernel.yama.ptrace_scope = 1
$ sudo sysctl -w kernel.yama.ptrace_scope=0 # disable ptrace_scope
kernel.yama.ptrace_scope = 0
7 Likes

@dmitriis - fyi, this is what we talked about the other day.

@jdstrand, thanks a lot, very useful information! I will definitely use that in practice as I have to do a lot of stracing.

Thanks for those details, @jdstrand.

We recently discussed the idea of integrating strace and gdb support as first class features. The idea would be running a command similar to:

$ snap run --strace snap.app

Or alternatively if we want to allow providing arbitrary flags into strace:

$ snap strace [options] snap.app

Same idea for gdb, except we may need some more integration to have that one working smoothly.

Would anyone be interested in working on this sooner rather than later?

3 Likes

FYI, pselect6 should also be excluded. I’ve updated the post to reflect that.

I see that parts of this are landing in master now, could we perhaps also have an option to turn it off for production Ubuntu Core images ?

If an attacker gets in we shouldnt also give him the tools for reverse engineering that pre-installed proprietary app snap right out of the box (worse enough that he got into the device :stuck_out_tongue: ).

Or am I to worried here ?

Is there really a situation where an attacker, who is already accessing your device, would be thwarted by switching this off? As a user, I’d love to be able to run strace and (especially) gdb on snapped apps when trying to work out what has gone wrong. Having more hoops to jump through isn’t much fun :wink:

1 Like

as a user you would not have to bother… but as a security anxious customer you perhaps want an option (i.e. in the gadget snap) to disable this feature on your IoT gateway images …

We can certainly add an option to disable this. However if an attacker gets into the system this attacker will usually have a toolbox available via scp or wget to get strace,gdb and more. So even if we stop snap run --strace we probably won’t stop the attacker from running strace.

Running strace is now supported directly via snap run --strace in 2.31.

5 Likes

I’ve used this yesterday! So nice. Thank you so much!

While going through the process of debugging an application crash today I saw many steps that could be automated, so I filed a bug in apport and it made me wonder how this would be solved for snap packages? Should all debug symbols be included in the snap? In another snap?

Depending on your needs, you can ship the debug symbols with the snap, or you can use a content interface and expose the symbols to the snap as needed. In the future we’ll have a first class mechanism for that, similar to how we just sorted out stracing, but for now it’s okay to just pick one of those choices.

1 Like

I’ve decided my crush for today is @mvo. hug

1 Like

Passing options is possible with the upcoming 2.32. This works via:

snap run --strace="-tt -r" snap.app

Enoy.

2 Likes

Hello, what about ltrace?

It’s currently unsupported. Should be straightforward to add just looking at the strace implementation.

1 Like

Leave a note for tracing the relocation issue:

snap run --strace='-e trace=stat,open,read,mmap' _snap_name_ 2>&1 | tee strace.log