Electron snap killed when using app.makeSingleInstance API

Hey folks! I’ve been working to package Mailspring (getmailspring.com) as a sandboxed Snap. You can view the snapcraft configuration file here: https://github.com/Foundry376/Mailspring/blob/master/snap/snapcraft.template.yaml.

It’s /so/ close to working nicely, but I’ve discovered a bit of a showstopper related to mailto:// link handling:

If I launch Mailspring (via a command prompt or via a shortcut, etc.) and then type mailspring on a second command prompt, one of two things happen:

  1. A new instance of Mailspring is started alongside the one already running. This is the correct behavior—It checks a lockfile, realizes another instance of Mailspring is running, sends the launch arguments to that instance to be handled, and exits.

  2. The existing instance of Mailspring is killed and the new one is started in it’s place. (In the terminal output of the running instance, I just see Killed.)

Scenario #2 happens about 50% of the time. Normally I imagine this wouldn’t be a big deal, but the system invokes mailspring mailto:... whenever a mailto link is clicked in another application and having the app restart is pretty infuriating.

I /think/ this is related to snapd because this behavior does not occur when Mailspring is installed outside of Snapcraft. (Scenario #1 works reliably on the same machine.)

Any help would be greatly appreciated!

Do you see anything in the syslog when this happens? The only time I’ve seen snapd kill something is when it made a syscall that wasn’t on the seccomp whitelist. You should see something about that in the log, though.

Hey! Ahh this is super interesting. Here’s what the syslog output looks like during a normal launch:


[ 7547.054893] audit: type=1400 audit(1509511805.480:87): apparmor="DENIED" operation="open" profile="snap.mailspring.mailspring" name="/etc/xdg/user-dirs.conf" pid=33761 comm="xdg-user-dirs-u" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 7547.109183] audit: type=1400 audit(1509511805.532:88): apparmor="DENIED" operation="file_mmap" profile="snap.mailspring.mailspring" name="/home/bengotow/.config/dconf/user" pid=33744 comm="mailspring" requested_mask="m" denied_mask="m" fsuid=1000 ouid=1000
[ 7547.365965] audit: type=1107 audit(1509511805.788:89): pid=1084 uid=106 auid=4294967295 ses=4294967295 msg='apparmor="DENIED" operation="dbus_method_call"  bus="system" path="/" interface="org.freedesktop.DBus.ObjectManager" member="GetManagedObjects" mask="send" name="org.bluez" pid=33744 label="snap.mailspring.mailspring" peer_pid=1008 peer_label="unconfined"
[ 7547.365965]  exe="/usr/bin/dbus-daemon" sauid=106 hostname=? addr=? terminal=?'

Here’s what it looks like when an instance is launched when another is already running (and the older instance is killed.):

[ 7513.434090] audit: type=1400 audit(1509511771.890:81): apparmor="DENIED" operation="open" profile="snap.mailspring.mailspring" name="/etc/xdg/user-dirs.conf" pid=33514 comm="xdg-user-dirs-u" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 7513.538599] audit: type=1400 audit(1509511771.998:82): apparmor="DENIED" operation="file_mmap" profile="snap.mailspring.mailspring" name="/home/bengotow/.config/dconf/user" pid=33495 comm="mailspring" requested_mask="m" denied_mask="m" fsuid=1000 ouid=1000
[ 7513.833014] audit: type=1400 audit(1509511772.293:83): apparmor="DENIED" operation="file_perm" profile="snap.mailspring.mailspring" name="/run/user/1000/snap.mailspring/.org.chromium.Chromium.7XwCRb/SS" pid=33495 comm="mailspring" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 7513.833022] audit: type=1400 audit(1509511772.293:84): apparmor="DENIED" operation="file_perm" profile="snap.mailspring.mailspring" name="/run/user/1000/snap.mailspring/.org.chromium.Chromium.7XwCRb/SS" pid=33495 comm="mailspring" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 7513.834170] audit: type=1400 audit(1509511772.293:85): apparmor="DENIED" operation="ptrace" profile="snap.mailspring.mailspring" pid=33495 comm="mailspring" requested_mask="trace" denied_mask="trace" peer="unconfined"
[ 7513.956821] audit: type=1107 audit(1509511772.417:86): pid=1084 uid=106 auid=4294967295 ses=4294967295 msg='apparmor="DENIED" operation="dbus_method_call"  bus="system" path="/" interface="org.freedesktop.DBus.ObjectManager" member="GetManagedObjects" mask="send" name="org.bluez" pid=33495 label="snap.mailspring.mailspring" peer_pid=1008 peer_label="unconfined"
[ 7513.956821]  exe="/usr/bin/dbus-daemon" sauid=106 hostname=? addr=? terminal=?'

That operation="ptrace" line stands out — looks like I need a trace permission?

Hmm - so investigating this further, I added the system-observe plug (to try to resolve the ptrace denial), attached the interface, and re-ran the test. That log line disappeared but the already-running instance of the app is still killed.

Some further testing. Uninstalling the snap and reinstalling it with --devmode resolves the issue, so it definitely seems to be a sandbox problem. When run in devmode, the syslog output looks like this, and the newly launched mailspring instance exits correctly and passes it’s launch params to the other running instance.

Oct 31 23:49:55 bengotow-virtual-machine kernel: [14736.808454] audit: type=1400 audit(1509518995.391:173): apparmor="ALLOWED" operation="open" profile="snap.mailspring.mailspring" name="/etc/xdg/user-dirs.conf" pid=41639 comm="xdg-user-dirs-u" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Oct 31 23:49:55 bengotow-virtual-machine kernel: [14736.858866] audit: type=1400 audit(1509518995.443:174): apparmor="ALLOWED" operation="file_mmap" profile="snap.mailspring.mailspring" name="/home/bengotow/.config/dconf/user" pid=41622 comm="mailspring" requested_mask="m" denied_mask="m" fsuid=1000 ouid=1000
Oct 31 23:49:55 bengotow-virtual-machine kernel: [14737.060676] audit: type=1400 audit(1509518995.643:175): apparmor="ALLOWED" operation="file_perm" profile="snap.mailspring.mailspring" name="/run/user/1000/snap.mailspring/.org.chromium.Chromium.Dytsxb/SS" pid=41622 comm="mailspring" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Oct 31 23:49:55 bengotow-virtual-machine kernel: [14737.060678] audit: type=1400 audit(1509518995.643:176): apparmor="ALLOWED" operation="file_perm" profile="snap.mailspring.mailspring" name="/run/user/1000/snap.mailspring/.org.chromium.Chromium.Dytsxb/SS" pid=41622 comm="mailspring" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

This seems like it might be similar to Would someone create an electron snap for this forum?.

EDIT: I also tried updating core to the very latest version (2.29+git446.aaee286) in case this might be fixed by this recent commit: https://github.com/snapcore/snapd/pull/4097/files/. No luck though.

I’m not sure where to look next @kyrofa —it seems like denying access to /home/bengotow/.config/dconf/user might be important, but I don’t know enough about dconf to know. (I’m a long time Mac + iOS developer but fairly noob linux user.) Mailspring is a fairly standard Electron application, so any changes required for this would likely also be relevant for other Electron apps.

the dconf issue should be solvable by adding (and making sure to have connected) the gsettings interface to your app…

readable ptrace support is in the system-observe interface (not sure if reading is enough in your case, but it is surely worth a try to add it)

The /etc/xdg/user-dirs.conf denial is fixed in 2.29 (the core snap in the beta channel should have this fix).

The bluez denials are just noise and likely not the cause of your issues. Feel free to plug the bluez interface if you want them to go away.

The ptrace denial is almost certainly due to cascading failures (eg something earlier failed and mailspring is trying to trace it, etc). The ‘ptrace trace peer=unconfined’ is not allowed by any interfaces (it would be a huge security issue: you would be able to control any unconfined process (eg, systemd), dozens of processes on a classic system, etc). But, like I said, I doubt you need it.

The dconf denial is for mmap(). When I was looking at mailspring wrt interface auto-connections, I reported that the snap suffers from having an executable stack:

$ snap-review ./mailspring_42.snap 
Warnings
--------
 - functional-snap-v2:execstack
	Found files with executable stack. This adds PROT_EXEC to mmap(2) during mediation which may cause security denials. Either adjust your program to not require an executable stack, strip it with 'execstack --clear-execstack ...' or remove the affected file from your snap. Affected files: usr/share/mailspring/mailspring
	https://forum.snapcraft.io/t/snap-and-executable-stacks/1812
./mailspring_42.snap: FAIL

(snap-review is from the ‘review-tools’ snap in edge).

Please see Snap and executable stacks and adjust your snap accordingly.

The cause of your problems I think are the /run/user denials. Your snap is allowed to read things in /run/user/*/snap.mailspring/* that the user owns, but notice the denial has ‘fsuid=1000 ouid=0’. This indicates that your snap is creating /run/user/1000/snap.mailspring/.org.chromium.Chromium.7XwCRb/SS with root permissions and that your non-root user process is trying to access it. You mention this is an electron app: it sounds like you have the internal security sandbox turned on. Please specify --disable-sandbox or otherwise configure your snap to not use the internal sandbox and simply rely upon snapd’s sandbox. I am not an electron developer (but understand the issues surrounding having its sandbox turned on within a snap), but it looks like https://github.com/electron/electron/blob/master/docs/api/sandbox-option.md may be of some help.

In short, I think if you remove the executable stack and disable the internal sandbox, things should start to work for you (and optionally start using the beta core snap).

Hey @ogra, @jdstrand - thanks for the replies:

  • Adding the system-observe, bluez, and process-control interfaces removed some of the warnings from the syslog output but didn’t fix the issue. (As expected)

  • I can confirm that the /etc/xdg/user-dirs.confg denial is fixed in 2.29.

  • Thanks for the info about the exectuable stack. When you say you reported it, should I have been notified somehow? I didn’t receive any feedback about the review—didn’t realize there was a snap-review command. I’ll see if I can fix that.

  • I don’t think I’m enabling the Chromium renderer process sandboxing (Mailspring doesn’t need it) but I’ll double check. Is there any other way around this other than disabling the sandbox mode? For apps that /do/ need it, Chromium’s sandbox mode is pretty important and isn’t equivalent to the snapd sandbox—asking folks to disable it should never be a general solution to sandboxing Electron snaps. (Snap’s OS-level sandbox gives my app access to the user’s keychain, files in their home directory, etc. The Chromium renderer process sandbox ensures that windows running untrusted javascript don’t have access to any of that.)

You mean like gnome-keyring? I’m not qualified to debate the rest of your points, but that’s actually covered by the password-manager-service interface, which isn’t automatically connected.

Hey! Yeah that’s a good point—the default Snap sandbox is pretty limited. Mailspring stores your email passwords in your system keyring, so the snap uses the password-manager-service interface.

UPDATE:

I fixed the execstack issue by adding execstack --clear-execstack <executable> to the build workflow and used execstack -q /snap/mailspring/current/usr/share/mailspring/mailspring to verify that it was fixed in the installed snap, but I’m still seeing the issue. Looking into the /run/user denials next.

I’ve also found that the /run/user reads + denials only occur when Mailspring uses Electron’s app.makeSingleInstance API. If I remove the code below, the denials go away and multiple copies of Mailspring run side by side and neither is killed.

    const otherInstanceRunning = app.makeSingleInstance(commandLine => {
      const otherOpts = parseCommandLine(commandLine);
      global.application.handleLaunchOptions(otherOpts);
    });

makeSingleInstance uses Chromium’s ProcessSingleton, so my guess is something in this implementation is acting as root, though nothing looks out of place offhand. Will keep digging in to this. I could discontinue use of this Electron API, but I’d need to implement a replacement and theirs seems very complete.

https://chromium.googlesource.com/chromium/chromium/+/master/chrome/browser/process_singleton_linux.cc

It looks like the ProcessSingleton implementation actually kills the “other running instance” deliberately if it’s unable to communicate with it through a shared socket [code], so it definitely makes sense that those AppArmor denials (for the shared socket file) result in the new process killing the old process. It looks like I just need to figure out why / how that .org.chromium.Chromium.Dytsxb/SS file is being created with root permissions. If anyone has any insight based on that source file, I’d appreciate it! This is well outside of my knowledge area :slight_smile:

I tried to provide that insight. There are only two ways this should be happening:

  • you enabled the chromium setuid sandbox and so it is starting as root. When I did the snap-review tool, it didn’t complain about setuid binaries, so this probably isn’t it
  • you are using ‘daemon’ as one of your commands which creates the socket as root

Which issue? This only would fix the mmap (‘m’) denial I referenced.

This is covered here:

The allow-sandbox attribute for the browser-support interface allows using the internal sandbox, but gives extra privileges to the app and triggers a manual review. Electron apps typically can rely on the snapd sandbox (in fact, electron-builder is already building snap swithout the sandbox).

The review happened as part of the request for interface auto-connections, which happened on your behalf via the snap advocacy team. I thought the information was forwarded to you, but seems it wasn’t. That’s ok, the snap-review tool was the extent of my feedback wrt executable stack.

Oh sorry! Since you brought it up in this thread I thought the mmap denial might be related to this issue.

I tried to provide that insight. There are only two ways this should be happening:

you enabled the chromium setuid sandbox and so it is starting as root. When I did the snap-review tool, it didn’t complain about setuid binaries, so this probably isn’t it you are using ‘daemon’ as one of your commands which creates the socket as root

Hey thanks for the reply!

  • I’m not using a daemon command. The snapcraft.yaml file is linked in my original post and only contains the app command.

  • I’ve verified that I am not enabling the Chromium sandbox. (It looks like --no-sandbox is the default now so passing it isn’t necessary [commit], and turning it on explicitly via mailspring --enable-sandbox causes the app to immediately crash. )

Since this bug is only exhibited when I use Electron’s app.makeSingleInstance API, I’ve updated the post title to reflect that and I’m going to keep digging in there.

I downloaded a few other Electron snaps and I’ve found the same problem occurs with mattermost-desktop, brave and wordpress-desktop. (Launch one and then run the launch command on a second command prompt. The running copy is killed and the new copy starts, which is not intended behavior.) I think it’s just more noticeable in Mailspring because the app is invoked frequently by mailto links. I imagine it will also prevent Brave from handling web links from other apps, though.

Thanks for this extra information. I haven’t yet had a chance to look further into this, but wanted to let you know I got the message.

Hey! Thanks for the reply—let me know if you get a chance to take a look and want me to try anything. Just for kicks, I tried packaging the snap with browser-support’s “allow-sandbox” option enabled, but it didn’t change the log output or fix the issue. (Not sure why it would, but I figured giving the app whatever extra permissions that entailed was worth a shot!)

I was able to reproduce this on Ubuntu 17.10 with gnome-shell/wayland. Importantly, it seems like an intermittent failure but truns out it is not (at least here). I launched mailspring from a terminal and it creates the socket with the correct permissions:

$ ls -ld /run/user/`id -u`/snap.mailspring/.org.chromium*
drwx------ 2 jamie jamie 80 Nov  8 10:11 /run/user/1000/snap.mailspring/.org.chromium.Chromium.g9eGF5

$ ls -l /run/user/`id -u`/snap.mailspring/.org.chromium*
total 0
lrwxrwxrwx 1 jamie jamie 19 Nov  8 10:11 SingletonCookie -> 3809248481036341864
srwxr-xr-x 1 jamie jamie  0 Nov  8 10:11 SS

If in a second terminal, without closing the first mailspring, when I launch another one from the command line, the first is not killed and the second says ‘Exiting because another instance of the app is already running.’. No apparmor denial for the socket. I tried this 30 times and it worked every time.

If I go to ‘File/Quit’ within mailspring, then the /run/user/1000/snap.mailspring/.org.chromium… is removed and cleaned up. In the first terminal I see the app is terminated. If in the first terminal I start the app and in a second try to start it, again it works fine (as described above) for 30 times.

However, I can reproduce the issue like so:

  • make sure the application is cleanly closed (eg, File/Quit)

  • in one terminal, launch mailspring. In this case, /run/user/1000/snap.mailspring/.org.chromium.Chromium.Aoy3tc is created and its socket has the correct permissions

  • click the ‘x’ button in the title bar (as opposed to File/Quit). The window goes away, but in the terminal we can see that the application is still running and the /run/user/1000/snap.mailspring/.org.chromium.Chromium.Aoy3tc is not cleaned up

  • launch mailspring in a second terminal. At this point, the mailspring in the first terminal exits, but a new socket directory is created: /run/user/1000/snap.mailspring/.org.chromium.Chromium.thlJJx and the first still exists. An apparmor denial is logged for the original socket directory:

    apparmor=“DENIED” operation=“file_perm” profile=“snap.mailspring.mailspring” name="/run/user/1000/snap.mailspring/.org.chromium.Chromium.Aoy3tc/SS" pid=17066 comm=“mailspring” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0

but looking at the socket after the fact, the file has the correct permissions;

 ls -ld /run/user/`id -u`/snap.mailspring/.org.chromium*
drwx------ 2 jamie jamie 80 Nov  8 10:19 /run/user/1000/snap.mailspring/.org.chromium.Chromium.Aoy3tc
drwx------ 2 jamie jamie 80 Nov  8 10:20 /run/user/1000/snap.mailspring/.org.chromium.Chromium.thlJJx

$ ls -l /run/user/`id -u`/snap.mailspring/.org.chromium*
/run/user/1000/snap.mailspring/.org.chromium.Chromium.Aoy3tc:
total 0
lrwxrwxrwx 1 jamie jamie 19 Nov  8 10:19 SingletonCookie -> 8465438638122226111
srwxr-xr-x 1 jamie jamie  0 Nov  8 10:19 SS

/run/user/1000/snap.mailspring/.org.chromium.Chromium.thlJJx:
total 0
lrwxrwxrwx 1 jamie jamie 20 Nov  8 10:20 SingletonCookie -> 10029366826429995547
srwxr-xr-x 1 jamie jamie  0 Nov  8 10:20 SS

This appears to be a bug in mailspring/electron not actually closing and cleaning up properly when receiving a window close event and not being able to find what it requires and thus kills off the process. Not sure why the kernel is reporting the socket as being owned by uid ‘0’ though, this may be a kernel bug.

Interestingly, if I add this to the policy (ie, no owner match):

/run/user/[0-9]*/snap.@{SNAP_NAME}/{,.}org.chromium.*/SS r,

then launching mailspring in the second terminal after window close works in that it doesn’t kill the first process, we see ‘Exiting because another instance of the app is already running.’ and a mailspring window pops up.

While I would argue that mailspring/electron is not doing the right thing on a window close event, I am going to add a workaround rule to the browser-support policy for the non-owner SS read. Will try to find a simplified reproducer to explore chasing down the kernel bug.

1 Like