@benfrancis - another gentle ping, just to let you know this is still on our radar, please don’t forget to update us on your progress when you get a chance to look at this further.
Apologies for the extended delay, the WebThings transition project I was working on took over for a while there!
@ijohnson I’ve had a stab at your proposed solution here. Can you tell me whether I’m going in the right direction?
My application already accepts a port number as a command line argument so it was easy to use a special port for the snap package by changing it from “80” to “systemd” in the launch command. Forgive my ignorance, but is “systemd” a magic value that Node.js is supposed to know about, or is there some other setup required for this to work?
The snap with the above configuration builds successfully but it won’t run. The logs are here.
Some lines that look like they might be relevant:
Dec 17 14:23:46 Unknown-b8-27-eb-f2-7b-0b krellian-kiosk.krellian-kiosk[3085]: _XSERVTransSocketCreateListener: failed to bind listener
Dec 17 14:23:46 Unknown-b8-27-eb-f2-7b-0b krellian-kiosk.krellian-kiosk[3085]: _XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
Dec 17 14:23:46 Unknown-b8-27-eb-f2-7b-0b krellian-kiosk.krellian-kiosk[3085]: _XSERVTransMakeAllCOTSServerListeners: failed to create listener for local
Dec 17 14:23:11 Unknown-b8-27-eb-f2-7b-0b krellian-kiosk.krellian-kiosk[2570]: /snap/krellian-kiosk/x2/bin/xwayland-kiosk-launch: line 151: /snap/krellian-kiosk/x2/bin/drop-snap-daemon.sh: Permission denied
Dec 17 14:23:47 Unknown-b8-27-eb-f2-7b-0b krellian-kiosk.krellian-kiosk[3085]: /snap/krellian-kiosk/x2/bin/xwayland-kiosk-launch: line 151: /snap/krellian-kiosk/x2/bin/drop-snap-daemon.sh: Permission denied
Do you have any further advice?
Much appreciated.
Ben
Hi again @benfrancis are you able to get system journal denials as well from journalctl --no-pager | grep DENIED ?
I’m curious what the denials are, as it appears to be trying to listen on the local socket but failing due to EPERM? Or perhaps the drop-snap-daemon.sh file is not executable ?
Also did you add the util-linux package to your stage-packages so that the setpriv utility is available from your snap?
Does this help?
Jan 13 16:48:53 Unknown-b8-27-eb-f2-7b-0b audit[2213]: AVC apparmor="DENIED" operation="open" profile="snap.krellian-kiosk.krellian-kiosk" name="/var/cache/fontconfig/" pid=2213 comm="ls" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 13 16:48:53 Unknown-b8-27-eb-f2-7b-0b kernel: audit: type=1400 audit(1610556533.545:427): apparmor="DENIED" operation="open" profile="snap.krellian-kiosk.krellian-kiosk" name="/var/cache/fontconfig/" pid=2213 comm="ls" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 13 16:48:53 Unknown-b8-27-eb-f2-7b-0b audit[2215]: AVC apparmor="DENIED" operation="capable" profile="snap.krellian-kiosk.krellian-kiosk" pid=2215 comm="Xwayland" capability=1 capname="dac_override"
Jan 13 16:48:53 Unknown-b8-27-eb-f2-7b-0b audit[2215]: AVC apparmor="DENIED" operation="bind" profile="snap.krellian-kiosk.krellian-kiosk" pid=2215 comm="Xwayland" family="unix" sock_type="stream" protocol=0 requested_mask="bind" denied_mask="bind" addr="@/tmp/.X11-unix/X63"
Jan 13 16:48:53 Unknown-b8-27-eb-f2-7b-0b kernel: audit: type=1400 audit(1610556533.981:428): apparmor="DENIED" operation="capable" profile="snap.krellian-kiosk.krellian-kiosk" pid=2215 comm="Xwayland" capability=1 capname="dac_override"
Jan 13 16:48:53 Unknown-b8-27-eb-f2-7b-0b kernel: audit: type=1400 audit(1610556533.981:429): apparmor="DENIED" operation="bind" profile="snap.krellian-kiosk.krellian-kiosk" pid=2215 comm="Xwayland" family="unix" sock_type="stream" protocol=0 requested_mask="bind" denied_mask="bind" addr="@/tmp/.X11-unix/X63"
Jan 13 16:48:54 Unknown-b8-27-eb-f2-7b-0b audit[2215]: AVC apparmor="DENIED" operation="capable" profile="snap.krellian-kiosk.krellian-kiosk" pid=2215 comm="Xwayland" capability=21 capname="sys_admin"
Jan 13 16:48:54 Unknown-b8-27-eb-f2-7b-0b kernel: audit: type=1400 audit(1610556534.037:430): apparmor="DENIED" operation="capable" profile="snap.krellian-kiosk.krellian-kiosk" pid=2215 comm="Xwayland" capability=21 capname="sys_admin"
I’m still curious about the magic “systemd” port.
Nope, I will try that now.
Maybe this was missed somewhere along the way, but you also need to specify require('systemd'); in your javascript somewhere, as that’s the package which will inherit the ports from systemd via listen-streams, etc.
this denial (and the others except the fontconfig one) are interesting but doesn’t seem to be related to inheriting the http socket from systemd, these errors seem to be related to your app trying to use XWayland things and not binding to the port that systemd is providing, does your snap work if you install it in devmode?
Also if you just use the systemd port specification but not drop privileges to snap_daemon user does it work? My guess is that your snap 1) works in devmode and 2) works when run as root, but not as snap_daemon because it seems that the snap is trying to do XWayland things which afaict need to be done as root.
Thanks for the ongoing help…
After adding the util-linux package to stage-packages, the full logs for the the krellian-kiosk snap are here.
The line…
/snap/krellian-kiosk/x1/bin/xwayland-kiosk-launch: line 151: bin/drop-snap-daemon.sh: No such file or directory
…suggests I did something wrong with drop-snap-daemon.sh
Maybe this was missed somewhere along the way, but you also need to specify
require('systemd');in your javascript somewhere, as that’s the package which will inherit the ports from systemd via listen-streams, etc.
I suspected I was missing something there! Unfortunately it appears that the systemd npm package may not work with the version of Node.js currently used by Electron, and is no longer maintained:
npm WARN notsup Unsupported engine for systemd@0.4.0: wanted: {"node":">=0.6.11 <0.11.0"} (current: {"node":"12.18.2","npm":"6.14.5"})
npm WARN notsup Not compatible with your version of node/npm: systemd@0.4.0
When my snap is installed using --devmode I don’t see the DENIED errors, but it’s still failing to start due to not being able to find drop-snap-daemon.sh (logs here).
it seems that the snap is trying to do XWayland things which afaict need to be done as root.
The application does use XWayland, as recommended in the Electron kiosk tutorial https://ubuntu.com/tutorials/electron-kiosk
If XWayland can not be used under snap_daemon then that’s going to be a problem.
@alan_g, @ogra: Do you know whether anyone has successfully run an Electron kiosk app (which uses XWayland on Ubuntu Core) as the snap_daemon user? I know @ogra has previously suggested this approach and @alan_g was updating tutorials to reflect current best practice.
you could try doing:
command: desktop-launch xwayland-kiosk-launch $SNAP/bin/drop-snap-daemon.sh $SNAP/krellian-kiosk/krellian-kiosk --no-sandbox -p systemd
in your app
Did you try the suggested workaround in this comment https://github.com/rubenv/node-systemd/issues/12#issuecomment-85501926 ?
nope, i don’t, but i know it is supposed to be possible to drop privileges to it … i guess you need to do this inside the application initialization or right before you start it since, as you noticed, XWayland needs to start as root first … but once you have a DISPLAY set you should be able to run as user
I think that’s what I started with but snapcraft kept telling me the $SNAP was redundant. Anyway I’ve added it back and will test.
I’ve rolled a patch for that and generated a new build with both fixes but I’m struggling to test it because I keep getting weird errors and timeouts on snap install/snap remove. Even at one point got “snap: command not found”!
All I can think is that microSD card in my Pi is wearing out, so I’ll try again next week with a fresh card + install.
@benfrancis what I think you should try first is to ignore the bin/drop-snap-daemon.sh bit for now and just work on getting listen-streams working to inherit the systemd file descriptors for your desired port. After you have confirmed that is working, then resume trying to get setpriv to work.
Could anyone point towards documentation for the “sockets” section of the snapcraft.yaml section @ijohnson proposed and exactly what it does?
I’ve been trying to get the systemd hand-off of the socket working as suggested, looking at the systemd configurations that snap is auto-generating and reading some background into this feature. The snapcraft.yaml reference mentions a “socket” (as opposed to “sockets”) feature, but doesn’t go into much detail as to how it should be used.
@ijohnson I note that the example project you linked to in your original email and the now un-maintained “systemd” npm library it uses are intended to provide “socket activation” for services, whereby services can be started on demand by systemd and shut down when not in use.
It occurs to me that whilst this might make sense for a simple Node.js server, it’s not really desirable for my Electron-based digital signage application which acts as both a web client (used to render web pages to the screen) and web server (used for remote control).
Whilst I haven’t yet been able to get the socket handoff working, I have noticed that the above configuration results in a snap which is no longer started on system boot, but is only started when a TCP connection is opened on port 80 (e.g. via an HTTP request). This isn’t going to work in my case, because the UI needs to be started on boot, and not terminated when the server hasn’t received any requests for a while.
I explained at the start of this thread why the (relatively simple) kiosk application I’m trying to package as a snap acts as both a web client and web server. That seemingly requires two things which ordinarily require root access:
- Starting a web server on port 80
- Starting a graphical interface using XWayland
In order to meet current snap store policies it seems my application can not be allowed to run as root, but so far the proposed solutions for making the server component run as a non-root user do not appear to work for the graphical client component of the application which uses XWayland and needs to run continuously from boot.
Have I understood this correctly, and can you offer any further advice on how I can modify my application so that it can be allowed into the snap store? I’m starting to wonder whether Ubuntu Core is a good fit for my use case after all 
Thanks
Sorry the documentation has a typo, this should be sockets. I’ll make sure this gets fixed and work on creating better examples in the docs…
That is part of the intended feature set, but that’s not why it was recommended for your project, for your project what is desirable about this is that your application doesn’t need to open port 80 by itself, and instead systemd opens that port and provides it via a file descriptor to your service when your service starts. Indeed, many projects (including some parts of snapd itself) rely on this in order to shut themselves down and start back up when a request is issued, but that is not strictly necessary. You can still have your service startup on boot and always be running when you use socket activation. Apologies if this was unclear. However read on…
Yes, it appears that we have stumbled onto a bug here, wherein there is not currently a way to actually start a daemon that is using socket activation except through actually socket activating it. I’m fairly certain this was unintentional and that this is a bug, but I guess what I expected to have worked is to have an install hook like this:
#!/bin/sh
snapctl start --enable "$SNAP_NAME.daemon"
which would manual start the daemon, however it appears looking at the code in snapd, in the case of there being a socket for the service “daemon”, we just plain skip over actually starting the daemon which is unintentional to say the least, and instead just make sure the socket is ready. I will file a bug to this effect and try to get that fixed for you. In the meantime, you could instead have something (which is hacky indeed) like this in your install hook that actually just manually socket activates the application:
#!/bin/bash -ex
snapctl start --enable "$SNAP_DAEMON.daemon"
attempts=0
until curl http://localhost:80/some-non-existent-route; do
sleep 1
attempts=$(( attempts + 1 ))
if [ "$attempts" = 10 ]; then
exit 1
fi
done
I say a non-existent route since you should at least then just trigger a 404 and not accidentally do something else that is unintended, and for the purposes of this hook you don’t even really need to make a complete HTTP request, you just need to try and write some data to that port so that systemd then activates the service. The loop is necessary since there is a race enabling the socket unit and trying to actually trigger the socket activation.
I made a full example of this here: https://github.com/anonymouse64/test-go-socket-activation-snap (but in Go not NodeJS since I don’t have much NodeJS experience). While doing so, I noticed that there is a slight error in the provided script to drop permissions, it should be calling exec on the setpriv command, since AFAICT only the PID that systemd launches is allowed to use the FD that systemd provides. So changing your drop-snap-daemon.sh script to this should get you further:
#!/bin/sh -e
exec "$SNAP/usr/bin/setpriv" --clear-groups --reuid snap_daemon --regid snap_daemon -- "$@"
If you wouldn’t mind giving that a try and seeing if you can make it further, I’m still hopeful we can make it work for your application, but do note that as mentioned originally, we still could allow your snap to use browser-sandbox in a daemon application with appropriate vetting if this still proves intractable.
To be clear, is this all done by the xwayland-kiosk-launch component of your script or does your NodeJS application need to do part of this as well? I am not familiar much with XWayland kiosks or electron for that matter…
Also, from above, you will need to ensure that all scripts / intermediary wrappers for your app end with an exec ... to ensure the same PID is used.
It is true, we’re lacking the means for expressing whether one wants to act on the socket, timer, or a corresponding service. There was some previous discussion in: Command line interface to manipulate services but the problem was not fully addressed then.
OTOH, if a service is socket activated, I expect it to be running only once it has been activated. While here we’re trying to use the socket activation as means of obtaining file descriptors bound to specific access-limited ports to avoid running as root.
1 Like
Might it make sense to extend the install-mode feature added by the following PR to cover this case?
While the current behaviour is probably the most common, there are certainly cases where you want socket activation and to be started on boot. For example, if you’ve got a set of co-dependent services then socket activation can be used as an alternative to before/after to order their start up.
1 Like
Yes, I think @jamesh is right in that we should extend install-mode to cover additional use cases, I guess my expectation is that socket activated services by default do not start up on their own and must be socket activated, but the bug I was trying to point out was that we have no way to actually start a socket activated service except for activating it which is awkward as can be seen in my examples above.
Thank you for the ongoing help. The fact that we’re uncovering bugs in the platform at least reassures me that I’m not just struggling with something obvious, but that we’re actually in slightly uncharted territory.
This is all handled by the xwayland-kiosk-launch command, which comes from the xwayland-kiosk-helper, as described in the Electron-based kiosk snap tutorial. I don’t fully understand all the ins and outs of how it works, but it launches “X11 applications on a Wayland kiosk, using Xwayland and a basic window manager to enforce fullscreen kiosk behaviour”.
OK, good news, I now have this working here https://github.com/krellian/kiosk/pull/75
OK, I made this change and I’m getting the same permissions error as before:
Feb 01 18:57:23 Unknown-b8-27-eb-f2-7b-0b-2 krellian-kiosk.krellian-kiosk[1917]: /snap/krellian-kiosk/x3/bin/xwayland-kiosk-launch: line 151: /snap/krellian-kiosk/x3/bin/drop-snap-daemon.sh: Permission denied
(Full output here.)
The output of sudo journalctl --no-pager | grep DENIED includes the following errors:
Feb 01 19:01:33 Unknown-b8-27-eb-f2-7b-0b-2 audit[6469]: AVC apparmor="DENIED" operation="open" profile="snap.krellian-kiosk.krellian-kiosk" name="/var/cache/fontconfig/" pid=6469 comm="ls" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Feb 01 19:01:33 Unknown-b8-27-eb-f2-7b-0b-2 kernel: audit: type=1400 audit(1612206093.283:501): apparmor="DENIED" operation="open" profile="snap.krellian-kiosk.krellian-kiosk" name="/var/cache/fontconfig/" pid=6469 comm="ls" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Feb 01 19:01:33 Unknown-b8-27-eb-f2-7b-0b-2 audit[6471]: AVC apparmor="DENIED" operation="bind" profile="snap.krellian-kiosk.krellian-kiosk" pid=6471 comm="Xwayland" family="unix" sock_type="stream" protocol=0 requested_mask="bind" denied_mask="bind" addr="@/tmp/.X11-unix/X24"
Feb 01 19:01:33 Unknown-b8-27-eb-f2-7b-0b-2 kernel: audit: type=1400 audit(1612206093.339:502): apparmor="DENIED" operation="bind" profile="snap.krellian-kiosk.krellian-kiosk" pid=6471 comm="Xwayland" family="unix" sock_type="stream" protocol=0 requested_mask="bind" denied_mask="bind" addr="@/tmp/.X11-unix/X24"
The output of ls /snap/krellian-kiosk/current/bin -al | grep drop-snap-daemon.sh is:
-rw-r--r-- 1 root root 105 Feb 1 17:44 drop-snap-daemon.sh
Suggesting the shell script is not in fact executable. How do I fix that? Is it sufficient to add the executable flag to the file in my local source tree and check it into git?
Thank you. Do you have a bug number I can follow so that I know when I can remove the hack?
I’ve not used hooks before. Can you explain why this should be in the install hook? My understanding from the documentation is that the install hook is only executed on first install, whereas presumably this socket activation needs to happen every time the application is started.
Can you explain what you mean by this? What edits do I need to make where exactly? The only shell script I’ve added myself is drop-snap-daemon.sh
Thanks
Given this, I think that your application probably doesn’t need to setup any kind of XWayland things and thus doesn’t need root for any of that. In any case we should find out as we get closer to the solution I think…
so that means that your listen-streams are working and your application is reading the file descriptors from systemd properly, and we just need to sort out the permission dropping bits.
this still looks suspicious to me, but is not related to dropping privileges AFAICT, so let’s ignore it for now…
Yes, you can run chmod +x drop-snap-daemon.sh and then commit the script to git and rebuild the snap. Let’s tackle this part next and see if this is sufficient to get a bit further.
Sure, here is the bug: Bug #1914121 “no way to actually start a socket activated servic...” : Bugs : snapd
Hmm, that’s a fair point I hadn’t thought about on reboots, I think to handle that case then unfortunately you would want to have the shell script in the install hook instead be a one-shot daemon that is also in your snap, so move the install hook script to somewhere in $SNAP/bin/activate-svc.sh or something (and make sure it is executable 
), then add something like this to your snapcraft.yaml in the apps section:
apps:
activator:
command: bin/activate-svc.sh
plugs:
- network
- network-bind # may not be necessary but just for good luck
but let’s not try this until we can get the current version you have with the hook working to make sure we can get the user permission dropping bits to work 
Then you are fine, the other scripts will already have this, and we just needed to add this to the drop-snap-daemon.sh script
@ijohnson Quick update:
I made drop-snap-daemon.sh executable (added to the PR), and I now get past the previous error to a new one:
Feb 02 18:11:15 Unknown-b8-27-eb-f2-7b-0b-2 krellian-kiosk.krellian-kiosk[2595]: /snap/krellian-kiosk/x5/bin/drop-snap-daemon.sh: 2: exec: /snap/krellian-kiosk/x5/usr/bin/setpriv: not found
What am I missing? Why is setpriv not included inside the snap? I added util-linux to stage-packages in the snapcraft.yaml as suggested (see here).