Running Snaps on WSL2 (Insiders only for now)

OMGee

Graphical apps seem to work correctly now when you pair with a Windows X11 server such as the one within MobaXterm. See firefox running in the screenshot.

image

YooToobe

https://youtu.be/cWlYe0CE2iU

(https://youtu.be/cWlYe0CE2iU)

Howto

Prerequisites

This requires WSL2 (the new generation of Windows System for Linux) which is only available to insiders on the Fast Track. You need to have installed WSL2 by the methods documented at https://docs.microsoft.com/en-us/windows/wsl/wsl2-install.

Ubuntu prerequisites

Once you have your Ubuntu 18.04 installed from the store and you’ve opened it the first time, you need to install some extra packages that aren’t part of the out-of-the-box filesystem:

  • daemonize
  • dbus-user-session
  • fontconfig

These can be installed by running the following in the bash shell:

sudo apt-get update && sudo apt-get install -yqq daemonize dbus-user-session fontconfig

/usr/sbin/start-systemd-namespace

To always enable systemd and by extension snapd when you start bash you need to create a file at /usr/sbin/start-systemd-namespace with the following content:

#!/bin/bash

SYSTEMD_PID=$(ps -ef | grep '/lib/systemd/systemd --system-unit=basic.target$' | grep -v unshare | awk '{print $2}')
if [ -z "$SYSTEMD_PID" ] || [ "$SYSTEMD_PID" != "1" ]; then
    export PRE_NAMESPACE_PATH="$PATH"
    (set -o posix; set) | \
        grep -v "^BASH" | \
        grep -v "^DIRSTACK=" | \
        grep -v "^EUID=" | \
        grep -v "^GROUPS=" | \
        grep -v "^HOME=" | \
        grep -v "^HOSTNAME=" | \
        grep -v "^HOSTTYPE=" | \
        grep -v "^IFS='.*"$'\n'"'" | \
        grep -v "^LANG=" | \
        grep -v "^LOGNAME=" | \
        grep -v "^MACHTYPE=" | \
        grep -v "^NAME=" | \
        grep -v "^OPTERR=" | \
        grep -v "^OPTIND=" | \
        grep -v "^OSTYPE=" | \
        grep -v "^PIPESTATUS=" | \
        grep -v "^POSIXLY_CORRECT=" | \
        grep -v "^PPID=" | \
        grep -v "^PS1=" | \
        grep -v "^PS4=" | \
        grep -v "^SHELL=" | \
        grep -v "^SHELLOPTS=" | \
        grep -v "^SHLVL=" | \
        grep -v "^SYSTEMD_PID=" | \
        grep -v "^UID=" | \
        grep -v "^USER=" | \
        grep -v "^_=" | \
        cat - > "$HOME/.systemd-env"
    echo "PATH='$PATH'" >> "$HOME/.systemd-env"
    exec sudo /usr/sbin/enter-systemd-namespace "$BASH_EXECUTION_STRING"
fi
if [ -n "$PRE_NAMESPACE_PATH" ]; then
    export PATH="$PRE_NAMESPACE_PATH"
fi

/usr/sbin/enter-systemd-namespace

Next you need to create a new file at /usr/sbin/enter-systemd-namespace, with executable permissions (chmod +x), and containing the following code:

#!/bin/bash

if [ "$UID" != 0 ]; then
    echo "You need to run $0 through sudo"
    exit 1
fi

SYSTEMD_PID="$(ps -ef | grep '/lib/systemd/systemd --system-unit=basic.target$' | grep -v unshare | awk '{print $2}')"
if [ -z "$SYSTEMD_PID" ]; then
    /usr/sbin/daemonize /usr/bin/unshare --fork --pid --mount-proc /lib/systemd/systemd --system-unit=basic.target
    while [ -z "$SYSTEMD_PID" ]; do
        SYSTEMD_PID="$(ps -ef | grep '/lib/systemd/systemd --system-unit=basic.target$' | grep -v unshare | awk '{print $2}')"
    done
fi

if [ -n "$SYSTEMD_PID" ] && [ "$SYSTEMD_PID" != "1" ]; then
    if [ -n "$1" ] && [ "$1" != "bash --login" ] && [ "$1" != "/bin/bash --login" ]; then
        exec /usr/bin/nsenter -t "$SYSTEMD_PID" -a \
            /usr/bin/sudo -H -u "$SUDO_USER" \
            /bin/bash -c 'set -a; source "$HOME/.systemd-env"; set +a; exec bash -c '"$(printf "%q" "$@")"
    else
        exec /usr/bin/nsenter -t "$SYSTEMD_PID" -a \
            /bin/login -p -f "$SUDO_USER" \
            $(/bin/cat "$HOME/.systemd-env" | grep -v "^PATH=")
    fi
    echo "Existential crisis"
fi

Sudoers

Using sudo visudo, add the following sudoers rule to the bottom of the sudoers file (you must use the visudo utility for this, not vi or nano directly, so that the changes are correctly adopted by your system):

Defaults        env_keep += WSLPATH
Defaults        env_keep += WSLENV
Defaults        env_keep += WSL_INTEROP
Defaults        env_keep += WSL_DISTRO_NAME
Defaults        env_keep += PRE_NAMESPACE_PATH
%sudo ALL=(ALL) NOPASSWD: /usr/sbin/enter-systemd-namespace

/etc/bash.bashrc

To tie these files together we need to add a line to /etc/bash.bashrc, which triggers the process of starting and entering the namespace for all interactive bash sessions. This was previously handled by /etc/profile.d/00-wsl2-systemd.sh, which you should now delete although it won’t hurt to leave it. Execute the following command to add the required line:

sudo sed -i 2a"# Start or enter a PID namespace in WSL2\nsource /usr/sbin/start-systemd-namespace\n" /etc/bash.bashrc

We can then add two environment variables to Windows which will cause bash to read the bashrc file for all sessions, both interactive and non-interactive. Run the following two commands to achieve this:

cmd.exe /C setx WSLENV BASH_ENV/u
cmd.exe /C setx BASH_ENV /etc/bash.bashrc

Visual Studio Code should now correctly enter the namespace when using the WSL Remote extension :slight_smile:

Optional: Strict confinement support (custom kernel)

Compile yourself

  1. Check out (git clone) the source at https://github.com/diddlesnaps/WSL2-Linux-Kernel/tree/snapd-support into an Ubuntu distro:
    sudo apt-get install git-core
    git clone -b snapd-support https://github.com/diddlesnaps/WSL2-Linux-Kernel
    
  2. Install some compilers:
    sudo apt-get install build-essential flex bison libssl-dev libelf-dev
    
  3. Change directory into the kernel source and compile:
    cd WSL2-Linux-Kernel
    make KCONFIG_CONFIG=Microsoft/config-wsl
    
  4. Copy the compiled kernel to your Windows user profile directory (replace diddledan with your Windows username):
    cp arch/x86/boot/bzImage /mnt/c/Users/diddledan/vmlinux-wsl2-snapd
    
  5. In Windows create or edit a file at %USERPROFILE%\.wslconfig, e.g. C:\Users\diddledan\.wslconfig with at least the following two lines. Change the username to match your Windows username. The double-backslashes \\ are required:
    [wsl2]
    kernel=C:\\Users\\diddledan\\vmlinux-wsl2-snapd
    
  6. Exit all WSL2 sessions and in cmd.exe or powershell run wsl --shutdown to ensure that the new kernel is used when you restart your WSL2 session(s).

Use precompiled binary

  1. Download https://github.com/diddlesnaps/WSL2-Linux-Kernel/releases/download/4.19-snapd-20190923/vmlinux-wsl2-snapd
  2. Copy the binary to %USERPROFILE%, e.g. C:\Users\diddledan\vmlinux-wsl2-snapd
  3. Edit or create a file alongside called .wslconfig with at least the following two lines. Change the username to match your Windows username. The double-backslashes \\ are required:
    [wsl2]
    kernel=C:\\Users\\diddledan\\vmlinux-wsl2-snapd
    
  4. Exit all WSL2 sessions and in cmd.exe or powershell run wsl --shutdown to ensure that the new kernel is used when you restart your WSL2 session(s).

Done

Now every time you start the terminal/bash it will automatically put you inside a pid namespace with snapd running.

Diagnostics of the environment by snapd

Below are some simple diagnostic listings that show the environment according to snapd.

Version details

$ snap version
snap    2.41
snapd   2.41
series  16
ubuntu  18.04
kernel  4.19.72-microsoft-standard+

Confinement

With stock Microsoft kernel
$ snap debug confinement
partial
With custom kernel
$ snap debug confinement
strict

Sandbox features

With stock Microsoft kernel
$ snap debug sandbox-features
confinement-options:  classic devmode
dbus:                 mediated-bus-access
kmod:                 mediated-modprobe
mount:                freezer-cgroup-v1 layouts mount-namespace per-snap-persistency per-snap-profiles per-snap-updates per-snap-user-profiles stale-base-invalidation
seccomp:              bpf-actlog bpf-argument-filtering kernel:allow kernel:errno kernel:kill_process kernel:kill_thread kernel:log kernel:trace kernel:trap
udev:                 device-cgroup-v1 tagging
With custom kernel
$ snap debug sandbox-features
apparmor:             kernel:caps kernel:dbus kernel:domain kernel:file kernel:mount kernel:namespaces kernel:network kernel:network_v8 kernel:policy kernel:ptrace kernel:query kernel:rlimit kernel:signal parser:unsafe policy:default support-level:full
confinement-options:  classic devmode strict
dbus:                 mediated-bus-access
kmod:                 mediated-modprobe
mount:                freezer-cgroup-v1 layouts mount-namespace per-snap-persistency per-snap-profiles per-snap-updates per-snap-user-profiles stale-base-invalidation
seccomp:              bpf-actlog bpf-argument-filtering kernel:allow kernel:errno kernel:kill_process kernel:kill_thread kernel:log kernel:trace kernel:trap
udev:                 device-cgroup-v1 tagging

Edits (newest edit is last)

Edited to improve profile script.

Edited to update scripts - the old scripts are below for reference

00-wsl2-systemd.sh

SYSTEMD_PID=$(ps -ef | grep '/lib/systemd/systemd --system-unit=basic.target$' | grep -v unshare | awk '{print $2}')

if [ -z "$SYSTEMD_PID" ]; then
   sudo /usr/sbin/daemonize /usr/bin/unshare --fork --pid --mount-proc /lib/systemd/systemd --system-unit=basic.target
   SYSTEMD_PID=$(ps -ef | grep '/lib/systemd/systemd --system-unit=basic.target$' | grep -v unshare | awk '{print $2}')
fi

if [ -n "$SYSTEMD_PID" ] && [ "$SYSTEMD_PID" != "1" ]; then
    exec sudo /usr/bin/nsenter -t $SYSTEMD_PID -a su - $LOGNAME
fi

sudoers

%sudo ALL=(ALL) NOPASSWD: /usr/sbin/daemonize /usr/bin/unshare --fork --pid --mount-proc /lib/systemd/systemd --system-unit=basic.target
%sudo ALL=(ALL) NOPASSWD: /usr/bin/nsenter -t [0-9]* -a su - [a-zA-Z0-9]*

Update dependencies and profile script

  • Add APT dependencies for Ubuntu
  • Replace su with login to initialise a user.slice and user@.service within the session
    • This starts a systemd user instance with dbus supported (dbus is provided by dbus-user-session in the Ubuntu Dependencies section)

Update enter-systemd-namespace script

  • Add while loop to wait for systemd to actually start

Update scripts for WSL Interoperability (can start Windows apps from the PID namespace)

  • Pass $WSL_INTEROP to enter-systemd-namespace in 00-wsl2-systemd.sh profile script
  • Use $WSL_INTEROP in enter-systemd-namespace when entering the PID namespace

Update sudo config and scripts for env-passthru

  • Use sudoers configuration for environment passthrough
  • Remove brittle command-line environment variable pass-through

Update sudoers and enter-systemd-namespace, Add custom kernel

  • Remove usage of $LOGNAME
  • Don’t propagate $LOGNAME into sudo session
  • Add custom kernel
  • Update diagnostic output to include custom kernel variants

Update entry method

  • Remove /etc/profile.d/00-wsl2-systemd.sh
  • Add /usr/sbin/start-systemd-namespace
  • Add line to /etc/bash.bashrc to call start-systemd-namespace file

Update for non-interactive sessions

  • Edit /usr/sbin/start-systemd-namespace to pass initial command
  • Edit /usr/sbin/enter-systemd-namespace to use passed initial command
  • Add instructions to set Windows environment variables

Working VSCode WSL Remote extension

  • Edit /usr/sbin/start-systemd-namespace to save initial environment (required to capture VSCode-specific variables)
  • Edit /usr/sbin/enter-systemd-namespace to restore initial environment (required to pass-through VSCode-specific variables)

Docker Tech Preview support

  • Edit /usr/sbin/enter-systemd-namespace to better quote the input command when non-interactive sessions are required
10 Likes

I’m unsure if it’s currently a bug in WSL2, or if the PID Namespace that I create for systemd is conflicting with the ability to execute Windows .exes from within the WSL2 environment. I’m getting segfaults when I attempt to do so, but it isn’t clear where the segfault is occuring and strace is not helping :slight_smile: I think it’s crashing within Microsoft’s init binary which is hooked-up as the handler for .exe files via the binfmt_misc capability.

I managed to get GUI apps running. Check the latest version of the scripts in the original post, along with some new pre-requisites that need adding via APT.

If you’ve already set your box up, replace the scripts with the ones above and run the APT commands to install the bare-minimum packages for gui apps to operate.

2 Likes

The problem is that anything that drops the environment (and hence the WSL_INTEROP environment variable that points at the socket the inits use to communicate with each other) breaks interop - which includes sudo and various other utilities.

There’s a fix inbound, but for the moment, using sudo -E et. al. to preserve the environment, or simply copying WSL_INTEROP into the new namespace/sudo session/etc. manually can serve as a workaround. See here: https://github.com/microsoft/WSL/issues/4465

1 Like

Thanks for the pointer. I’ve updated the scripts to pass the $WSL_INTEROP variable into the PID namespace now.

Updated the scripts again to use sudo to pass through the environment variables from the initial shell into the root shell used to spawn systemd.

Wow, I am so down with this

What are your CPU/RAM utilization numbers in Windows before & after launching a GUI snap via x11 server?

Edit: Sorry I thought of another question, too - which version of Windows are you using?

Would really like to attempt this. Currently on 18363.356 (slow) insider build. Haven’t been able to get snapd to launch, does it require Linux (non-microsoft) kernel in WSL?

1 Like

Don’t think y’all’ve got WSL 2 in slow ring yet, have you (WSL 2 is in build 18917 or higher, latest is 18980)?

Last I checked, it was still 19H2 builds, with the 20H1 changeover not due until later this month/next month.

Also, when I got snapd to work under WSL 2, I had apparmor in the kernel, but I’m still only getting partial confinement. Anyone out there who can suggest what might need tweaking to get to strict?

Relevant output:

❯ snap debug confinement
partial

❯ snap debug sandbox-features
apparmor:             kernel:caps kernel:domain kernel:file kernel:mount kernel:namespaces kernel:network_v8 kernel:policy kernel:ptrace kernel:query kernel:rlimit kernel:signal parser:unsafe policy:downgraded support-level:partial
confinement-options:  classic devmode
dbus:                 mediated-bus-access
kmod:                 mediated-modprobe
mount:                freezer-cgroup-v1 layouts mount-namespace per-snap-persistency per-snap-profiles per-snap-updates per-snap-user-profiles stale-base-invalidation
seccomp:              bpf-actlog bpf-argument-filtering kernel:allow kernel:errno kernel:kill_process kernel:kill_thread kernel:log kernel:trace kernel:trap
udev:                 device-cgroup-v1 tagging

Hi, yes I have never tried Insiders before so I just went with what it recommended at first

Did some research and realized I needed 18862+ (I believe) so I set it to “fast” and now I’m at 18990 and have WSL2

Still working on getting snapd to work, have to set up the systemd container to spoof the init requirement, just haven’t gotten around to it yet

Thanks for your help :smiley:

I’ve developed a tool to do all the systemd container-ing for you in one go, if that helps any:

rad yes thank you! :smiley:

I’ve added details on using a custom kernel which I’ve preconfigured to support apparmor suitable for strict confinement of snaps. You can either download the one I built or build your own.

Also, I’ve removed the passthrough of $LOGNAME into the sudo session to improve the security stance. Instead we use $SUDO_USER to determine the original user to use when entering the namespace, which is more secure.

image

1 Like

More modifications this morning. VSCode’s WSL Remote extension should now run inside the systemd PID namespace.

It definitely works now with another set of modifications after this reply was written.

Hi there,

Thanks for all your work on this!!! But, I think I’ve borked something along the way. I followed the instructions successfully. I did not do the “optional” part. Snap is working now (thank you sir!):

snap    2.40+18.04
snapd   2.40+18.04
series  16
ubuntu  18.04
kernel  4.19.67-microsoft-standard

Then I tried to build my Docker image. Shortly after it gave up and provided this error message:

failed to build: cgroups: cannot find cgroup mount destination: unknown

I restarted. When I try to open a WSL project in VSCode I get the following dialog box:

“VS Code Server for WSL failed to start. No messages received for 90s Check WSL terminal for more details”

The terminal says:

[2019-10-18 19:50:23.859] Starting VS Code Server inside WSL (ubuntu)
[2019-10-18 19:50:23.860] Extension version: 0.39.9, Windows build: 18980. Multi distro support: enabled. WSL path support: enabled
[2019-10-18 19:50:23.968] Probing if server is already installed: C:\WINDOWS\System32\wsl.exe -d ubuntu -e sh -c “[ -d ~/.vscode-server-insiders/bin/5aec74ad3b55bb9463e717ec950e665288accd0f ] && echo found || ([ -f /etc/alpine-release ] && echo alpine || echo x64)”
[2019-10-18 19:50:24.045] Server install found in WSL
[2019-10-18 19:50:24.045] Launching C:\WINDOWS\System32\wsl.exe -d ubuntu sh -c '"$VSCODE_WSL_EXT_LOCATION/scripts/wslServer.sh" 5aec74ad3b55bb9463e717ec950e665288accd0f insider .vscode-server-insiders 0 ’ in c:\Users\Joel.vscode-insiders\extensions\ms-vscode-remote.remote-wsl-0.39.9
[2019-10-18 19:50:24.267] your 131072x1 screen size is bogus. expect trouble

I am on Windows Build 18980.vb_release.190907-1301

Can you give me a few ideas on how to unbork it?

For the visual studio code issue, double-check that the two scripts (enter-systemd-namespace and start-systemd-namespace) are as written above. I’ve been updating the original post and you possibly have a previous iteration. It might be worth recreating the two scripts via copy+paste again to be sure that they’re correct.

I’m not sure what’s wrong with docker just yet.

Thanks daniel. I went through all the steps again and recreated those two scripts as suggested. I’m sure those are there and correct now.

I did mess up one step the first time I went through the steps. I pasted the cmd.exe ... in the Ubuntu terminal instead of the Windows terminal. So I have fixed that.

I am getting a diff error message this time, which is probably a good thing:

During startup of VSCode, as it is Opening Remote, I am still getting the Starting VS Code in WSL (ubuntu): Installing WSL components message in the corner.

Followed by this error:

[2019-10-21 14:02:37.046] Starting VS Code Server inside WSL (ubuntu) [2019-10-21 14:02:37.047] Extension version: 0.39.9, Windows build: 18980. Multi distro support: enabled. WSL path support: enabled [2019-10-21 14:02:37.509] Probing if server is already installed: C:\WINDOWS\System32\wsl.exe -d ubuntu -e sh -c "[ -d ~/.vscode-server-insiders/bin/82ca6ba87f450245f665ab7a5f3a686e2cd00045 ] && echo found || ([ -f /etc/alpine-release ] && echo alpine || echo x64)" [2019-10-21 14:02:37.885] No server install found in WSL, needs x64 [2019-10-21 14:02:37.887] Launching C:\WINDOWS\System32\wsl.exe -d ubuntu sh -c '"$VSCODE_WSL_EXT_LOCATION/scripts/wslServer.sh" 82ca6ba87f450245f665ab7a5f3a686e2cd00045 insider .vscode-server-insiders 0 ' in c:\Users\Joel\.vscode-insiders\extensions\ms-vscode-remote.remote-wsl-0.39.9 [2019-10-21 14:02:38.222] your 131072x1 screen size is bogus. expect trouble [2019-10-21 14:04:10.895] VS Code Server for WSL failed to start. No messages received for 90s [2019-10-21 14:04:10.895] For help with startup problems, go to [2019-10-21 14:04:10.895] https://code.visualstudio.com/docs/remote/troubleshooting#_wsl-tips

The only diff between those messages is

Before:

[2019-10-18 19:50:24.045] Server install found in WSL

After:

[2019-10-21 14:02:37.885] No server install found in WSL, needs x64

And this at the end:

[2019-10-21 14:04:10.895] VS Code Server for WSL failed to start. No messages received for 90s [2019-10-21 14:04:10.895] For help with startup problems, go to [2019-10-21 14:04:10.895] https://code.visualstudio.com/docs/remote/troubleshooting#_wsl-tips

Do you have any other suggestions? Thank you for your time.

As a debugging step, can you open a terminal in your Ubuntu-based WSL. Separately, tell VSCode to connect so that you reproduce the timeout. Once you’ve hit the timeout run the following in the terminal you opened first:

$ grep WSLENV ~/.systemd-env

We’re looking for the output of grep to read similar to:

WSLENV=VSCODE_WSL_EXT_LOCATION/up:BASH_ENV/u

If it is there, run another grep:

$ grep VSCODE_WSL_EXT_LOCATION= ~/.systemd-env

This is expected to show something similar to below, with the path for your user folder instead of mine:

VSCODE_WSL_EXT_LOCATION=/mnt/c/Users/yabea/.vscode-insiders/extensions/ms-vscode-remote.remote-wsl-0.39.9

Thank you daniel

I screwed up by editing the visudo file in vim instead of visudo. After I fixed that, everything seems to be working. So, everything in the tutorial seems right on. Thank yoiu.

This tutorial is not about Docker so feel free to ignore this question, but when I try to open Docker in windows I get a message that says,

“This is a command line tool.
You need to open cmd.exe and run it from there.”

Edit: When I use a docker command like docker ps it says

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

After googling an hour I think it might be related to it looking for it in the old location, and we are sorta running in a different way.

Again, feel free to ignore this Docker question. It a little is off-topic. Thank you for your help.

1 Like

Well… things are a little more borked that I thought. I can’t get the “WSL 2 Tech Preview” Docker 2.1.00 to run anymore. When I click the “start” button in the Tech Preview GUI It says

Error from Lifecycle Server
exit status 1

Here is the log from Docker Lifecycle:

Here is the log from Docker Daemon:

The PATH file does seem to have unescaped characters in it. It was not an issue before, and I don’t think they were introduced in these instructions.

Also worth noting. When I tried the troubleshooting greps, those enteries were all missing. I added them to the file manually. Not sure why they were missing or if other things were missing too. I tried to followed the instructions to a T.

Any help debugging would be helpful. I thought it would be helpful to add this comment for other people trying this.

I’ve been trying to debug it for a few hours. I had a co-worker who is much smarter than me try to help me fix this for an hour, and he gave up and suggested I nuke it all and remove any new files that were introduced. If I have lost the ability to use VSCode with WSL 2, its not worth the trade off. High stakes to get Snap.

This is all cutting edge experimental stuff with WSL 2. I’m sure its me and not the instructions. It is just part of being on the latest and greatest new stuff.