Cannot locate base snap core18: Permission denied

Running $ sudo dmesg | grep DENIED on the host does not return any values.

I am observing bizarre behaviour now, which I cannot plausibly explain.

When I created this post initially, I created a container and then another one when the first troubleshooting suggestions arrived.

Today, I see that those two “old” containers do not inhabit this error cannot locate base snap core18: Permission denied anymore but instead successfully launch the Chromium browser.

To investigate this further, I have now created a new, a third, container named ‘lxdcontainer3’ and followed the same steps as previously outlined. There I get the cannot locate base snap core18: Permission denied error again.

I then stopped the LXD service on the laptop (LXD host) with sudo snap stop lxd and then started it again with sudo snap start lxd. After this, I connected to the container ‘lxdcontainer3’ again, and I still got the cannot locate base snap core18: Permission denied error when trying to launch the Chromium browser. :frowning:

Since my laptop on which those containers run gets shut down when not used I suspected that a shut down might affect the behaviour observed. In the light of this, I shut down and booted my laptop and tested this with the ‘lxdcontainer3’ container again but the problem persisted.

As of the behaviour observed so far, I have the hunch that ‘lxdcontainer3’ may work tomorrow…

CC @pstolowski for his insight into disabling LXD

My laptop was shutdown over the weekend and I now tried to launch Chromium that is installed in ‘lxdcontainer3’ and it worked.

Behaviour summary as currently observed:

  1. When restarting the entire Ubuntu LXD host (laptop with snap LXD installed), the problem persists with error cannot locate base snap core18: Permission denied.
  2. When shutting down the Ubuntu LXD host and leave it until the next day (another calendar date), launching Chromium in the LXD container works.

I’ll look at it soon and see if I can reproduce.

I didn’t have luck reproducing this (I tried with a few 18.04 lxd containers; lxd snap 4.7 rev 18137, running on groovy as a host, restarted the host a couple of times); I didn’t have luck setting them for GUI apps following that tutorial though, but I think this may not be relevant for the problem; FWTW I had chromium and firefox snaps inside containers, they would run from snaps and were just failing accessing X.

One possible explanation (but it may be a long shot) is that something fails during an automatic snap refresh, and current symlink for core18 is lost; could you please:

  • see (now) if there are any errors reported by snap changes (and if there is an error, run snap change <id> where id is the id of the failed change and attach the output)
  • if the problems happens again, check snap changes again and also check if /snap/core18/current is a valid symlink?

Thank you for looking into this @pstolowski

As you mentioned that you tried with Ubuntu 18.04 LXD containers, I have tested this with Ubuntu 18.04 LXD containers now and can confirm that Chromium works with Ubuntu 18.04 LXD containers.

When testing with 18.04 LXD I noticed that snap list, on a vanilla 18.04 LXD container, shows no Snaps installed

ubuntu@lxdcontainer5:~$ snap list
No snaps are installed yet. Try 'snap install hello-world'.

Whereas snap list, on a vanilla 20.10 LXD container, shows;

ubuntu@lxdcontainer6:~$ sudo snap list
Name    Version   Rev    Tracking         Publisher   Notes
core18  20200929  1932   latest/stable    canonical✓  base
lxd     4.7       18137  latest/stable/…  canonical✓  -
snapd   2.47.1    9721   latest/stable    canonical✓  snap

Based on this, I figured out a workaround to make Chromium work immediately after it was installed, avoiding the Cannot locate base snap core18: Permission denied error.

The workaround is to basically uninstall the LXD and the core18 snaps and then install Chromium.

Below are is the sequence of commands to work around this error.

$ export LXD_CONTAINER_NAME="lxdcontainer6"

$ lxc launch ubuntu:20.10 --profile default $LXD_CONTAINER_NAME
Creating lxdcontainer6
Starting lxdcontainer6

$ lxc exec $LXD_CONTAINER_NAME -- sudo --user ubuntu --login
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@lxdcontainer6:~$ sudo snap list
Name    Version   Rev    Tracking         Publisher   Notes
core18  20200929  1932   latest/stable    canonical✓  base
lxd     4.7       18137  latest/stable/…  canonical✓  -
snapd   2.47.1    9721   latest/stable    canonical✓  snapd
ubuntu@lxdcontainer6:~$ sudo snap remove lxd
lxd removed
ubuntu@lxdcontainer6:~$ sudo snap remove core18
core18 removed
ubuntu@lxdcontainer6:~$ sudo snap list
Name   Version  Rev   Tracking       Publisher   Notes
snapd  2.47.1   9721  latest/stable  canonical✓  snapd
ubuntu@lxdcontainer6:~$ sudo snap changes
ID   Status  Spawn                   Ready               Summary
1    Done    yesterday at 11:33 UTC  today at 00:10 UTC  Initialize system state
2    Done    today at 00:10 UTC      today at 00:10 UTC  Initialize device
3    Done    today at 00:10 UTC      today at 00:10 UTC  Remove "lxd" snap
4    Done    today at 00:10 UTC      today at 00:10 UTC  Remove "core18" snap

ubuntu@lxdcontainer6:~$ sudo snap install chromium
chromium 86.0.4240.111 from Canonical✓ installed
ubuntu@lxdcontainer6:~$ sudo snap changes
ID   Status  Spawn                   Ready               Summary
1    Done    yesterday at 11:33 UTC  today at 00:10 UTC  Initialize system state
2    Done    today at 00:10 UTC      today at 00:10 UTC  Initialize device
3    Done    today at 00:10 UTC      today at 00:10 UTC  Remove "lxd" snap
4    Done    today at 00:10 UTC      today at 00:10 UTC  Remove "core18" snap
5    Done    today at 00:10 UTC      today at 00:14 UTC  Install "chromium" snap

ubuntu@lxdcontainer6:~$ sudo snap list
Name               Version          Rev   Tracking       Publisher   Notes
chromium           86.0.4240.111    1373  latest/stable  canonical✓  -
core18             20200929         1932  latest/stable  canonical✓  base
gtk-common-themes  0.1-36-gc75f853  1506  latest/stable  canonical✓  -
snapd              2.47.1           9721  latest/stable  canonical✓  snapd

ubuntu@lxdcontainer6:~$ ls -lah /snap/core18/1932
total 0
lrwxrwxrwx  1 root root 15 Sep 29 10:44 .disk -> /writable/.disk
drwxr-xr-x  2 root root  0 Sep 29 10:45 bin
drwxr-xr-x  6 root root  0 Sep 29 10:45 boot
drwxr-xr-x  2 root root  0 Sep 29 10:45 dev
drwxr-xr-x 42 root root  0 Sep 29 10:45 etc
drwxr-xr-x  2 root root  0 Apr 24  2018 home
drwxr-xr-x 14 root root  0 Sep 29 10:45 lib
drwxr-xr-x  2 root root  0 Sep 29 10:45 lib64
drwxr-xr-x  2 root root  0 Aug  6 22:33 media
drwxr-xr-x  2 root root  0 Sep 29 10:45 meta
drwxr-xr-x  2 root root  0 Aug  6 22:33 mnt
drwxr-xr-x  2 root root  0 Aug  6 22:33 opt
drwxr-xr-x  2 root root  0 Apr 24  2018 proc
drwx------  2 root root  0 Sep 29 10:45 root
drwxr-xr-x  2 root root  0 Sep 29 10:44 run
drwxr-xr-x  2 root root  0 Sep 29 10:45 sbin
drwxr-xr-x  2 root root  0 Sep 29 10:45 snap
drwxr-xr-x  2 root root  0 Aug  6 22:33 srv
-rw-r--r--  1 root root 53 Sep 29 10:45 stdout
drwxr-xr-x  2 root root  0 Apr 24  2018 sys
drwxrwxrwt  2 root root  0 Sep 29 10:45 tmp
drwxr-xr-x 11 root root  0 Sep 29 10:45 usr
drwxr-xr-x 12 root root  0 Sep 29 10:45 var
drwxr-xr-x  2 root root  0 Sep 29 10:44 writable

ubuntu@lxdcontainer6:~$ chromium
mkdir: cannot create directory ‘/run/user/1000’: Permission denied
[1457:1457:1104/001543.990873:ERROR:browser_main_loop.cc(1426)] Unable to open X display.

Since this problem reported with this post is not related to the X display forwarding, I am using the default LXD profile here and therefore the Unable to open X display error shown above is expected and not of relevance.

In answer to your question, to verify that /snap/core18/current is a valid symlink, I have observed the following:

The symlink shows correctly on the LXD host and the LXD containers:

ubuntu@lxdcontainer6:~$ ls -lah /snap/core18/current
lrwxrwxrwx 1 root root 4 Nov  4 00:11 /snap/core18/current -> 1932

However, the target directory 1932 shows permission denied when run from within the container:

ubuntu@lxdcontainer8:~$ ls -lah /snap/core18/1932 
ls: cannot access '/snap/core18/1932': Permission denied

When doing an ls for the /snap/core18 directory from within the LXD container, I get:

ubuntu@lxdcontainer8:~$ ls -lah /snap/core18/
ls: cannot access '/snap/core18/1932': Permission denied
total 2.5K
drwxr-xr-x 3 root root 4 Nov  3 11:33 .
drwxr-xr-x 8 root root 9 Nov  4 00:44 ..
d????????? ? ?    ?    ?            ? 1932
lrwxrwxrwx 1 root root 4 Nov  3 11:33 current -> 1932

Given the workaround I posted, I now got the impression this problem is not a problem with the Chromium snap but possibly a problem as to how core18 is installed on the LXD base image for Ubuntu 20:10.

What do you think?

For brevity, here is the sequence of commands to reproduce the error.

$ export LXD_CONTAINER_NAME="lxdcontainer8"

$ lxc launch ubuntu:20.10 --profile default $LXD_CONTAINER_NAME
Creating lxdcontainer8
Starting lxdcontainer8

$ lxc exec $LXD_CONTAINER_NAME -- sudo --user ubuntu --login

ubuntu@lxdcontainer8:~$ sudo snap list
Name    Version   Rev    Tracking         Publisher   Notes
core18  20200929  1932   latest/stable    canonical✓  base
lxd     4.7       18137  latest/stable/…  canonical✓  -
snapd   2.47.1    9721   latest/stable    canonical✓  snapd
ubuntu@lxdcontainer8:~$ ls -lah /snap/core18/
ls: cannot access '/snap/core18/1932': Permission denied
total 2.5K
drwxr-xr-x 3 root root 4 Nov  3 11:33 .
drwxr-xr-x 6 root root 7 Nov  3 11:33 ..
d????????? ? ?    ?    ?            ? 1932
lrwxrwxrwx 1 root root 4 Nov  3 11:33 current -> 1932
ubuntu@lxdcontainer8:~$ sudo snap changes
ID   Status  Spawn                   Ready               Summary
1    Done    yesterday at 11:33 UTC  today at 00:39 UTC  Initialize system state
2    Done    today at 00:39 UTC      today at 00:40 UTC  Initialize device

ubuntu@lxdcontainer8:~$ sudo snap install chromium
chromium 86.0.4240.111 from Canonical✓ installed
ubuntu@lxdcontainer8:~$ sudo snap list
Name               Version          Rev    Tracking         Publisher   Notes
chromium           86.0.4240.111    1373   latest/stable    canonical✓  -
core18             20200929         1932   latest/stable    canonical✓  base
gtk-common-themes  0.1-36-gc75f853  1506   latest/stable    canonical✓  -
lxd                4.7              18137  latest/stable/…  canonical✓  -
snapd              2.47.1           9721   latest/stable    canonical✓  snapd
ubuntu@lxdcontainer8:~$ sudo snap changes
ID   Status  Spawn                   Ready               Summary
1    Done    yesterday at 11:33 UTC  today at 00:39 UTC  Initialize system state
2    Done    today at 00:39 UTC      today at 00:40 UTC  Initialize device
3    Done    today at 00:42 UTC      today at 00:44 UTC  Install "chromium" snap

ubuntu@lxdcontainer8:~$ chromium
cannot locate base snap core18: Permission denied
ubuntu@lxdcontainer8:~$ ls -lah /snap/core18/
ls: cannot access '/snap/core18/1932': Permission denied
total 2.5K
drwxr-xr-x 3 root root 4 Nov  3 11:33 .
drwxr-xr-x 8 root root 9 Nov  4 00:44 ..
d????????? ? ?    ?    ?            ? 1932
lrwxrwxrwx 1 root root 4 Nov  3 11:33 current -> 1932
ubuntu@lxdcontainer8:~$ sudo snap change 1
Status  Spawn                   Ready                   Summary
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Ensure prerequisites for "snapd" are available
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Prepare snap "/var/lib/snapd/seed/snaps/snapd_9721.snap" (9721)
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Mount snap "snapd" (9721)
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Copy snap "snapd" data
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Setup snap "snapd" (9721) security profiles
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Make snap "snapd" (9721) available to the system
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Automatically connect eligible plugs and slots of snap "snapd"
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Set automatic aliases for snap "snapd"
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Setup snap "snapd" aliases
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Run install hook of "snapd" snap if present
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Start snap "snapd" (9721) services
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Run configure hook of "core" snap if present
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Ensure prerequisites for "core18" are available
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Prepare snap "/var/lib/snapd/seed/snaps/core18_1932.snap" (1932)
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Mount snap "core18" (1932)
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Copy snap "core18" data
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Setup snap "core18" (1932) security profiles
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Make snap "core18" (1932) available to the system
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Automatically connect eligible plugs and slots of snap "core18"
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Set automatic aliases for snap "core18"
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Setup snap "core18" aliases
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Run install hook of "core18" snap if present
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Start snap "core18" (1932) services
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Run health check of "core18" snap
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Ensure prerequisites for "lxd" are available
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Prepare snap "/var/lib/snapd/seed/snaps/lxd_18137.snap" (18137)
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Mount snap "lxd" (18137)
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Copy snap "lxd" data
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Setup snap "lxd" (18137) security profiles
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Make snap "lxd" (18137) available to the system
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Automatically connect eligible plugs and slots of snap "lxd"
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Set automatic aliases for snap "lxd"
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Setup snap "lxd" aliases
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Run install hook of "lxd" snap if present
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Start snap "lxd" (18137) services
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Run configure hook of "lxd" snap if present
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Run health check of "lxd" snap
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Mark system pre-seeded
Done    yesterday at 11:33 UTC  today at 00:39 UTC      Mark system seeded
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Connect lxd:lxd-support to snapd:lxd-support
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Connect lxd:network to snapd:network
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Connect lxd:network-bind to snapd:network-bind
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Connect lxd:system-observe to snapd:system-observe
Done    yesterday at 11:33 UTC  yesterday at 11:33 UTC  Setup snap "lxd" (18137) security profiles for auto-connections

ubuntu@lxdcontainer8:~$ sudo snap change 2
Status  Spawn               Ready               Summary
Done    today at 00:39 UTC  today at 00:39 UTC  Generate device key
Done    today at 00:39 UTC  today at 00:40 UTC  Request device serial

ubuntu@lxdcontainer8:~$ sudo snap change 3
Status  Spawn               Ready               Summary
Done    today at 00:42 UTC  today at 00:42 UTC  Ensure prerequisites for "chromium" are available
Done    today at 00:42 UTC  today at 00:44 UTC  Download snap "chromium" (1373) from channel "stable"
Done    today at 00:42 UTC  today at 00:44 UTC  Fetch and check assertions for snap "chromium" (1373)
Done    today at 00:42 UTC  today at 00:44 UTC  Mount snap "chromium" (1373)
Done    today at 00:42 UTC  today at 00:44 UTC  Copy snap "chromium" data
Done    today at 00:42 UTC  today at 00:44 UTC  Setup snap "chromium" (1373) security profiles
Done    today at 00:42 UTC  today at 00:44 UTC  Make snap "chromium" (1373) available to the system
Done    today at 00:42 UTC  today at 00:44 UTC  Automatically connect eligible plugs and slots of snap "chromium"
Done    today at 00:42 UTC  today at 00:44 UTC  Set automatic aliases for snap "chromium"
Done    today at 00:42 UTC  today at 00:44 UTC  Setup snap "chromium" aliases
Done    today at 00:42 UTC  today at 00:44 UTC  Run install hook of "chromium" snap if present
Done    today at 00:42 UTC  today at 00:44 UTC  Start snap "chromium" (1373) services
Done    today at 00:42 UTC  today at 00:44 UTC  Run configure hook of "chromium" snap if present
Done    today at 00:42 UTC  today at 00:44 UTC  Run health check of "chromium" snap
Done    today at 00:42 UTC  today at 00:42 UTC  Ensure prerequisites for "gtk-common-themes" are available
Done    today at 00:42 UTC  today at 00:43 UTC  Download snap "gtk-common-themes" (1506) from channel "stable"
Done    today at 00:42 UTC  today at 00:43 UTC  Fetch and check assertions for snap "gtk-common-themes" (1506)
Done    today at 00:42 UTC  today at 00:43 UTC  Mount snap "gtk-common-themes" (1506)
Done    today at 00:42 UTC  today at 00:43 UTC  Copy snap "gtk-common-themes" data
Done    today at 00:42 UTC  today at 00:43 UTC  Setup snap "gtk-common-themes" (1506) security profiles
Done    today at 00:42 UTC  today at 00:43 UTC  Make snap "gtk-common-themes" (1506) available to the system
Done    today at 00:42 UTC  today at 00:43 UTC  Automatically connect eligible plugs and slots of snap "gtk-common-themes"
Done    today at 00:42 UTC  today at 00:43 UTC  Set automatic aliases for snap "gtk-common-themes"
Done    today at 00:42 UTC  today at 00:43 UTC  Setup snap "gtk-common-themes" aliases
Done    today at 00:42 UTC  today at 00:43 UTC  Run install hook of "gtk-common-themes" snap if present
Done    today at 00:42 UTC  today at 00:43 UTC  Start snap "gtk-common-themes" (1506) services
Done    today at 00:42 UTC  today at 00:43 UTC  Run configure hook of "gtk-common-themes" snap if present
Done    today at 00:42 UTC  today at 00:43 UTC  Run health check of "gtk-common-themes" snap
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:audio-record to snapd:audio-record
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:camera to snapd:camera
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:desktop to snapd:desktop
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:removable-media to snapd:removable-media
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:screen-inhibit-control to snapd:screen-inhibit-control
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:x11 to snapd:x11
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:browser-sandbox to snapd:browser-support
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:chromium-config to snapd:personal-files
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:network-bind to snapd:network-bind
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:unity7 to snapd:unity7
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:bluez to snapd:bluez
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:etc-chromium-browser-policies to snapd:system-files
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:joystick to snapd:joystick
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:cups-control to snapd:cups-control
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:system-packages-doc to snapd:system-packages-doc
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:u2f-devices to snapd:u2f-devices
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:upower-observe to snapd:upower-observe
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:audio-playback to snapd:audio-playback
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:gsettings to snapd:gsettings
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:network to snapd:network
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:opengl to snapd:opengl
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:sound-themes to gtk-common-themes:sound-themes
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:gtk-3-themes to gtk-common-themes:gtk-3-themes
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:home to snapd:home
Done    today at 00:44 UTC  today at 00:44 UTC  Connect chromium:icon-themes to gtk-common-themes:icon-themes
Done    today at 00:44 UTC  today at 00:44 UTC  Setup snap "chromium" (1373) security profiles for auto-connections

wild guess here, but this looks more like a kernel bug or an issue with snapfuse … what is the host system running exactly ?

what is the output of snap version inside and outside of the container ?

I managed to reproduce it while playing with the containers (the key point was to create 20.10 container which I missed previously; unfortunately I’ve no clear steps and couldn’t reproduce in a clean VM :frowning:

Key observations - summary:

  • the host system is 20.10 , installed lxd 4.7 snap (rev 18137) on it
  • I created a mix of 18.04 and 20.10 lxc containers
  • I used lxdguiprofile.txt (from the tutorial linked above) for them, but I think this is irrelavant for the problem (it only sets up stuff around X11 / pulseaudio sockets).
  • in the 18.04 containers I installed firefox or chromium snaps (so that core18 base is pulled; not neccessary with 20.10 containers as they come with core18 already installed).
  • at some point, inside 20.10 container I saw there was a problem with core18 snap:
d????????? ? ?    ?    ?            ? 1932
  • snapfuse process for this revision of core18 was running.
  • no denials and no snapfuse errors in the log.

This indeed looks like an issue with kernel / snapfuse and/or interaction with lxd. CC @stgraber for ideas.

Below commands have been run on the LXD host and the container. The OS for the LXD host and the containers are the same, Ubuntu 20.10. All commands returned the same output on the LXD host and the container.

lsb_release -a:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.10
Release:	20.10
Codename:	groovy

uname -a:

$ uname -a
Linux machinename 5.8.0-26-generic #27-Ubuntu SMP Wed Oct 21 22:29:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

snap version:

$ snap version
snap    2.47.1+20.10.1
snapd   2.47.1+20.10.1
series  16
ubuntu  20.10
kernel  5.8.0-26-generic

looks like pawel is on the right track here …

Hmmm, so this seems to be related to use of ubuntu user with sudo, compare the two lxc exec commands for the same container:

pawel@pawel-virtual-machine:~$ lxc exec u4 -- /bin/bash
root@u4:~# ls -l /snap/core18
total 0
drwxr-xr-x 24 root root 0 Sep 29 10:45 1932
lrwxrwxrwx  1 root root 4 Nov  3 11:33 current -> 1932
root@u4:~# exit

pawel@pawel-virtual-machine:~$ lxc exec u4 -- sudo --user ubuntu --login
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@u4:~$ ls -l /snap/core18
ls: cannot access '/snap/core18/1932': Permission denied
total 0
d????????? ? ?    ?    ?            ? 1932
lrwxrwxrwx 1 root root 4 Nov  3 11:33 current -> 1932

Not seeing this problem here:

root@lxd-build:~# sudo --user ubuntu --login
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@lxd-build:~$ ls -l /snap/
total 16
-r--r--r-- 1 root root 548 Sep 24 15:24 README
drwxr-xr-x 2 root root   7 Nov  4 14:31 bin
drwxr-xr-x 3 root root   4 Nov  4 14:31 chromium
drwxr-xr-x 4 root root   5 Oct 20 17:43 core
drwxr-xr-x 3 root root   4 Nov  4 14:29 core18
drwxr-xr-x 4 root root   5 Oct 19 15:48 go
drwxr-xr-x 3 root root   4 Nov  4 14:29 gtk-common-themes
ubuntu@lxd-build:~$ ls -l /snap/core18/
total 1
drwxr-xr-x 24 root root 0 Sep 29 10:45 1932
lrwxrwxrwx  1 root root 4 Nov  4 14:29 current -> 1932

Can you show ps fauxww in that container as well as uname -a?
Also lxc config show --expanded NAME for that container.

Commands for launching container and accessing it as root.

$ export LXD_CONTAINER_NAME="container1"

$ lxc launch ubuntu:20.10 --profile default $LXD_CONTAINER_NAME                         
Creating container1
Starting container1

$ lxc exec $LXD_CONTAINER_NAME -- /bin/bash   

root@container1:~# ls -lah /snap/
total 7.0K
drwxr-xr-x  6 root root   7 Nov  3 11:33 .
drwxr-xr-x 18 root root  24 Nov  3 11:37 ..
-r--r--r--  1 root root 548 Nov  3 11:33 README
drwxr-xr-x  2 root root  10 Nov  3 11:33 bin
drwxr-xr-x  3 root root   4 Nov  3 11:33 core18
drwxr-xr-x  3 root root   4 Nov  3 11:33 lxd
drwxr-xr-x  3 root root   4 Nov  3 11:33 snapd

root@container1:~# ls -lah /snap/core18/
total 2.5K
drwxr-xr-x  3 root root 4 Nov  3 11:33 .
drwxr-xr-x  6 root root 7 Nov  3 11:33 ..
drwxr-xr-x 24 root root 0 Sep 29 10:45 1932
lrwxrwxrwx  1 root root 4 Nov  3 11:33 current -> 1932

root@container1:~# ls -lah /snap/core18/1932/
total 0
lrwxrwxrwx  1 root root 15 Sep 29 10:44 .disk -> /writable/.disk
drwxr-xr-x  2 root root  0 Sep 29 10:45 bin
drwxr-xr-x  6 root root  0 Sep 29 10:45 boot
drwxr-xr-x  2 root root  0 Sep 29 10:45 dev
drwxr-xr-x 42 root root  0 Sep 29 10:45 etc
drwxr-xr-x  2 root root  0 Apr 24  2018 home
drwxr-xr-x 14 root root  0 Sep 29 10:45 lib
drwxr-xr-x  2 root root  0 Sep 29 10:45 lib64
drwxr-xr-x  2 root root  0 Aug  6 22:33 media
drwxr-xr-x  2 root root  0 Sep 29 10:45 meta
drwxr-xr-x  2 root root  0 Aug  6 22:33 mnt
drwxr-xr-x  2 root root  0 Aug  6 22:33 opt
drwxr-xr-x  2 root root  0 Apr 24  2018 proc
drwx------  2 root root  0 Sep 29 10:45 root
drwxr-xr-x  2 root root  0 Sep 29 10:44 run
drwxr-xr-x  2 root root  0 Sep 29 10:45 sbin
drwxr-xr-x  2 root root  0 Sep 29 10:45 snap
drwxr-xr-x  2 root root  0 Aug  6 22:33 srv
-rw-r--r--  1 root root 53 Sep 29 10:45 stdout
drwxr-xr-x  2 root root  0 Apr 24  2018 sys
drwxrwxrwt  2 root root  0 Sep 29 10:45 tmp
drwxr-xr-x 11 root root  0 Sep 29 10:45 usr
drwxr-xr-x 12 root root  0 Sep 29 10:45 var
drwxr-xr-x  2 root root  0 Sep 29 10:44 writable

root@container1:~# ps fauxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         702  0.0  0.0   8952  2844 pts/0    Ss   00:14   0:00 /bin/bash
root         709  0.0  0.0  11476  2200 pts/0    R+   00:14   0:00  \_ ps fauxww
root           1  0.0  0.0 170676  8228 ?        Ss   00:07   0:00 /sbin/init
root          62  0.0  0.0  36240  9528 ?        Ss   00:07   0:00 /lib/systemd/systemd-journald
root          94  0.0  0.0  20452  3364 ?        Ss   00:07   0:00 /lib/systemd/systemd-udevd
root         113  0.0  0.0   3692  1236 ?        Ss   00:07   0:00 snapfuse /var/lib/snapd/snaps/lxd_18137.snap /snap/lxd/18137 -o ro,nodev,suid
root         114  0.0  0.0   3624  1188 ?        Ss   00:07   0:00 snapfuse /var/lib/snapd/snaps/snapd_9721.snap /snap/snapd/9721 -o ro,nodev,suid
root         115  0.0  0.0   3732  1412 ?        Ss   00:07   0:00 snapfuse /var/lib/snapd/snaps/core18_1932.snap /snap/core18/1932 -o ro,nodev,suid
systemd+     183  0.0  0.0  28012  5852 ?        Ss   00:07   0:00 /lib/systemd/systemd-networkd
systemd+     185  0.0  0.0  25156  9920 ?        Ss   00:07   0:00 /lib/systemd/systemd-resolved
root         220  0.0  0.0 238208  4744 ?        Ssl  00:07   0:00 /usr/lib/accountsservice/accounts-daemon
root         223  0.0  0.0   8524  1696 ?        Ss   00:07   0:00 /usr/sbin/cron -f
message+     224  0.0  0.0   8256  2884 ?        Ss   00:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         230  0.0  0.0  29928 14364 ?        Ss   00:07   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
syslog       231  0.0  0.0 151608  3084 ?        Ssl  00:07   0:00 /usr/sbin/rsyslogd -n -iNONE
root         234  0.0  0.0 2059004 29060 ?       Ssl  00:07   0:01 /usr/lib/snapd/snapd
root         236  0.0  0.0  17816  5620 ?        Ss   00:07   0:00 /lib/systemd/systemd-logind
daemon       238  0.0  0.0   3776  1540 ?        Ss   00:07   0:00 /usr/sbin/atd -f
root         248  0.0  0.0   7348  1444 pts/0    Ss+  00:07   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         250  0.0  0.0  13068  4500 ?        Ss   00:07   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root         260  0.0  0.0 233592  4804 ?        Ssl  00:07   0:00 /usr/libexec/polkitd --no-debug
root         269  0.0  0.0 108196 15804 ?        Ssl  00:07   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal

root@container1:~# exit
exit

Commands when container is accessed with Sudo

$ lxc exec $LXD_CONTAINER_NAME -- sudo --user ubuntu --login
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@container1:~$ ls -lah /snap/core18/
ls: cannot access '/snap/core18/1932': Permission denied
total 2.5K
drwxr-xr-x 3 root root 4 Nov  3 11:33 .
drwxr-xr-x 6 root root 7 Nov  3 11:33 ..
d????????? ? ?    ?    ?            ? 1932
lrwxrwxrwx 1 root root 4 Nov  3 11:33 current -> 1932

ubuntu@container1:~$ uname -a
Linux container1 5.8.0-26-generic #27-Ubuntu SMP Wed Oct 21 22:29:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@container1:~$ ps fauxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         680  0.0  0.0  11680  3232 pts/0    Ss   00:08   0:00 sudo --user ubuntu --login
ubuntu       681  0.0  0.0  10056  4180 pts/0    S    00:08   0:00  \_ -bash
ubuntu       698  0.0  0.0  11476  2236 pts/0    R+   00:09   0:00      \_ ps fauxww
root           1  0.0  0.0 170676  8224 ?        Ss   00:07   0:00 /sbin/init
root          62  0.0  0.0  36240  9524 ?        Ss   00:07   0:00 /lib/systemd/systemd-journald
root          94  0.0  0.0  20452  3364 ?        Ss   00:07   0:00 /lib/systemd/systemd-udevd
root         113  0.0  0.0   3692  1236 ?        Ss   00:07   0:00 snapfuse /var/lib/snapd/snaps/lxd_18137.snap /snap/lxd/18137 -o ro,nodev,suid
root         114  0.0  0.0   3624  1188 ?        Ss   00:07   0:00 snapfuse /var/lib/snapd/snaps/snapd_9721.snap /snap/snapd/9721 -o ro,nodev,suid
root         115  0.0  0.0   3732  1412 ?        Ss   00:07   0:00 snapfuse /var/lib/snapd/snaps/core18_1932.snap /snap/core18/1932 -o ro,nodev,suid
systemd+     183  0.0  0.0  28012  5852 ?        Ss   00:07   0:00 /lib/systemd/systemd-networkd
systemd+     185  0.0  0.0  25156  9920 ?        Ss   00:07   0:00 /lib/systemd/systemd-resolved
root         220  0.0  0.0 238208  4744 ?        Ssl  00:07   0:00 /usr/lib/accountsservice/accounts-daemon
root         223  0.0  0.0   8524  1696 ?        Ss   00:07   0:00 /usr/sbin/cron -f
message+     224  0.0  0.0   8256  2884 ?        Ss   00:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         230  0.0  0.0  29928 14364 ?        Ss   00:07   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
syslog       231  0.0  0.0 151608  3084 ?        Ssl  00:07   0:00 /usr/sbin/rsyslogd -n -iNONE
root         234  0.0  0.0 2059004 29060 ?       Ssl  00:07   0:01 /usr/lib/snapd/snapd
root         236  0.0  0.0  17816  5620 ?        Ss   00:07   0:00 /lib/systemd/systemd-logind
daemon       238  0.0  0.0   3776  1540 ?        Ss   00:07   0:00 /usr/sbin/atd -f
root         248  0.0  0.0   7348  1444 pts/0    Ss+  00:07   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         250  0.0  0.0  13068  4500 ?        Ss   00:07   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root         260  0.0  0.0 233592  4804 ?        Ssl  00:07   0:00 /usr/libexec/polkitd --no-debug
root         269  0.0  0.0 108196 15804 ?        Ssl  00:07   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal

lxc config show for container

$ lxc config show --expanded $LXD_CONTAINER_NAME
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.10 amd64 (release) (20201103)
  image.label: release
  image.os: ubuntu
  image.release: groovy
  image.serial: "20201103"
  image.type: squashfs
  image.version: "20.10"
  volatile.base_image: c010961d207c567fdb7cdd72405812d627151140f436835b115f9b673d0c02f6
  volatile.eth0.host_name: veth424c96e7
  volatile.eth0.hwaddr: 00:16:3e:93:2f:34
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

I am observing this behaviour with LXD, both stable and candidate.

$ sudo snap list lxd
Name  Version  Rev    Tracking          Publisher   Notes
lxd   4.7      18169  latest/candidate  canonical✓  -


$ sudo snap list lxd
Name  Version  Rev    Tracking       Publisher   Notes
lxd   4.7      18137  latest/stable  canonical✓  -

The issue is snapfuse, it’s not started with allow_other which is required for non-root users to have access to the snap data. This must be a snapd bug as at least here I’m certainly seeing it run with that particular option.

@pstolowski

stgraber@castiana:~/data/code/lxc/lxc-ci/bin (master)$ lxc exec lxd-build bash
root@lxd-build:~# ps fauxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         514  0.0  0.0  23204  2488 pts/0    Ss   01:12   0:00 bash
root         524  0.0  0.0  39100  2172 pts/0    R+   01:12   0:00  \_ ps fauxww
root           1  0.0  0.0 225284  6364 ?        Ss   Nov04   0:00 /sbin/init
root          49  0.0  0.0  78492  7336 ?        Ss   Nov04   0:00 /lib/systemd/systemd-journald
root          55  0.0  0.0  33396  2044 ?        Ss   Nov04   0:00 /lib/systemd/systemd-udevd
root         100  0.0  0.0  18152  1536 ?        Ss   Nov04   0:00 snapfuse /var/lib/snapd/snaps/core_10185.snap /snap/core/10185 -o ro,nodev,allow_other,suid
root         106  0.0  0.0  17908  1088 ?        Ss   Nov04   0:00 snapfuse /var/lib/snapd/snaps/go_6633.snap /snap/go/6633 -o ro,nodev,allow_other,suid
root         110  0.0  0.0  17776   164 ?        Ss   Nov04   0:00 snapfuse /var/lib/snapd/snaps/core18_1932.snap /snap/core18/1932 -o ro,nodev,allow_other,suid
root         113  0.0  0.0  17908   164 ?        Ss   Nov04   0:00 snapfuse /var/lib/snapd/snaps/chromium_1373.snap /snap/chromium/1373 -o ro,nodev,allow_other,suid
root         118  0.0  0.0  17776   160 ?        Ss   Nov04   0:00 snapfuse /var/lib/snapd/snaps/go_6439.snap /snap/go/6439 -o ro,nodev,allow_other,suid
root         119  0.0  0.0  17776   160 ?        Ss   Nov04   0:00 snapfuse /var/lib/snapd/snaps/core_10126.snap /snap/core/10126 -o ro,nodev,allow_other,suid
root         122  0.0  0.0  17908  1116 ?        Ss   Nov04   0:00 snapfuse /var/lib/snapd/snaps/gtk-common-themes_1506.snap /snap/gtk-common-themes/1506 -o ro,nodev,allow_other,suid
systemd+     126  0.0  0.0  80096  3652 ?        Ss   Nov04   0:00 /lib/systemd/systemd-networkd
systemd+     165  0.0  0.0  70676  3696 ?        Ss   Nov04   0:00 /lib/systemd/systemd-resolved
root         228  0.0  0.0 170384 13100 ?        Ssl  Nov04   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
syslog       229  0.0  0.0 193412  2416 ?        Ssl  Nov04   0:00 /usr/sbin/rsyslogd -n
root         230  0.0  0.0  31304  1456 ?        Ss   Nov04   0:00 /usr/sbin/cron -f
root         231  0.0  0.0  62064  3556 ?        Ss   Nov04   0:00 /lib/systemd/systemd-logind
root         232  0.0  0.1 1309384 24764 ?       Ssl  Nov04   0:11 /usr/lib/snapd/snapd
message+     233  0.0  0.0  49940  2444 ?        Ss   Nov04   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         237  0.0  0.0  15968  1056 pts/0    Ss+  Nov04   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         239  0.0  0.0 187100 14704 ?        Ssl  Nov04   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root         396  0.0  0.0  17968   808 ?        Ss   Nov04   0:00 snapfuse /var/lib/snapd/snaps/chromium_1382.snap /snap/chromium/1382 -o ro,nodev,allow_other,suid
root@lxd-build:~# 

As you can see, the allow_other is passed in my case.

2 Likes

Thanks @stgraber! I’ll check this.

I’ve tracked the problem down to snapd-generator for systemd, it’s affecting lxd images (snaps that come preinstalled on the these images) prepared with snap-preseed, the fix is proposed here:

https://github.com/snapcore/snapd/pull/9613

Once landed future groovy images for lxd should not be affected anymore. The problem should also disappear after core18 snap and snapd snap get refreshed, or snapd deb is upgraded inside the container.

1 Like

@pstolowski Thank you for fixing this!