Snapped LXD has stopped working - aa-exec: Permission denied


#21

webbrowser-app was installed by default via apt (I have never used it, nor desired to find out how to install it, so it wasn’t installed by me manually!), so why does it not depend on it’s required dependencies?!


#22

@zyga do you have any further ideas for troubleshooting? if not then I’ll do a reboot, but I’m loathed to do that if there’s still more info I can gather from the broken setup.


#23

No, I don’t have any other ideas. Sorry


#24

it seems the reboot successfully brought everything back into operation again. I’m thinking I must have hit a very rare race-condition on my previous reboot which caused snapd to get it’s knickers in a twist leading to the behaviour above.

TL;DR: the behaviour manifested immediately after bootup where nothing installed via snap was functional. Rebooting again fixed it.


#25

FYI, this is https://launchpad.net/bugs/1756800 and it is now fixed in 18.04 (as of yesterday). Please note that it does not affect snapd.


#26

I saw this issue yesterday on 18.04 with core from the beta channel (r4376). I saw the same apparmor errors. I then did ‘sudo snap interfaces’ and noticed that the lxd-support interface was not there. So I did sudo systemctl stop snapd ; sudo systemctl start snapd and the interfaces showed up again but the policy wan’t applied. So I tried to load the policy via apparmor_parser but the rules for lxd-support were not in there, so I had to sudo snap disconnect lxd:lxd-support ; sudo snap connect lxd-support to get the snap to work again.


#27

Please see this paste: https://paste.ubuntu.com/p/ySSrxXRgyK/. Timestamps are -0500. The problem came in between Apr 02 16:30 (when I believe I last successfully used lxd) and Apr 03 11:33 (when I noticed the apparmor errors and lxd did not work)

Looking at snap changes, I see:

2708  Done    2018-04-03T15:24:34Z  2018-04-03T15:26:04Z  Auto-refresh snaps "chromium", "lxd", "core"
2709  Done    2018-04-03T16:39:13Z  2018-04-03T16:39:14Z  Disconnect : from lxd:lxd-support
2710  Done    2018-04-03T16:39:17Z  2018-04-03T16:39:19Z  Connect lxd:lxd-support to core:lxd-support
2711  Done    2018-04-03T18:37:53Z  2018-04-03T18:38:36Z  Auto-refresh snap "lxd"
2712  Done    2018-04-04T04:47:54Z  2018-04-04T04:48:04Z  Auto-refresh snap "lxd"

which translates to Apr 03 10:24:34 for a refresh of core and lxd at the same time. So looking at the paste, at 11:37 is when I manually stop/started snapd and looking at snap changes I disconnected/connected lxd-support at 11:39.

Snaps did not show up as ‘broken’ when I noticed the problem.


#28

We roughly understand the mechanism that may explain this but need to come up with a plausible theory of why it could happen. I will update this bug with more information soon.


#29

Also note, I do not have the dotnet-sdk snap installed any more. I believe this is Please help testing 2.32 in beta


#30

FYI, I’ve been using the lxd snap for a long time. It may have plugged ‘home’ in the past but looking at /snap/lxd/current/meta/snap.yaml, it doesn’t plug ‘home’ any more.


#31

Finally, looking at the paste, it seems that chromium was likely also affected since it was refreshed at the same time as core and lxd. Unfortunately, I don’t use the chromium snap regularly so I don’t know if it was misbehaving.

It seems that one could try to reproduce this by installing older versions of lxd and core, then waiting for auto-refresh. Perhaps this could be done by installing stable of both, then snap switching both to edge, then running snap refresh.


#32

FYI, I’ve updated the topic since this issue has nothing to do with aa-exec not being in the snap. It has to do with aa-exec getting an EPERM.


#33

Some interesting log from IRC:

zyga> another bug getting in the way https://www.irccloud.com/pastebin/Q6aoFA0A/

mvo: zyga@t470:~/go/src/github.com/snapcore/snapd$ sudo ./bug.sh 
+ snap install core lxd
core already installed
lxd already installed
+ snap switch --edge core
"core" switched to the "edge" channel
+ snap switch --edge lxd
"lxd" switched to the "edge" channel
+ snap refresh
lxd (edge) git-b4ddcb9 from 'canonical' refreshed
core (edge) 16-2.32.2+git644.abf4628 from 'canonical' refreshed
+ snap interface lxd-support
name:    lxd-support
summary: allows operating as the LXD service
plugs:
  - lxd
slots:
  - core
+ snap switch --stable core
"core" switched to the "stable" channel
+ snap switch --stable lxd
"lxd" switched to the "stable" channel
+ snap refresh
error: cannot perform the following tasks:
- Run configure hook of "lxd" snap if present (run hook "configure": 
-----
error: cannot communicate with server: Post http://localhost/v2/snapctl: dial unix /run/snapd-snap.socket: connect: no such file or directory
error: cannot communicate with server: Post http://localhost/v2/snapctl: dial unix /run/snapd-snap.socket: connect: no such file or directory
-----)

zyga@t470:~/go/src/github.com/snapcore/snapd$ cat bug.sh 
#!/bin/sh
set -uxe
snap install core lxd
snap switch --edge core
snap switch --edge lxd
snap refresh
snap interface lxd-support
snap switch --stable core
snap switch --stable lxd
snap refresh
snap interface lxd-support

zyga: uhhh
I think we must add some code that waits for core restart with rest of setup
pedronis: ^ not sure what you would suggest for this, I’m looking at the snapd restart code now
mvo: FYI, I ran bug.sh exactly once :confused:
mvo: holly cow
reproduced the bigger bug
2nd run :expressionless:
I have lxd without lxd-support now
lxd-support bug trivially reproduced https://www.irccloud.com/pastebin/rootNBq3/

zyga@t470:~/go/src/github.com/snapcore/snapd$ sudo ./bug.sh 
+ snap install core lxd
core already installed
lxd already installed
+ snap switch --edge core
"core" switched to the "edge" channel
+ snap switch --edge lxd
"lxd" switched to the "edge" channel
+ snap refresh
core (edge) 16-2.32.2+git644.abf4628 from 'canonical' refreshed
+ snap interface lxd-support
name:    lxd-support
summary: allows operating as the LXD service
plugs:
  - lxd
slots:
  - core
+ snap switch --stable core
"core" switched to the "stable" channel
+ snap switch --stable lxd
"lxd" switched to the "stable" channel
+ snap refresh
core 16-2.32.1 from 'canonical' refreshed
lxd 3.0.0 from 'canonical' refreshed
+ snap interface lxd-support
name:    lxd-support
summary: allows operating as the LXD service
slots:
  - core
zyga@t470:~/go/src/github.com/snapcore/snapd$ snap infoc^C
zyga@t470:~/go/src/github.com/snapcore/snapd$ snap interfaces
Slot                                Plug
:account-control                    -
:accounts-service                   -
:alsa                               -
:autopilot-introspection            -
:avahi-control                      -
:avahi-observe                      firefox,mailspring
:bluetooth-control                  -
:bluez                              -
:broadcom-asic-control              -
:browser-support                    firefox:browser-sandbox,gitkraken,irccloud-desktop,mailspring,spotify
:camera                             firefox
:classic-support                    -
:core-support                       core:core-support-plug
:cups-control                       firefox,mailspring
:dcdbas-control                     -
:desktop                            firefox,gitkraken,irccloud-desktop,mailspring,minecraft,openra,spotify,telegram-desktop,vlc
:desktop-legacy                     firefox,irccloud-desktop,minecraft,telegram-desktop,vlc
:docker-support                     -
:firewall-control                   -
:framebuffer                        -
:fuse-support                       -
:gpg-keys                           -
:gpg-public-keys                    -
:gpio-memory-control                -
:greengrass-support                 -
:gsettings                          firefox,gitkraken,irccloud-desktop,mailspring,spotify,telegram-desktop
:hardware-observe                   -
:hardware-random-control            -
:hardware-random-observe            -
:home                               firefox,gitkraken,hugo,irccloud-desktop,mailspring,minecraft,spotify,telegram-desktop,test-snapd-sh,vlc
:io-ports-control                   -
:joystick                           -
:kernel-module-control              -
:kubernetes-support                 -
:kvm                                -
:libvirt                            -
:locale-control                     -
:log-observe                        -
:lxd-support                        -
:modem-manager                      -
:mount-observe                      vlc
:netlink-audit                      -
:netlink-connector                  -
:network                            0ad,boa,firefox,gitkraken,google-cloud-sdk,irccloud-desktop,mailspring,minecraft,openra,spotify,telegram-desktop,vlc
:network-bind                       0ad,google-cloud-sdk,hugo,irccloud-desktop,mailspring,minecraft,openra,telegram-desktop,vlc
:network-control                    -
:network-manager                    telegram-desktop
:network-observe                    mailspring
:network-setup-control              -
:network-setup-observe              -
:ofono                              -
:opengl                             0ad,boa,firefox,gitkraken,irccloud-desktop,minecraft,openra,spotify,vlc
:openvswitch                        -
:openvswitch-support                -
:optical-drive                      vlc
:password-manager-service           mailspring
:physical-memory-control            -
:physical-memory-observe            -
:ppp                                -
:process-control                    0ad,htop
:pulseaudio                         0ad,boa,firefox,irccloud-desktop,mailspring,minecraft,openra,spotify,telegram-desktop,vlc
:raw-usb                            -
:removable-media                    mailspring,vlc
:screen-inhibit-control             0ad,firefox,mailspring,vlc
:shutdown                           -
:snapd-control                      -
:ssh-keys                           -
:ssh-public-keys                    -
:system-observe                     classic-snap-analyzer,htop
:system-trace                       -
:time-control                       -
:timeserver-control                 -
:timezone-control                   -
:tpm                                -
:uhid                               -
:unity7                             boa,firefox,gitkraken,irccloud-desktop,mailspring,minecraft,spotify,telegram-desktop,vlc
:upower-observe                     firefox
:wayland                            gitkraken,irccloud-desktop,minecraft,openra
:x11                                0ad,firefox,gitkraken,mailspring,minecraft,openra,spotify,vlc
spotify:spotify-mpris               -
test-snapd-dbus-provider:dbus-test  -
vlc:mpris                           -
-                                   boa:joystick
-                                   firefox:network-observe
-                                   gitkraken:removable-media
-                                   hugo:removable-media
-                                   irccloud-desktop:mount-observe
-                                   mailspring:mount-observe
-                                   minecraft:joystick
-                                   openra:mount-observe
-                                   openra:removable-media
-                                   spotify:mount-observe
-                                   vlc:camera
zyga@t470:~/go/src/github.com/snapcore/snapd$ 
zyga@t470:~/go/src/github.com/snapcore/snapd$ cat /snap/lxd/current/meta/snap.yaml 
name: lxd
version: 3.0.0
summary: LXD - the container lightervisor
description: |-
  LXD is a container manager for system containers.

  It offers a REST API to remotely manage containers over the network,
  using an image based workflow and with support for live migration.

  Images are available for all Ubuntu releases and architectures as well
  as for a wide number of other Linux distributions.

  LXD containers are lightweight, secure by default and a great
  alternative to virtual machines.

  Supported configuration options (snap set lxd [<key>=<value>...]):
    - criu.enable: Enable experimental live-migration support [default=false]
    - daemon.debug: Increases logging to debug level [default=false]
    - daemon.group: Group of users that can interact with LXD [default=lxd]
    - ceph.builtin: Use snap-specific ceph configuration [default=false]
    - openvswitch.builtin: Run a snap-specific OVS daemon [default=false]
    - waitready.timeout: How long to wait for LXD to be ready [default=600]
architectures:
- amd64
confinement: strict
grade: stable
hooks:
  configure:
    plugs:
    - network
apps:
  benchmark:
    command: command-benchmark.wrapper
    plugs:
    - lxd-support
    - system-observe
  check-kernel:
    command: command-check-kernel.wrapper
    plugs:
    - lxd-support
    - system-observe
  daemon:
    command: command-daemon.wrapper
    daemon: simple
    plugs:
    - lxd-support
    - system-observe
    reload-command: commands/daemon.reload
    restart-condition: always
    slots:
    - lxd
    stop-command: stop-command-daemon.wrapper
    stop-timeout: 600s
  database:
    command: command-database.wrapper
    plugs:
    - lxd-support
    - system-observe
  lxc:
    command: command-lxc.wrapper
    completer: etc/bash_completion.d/snap.lxd.lxc
    plugs:
    - lxd-support
    - system-observe
  lxd:
    command: command-lxd.wrapper
    plugs:
    - lxd-support
    - system-observe
  migrate:
    command: command-migrate.wrapper
    plugs:
    - lxd-support
    - system-observe

note that spawn and ready times are more interesting than task order


#34

To summarize the post above:

  • refreshing core and lxd together via “snap switch --stable {core,lxd} && snap refresh” trick
  • refresh succeeds but the the lxd snap has no plugs at all
  • the yaml is correct

NOTE: we don’t return errors if we cannot read snap.yaml, we just return a snap.Info with Broken field. Hence nothing fails.


Snappy packaged automation + snap LXD "permission denied" error