webbrowser-app was installed by default via apt (I have never used it, nor desired to find out how to install it, so it wasn’t installed by me manually!), so why does it not depend on it’s required dependencies?!
@zyga-snapd do you have any further ideas for troubleshooting? if not then I’ll do a reboot, but I’m loathed to do that if there’s still more info I can gather from the broken setup.
No, I don’t have any other ideas. Sorry
it seems the reboot successfully brought everything back into operation again. I’m thinking I must have hit a very rare race-condition on my previous reboot which caused snapd to get it’s knickers in a twist leading to the behaviour above.
TL;DR: the behaviour manifested immediately after bootup where nothing installed via snap was functional. Rebooting again fixed it.
1 Like
FYI, this is https://launchpad.net/bugs/1756800 and it is now fixed in 18.04 (as of yesterday). Please note that it does not affect snapd.
I saw this issue yesterday on 18.04 with core from the beta channel (r4376). I saw the same apparmor errors. I then did ‘sudo snap interfaces’ and noticed that the lxd-support interface was not there. So I did sudo systemctl stop snapd ; sudo systemctl start snapd and the interfaces showed up again but the policy wan’t applied. So I tried to load the policy via apparmor_parser but the rules for lxd-support were not in there, so I had to sudo snap disconnect lxd:lxd-support ; sudo snap connect lxd-support to get the snap to work again.
Please see this paste: https://paste.ubuntu.com/p/ySSrxXRgyK/. Timestamps are -0500. The problem came in between Apr 02 16:30 (when I believe I last successfully used lxd) and Apr 03 11:33 (when I noticed the apparmor errors and lxd did not work)
Looking at snap changes, I see:
2708 Done 2018-04-03T15:24:34Z 2018-04-03T15:26:04Z Auto-refresh snaps "chromium", "lxd", "core"
2709 Done 2018-04-03T16:39:13Z 2018-04-03T16:39:14Z Disconnect : from lxd:lxd-support
2710 Done 2018-04-03T16:39:17Z 2018-04-03T16:39:19Z Connect lxd:lxd-support to core:lxd-support
2711 Done 2018-04-03T18:37:53Z 2018-04-03T18:38:36Z Auto-refresh snap "lxd"
2712 Done 2018-04-04T04:47:54Z 2018-04-04T04:48:04Z Auto-refresh snap "lxd"
which translates to Apr 03 10:24:34 for a refresh of core and lxd at the same time. So looking at the paste, at 11:37 is when I manually stop/started snapd and looking at snap changes I disconnected/connected lxd-support at 11:39.
Snaps did not show up as ‘broken’ when I noticed the problem.
We roughly understand the mechanism that may explain this but need to come up with a plausible theory of why it could happen. I will update this bug with more information soon.
Also note, I do not have the dotnet-sdk snap installed any more. I believe this is Please help testing 2.32 in beta
FYI, I’ve been using the lxd snap for a long time. It may have plugged ‘home’ in the past but looking at /snap/lxd/current/meta/snap.yaml, it doesn’t plug ‘home’ any more.
Finally, looking at the paste, it seems that chromium was likely also affected since it was refreshed at the same time as core and lxd. Unfortunately, I don’t use the chromium snap regularly so I don’t know if it was misbehaving.
It seems that one could try to reproduce this by installing older versions of lxd and core, then waiting for auto-refresh. Perhaps this could be done by installing stable of both, then snap switching both to edge, then running snap refresh.
FYI, I’ve updated the topic since this issue has nothing to do with aa-exec not being in the snap. It has to do with aa-exec getting an EPERM.
Some interesting log from IRC:
zyga> another bug getting in the way https://www.irccloud.com/pastebin/Q6aoFA0A/
mvo: zyga@t470:~/go/src/github.com/snapcore/snapd$ sudo ./bug.sh
+ snap install core lxd
core already installed
lxd already installed
+ snap switch --edge core
"core" switched to the "edge" channel
+ snap switch --edge lxd
"lxd" switched to the "edge" channel
+ snap refresh
lxd (edge) git-b4ddcb9 from 'canonical' refreshed
core (edge) 16-2.32.2+git644.abf4628 from 'canonical' refreshed
+ snap interface lxd-support
name: lxd-support
summary: allows operating as the LXD service
plugs:
- lxd
slots:
- core
+ snap switch --stable core
"core" switched to the "stable" channel
+ snap switch --stable lxd
"lxd" switched to the "stable" channel
+ snap refresh
error: cannot perform the following tasks:
- Run configure hook of "lxd" snap if present (run hook "configure":
-----
error: cannot communicate with server: Post http://localhost/v2/snapctl: dial unix /run/snapd-snap.socket: connect: no such file or directory
error: cannot communicate with server: Post http://localhost/v2/snapctl: dial unix /run/snapd-snap.socket: connect: no such file or directory
-----)
zyga@t470:~/go/src/github.com/snapcore/snapd$ cat bug.sh
#!/bin/sh
set -uxe
snap install core lxd
snap switch --edge core
snap switch --edge lxd
snap refresh
snap interface lxd-support
snap switch --stable core
snap switch --stable lxd
snap refresh
snap interface lxd-support
zyga: uhhh I think we must add some code that waits for core restart with rest of setup pedronis: ^ not sure what you would suggest for this, I’m looking at the snapd restart code now mvo: FYI, I ran bug.sh exactly once
mvo: holly cow reproduced the bigger bug 2nd run
I have lxd without lxd-support now lxd-support bug trivially reproduced https://www.irccloud.com/pastebin/rootNBq3/
zyga@t470:~/go/src/github.com/snapcore/snapd$ sudo ./bug.sh
+ snap install core lxd
core already installed
lxd already installed
+ snap switch --edge core
"core" switched to the "edge" channel
+ snap switch --edge lxd
"lxd" switched to the "edge" channel
+ snap refresh
core (edge) 16-2.32.2+git644.abf4628 from 'canonical' refreshed
+ snap interface lxd-support
name: lxd-support
summary: allows operating as the LXD service
plugs:
- lxd
slots:
- core
+ snap switch --stable core
"core" switched to the "stable" channel
+ snap switch --stable lxd
"lxd" switched to the "stable" channel
+ snap refresh
core 16-2.32.1 from 'canonical' refreshed
lxd 3.0.0 from 'canonical' refreshed
+ snap interface lxd-support
name: lxd-support
summary: allows operating as the LXD service
slots:
- core
zyga@t470:~/go/src/github.com/snapcore/snapd$ snap infoc^C
zyga@t470:~/go/src/github.com/snapcore/snapd$ snap interfaces
Slot Plug
:account-control -
:accounts-service -
:alsa -
:autopilot-introspection -
:avahi-control -
:avahi-observe firefox,mailspring
:bluetooth-control -
:bluez -
:broadcom-asic-control -
:browser-support firefox:browser-sandbox,gitkraken,irccloud-desktop,mailspring,spotify
:camera firefox
:classic-support -
:core-support core:core-support-plug
:cups-control firefox,mailspring
:dcdbas-control -
:desktop firefox,gitkraken,irccloud-desktop,mailspring,minecraft,openra,spotify,telegram-desktop,vlc
:desktop-legacy firefox,irccloud-desktop,minecraft,telegram-desktop,vlc
:docker-support -
:firewall-control -
:framebuffer -
:fuse-support -
:gpg-keys -
:gpg-public-keys -
:gpio-memory-control -
:greengrass-support -
:gsettings firefox,gitkraken,irccloud-desktop,mailspring,spotify,telegram-desktop
:hardware-observe -
:hardware-random-control -
:hardware-random-observe -
:home firefox,gitkraken,hugo,irccloud-desktop,mailspring,minecraft,spotify,telegram-desktop,test-snapd-sh,vlc
:io-ports-control -
:joystick -
:kernel-module-control -
:kubernetes-support -
:kvm -
:libvirt -
:locale-control -
:log-observe -
:lxd-support -
:modem-manager -
:mount-observe vlc
:netlink-audit -
:netlink-connector -
:network 0ad,boa,firefox,gitkraken,google-cloud-sdk,irccloud-desktop,mailspring,minecraft,openra,spotify,telegram-desktop,vlc
:network-bind 0ad,google-cloud-sdk,hugo,irccloud-desktop,mailspring,minecraft,openra,telegram-desktop,vlc
:network-control -
:network-manager telegram-desktop
:network-observe mailspring
:network-setup-control -
:network-setup-observe -
:ofono -
:opengl 0ad,boa,firefox,gitkraken,irccloud-desktop,minecraft,openra,spotify,vlc
:openvswitch -
:openvswitch-support -
:optical-drive vlc
:password-manager-service mailspring
:physical-memory-control -
:physical-memory-observe -
:ppp -
:process-control 0ad,htop
:pulseaudio 0ad,boa,firefox,irccloud-desktop,mailspring,minecraft,openra,spotify,telegram-desktop,vlc
:raw-usb -
:removable-media mailspring,vlc
:screen-inhibit-control 0ad,firefox,mailspring,vlc
:shutdown -
:snapd-control -
:ssh-keys -
:ssh-public-keys -
:system-observe classic-snap-analyzer,htop
:system-trace -
:time-control -
:timeserver-control -
:timezone-control -
:tpm -
:uhid -
:unity7 boa,firefox,gitkraken,irccloud-desktop,mailspring,minecraft,spotify,telegram-desktop,vlc
:upower-observe firefox
:wayland gitkraken,irccloud-desktop,minecraft,openra
:x11 0ad,firefox,gitkraken,mailspring,minecraft,openra,spotify,vlc
spotify:spotify-mpris -
test-snapd-dbus-provider:dbus-test -
vlc:mpris -
- boa:joystick
- firefox:network-observe
- gitkraken:removable-media
- hugo:removable-media
- irccloud-desktop:mount-observe
- mailspring:mount-observe
- minecraft:joystick
- openra:mount-observe
- openra:removable-media
- spotify:mount-observe
- vlc:camera
zyga@t470:~/go/src/github.com/snapcore/snapd$
zyga@t470:~/go/src/github.com/snapcore/snapd$ cat /snap/lxd/current/meta/snap.yaml
name: lxd
version: 3.0.0
summary: LXD - the container lightervisor
description: |-
LXD is a container manager for system containers.
It offers a REST API to remotely manage containers over the network,
using an image based workflow and with support for live migration.
Images are available for all Ubuntu releases and architectures as well
as for a wide number of other Linux distributions.
LXD containers are lightweight, secure by default and a great
alternative to virtual machines.
Supported configuration options (snap set lxd [<key>=<value>...]):
- criu.enable: Enable experimental live-migration support [default=false]
- daemon.debug: Increases logging to debug level [default=false]
- daemon.group: Group of users that can interact with LXD [default=lxd]
- ceph.builtin: Use snap-specific ceph configuration [default=false]
- openvswitch.builtin: Run a snap-specific OVS daemon [default=false]
- waitready.timeout: How long to wait for LXD to be ready [default=600]
architectures:
- amd64
confinement: strict
grade: stable
hooks:
configure:
plugs:
- network
apps:
benchmark:
command: command-benchmark.wrapper
plugs:
- lxd-support
- system-observe
check-kernel:
command: command-check-kernel.wrapper
plugs:
- lxd-support
- system-observe
daemon:
command: command-daemon.wrapper
daemon: simple
plugs:
- lxd-support
- system-observe
reload-command: commands/daemon.reload
restart-condition: always
slots:
- lxd
stop-command: stop-command-daemon.wrapper
stop-timeout: 600s
database:
command: command-database.wrapper
plugs:
- lxd-support
- system-observe
lxc:
command: command-lxc.wrapper
completer: etc/bash_completion.d/snap.lxd.lxc
plugs:
- lxd-support
- system-observe
lxd:
command: command-lxd.wrapper
plugs:
- lxd-support
- system-observe
migrate:
command: command-migrate.wrapper
plugs:
- lxd-support
- system-observe
note that spawn and ready times are more interesting than task order
To summarize the post above:
- refreshing core and lxd together via “snap switch --stable {core,lxd} && snap refresh” trick
- refresh succeeds but the the lxd snap has no plugs at all
- the yaml is correct
NOTE: we don’t return errors if we cannot read snap.yaml, we just return a snap.Info with Broken field. Hence nothing fails.
For what it’s worth, I ran into this a couple of days ago. My log was spammed with:
aug 24 00:53:34 rossak snapd[4517]: snapmgr.go:261: cannot read snap info of snap "core" at revision 7396: cannot find installed snap "core" at revision 7396: missing file /snap/core/7396/meta/snap.yaml
and pretty much nothing worked. The symptoms were similar to the ones already detailed in this thread. Ended up fixing it by doing:
sudo snap revert core
sudo snap refresh core
Everything worked afterwords.
Reviving this thread as this issue has been recently affecting the snapcraft, charmcraft, and rockcraft workflows.
Issue
When running spread tests for *craft applications, we occasionally see the following failure:
+ snap install lxd
error: cannot perform the following tasks:
- Run configure hook of "lxd" snap if present (run hook "configure":
-----
cat: /proc/self/attr/current: Permission denied
/snap/lxd/23680/snap/hooks/configure: 5: exec: aa-exec: Permission denied
-----)
The failure is occurring in clean VMs with a stock Ubuntu image (18.04, 20.04, and 22.04). The failure rate for these steps is very low - less than 1% of the time. In our github workflow, we run ~350 spread tests, so the failure of the overall work flow is around 10-15% (although some weeks it has been closer to 50%).
The environment setup is brief. We apply the following steps to the image:
apt-get install -y snapd
snap install snapd
snap wait system seed.loaded
if [ "$SPREAD_SYSTEM" = "ubuntu-18.04-64" ] || [ "$SPREAD_SYSTEM" = "ubuntu-20.04-64" ]; then
# Remove lxd and lxd-client deb packages as our implementation (pylxd) does not
# nicely handle the snap and deb being installed at the same time.
apt-get remove --purge --yes lxd lxd-client
fi
# Install and setup the lxd snap
snap install lxd
Attempted solutions
I tried the 3 primary suggestions from this thread with no luck:
-
Running
snap interfaces(nowsnap connections) This was suggested to check if the LXD snap is connected tolxd-supportinterface. This displays nothing, because the LXD snap is not installed yet. -
Running
snap install core20andsnap revert core20This gives the error:
+ snap revert core20
error: cannot revert "core20": no revision to revert to
-
Running
snap install core20andsnap refresh core20This still produces the sameexec: aa-exec: Permission deniederror.
I’ve captured the journal of a failure, with the failing section here:
oct142128-679423 audit[1763]: AVC apparmor="DENIED" operation="open" profile="snap.lxd.activate" name="/proc/1763/attr/current" pid=1763 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
oct142128-679423 audit[1764]: AVC apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 audit[1764]: AVC apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 kernel: kauditd_printk_skb: 4 callbacks suppressed
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:46): apparmor="DENIED" operation="open" profile="snap.lxd.activate" name="/proc/1763/attr/current" pid=1763 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:47): apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:48): apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 lxd.activate[1744]: The LXD snap was unable to run aa-exec, this usually indicates a LXD sideload.
oct142128-679423 lxd.activate[1744]: When sideloading, make sure to manually connect all interfaces.
oct142128-679423 systemd[1]: snap.lxd.activate.service: Deactivated successfully.
I’ve also cross-posted on the LXD forum. I also updated the existing launchpad bug.
Any advice or suggestions from the snapd team? Thanks!
1 Like