popey
November 14, 2019, 8:36am
1
I frequently spin up clean lxd containers in which to build software. Today (maybe started before today, but certainly today) it’s broken. I can’t install core inside a 16.04 clean lxd container.
# snap version
snap 2.27.5
snapd 2.27.5
series 16
ubuntu 16.04
kernel 5.0.0-34-generic
# snap install node --classic --channel=12/stable
error: cannot perform the following tasks:
- Mount snap "core" (8039) ([start snap-core-8039.mount] failed with exit status 1: Job for snap-core-8039.mount failed. See "systemctl status snap-core-8039.mount" and "journalctl -xe" for details.
)
root@signal-desktop-20191114-082626:~# journalxtl -xe
No command 'journalxtl' found, did you mean:
Command 'journalctl' from package 'systemd' (main)
journalxtl: command not found
root@signal-desktop-20191114-082626:~# systemctl status snap-core-8039.mount
● snap-core-8039.mount - Mount unit for core
Loaded: loaded (/etc/systemd/system/snap-core-8039.mount; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2019-11-14 08:32:01 UTC; 27s ago
Where: /snap/core/8039
What: /var/lib/snapd/snaps/core_8039.snap
Process: 1019 ExecMount=/bin/mount /var/lib/snapd/snaps/core_8039.snap /snap/core/8039 -t squashfs -o nodev,ro (code=exited, status=32)
Nov 14 08:32:01 signal-desktop-20191114-082626 systemd[1]: Mounting Mount unit for core...
Nov 14 08:32:01 signal-desktop-20191114-082626 mount[1019]: mount: /snap/core/8039: mount failed: Unknown error -1
Nov 14 08:32:01 signal-desktop-20191114-082626 systemd[1]: snap-core-8039.mount: Mount process exited, code=exited status=32
Nov 14 08:32:01 signal-desktop-20191114-082626 systemd[1]: Failed to mount Mount unit for core.
Nov 14 08:32:01 signal-desktop-20191114-082626 systemd[1]: snap-core-8039.mount: Unit entered failed state.
Host is 19.10, lxd is installed from snap, stable channel.
Can you run this line manually?
If that fails, try adding LIBMOUNT_DEBUG=all LOOPDEV_DEBUG=all in the environment.
popey
November 14, 2019, 10:34am
3
# /bin/mount /var/lib/snapd/snaps/core_8039.snap /snap/core/8039 -t squashfs -o nodev,ro
mount: /snap/core/8039: mount failed: Unknown error -1
# LIBMOUNT_DEBUG=all LOOPDEV_DEBUG=all /bin/mount /var/lib/snapd/snaps/core_8039.snap /snap/core/8039 -t squashfs -o nodev,ro
1223: libmount: INIT: library debug mask: 0xffff
1223: libmount: INIT: library version: 2.27.0
1223: libmount: INIT: feature: selinux
1223: libmount: INIT: feature: assert
1223: libmount: INIT: feature: debug
Available "LIBMOUNT_DEBUG=<name>[,...]|<mask>" debug masks:
all [0xffff] : info about all subsystems
cache [0x0004] : paths and tags cache
cxt [0x0200] : library context (handler)
diff [0x0400] : mountinfo changes tracking
fs [0x0040] : FS abstraction
help [0x0001] : this help
locks [0x0010] : mtab and utab locking
options [0x0008] : mount options parsing
tab [0x0020] : fstab, mtab, mounninfo routines
update [0x0080] : mtab, utab updates
utils [0x0100] : misc library utils
monitor [0x0800] : mount tables monitor
1223: libmount: CXT: [0xa3f050]: ----> allocate
1223: libmount: CXT: [0xa3f050]: mount: preparing
1223: libmount: CXT: [0xa3f050]: use default optsmode
1223: libmount: CXT: [0xa3f050]: OPTSMODE: ignore=0, append=0, prepend=1, replace=0, force=0, fstab=1, mtab=1
1223: libmount: CXT: [0xa3f050]: fstab not required -- skip
1223: libmount: CXT: [0xa3f050]: merging mount flags
1223: libmount: CXT: [0xa3f050]: final flags: VFS=00000005 user=00000000
1223: libmount: CXT: [0xa3f050]: mount: evaluating permissions
1223: libmount: CXT: [0xa3f050]: mount: fixing optstr
1223: libmount: CXT: [0xa3f050]: mount: fixing vfs optstr
1223: libmount: CXT: applying 0x00000005 flags to 'nodev,ro'
1223: libmount: CXT: new optstr 'ro,nodev'
1223: libmount: CXT: [0xa3f050]: mount: fixing user optstr
1223: libmount: CXT: applying 0x00000000 flags to '(null)'
1223: libmount: CXT: new optstr '(null)'
1223: libmount: CXT: [0xa3f050]: fixed options [rc=0]: vfs: 'ro,nodev' fs: '(null)' user: '(null)', optstr: 'ro,nodev'
1223: libmount: CXT: [0xa3f050]: preparing source path
1223: libmount: CXT: [0xa3f050]: srcpath '/var/lib/snapd/snaps/core_8039.snap'
1223: libmount: CACHE: [0xa3f2e0]: alloc
1223: libmount: CACHE: [0xa3f2e0]: canonicalize path /var/lib/snapd/snaps/core_8039.snap
1223: libmount: CACHE: [0xa3f2e0]: add entry [ 1] (path): /var/lib/snapd/snaps/core_8039.snap: /var/lib/snapd/snaps/core_8039.snap
1223: libmount: CXT: [0xa3f050]: automatically enabling loop= option
1223: libmount: CXT: [0xa3f050]: trying to setup loopdev for /var/lib/snapd/snaps/core_8039.snap
1223: libmount: CXT: [0xa3f050]: enabling READ-ONLY flag
1223: loopdev: CXT: [0x7ffd67546450]: initialize context
1223: loopdev: CXT: [0x7ffd67546450]: init: ignore ioctls
1223: libmount: TAB: [0xa40f40]: alloc
1223: libmount: TAB: [0xa40f40]: mtab parse: ignore /etc/mtab
1223: libmount: TAB: [0xa40f40]: mtab parse: #1 read mountinfo
1223: libmount: TAB: [0xa40f40]: /proc/self/mountinfo: start parsing [entries=0, filter=not]
1223: libmount: TAB: [0xa40f40]: add entry: default/containers/signal-desktop-20191114-082626 /
1223: libmount: CACHE: canonicalize path /proc/self/mountinfo
1223: libmount: TAB: TID for /proc/self/mountinfo is 1223
1223: libmount: TAB: [0xa40f40]: add entry: none /dev
1223: libmount: TAB: [0xa40f40]: add entry: proc /proc
1223: libmount: TAB: [0xa40f40]: add entry: sysfs /sys
1223: libmount: TAB: [0xa40f40]: add entry: udev /dev/fuse
1223: libmount: TAB: [0xa40f40]: add entry: udev /dev/net/tun
1223: libmount: TAB: [0xa40f40]: add entry: binfmt_misc /proc/sys/fs/binfmt_misc
1223: libmount: TAB: [0xa40f40]: add entry: fusectl /sys/fs/fuse/connections
1223: libmount: TAB: [0xa40f40]: add entry: pstore /sys/fs/pstore
1223: libmount: TAB: [0xa40f40]: add entry: debugfs /sys/kernel/debug
1223: libmount: TAB: [0xa40f40]: add entry: securityfs /sys/kernel/security
1223: libmount: TAB: [0xa40f40]: add entry: mqueue /dev/mqueue
1223: libmount: TAB: [0xa40f40]: add entry: tmpfs /dev/lxd
1223: libmount: TAB: [0xa40f40]: add entry: tmpfs /dev/.lxd-mounts
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /proc/cpuinfo
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /proc/diskstats
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /proc/loadavg
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /proc/meminfo
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /proc/stat
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /proc/swaps
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /proc/uptime
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /sys/devices/system/cpu/online
1223: libmount: TAB: [0xa40f40]: add entry: lxcfs /var/lib/lxcfs
1223: libmount: TAB: [0xa40f40]: add entry: udev /dev/full
1223: libmount: TAB: [0xa40f40]: add entry: udev /dev/null
1223: libmount: TAB: [0xa40f40]: add entry: udev /dev/random
1223: libmount: TAB: [0xa40f40]: add entry: udev /dev/tty
1223: libmount: TAB: [0xa40f40]: add entry: udev /dev/urandom
1223: libmount: TAB: [0xa40f40]: add entry: udev /dev/zero
1223: libmount: TAB: [0xa40f40]: add entry: devpts /dev/console
1223: libmount: TAB: [0xa40f40]: add entry: none /proc/sys/kernel/random/boot_id
1223: libmount: TAB: [0xa40f40]: add entry: devpts /dev/pts
1223: libmount: TAB: [0xa40f40]: add entry: devpts /dev/ptmx
1223: libmount: TAB: [0xa40f40]: add entry: tmpfs /dev/shm
1223: libmount: TAB: [0xa40f40]: add entry: tmpfs /run
1223: libmount: TAB: [0xa40f40]: add entry: tmpfs /run/lock
1223: libmount: TAB: [0xa40f40]: add entry: tmpfs /sys/fs/cgroup
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/systemd
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/net_cls,net_prio
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/pids
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/blkio
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/devices
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/hugetlb
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/memory
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/perf_event
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/freezer
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/rdma
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/cpuset
1223: libmount: TAB: [0xa40f40]: add entry: cgroup /sys/fs/cgroup/cpu,cpuacct
1223: libmount: TAB: [0xa40f40]: add entry: tmpfs /run/user/0
1223: libmount: FS: [0xa4ca70]: free [refcount=0]
1223: libmount: TAB: [0xa40f40]: /proc/self/mountinfo: stop parsing (50 entries)
1223: libmount: TAB: [0xa40f40]: parsing done [filename=/proc/self/mountinfo, rc=0]
1223: libmount: TAB: [0xa40f40]: mtab parse: #2 read utab
1223: libmount: CXT: [0xa3f050]: mtab requested [nents=50]
1223: libmount: CXT: [0xa3f050]: checking if /var/lib/snapd/snaps/core_8039.snap mounted on /snap/core/8039
1223: libmount: CACHE: [0xa3f2e0]: canonicalize path /snap/core/8039
1223: libmount: CACHE: [0xa3f2e0]: add entry [ 2] (path): /snap/core/8039: /snap/core/8039
1223: libmount: CXT: [0xa3f050]: enabling AUTOCLEAR flag
1223: loopdev: CXT: [0x7ffd67546450]: find_unused requested
1223: loopdev: ITER: [0x7ffd67546608]: initialize
1223: loopdev: ITER: [0x7ffd67546608]: next
1223: loopdev: ITER: [0x7ffd67546608]: next: default check
1223: loopdev: CXT: [0x7ffd67546450]: loop0 name assigned
1223: loopdev: ITER: [0x7ffd67546608]: /dev/loop0 does not exist
1223: loopdev: CXT: [0x7ffd67546450]: loop1 name assigned
1223: loopdev: ITER: [0x7ffd67546608]: /dev/loop1 does not exist
1223: loopdev: CXT: [0x7ffd67546450]: loop2 name assigned
1223: loopdev: ITER: [0x7ffd67546608]: /dev/loop2 does not exist
1223: loopdev: CXT: [0x7ffd67546450]: loop3 name assigned
1223: loopdev: ITER: [0x7ffd67546608]: /dev/loop3 does not exist
1223: loopdev: CXT: [0x7ffd67546450]: loop4 name assigned
1223: loopdev: ITER: [0x7ffd67546608]: /dev/loop4 does not exist
1223: loopdev: CXT: [0x7ffd67546450]: loop5 name assigned
1223: loopdev: ITER: [0x7ffd67546608]: /dev/loop5 does not exist
1223: loopdev: CXT: [0x7ffd67546450]: loop6 name assigned
1223: loopdev: ITER: [0x7ffd67546608]: /dev/loop6 does not exist
1223: loopdev: CXT: [0x7ffd67546450]: loop7 name assigned
1223: loopdev: ITER: [0x7ffd67546608]: /dev/loop7 does not exist
1223: loopdev: ITER: [0x7ffd67546608]: next: scanning /dev
1223: loopdev: ITER: scan dir: /dev/
1223: loopdev: ITER: [0x7ffd67546608]: de-initialize
1223: loopdev: ITER: [0x7ffd67546608]: de-initialize
1223: loopdev: CXT: [0x7ffd67546450]: find_unused by scan [rc=1]
1223: loopdev: CXT: [0x7ffd67546450]: de-initialize
1223: loopdev: ITER: [0x7ffd67546608]: de-initialize
1223: libmount: CXT: [0xa3f050]: mount: preparing failed
mount: /snap/core/8039: mount failed: Unknown error -1
1223: libmount: CXT: [0xa3f050]: <---- reset [status=0] ---->
1223: libmount: FS: [0xa3f150]: free [refcount=0]
1223: libmount: TAB: [0xa40f40]: reset
1223: libmount: FS: [0xa411d0]: free [refcount=0]
1223: libmount: FS: [0xa41ab0]: free [refcount=0]
1223: libmount: FS: [0xa41dc0]: free [refcount=0]
1223: libmount: FS: [0xa42140]: free [refcount=0]
1223: libmount: FS: [0xa424c0]: free [refcount=0]
1223: libmount: FS: [0xa42840]: free [refcount=0]
1223: libmount: FS: [0xa42bc0]: free [refcount=0]
1223: libmount: FS: [0xa42f40]: free [refcount=0]
1223: libmount: FS: [0xa432c0]: free [refcount=0]
1223: libmount: FS: [0xa43640]: free [refcount=0]
1223: libmount: FS: [0xa439f0]: free [refcount=0]
1223: libmount: FS: [0xa43dc0]: free [refcount=0]
1223: libmount: FS: [0xa44190]: free [refcount=0]
1223: libmount: FS: [0xa44540]: free [refcount=0]
1223: libmount: FS: [0xa44910]: free [refcount=0]
1223: libmount: FS: [0xa44cb0]: free [refcount=0]
1223: libmount: FS: [0xa45050]: free [refcount=0]
1223: libmount: FS: [0xa453f0]: free [refcount=0]
1223: libmount: FS: [0xa45790]: free [refcount=0]
1223: libmount: FS: [0xa45b30]: free [refcount=0]
1223: libmount: FS: [0xa45ed0]: free [refcount=0]
1223: libmount: FS: [0xa46270]: free [refcount=0]
1223: libmount: FS: [0xa46610]: free [refcount=0]
1223: libmount: FS: [0xa469b0]: free [refcount=0]
1223: libmount: FS: [0xa46d50]: free [refcount=0]
1223: libmount: FS: [0xa470f0]: free [refcount=0]
1223: libmount: FS: [0xa47490]: free [refcount=0]
1223: libmount: FS: [0xa47830]: free [refcount=0]
1223: libmount: FS: [0xa47bd0]: free [refcount=0]
1223: libmount: FS: [0xa47f70]: free [refcount=0]
1223: libmount: FS: [0xa482f0]: free [refcount=0]
1223: libmount: FS: [0xa48670]: free [refcount=0]
1223: libmount: FS: [0xa48a50]: free [refcount=0]
1223: libmount: FS: [0xa48e30]: free [refcount=0]
1223: libmount: FS: [0xa491b0]: free [refcount=0]
1223: libmount: FS: [0xa49530]: free [refcount=0]
1223: libmount: FS: [0xa49910]: free [refcount=0]
1223: libmount: FS: [0xa49c90]: free [refcount=0]
1223: libmount: FS: [0xa4a010]: free [refcount=0]
1223: libmount: FS: [0xa4a390]: free [refcount=0]
1223: libmount: FS: [0xa4a710]: free [refcount=0]
1223: libmount: FS: [0xa4aa90]: free [refcount=0]
1223: libmount: FS: [0xa4ae10]: free [refcount=0]
1223: libmount: FS: [0xa4b190]: free [refcount=0]
1223: libmount: FS: [0xa4b510]: free [refcount=0]
1223: libmount: FS: [0xa4b890]: free [refcount=0]
1223: libmount: FS: [0xa4bc10]: free [refcount=0]
1223: libmount: FS: [0xa4bf90]: free [refcount=0]
1223: libmount: FS: [0xa4c310]: free [refcount=0]
1223: libmount: FS: [0xa4c690]: free [refcount=0]
1223: libmount: TAB: [0xa40f40]: free [refcount=0]
1223: libmount: CACHE: [0xa3f2e0]: free [refcount=0]
1223: libmount: CXT: [0xa3f050]: <---- free
Thanks. This is actually correct, since we’re in container. If there was a persistent problem with mounting snaps, it would have been caught by sanity checks and snap installs would fail right away.
Can you check the following please:
ls -l /dev/fuseis /usr/bin/snapfuse present?
the output of systemd-detect-virt --container, should be lxc
You may also try to mount the snap using snapfuse like so:
mount /var/lib/snapd/snaps/core_8039.snap /snap/core/8039 -t fuse.snapfuse -o nodev,ro
popey
November 14, 2019, 10:58am
5
# ls -l /dev/fuse
crw-rw-rw- 1 nobody nogroup 10, 229 Nov 14 08:26 /dev/fuse
# ls /usr/bin/snapfuse
ls: cannot access '/usr/bin/snapfuse': No such file or directory
# systemd-detect-virt --container
lxc
# mount /var/lib/snapd/snaps/core_8039.snap /snap/core/8039 -t fuse.snapfuse -o nodev,ro
/bin/sh: 1: snapfuse: not found
Does squashfuse exist at least?
Btw. I’ve launched the container locally using the latest ubuntu:16.04 LXD image, whcih came with snapd 2.40.
popey
November 14, 2019, 11:34am
7
So it looks like my lxc container is outdated, but I don’t know how this worked last week but no longer does.
popey
November 14, 2019, 11:55am
8
I ended up nuking my lxd install and reset it, because I was in a hurry, sorry. So can’t debug further.
