Mounts broken with kernel 5.2

The Linux kernel in Arch got recently updated to 5.2.0 (5.2.0-arch2-1-ARCH in detail). Today I noticed some unexcpected breakage after a clean boot, specifically some snaps are listed as broken:

Name          Version      Rev    Tracking        Publisher        Notes
...
gnome-logs                 61     stable          canonical✓       broken
...
http                       25     stable          chipaca          broken
icdiff        1.9.4-1      x1     -               -                -
love-brlin                 27     stable          brlin            broken
...
mutt                       125    stable          snapcrafters     broken
...

Further investigation shows the mount units failed:

$ systemctl --all --failed |grep var-lib-snapd
● var-lib-snapd-snap-gnome\x2d3\x2d26\x2d1604-88.mount loaded failed failed Mount unit for gnome-3-26-1604, revision 88  
● var-lib-snapd-snap-gnome\x2dcalculator-352.mount     loaded failed failed Mount unit for gnome-calculator, revision 352
● var-lib-snapd-snap-hello\x2dworld-27.mount           loaded failed failed Mount unit for hello-world, revision 27      
● var-lib-snapd-snap-http-25.mount                     loaded failed failed Mount unit for http, revision 25             
● var-lib-snapd-snap-love\x2dbrlin-27.mount            loaded failed failed Mount unit for love-brlin, revision 27       
● var-lib-snapd-snap-mutt-125.mount                    loaded failed failed Mount unit for mutt, revision 125  

Investigating the units:

$ systemctl status var-lib-snapd-snap-mutt-125.mount
● var-lib-snapd-snap-mutt-125.mount - Mount unit for mutt, revision 125
   Loaded: loaded (/etc/systemd/system/var-lib-snapd-snap-mutt-125.mount; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2019-07-12 14:18:58 CEST; 2min 28s ago
    Where: /var/lib/snapd/snap/mutt/125
     What: /var/lib/snapd/snaps/mutt_125.snap

lip 12 14:18:56 corsair systemd[1]: Mounting Mount unit for mutt, revision 125...
lip 12 14:18:57 corsair mount[718]: mount: /var/lib/snapd/snap/mutt/125: /dev/loop26 already mounted or mount point busy.
lip 12 14:18:58 corsair systemd[1]: var-lib-snapd-snap-mutt-125.mount: Mount process exited, code=exited, status=32/n/a
lip 12 14:18:58 corsair systemd[1]: var-lib-snapd-snap-mutt-125.mount: Failed with result 'exit-code'.
lip 12 14:18:58 corsair systemd[1]: Failed to mount Mount unit for mutt, revision 125.
$ systemctl status var-lib-snapd-snap-http-25.mount
● var-lib-snapd-snap-http-25.mount - Mount unit for http, revision 25
   Loaded: loaded (/etc/systemd/system/var-lib-snapd-snap-http-25.mount; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2019-07-12 14:18:57 CEST; 2min 37s ago
    Where: /var/lib/snapd/snap/http/25
     What: /var/lib/snapd/snaps/http_25.snap

lip 12 14:18:56 corsair systemd[1]: Mounting Mount unit for http, revision 25...
lip 12 14:18:57 corsair mount[659]: mount: /var/lib/snapd/snap/http/25: /dev/loop15 already mounted or mount point busy.
lip 12 14:18:57 corsair systemd[1]: var-lib-snapd-snap-http-25.mount: Mount process exited, code=exited, status=32/n/a
lip 12 14:18:57 corsair systemd[1]: var-lib-snapd-snap-http-25.mount: Failed with result 'exit-code'.
lip 12 14:18:57 corsair systemd[1]: Failed to mount Mount unit for http, revision 25.

So let’s see what /dev/loop15 and /dev/loop26 are:

$ losetup|grep -e loop15 -e loop26
/dev/loop15         0      0         1  1 /var/lib/snapd/snaps/gnome-3-28-1804_59.snap                 0     512
/dev/loop26         0      0         1  1 /var/lib/snapd/snaps/mutt_115.snap                           0     512

This is somewhat unexpected. Status log shows that http snap, rev 25 was supposed to be mounted with /dev/loop26, but that is assigned to the gnome-3-28-1804 runtime snap. Clearly something went wrong with loopback device allocation. Perhaps a regression in 5.2.0 or systemd.

1 Like

I can try the mainline kernel on my daily Ubuntu laptop to see what I get.

1 Like

Note this (or something that looks like this) is biting eoan as well, now that it’s on 5.2:

Here’s a reproducer for the bug, no snapd needed:

#!/bin/bash

if [ "$EUID" != 0 ]; then
    echo "Run this as root please." >&2
    exit 1
fi

set -e

echo Preparing...
for i in {a..z}{a..z}; do
    echo $i > canary.txt
    mksquashfs canary.txt $i.squash -no-progress >/dev/null
    mkdir /mnt/$i
done

echo Running:
for i in {a..z}{a..z}; do
    mount $i.squash /mnt/$i &
done

wait

This is now reported as lp:1836914.

1 Like

I have the same problem, but on kernel 5.2 and systemd 2.40.
If I go back to the 5.0 kernel, everything works again.

apt list systemd
Elencazione… Fatto
systemd/eoan,now 240-6ubuntu9 amd64 [installato, automatico]

snap version
snap 2.39.3
snapd 2.39.3
series 16
ubuntu 19.10
kernel 5.0.0-15-generic

FWIW,

I cannot launch any snap anymore on Ubuntu Eoan with kernel 5.2

$ snap version
snap    2.39.2+19.10ubuntu1
snapd   2.39.2+19.10ubuntu1
series  16
ubuntu  19.10
kernel  5.2.0-8-generic

core18 has been auto-refreshed yesterday @10:26PM

Everything works fine with kernel 5.0

For example:

$ spotify 
internal error, please report: running "spotify" failed: cannot find installed snap "spotify" at revision 35: missing file /snap/spotify/35/meta/snap.yaml
 
$ code
/snap/code/11/electron-launch: line 28: /snap/code/11/usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/gdk-pixbuf-query-loaders: No such file or directory
/snap/code/11/usr/share/code/bin/code: line 44: /snap/code/11/usr/share/code/bin/../code: No such file or directory

$ /snap/bin/google-chrome 
/snap/google-chrome/x1/bin/desktop-launch: line 29: /home/j-lallement/.config/user-dirs.dirs: Permission denied
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
Trace/breakpoint trap (core dumped)

$ sweethome3d-homedesign 
internal error, please report: running "sweethome3d-homedesign" failed: cannot find installed snap "sweethome3d-homedesign" at revision 9: missing file /snap/sweethome3d-homedesign/9/meta/snap.yaml

$ /snap/bin/hello
cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_neFJcx//dev: No such file or directory

What’s the output of journalctl -u snapd? What’s the output of snap list? What’s the output of snap debug sandbox-features?

$ cat /tmp/snap.list 
Name                    Version                     Rev    Tracking  Publisher    Notes
code                    2213894e                    11     stable    vscode*      classic
core                                                7270   stable    canonical*   broken
core18                  20190709                    1066   stable    canonical*   base
firefox                                             242    beta      mozilla*     broken
freemind                1.1.0-Beta-2                4      edge      jibel        -
freeplane-mindmapping   1.7.9                       14     edge      jibel        -
gnome-3-26-1604         3.26.0.20190705             90     stable    canonical*   -
gnome-3-28-1804         3.28.0-10-gaa70833.aa70833  67     stable    canonical*   -
google-chrome           75.0.3770.100-1             x1     -         -            -
gtk-common-themes       0.1-22-gab0a26b             1313   stable    canonical*   -
hello                   2.10                        38     stable    canonical*   -
lxd                     3.14                        11098  stable/…  canonical*   -
multipass                                           908    beta      canonical*   broken
pycharm-community       2019.1.3                    132    stable    jetbrains*   classic
skype                   8.34.0.78                   66     stable    skype*       classic
snapcraft               3.6                         3059   stable    canonical*   classic
spotify                                             35     stable    spotify*     broken
squirrelsql             snapshot-20190714-0016      44     edge      jibel        -
sweethome3d-homedesign                              9      edge      jibel        broken
transporter             0+git.2c1573c               75     stable    ken-vandine  -
vlc                     3.0.7                       1049   stable    videolan*    -

journalctl -u

juil. 17 11:20:41 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "core" at revision 7270: cannot find installed snap "core" at revision 7270: missing file /snap/core/7270/meta/snap.yaml
[...]
juil. 17 11:20:41 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "core" at revision 7270: cannot find installed snap "core" at revision 7270: missing file /snap/core/7270/meta/snap.yaml
juil. 17 11:21:47 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "spotify" at revision 35: cannot find installed snap "spotify" at revision 35: missing file /snap/spotify/35/meta/snap.yaml
juil. 17 11:21:47 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "core" at revision 7270: cannot find installed snap "core" at revision 7270: missing file /snap/core/7270/meta/snap.yaml
juil. 17 11:21:47 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "sweethome3d-homedesign" at revision 9: cannot find installed snap "sweethome3d-homedesign" at revision 9: missing file /snap/sweethome3d-homedesign/9/meta/snap.yaml
juil. 17 11:21:47 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "multipass" at revision 908: cannot find installed snap "multipass" at revision 908: missing file /snap/multipass/908/meta/snap.yaml
juil. 17 11:21:47 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "firefox" at revision 242: cannot find installed snap "firefox" at revision 242: missing file /snap/firefox/242/meta/snap.yaml
juil. 17 11:22:07 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "spotify" at revision 35: cannot find installed snap "spotify" at revision 35: missing file /snap/spotify/35/meta/snap.yaml
juil. 17 11:22:08 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "core" at revision 7270: cannot find installed snap "core" at revision 7270: missing file /snap/core/7270/meta/snap.yaml
juil. 17 11:22:08 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "sweethome3d-homedesign" at revision 9: cannot find installed snap "sweethome3d-homedesign" at revision 9: missing file /snap/sweethome3d-homedesign/9/meta/snap.yaml
juil. 17 11:22:08 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "multipass" at revision 908: cannot find installed snap "multipass" at revision 908: missing file /snap/multipass/908/meta/snap.yaml
juil. 17 11:22:09 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "firefox" at revision 242: cannot find installed snap "firefox" at revision 242: missing file /snap/firefox/242/meta/snap.yaml
juil. 17 11:25:46 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "core" at revision 7270: cannot find installed snap "core" at revision 7270: missing file /snap/core/7270/meta/snap.yaml
juil. 17 11:55:44 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "core" at revision 7270: cannot find installed snap "core" at revision 7270: missing file /snap/core/7270/meta/snap.yaml
juil. 17 11:55:44 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "sweethome3d-homedesign" at revision 9: cannot find installed snap "sweethome3d-homedesign" at revision 9: missing file /snap/sweethome3d-homedesign/9/meta/snap.yaml
juil. 17 11:55:44 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "spotify" at revision 35: cannot find installed snap "spotify" at revision 35: missing file /snap/spotify/35/meta/snap.yaml
juil. 17 11:55:44 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "firefox" at revision 242: cannot find installed snap "firefox" at revision 242: missing file /snap/firefox/242/meta/snap.yaml
juil. 17 11:55:44 sark snapd[1553]: snapmgr.go:258: cannot read snap info of snap "multipass" at revision 908: cannot find installed snap "multipass" at revision 908: missing file /snap/multipass/908/meta/snap.yaml

snap debug sandbox features

apparmor:             kernel:caps kernel:dbus kernel:domain kernel:file kernel:mount kernel:namespaces kernel:network kernel:network_v8 kernel:policy kernel:ptrace kernel:query kernel:rlimit kernel:signal parser:unsafe policy:default support-level:full
confinement-options:  classic devmode strict
dbus:                 mediated-bus-access
kmod:                 mediated-modprobe
mount:                freezer-cgroup-v1 layouts mount-namespace per-snap-persistency per-snap-profiles per-snap-updates per-snap-user-profiles stale-base-invalidation
seccomp:              bpf-argument-filtering kernel:allow kernel:errno kernel:kill_process kernel:kill_thread kernel:log kernel:trace kernel:trap kernel:user_notif
udev:                 device-cgroup-v1 tagging

Thank you. Was there an updated systemd over your reboot? (can you share the systemd version now, and before the reboot please?)

FTR this looks suspiciously like

but the systemd version here is 240-6ubuntu9.

from IRC, here’s @jbl’s failed mounts:

● snap-core-7270.mount                                               loaded failed failed Mount unit for core, revision 7270                
● snap-firefox-242.mount                                             loaded failed failed Mount unit for firefox, revision 242              
● snap-gtk\x2dcommon\x2dthemes-1198.mount                            loaded failed failed Mount unit for gtk-common-themes, revision 1198   
● snap-multipass-908.mount                                           loaded failed failed Mount unit for multipass, revision 908            
● snap-spotify-35.mount                                              loaded failed failed Mount unit for spotify, revision 35               
● snap-sweethome3d\x2dhomedesign-7.mount                             loaded failed failed Mount unit for sweethome3d-homedesign, revision 7 
● snap-sweethome3d\x2dhomedesign-9.mount                             loaded failed failed Mount unit for sweethome3d-homedesign, revision 9 
● snaps-cap\x2dtest.mvo-2.2b1\x2d3.16.04.5.mount                     loaded failed failed Squashfs mount unit for cap-test                  
● snaps-ubuntu\x2dcalculator\x2dapp.ubuntucoredev-2.1\x2bsnap0.mount loaded failed failed Squashfs mount unit for ubuntu-calculator-app     
● snaps-ubuntu\x2dcalculator\x2dapp.ubuntucoredev-2.1\x2bsnap2.mount loaded failed failed Squashfs mount unit for ubuntu-calculator-app     
● snaps-ubuntu\x2dclock\x2dapp.ubuntucoredev-3.6\x2bsnap2.mount      loaded failed failed Squashfs mount unit for ubuntu-clock-app          
● snaps-ubuntu\x2dcore.canonical-16.04.0\x2d25.mount                 loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160321.17\x2d27.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160405.11\x2d54.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160409.06\x2d32.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160409.16\x2d42.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160409.18\x2d05.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160409.20\x2d34.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160409.21\x2d21.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160409.21\x2d59.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2b20160410.15\x2d09.mount    loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2d20160321\x2d05\x2d01.mount loaded failed failed Squashfs mount unit for ubuntu-core               
● snaps-ubuntu\x2dcore.canonical-16.04\x2d20160321\x2d14\x2d15.mount loaded failed failed Squashfs mount unit for ubuntu-core               
● snap.lxd.activate.service                                          loaded failed failed Service for snap application lxd.activate

This is the same bug than Arch.

For example vscode attempts to mount the snap on loop21:

$ systemctl status snap-code-10.mount
● snap-code-10.mount - Mount unit for code, revision 10
   Loaded: loaded (/etc/systemd/system/snap-code-10.mount; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-07-17 12:20:38 CEST; 42min ago
    Where: /snap/code/10
     What: /var/lib/snapd/snaps/code_10.snap

juil. 17 12:20:36 sark systemd[1]: Mounting Mount unit for code, revision 10...
juil. 17 12:20:38 sark mount[1094]: mount: /snap/code/10: /dev/loop21 already mounted or mount point busy.

But it’s already assigned to another snap

$ losetup /dev/loop21 
/dev/loop21: []: (/var/lib/snapd/snaps/freemind_3.snap)

This is also reported in launchpad as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1836914

I merged two topics together, but the ordering got a bit messy.

The current status is that it’s understood that linux 5.2 has an issue, and it’s being worked on.
More here:


HTH

1 Like

On Tue, 30 Jul 2019 at 20:17, Jens Axboe axboe@kernel.dk wrote:

Jan, I’ve applied this patch - and also marked it for stable, so it’ll
end up in 5.2-stable. Thanks.

http://i.imgur.com/X9Ob9wJ.gifv

1 Like

So I can upgrade to Eoan now? :-p What do you mean the eoan distro is still pre-release and the kernel patch has only just this second been applied?!