All snaps showing as ‘broken’ (ubuntu 20.04 + zfs)

ghostbuster91 · September 12, 2020, 5:40am

Hi, I’ve been using snap for at least over a year now without any problems and it was great.

Recently I decided to reinstall my system to ubuntu 20.04 but with ZFS as a file system. Since then I have weird problems with snap. After a restart I cannot launch any of my applications installed via snap. They also appear as broken in snap list output.

snap list                
Name                    Version  Rev   Tracking         Publisher   Notes
canonical-livepatch              95    latest/stable    canonical✓  broken
core                             9804  latest/stable    canonical✓  broken
core18                           1885  latest/stable    canonical✓  broken
gnome-3-34-1804                  36    latest/stable/…  canonical✓  broken
gtk-common-themes                1506  latest/stable/…  canonical✓  broken
intellij-idea-ultimate           245   latest/stable    jetbrains✓  broken
skype                            148   latest/stable    skype✓      broken
snap-store                       467   latest/stable/…  canonical✓  broken
snapd                            8790  latest/stable    canonical✓  broken

–

lla /var/lib/snapd/snaps
.rw------- 9,5M root  7 wrz  1:25 canonical-livepatch_95.snap
.rw-------  57M root 31 lip 18:35 core18_1880.snap
.rw-------  58M root  7 wrz  9:13 core18_1885.snap
.rw------- 101M root  7 wrz  1:25 core_9804.snap
.rw------- 267M root 31 lip 18:35 gnome-3-34-1804_36.snap
.rw-------  65M root 31 lip 18:35 gtk-common-themes_1506.snap
.rw------- 755M root  7 wrz 11:09 intellij-idea-ultimate_245.snap
drwxr-xr-x    - root 10 lip 15:59 partial
.rw------- 188M root  9 wrz 22:45 skype_147.snap
.rw------- 188M root 11 wrz 18:56 skype_148.snap
.rw-------  52M root 31 lip 18:35 snap-store_467.snap
.rw-------  31M root 31 lip 18:35 snapd_8542.snap
.rw-------  31M root  7 wrz  9:13 snapd_8790.snap

–

service snapd status 
● snapd.service - Snap Daemon
     Loaded: loaded (/lib/systemd/system/snapd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-09-12 07:27:56 CEST; 9min ago
TriggeredBy: ● snapd.socket
   Main PID: 32026 (snapd)
      Tasks: 22 (limit: 38174)
     Memory: 20.8M
     CGroup: /system.slice/snapd.service
             └─32026 /usr/lib/snapd/snapd

wrz 12 07:27:57 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "core" at revision 9804: cannot find installed snap "core" at revision 9804: missing file /snap/core/9804/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "snapd" at revision 8790: cannot find installed snap "snapd" at revision 8790: missing file /snap/snapd/8790/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "canonical-livepatch" at revision 95: cannot find installed snap "canonical-livepatch" at revision 95: missing file /snap/canonical-livepatch/95/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "skype" at revision 148: cannot find installed snap "skype" at revision 148: missing file /snap/skype/148/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "snap-store" at revision 467: cannot find installed snap "snap-store" at revision 467: missing file /snap/snap-store/467/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "core" at revision 9804: cannot find installed snap "core" at revision 9804: missing file /snap/core/9804/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "gnome-3-34-1804" at revision 36: cannot find installed snap "gnome-3-34-1804" at revision 36: missing file /snap/gnome-3-34-1804/36/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "gtk-common-themes" at revision 1506: cannot find installed snap "gtk-common-themes" at revision 1506: missing file /snap/gtk-common-themes/1506/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "core18" at revision 1885: cannot find installed snap "core18" at revision 1885: missing file /snap/core18/1885/meta/snap.yaml
wrz 12 07:29:25 precision-7530 snapd[32026]: snapmgr.go:298: cannot read snap info of snap "intellij-idea-ultimate" at revision 245: cannot find installed snap "intellij-idea-ultimate" at revision 245: missing file /snap/intellij-idea-ultimate/245/meta/snap.yaml

–

mount | grep snap
rpool/ROOT/ubuntu_i8m3zy/var/snap on /var/snap type zfs (rw,noatime,xattr,posixacl)

–

zfs list  
NAME                                               USED  AVAIL     REFER  MOUNTPOINT
bpool                                              352M  1,41G       96K  /boot
bpool/BOOT                                         350M  1,41G       96K  none
bpool/BOOT/ubuntu_i8m3zy                           350M  1,41G      182M  /boot
rpool                                             66,9G   390G      192K  /
rpool/ROOT                                        9,23G   390G      192K  none
rpool/ROOT/ubuntu_i8m3zy                          9,23G   390G     3,73G  /
rpool/ROOT/ubuntu_i8m3zy/srv                       192K   390G      192K  /srv
rpool/ROOT/ubuntu_i8m3zy/usr                      1,89M   390G      192K  /usr
rpool/ROOT/ubuntu_i8m3zy/usr/local                1,70M   390G     1,37M  /usr/local
rpool/ROOT/ubuntu_i8m3zy/var                      4,12G   390G      192K  /var
rpool/ROOT/ubuntu_i8m3zy/var/games                 192K   390G      192K  /var/games
rpool/ROOT/ubuntu_i8m3zy/var/lib                  3,88G   390G     2,75G  /var/lib
rpool/ROOT/ubuntu_i8m3zy/var/lib/AccountsService  1,56M   390G      280K  /var/lib/AccountsService
rpool/ROOT/ubuntu_i8m3zy/var/lib/NetworkManager   4,19M   390G      360K  /var/lib/NetworkManager
rpool/ROOT/ubuntu_i8m3zy/var/lib/apt              78,5M   390G     65,3M  /var/lib/apt
rpool/ROOT/ubuntu_i8m3zy/var/lib/dpkg              107M   390G     51,3M  /var/lib/dpkg
rpool/ROOT/ubuntu_i8m3zy/var/log                   247M   390G     26,9M  /var/log
rpool/ROOT/ubuntu_i8m3zy/var/mail                  192K   390G      192K  /var/mail
rpool/ROOT/ubuntu_i8m3zy/var/snap                 2,14M   390G      728K  /var/snap
rpool/ROOT/ubuntu_i8m3zy/var/spool                1,34M   390G      288K  /var/spool
rpool/ROOT/ubuntu_i8m3zy/var/www                   192K   390G      192K  /var/www
rpool/USERDATA                                    15,1G   390G      192K  /
rpool/USERDATA/kghost_c729ro                      15,1G   390G     9,72G  /home/kghost
rpool/USERDATA/root_c729ro                         700K   390G      244K  /root
rpool/swap2                                       42,5G   433G      116K  -
rpool/var                                          988K   390G      192K  /var
rpool/var/lib                                      796K   390G      192K  /var/lib
rpool/var/lib/docker                               604K   390G      604K  /var/lib/docker

I have found similar problems but they were quite old ~2018/2019 and mostly affecting arch linux users.

ghostbuster91 · September 17, 2020, 8:40am

Nobody? …

ijohnson · September 17, 2020, 3:34pm

Hi, can you provide full system journal for snapd with:

journalctl --no-pager -u snapd

and also

snap version

ghostbuster91 · September 18, 2020, 9:16am

Hi @ijohnson ,

Sure, here you are:

$ snap version
snap    2.45.1+20.04.2
snapd   2.45.1+20.04.2
series  16
ubuntu  20.04
kernel  5.4.0-47-generic

$ journalctl --no-pager -u snapd

https://pastebin.pl/view/c56689a3

Thanks for looking into it, I really appreciate that.

ijohnson · September 18, 2020, 11:32am

What’s the output of

ls -lah /snap/snapd/

Also, try running

snap install snapd || snap refresh snapd --candidate

ghostbuster91 · September 18, 2020, 11:55am

ls -lah /snap/snapd/
Permissions Size User Date Modified Name
drwxr-xr-x     - root  7 wrz  9:14  8790
drwxr-xr-x     - root 15 wrz  1:03  9279
lrwxrwxrwx     4 root 15 wrz  1:03  current -> 9279

$ snap install snapd || snap refresh snapd --candidate

snap "snapd" is already installed, see 'snap help refresh'

The same thing happens if I try to install any snap which was previously installed. Uninstalling and installing it again fixes the problem but that’s not a solution as after restart the problem comes back.

ijohnson · September 18, 2020, 12:25pm

After a restart when the problem comes back, what is cat /proc/self/mountinfo?

ghostbuster91 · September 18, 2020, 12:29pm

$ cat /proc/self/mountinfo 
25 32 0:23 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw
26 32 0:5 / /proc rw,nosuid,nodev,noexec,relatime shared:15 - proc proc rw
27 32 0:6 / /dev rw,nosuid,noexec,relatime shared:2 - devtmpfs udev rw,size=16287732k,nr_inodes=4071933,mode=755
28 27 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
29 32 0:25 / /run rw,nosuid,nodev,noexec,relatime shared:5 - tmpfs tmpfs rw,size=3266480k,mode=755
32 1 0:28 / / rw,relatime shared:1 - zfs rpool/ROOT/ubuntu_i8m3zy rw,xattr,posixacl
33 25 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
34 27 0:29 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw
35 29 0:30 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k
36 25 0:31 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755
37 36 0:32 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:10 - cgroup2 cgroup2 rw,nsdelegate
38 36 0:33 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,xattr,name=systemd
39 25 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:12 - pstore pstore rw
40 25 0:35 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:13 - efivarfs efivarfs rw
41 25 0:36 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:14 - bpf none rw,mode=700
42 36 0:37 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,rdma
43 36 0:38 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,cpuset
44 36 0:39 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,freezer
45 36 0:40 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,blkio
46 36 0:41 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,devices
47 36 0:42 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:21 - cgroup cgroup rw,hugetlb
48 36 0:43 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:22 - cgroup cgroup rw,net_cls,net_prio
49 36 0:44 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:23 - cgroup cgroup rw,perf_event
50 36 0:45 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:24 - cgroup cgroup rw,cpu,cpuacct
51 36 0:46 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:25 - cgroup cgroup rw,pids
52 36 0:47 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:26 - cgroup cgroup rw,memory
53 26 0:48 / /proc/sys/fs/binfmt_misc rw,relatime shared:27 - autofs systemd-1 rw,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=22325
54 27 0:21 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:28 - mqueue mqueue rw
55 27 0:49 / /dev/hugepages rw,relatime shared:29 - hugetlbfs hugetlbfs rw,pagesize=2M
56 25 0:8 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:30 - debugfs debugfs rw
57 25 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:31 - tracefs tracefs rw
58 25 0:50 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:32 - fusectl fusectl rw
59 25 0:22 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:33 - configfs configfs rw
129 32 0:51 / /boot rw,nodev,relatime shared:69 - zfs bpool/BOOT/ubuntu_i8m3zy rw,xattr,posixacl
132 32 0:52 / /root rw,noatime shared:71 - zfs rpool/USERDATA/root_c729ro rw,xattr,posixacl
135 32 0:55 / /srv rw,noatime shared:73 - zfs rpool/ROOT/ubuntu_i8m3zy/srv rw,xattr,posixacl
138 32 0:53 / /home/kghost rw,noatime shared:75 - zfs rpool/USERDATA/kghost_c729ro rw,xattr,posixacl
141 32 0:54 / /usr/local rw,noatime shared:77 - zfs rpool/ROOT/ubuntu_i8m3zy/usr/local rw,xattr,posixacl
144 32 0:56 / /var/lib rw,noatime shared:79 - zfs rpool/ROOT/ubuntu_i8m3zy/var/lib rw,xattr,posixacl
147 32 0:58 / /var/snap rw,noatime shared:81 - zfs rpool/ROOT/ubuntu_i8m3zy/var/snap rw,xattr,posixacl
150 32 0:60 / /var/mail rw,noatime shared:83 - zfs rpool/ROOT/ubuntu_i8m3zy/var/mail rw,xattr,posixacl
153 32 0:57 / /var/games rw,noatime shared:85 - zfs rpool/ROOT/ubuntu_i8m3zy/var/games rw,xattr,posixacl
156 129 259:1 / /boot/efi rw,relatime shared:87 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
159 32 0:59 / /var/spool rw,noatime shared:89 - zfs rpool/ROOT/ubuntu_i8m3zy/var/spool rw,xattr,posixacl
162 32 0:61 / /var/www rw,noatime shared:91 - zfs rpool/ROOT/ubuntu_i8m3zy/var/www rw,xattr,posixacl
165 129 259:1 /grub /boot/grub rw,relatime shared:87 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
168 144 0:63 / /var/lib/apt rw,noatime shared:94 - zfs rpool/ROOT/ubuntu_i8m3zy/var/lib/apt rw,xattr,posixacl
171 144 0:64 / /var/lib/AccountsService rw,noatime shared:96 - zfs rpool/ROOT/ubuntu_i8m3zy/var/lib/AccountsService rw,xattr,posixacl
174 144 0:65 / /var/lib/docker rw,noatime shared:98 - zfs rpool/var/lib/docker rw,xattr,posixacl
177 144 0:66 / /var/lib/dpkg rw,noatime shared:100 - zfs rpool/ROOT/ubuntu_i8m3zy/var/lib/dpkg rw,xattr,posixacl
180 144 0:67 / /var/lib/NetworkManager rw,noatime shared:102 - zfs rpool/ROOT/ubuntu_i8m3zy/var/lib/NetworkManager rw,xattr,posixacl
183 32 0:62 / /var/log rw,noatime shared:104 - zfs rpool/ROOT/ubuntu_i8m3zy/var/log rw,xattr,posixacl
1860 29 0:74 / /run/user/1000 rw,nosuid,nodev,relatime shared:1008 - tmpfs tmpfs rw,size=3266480k,mode=700,uid=1000,gid=1000
1892 1860 0:75 / /run/user/1000/gvfs rw,nosuid,nodev,relatime shared:1025 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
652 1860 0:70 / /run/user/1000/doc rw,nosuid,nodev,relatime shared:471 - fuse /dev/fuse rw,user_id=1000,group_id=1000

ijohnson · September 18, 2020, 12:43pm

Interesting, somehow none of the snap squashfs files are mounted, what is the output of

systemctl status snap-*.mount

To try fixing it, you could manually mount the snaps, try for example:

sudo mount /var/lib/snapd/snaps/snapd_9279.snap /snap/snapd/9279

but obviously this should happen automatically through the systemd mount unit which for some reason didn’t run properly

ghostbuster91 · September 18, 2020, 1:03pm

Yes, you are right. Also restarting these ‘mounting’ services helps

$ systemctl restart snap-intellij\\x2didea\\x2dultimate-248.mount

Any idea how can we debug it further?

ijohnson · September 18, 2020, 3:02pm

So after a fresh reboot, what is the status of these mount units with the systemctl status command I posted before? It seems somehow either they are not enabled or otherwise failing when they initially run

ghostbuster91 · September 21, 2020, 12:24pm

Hmm that’s weird. So, I checked the status before restarting and all of them were turned off with some indication that they were ‘dead’ whatever that means. Then, I restarted one of them (the one I mentioned) and it fixed IntellijIdea. After that I restarted PC to collect the information you asked about, but to my surprise all of them were mounted correctly. I dunno why, but in the meantime I ran apt upgrade, so maybe that has somehow fixed the problem.

Thank you for the help(I learned couple of useful command) but for now it seems that the problem magically disappeared.

ijohnson · September 21, 2020, 1:55pm

Hmm, do you have any apt logs of the upgrade of the snapd package? Perhaps something there went awry

ghostbuster91 · September 21, 2020, 2:11pm

Probably but tbh I am not sure how to collect them easily. I suppose that some track of it should be in the journalctl.

ijohnson · September 21, 2020, 11:58pm

You could look at /var/log/apt/term.log as well as /var/log/apt/history.log, they may have relevant information on the snapd package upgrade

ghostbuster91 · September 22, 2020, 7:23pm

I was just going to do, but I forgot that I had reinstalled my system yesterday because of another issue, so it won’t be much of a help

fencingparry4 · December 17, 2020, 2:27am

This may be separate from the issue OP was having, but I had a nearly identical problem when updating to 20.04. For any future people trying to solve this, the solution for me actually was seemingly unrelated.

I booted into windows (which I had installed on another disk) through Grub.
Windows showed a “repairing drive” on boot, and then booted fine.
I then did a chkdsk C: /r on the windows boot disk (seemingly entirely unrelated). It got hung at 30% but I just let it sit for awhile
When I rebooted into Ubuntu, all my snaps were fixed

I thought I would have to reinstall the OS, but this seemingly unrelated workflow fixed it.