A more graceful way for snapd to fail when /snap is missing

[zyga]

Note that if you are low on space and have an internal (mounted a boot) disk you can just mount a partition on /var/lib/snapd/snaps and copy the snaps over. This is where the space is really used.

[veritanuda]

Note that if you are low on space and have an internal (mounted a boot) disk you can just mount a partition on /var/lib/snapd/snaps and copy the snaps over. This is where the space is really used.

Ok now it becomes officially weird.

ls -la /var/lib/snapd/snaps/
total 1446636
drwxr-xr-x  3 root root      4096 Nov 13 16:15 .
drwxr-xr-x 15 root root      4096 Nov 13 16:20 ..
-rw-r--r--  1 root root 122626048 Aug 30 16:18 castersoundboard_21.snap
-rw-r--r--  1 root root 122671104 Sep  5 22:42 castersoundboard_34.snap
-rw-r--r--  1 root root  85348352 Sep 19 03:21 core_2898.snap
-rw-r--r--  1 root root  87080960 Oct 10 18:34 core_3017.snap
-rw-r--r--  1 root root  87089152 Oct 23 20:43 core_3247.snap
-rw-r--r--  1 root root   3350528 Sep 13 11:30 doctl_13.snap
-rw-r--r--  1 root root 114561024 Sep  6 00:39 krita_25.snap
-rw-r--r--  1 root root 114900992 Sep  6 00:33 krita_26.snap
-rw-r--r--  1 root root 114946048 Oct 10 12:04 krita_27.snap
-rw-r--r--  1 root root 130473984 Sep  6 13:48 lonewolf_32.snap
-rw-r--r--  1 root root  63033344 Sep 15 15:59 magic-device-tool_286.snap
drwxr-xr-x  2 root root      4096 Sep 26 18:41 partial
-rw-r--r--  1 root root   9011200 Oct 10 23:04 pulsemixer_1.snap
-rw-r--r--  1 root root 311910400 Nov 10 23:47 shotcut_15.snap
-rw-r--r--  1 root root   5074944 Nov 11 14:17 snapweb_300.snap
-rw-r--r--  1 root root   5074944 Nov 13 16:13 snapweb_307.snap
-rw-r--r--  1 root root   3616768 Sep 23 11:30 teleconsole_1.snap
-rw-r--r--  1 root root   3661824 Oct  3 10:07 teleconsole_3.snap
-rw-r--r--  1 root root  10481664 Oct 18 13:00 uappexplorer-cli_2.snap
-rw-r--r--  1 root root  86364160 Sep 24 16:23 ultimate-media-downloader_3.snap


du -h  /var/lib/snapd/snaps/
4.0K	/var/lib/snapd/snaps/partial
1.4G	/var/lib/snapd/snaps/

So 1.4Gb of snaps takes up 4.3Gb of real space?

and I am officially confused.

[zyga]

and I am officially confused.

Snaps are heavily compressed. When you look at them they are uncompressed in /snap. When you look at the .snap files you see the real thing.

[ogra]

It was literally an rsync -av --partial --progress /snap .

If you did that command on that target disk (/mnt/containers/snap/) then you definitely doubled up (likely even more because the target will have the data uncompressed) the space your snaps use by copying the content of the mounted compressed squashfs’es out of the mountpoints into the target disk …

[veritanuda]

Ok I guess I have had enough of this. I will blow away snapd completely and start again. This time looking for where it puts actual snaps and where is looback mounts them. Because I swear it was freeing up space by removing /snap

[ogra]

So 1.4Gb of snaps takes up 4.3Gb of real space?

the kernel is clever enough and exposes the content of the mounted squashfs files, thats not actually the real disk space used

[niemeyer]

So 1.4Gb of snaps takes up 4.3Gb of real space?

Exactly the opposite: 4.3GB of “real space” takes up 1.4GB of disk space!

Nice, isn’t it?

[popey]

@veritanuda you have copied the contents of snap files to another disk. Not the snaps themselves. This won’t end well.

The disk space consumed by snaps is all in /var/lib/snapd/snaps, not /snaps

alan@hal:~$ du -hs /var/lib/snapd/snaps
16G	/var/lib/snapd/snaps

When you ‘snap install’ something, you get a .snap file in /var/lib/snapd/snaps/ - e.g. for teleconsole I have

 alan@hal:~$ ls -l /var/lib/snapd/snaps/teleconsole_3.snap 
-rw-r--r-- 1 root root 3661824 Oct  3 04:04 /var/lib/snapd/snaps/teleconsole_3.snap

That is the only disk space the snap takes up. That compressed file is mounted (not unpacked) in /snap/teleconsole/ and a symlink at /snap/teleconsole/current points to that

alan@hal:~$ mount | grep teleconsole
/var/lib/snapd/snaps/teleconsole_3.snap on /snap/teleconsole/3 type squashfs (ro,nodev,relatime)

alan@hal:~$ ls -l /snap/teleconsole/current
lrwxrwxrwx 1 root root 1 Oct  3 04:04 /snap/teleconsole/current -> 3

At runtime, yes, you will see files in /snap/teleconsole/3

alan@hal:~$ du -hs /snap/teleconsole/3/
15M	/snap/teleconsole/3/

However, if you shut the machine down, and examine the disk, you’ll see nothing in /snap, because they only get mounted when the system is started.

So what you’ve done is copy files from the insides of those .snap files to an external/separate disk. The snaps which get updated in /var/lib/snapd/snaps will have new and updated files which you will not have in your copies. You’ve basically broken the snap installs.

[veritanuda]

Yeah well I purged snapd now, made symlink from /var/lib/snapd/snaps -> /mnt/containers/snap and am installing warsone2100 now see where it all ends up. If as you say all the magic is in /var/lib/snapd/snaps then it should not take up any more space on root but on the /mnt/containers/snap instead. In which case my snapd.mount services is still needed to stop snapd from breaking boot but it just points to a different symlink then.

Hey ho.

[veritanuda]

Yup, ok that seems to hold true.

ls -la  /mnt/containers/snap/
total 263052
drwxr-xr-x 3 root root        70 Nov 13 16:52 .
drwxr-xr-x 4 root root       127 Nov  5 23:05 ..
-rw-r--r-- 1 root root  87089152 Nov 13 16:47 core_3247.snap
drwxr-xr-x 2 root root         6 Sep 26 18:41 partial
-rw-r--r-- 1 root root 182276096 Nov 13 16:52 warzone2100_12.snap

and

mount |grep snap
/dev/mapper/vg_ssd-snap on /mnt/containers type xfs (rw,relatime,attr2,inode64,noquota)
/mnt/containers/snap/core_3247.snap on /snap/core/3247 type squashfs (ro,nodev,relatime)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,noexec,relatime,size=3270728k,mode=755)
nsfs on /run/snapd/ns/core.mnt type nsfs (rw)
/mnt/containers/snap/warzone2100_12.snap on /snap/warzone2100/12 type squashfs (ro,nodev,relatime)

So I guess I should really retitle this post to

A more graceful way for snapd to fail when /snap or /var/lib/snapd/snaps is missing.

[zyga]

A symlink will likely not work because of confinemnt. You need a mount or a bind mount at the very least. You can add a systemd mount unit to do the right thing.

[veritanuda]

A symlink will likely not work because of confinemnt. You need a mount or a bind mount at the very least. You can add a systemd mount unit to do the right thing.

Hmm atm I am sysytemd.mount /mnt/containers I have both flatpak and snap folders there. I spose I could just create a couple more LV’s and mount the separately but talk about added complexity.

[zyga]

You don’t need a partition, a bind mount will suffice and will be always working (unlike a symlink).

[veritanuda]

You don’t need a partition, a bind mount will suffice

Hmm ok that sounds plausible. It is all to do with timing really. Snapd starts up as soon as systemd.pre-mount finishes and root is mounted, meaning if /var/lib/snapd/snaps is not there it barfs.

I don’t spose snapd is smart enough to just check /var/lib/snapd/snaps for any realtime changes is it? Otherwise I could just leave snaps empty and bindmount over it later in the boot process and restart snapd

[zyga]

Hmm ok that sounds plausible. It is all to do with timing really. Snapd starts up as soon as systemd.pre-mount finishes and root is mounted, meaning if /var/lib/snapd/snaps is not there it barfs.

You can just say Before=snapd.service

[veritanuda]

You can just say Before=snapd.service

I already do that. I was just mulling over putting a lazy script unit in to do the bind mounts after it.

[ogra]

many snaps ship services and snapd generates a systemd service file on install … they would all fall flat on their face

if you would do realtime stuff here that would require a lot of additional plumbing to manage the services etc …

[veritanuda]

Meh… decided to keep it obvious. Edited /etc/fstab and added x-systemd.requires= and x-systemd.before= mount options.

means don’t really have to worry about change to systemd units , reboots etc.

Ok feel free to close the topic if you want. Think I have gotten the answer I need now.

[zyga]

FYI: I made this patch to snapd as a side-effect of this conversation: https://github.com/snapcore/snapd/pull/4210

We will now actively inform users casually browsing the filesystem about what is in /snap

[veritanuda]

FYI: I made this patch to snapd as a side-effect of this conversation: https://github.com/snapcore/snapd/pull/4210

We will now actively inform users casually browsing the filesystem about what is in /snap

Nice. Thanks!