Hardlinking snaps wastes 400 MB tmpfs RAM in live CDs

I’m not sure if snap developers monitor launchpad or not, so I thought I’d report LP bug #1867415 here as well:

Boot with e.g. the current ubuntu-20.04-desktop-amd64.iso. Open a terminal and run df -h. You’ll notice:

Filesystem Size Used Avail Use% Mounted on
/cow       2.0G 434M 1.5G  23%  /

This means that the tmpfs file system needs 400 MB RAM for snaps. Xenial didn’t have snaps and needed only 14 MB tmpfs. Snap shouldn’t consume tmpfs RAM space, it should only use squashfs disk space.

I think this is caused by function Install() in snap/squashfs/squashfs.go, which hardlinks /var/lib/snapd/seed/snaps/* to /var/lib/snapd/snaps/*.
Hardlinks cause overlayfs copy-ups in live CDs and LTSP.

Maybe the next option, symlinks, should be preferred in overlayfs.

I’m sure, if they have time, they do :wink:

@alkisg: How can i reproduce this ? Sounds like to install a core-image ?

Hi @sdhd,

to reproduce it, you just need to boot the live CD, nothing more:

  1. Download any Ubuntu live CD, for example the current ubuntu focal live CD from here.

  2. Boot it and wait a minute until snap is populated:

# snap list
Name               Version                     Rev   Tracking  Publisher   Notes
core               16-2.42.5                   8268  stable    canonical*  core
core18             20191212                    1288  stable    canonical*  base
gnome-3-28-1804    3.28.0-16-g27c9498.27c9498  110   stable/…  canonical*  -
gnome-calculator   3.34.1+git1.d34dc842        544   stable/…  canonical*  -
gnome-characters   v3.32.1+git3.b9120df        375   stable/…  canonical*  -
gnome-logs         3.34.0                      81    stable/…  canonical*  -
gtk-common-themes  0.1-25-gcc83164             1353  stable/…  canonical*  -

These snaps are preinstalled in the live CD, you don’t need to install anything.

  1. Check the disk space:
# df -h
Filesystem Size Used Avail Use% Mounted on
/cow       2.0G 434M 1.5G  23%  /

These 400+ MB there shouldn’t exist, the tmpfs use should be less than 20 MB. They are caused by this line in squashfs.go:

if err := osLink(s.path, targetPath); err == nil {

When using ln instead of ln -s in overlayfs, the source file is modified in order to update its hardlink count. This causes a “copy-up” to RAM. So if a live CD has 400 MB worth of snaps, it causes 400 MB of RAM to be wasted because of those copy-ups.

That logic in squashfs.go should be reworked, so that symlinks and not hard links are used in live CDs, i.e. the next line in squashfs.go:

if filepath.Dir(s.path) == dirs.SnapSeedDir && os.Symlink(s.path, targetPath) == nil {

The logic now is:
try hardlinks, then symlinks, then cp.

It should be possible to just drop the hardlink call:
try symlinks, then cp.

Hardlinks have many issues, like the one described above; also they’re not supported in all file systems; and finally they won’t work at all if the target dir is on a different file system than the source dir. Maybe defaulting to relative symlinks is better in all cases.

There is work on going to do some of the things seeding does at image build time. I’m not sure if creating these hard links is part of that, if it is then this issue will go away. I’m also not sure if that’s going to be ready for focal though.

mwhudson, if snap can’t do it in time for focal, maybe it would be worth it to put “casper” in the affected packages, to do a quick ln -s on boot, e.g. at the end of 55disable_snap_refresh?

400 MB is a lot of additional RAM in the “minimal requirements” for running the Ubuntu live CD…

We have a fix for this in master now https://github.com/snapcore/snapd/pull/8279

sorry, i read your message just now.
Hope the patch above, fix the problem ?

Hi sdhd, thank you for the input, I will test the patch when focal daily live CDs include it.