Proposal to disable squashfs fragments in snaps

I’ve been looking into how we can run a snap through unsquashfs, rebuild it using mksquashfs, and finally compare the result to the original snap to see if there are any differences. This is a useful thing to be able to do in the automated review-tools to detect any anomalies in the uploaded snap. The ability to do this requires that the generation of a squashfs/snap file being predictable and reproducible. The same input should produce the same output.

One issue that I’ve ran into when trying to rebuild a snap in a predictable manner is the fragment handling threads used inside mksquashfs. They introduce unpredictability in the resulting squashfs image because they’re doing work to gather, merge, and compress “fragments” in parallel. Fragments are files that are less than the squashfs block size (131072 bytes by default) that are merged together inside of a single block and then compressed together to reduce the size of the resulting squashfs file.

There are three possible solutions to this problem:

  1. Remove the multi-threaded nature of the fragment handling threads
  2. Adjust the behavior of the fragment handling threads to produce consistent results
  3. Don’t use squashfs fragments in snaps

#1 and #2 are both problematic in that they require code changes to mksquashfs. This could be difficult to roll out everywhere (thinking cross-distro) since snapcraft uses the system mksquashfs.

Someone has already implemented a patch which does #1 but it significantly impacts performance.

To my knowledge, nobody has previously attempted #2. There’s an upstream feature request with no action taken. This option doesn’t look trivial, could take considerable work, and we still have the problem of getting the patched mksquashfs out to the world.

#3 is my favored solution. The negative here is that disabling fragments in mksquashfs (using the -no-fragments) option increases the size of the squashfs file. I was hoping that the increase in snap size would be offset by smaller xdelta3 binary diffs since the snaps would be more predictable since today, if you rebuild the same snap twice on a system with a fair number of cores, you’ll likely see a difference between the two snaps. However, I’ve analyzed a sequence of LXD and atom snaps but I’m not seeing the xdelta3 diffs consistently being any smaller. Here are the results where snaps built with the mksquashfs -no-fragments option containing “nofrag” in the name:

Atom
$ analyze-nofrag.sh atom_76.snap atom_88.snap atom_91.snap atom_94.snap atom_97.snap
### Resquashing atom_76.snap with no fragments
173M    atom_76.snap
178M    atom_76-nofrag.snap
### Resquashing atom_88.snap with no fragments
173M    atom_88.snap
178M    atom_88-nofrag.snap
816K    atom_76-to-atom_88.xdelta3
708K    atom_76-nofrag-to-atom_88-nofrag.xdelta3
### Resquashing atom_91.snap with no fragments
173M    atom_91.snap
178M    atom_91-nofrag.snap
408K    atom_88-to-atom_91.xdelta3
468K    atom_88-nofrag-to-atom_91-nofrag.xdelta3
### Resquashing atom_94.snap with no fragments
173M    atom_94.snap
178M    atom_94-nofrag.snap
492K    atom_91-to-atom_94.xdelta3
456K    atom_91-nofrag-to-atom_94-nofrag.xdelta3
### Resquashing atom_97.snap with no fragments
173M    atom_97.snap
178M    atom_97-nofrag.snap
1.3M    atom_94-to-atom_97.xdelta3
1.2M    atom_94-nofrag-to-atom_97-nofrag.xdelta3
LXD
$ analyze-nofrag.sh lxd_5041.snap lxd_5061.snap lxd_5072.snap lxd_5182.snap lxd_5235.snap
### Resquashing lxd_5041.snap with no fragments
43M     lxd_5041.snap
43M     lxd_5041-nofrag.snap
### Resquashing lxd_5061.snap with no fragments
43M     lxd_5061.snap
43M     lxd_5061-nofrag.snap
16K     lxd_5041-to-lxd_5061.xdelta3
28K     lxd_5041-nofrag-to-lxd_5061-nofrag.xdelta3
### Resquashing lxd_5072.snap with no fragments
43M     lxd_5072.snap
43M     lxd_5072-nofrag.snap
20K     lxd_5061-to-lxd_5072.xdelta3
24K     lxd_5061-nofrag-to-lxd_5072-nofrag.xdelta3
### Resquashing lxd_5182.snap with no fragments
43M     lxd_5182.snap
43M     lxd_5182-nofrag.snap
17M     lxd_5072-to-lxd_5182.xdelta3
17M     lxd_5072-nofrag-to-lxd_5182-nofrag.xdelta3
### Resquashing lxd_5235.snap with no fragments
43M     lxd_5235.snap
43M     lxd_5235-nofrag.snap
14M     lxd_5182-to-lxd_5235.xdelta3
14M     lxd_5182-nofrag-to-lxd_5235-nofrag.xdelta3

The results show a small increase in size for the large Atom snap and roughly the same size when comparing each xdelta3 binary diff (some slightly larger, some slightly smaller).

You can find the analyze-nofrag.sh script here: https://gist.github.com/tyhicks/0d23aa4d01bacb2e18e7f6a5a628d157

If this small increase in snap size is deemed acceptable, I’ll propose a change to snapcraft to make use of the -no-fragments option.

I take it this is not something that can be done via -processors 1?

More data from a few large snaps that I maintain (libreoffice, chromium, 0ad):

$ ./analyze-nofrag.sh libreoffice_5.4.2.2_amd64.snap libreoffice_5.4.3.2_amd64.snap

Resquashing libreoffice_5.4.2.2_amd64.snap with no fragments

500M libreoffice_5.4.2.2_amd64.snap
511M libreoffice_5.4.2.2_amd64-nofrag.snap

Resquashing libreoffice_5.4.3.2_amd64.snap with no fragments

412M libreoffice_5.4.3.2_amd64.snap
424M libreoffice_5.4.3.2_amd64-nofrag.snap
197M libreoffice_5.4.2.2_amd64-to-libreoffice_5.4.3.2_amd64.xdelta3
198M libreoffice_5.4.2.2_amd64-nofrag-to-libreoffice_5.4.3.2_amd64-nofrag.xdelta3

$ ./analyze-nofrag.sh chromium_63.0.3239.59_amd64.snap chromium_63.0.3239.70_amd64.snap chromium_63.0.3239.84_amd64.snap

Resquashing chromium_63.0.3239.59_amd64.snap with no fragments

156M chromium_63.0.3239.59_amd64.snap
160M chromium_63.0.3239.59_amd64-nofrag.snap

Resquashing chromium_63.0.3239.70_amd64.snap with no fragments

156M chromium_63.0.3239.70_amd64.snap
160M chromium_63.0.3239.70_amd64-nofrag.snap
74M chromium_63.0.3239.59_amd64-to-chromium_63.0.3239.70_amd64.xdelta3
73M chromium_63.0.3239.59_amd64-nofrag-to-chromium_63.0.3239.70_amd64-nofrag.xdelta3

Resquashing chromium_63.0.3239.84_amd64.snap with no fragments

156M chromium_63.0.3239.84_amd64.snap
160M chromium_63.0.3239.84_amd64-nofrag.snap
68M chromium_63.0.3239.70_amd64-to-chromium_63.0.3239.84_amd64.xdelta3
69M chromium_63.0.3239.70_amd64-nofrag-to-chromium_63.0.3239.84_amd64-nofrag.xdelta3

$ ./analyze-nofrag.sh play0ad_0.0.21-alpha_amd64.snap play0ad_0.0.22-alpha_amd64.snap

Resquashing play0ad_0.0.21-alpha_amd64.snap with no fragments

764M play0ad_0.0.21-alpha_amd64.snap
765M play0ad_0.0.21-alpha_amd64-nofrag.snap

Resquashing play0ad_0.0.22-alpha_amd64.snap with no fragments

826M play0ad_0.0.22-alpha_amd64.snap
827M play0ad_0.0.22-alpha_amd64-nofrag.snap
746M play0ad_0.0.21-alpha_amd64-to-play0ad_0.0.22-alpha_amd64.xdelta3
747M play0ad_0.0.21-alpha_amd64-nofrag-to-play0ad_0.0.22-alpha_amd64-nofrag.xdelta3

That’s correct. mksquashfs has six threads, one of which is the fragment deflator/compressor thread, even when -processors 1 is specified. (9 threads with -processors 2, 12 threads with -processors 3, etc.)

Thanks for the libreoffice, chromium, and play0ad numbers. The relatively small increase in size makes me feel more comfortable with the change.

I’ve created a snapcraft pull request:

https://github.com/snapcore/snapcraft/pull/1805

If someone feels like there needs to be more discussion about this change, please speak up soon.

1 Like

note snap pack probably also needs a patch.

snap pack PR: https://github.com/snapcore/snapd/pull/4396