It should be possible to use ZSTD compression for snaps. It offers better compression/performance ratio. I do understand the reason for not exposing that currently because systems running older kernel may not support ZSTD (that’s understandable).
Since Snap store only support xz and lzo compressed snaps, I don’t see any hard reason in snapcraft not supporting that for the snap archive. This can be useful for custom snap stores, where we “know” on what hardware our apps are going to run.
Also: Can we please revisit the default compression format again. Especially is snapd still running on distros (and Canonical customer devices) that are pre-linux 4.14 ?
My proposal about using desync for Delta updates also suggests that only a single instance of a snap should live on storage AND delta chunks for older revisions should live on storage and snap revert should reconstruct the old revision using those chunks, instead of “wasting” disk space as we have today.
The issue with enabling this is that allowing snaps to be uploaded with compression formats that are not universally supported like XZ and LZO introduces the possibility of folks uploading snaps that cannot be used on platforms that we, Canonical and the snapd team, specifically support.
You have mentioned a custom snap store, if you have a brand store I would suggest reaching out to your sales rep folks about the possibility of enabling this for your brand store, it may be something we can do for specific brand stores until we have a more general solution in place which allows us to provide a better user experience / story for users on distros which may not support new compression formats like zstd, etc. See my explanation here for more details: https://ubuntu.com/blog/why-lzo-was-chosen-as-the-new-compression-method
Regarding the patch to snapd, I’m sure it’s extremely simple, but we can’t accept it until we have a better story for users on old distributions that don’t support the new formats.
We could easily probably add support for GZIP and “none” compression methods since those should be easily supported everywhere but indeed the more interesting thing is ZSTD which brings the best of both worlds for both size and speed.
I believe @om26er was referring to a non-Canonical Snap Store when they mention “custom” - i.e. not the Public store, and not a Brand store. They’re working on reimplementing the Snap Store API in an open source implementation over on GitHug.
If you attempt to limit what the capability of the snap project is to achieve an arbitrary amount of backward compatibility, this project shall stagnate. Addition of modules that allow for compatibility should be the most ideal method of ensuring compatibility, but if they hinder progress, I shall greatly support their removal. Different configurations of GNU/Linux should be what this project is attempting to support rather than old packages, because the latter is what has caused Windows to become as bloated as it is, and shall divert precious resources to something that is known to be incredibly difficult to maintain whilst performing improvement.
For what it is worth, if you had a suitably patched implementation of snap pack, it would be possible to have an unpatched Snapcraft build such a snap with the following in the snapcraft.yaml file:
passthrough:
compression: zstd
This won’t generate a specific warning, but it does signal “I know what I’m doing isn’t ready for prime time”.
There are checks for the compression method in the automated review tool which don’t rely on “snap pack”, so in theory the snapd change could be accepted without risk of zstd snaps being published to the store before the compatibility issues have been evaluated.
That is true and also why I mentioned that in my initial comment that, there is no risk in a developer publishing their snaps using wrong compression, because the store does not allow that.
We could also make zstd support hidden i.e. by keeping snap pack --help unchanged.
The store implementation that I have is currently not open-source but I will do that eventually. We are using that in our company internally.
The basic architecture is:
Patched snapd to support custom signing authority
Django based web service, that signs snaps (creates assert file) and uploads to S3
A cli tool (written in python) that talks to snapd sock but is also able to talk to our django based service to install/update snaps (our store takes precedence over Snap store)
The service also supports delta updates using desync
In general we try not to add features partially like supporting this new compression method without having a plan for the rest of the feature as then it results in us maintaining a bunch of half-finished features which is not productive and we already have this problem too much, so I would be -1 to a patch to snapd allowing zstd until we have more of a plan in place unfortunately.
It would be interested to find out where people stand on this topic. I believe this is a very safe change given all the changes just need to be done to pass the right flag to mksquashfs so this doesn’t really increase any “debt” on snapd IMO.
