Almost exactly two years ago, I initiated a brief discussion on the status of delta vs full snap downloads, especially in the context of armhf: Refresh causes download of full snap file instead of delta. The conclusion at that time was that (a) deltas work, though (b) for armhf they are disabled unless an environment flag is set, yet ( c) they should probably also be enabled by default for armhf (last post in that thread).
Earlier today I refreshed a snap on a Raspbery Pi running the latest version of snapd (2.43.2), and noticed that the full snap was downloaded rather than a delta. Without having done any further investigation, that got me wondering where things stand, two years down the line. I have two specific questions:
Are delta downloads the default behavior for armhf, or is an environment flag still required?
I assume deltas are generated by the store whenever a new snap revision is uploaded. Is the logic applied to this documented somewhere?
For the second question what I have in mind is the following. Let’s say I release snap revision 1 to the edge channel, and promote it to stable. Then I also release revisions 2, 3, 4, 5, 6, 7, and 8 to edge. I figure that 8 is good enough, so promote it to stable. Will a diff from 1->8 be generated and provided to any device that’s tracking the stable channel? Or will that not happen due to the “gap” in revisions?
One clarification: the device where I noticed the full download (rather than delta) is running Raspbian, and not Ubuntu Core. It has the “core18” and “snapd” snaps installed, but no “core” snap.
I noticed in the codebase of snapd that it appears to look for an xdelta3 command in /snap/core/current/usr/bin/xdelta3 - which obviously doesn’t exist in the absence of the “core” snap. On the other hand /snap/snapd/current/usr/bin/xdelta3 does exist (but is not in the system path). I wonder if that may be the reason…
UPDATE: Yes, after a bit of poking around, the above does appear to be the reason for deltas not being used. If I had to guess, this actually affects all devices that don’e either have the “core” snap installed, or have xdelta3 present in $PATH through another means. For the Raspberry Pi in question, installing xdelta3 via apt magically enabled delta downloads. I guess the hardcoded dependency on xdelta3 under /snap/core/ is a bug?
Also to answer my own question: based on the codebase, setting SNAPD_USE_DELTAS_EXPERIMENTAL to true is no longer necessary. Using deltas is the default on all architectures. ThoughSNAPD_USE_DELTAS_EXPERIMENTAL can be set to false to disable deltas. This is the case for snapd 2.43 at least.
No, because the logic is subject to change in order to generate more relevant deltas that cover as many refresh scenarios as possible.
The most common scenario (a snap is released to stable) is covered by automatic last stable -> new stable delta generation, for all architectures and all snaps.
If the resulting delta is too close in size to a full-size snap, it gets discarded and a full download is used instead.
In your scenario, the 1->8 diff/delta will indeed be generated, since 1 was the previous revision on stable.
Thanks guys! I appreciate the help, and makes a lot of sense. Quick responses:
This happened with respect to a different snap (published by me), not the core snap. Core hasn’t been installed on this device.
That would…make a lot of sense :). As I said, though, on a system with snapd installed, I observed a perfect correlation between not having xdelta3 installed in $PATH and not getting delta updates, and getting delta updates as soon as xdelta3 was installed to $PATH using apt.
Maybe a stupid (and definitely tangential) question, but on Raspbian - or another “classic” system - how would the snapd daemon know to look for an xdelta3 binary in /snap/snapd/current/usr/bin/xdelta3? The snapd daemon binary is at /usr/lib/snapd/snapd, and I can’t see it running in an environment that would make it aware of the presence of an xdelta3 binary within that snap subdirectory (needless to say it’s not in $PATH).
For what it’s worth, I have snapd 2.42.1 installed via apt, and 2.43.2 installed “on top of that” via snap install snapd. Correspondingly snap verison shows
snap 2.43.2
snapd 2.43.2
series 16
raspbian 10
kernel 4.19.97-v7+
Thanks, that clarifies things and answers my question. Sounds like you guys have thought it through!
However reading through that code, we prefer xdelta3 from $PATH, and if that doesn’t exist we check the core snap for xdelta3 even though we could not have the core snap like in your system. I think you have found a bug
Haha yep! That’s the exact same bit of code I’d looked at when trying to work out what’s going on …wondered if I’m just missing the magic. Would Bugs : snapd be the right place to file this? Can’t see anything about it there at present.
…wondered if I’m just missing the magic. Would