It’s been several months since we turned download deltas on in the snap store. I realised that we never published a single complete overview of what the feature is & how it works, so this is a rather belated announcement of a feature that people have been using for months now:
If anyone has any questions, please don’t hesitate to ask. If there are other aspects of the snap store that people would appreciate a blog post about, please let me know.
Great news!
What is the minimal version of snapd that supports deltas?
Any plans to integrate deltas in snapcraft? So that uploads to the store would also be fast.
Are deltas calculated only between two consecutive revisions? It would be great to have deltas between all current revisions in different channels for popular snaps like core snap: to faster switch between channels.
Deltas were enabled by default in snapd 2.23, we had dark-launched prior to that with a feature-flag. Upload deltas have been in snapcraft since 2.28.
@thomir can elaborate on the selection of which pairs of revisions we generate deltas for, but our intent has been to gather more data so we can base tweaks of that algorithm on actual usage. Thanks for your suggestion about core.
As Bret mentioned below, snapd 2.23 is the minimum version you need in order to use download deltas.
This is actually already in place, but I’ll let @blr write up a feature report for that.
The language here is a little complex - they’re calculated between two successive releases. For example, if revision 15 is in stable, and you go to release revision 23 into stable, we’ll generate a delta between 15 and 23. We do this for all channels, but we don’t currently do it across channels: I’d need to be convinced that this use case was popular enough to warrant supporting it. One of my tasks for this week is to improve the KPIs we’re collecting for the delta feature so I can drill down and see per-snap statistics (currently we only collect per-channel stats). This will allow me to better understand where we’re missing deltas and plug that gap.
Thanks for a great write-up! I have a clarifying question that is hopefully not out of place in this thread: As of today (snapd 2.30), are there any specific requirements to get snap updates to be downloaded as deltas instead of full snaps?
I’m having trouble getting updates of an app that I’m publishing myself to download as deltas. It’s a python script + libraries that result in a 23MB snap. If I make a minor change to the codebase, snapcraft successfully generates a small (~300KB) delta and pushes that delta to the store when I run snapcraft push <snapname>.snap, so that’s good. I then release the snap to “edge”.
However, when I run snap refresh on a client, the full 23MB snap is downloaded, instead of the ~300KB delta. I tried a few times on sequential releases, with the same result each time. I’d like to understand what it is about my setup that’s causing the full snap to be downloaded rather than the delta.
Some potentially relevant facts:
As mentioned, this is a private snap in the “edge” channel
My snapcraft.yaml file has grade: devel and confinement: strict set. Confinement is not overridden during install.
I have only carried out the above-mentioned updates by running snap refresh manually, rather than waiting for auto-refresh.
Perhaps unsurprisingly, sudo journalctl -u snapd doesn’t throw up anything related to these refreshes. It’s not clear to me where to look for the log messages pasted in the original link (e.g. Deltas enabled.), or indeed how to go about troubleshooting this further.
Let me know if the question should be moved to a separate thread - I’m posting here in the hope that it’s a simple answer that can contribute to the existing discussion around functionality. Thanks in advance for any help anyone can offer!
While not an answer to your question, note that what’s happening here is that you’re generating an upload delta (this saves you, the developer, upload bandwidth), which is distinct from the download deltas which are generated store-side.
Thanks, that’s a good point. I guess it was comforting to see that there doesn’t appear to be a fundamental issue with generating deltas (of a small size) - but you’re right that it’s not those same uploaded deltas that will be downloaded from the store during a refresh.
So I went digging into store logs and it seems all deltas were generated correctly for your snap. I saw, at the very least, deltas to go from r7 to r8 and from r8 to r9. So if someone has r7/8 they should be able to go to r8/9 respectively.
One thing you can do to get more info about which deltas are being used is asking snapd to give you debugging messages. From the snapd HACKING.md:
Thanks for taking a closer look @roadmr! Also very useful pointer on how to get the debug info!
I’ve now tried a few more revisions; the behavior in terms of downloading full snaps instead of deltas is the same as before. I can’t really see any tell-tale signs of what’s going wrong from the logs: a few successful API requests followed by a download of the full .snap file. No mention of deltas anywhere. I’m of course happy to share the logs, though I assume it that I should sanitize the authorization strings from the HTTP requests before doing so?
oh bummer… yes, a look at the logs might be enlightening, but yes, please remove the auth strings and any other info you don’t want to be published, such as your IP address
How long are you waiting before executing the refresh? Download delta generation is an asynchronous process, and on average can take a minute or two.
It’s also worth ensuring that your machine has the previously released snap revision installed, since we currently only generate deltas between the ‘N’ and ‘N+1’ releases.
After some more investigation I also can’t see any issues on the server-side that might explain you not getting download deltas. If you could please try again, and collect for me:
The name of the snap you’re trying to refresh
The revision of the snap you have installed
The time (in UTC) that you ran the update
With that information I should be able to pin down why you’re not seeing download deltas. Feel free to send that information to me privately, if you prefer.
Thanks both for the support, I really appreciate your responsiveness! I’m pretty tied up today but should get the chance to post full client-site debug info tomorrow. Will probably do that in a separate thread as this appears to be potentially specific to my setup rather than “of general interest”.
To answer some of your questions in the meantime:
I’ve generally been waiting about 10-15 minutes between the release of a new version and a refresh attempt. In at least one case it was closer to an hour - so hopefully enough time for deltas to get generated on the server.
I’ve been carrying out upgrades to successive revisions. E.g. install revision 8 on device, release revision 9 to store, wait a while, refresh device so it pulls 9, etc.
The app name is “stromm”. As I mentioned it’s set to private, but I’m not against making it public if that might help.
