We have devices running snaps in some pretty remote locations with poor connectivity. Sometimes we observe that during refresh, snap downloads error out - with an “unexpected EOF” error.
Doing a refresh manually on a particular device in Sierra Leone, I observe that the ~50MB download chugs along fine for a while at ~25kB/sec, but then about a quarter of the way through the dreaded “unexpected EOF” comes up and everything drops dead.
error: cannot perform the following tasks:
- Download snap "some-snap" (430) from channel "stable" (unexpected EOF)
Now, this is clearly due to some sort of network connection reset, but after 4-5 retries it seems to happen pretty consistently. This behavior may end up being particularly problematic due to the amount of (expensive mobile) data that would get eaten up over a number of auto-retries - though that’s a separate point.
For the moment I’m trying to brainstorm a work-around that allows me to push the update through despite the network disconnects. I’d love to get the community’s feedback on either of the following ideas:
Somehow having snapd resume a previously interrupted download, rather than starting over each time. I’m confident that something like this would manage to pull the full file through after a couple of attempts - as the connection is at least good enough for chunks of 10-20MB to be downloaded at a time.
Copying the snap file onto the device manually (e.g. as a chunked zip file) and installing it from that. Perhaps I can use the method outlined in Offline snap installers and possibility to update in order to install the snap in an “approved” way (i.e. without --dangerous) that would still make it look like the snap came from the store, and make future refreshes possible. Hopefully the assertion file is portable across devices.
Clearly #1 would be preferable (and more easily scalable to a larger number of devices), but I figured I’d put the general question out there for any thoughts.
Hmm, I’m not sure if we do resumes for auto-refreshes, if we don’t then this is a very reasonable suggestion.
What you would do here is to download both the snap and the assertion with snap download <name> and then copy both the .assert and .snap file to the machine, and run snap ack <name>.assert and then you can install asserted with snap install <name>.snap.
Just wanted to follow up and say that chunking the 50MB snap into 10MB parts and transferring it bit by bit - and then re-combining and installing on the device (after ack-ing) worked very well.
It does bring me to the point about potential resumption of interrupted downloads. I’m conscious that the directory /snap/core/current/var/lib/snapd/snaps/partial exists, though I’ve not seen it be populated. Looking at the snapd code, it looks like partial downloads should be saved and subsequently resumed. In the most recent version, there is also a specific LeavePartialOnError option (seemingly defaulting to true) that controls that.
I may try to do some more testing with respect to interruption/resumption of downloads, to get a clearer view of the situation. But if anyone has ideas in the meantime - and/or can confirm what the behavior should be - then do let me know!
Do you mean by issuing snap download xyz? I did not try that.
I did try snap refresh xyz multiple times, and the download always started from scratch. Though now that I think of it, the core snap on that system would likely have also been out of date due to its inability to download updates (just checked: it’s 2.40), so maybe it was just suffering from the issue you mentioned.
I’ll see if I can reproduce the issue anywhere, but for now I’m going to assume it’s been fixed.
@roadmr I did some more testing here, with the latest snapd (2.43.3). This is the behavior I observed:
If snap download xyz is run manually, it saves interim data to a xyz_123.snap.partial file. This is left in place in case of interruption by e.g. a connection error (usually “unexpected EOF” in my case). Running snap download xyz again uses that partial file to resume the download - so that’s all good.
On the other hand, if a refresh is carried out - either by running snap refresh xyz manually, or as part of an automatic refresh, I do not see any evidence of partial downloads being saved - or of interrupted downloads being resumed. That’s based on two observations:
I do not find any .partial files left anywhere on the device, e.g. in /var/lib/snapd/snaps
Looking at snap changes, I can see download operations retried over and over and over again - over multiple days/weeks for example…and consistently failing with “unexpected EOF”. If these downloads were being resumed, I would expect them to finally complete after a finite number of attempts.
Is it possible to confirm what the functionality should be, in the context of (auto-)refresh? It’s possible that I’m just not diagnosing this correctly. I checked Launchpad for bug reports, and found this: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1790964. In principle relevant, but the discussion somehow got sidetracked and then died off inconclusively.
snap refresh not resuming downloads (and potentially snap install as well - wonder if this could be tested?) sound like snapd bugs. I’ve groomed the one you linked to a bit, hopefully will get more attention from the snapd team.
If snap download works, it means the store-side functionality is working as intended (meaning I’m of the hook WOOHOO!). No, seriously - at least this reassures me that the store-side regression from a couple of weeks ago is properly fixed and we can focus on checking snapd’s behavior.
Thanks @roadmr! I’m doing some more tests and will add any further info to the bug page if anything different emerges. Though for the time being what I said above appears to be the case. Either way, definitely appears to be a snapd issue rather than a store back-end issue.
In the meantime, if anyone is in a situation like mine where a device’s connection is so poor that it can’t complete a snap download, the following works nicely - assuming you have (at least intermittent) shell access.
Make a shell script with the following, where snap-name is the name of the snap:
#!/bin/bash
until snap download snap-name; do
echo Download disrupted, resuming in 5 seconds...
sleep 5
done
Run this with nohup so it keeps going if your shell session is interrupted. The loop will keep trying to download (and resume any interrupted downloads) until the full snap+assert files have been downloaded. At that point you can log back in and snap ack+snap install the downloaded files.
@svet that script worked like a charm. thank you for your diligence and the work you put in. my area is currently having some internet issues and this did the trick.
Thanks @henry for the kind words. Many on this forum have been a fount of support and advice for me, so it’s nice to be able to give back to the community occasionally!
