snap refresh amazon-ssm-agent
error: cannot refresh "amazon-ssm-agent": unexpectedly empty response from the server (try again
later)
We’re seeing this error across our entire fleet of servers, which hasn’t been happening in the past. The status page shows that there was downtime the past two days, but it reports everything is okay today. Can anybody tell me what’s going on with this? Thanks.
Hi there,
Can you retry now? I spotted a configuration mistake (which I did in the first place, apologies) which caused requests for some snaps to intermittently return an empty result (I bet you did get refreshes on some of your servers, just not all of them - intermittent 
). It’s fixed now, all should be OK.
Yep, seems to work now on a server which was broken at the time of my initial post.
Thanks for the quick fix!
Hmm, perhaps I spoke too soon. It did work on one of the servers I tried, but we’re still seeing errors on other servers. This is from about two minutes ago:
root@ip-10-3-0-94-staging:/home/me# date && snap refresh amazon-ssm-agent
Wed Mar 17 14:35:32 UTC 2021
error: cannot refresh "amazon-ssm-agent": unexpectedly empty response from the server (try again
later)
I’ve been seeing what I believe is the same problem in some Github Actions runs, such as:
In this case, the snapcore/action-build does a snap refresh lxd early on, which is failing with the same error message:
/usr/bin/sudo snap refresh lxd
error: cannot refresh "lxd": unexpectedly empty response from the server (try
again later)
Error: The process '/usr/bin/sudo' failed with exit code 1
I kicked off a retry of the run after reading your message, so it still seems to be happening.
I haven’t noticed any similar refresh errors today, so presumably it just took a while for the fix to propagate.
I got the same error today.
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)Just wanted to chime in that I was still getting this issue occasionally today. It did go through after a bit but happened a few times before the build would go through. Possibly interesting was that there weren’t any updates to core18 when it did go through:
snap “core18” has no updates available
My understanding (@roadmr can confirm) is that this typically happens when the store is throttling a particular snap, often because the store infrastructure is being overwhelmed.
This is causing a lot of issues for us (LXD) and our users as whenever we get throttled, our CI fails, causing it to retry, causing even more load. I don’t think the current store side throttling mechanism is really well suited for this.
Could we maybe have the store set some kind of header with a timeout for when to retry in such cases and have snapd then do the waiting and re-trying? (@mvo)
That feels like it could be a pretty cheap fix that would come with an actual further reduction of load by not having automated systems (and users) retry things in tight loops until the store feels like providing the snap again
I’m seeing the same error (with core18) intermittently this morning.
I am also seeing this error today.
@roadmr I am also seeing this same error at the moment when trying to build a base: core18 snap via snapcraft - is core18 currently being throttled in the store?
$ SNAPCRAFT_BUILD_ENVIRONMENT=lxd snapcraft
Launching a container.
Waiting for container to be ready
Waiting for network to be ready...
snapd is not logged in, snap install commands will use sudo
error: cannot refresh "core18": unexpectedly empty response from the server
(try again later)
Error while refreshing snap 'core18' to channel 'latest/stable'
Same here, it has been happening since 2021-06-22T15:28:00Z
error: cannot refresh "core18": unexpectedly empty response from the server
(try again later)
Error while refreshing snap 'core18' to channel 'latest/stable'
If i manually try to update I get this until it does work.
user@machine:~$ sudo snap refresh core18
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
core18 20210611 from Canonical✓ refreshed
There is no retry option when using multipass to build a snap.
I am also running into the same problem. Snapcraft is awesome, I love what it does, but the timing of this is unfortunate for me. Any way to get past this and get snaps building again? I am building on a local machine, and network is connected and fully operational, but seems like there there is a problem with communication. What can I do to get past this? I need to get this out the door and into production soon.
error: cannot refresh “core18”: unexpectedly empty response from the server (try again later)
Error while refreshing snap ‘core18’ to channel ‘latest/stable’
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.
Edit:
Looks like there is a service outage for snap refresh:
“Intermittent access issue in few services for 1 Mins 42 Secs”
But still cant seem to pull.
Edit 2:
After some time and multiple attempts, it eventually passed. Took a while. If you see this issue it may be an external issue. Check the status page and give it some time and try again and again until it works.
FTR we are having such issues with the snapcore/action-build@v1 action:
Run snapcore/action-build@v1
with:
path: .
build-info: true
snapcraft-channel: stable
Building Snapcraft project in "."...
Installing Snapcraft plus dependencies
Ensuring runner is in the lxd group...
/usr/bin/sudo groupadd --force --system lxd
/usr/bin/sudo usermod --append --groups lxd runner
Installing LXD...
/usr/bin/sudo snap refresh lxd
error: cannot refresh "lxd": unexpectedly empty response from the server (try
again later)
lxd had a new release yesterday that switched it to base: core20 … this in turn generates a huge download load, so lxd downloads are throttled by the store currently … if you retry often enough it will actually work, but fact is that quite a few queries will be refused with that message after all
That seems a really unfortunate way to handle congestion, I understand there’s practicalities that outweigh convienience and at the end of the day the service is gratis, but at times the store/CDN seems to play up in ways that give off some bad impressions, straight up denying requests by not responding to them at all seems unfortunate, but then if the alternative is staggeringly slow downloads, maybe it’s better overall.
I’m unsure if this is entirely related to the LXD specific congestion control, but last night I was helping an upstream project with their snap, and the gnome extension SDK snaps were throttling so hard the builds were impractical. When the resulting snap was finally created, it was pushed to a branch release and the test user couldn’t even download it. 99% of the time the store is reliable but it feels like every so often it just acts up, in this case, for a matter of hours, world wide.
It feels bad saying it while knowing I’m paying nothing and I’m probably a significant burden on these resources even by myself, but I think at least saying it is probably worthwhile just to voice the opinion. (Also thanks for the free hosting 
)
i think everyone agrees on that
lxd is supposed to use cohorts in the future to prevent exactly this issue (and span a release over 90 days automatically) … for cohorts you need a key though … and the current release is the first one to support this feature … it is a catch22 …
there are millions and millions of lxd based cloud instances out there in the world and to prevent a meltdown there is manual throttling in place now … this will be a one timer though, cohorts will save us in the future.
I proposed
I don’t own the action, so will need to wait for an owner to merge
Which just has been!