Unexpectedly empty response from the server

Just wanted to chime in that I was still getting this issue occasionally today. It did go through after a bit but happened a few times before the build would go through. Possibly interesting was that there weren’t any updates to core18 when it did go through:

snap “core18” has no updates available

My understanding (@roadmr can confirm) is that this typically happens when the store is throttling a particular snap, often because the store infrastructure is being overwhelmed.

This is causing a lot of issues for us (LXD) and our users as whenever we get throttled, our CI fails, causing it to retry, causing even more load. I don’t think the current store side throttling mechanism is really well suited for this.

Could we maybe have the store set some kind of header with a timeout for when to retry in such cases and have snapd then do the waiting and re-trying? (@mvo)

That feels like it could be a pretty cheap fix that would come with an actual further reduction of load by not having automated systems (and users) retry things in tight loops until the store feels like providing the snap again :slight_smile:

1 Like

I’m seeing the same error (with core18) intermittently this morning.

I am also seeing this error today.

@roadmr I am also seeing this same error at the moment when trying to build a base: core18 snap via snapcraft - is core18 currently being throttled in the store?

$ SNAPCRAFT_BUILD_ENVIRONMENT=lxd snapcraft
Launching a container.
Waiting for container to be ready
Waiting for network to be ready...
snapd is not logged in, snap install commands will use sudo
error: cannot refresh "core18": unexpectedly empty response from the server
       (try again later)
Error while refreshing snap 'core18' to channel 'latest/stable'

Same here, it has been happening since 2021-06-22T15:28:00Z

error: cannot refresh "core18": unexpectedly empty response from the server
       (try again later)
Error while refreshing snap 'core18' to channel 'latest/stable'

If i manually try to update I get this until it does work.

user@machine:~$ sudo snap refresh core18 
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18 
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18 
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18 
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18 
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18 
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18 
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18 
error: cannot refresh "core18": unexpectedly empty response from the server (try again later)
user@machine:~$ sudo snap refresh core18
core18 20210611 from Canonical✓ refreshed

There is no retry option when using multipass to build a snap.

I am also running into the same problem. Snapcraft is awesome, I love what it does, but the timing of this is unfortunate for me. Any way to get past this and get snaps building again? I am building on a local machine, and network is connected and fully operational, but seems like there there is a problem with communication. What can I do to get past this? I need to get this out the door and into production soon.

error: cannot refresh “core18”: unexpectedly empty response from the server (try again later)
Error while refreshing snap ‘core18’ to channel ‘latest/stable’
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.

Edit:

Looks like there is a service outage for snap refresh:
“Intermittent access issue in few services for 1 Mins 42 Secs”

But still cant seem to pull.

Edit 2:
After some time and multiple attempts, it eventually passed. Took a while. If you see this issue it may be an external issue. Check the status page and give it some time and try again and again until it works.

FTR we are having such issues with the snapcore/action-build@v1 action:

Run snapcore/action-build@v1
  with:
    path: .
    build-info: true
    snapcraft-channel: stable

Building Snapcraft project in "."...
Installing Snapcraft plus dependencies
  Ensuring runner is in the lxd group...
  /usr/bin/sudo groupadd --force --system lxd
  /usr/bin/sudo usermod --append --groups lxd runner
  Installing LXD...
  /usr/bin/sudo snap refresh lxd
  error: cannot refresh "lxd": unexpectedly empty response from the server (try
         again later)

lxd had a new release yesterday that switched it to base: core20 … this in turn generates a huge download load, so lxd downloads are throttled by the store currently … if you retry often enough it will actually work, but fact is that quite a few queries will be refused with that message after all

2 Likes

That seems a really unfortunate way to handle congestion, I understand there’s practicalities that outweigh convienience and at the end of the day the service is gratis, but at times the store/CDN seems to play up in ways that give off some bad impressions, straight up denying requests by not responding to them at all seems unfortunate, but then if the alternative is staggeringly slow downloads, maybe it’s better overall.

I’m unsure if this is entirely related to the LXD specific congestion control, but last night I was helping an upstream project with their snap, and the gnome extension SDK snaps were throttling so hard the builds were impractical. When the resulting snap was finally created, it was pushed to a branch release and the test user couldn’t even download it. 99% of the time the store is reliable but it feels like every so often it just acts up, in this case, for a matter of hours, world wide.

It feels bad saying it while knowing I’m paying nothing and I’m probably a significant burden on these resources even by myself, but I think at least saying it is probably worthwhile just to voice the opinion. (Also thanks for the free hosting :wink: )

2 Likes

i think everyone agrees on that :slight_smile:

lxd is supposed to use cohorts in the future to prevent exactly this issue (and span a release over 90 days automatically) … for cohorts you need a key though … and the current release is the first one to support this feature … it is a catch22 …

there are millions and millions of lxd based cloud instances out there in the world and to prevent a meltdown there is manual throttling in place now … this will be a one timer though, cohorts will save us in the future.

1 Like

I proposed

I don’t own the action, so will need to wait for an owner to merge

1 Like

Which just has been!

2 Likes

I am trying to use snapcraft and get a similar error related to core20:

error: cannot refresh "core20": unexpectedly empty response from the server
       (try again later)
Error while refreshing snap 'core20' to channel 'latest/stable'
1 Like

Same issue here! Unable to build a snap.

I am still getting error: cannot refresh "core20": unexpectedly empty response from the server (try again later) very frequently. I can install core20 on the host manually without issues.

Since snapcraft is just launching the old VM (and not creating a new one), I actually do not understand why it has to download core20 again? Is core20 not available anymore? Can someone please explain what is going on here? Is all of snapcraft broken now and no one can build new snaps?

1 Like

I made a thread for this:

Looks like the answer is to just wait? This is frustrating because it is impacting my entire dev team. Is there an ETA on when the store will be running back to normal again?

There’s a client side fix upcoming, currently on Snapcraft on edge, making its way to stable in a couple of hours

Thanks for addressing this issue. I hope this will also prevent these kinds of issues for future base packages (i.e. core22, core24).

It should, that has been accounted for. There will also be a snapd/snap store feature added to allow for manual refreshes.