Cannot decode new commands catalog - unexpected 403 error

Over the last couple of day’s we’ve seen the below error in our logs over multiple systems, I’ve been unable to replicate running any specific commands. However something is triggering it (semi-regularly).

snapd[629]: 2018/11/19 16:50:33.212210 stateengine.go:101: state ensure error: cannot decode new commands catalog: got unexpected HTTP status code 403 via GET to “https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic

Has anyone else seen this? Are they any extra steps I can take to debug?

snapd will regularly (once every 24 hours) attempt a “catalog refresh”, which is an update of the databases it uses for quick tab completion of stable snaps’ names (for snap install), commands (for command-not-found integration) and sections (for snap find --section=...).

The store was struggling yesterday, so it shut down that endpoint, returning a plain 403 before hitting the appserver. It took longer than usual to come recover, so it was (I think, I don’t have numbers yet) a little over 24 hours away so if you were unlucky you might have two failures in any given machine. It should be back to being OK today. If there was something you were expecting to be cached that isn’t, restarting snapd will trigger a refresh as well.

HTH, HAND.

That would make sense, thank you.
When we first saw this issue I did check this status page but it didn’t mention this at all. :frowning: I see it’s now been update 48hrs later.