504 timeouts on snap upload

Occasionally I’ll publish snaps to the store, and the store will return error 504. While simply resubmitting the snap isn’t much of a problem, I’m curious as to whether there’s any identifiable reason as to what the error is, it’s just mildly irritating. And since I’m naturally lazy, this often happens in the context of CI where the entire build process gets repeated, often for hours at a time.

This mostly happens with one of my snaps in particular, it’s a fairly large snap (1.3GB) but smaller than some other snaps that don’t seem to suffer as badly. The snap is called root-framework, and with the nightly builds, about 10% of them just never deploy because the store returns an error.

Occasionally however, there’ll be scenarios like today where the package needs rebuilding to pull in security updates, and the same track fails to publish repeatedly (at the time of writing this, we’re on attempt 4!), e.g:

image image image

These are all cases of the build going fine, but the store rejecting the submission after reaching 100% upload.

This isn’t an isolated incident, it’s been happening for a long time, but I thought it might be worth bringing it up just incase there is something that can be done on the store side. Otherwise, maybe Snapcraft could be updated to retry uploads a few times before aborting & etc.

Is there anything that stands out as problematic with large snaps or this snap in particular?

@roadmr can you provide insight into this issue?

As an update to this, although the problem had been prevalent for months to years when I’d written this originally, the stability of the upload process has increased significantly recently, and I haven’t had a single upload failure for the same snap (or any other snaps) which would usually be as high as 10% of uploads being rejected.

So I’m unsure if there’s been anything targetted specifically for this kind of problem, but whether it’s a change on the snapcraft side or the store side, the problem seems to have effectively disappeared on a fairly large sample size (given nightlies, security rebuilds, etc).

1 Like