Hello snapcrafters,
our travis build currently fails with “SnapRevs update failed” at one of the last steps before releasing the snap in the store:
After many manual restarts of the travis build this gets surpassed and the upload works. But this can take several dozen manual restarts until this happens.
Does anyone here has any idea why this happens and how we can solve this from our side?
Best regards and thanks in advance,
Frederik
We are investigating issues related with the communitheme snap (and FWIW other snaps releasing PRs to branches). Essentially, extensive use of branches imposes an unexpected burden on channelmap operations.
We are working on immediate mitigation actions on the SnapStore side, you don’t have to change anything in your workflow. I will update this thread when the improvements are available.
Thanks for the feedback so far.
[]
Thank you very much for this fast update!
We have landed a set of mitigation actions in production SnapStore and in the last 6h the release operations became faster and more stable (across different sizes of channelmap).
Of course it’s the weekend already and we have to wait a couple of days for consistent traffic/load to call it a win.
Please let us know if you have a different perception from your travis jobs.
[]
Sadly we still can not finish the builds. Now they break with different errors than before:
Sorry, Snapcraft ran into an error when trying to running through its
lifecycle that generated a trace that has been put in ‘/tmp/tmpmc4l2dvx/trace.txt’.
"Submitting this error to the Snapcraft developers is not possible through the CLI
without Raven installed.
If you wish to report this issue, please copy the contents of the previous traceback
and submit manually at https://launchpad.net/snapcraft/+filebug.
The command “build-helpers/prepare-build-snap” exited with 1.
Thanks in advance and best regards,
Frederik
@frederik-f this last problem seems to be completely unrelated to the one originally reported. I suggest you follow the direction and file a bug report in LP with the corresponding file contents.
The first problem re appeared and it feels even worse than last week. After several manual restarts it may finnish successfully and the pr channel is updated with a new snap. But mostly it fails with the error in the opening post.
I manually restarted the build for at least 20 times and still no success: https://travis-ci.org/ubuntu/yaru/builds/425497576#L1028
We are using this feature to have snaps build out of pull requests since several months and we were using it really often with parallel PRs and it worked all the time after every commit without any problem.
Does the amount of uses of this feature somehow decreases it success rate? Meaning, did we use it too often?
That’s right – at the moment each branch slows down releases slightly. This isn’t a fundamental behaviour of the system, and it’s something we’ve always planned to improve once snaps started using lots of branches and running into problems.
We’ll roll out a quick fix early next week, and then develop a long-term fix which improves the scalability in terms of number of branches.
Thanks @wgrant and @cprov. Please keep us posted on this thread when the fix is deployed, as this is hitting us quite hard 
@didrocks another round of quick-fixes (basically relaxing internal timeouts temporarily) is effective since 3:00 UTC.
The last PR failure on communitheme [1] seems to be unrelated and the last success was 14h ago [2]. I will continue to monitor you jobs, additionally to our internal metrics, and keep the thread updated if anything goes south again.
[1] https://travis-ci.org/ubuntu/yaru/pull_requests
[2] https://travis-ci.org/ubuntu/yaru/builds/427736409
For the last few days, the “SnapRevs update failed.” issue is back, like in https://travis-ci.org/ubuntu/yaru/builds/439021584 (in particular on gtk-common-themes). Any idea if the quick fix was reverted by error?
@didrocks the workaround is still in place but it is just not effective anymore. We will work on it ASAP.
@didrocks we have just released the appropriate fix in production. Now releasing and closing operations are scoped to the affected channels and their performance will not be impacted by the number of active tracks or branches the snap has.
I’m running into this issue now, but just using the snapcraft release command - but this only happens when releasing to the seed channels - ie: stable/ubuntu-18.04. Anyone else running into this
What were the exact commands that failed, and the ones that worked?
it was snapcraft release ubuntu-budgie-welcome 122 stable that worked
when we ran
snapcraft release ubuntu-budgie-welcome 122 stable/ubuntu-18.04 we got this error (same for stable/ubuntu-18.10 stable/ubuntu-19.04 stable/ubuntu-19.10 as well)
Thanks for the details. I’ve tracked down the bug, and we’ll have it fixed this week. If you need to release something to the branches immediately, we can apply a workaround to particular branches.
no hurry … and thanks for tracking this down!
This is now fixed. Let us know if you see any more issues.
