We are investigating issues related with the communitheme snap (and FWIW other snaps releasing PRs to branches). Essentially, extensive use of branches imposes an unexpected burden on channelmap operations.
We are working on immediate mitigation actions on the SnapStore side, you don’t have to change anything in your workflow. I will update this thread when the improvements are available.
Sadly we still can not finish the builds. Now they break with different errors than before:
Sorry, Snapcraft ran into an error when trying to running through its
lifecycle that generated a trace that has been put in ‘/tmp/tmpmc4l2dvx/trace.txt’.
"Submitting this error to the Snapcraft developers is not possible through the CLI
without Raven installed.
If you wish to report this issue, please copy the contents of the previous traceback
and submit manually at https://launchpad.net/snapcraft/+filebug.
The command “build-helpers/prepare-build-snap” exited with 1.
The first problem re appeared and it feels even worse than last week. After several manual restarts it may finnish successfully and the pr channel is updated with a new snap. But mostly it fails with the error in the opening post.
I manually restarted the build for at least 20 times and still no success: https://travis-ci.org/ubuntu/yaru/builds/425497576#L1028
We are using this feature to have snaps build out of pull requests since several months and we were using it really often with parallel PRs and it worked all the time after every commit without any problem.
Does the amount of uses of this feature somehow decreases it success rate? Meaning, did we use it too often?
That’s right – at the moment each branch slows down releases slightly. This isn’t a fundamental behaviour of the system, and it’s something we’ve always planned to improve once snaps started using lots of branches and running into problems.
We’ll roll out a quick fix early next week, and then develop a long-term fix which improves the scalability in terms of number of branches.
@didrocks another round of quick-fixes (basically relaxing internal timeouts temporarily) is effective since 3:00 UTC.
The last PR failure on communitheme  seems to be unrelated and the last success was 14h ago . I will continue to monitor you jobs, additionally to our internal metrics, and keep the thread updated if anything goes south again.
@didrocks we have just released the appropriate fix in production. Now releasing and closing operations are scoped to the affected channels and their performance will not be impacted by the number of active tracks or branches the snap has.