@zyga-snapd thanks a lot for putting this together! 
WRT to spread-cron https://travis-ci.org/snapcore/spread-cron/branches we have currently 2 scenarios consistently failing:
-
snapd-reexec-2.21-vs-edge, the test is triggered after each core publication to edge and sets up the suite to download snapd 2.21, install an unmodified core snap from edge, this is done here https://github.com/snapcore/spread-cron/blob/snapd-reexec-2.21-vs-edge/run-checks and this is the last execution https://travis-ci.org/snapcore/spread-cron/builds/219549331
-
snapd-nightly-suite: this scenario runs the nightly suite in from snapd project, is executed daily this is the last failed execution https://travis-ci.org/snapcore/spread-cron/builds/219525601 and will be hopefully fixed once https://github.com/snapcore/snapd/pull/3104 lands.
I also wanted to mention a really elusive, flaky test that hits us eventually in snapd’s suite, sometimes tests/main/refresh-delta-from-core fails like in this execution: Ubuntu Pastebin
According to this part of the log maybe a retry is involved:
Apr 05 09:11:26 ubuntu /usr/lib/snapd/snapd[25130]: daemon.go:176: DEBUG: uid=0;@ GET /v2/snaps?snaps=test-snapd-delta-refresh 788.216µs 200
Apr 05 09:11:26 ubuntu /usr/lib/snapd/snapd[25130]: retry.go:47: DEBUG: Retrying https://assertions.ubuntu.com/v1/assertions/snap-declaration/16/99T7MUlRhtI3U0QFgl5mXXESAiSwt776?max-format=2, attempt 1, elapsed time=12.365µs
Apr 05 09:11:27 ubuntu /usr/lib/snapd/snapd[25130]: daemon.go:176: DEBUG: uid=0;@ GET /v2/snaps?snaps=test-snapd-delta-refresh 685.31µs 200
Cheers!