To build the Certbot snaps maintained by the “Certbot Project” we make heavy use of snapcraft remote-build. One problem we’ve seen with increasing frequency lately are builds indefinitely hanging in the “Currently building” state. When this happens, the build log shown on launchpad always looks like this.
We saw this problem for the first time back in January, but have seen it 3 times in the last week. We build on amd64, arm64, and armhf and have seen the problem on both ARM architectures. We haven’t seen it on amd64 (yet).
What can we do to fix this problem? Is this a bug in snapd/launchpad that I should report or track elsewhere?
Another option that’d help us would be to add a timeout option to snapcraft remote-build to have it delete the builds after this timeout has been reached if something like this is supported by launchpad’s API. Once our builds get in this state, they remain stuck there and future runs of snapcraft remote-build will reconnect to them until our source code changes or we manually delete the build off launchpad. I do not believe snapcraft remote-build's current --timeout option works for us because its help text says “Note that the build will continue on Launchpad and can be resumed later.”
Any help in fixing this problem is much appreciated.
Looks like it may be some sort of hang with snapd given the last message of 2021-01-25T04:38:26Z INFO Waiting for automatic snapd restart.... @sergiusens@ijohnson any thoughts?
Without more logs, I’m inclined to punt to the store and assume it’s probably fallout from the store issues this week. If you have more logs from what snapd was trying to do like journalctl --no-pager -u snapd or snap changes or even the full system journal maybe I could say more, but without more logs it’s unclear what the issue is
Unfortunately I don’t see a way for me to get more logs at least without modifying snapcraft. This the build log created from running snapcraft remote-build and I believe it hangs before it even starts using my snapcraft.yaml file.
We’re still seeing this problem sometimes. Can anything be done to fix it? I’m happy to provide more logs from launchpad/snapcraft remote-build, but like I said before, I’m not sure how to get the logs you want without at least modifying snapcraft’s source code somewhere.
