Snapcraft Remote Build - Failure Regression?

snapcraft
1

Hi world,

After a over a decade of not daily driving Linux, I tried return recently with the Ubuntu 24.04 release and, unfortunately it looks like that journey will end soon because of hardware instability that makes my main computer useless for any serious work.

What I have noticed before I swap back is a very interesting quirk between Snapcraft Remote-Builds and internet connectivity, at least as far as the newer Snapcraft 8 goes.

For example:

james@James-Desktop:~/joplin-desktop-snap$ snapcraft remote-build
All data sent to remote builders will be publicly available. Are you sure you want to continue? [y/N]: y
remote-build is experimental and is subject to change. Use with caution.                                                                                                                                                                               
The authorization page:
 (*REDACTED*)
should be opening in your browser. Use your browser to authorize
this program to access Launchpad on your behalf.
Waiting to hear from Launchpad about your decision...
(*about an hour of generic processing*)
snapcraft internal error: gaierror(-3, 'Temporary failure in name resolution')                                                                                                                                                                         
Full execution log: '/home/james/.local/state/snapcraft/log/snapcraft-20240523-125416.424344.log'

So, one of my instabilities is with the wifi driver. (Classic!). Although the speed of the connection is generally good (coming in at half that at Windows but a solid 5 times faster than the UK average), I randomly lose all connections at once, constantly, every 10-15 minutes. The connections begin working again 20 seconds later, and generally things move on with some curses about why my video is buffering.

However not Snapcraft, I’ve been trying to use remote build to patch a security bug, and every single time on a long enough build I’m not able to finish it.

Older versions of Snapcraft used to provide a build ID that could recover a lost build session. However the new behaviour, in this specific set of circumstances, doesn’t recover the build but restart it entirely.

This is now the third time I’ve waited an hour to have a wifi blip for 20 seconds and Launchpad/Snapcraft delete the previous build entirely rather than continuing it.

Is this the expected behaviour? It feels like a relative niche but the relative consequences here could be dire depending upon situations, and the old behaviour generally never had a problem.

2

@lengau might be able to help more. My initial suspicion is that the reason you cannot recover is because the project/repo/recipe was not successfully created in order to have a build id.

1 Like
3

I’m not super solid with the build-id behaviour, but for what it’s worth, when the wifi doesn’t drop, I do get usable results. When the wifi drops, I can even see the build continuing on Launchpad. I don’t believe it’s until the second invocation of snapcraft remote-build where Launchpad suddenly loses the old build and begins generating a new one.

I’ll try and create this error and leave the store build in place if that helps work out what’s going on :).

Edit: snapcraft-joplin-desktop-50fb6c1a57b74e6846330cc8bea56206 : Snap packages : James Carroll has just been submitted, I’ll let you know if the correct circumstances happen or keep trying until it does (in my instance it just seems to be duration will eventually ensure a failure, given enough dice are being thrown, one will eventually land).

4

An update:

The AMD64 build actually built fine that run and didn’t happen to be hit by wifi dropouts at inconvenient times. (Which is a shame, I actually needed that).

I’ve uploaded the ARM64 release just now which takes significantly longer. Hopefully this will produce the error (or alternatively at least I’ll be able to upload the result).

What did strike me as curious is that rather than appear on Launchpad with a new URL, it actually just replaced the existing URL for the AMD64 run entirely. I’m not sure if that actually is any different from previous behaviour but it stood out to me.

(To be clear, I’d submitted these as 2 different builds because of a regression with the --build-for= syntax returning).

5

Aaaand I’ve caught one!

Though it’s a bit different to what I expected was happening. I’d submitted a build, had an afternoon siesta, came back, and have the same error as originally.

I’d then checked the Launchpad build (still snapcraft-joplin-desktop-50fb6c1a57b74e6846330cc8bea56206 : Snap packages : James Carroll) - and it’s restarted building before I’d issued the second snap request.

It feels like in the last few seconds of snapcraft crashing due to the internet instability, it still manages to get a single packet out causing complications.

More explicitly will have been the machine running in the X11 lockscreen whereupon the restart of the build doesn’t happen until I enter the password to log back into the session. The restart coincides with logging back into the machine; rather than whenever during my nap the error actually occurred.