Any way to see "full execution log" on Launchpad builder?

I’m trying to build a snap with a Rust part - this works when I build locally, but fails on the Launchpad builder. The snapcraft version appears to be 7.0.11on both, so I’m not sure what’s happening.

An example build log is here: https://launchpadlibrarian.net/613728839/buildlog_snap_ubuntu_jammy_arm64_908277a77436b2b858d32d6cd0bc859a_BUILDING.txt.gz (and FWIW the source for the snap is here). Unfortunately it terminates with

Executing parts lifecycle: build ae                                            [19/Jul/2022:11:44:45 +0000] "CONNECT crates.io:443 HTTP/1.1" 200 80783 "-" "cargo 1.59.0"
[19/Jul/2022:11:44:45 +0000] "CONNECT crates.io:443 HTTP/1.1" 200 6446 "-" "cargo 1.59.0"
[19/Jul/2022:11:44:45 +0000] "CONNECT static.crates.io:443 HTTP/1.1" 200 10230377 "-" "cargo 1.59.0"
[19/Jul/2022:11:44:45 +0000] "CONNECT static.crates.io:443 HTTP/1.1" 200 24229 "-" "cargo 1.59.0"
[19/Jul/2022:11:44:45 +0000] "CONNECT github.com:443 HTTP/1.1" 200 82851254 "-" "git/1.0 (git2-curl 0.14.1)"

Failed to run the build script for part 'ae'.                                  
Full execution log: '/root/.cache/snapcraft/log/snapcraft-20220719-114314.355655.log'                                                                          
Build failed

Is there any way for me to access the full execution log referenced at the end? Or trigger the build so this is somehow exposed in the “main” build log? I’m not really sure how to troubleshoot further without this…

(I wrote a bit more about my attempts to build the snap here: Correct snapcraft syntax for Rust-based snap on core22. That was more in the context of “how do I build a Rust snap?” rather than “how do I see the build logs?” so hopefully not a duplicate…)

1 Like

This is unfortunately starting to drive me up the wall a bit. After some iteration I figured that a missing dependency (curl) wasn’t available and adding it to build-packages fixed the immediate issue.

But now I’m running into a similar error for another (more complex) part, and all I have to go on is

Failed to run the build script for part 'ammp-edge'.                           
Full execution log: '/root/.cache/snapcraft/log/snapcraft-20220720-111643.827022.log'                                                                          
Build failed

This is virtually impossible to troubleshoot without seeing the actual log. Is there any way to access this? (Again, local build works just fine for this).

Hi there @svet, could you provide the commands you are using to launch the remote build? For example, are you using the snapcraft remote-build command?

Thanks for helping @nuccitheboss! The examples above are from Launchpad builds that were triggered via a Github hook from the repo. I’ve had a go with snapcraft remote-build as well, with very similar results. Namely, on my machine that process exits with

...
Build status as of 2022-07-20 16:15:15.395475:
	arch=amd64	state=Currently building
Build status as of 2022-07-20 16:15:45.989201:
	arch=amd64	state=Failed to build
Snap file not available for arch 'amd64'.
Build log available at 'ammp-edge_amd64.txt'
Build failed for arch 'amd64'.
Build complete.

The ammp-edge_amd64.txt that then appears on my machine has equivalent contents to the build log available on the Launchpad builder. So quite a lengthy pull/build log, that ultimately terminates with namely something like

Failed to run the build script for part 'ammp-edge'.                           
Full execution log: '/root/.cache/snapcraft/log/snapcraft-20220720-161332.318625.log'                                                                          
Build failed

and then a traceback. Now, the /root/.cache/snapcraft/log/snapcraft-20220720-161332.318625.log is what I’d really like to look at - since I want to see what’s actually causing the error. But AFAICT there isn’t a way for me to access this.

I did note that if I run the snapcraft build locally, I can pass a -v flag to get verbose output directly on stdout (rather than locked away in a log file), but I don’t see a way to do that for a remote build.

[Also for the avoidance of doubt, the messages immediately preceding the error are routine PyPI connection messages (since it’s a Python part that’s being built) - those are all successful and offer no clues as to the reason for the error]

Hmmmm. Interesting. From what I can gather here, it looks like you are being given an abridged log for your remote build.

Right off the bat, the easiest way I could see you getting the full Launchpad build log is to use the builder service provided by the Snap Store. For example, I have my snapcrafters repository (still a work in progress) connected to the snap store.

Once I start a build, I usually just click on the ID # which gives me the build log:

If I want to copy the whole log, I just click “View Raw” and then Crtl+A Crtl+C into a local file on my machine

You can also download the log file locally using curl using the long URL you are given by the librarian.

Let me know if this helps you at all. I think there is way you can directly build snaps through your Launchpad account, but I haven’t looked deeply enough into it.

It is a known issue, which I believe is being worked on right now, that the LP builder is unable to pull the log that Snapcraft creates when building a core22 snap package. Earlier core* builds will still expose the log in the LP output as it always has.

Snapcraft was changed when core22 was added to save the log to a file rather than spew it onto the console - LP only records the console output so this log file is not included in the LP logs. This only affects core22 builds because the earlier bases use a different code-path to maintain backwards compatibility.

2 Likes

Thanks for the explanation @lucyllewy. Not quite what I was hoping to hear, but I guess I’ll have to love with it for now and try to find some other creative way to troubleshoot. I suppose a workaround for now is to build locally and push to the store manually, rather than using CI and the Launchpad builders. (And this was indeed not a problem before I switched to core22)

@nuccitheboss thanks for the tips. I think the workflow you describe is basically what I’m following, and the hyperlinked log is what I’m looking at - but it’s basically not the “full execution log”, which is saved elsewhere.

I appreciate your time helping look into this!

1 Like

By the way @lucyllewy do you know where I can track this issue? I quickly checked out the launchpad bug tracker but couldn’t find anything relevant.

We’ve put this snap on hold for now due to the issue but would be keen to restart when it’s solved.

@svet I believe the issue is being tracked internally rather than on the Launchpad bug tracker, but the snapcraft team is working on the issue.

I think it is slated to be part of the next bug fix release of snapcraft given that the recent call for testing added more supported for building core22 desktop snaps with the addition of the GNOME 42 extension. You can track the changelog at the snapcore/snapcraft on GitHub. I also do believe the snapcraft team will post a call for testing here on the Forum once the next release is ready.

1 Like

It’s being tracked here https://bugs.launchpad.net/snapcraft/+bug/1981072

2 Likes

I’m running into the same issue.

I saw that something was merged back in August regarding this:

It allows users to run snapcraft like this:

SNAPCRAFT_ENABLE_DEVELOPER_DEBUG=1 snapcraft

Everything will be output to the standard output. The problem is that these create extremely verbose logs (almost 90 000 lines in my case).

I’ve also seen this Launchpad issue, where a fix was committed a month ago (but not yet available in the Snapcraft 7.x stable channel): https://bugs.launchpad.net/snapcraft/+bug/1980515

I’m not sure if there is an option to see enough logs without going into DEBUG mode (with SNAPCRAFT_ENABLE_DEVELOPER_DEBUG environment variable). Does anyone know?