Launchpad now builds snaps in LXD containers, in support of "build-snaps"

As of launchpad-buildd 149, deployed to production on 2017-09-04, Launchpad’s build farm now uses LXD containers rather than chroots to build snaps. This applies both to snaps configured directly on Launchpad and those built via build.snapcraft.io. Let us know if you see any odd build failures that didn’t happen before and that aren’t reproducible using snapcraft cleanbuild, but so far this seems to have been fairly smooth.

The background for this is that we want to support build-snaps to allow snaps to be used as build-dependencies, and that isn’t very practical when using chroots. We’ve also switched live filesystem builds to use LXD containers in the same way, in order that they can start consuming build tools as snaps.

This work involved a lot of refactoring to convert chunks of launchpad-buildd’s build logic from ad-hoc shell scripts to unit-tested Python code, so a useful side-effect should be that future changes will be easier and safer.

6 Likes

Is this breaking builds? I’ve just had a report against Warzone 2100, duplicating against castersoundboard, that it is depending on ubuntu-core when attempting to install. The snap was build approximately an hour ago so I’m assuming that means it was using the LXD builders.

Ubuntu-core went away months ago afaict so the buildd shouldn’t be embedding any references to it anymore?

Sad to say it also applies to Krita.

krita
cannot locate the base snap: ubuntu-core: No such file or directory

I wonder how many other snaps are broken?

I’m not really sure how changing from chroots to containers could have that effect; it sounds more like the sort of thing that would be introduced at the snapcraft level, or maybe by a part. Do you know for sure that this is a new issue?

Is the krita snap built on Launchpad? I was under the impression that it was built by upstream on their infrastructure. If so, that would suggest that this is a broader issue not related to our change in build infrastructure.

This seems to be a snapd/snap-confine regression in the newly introduced handling of base snaps, very unlikely to be related to the build system at all …

This seems to be a snapd/snap-confine regression in the newly introduced handling of base snaps, very unlikely to be related to the build system at all …

That sounds plausible. Wonder why it was not caught though? Seems checks for dependent base snaps fails spectacularly given that it installs ok but won’t then run on those listed dependencies.

I know very little about snaps or snapd or even snap confinement, but it occurs to me there is some translation function from ubuntu-core > core while installing without the same corresponding function during execution.

Oh well.

If you are affected can you please give more details about your system? I cannot reproduce this in any way so far.

Bit of deja-vu here but of course.

snap version
snap 2.27.4-1
snapd 2.27.4-1
series 16
debian unknown
kernel 4.12.0-1-amd64

and

snap list
Name Version Rev Developer Notes
castersoundboard 0+git.c41a4f2 34 rikai -
core 16-2.26.14 2462 canonical core,devmode
krita 3.2.0 26 krita -
warzone2100 3.2.3 5 diddledan -

I think I jumped the gun. It appears from the other thread that it was a local issue to @veritanuda’s system somehow getting into a weird state regarding the core snap.

Yes… Turns out that core had an update that sorted things out but no clue as to why snap refresh did not do that automatically.

Bottom line is with core 16-2.27.5 Rev 2774

All is working as it should.

Sorry if this caused some confusion but you can’t really blame me given the unhelpful error messages I was experiencing.

1 Like

No blaming and thanks for the helpful replies you have been giving to help narrow down the issue. We will continue to look into it.

Great, I just tried running an unreleased version of snapcraft against this, the log for a simple project indicates that we cannot talk to the CDN, which makes sense given that we can access the store directly from the infrastructure launchpad-buildd runs in.

I know there is a way to tell snapd to not redirect to the CDN, would that be enough and should snapcraft special case launchpad here or should launchpad configure the snapd environment for this?

1 Like

Oh, right - yes, we’ll need to do something about that. @wgrant, do you think it would be reasonable to configure snapd with SNAPPY_STORE_NO_CDN=1 in LP builds? I think that’s probably easier than dealing with DNS views and firewall holes for the CDN.

1 Like

We should definitely set SNAPPY_STORE_NO_CDN; we can’t allow firewall access to the CDN from the buildds.

1 Like

This should work for you now; I got it working on dogfood earlier and we’ve now rolled that out to production. Thanks for reporting the problem.

Thanks, just wanted to follow up that it worked https://code.launchpad.net/~sergiusens/+snap/build-snaps-test/+build/79201

My 0AD snap (built from a git mirror of the upstream project’s trunk) used to build fine but now fails with the following error when building an embedded copy of spidermonkey:

checking Python environment is Mozilla virtualenv... Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/build/0ad-svn/parts/0ad-svn-build/build/libraries/source/spidermonkey/mozjs-38.0.0/python/mozbuild/mozbuild/base.py", line 17, in <module>
    from mach.mixin.process import ProcessExecutionMixin
  File "/build/0ad-svn/parts/0ad-svn-build/build/libraries/source/spidermonkey/mozjs-38.0.0/python/mach/mach/mixin/process.py", line 29, in <module>
    raise Exception('Could not detect environment shell!')
Exception: Could not detect environment shell!

The code that raises that exception is the following:

# Perform detection of operating system environment. This is used by command
# execution. We only do this once to save redundancy. Yes, this can fail module
# loading. That is arguably OK.
if 'SHELL' in os.environ:
    _current_shell = os.environ['SHELL']
elif 'MOZILLABUILD' in os.environ:
    _current_shell = os.environ['MOZILLABUILD'] + '/msys/bin/sh.exe'
elif 'COMSPEC' in os.environ:
    _current_shell = os.environ['COMSPEC']
else:
    raise Exception('Could not detect environment shell!')

It looks like $SHELL is not defined in the launchpad builders’ environment, I’m wondering whether that could be a regression (or an intended change maybe) introduced by the use of LXD containers?

Note that I have successfully built the snap locally in a clean LXD container, where $SHELL is defined and its value is /bin/bash.

Corresponding build: https://code.launchpad.net/~osomon/+snap/0ad-svn/+build/79425.

I think this is a poor assumption for the upstream build system to make, but it’s true that lxc exec doesn’t set $SHELL for the command that it runs inside the container. Could you please file a launchpad-buildd bug to remind us to set $SHELL manually, since that probably more or less makes sense?

1 Like

Thanks!

I agree that this is a poor assumption, but there it is, and I’d rather not have to patch the source to fix the build.

Here is the bug report: bug #1716739.

2 Likes