As of launchpad-buildd 149, deployed to production on 2017-09-04, Launchpad’s build farm now uses LXD containers rather than chroots to build snaps. This applies both to snaps configured directly on Launchpad and those built via build.snapcraft.io. Let us know if you see any odd build failures that didn’t happen before and that aren’t reproducible using snapcraft cleanbuild, but so far this seems to have been fairly smooth.
The background for this is that we want to support build-snaps to allow snaps to be used as build-dependencies, and that isn’t very practical when using chroots. We’ve also switched live filesystem builds to use LXD containers in the same way, in order that they can start consuming build tools as snaps.
This work involved a lot of refactoring to convert chunks of launchpad-buildd’s build logic from ad-hoc shell scripts to unit-tested Python code, so a useful side-effect should be that future changes will be easier and safer.
Is this breaking builds? I’ve just had a report against Warzone 2100, duplicating against castersoundboard, that it is depending on ubuntu-core when attempting to install. The snap was build approximately an hour ago so I’m assuming that means it was using the LXD builders.
Ubuntu-core went away months ago afaict so the buildd shouldn’t be embedding any references to it anymore?
I’m not really sure how changing from chroots to containers could have that effect; it sounds more like the sort of thing that would be introduced at the snapcraft level, or maybe by a part. Do you know for sure that this is a new issue?
Is the krita snap built on Launchpad? I was under the impression that it was built by upstream on their infrastructure. If so, that would suggest that this is a broader issue not related to our change in build infrastructure.
This seems to be a snapd/snap-confine regression in the newly introduced handling of base snaps, very unlikely to be related to the build system at all …
This seems to be a snapd/snap-confine regression in the newly introduced handling of base snaps, very unlikely to be related to the build system at all …
That sounds plausible. Wonder why it was not caught though? Seems checks for dependent base snaps fails spectacularly given that it installs ok but won’t then run on those listed dependencies.
I know very little about snaps or snapd or even snap confinement, but it occurs to me there is some translation function from ubuntu-core > core while installing without the same corresponding function during execution.
I think I jumped the gun. It appears from the other thread that it was a local issue to @veritanuda’s system somehow getting into a weird state regarding the core snap.
Great, I just tried running an unreleased version of snapcraft against this, the log for a simple project indicates that we cannot talk to the CDN, which makes sense given that we can access the store directly from the infrastructure launchpad-buildd runs in.
I know there is a way to tell snapd to not redirect to the CDN, would that be enough and should snapcraft special case launchpad here or should launchpad configure the snapd environment for this?
Oh, right - yes, we’ll need to do something about that. @wgrant, do you think it would be reasonable to configure snapd with SNAPPY_STORE_NO_CDN=1 in LP builds? I think that’s probably easier than dealing with DNS views and firewall holes for the CDN.
My 0AD snap (built from a git mirror of the upstream project’s trunk) used to build fine but now fails with the following error when building an embedded copy of spidermonkey:
checking Python environment is Mozilla virtualenv... Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/build/0ad-svn/parts/0ad-svn-build/build/libraries/source/spidermonkey/mozjs-38.0.0/python/mozbuild/mozbuild/base.py", line 17, in <module>
from mach.mixin.process import ProcessExecutionMixin
File "/build/0ad-svn/parts/0ad-svn-build/build/libraries/source/spidermonkey/mozjs-38.0.0/python/mach/mach/mixin/process.py", line 29, in <module>
raise Exception('Could not detect environment shell!')
Exception: Could not detect environment shell!
The code that raises that exception is the following:
# Perform detection of operating system environment. This is used by command
# execution. We only do this once to save redundancy. Yes, this can fail module
# loading. That is arguably OK.
if 'SHELL' in os.environ:
_current_shell = os.environ['SHELL']
elif 'MOZILLABUILD' in os.environ:
_current_shell = os.environ['MOZILLABUILD'] + '/msys/bin/sh.exe'
elif 'COMSPEC' in os.environ:
_current_shell = os.environ['COMSPEC']
else:
raise Exception('Could not detect environment shell!')
It looks like $SHELL is not defined in the launchpad builders’ environment, I’m wondering whether that could be a regression (or an intended change maybe) introduced by the use of LXD containers?
Note that I have successfully built the snap locally in a clean LXD container, where $SHELL is defined and its value is /bin/bash.
I think this is a poor assumption for the upstream build system to make, but it’s true that lxc exec doesn’t set $SHELL for the command that it runs inside the container. Could you please file a launchpad-buildd bug to remind us to set $SHELL manually, since that probably more or less makes sense?
