Build.snapcraft - stage-package not found

bsutton · February 21, 2018, 7:29am

I have a snap which uses an apt package ‘certbot’.

Certbot isn’t in the default 16.04 repositories so needs to be added manually.

does this mean I can’t use the build server or is there some way to specify these alternate repos?
I’m guessing I can’t do this myself as it sounds like a security risk.

Brett

ogra · February 21, 2018, 9:17am

You have several options:

use a scriptlet to wget/curl the package from a non-archive place and “dpkg -x” it (indeed this requires that the binary deb was built against 16.04)
use a scriptlet to add a PPA to the build host and install the package from the PPA (taints the build host indeed)
build certbot from source in its own snapcraft.yaml part

(all of the above have been discussed before in this forum, look around a little and you should find some instructions)

bsutton · February 22, 2018, 2:58am

I’m using option two which works fine on my local box, but how do I make this work on the build server.

parts:
  tomcat: 
    ...
    prepare: |
      sudo apt-get update
      sudo apt-get install software-properties-common
      sudo add-apt-repository --yes ppa:certbot/certbot
      sudo apt-get update
      sudo apt-get install --yes certbot

apt-get install requires sudo?

If I run snapcraft with sudo in the script it just asks me for a password which is fine. But thats not going to work on the build server.

lucyllewy · February 22, 2018, 3:09pm

The build server runs everything as root, so you can remove the sudo calls and it’ll work fine.

ogra · February 22, 2018, 4:31pm

and at the risk of sounding like a broken record … snapcraft cleanbuild does too

lucyllewy · February 22, 2018, 4:43pm

Break more records! Break ALL the records! :-p

bsutton · February 22, 2018, 11:14pm

no records broken.
I’m going to give the undocumented lcx build option that use the environment var.

bsutton · February 22, 2018, 11:16pm

I’ve removed the sudo’s and the build server is still failing.
It appears that its not running the prepare script.
On my localhost the prepare scripts run and install certbot.
On the build server it says that certbot isn’t found and the logs give no hint that it was installed?

Can you see the above link?

lucyllewy · February 23, 2018, 12:13am

It is running the script, but prepare is run after the stage packages are collected so it can’t find certbot at the time it’s looking for it. The solution, while hacky, is to move the addition of the repo to a second part which you set your installation to run after:

parts:
  setup-repo:
    prepare: |
      add-apt-repository ...
      apt-get update
      apt-get upgrade
      apt-get install -yqq certbot

  my-part:
    after: [setup-repo]
    stage-packages: [certbot]
    ...

bsutton · February 23, 2018, 1:44am

This still failed on the build server, however it appears that it works on my local lxd install.

reparing to pull setup-repo
Pulling setup-repo
‘getcert’ has prerequisites that need to be staged: setup-repo
Preparing to build setup-repo
Building setup-repo
Hit:1 http://ppa.launchpad.net/snappy-dev/tools/ubuntu xenial InRelease
Hit:2 http://ftpmaster.internal/ubuntu xenial InRelease
Hit:3 http://ftpmaster.internal/ubuntu xenial-security InRelease
Hit:4 http://ftpmaster.internal/ubuntu xenial-updates InRelease
Reading package lists…
Reading package lists…
Building dependency tree…
Reading state information…
The following packages were automatically installed and are no longer required:
ant ca-certificates-java default-jre-headless fontconfig-config
fonts-dejavu-core java-common libavahi-client3 libavahi-common-data
libavahi-common3 libcups2 libfontconfig1 libfreetype6 libjpeg-turbo8
libjpeg8 liblcms2-2 libnspr4 libnss3 libnss3-nssdb libx11-6 libx11-data
libxau6 libxcb1 libxdmcp6 libxext6 libxi6 libxrender1 libxtst6
openjdk-8-jre-headless ucf x11-common
Use ‘apt autoremove’ to remove them.
The following additional packages will be installed:
distro-info-data gir1.2-glib-2.0 iso-codes libdbus-glib-1-2
libgirepository-1.0-1 libglib2.0-0 lsb-release python3-dbus python3-gi
python3-pycurl python3-software-properties
Suggested packages:
isoquery lsb python-dbus-doc python3-dbus-dbg libcurl4-gnutls-dev
python-pycurl-doc python3-pycurl-dbg
Recommended packages:
libglib2.0-data shared-mime-info xdg-user-dirs unattended-upgrades
The following NEW packages will be installed:
distro-info-data gir1.2-glib-2.0 iso-codes libdbus-glib-1-2
libgirepository-1.0-1 libglib2.0-0 lsb-release python3-dbus python3-gi
python3-pycurl python3-software-properties software-properties-common
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 4091 kB of archives.
After this operation, 23.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] Abort.
Traceback (most recent call last):
File “/usr/bin/snapcraft”, line 9, in
load_entry_point(‘snapcraft==2.35’, ‘console_scripts’, ‘snapcraft’)()
File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 542, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2569, in load_entry_point
return ep.load()
File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2229, in load
return self.resolve()
File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2235, in resolve
module = import(self.module_name, fromlist=[‘name’], level=0)
File “/usr/lib/python3/dist-packages/snapcraft/cli/main.py”, line 19, in
run(prog_name=‘snapcraft’)
File “/usr/lib/python3/dist-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/usr/lib/python3/dist-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/usr/lib/python3/dist-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/lib/python3/dist-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/lib/python3/dist-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/usr/lib/python3/dist-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/lib/python3/dist-packages/snapcraft/cli/lifecycle.py”, line 67, in pull
_execute(‘pull’, parts, **kwargs)
File “/usr/lib/python3/dist-packages/snapcraft/cli/lifecycle.py”, line 34, in _execute
lifecycle.execute(command, project_options, parts)
File “/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py”, line 80, in execute
_Executor(config, project_options).run(step, part_names)
File “/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py”, line 175, in run
self._run_step(step, part, part_names)
File “/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py”, line 201, in _run_step
self.run(‘stage’, unstaged_prereqs)
File “/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py”, line 175, in run
self._run_step(step, part, part_names)
File “/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py”, line 212, in _run_step
getattr(part, step)()
File “/usr/lib/python3/dist-packages/snapcraft/internal/pluginhandler/init.py”, line 325, in build
script_runner.run(scriptlet=self._part_properties.get(‘prepare’))
File “/usr/lib/python3/dist-packages/snapcraft/internal/pluginhandler/_scriptlets.py”, line 43, in run
run([scriptlet_path], cwd=self._builddir)
File “/usr/lib/python3/dist-packages/snapcraft/internal/common.py”, line 63, in run
subprocess.check_call([’/bin/sh’, f.name] + cmd, **kwargs)
File “/usr/lib/python3.5/subprocess.py”, line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[’/bin/sh’, ‘/tmp/tmp94j_yxd9’, ‘/tmp/tmpfjhndf3z’]’ returned non-zero exit status 1
Build failed
Traceback (most recent call last):
File “/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py”, line 225, in run
self.pull()
File “/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py”, line 184, in pull
env=env)
File “/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py”, line 88, in run_build_command
return self.backend.run(args, cwd=cwd, env=full_env, **kwargs)
File “/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py”, line 448, in run
subprocess.check_call(cmd, **kwargs)
File “/usr/lib/python2.7/subprocess.py”, line 541, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command ‘[‘lxc’, ‘exec’, ‘lp-xenial-i386’, ‘–env’, ‘LANG=C.UTF-8’, ‘–env’, ‘SHELL=/bin/sh’, ‘–env’, ‘SNAPCRAFT_LOCAL_SOURCES=1’, ‘–env’, ‘SNAPCRAFT_SETUP_CORE=1’, ‘–env’, ‘SNAPCRAFT_BUILD_INFO=1’, ‘–env’, ‘http_proxy=http://snap-proxy.launchpad.net:3128’, ‘–env’, ‘https_proxy=http://snap-proxy.launchpad.net:3128’, ‘–env’, ‘GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy’, ‘–’, ‘/bin/sh’, ‘-c’, ‘cd /build/e4039a0b32e4bca67d290294684c86d6-xenial && linux32 snapcraft pull’]’ returned non-zero exit status 1
Revoking proxy token…
RUN: /usr/share/launchpad-buildd/slavebin/in-target scan-for-processes --backend=lxd --series=xenial --arch=i386 SNAPBUILD-154317
Scanning for processes to kill in build SNAPBUILD-154317
RUN: /usr/share/launchpad-buildd/slavebin/in-target umount-chroot --backend=lxd --series=xenial --arch=i386 SNAPBUILD-154317
Stopping target for build SNAPBUILD-154317
RUN: /usr/share/launchpad-buildd/slavebin/in-target remove-build --backend=lxd --series=xenial --arch=i386 SNAPBUILD-154317

bsutton · February 23, 2018, 1:45am

FYI the actual entry I used is:

setup-repo:
  plugin: nil
  prepare: |
    apt-get update
    apt-get install software-properties-common
    add-apt-repository --yes ppa:certbot/certbot
    apt-get update
    apt-get install --yes certbot