Help packaging swift compiler in a snap


#21

Thank you,

Ok, so I can create a removeexecstack.sh, add execstack -c $SNAPCRAFT_PART_INSTALL/usr/lib/arm-linux-gnueabihf/libBlocksRuntime.so.0.0.0 into it, and finally add a build-packages: into the yaml like so:

parts:
  swift:
    plugin: dump
    source: https://swift.org/builds/swift-4.0.3-release/ubuntu1604/swift-4.0.3-RELEASE/swift-4.0.3-RELEASE-ubuntu16.04.tar.gz
    stage-packages:
      - clang
      - libblocksruntime0
      - libbsd0
      - libedit2
      - libicu-dev          # Yes, the -dev package is deliberately staged.
      - libncursesw5
      - libpython2.7
      - libsqlite3-0amd64
      - libuutil1linux
      - libxml2
      - python2.7
    build-packages:
      - removeexecstack.sh
  • Now, where I place this script within the filesystem?

#22

This wouldnt work … build-packages: only takes names of deb packages from the archive …

Here is the documentation about scriptlets:

https://docs.snapcraft.io/build-snaps/scriptlets

For your case the install: | scriptlet seems right …


#23

It seems execstack is not installed in the linux container created…

Building swift 
/tmp/tmp6u_1e98d: 2: /tmp/tmp6u_1e98d: execstack: not found
Traceback (most recent call last):
  File "/usr/bin/snapcraft", line 9, in <module>
    load_entry_point('snapcraft==2.35', 'console_scripts', 'snapcraft')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/__main__.py", line 19, in <module>
    run(prog_name='snapcraft')
  File "/usr/lib/python3/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1037, in invoke
    return Command.invoke(self, ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/__init__.py", line 124, in run
    ctx.forward(lifecyclecli.commands['snap'])
  File "/usr/lib/python3/dist-packages/click/core.py", line 552, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/lifecycle.py", line 140, in snap
    project_options, directory=directory, output=output)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_packer.py", line 45, in snap
    execute('prime', project_options)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 80, in execute
    _Executor(config, project_options).run(step, part_names)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 175, in run
    self._run_step(step, part, part_names)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 212, in _run_step
    getattr(part, step)()
  File "/usr/lib/python3/dist-packages/snapcraft/internal/pluginhandler/__init__.py", line 331, in build
    script_runner.run(scriptlet=self._part_properties.get('install'))
  File "/usr/lib/python3/dist-packages/snapcraft/internal/pluginhandler/_scriptlets.py", line 43, in run
    run([scriptlet_path], cwd=self._builddir)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/common.py", line 63, in run
    subprocess.check_call(['/bin/sh', f.name] + cmd, **kwargs)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/sh', '/tmp/tmpy5dc80a3', '/tmp/tmp6u_1e98d']' returned non-zero exit status 127
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 226, in run
    self.build()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 200, in build
    env=env)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 88, in run_build_command
    return self.backend.run(args, cwd=cwd, env=full_env, **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 448, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-xenial-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'http_proxy=http://snap-proxy.launchpad.net:3128', '--env', 'https_proxy=http://snap-proxy.launchpad.net:3128', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/f3f03c80e7fff860ec06266b8e1d075d-xenial && linux64 snapcraft']' returned non-zero exit status 1
Revoking proxy token...
RUN: /usr/share/launchpad-buildd/slavebin/in-target scan-for-processes --backend=lxd --series=xenial --arch=amd64 SNAPBUILD-152104
Scanning for processes to kill in build SNAPBUILD-152104
RUN: /usr/share/launchpad-buildd/slavebin/in-target umount-chroot --backend=lxd --series=xenial --arch=amd64 SNAPBUILD-152104
Stopping target for build SNAPBUILD-152104
RUN: /usr/share/launchpad-buildd/slavebin/in-target remove-build --backend=lxd --series=xenial --arch=amd64 SNAPBUILD-152104


#24

did you add it to build-packages ?


#25

Thank you,

It has improved, there is, however, some problem with the relative path. I see you used the variable $SNAPCRAFT_PART_INSTALL, where can I find documentation for these environment variables?

The error:

execstack: cannot open "/build/f3f03c80e7fff860ec06266b8e1d075d-xenial/parts/swift/install/usr/lib/arm-linux-gnueabihf/libBlocksRuntime.so.0.0.0": No such file or directory

#26

this seems to be an arm build (sorry for not noticing earlier), are you sure the swift binary yòu are downloading via source can even run on armhf ?

to make the package multi-arch capable that lib path needs to be dynamically set… i.e. with something like:

  install: |
      if [ "$(arch)" = "x86_64" ]; then
        EXECSTACK_PATH="usr/lib/x86_64-linux-gnu"
      elif [ "$(arch)" = "[ "armv7l" ]; then
        EXECSTACK_PATH="usr/lib/arm-linux-gnueabihf"
      else
        echo "unkonwn architecture"
        exit 1
      fi
      execstack -c $SNAPCRAFT_PART_INSTALL/$EXECSTACK_PATH/......

#27

No, these are only 64bit binaries, but adding architectures: architectures: [amd64] does not preclude build.snapcraft to build other architectures.

  • How can I make it only build the 64bit architecture?

#28

Currently you can not, just ignore the armhf build then and dont release it …


#29

@ogra Thank you again for the help, the package is now working under 64bit architectures, in my local testings it is working after testing with real code.

Many thanks.

Will soon release with the Swift namespace.


#30

Hi everyone,

time to update the snap, and I have a question.

While testing I have to slightly change the yaml with extra stage-packages, when I └─λ sudo snap refresh --edge swift --revision=X sometimes links to old binaries are still there. Refreshing is not really refreshing. How do I make sure that after the refresh I am using the newly pulled binaries?

Thanks


#31

I just installed revision 41 from edge and there’s something amiss with the libraries bundled in the snap.

The swift binary itself is x86_64, but there’s some armhf libraries bundled, which confuses things a bit.

alan@KinkPad-K450:~$ swift
/snap/swift/18/bin/swift: error while loading shared libraries: libatomic.so.1: wrong ELF class: ELFCLASS32
alan@KinkPad-K450:~$ which swift
/snap/bin/swift
alan@KinkPad-K450:~$ file /snap/swift/current/bin/swift
/snap/swift/current/bin/swift: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=186dbd8214de76fc46e1b2cdba1f1f37523a205d, not stripped
alan@KinkPad-K450:~$ file /snap/swift/current/usr/lib/arm-linux-gnueabihf/libatomic.so.1.1.0
/snap/swift/current/usr/lib/arm-linux-gnueabihf/libatomic.so.1.1.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=24bcd08007f1ca3055c76a57824fa4a5ae288b03, stripped

The wrapper looks to blame.

alan@KinkPad-K450:~$ cat /snap/swift/current/command-swift.wrapper
#!/bin/sh
export PATH="$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/arm-linux-gnueabihf:$SNAP/usr/lib/arm-linux-gnueabihf"
export LD_LIBRARY_PATH="$SNAP/usr/lib/arm-linux-gnueabihf:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH=$SNAP_LIBRARY_PATH:$LD_LIBRARY_PATH
exec "$SNAP/bin/swift" "$@"

#32

Thanks, I am looking into it right now… It seems there is a link to a 32bit library, am I right? How do I package a 64bit version of libatomic?


#33

Do you have the yaml and wrappers in a repo somewhere we can see? Much easier to debug that way. I’ll take a look in the morning if so.


#34

yep, https://github.com/lf-araujo/swift-snap/blob/master/snap/snapcraft.yaml

I have been working in the edge channel…


#35

With the current yaml file, I am getting:

Priming swift 
Files from the build host were migrated into the snap to satisfy dependencies that would otherwise not be met. This feature will be removed in a future release. If these libraries are needed in the final snap, ensure that the following are either satisfied by a stage-packages entry or through a part:
usr/lib/x86_64-linux-gnu/libdb-5.3.so
Traceback (most recent call last):
  File "/usr/bin/snapcraft", line 9, in <module>
    load_entry_point('snapcraft==2.40', 'console_scripts', 'snapcraft')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/__main__.py", line 43, in <module>
    run(prog_name='snapcraft')
  File "/usr/lib/python3/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1037, in invoke
    return Command.invoke(self, ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/_runner.py", line 75, in run
    ctx.forward(lifecyclecli.commands['snap'])
  File "/usr/lib/python3/dist-packages/click/core.py", line 552, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/snapcraft/cli/lifecycle.py", line 139, in snap
    project_options, directory=directory, output=output)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_packer.py", line 46, in snap
    execute('prime', project_options)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 80, in execute
    _Executor(config, project_options).run(step, part_names)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 194, in run
    self._create_meta(step, part_names)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/lifecycle/_runner.py", line 243, in _create_meta
    self.config.snapcraft_yaml_path)
  File "/usr/lib/python3/dist-packages/snapcraft/internal/meta/_snap_packaging.py", line 106, in create_snap_packaging
    packaging.write_snap_yaml()
  File "/usr/lib/python3/dist-packages/snapcraft/internal/meta/_snap_packaging.py", line 248, in write_snap_yaml
    snap_yaml = self._compose_snap_yaml()
  File "/usr/lib/python3/dist-packages/snapcraft/internal/meta/_snap_packaging.py", line 400, in _compose_snap_yaml
    snap_yaml['apps'] = self._wrap_apps(self._config_data['apps'])
  File "/usr/lib/python3/dist-packages/snapcraft/internal/meta/_snap_packaging.py", line 495, in _wrap_apps
    self._wrap_app(app, apps[app])
  File "/usr/lib/python3/dist-packages/snapcraft/internal/meta/_snap_packaging.py", line 503, in _wrap_app
    app[k] = self._wrap_exe(app[k], '{}-{}'.format(k, name))
  File "/usr/lib/python3/dist-packages/snapcraft/internal/meta/_snap_packaging.py", line 473, in _wrap_exe
    with open(exepath, 'rb') as exefile:
FileNotFoundError: [Errno 2] No such file or directory: '/build/f3f03c80e7fff860ec06266b8e1d075d-xenial/prime/bin/repl_swift'
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 248, in run
    self.build()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 222, in build
    env=env)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 96, in run_build_command
    return self.backend.run(args, cwd=cwd, env=full_env, **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 456, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-xenial-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'http_proxy=http://snap-proxy.launchpad.net:3128', '--env', 'https_proxy=http://snap-proxy.launchpad.net:3128', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/f3f03c80e7fff860ec06266b8e1d075d-xenial && linux64 snapcraft']' returned non-zero exit status 1
Revoking proxy token...
RUN: /usr/share/launchpad-buildd/slavebin/in-target scan-for-processes --backend=lxd --series=xenial --arch=amd64 SNAPBUILD-192648
Scanning for processes to kill in build SNAPBUILD-192648
RUN: /usr/share/launchpad-buildd/slavebin/in-target umount-chroot --backend=lxd --series=xenial --arch=amd64 SNAPBUILD-192648
Stopping target for build SNAPBUILD-192648
/usr/lib/python2.7/dist-packages/pylxd/model.py:116: UserWarning: Attempted to set unknown attribute "description" on instance of "Container"
  key, self.__class__.__name__
RUN: /usr/share/launchpad-buildd/slavebin/in-target remove-build --backend=lxd --series=xenial --arch=amd64 SNAPBUILD-192648

#36

Hm, it fails differently here.

Skipping prime swift (already ran)
Sorry, Snapcraft ran into an error when trying to running through its
lifecycle that generated the following traceback:
Traceback (most recent call last):
  File "/snap/snapcraft/1337/bin/snapcraft", line 11, in <module>
    load_entry_point('snapcraft==2.40', 'console_scripts', 'snapcraft')()
  File "/snap/snapcraft/1337/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/snap/snapcraft/1337/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/snap/snapcraft/1337/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/snap/snapcraft/1337/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/cli/__main__.py", line 43, in <module>
    run(prog_name='snapcraft')
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/cli/lifecycle.py", line 139, in snap
    project_options, directory=directory, output=output)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_packer.py", line 46, in snap
    execute('prime', project_options)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 80, in execute
    _Executor(config, project_options).run(step, part_names)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 194, in run
    self._create_meta(step, part_names)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 243, in _create_meta
    self.config.snapcraft_yaml_path)
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/meta/_snap_packaging.py", line 106, in create_snap_packaging
    packaging.write_snap_yaml()
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/meta/_snap_packaging.py", line 248, in write_snap_yaml
    snap_yaml = self._compose_snap_yaml()
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/meta/_snap_packaging.py", line 400, in _compose_snap_yaml
    snap_yaml['apps'] = self._wrap_apps(self._config_data['apps'])
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/meta/_snap_packaging.py", line 495, in _wrap_apps
    self._wrap_app(app, apps[app])
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/meta/_snap_packaging.py", line 503, in _wrap_app
    app[k] = self._wrap_exe(app[k], '{}-{}'.format(k, name))
  File "/snap/snapcraft/1337/lib/python3.6/site-packages/snapcraft/internal/meta/_snap_packaging.py", line 473, in _wrap_exe
    with open(exepath, 'rb') as exefile:
FileNotFoundError: [Errno 2] No such file or directory: '/root/build_swift/prime/bin/lldb'

Strange that it can’t find lldb, which very clearly is in parts/swift/build/bin/lldb but isn’t copied to stage perhaps? Maybe we need assistance from @kalikiana :slight_smile:


#37

This is because I made some changes earlier today, the yaml is back to the one which is building but pulling 32bit libraries instead of 64bit.

name: swift
version: '4.1'
summary: The Swift Programming Language
description: |
  Swift is a high-performance system programming language. It has a clean and
  modern syntax, offers seamless access to existing C and Objective-C code and
  frameworks, and is memory safe by default.

# TODO:
#  - Build from source http://paste.ubuntu.com/25564789/

confinement: strict
grade: devel
architectures: [ amd64 ]

parts:
  swift:
    plugin: dump
    source: https://swift.org/builds/swift-4.1-release/ubuntu1604/swift-4.1-RELEASE/swift-4.1-RELEASE-ubuntu16.04.tar.gz
    stage-packages:
      - clang
      - libblocksruntime0
      - libbsd0
      - libedit2
      - libicu-dev          # Yes, the -dev package is deliberately staged.
      - libncursesw5
      - libpython2.7
      - libsqlite3-0
      - libuutil1linux
      - libxml2
      - python2.7
      - libcurl3
      - libatomic1
      
apps:
  lldb:
    command: bin/lldb
  lldb-argdumper:
    command: bin/lldb-argdumper
  lldb-mi:
    command: bin/lldb-mi
  lldb-server:
    command: bin/lldb-server
  repl-swift:
    command: bin/repl_swift
  swift:
    command: bin/swift
  swift-autolink-extract:
    command: bin/swift-autolink-extract
  swift-build:
    command: bin/swift-build
  swift-build-tool:
    command: bin/swift-build-tool
  swift-demangle:
    command: bin/swift-demangle
  swift-package:
    command: bin/swift-package
  swift-test:
    command: bin/swift-test
  swiftc:
    command: bin/swiftc
  swift-run:
    command: bin/swift-run
  swift-format:
    command: bin/swift-format

#38

The above line tells snapd that any package created with this yaml will run on amd64. The builder will take the yaml and build on BOTH i386 AND amd64 (and attempt to build on armhf), and BOTH of (all all three if armhf succeeds) the built packages claim they will run on amd64. To fix this simply remove the architectures line.


#39

Thank you @daniel,

Now what can I do so the shared libraries are accessible by the compiler in the snap?

I tried stage: usr/lib/* to no avail, how do I do this?

Best,
lf


#40

Which libraries are inaccessible? Where are they located? I built your yaml (from two posts ago) and can’t see any problems with libraries.