Copy file problem when staging files with docker (for arm)

snapcraft
1

Hello,

I have a little problem with staging files when running snapcraft with docker (for arm).
My setting is the following:

  • Running snapcraft with a docker for armhf architecture (compare https://hub.docker.com/repository/docker/flrkli/snapcraft/general). Do not get confused by the name flrkli/snapcraft:3.8-armhf. It is the latest snapcraft version and I’ve aligned to the official snapcraft docker.
  • Command is: docker run -v “$PWD”:/build -w /build flrkli/snapcraft:3.8-armhf sh -c “snapcraft”
  • Building an autotool based part (I’ve attached an example in [1] but it also fails for other stuff)
  • Running docker on mac (amd64)

Now the build is running through but once the staging is started I get a weird error (compare [2]). It seems that something is going wrong when the files (and symlinks) are copied. When I compare the parts with the stage folder I see the following:

./parts/fdk-aac/install/usr/lib/

-rwxr-xr-x  1 florian  staff      938 Jan 27 21:05 libfdk-aac.la
lrwxrwxrwx  1 florian  staff       19 Jan 27 21:05 libfdk-aac.so -> libfdk-aac.so.2.0.0
lrwxrwxrwx  1 florian  staff       19 Jan 27 21:05 libfdk-aac.so.2 -> libfdk-aac.so.2.0.0
-rwxr-xr-x  2 florian  staff  5648488 Jan 27 21:05 libfdk-aac.so.2.0.0
drwxr-xr-x  3 florian  staff       96 Jan 27 21:05 pkgconfig

./stage/usr/lib/

-rwxr-xr-x  2 florian  staff  5648488 Jan 27 21:05 libfdk-aac.so
drwxr-xr-x@ 2 florian  staff       64 Jan 27 21:05 pkgconfig

So it seems that instead of the symlink libfdk-aac.so the complete file is copied and afterwards we receive the error.

Could you help on this? Is this a problem with snapcraft, docker or something else?

Thanks a lot

[1]

parts:
  fdk-aac:
    plugin: autotools
    source: https://github.com/mstorsjo/fdk-aac/archive/v2.0.0.tar.gz
    build-packages:
      - g++
    configflags:
      - --prefix=/usr
      - --disable-static
    prime:
      - usr/lib
      - usr/lib/pkgconfig
      - usr/include

[2]

make install DESTDIR=/build/parts/fdk-aac/install
make[1]: Entering directory '/build/parts/fdk-aac/build'
 /bin/mkdir -p '/build/parts/fdk-aac/install/usr/lib'
 /bin/bash ./libtool   --mode=install /usr/bin/install -c   libfdk-aac.la '/build/parts/fdk-aac/install/usr/lib'
libtool: install: /usr/bin/install -c .libs/libfdk-aac.so.2.0.0 /build/parts/fdk-aac/install/usr/lib/libfdk-aac.so.2.0.0
libtool: install: (cd /build/parts/fdk-aac/install/usr/lib && { ln -s -f libfdk-aac.so.2.0.0 libfdk-aac.so.2 || { rm -f libfdk-aac.so.2 && ln -s libfdk-aac.so.2.0.0 libfdk-aac.so.2; }; })
libtool: install: (cd /build/parts/fdk-aac/install/usr/lib && { ln -s -f libfdk-aac.so.2.0.0 libfdk-aac.so || { rm -f libfdk-aac.so && ln -s libfdk-aac.so.2.0.0 libfdk-aac.so; }; })
libtool: install: /usr/bin/install -c .libs/libfdk-aac.lai /build/parts/fdk-aac/install/usr/lib/libfdk-aac.la
libtool: warning: remember to run 'libtool --finish /usr/lib'
 /bin/mkdir -p '/build/parts/fdk-aac/install/usr/include/fdk-aac'
 /usr/bin/install -c -m 644 ./libSYS/include/machine_type.h ./libSYS/include/genericStds.h ./libSYS/include/FDK_audio.h ./libSYS/include/syslib_channelMapDescr.h ./libAACenc/include/aacenc_lib.h ./libAACdec/include/aacdecoder_lib.h '/build/parts/fdk-aac/install/usr/include/fdk-aac'
 /bin/mkdir -p '/build/parts/fdk-aac/install/usr/lib/pkgconfig'
 /usr/bin/install -c -m 644 fdk-aac.pc '/build/parts/fdk-aac/install/usr/lib/pkgconfig'
make[1]: Leaving directory '/build/parts/fdk-aac/build'
Staging fdk-aac 
snapcraft is running as a snap True, SNAP_NAME set to 'snapcraft'
Sorry, an error occurred in Snapcraft:
[Errno 17] File exists: 'libfdk-aac.so.2.0.0' -> '/build/stage/usr/lib/libfdk-aac.so'
Traceback (most recent call last):
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/file_utils.py", line 110, in link_or_copy
    link(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/file_utils.py", line 144, in link
    os.link(source_path, destination, follow_symlinks=False)
OSError: [Errno 5] Input/output error: '/build/parts/fdk-aac/install/usr/lib/libfdk-aac.so' -> '/build/stage/usr/lib/libfdk-aac.so'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/snapcraft/current/bin/snapcraft", line 11, in <module>
    load_entry_point('snapcraft==3.9.1', 'console_scripts', 'snapcraft')()
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 1114, in invoke
    return Command.invoke(self, ctx)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/_runner.py", line 109, in run
    snap_command.invoke(ctx)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/_command.py", line 87, in invoke
    return super().invoke(ctx)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/lifecycle.py", line 261, in snap
    _execute(steps.PRIME, parts=[], pack_project=True, output=output, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/lifecycle.py", line 66, in _execute
    lifecycle.execute(step, project_config, parts)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 132, in execute
    executor.run(step, part_names)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 186, in run
    self._handle_step(part_names, part, step, current_step, cli_config)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 200, in _handle_step
    getattr(self, "_run_{}".format(current_step.name))(part)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 264, in _run_stage
    self._run_step(step=steps.STAGE, part=part, progress="Staging")
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 319, in _run_step
    getattr(part, step.name)()
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 743, in stage
    self._do_runner_step(steps.STAGE)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 245, in _do_runner_step
    return getattr(self._runner, "{}".format(step.name))()
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 84, in stage
    "override-stage", self._override_stage_scriptlet, self._stagedir
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 137, in _run_scriptlet
    scriptlet_name, function_call.strip()
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 193, in _handle_builtin_function
    function(**function_args)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 765, in _do_stage
    fixup_func=fixup_func,
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 1140, in _migrate_files
    file_utils.link_or_copy(src, dst, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/file_utils.py", line 118, in link_or_copy
    copy(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/file_utils.py", line 167, in copy
    shutil.copy2(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/usr/lib/python3.6/shutil.py", line 263, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/usr/lib/python3.6/shutil.py", line 118, in copyfile
    os.symlink(os.readlink(src), dst)
FileExistsError: [Errno 17] File exists: 'libfdk-aac.so.2.0.0' -> '/build/stage/usr/lib/libfdk-aac.so'
snapcraft is running as a snap True, SNAP_NAME set to 'snapcraft'
2

I’ve done some digging. This appears to be an interaction between python used by snapcraft and the Docker filesystem proxy used by Docker for Mac. Docker for Windows is unaffected. The situation is that the filesystem proxy that Docker sets up is causing Python to follow the symlinks and copying their target instead of copying the links as links.

cc/ @sergiusens and @cmatsuoka for Snapcraft developers’ eyes, but I think this is a Docker bug, which is exposed by Python’s file_utils.link_or_copy(), or possibly but probably not a Python bug directly.

3

I should of probably commented here instead of random chat groups :wink:

There is an open Snapcraft bug for this, there is not much we can do about it as the issue happens one layer below the tool. There are some side effects paths we could take for these situations, but then you’d get a different snap than when building somewhere else.

The best solution is to checkout the code inside the docker instance.

4

Thanks a lot for your quick feedback. I ran it in Ubuntu and it was working out. So as mentioned this seems to be a mac problem.
Great that you are working on it. Let me know once it is resolved so I can run it on my mac as well :).
Best regards

1 Like