Snapcrafting on Mac with Docker

Continuing with another of my very niche kind of problems…
I am today trying to build a snap that uses core18 on Mac via Docker.

I read the docs and created an image from this Dockerfile with the following command:

docker build --no-cache -f stable.Dockerfile --label mycustomimage --tag mycustomimage:stable --network host .

If I than tried to follow the “first snap” example for python i.e. OfflineIMAP from this snapcraft.yaml at the staging part I get the following error:

Sorry, an error occurred in Snapcraft:
[Errno 17] File exists: 'libexpat.so.1.6.7' -> '/src/stage/lib/x86_64-linux-gnu/libexpat.so.1'

The thing that really leaves me quite confused is that, if I ask a friend who owns an Ubuntu Desktop laptop to reproduce the issue he can compile alright with no errors whatsoever.

I thought that the issue might be in the Dockerfile so I pulled this image and tried again, the exact error comes up at the staging step. I again asked my mate with Ubuntu to pull the same image and compile from the same yaml and again he succeeded with no issue.

At this point I am very puzzled. I thought that the whole point of using a container was that one can reproduce an environment independently on what OS is running the container from.

What am I doing wrong?

My Docker version is 19.03.8, build afacb8b
My Mac is running Catalina 10.15.5

Thanks for your help!

We’re gonna need more of the log output than those two lines, so that we have a better understanding of when this occurs.

I’ve just cloned the repo and run through a docker-based build. I did not hit the same problem, so something is amiss either with the images you tried or your command line, maybe.

Sure. After the initial apt install run I get to this part… sorry this is a lot of log

Pulling test-offlineimap 
Cloning into '/src/parts/test-offlineimap/src'...
remote: Enumerating objects: 12860, done.
remote: Total 12860 (delta 0), reused 0 (delta 0), pack-reused 12860
Receiving objects: 100% (12860/12860), 5.35 MiB | 1.86 MiB/s, done.
Resolving deltas: 100% (8229/8229), done.
Checking connectivity... done.
Fetching and installing pip...
Collecting pip
Downloading https://files.pythonhosted.org/packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl (1.5MB)
100% |████████████████████████████████| 1.5MB 714kB/s 
Saved ./parts/test-offlineimap/python-packages/pip-20.1.1-py2.py3-none-any.whl
Successfully downloaded pip
Ignoring indexes: https://pypi.python.org/simple
Collecting pip
Installing collected packages: pip
Successfully installed pip-8.1.1
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Fetching and installing wheel...
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting wheel
Downloading wheel-0.34.2-py2.py3-none-any.whl (26 kB)
Saved ./parts/test-offlineimap/python-packages/wheel-0.34.2-py2.py3-none-any.whl
Successfully downloaded wheel
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Looking in links: /src/parts/test-offlineimap/python-packages
Processing ./parts/test-offlineimap/python-packages/wheel-0.34.2-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.34.2
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Fetching and installing setuptools...
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting setuptools
Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
|████████████████████████████████| 583 kB 5.7 MB/s 
Saved ./parts/test-offlineimap/python-packages/setuptools-44.1.1-py2.py3-none-any.whl
Successfully downloaded setuptools
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Looking in links: /src/parts/test-offlineimap/python-packages
Processing ./parts/test-offlineimap/python-packages/setuptools-44.1.1-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-44.1.1
Building test-offlineimap 
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Looking in links: /src/parts/test-offlineimap/python-packages
Processing /src/parts/test-offlineimap/build
Building wheels for collected packages: offlineimap
Building wheel for offlineimap (setup.py) ... done
Created wheel for offlineimap: filename=offlineimap-7.1.1-py2-none-any.whl size=171717 sha256=a8261f621b83c475539d10dfe5190b1122b29c8f2c9cea2b7d0149496a60e7ea
Stored in directory: /tmp/pip-ephem-wheel-cache-cwMUqv/wheels/e8/05/b1/00914fe507607013c7d4bb2a5fc7d482a0c578614202fd0673
Successfully built offlineimap
Installing collected packages: offlineimap
Successfully installed offlineimap-7.1.1
/src/parts/test-offlineimap/install/usr/bin/python2 setup.py --no-user-cfg install --single-version-externally-managed --user --record install.txt
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: option --single-version-externally-managed not recognized
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Staging test-offlineimap 
Sorry, an error occurred in Snapcraft:
[Errno 17] File exists: 'libexpat.so.1.6.0' -> '/src/stage/lib/x86_64-linux-gnu/libexpat.so.1'

Again thanks!

Can you run the following command from the directory you are building and paste the output, please? This is most odd with such a simple snap as offlineimap.

ls -l parts/offlineimap/install/lib/x86_64-linux-gnu/libexpat* stage/lib/x86_64-linux-gnu/libexpat*

Also could you share the full docker command you used, maybe you’re running it differently to me :slight_smile:

For reference, I used my own Container Image with the following command:

docker run --rm -it -v $PWD:$PWD -w $PWD diddledan/snapcraft:core18 bash -c 'apt-get update; snapcraft'

There you go:

lrwxrwxrwx@ 1 turux  staff      17 12 Sep  2019 parts/test-offlineimap/install/lib/x86_64-linux-gnu/libexpat.so.1 -> libexpat.so.1.6.0
-rw-r--r--@ 2 turux  staff  166032 12 Sep  2019 parts/test-offlineimap/install/lib/x86_64-linux-gnu/libexpat.so.1.6.0
-rw-r--r--@ 2 turux  staff  166032 12 Sep  2019 stage/lib/x86_64-linux-gnu/libexpat.so.1

also this is the command I am using

docker run -it -v $PWD:/src:Z -w /src mycustomimage:stable snapcraft

A quick update… Just tried with your image and your command and I get the exact same error :frowning_face:

Hmm, I wonder if this is something funky with the macOS-specific filesystem proxy that Docker are using? I’m going to try replicating on a macOS install rather than Ubuntu to see if that makes the difference.

I’m pretty sure this is some “macOS-specific-crap”. I have already tried on Ubuntu and works perfectly fine…

Yes, I can replicate your issue using my container image in macOS. It looks to me like the symlink is being screwed-up somewhere by the osxfs in Docker for Mac.

1 Like

This works, albeit not an ideal solution:

docker run --rm -it -v $PWD:$PWD -w $PWD diddledan/snapcraft:core18 bash -c '
set -e
apt-get update
apt-get install -yqq rsync
rsync -rv ./ /src/
pushd /src
snapcraft
popd 
cp -v /src/*.snap .
'
1 Like

Thanks for the heads up!
I will give it a try asap… but yeah… pretty convoluted and possibly not behaving as expected.

Is this linked to the latest version of macOS?