Docker images are broken for stage-packages

abitrolly · May 26, 2020, 4:52pm

EDIT: This is an attempt to use snapcraft on Fedora 32 using podman.

Trying to build snap with stage-packages using docker image fails.

Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [8158 B]                                                                                
Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [8286 B]                                                                                    
Fetched 18.0 MB in 6s (2907 kB/s)                                                                                                                                        
Get:1 libkrb5-3_1.16-2ubuntu0.1_amd64.deb [279 kB]                                                                                                                       
Fetched 279 kB in 0s (0 B/s)                                                                                                                                             
Sorry, an error occurred in Snapcraft:
[Errno 13] Permission denied: '/root/myip/parts/myip/ubuntu/download/libkrb5-3_1.16-2ubuntu0.1_amd64.deb'
We would appreciate it if you anonymously reported this issue.
No other data than the traceback and the version of snapcraft in use will be sent.
Would you like to send this error data? (Yes/No/Always/View) [no]: v
Traceback (most recent call last):
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/file_utils.py", line 109, in link_or_copy
    link(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/file_utils.py", line 143, in link
    os.link(source_path, destination, follow_symlinks=False)
OSError: [Errno 18] Invalid cross-device link: '/root/.cache/snapcraft/stage-packages/apt/c973f474f834adabf37c656562b00de411dc8c2b38c28b651fe750afe10f0498585c6d804b67a5f420398e78b7347456/var/cache/apt/archives/libkrb5-3_1.16-2ubuntu0.1_amd64.deb' -> '/root/myip/parts/myip/ubuntu/download/libkrb5-3_1.16-2ubuntu0.1_amd64.deb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/snapcraft/current/bin/snapcraft", line 11, in <module>
    load_entry_point('snapcraft==3.9.1', 'console_scripts', 'snapcraft')()
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/_command.py", line 88, in invoke
    return super().invoke(ctx)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/lifecycle.py", line 207, in build
    _execute(steps.BUILD, parts, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/lifecycle.py", line 78, in _execute
    lifecycle.execute(step, project_config, parts)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 131, in execute
    executor.run(step, part_names)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 185, in run
    self._handle_step(part_names, part, step, current_step, cli_config)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 199, in _handle_step
    getattr(self, "_run_{}".format(current_step.name))(part)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 241, in _run_pull
    self._run_step(step=steps.PULL, part=part, progress="Pulling")
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 315, in _run_step
    self._prepare_step(step=step, part=part)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 307, in _prepare_step
    preparation_function()
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 449, in prepare_pull
    self._fetch_stage_packages()
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 435, in _fetch_stage_packages
    self.stage_packages = self._stage_packages_repo.get(stage_packages)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/repo/_deb.py", line 488, in get
    return self._get(apt_cache)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/repo/_deb.py", line 556, in _get
    file_utils.link_or_copy(source, destination)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/file_utils.py", line 117, in link_or_copy
    copy(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/file_utils.py", line 166, in copy
    shutil.copy2(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/usr/lib/python3.6/shutil.py", line 264, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/usr/lib/python3.6/shutil.py", line 229, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/snap/snapcraft/current/usr/lib/python3.6/shutil.py", line 165, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: '/root/myip/parts/myip/ubuntu/download/libkrb5-3_1.16-2ubuntu0.1_amd64.deb'

The cause of the problem is that /root/.cache is inside container filesystem, and /root/myip is a mounted volume. snapcraft tries to create hard link between filesystems, which is not possible.

EDIT: An when the link fails, shutil.copy2 tries set attributes on SELinux enabled volume and fails too, because it doesn’t know how to ignore EACCES errors https://bugs.python.org/issue38893#msg364011

cjp256 · May 26, 2020, 7:32pm

So the filesystem you have mounted in the docker container will not allow snapcraft process to set extended attributes?

Snapcraft currently takes advantage of extended attributes (user) for tracking stage packages, so even if it makes it past this step, we may hit other issues.

Can you provide some additional context to help me reproduce?

If you can, consider trying the edge version of snapcraft and see what happens? Stage packages are handled a little differently now, so you may have some luck there because that specific cross-boundary copy should no longer be happening IIRC…

abitrolly · May 26, 2020, 7:53pm

Just run it on Fedora 32. Looks like multipass should allow this to be easy.

Here is some evidence that it is Python bug https://github.com/containers/libpod/issues/4963

abitrolly · May 26, 2020, 7:58pm

The command to run.

podman run -v -v "$(pwd):/root/$NAME":Z:/root/myip:Z -w /root/myip -it --rm snapcore/snapcraft:edge snapcraft

cjp256 · May 26, 2020, 8:03pm

Ah, makes sense - thanks for the link!

Glad multipass is working for you

abitrolly · May 26, 2020, 8:05pm

No its not.

✗ multipass list
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
list failed: multipass socket access denied
Please check that you have read/write permissions to '/var/snap/multipass/common/multipass_socket'

cjp256 · May 26, 2020, 8:22pm

@Saviq suggests that may be a firewalld configuration issue?

abitrolly · May 26, 2020, 8:24pm

How to check? I am using latest multipass.

multipass  1.3.0-dev.55+gb82d487

Saviq · May 26, 2020, 8:25pm

Actually that’s not it.

@abitrolly what’s the ownership on the socket the error mentions?

It should be one of sudo, adm, wheel. Have a look and make sure your user is a member.

abitrolly · May 26, 2020, 8:27pm

➜  ~ ls -la /var/snap/multipass/common/multipass_socket
srw-rw----. 1 root adm 0 May 26 20:42 /var/snap/multipass/common/multipass_socket
➜  ~ groups
anatoli wheel lxd libvirt docker

Saviq · May 26, 2020, 8:30pm

That’s it, then, you’ll need to be part of the adm group. It’s a security measure, since Multipass has access to the whole host.

abitrolly · May 26, 2020, 8:32pm

I can not find any reference on the usage of adm group on Fedora. Should the snap set it to be wheel?

Saviq · May 26, 2020, 8:35pm

Yes, we should prefer wheel, I could’ve sworn we have a bug about it already.

You can obviously change the perms, but they will reset on Multipass restart.

abitrolly · May 26, 2020, 8:41pm

Regarding wheel group I found only this.

In https://wiki.debian.org/SystemGroups it is said that adm groups is only for system monitoring.

Saviq · May 26, 2020, 8:44pm

It’s a Multipass issue, not a snapd one, FWIW.

Saviq · May 26, 2020, 8:49pm

@abitrolly I’ve filed a PR to fix this:

You will be able to snap refresh multipass --channel edge/pr1559 in 20 minutes or so.

abitrolly · May 26, 2020, 11:10pm

multipass works now.

~ multipass list       
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
No instances found.

abitrolly · May 27, 2020, 7:52am

However, the original issue with stage-packages when linking/copying them from container cache to volumes located on different filesystem is not solved.