Docker images are broken for stage-packages

EDIT: This is an attempt to use snapcraft on Fedora 32 using podman.

Trying to build snap with stage-packages using docker image fails.

Get:17 bionic-backports/universe amd64 Packages [8158 B]                                                                                
Get:18 bionic-backports/main amd64 Packages [8286 B]                                                                                    
Fetched 18.0 MB in 6s (2907 kB/s)                                                                                                                                        
Get:1 libkrb5-3_1.16-2ubuntu0.1_amd64.deb [279 kB]                                                                                                                       
Fetched 279 kB in 0s (0 B/s)                                                                                                                                             
Sorry, an error occurred in Snapcraft:
[Errno 13] Permission denied: '/root/myip/parts/myip/ubuntu/download/libkrb5-3_1.16-2ubuntu0.1_amd64.deb'
We would appreciate it if you anonymously reported this issue.
No other data than the traceback and the version of snapcraft in use will be sent.
Would you like to send this error data? (Yes/No/Always/View) [no]: v
Traceback (most recent call last):
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/", line 109, in link_or_copy
    link(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/", line 143, in link, destination, follow_symlinks=False)
OSError: [Errno 18] Invalid cross-device link: '/root/.cache/snapcraft/stage-packages/apt/c973f474f834adabf37c656562b00de411dc8c2b38c28b651fe750afe10f0498585c6d804b67a5f420398e78b7347456/var/cache/apt/archives/libkrb5-3_1.16-2ubuntu0.1_amd64.deb' -> '/root/myip/parts/myip/ubuntu/download/libkrb5-3_1.16-2ubuntu0.1_amd64.deb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/snapcraft/current/bin/snapcraft", line 11, in <module>
    load_entry_point('snapcraft==3.9.1', 'console_scripts', 'snapcraft')()
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/", line 782, in main
    rv = self.invoke(ctx)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/", line 88, in invoke
    return super().invoke(ctx)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/click/", line 610, in invoke
    return callback(*args, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/", line 207, in build
    _execute(steps.BUILD, parts, **kwargs)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/cli/", line 78, in _execute
    lifecycle.execute(step, project_config, parts)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/", line 131, in execute, part_names)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/", line 185, in run
    self._handle_step(part_names, part, step, current_step, cli_config)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/", line 199, in _handle_step
    getattr(self, "_run_{}".format(
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/", line 241, in _run_pull
    self._run_step(step=steps.PULL, part=part, progress="Pulling")
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/", line 315, in _run_step
    self._prepare_step(step=step, part=part)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/lifecycle/", line 307, in _prepare_step
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/", line 449, in prepare_pull
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/pluginhandler/", line 435, in _fetch_stage_packages
    self.stage_packages = self._stage_packages_repo.get(stage_packages)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/repo/", line 488, in get
    return self._get(apt_cache)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/internal/repo/", line 556, in _get
    file_utils.link_or_copy(source, destination)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/", line 117, in link_or_copy
    copy(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/lib/python3.6/site-packages/snapcraft/", line 166, in copy
    shutil.copy2(source, destination, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/usr/lib/python3.6/", line 264, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/usr/lib/python3.6/", line 229, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/snap/snapcraft/current/usr/lib/python3.6/", line 165, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: '/root/myip/parts/myip/ubuntu/download/libkrb5-3_1.16-2ubuntu0.1_amd64.deb'

The cause of the problem is that /root/.cache is inside container filesystem, and /root/myip is a mounted volume. snapcraft tries to create hard link between filesystems, which is not possible.

EDIT: An when the link fails, shutil.copy2 tries set attributes on SELinux enabled volume and fails too, because it doesn’t know how to ignore EACCES errors

So the filesystem you have mounted in the docker container will not allow snapcraft process to set extended attributes?

Snapcraft currently takes advantage of extended attributes (user) for tracking stage packages, so even if it makes it past this step, we may hit other issues.

Can you provide some additional context to help me reproduce?

If you can, consider trying the edge version of snapcraft and see what happens? Stage packages are handled a little differently now, so you may have some luck there because that specific cross-boundary copy should no longer be happening IIRC…

Just run it on Fedora 32. Looks like multipass should allow this to be easy.

Here is some evidence that it is Python bug

1 Like

The command to run.

podman run -v -v "$(pwd):/root/$NAME":Z:/root/myip:Z -w /root/myip -it --rm snapcore/snapcraft:edge snapcraft

Ah, makes sense - thanks for the link!

Glad multipass is working for you :slight_smile:

No its not. :smiley:

✗ multipass list
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
list failed: multipass socket access denied
Please check that you have read/write permissions to '/var/snap/multipass/common/multipass_socket'
1 Like

@Saviq suggests that may be a firewalld configuration issue?

How to check? I am using latest multipass.

multipass  1.3.0-dev.55+gb82d487

Actually that’s not it.

@abitrolly what’s the ownership on the socket the error mentions?

It should be one of sudo, adm, wheel. Have a look and make sure your user is a member.

➜  ~ ls -la /var/snap/multipass/common/multipass_socket
srw-rw----. 1 root adm 0 May 26 20:42 /var/snap/multipass/common/multipass_socket
➜  ~ groups
anatoli wheel lxd libvirt docker

That’s it, then, you’ll need to be part of the adm group. It’s a security measure, since Multipass has access to the whole host.

I can not find any reference on the usage of adm group on Fedora. Should the snap set it to be wheel?

Yes, we should prefer wheel, I could’ve sworn we have a bug about it already.

You can obviously change the perms, but they will reset on Multipass restart.

Regarding wheel group I found only this.

In it is said that adm groups is only for system monitoring.

It’s a Multipass issue, not a snapd one, FWIW.

@abitrolly I’ve filed a PR to fix this:

You will be able to snap refresh multipass --channel edge/pr1559 in 20 minutes or so.

1 Like

multipass works now. :smiley:

~ multipass list       
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
No instances found.

However, the original issue with stage-packages when linking/copying them from container cache to volumes located on different filesystem is not solved.