Snapcraft using LXD seems to break with local umask 0027

snapcraft
#1

Hi,

I have a slightly more secure umask than is common, 0027, compared the to more standard 0022

I recently tried switching snapcraft local builds to using lxd, after multipass builds seemed to stop working reliably. I initially got the following error from snapcraft:

2023-09-21 18:59:29.703 :: 2023-09-21 17:59:27.939 Starting Snapcraft 7.5.2                                                                                                            
2023-09-21 18:59:29.703 :: 2023-09-21 17:59:27.941 lifecycle command: 'pull', arguments: Namespace(bind_ssh=False, build_for='amd64', debug=True, destructive_mode=False, enable_developer_debug=False, enable_experimental_extensions=False, enable_experimental_plugins=False, enable_experimental_target_arch=False, enable_experimental_ua_services=False, enable_manifest=False, http_proxy=None, https_proxy=None, manifest_image_information=None, parts=[], provider=None, shell=False, shell_after=False, target_arch=None, ua_token=None, use_lxd=False)     
2023-09-21 18:59:29.703 :: 2023-09-21 17:59:27.941 command: pull, arguments: Namespace(bind_ssh=False, build_for='amd64', debug=True, destructive_mode=False, enable_developer_debug=False, enable_experimental_extensions=False, enable_experimental_plugins=False, enable_experimental_target_arch=False, enable_experimental_ua_services=False, enable_manifest=False, http_proxy=None, https_proxy=None, manifest_image_information=None, parts=[], provider=None, shell=False, shell_after=False, target_arch=None, ua_token=None, use_lxd=False)                 
2023-09-21 18:59:29.703 :: 2023-09-21 17:59:27.941 Could not find snap/snapcraft.yaml. Are you sure you are in the right directory?                                                    
2023-09-21 18:59:29.703 :: 2023-09-21 17:59:27.941 Recommended resolution: To start a new project, use `snapcraft init`              
2023-09-21 18:59:32.253 Failed to execute pull in instance.                                                                                                                            
2023-09-21 18:59:32.254 Detailed information: Run the same command again with --debug to shell into the environment if you wish to introspect this failure.                            
2023-09-21 18:59:32.256 Traceback (most recent call last):                                                                                                                             
2023-09-21 18:59:32.256   File "/snap/snapcraft/9510/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 574, in _run_in_provider                                          
2023-09-21 18:59:32.256     instance.execute_run(cmd, check=True, cwd=output_dir)                                                                                                      
2023-09-21 18:59:32.256   File "/snap/snapcraft/9510/lib/python3.8/site-packages/craft_providers/lxd/lxd_instance.py", line 289, in execute_run                                        
2023-09-21 18:59:32.256     return self.lxc.exec(                                                                                                                                      
2023-09-21 18:59:32.256   File "/snap/snapcraft/9510/lib/python3.8/site-packages/craft_providers/lxd/lxc.py", line 371, in exec                                                        
2023-09-21 18:59:32.257     return runner(final_cmd, **kwargs)  # pylint: disable=subprocess-run-check                                                                                 
2023-09-21 18:59:32.257   File "/snap/snapcraft/9510/usr/lib/python3.8/subprocess.py", line 516, in run                                                                                
2023-09-21 18:59:32.257     raise CalledProcessError(retcode, process.args,                                                                                                            
2023-09-21 18:59:32.257 subprocess.CalledProcessError: Command '['lxc', '--project', 'snapcraft', 'exec', 'local:snapcraft-osp-nomad-on-amd64-for-amd64-13197989', '--cwd', '/root/project', '--', 'env', 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'SNAPCRAFT_MANAGED_MODE=1', 'snapcraft', 'pull', '--verbosity=debug', '--debug', '--build-for', 'amd64']' returned non-zero exit status 1.

After a bit of digging, I realised the the problem seemed to be permissions based.

Logging into the lxd instance while it was running, I noted the perms where:

# find -ls
  2031747      4 drwx------   5 root     root         4096 Sep 21 17:59 .
  2031865      4 drwx------   3 root     root         4096 Sep 14 16:59 ./snap
  2031866      4 drwxr-xr-x   5 root     root         4096 Sep 21 17:57 ./snap/snapcraft
  2040619      4 drwxr-xr-x   2 root     root         4096 Sep 14 16:59 ./snap/snapcraft/9542
  2031868      4 drwxr-xr-x   2 root     root         4096 Sep 14 16:59 ./snap/snapcraft/common
  2031867      4 drwxr-xr-x   2 root     root         4096 Sep 14 16:59 ./snap/snapcraft/9510
  1971635      0 lrwxrwxrwx   1 root     root            4 Sep 21 17:57 ./snap/snapcraft/current -> 9542
 13197989      4 drwxr-x---   3 root     nogroup      4096 Sep 21 16:34 ./project
 13197990      4 drwxr-x---   3 root     nogroup      4096 Sep 21 16:53 ./project/snap
 13197991      4 drwxr-x---   2 root     nogroup      4096 Sep 21 16:36 ./project/snap/hooks
 13141832      4 -rwxr-xr-x   1 root     nogroup        30 Sep 21 16:36 ./project/snap/hooks/configure
 13141833      4 -rw-r-----   1 root     nogroup       450 Sep 21 16:53 ./project/snap/snapcraft.yaml
  2030894      4 -rw-------   1 root     root           82 Sep 21 17:59 ./.bash_history
  1973377      4 -rw-r--r--   1 root     root          161 Jul  9  2019 ./.profile
  1973376      4 -rw-r--r--   1 root     root         1633 Sep 21 17:59 ./.bashrc
  2040622      4 drwxr-xr-x   3 root     root         4096 Sep 21 17:57 ./.cache
  2040623      4 drwxr-xr-x   4 root     root         4096 Sep 21 17:57 ./.cache/snapcraft
  2040625      4 drwxr-xr-x   2 root     root         4096 Sep 21 17:57 ./.cache/snapcraft/stage-packages
  2040624      4 drwxr-xr-x   2 root     root         4096 Sep 21 17:57 ./.cache/snapcraft/download
  2029735      4 -rw-r--r--   1 root     root          272 Sep 21 17:58 ./environment.sh

Changing the perms in my local project dir to include appropriate r or rx modes for o, and trying again, the snapcraft build works.

Here it is inside the instance with the corrected perms:

# find -ls
  2031747      4 drwx------   5 root     root         4096 Sep 21 18:01 .
  2031865      4 drwx------   3 root     root         4096 Sep 14 16:59 ./snap
  2031866      4 drwxr-xr-x   5 root     root         4096 Sep 21 17:57 ./snap/snapcraft
  2040619      4 drwxr-xr-x   2 root     root         4096 Sep 14 16:59 ./snap/snapcraft/9542
  2031868      4 drwxr-xr-x   2 root     root         4096 Sep 14 16:59 ./snap/snapcraft/common
  2031867      4 drwxr-xr-x   2 root     root         4096 Sep 14 16:59 ./snap/snapcraft/9510
  1971635      0 lrwxrwxrwx   1 root     root            4 Sep 21 17:57 ./snap/snapcraft/current -> 9542
 13197989      4 drwxr-xr-x   3 root     nogroup      4096 Sep 21 16:34 ./project
 13197990      4 drwxr-xr-x   3 root     nogroup      4096 Sep 21 16:53 ./project/snap
 13197991      4 drwxr-xr-x   2 root     nogroup      4096 Sep 21 16:36 ./project/snap/hooks
 13141832      4 -rwxr-xr-x   1 root     nogroup        30 Sep 21 16:36 ./project/snap/hooks/configure
 13141833      4 -rw-r--r--   1 root     nogroup       450 Sep 21 16:53 ./project/snap/snapcraft.yaml
  2030894      4 -rw-------   1 root     root          142 Sep 21 17:59 ./.bash_history
  1973377      4 -rw-r--r--   1 root     root          161 Jul  9  2019 ./.profile
  1973376      4 -rw-r--r--   1 root     root         1633 Sep 21 18:01 ./.bashrc
  2040622      4 drwxr-xr-x   3 root     root         4096 Sep 21 17:57 ./.cache
  2040623      4 drwxr-xr-x   4 root     root         4096 Sep 21 17:57 ./.cache/snapcraft
  2040625      4 drwxr-xr-x   2 root     root         4096 Sep 21 17:57 ./.cache/snapcraft/stage-packages
  2040624      4 drwxr-xr-x   2 root     root         4096 Sep 21 17:57 ./.cache/snapcraft/download
  2029735      4 -rw-r--r--   1 root     root          272 Sep 21 17:58 ./environment.sh

Firstly, as the snapcraft process seems to run as root, I’m unsure why they can’t access the files without extra permission on other user. Does anyone have any insight ?

Secondly, it would be nice if snapcraft did support the above umask, so I would be interested to understand if it should be possible, or if this is simply a big that should be fixed.

Thanks!

Cheers, Just

#2

Hey @jocado,

I’m having a hard time reproducing this. Here’s what I’m doing:

umask 0027
mkdir new-project
vim snapcraft.yaml # create a snapcraft.yaml, which gets 640 permissions by umask
snapcraft

The snapcraft.yaml is very simple:

name: hello-world
base: core20
version: '1.0'
summary: 'hello-world'
description: 'hello-world'
grade: devel
confinement: strict

parts:
  hello-world:
    plugin: dump
    source: .

The build works and I don’t see the nogroup problem when entering the lxd instance.

Are you using core20 or core22? The logs look like a core22 snap but the filetrees look like a core20 snap.

What are the permissions of the project directory on your machine?

#3

Hi @mr_cal - thanks for taking a look :+1:

Just to confirm, it’s actually a core22 snap. I just tried with core20, and it worked without error.

Here is my full repro script:

umask 0027
mkdir new-project
cd new-project
cat << END > snapcraft.yaml
name: hello-world
base: core22
version: '1.0'
summary: 'hello-world'
description: 'hello-world'
grade: devel
confinement: strict

parts:
  hello-world:
    plugin: dump
    source: .
END
snapcraft --use-lxd --verbosity=debug

Also, I’m using snapcraft 7.5.3

BTW - I’m curious what makes you think it might of been core20, is it because the pf the path snap/snapcraft.yaml ?

Thanks!

Cheers, Just

#4

I still can’t reproduce the nogroup problem. @cmatsuoka - do you mind trying to the repro steps?

PS - I was mistaken when looking at the output of find -ls, there wasn’t anything there to indicate is was core20.

#5

I was trying to think of local LXD config which might matter.

Perhaps storage [ see below, slightly edited ], but if you can think of any other info I can add please let me know.

$ lxc storage list
+---------+--------+------------------------------------------------+-------------+---------+---------+
|  NAME   | DRIVER |                     SOURCE                     | DESCRIPTION | USED BY |  STATE  |
+---------+--------+------------------------------------------------+-------------+---------+---------+
| default | dir    | /var/snap/lxd/common/lxd/storage-pools/default |             | 8       | CREATED |
+---------+--------+------------------------------------------------+-------------+---------+---------+

$ lxc storage show default
config:
  source: /var/snap/lxd/common/lxd/storage-pools/default
description: ""
name: default
driver: dir
used_by:
- /1.0/instances/base-instance-snapcraft-buildd-base-v00--a12c0d0071c13c372948?project=snapcraft
- /1.0/instances/base-instance-snapcraft-buildd-base-v10--2d55678e0339df02cd48?project=snapcraft
- /1.0/instances/snapcraft-~8<~?project=snapcraft
- /1.0/instances/snapcraft-~8<~?project=snapcraft
- /1.0/instances/snapcraft-~8<~?project=snapcraft
- /1.0/instances/test
- /1.0/profiles/default
- /1.0/profiles/default?project=snapcraft
status: Created
locations:
- none

Cheers, Just

#6

Couldn’t reproduce it locally with the reproducer script using stable or edge Snapcraft, the test just runs correctly. The only difference seems to be that my storage is on btrfs.

config:
  source: /storage/lxd/storage-pools/fast
  volatile.initial_source: /storage/lxd/storage-pools/fast
description: ""
name: fast
driver: btrfs
used_by:
   - ...
#7

Well, it get’s stranger.

I thought it would be useful to prepare a full virtual test case, using a multicast VM.

This is the cloud init I created:

cat multipass/cloud-init/lxd-test.yaml
#cloud-config

# add each entry to ~/.ssh/authorized_keys for the configured user or the
# first user defined in the user definition directive.
ssh_authorized_keys:
  - {my-ssh-key}

write_files:
  - content: |
      config:
        images.auto_update_interval: '0'
      networks:
        - name: lxdbr0
          type: bridge
          config:
            ipv4.address: auto
            ipv4.nat: true
            ipv6.address: none
      storage_pools:
        - name: default
          driver: dir
      profiles:
        - name: default
          devices:
            root:
              path: /
              pool: default
              type: disk
            eth0:
              name: eth0
              nictype: bridged
              parent: lxdbr0
              type: nic

    path: /usr/local/share/lxd-preseed.yaml
    permissions: '0644'
  - content: |
      #!/bin/bash
      umask 0027
      mkdir new-project
      cd new-project
      cat << END > snapcraft.yaml
      name: hello-world
      base: core22
      version: '1.0'
      summary: 'hello-world'
      description: 'hello-world'
      grade: devel
      confinement: strict
      
      parts:
        hello-world:
          plugin: dump
          source: .
      END
      snapcraft --use-lxd --verbosity=debug

    path: /usr/local/share/snapcraft-test.sh
    permissions: '0755'

runcmd:
  - [ snap, install, --classic, snapcraft ]
  - [ sh, -c, "lxd init --preseed < /usr/local/share/lxd-preseed.yaml"]

Then launched an instance:

multipass launch -n lxd-test -d 10g --cloud-init multipass/cloud-init/lxd-test.yaml jammy

But, running /usr/local/share/snapcraft-test.sh works !

I started thinking that it must be some local LXD config issue, so I purged my LXD install and started again.

snap remove --purge lxd
2023-10-25T14:14:57+01:00 INFO Waiting for "snap.lxd.daemon.service" to stop.
lxd removed

snap install lxd
lxd 5.19-31ff7b6 from Canonical✓ installed

Then initialised with the same pressed config as above:

sudo lxd init --preseed < lxd-preseed.yaml

…and I still get the same error!

Cheers, Just