Snapcraft launch never finishes (but the latest edge version does...)

I was able to build core22 snaps on my Ubuntu 20.04 workstation just fine but once I upgraded to Ubuntu 22.04, I am no longer able to build the same snaps. I installed the latest edge version of snapcraft (snapcraft 7.5.2.post62+git5ec65724) to get better diagnostics and if I run:

snapcraft --debug --verbosity=trace --use-lxd

it fails with the error:

> subprocess.TimeoutExpired: Command '['lxc', '--project', 'snapcraft', 'exec', 'local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667', '--', 'env', 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'SNAPCRAFT_MANAGED_MODE=1', 'getent', 'hosts', 'snapcraft.io']' timed out after 10 seconds

Note: If I use multipass as the provider this problem does not occur.

How do I debug this? Any help is appreciated…

The full output pad is:

> 2023-08-16 13:38:13.610 Starting Snapcraft 7.5.2.post62+git5ec65724                                                                                               
> 2023-08-16 13:38:13.610 Logging execution to '/home/denis/.local/state/snapcraft/log/snapcraft-20230816-133813.609574.log'                                        
> 2023-08-16 13:38:13.611 Raw pre-parsed sysargs: args={'help': False, 'verbose': False, 'quiet': False, 'verbosity': 'trace', 'version': False, 'trace': False} filtered=['--debug', '--use-lxd']                                                                                                                                     
> 2023-08-16 13:38:13.612 Using default command: 'pack'                                                                                                             
> 2023-08-16 13:38:13.612 General parsed sysargs: command='pack' args=['--debug', '--use-lxd']                                                                      
> 2023-08-16 13:38:13.613 Command parsed sysargs: Namespace(destructive_mode=False, use_lxd=True, debug=True, enable_manifest=False, manifest_image_information=None, bind_ssh=False, build_for=None, http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None)
> 2023-08-16 13:38:13.613 lifecycle command: 'pack', arguments: Namespace(destructive_mode=False, use_lxd=True, debug=True, enable_manifest=False, manifest_image_information=None, bind_ssh=False, build_for=None, http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None)                                                                                                                                                     
> 2023-08-16 13:38:13.613 command: pack, arguments: Namespace(destructive_mode=False, use_lxd=True, debug=True, enable_manifest=False, manifest_image_information=None, bind_ssh=False, build_for=None, http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None)                                                                                                                                                                 
> 2023-08-16 13:38:13.621 Created build plan:
>   build-on: amd64 build-for: amd64                                                                                    
> 2023-08-16 13:38:13.621 CPU count (from process affinity): 6                                                                                                      
> 2023-08-16 13:38:13.621 Invalid SNAPCRAFT_MAX_PARALLEL_BUILD_COUNT ''                                                                                             
> 2023-08-16 13:38:13.621 Running on amd64 for amd64                                                                                                                
> 2023-08-16 13:38:13.622 Checking build provider availability                                                                                                      
> 2023-08-16 13:38:13.631 Retrieved snap config: {}                                                                                                                 
> 2023-08-16 13:38:13.632 Using provider 'lxd' passed as an argument.                                                                                               
> 2023-08-16 13:38:13.786 Executing on host: lxc --project default profile show local:default                                                                       
> 2023-08-16 13:38:13.893 Using hostname 'snapcraft-optumi-agent-on-amd64-for-amd64-7020667'                                                                        
> 2023-08-16 13:38:13.893 Launching instance...                                                                                                                     
> 2023-08-16 13:38:13.893 Executing on host: lxc remote list --format=yaml                                                                                          
> 2023-08-16 13:38:14.000 Remote 'craft-com.ubuntu.cloud-buildd' already exists.                                                                                    
> 2023-08-16 13:38:14.000 Executing on host: lxc project list local: --format=yaml                                                                                  
> 2023-08-16 13:38:14.076 Set LXD instance name to 'snapcraft-optumi-agent-on-amd64-for-amd64-7020667'                                                              
> 2023-08-16 13:38:14.076 Checking for instance 'snapcraft-optumi-agent-on-amd64-for-amd64-7020667' in project 'snapcraft' in remote 'local'                        
> 2023-08-16 13:38:14.076 Executing on host: lxc --project snapcraft list local: --format=yaml                                                                      
> 2023-08-16 13:38:14.183 Executing on host: lxc --project snapcraft config get local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 raw.idmap                   
> 2023-08-16 13:38:14.280 Unexpected id map for 'snapcraft-optumi-agent-on-amd64-for-amd64-7020667' (expected 'both 1001 0', got '').                               
> 2023-08-16 13:38:14.280 Cleaning incompatible instance 'snapcraft-optumi-agent-on-amd64-for-amd64-7020667' (reason: the instance's id map ('raw.idmap') is not configured as expected).                                                                                                                                              
> 2023-08-16 13:38:14.280 Executing on host: lxc --project snapcraft delete local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 --force                         
> 2023-08-16 13:38:15.890 Instance 'snapcraft-optumi-agent-on-amd64-for-amd64-7020667' does not exist.                                                              
> 2023-08-16 13:38:15.890 Set LXD instance name to 'base-instance-snapcraft-buildd-base-v10--2d55678e0339df02cd48'                                                  
> 2023-08-16 13:38:15.890 Checking for base instance 'base-instance-snapcraft-buildd-base-v10--2d55678e0339df02cd48' in project 'snapcraft' in remote 'local'       
> 2023-08-16 13:38:15.890 Executing on host: lxc --project snapcraft list local: --format=yaml                                                                      
> 2023-08-16 13:38:15.971 Base instance 'base-instance-snapcraft-buildd-base-v10--2d55678e0339df02cd48' does not exist.                                             
> 2023-08-16 13:38:15.971 Creating new instance from image 'core22' from remote 'craft-com.ubuntu.cloud-buildd'.                                                    
> 2023-08-16 13:38:15.971 Executing on host: lxc --project snapcraft info local:                                                                                    
> 2023-08-16 13:38:16.073 Executing on host: lxc --project snapcraft launch craft-com.ubuntu.cloud-buildd:core22 local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 --config security.syscalls.intercept.mknod=true                                                                                                               
> 2023-08-16 13:38:21.350 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 test -f /etc/craft-instance.conf                                                    
> 2023-08-16 13:38:21.516 Executing on host: lxc --project snapcraft file push /tmp/tmpeza4k0r7 local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667/etc/craft-instance.conf --mode=0644                                                                                                                                            
> 2023-08-16 13:38:21.633 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 chown root:root /etc/craft-instance.conf                                            
> 2023-08-16 13:38:21.852 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 cat /etc/os-release                                                                 
> 2023-08-16 13:38:22.077 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 test -f /etc/craft-instance.conf                                                    
> 2023-08-16 13:38:22.243 Executing on host: lxc --project snapcraft file pull local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667/etc/craft-instance.conf /home/denis/tmphggc_fni.tmp-craft/tmphyz_nh3i                                                                                                                           
> 2023-08-16 13:38:22.335 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 test -f /etc/craft-instance.conf                                                    
> 2023-08-16 13:38:22.501 Executing on host: lxc --project snapcraft file pull local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667/etc/craft-instance.conf /home/denis/tmp264_ecde.tmp-craft/tmpaexijn29                                                                                                                           
> 2023-08-16 13:38:22.590 Executing on host: lxc --project snapcraft file push /tmp/tmpzwg_taxl local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667/etc/craft-instance.conf --mode=0644                                                                                                                                            
> 2023-08-16 13:38:22.685 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 chown root:root /etc/craft-instance.conf                                            
> 2023-08-16 13:38:22.829 Executing on host: lxc --project snapcraft file push /tmp/tmpxyfw9soc local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667/etc/environment --mode=0644                                                                                                                                                    
> 2023-08-16 13:38:22.959 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 chown root:root /etc/environment                                                    
> 2023-08-16 13:38:23.105 Executing on host: lxc --project snapcraft file push /tmp/tmpphbqulv4 local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667/etc/apt/apt.conf.d/20auto-upgrades --mode=0644                                                                                                                                 
> 2023-08-16 13:38:23.187 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 chown root:root /etc/apt/apt.conf.d/20auto-upgrades                                 
> 2023-08-16 13:38:23.322 Waiting for environment to be ready...                                                                                                    
> 2023-08-16 13:38:23.322 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 systemctl is-system-running                                                         
> 2023-08-16 13:38:23.452 Executing on host: lxc --project snapcraft file push /tmp/tmpkq1p30il local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667/etc/hostname --mode=0644                                                                                                                                                       
> 2023-08-16 13:38:23.547 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 chown root:root /etc/hostname                                                       
> 2023-08-16 13:38:23.706 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 hostname -F /etc/hostname                                                           
> 2023-08-16 13:38:23.886 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf                            
> 2023-08-16 13:38:24.013 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 systemctl enable systemd-resolved                                                   
> 2023-08-16 13:38:24.206 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 systemctl restart systemd-resolved                                                  
> 2023-08-16 13:38:24.391 Executing on host: lxc --project snapcraft file push /tmp/tmp5wutfchl local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667/etc/systemd/network/10-eth0.network --mode=0644                                                                                                                                
> 2023-08-16 13:38:24.476 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 chown root:root /etc/systemd/network/10-eth0.network                                
> 2023-08-16 13:38:24.605 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 systemctl enable systemd-networkd                                                   
> 2023-08-16 13:38:24.846 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 systemctl restart systemd-networkd                                                  
> 2023-08-16 13:38:25.049 Waiting for networking to be ready...                                                                                                     
> 2023-08-16 13:38:25.050 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 getent hosts snapcraft.io                                                           
> 2023-08-16 13:38:25.437 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 getent hosts snapcraft.io                                                           
> 2023-08-16 13:38:25.848 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 getent hosts snapcraft.io                                                           
> 2023-08-16 13:38:26.232 Executing in container: lxc --project snapcraft exec local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAPCRAFT_MANAGED_MODE=1 getent hosts snapcraft.io / (9.9s)                                                  Traceback (most recent call last):
>   File "/snap/snapcraft/9577/bin/snapcraft", line 8, in <module>
>     sys.exit(run())
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/snapcraft/cli.py", line 255, in run
>     _run_dispatcher(dispatcher, global_args)
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/snapcraft/cli.py", line 228, in _run_dispatcher
>     dispatcher.run()
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 462, in run
>     return self._loaded_command.run(self._parsed_command_args)
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/snapcraft/commands/lifecycle.py", line 265, in run
>     super().run(parsed_args)
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/snapcraft/commands/lifecycle.py", line 138, in run
>     parts_lifecycle.run(self.name, parsed_args)
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/snapcraft/parts/lifecycle.py", line 216, in run
>     _run_command(
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/snapcraft/parts/lifecycle.py", line 262, in _run_command
>     _run_in_provider(project, command_name, parsed_args)
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/snapcraft/parts/lifecycle.py", line 556, in _run_in_provider
>     with provider.launched_environment(
>   File "/snap/snapcraft/9577/usr/lib/python3.10/contextlib.py", line 135, in __enter__
>     return next(self.gen)
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_providers/lxd/lxd_provider.py", line 145, in launched_environment
>     instance = launch(
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_providers/lxd/launcher.py", line 545, in launch
>     _create_instance(
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_providers/lxd/launcher.py", line 74, in _create_instance
>     base_configuration.setup(executor=instance)
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_providers/base.py", line 970, in setup
>     self._setup_wait_for_network(executor=executor)
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_providers/base.py", line 438, in _setup_wait_for_network
>     proc = self._execute_run(
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_providers/base.py", line 1086, in _execute_run
>     proc = executor.execute_run(
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_providers/lxd/lxd_instance.py", line 293, in execute_run
>     return self.lxc.exec(
>   File "/snap/snapcraft/9577/lib/python3.10/site-packages/craft_providers/lxd/lxc.py", line 374, in exec
>     return runner(  # pylint: disable=subprocess-run-check
>   File "/snap/snapcraft/9577/usr/lib/python3.10/subprocess.py", line 505, in run
>     stdout, stderr = process.communicate(input, timeout=timeout)
>   File "/snap/snapcraft/9577/usr/lib/python3.10/subprocess.py", line 1154, in communicate
>     stdout, stderr = self._communicate(input, endtime, timeout)
>   File "/snap/snapcraft/9577/usr/lib/python3.10/subprocess.py", line 2022, in _communicate
>     self._check_timeout(endtime, orig_timeout, stdout, stderr)
>   File "/snap/snapcraft/9577/usr/lib/python3.10/subprocess.py", line 1198, in _check_timeout
>     raise TimeoutExpired(
> subprocess.TimeoutExpired: Command '['lxc', '--project', 'snapcraft', 'exec', 'local:snapcraft-optumi-agent-on-amd64-for-amd64-7020667', '--', 'env', 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'SNAPCRAFT_MANAGED_MODE=1', 'getent', 'hosts', 'snapcraft.io']' timed out after 10 seconds

We run getent hosts snapcraft.io to verify network connectivity inside the container, which unfortunately it looks like is unavailable.

Do you happen to have Docker installed? That is the most common culprit for interfering with LXD network connectivity.

Hi Cal,

I have 6 workstations running Ubuntu 22.04 and only one of them has this problem - there is probably some installation history to explain that. And yes, it has Docker installed and I probably mistakenly thought that because multipass works fine, that it was not a network connectivity issue.

I have tried to Google the various Docker fixes but wind up running tinto broken links now that LXD has moved back to Canonical. Can you point me to a doc that can enlighten me on how to remedy this?

Much appreciate your help! /denis

Sure, here’s a new link with some information on resolving docker+lxd compatibility issues: