Disconnected issue, a cocktail of running `snapcraft cleanbuild` in `multipass` with the `LXD` snap

I have created a multipass 16.04 image, inside I setup LXD with local storage:

ubuntu@snapcraft-xenial:~$ lxc storage list
+---------+-------------+--------+------------------------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |                     SOURCE                     | USED BY |
+---------+-------------+--------+------------------------------------------------+---------+
| default |             | dir    | /var/snap/lxd/common/lxd/storage-pools/default | 2       |
+---------+-------------+--------+------------------------------------------------+---------+

From the path you can see LXD is a snap.

ubuntu@snapcraft-xenial:~$ snap info lxd
name:      lxd
summary:   System container manager and API
publisher: canonical
contact:   https://github.com/lxc/lxd/issues
description: |
  LXD is a container manager for system containers.
  
  It offers a REST API to remotely manage containers over the network, using an
  image based workflow and with support for live migration.
  
  Images are available for all Ubuntu releases and architectures as well as for
  a wide number of other Linux distributions.
  
  LXD containers are lightweight, secure by default and a great alternative to
  virtual machines.
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
commands:
  - lxd.benchmark
  - lxd.check-kernel
  - lxd.lxc
  - lxd
  - lxd.migrate
services:
  lxd.daemon:    simple, enabled, active
tracking:        stable
installed:       2.21 (5556) 47MB -
refreshed:       2018-02-02 05:48:02 -0800 PST
channels:                           
  stable:        2.21        (5556) 47MB -
  candidate:     2.21        (5556) 47MB -
  beta:          ā†‘                       
  edge:          git-f966fcc (5589) 48MB -
  2.0/stable:    2.0.11      (5384) 21MB -
  2.0/candidate: 2.0.11      (5384) 21MB -
  2.0/beta:      ā†‘                       
  2.0/edge:      git-cdc6fc1 (5574) 20MB -

When I run snapcraft cleanbuild with --debug to get into a shell (error occurs either way) I get things like:

OSError: [Errno 107] Transport endpoint is not connected: '/snap/snapcraft/x1/usr/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py'

Manually while in the shell I can see:

root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# snapcraft 
internal error, please report: running "snapcraft" failed: open /snap/snapcraft/x1/meta/snap.yaml: transport endpoint is not connected

and,

root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# ls /snap/snapcraft/x1/meta
ls: cannot open directory '/snap/snapcraft/x1/meta': Transport endpoint is not connected

Without much knowledge, Iā€™d think this is a multipass/VM issue where my disk has been disconnected. Ideas?

Full output follows,

ubuntu@snapcraft-xenial:~/7z-hello$ snapcraft cleanbuild --debugCreating snapcraft-anthologically-transomed-taina
Starting snapcraft-anthologically-transomed-taina
Waiting for a network connection...
Network connection established
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://security.ubuntu.com/ubuntu xenial-security/main Sources [107 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/restricted Sources [2116 B]
Get:7 http://archive.ubuntu.com/ubuntu xenial/main Sources [868 kB]             
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [49.6 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [1520 B]       
Get:10 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [435 kB]     
Get:11 http://archive.ubuntu.com/ubuntu xenial/restricted Sources [4808 B]                
Get:12 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [189 kB]     
Get:13 http://archive.ubuntu.com/ubuntu xenial/universe Sources [7728 kB]                 
Get:14 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [201 kB] 
Get:15 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [102 kB] 
Get:16 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3208 B]
Get:17 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [1408 B]
Get:18 http://archive.ubuntu.com/ubuntu xenial/multiverse Sources [179 kB]                
Get:19 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7532 kB]          
Get:20 http://archive.ubuntu.com/ubuntu xenial/universe Translation-en [4354 kB]          
Get:21 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [144 kB]         
Get:22 http://archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB]         
Get:23 http://archive.ubuntu.com/ubuntu xenial-updates/main Sources [292 kB]              
Get:24 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Sources [2524 B]        
Get:25 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [190 kB]          
Get:26 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [7972 B]        
Get:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [713 kB]       
Get:28 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [296 kB]       
Get:29 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [581 kB]   
Get:30 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [234 kB]   
Get:31 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [16.2 kB]
Get:32 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [8052 B] 
Get:33 http://archive.ubuntu.com/ubuntu xenial-backports/main Sources [3436 B]            
Get:34 http://archive.ubuntu.com/ubuntu xenial-backports/universe Sources [4904 B]        
Get:35 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [4840 B]     
Get:36 http://archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [3220 B]     
Get:37 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [6628 B] 
Get:38 http://archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [3768 B] 
Fetched 24.7 MB in 45s (537 kB/s)                                                         
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  squashfuse
0 upgraded, 1 newly installed, 0 to remove and 10 not upgraded.
Need to get 17.4 kB of archives.
After this operation, 54.3 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 squashfuse amd64 0.1.100-0ubuntu1~ubuntu16.04.1 [17.4 kB]
Fetched 17.4 kB in 0s (1040 kB/s)
Selecting previously unselected package squashfuse.
(Reading database ... 25537 files and directories currently installed.)
Preparing to unpack .../squashfuse_0.1.100-0ubuntu1~ubuntu16.04.1_amd64.deb ...
Unpacking squashfuse (0.1.100-0ubuntu1~ubuntu16.04.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up squashfuse (0.1.100-0ubuntu1~ubuntu16.04.1) ...
Looking up assertion ['account-key', 'public-key-sha3-384=BWDEoaqyr25nF5SNCvEv2v7QnM9QsfCc0PBMYD_i2NGSQ32EF2d4D0hqUel3m8ul']
Looking up assertion ['snap-declaration', 'snap-name=core']
Looking up assertion ['snap-revision', 'snap-revision=3887', 'snap-id=99T7MUlRhtI3U0QFgl5mXXESAiSwt776']
Adding assertion core_3887.assert
Installing /run/core_3887.snap
2018-02-06T20:06:46Z INFO Waiting for restart...
core 16-2.30 from 'canonical' installed
Making snapcraft_x3.snap user-accessible
Installing /run/snapcraft_x3.snap
snapcraft 2.39+git17.5342d24 installed
Setting up container with project assets
./
./snap/
./test.7z
./snapcraft.yaml
Traceback (most recent call last):
  File "/snap/snapcraft/x1/bin/snapcraft", line 6, in <module>
  File "/snap/snapcraft/x1/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 74, in <module>
  File "/snap/snapcraft/x1/usr/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/snap/snapcraft/x1/usr/lib/python3.6/site-packages/pkg_resources/extern/__init__.py", line 43, in load_module
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 780, in get_code
  File "<frozen importlib._bootstrap_external>", line 832, in get_data
OSError: [Errno 107] Transport endpoint is not connected: '/snap/snapcraft/x1/usr/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py'
Debug mode enabled, dropping into a shell
root@snapcraft-anthologically-transomed-taina:~# ls
build_7z-hello  snap
root@snapcraft-anthologically-transomed-taina:~# ls snap/
snapcraft
root@snapcraft-anthologically-transomed-taina:~# ls snap/snapcraft/
common  current  x1
root@snapcraft-anthologically-transomed-taina:~# cd build_7z-hello/
root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# ls
7z-hello_0.1_source.tar.bz2  snap  snapcraft.yaml  test.7z
root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# snapcraft 
internal error, please report: running "snapcraft" failed: open /snap/snapcraft/x1/meta/snap.yaml: transport endpoint is not connected
root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# ls /snap/
README  bin  core  snapcraft
root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# ls /snap/snapcraft/x1/meta
ls: cannot open directory '/snap/snapcraft/x1/meta': Transport endpoint is not connected
root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# ls /snap/snapcraft/x1/
ls: cannot open directory '/snap/snapcraft/x1/': Transport endpoint is not connected
root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# mount
/dev/sda1 on / type ext4 (rw,relatime,data=ordered)
none on /dev type tmpfs (rw,nodev,relatime,size=492k,mode=755,uid=1000000,gid=1000000)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nodev,relatime)
udev on /dev/fuse type devtmpfs (rw,nosuid,relatime,size=498700k,nr_inodes=124675,mode=755)
udev on /dev/net/tun type devtmpfs (rw,nosuid,relatime,size=498700k,nr_inodes=124675,mode=755)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
tmpfs on /dev/lxd type tmpfs (rw,relatime,size=100k,mode=755)
tmpfs on /dev/.lxd-mounts type tmpfs (rw,relatime,size=100k,mode=711)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
udev on /dev/null type devtmpfs (rw,nosuid,relatime,size=498700k,nr_inodes=124675,mode=755)
udev on /dev/zero type devtmpfs (rw,nosuid,relatime,size=498700k,nr_inodes=124675,mode=755)
udev on /dev/full type devtmpfs (rw,nosuid,relatime,size=498700k,nr_inodes=124675,mode=755)
udev on /dev/urandom type devtmpfs (rw,nosuid,relatime,size=498700k,nr_inodes=124675,mode=755)
udev on /dev/random type devtmpfs (rw,nosuid,relatime,size=498700k,nr_inodes=124675,mode=755)
udev on /dev/tty type devtmpfs (rw,nosuid,relatime,size=498700k,nr_inodes=124675,mode=755)
devpts on /dev/console type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/pts type devpts (rw,relatime,gid=1000005,mode=620,ptmxmode=666,max=1024)
devpts on /dev/ptmx type devpts (rw,relatime,gid=1000005,mode=620,ptmxmode=666,max=1024)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,uid=1000000,gid=1000000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755,uid=1000000,gid=1000000)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,uid=1000000,gid=1000000)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755,uid=1000000,gid=1000000)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
squashfuse on /snap/core/3887 type fuse.squashfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
squashfuse on /snap/snapcraft/x1 type fuse.squashfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
root@snapcraft-anthologically-transomed-taina:~/build_7z-hello# mount|grep 

OK, maybe not VM/disk related, much more interesting:

[ 1021.767666] squashfuse[5246]: segfault at 8 ip 00005643bf63d7c4 sp 00007ffd5666e000 error 4 in squashfuse[5643bf63a000+8000]

That is running one

Linux snapcraft-anthologically-transomed-taina 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Sergio and I were able to track down the problem. He downloaded a snap over flaky wifi via a wget command. The bad copy of the snap was then uploaded to the store. This caused problems for squashfuse (the squashfs kernel driver handled it a more gracefully).

1 Like

Is there a bug for squashfuse that needs filing then?

1 Like