Ubuntu-image 'No space left on device'

Ultimately, I’m trying to build an Ubuntu Core image for an Atmel SAMA5D36-based dev-kit. Since this is my first foray into Snappy, I figured I’d start with a Raspberry Pi 3 first, since it’s officially supported. I’ve managed to get Ubuntu Core up and running just fine on the Pi3, had a bit of a play with it, and am happy enough that it’s working.

Next, I thought I would try and build my own image for the Pi3, and then apply that knowledge to the Atmel board. Following the guide on the Ubuntu docs site, I thought I would try build first using a Canonical signed model assertion just to check I had everything in place to build and run Ubuntu Core on the Pi3.

I have ubuntu-image installed from a Snap package.

$ mkdir pi3snap
$ cd pi3snap/
$ snap known --remote model series=16 brand-id=canonical model=pi3 > pi3.model
$ sudo ubuntu-image snap -c beta pi3.model

I added the snap keyword to the ubuntu-image command, since the docs seem to be slightly out-of-date - but other than that the command seems to be correct.

It starts running, downloads the pieces, but then crashes with this:

Fetching core
Fetching pi2-kernel
Fetching pi3
COMMAND FAILED: sudo cp -dR --preserve=mode,timestamps,ownership /tmp/tmpac3_hgzw/root/* /tmp/tmpjhuelyyw/root-mount

cp: error writing '/tmp/tmpjhuelyyw/root-mount/system-data/var/lib/snapd/snaps/core_4020.snap': No space left on device
cp: error writing '/tmp/tmpjhuelyyw/root-mount/system-data/var/lib/snapd/snaps/pi2-kernel_50.snap': No space left on device
cp: cannot create directory '/tmp/tmpjhuelyyw/root-mount/system-data/var/lib/snapd/seed': No space left on device
cp: cannot create directory '/tmp/tmpjhuelyyw/root-mount/system-data/boot': No space left on device

Crash in state machine
Traceback (most recent call last):
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/__main__.py", line 341, in main
    list(state_machine)
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/state.py", line 82, in __next__
    step()
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/common_builder.py", line 337, in populate_filesystems
    self._populate_one_volume(name, volume)
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/common_builder.py", line 290, in _populate_one_volume
    preserve_ownership=True)
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/helpers.py", line 211, in mkfs_ext4
    preserve_flags, contents_dir, mountpoint), shell=True)
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/helpers.py", line 112, in run
    proc.check_returncode()
  File "/snap/ubuntu-image/98/usr/lib/python3.5/subprocess.py", line 349, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command 'sudo cp -dR --preserve=mode,timestamps,ownership /tmp/tmpac3_hgzw/root/* /tmp/tmpjhuelyyw/root-mount' returned non-zero exit status 1

There’s no way I don’t have enough disk space:

Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           1.6G  9.7M  1.6G   1% /run
/dev/sda1       432G  270G  146G  65% /
tmpfs           7.8G  136M  7.7G   2% /dev/shm
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/loop0       84M   84M     0 100% /snap/core/3604
...
/dev/loop5       84M   84M     0 100% /snap/core/3748
/dev/loop1       82M   82M     0 100% /snap/core/3887
/dev/loop7       33M   33M     0 100% /snap/ubuntu-image/98
/dev/nvme0n1p4  227G  179G   48G  79% /mnt/windows
/dev/nvme0n1p2  356M   77M  280M  22% /boot/efi
/dev/sda1       432G  270G  146G  65% /home
...
tmpfs           1.6G  540K  1.6G   1% /run/user/1000
/dev/loop8       55M   55M     0 100% /snap/snapcraft/1094

Once when trying this and spamming df -h, I got this: /dev/loop8 3.9M 52K 3.3M 2% /tmp/tmpvn70tzdq/root-mount, seemingly suggesting that the loop device being created is only 3.9 MB.

What’s going on here? Am I doing something wrong or is this a bug? I’ve tried it from two machines, one with 16.04 and one with 17.10 - both have the same issue.

Try -w build option for ubuntu-image.

the snap keyword is the default anyway, so you dont need it (but it wont do any harm either).

which ubuntu-image and snapd versions do you have installed?
check with:

snap info ubuntu-image
...
snap version

loop devices are not involved in building the snap image, just ignore them when debugging (technically not even sudo is required to run it, but that makes some permissions inside the image to be wrong and triggers a cleanup process on first boot of the image which will just be skipped when you build with sudo)

ubuntu-image builds images in /tmp which on a typical ubuntu machine is a tmpfs, i dont see /tmp in your df -h output, was it in the cut out part or did you disable /tmp being a tmpfs on your system … ?

if you use -w like @thymythos said, it will not use /tmp but the dir specified, so you can pick a place where you definitely have space … nontheless ubuntu-image should better check for free space before blindly running its head against the wall, you should file a bug about this at:

Thanks both,

I’ve tried the -w flag, but still see the issue:

$ sudo ubuntu-image snap -w /home/sean/pi3snap pi3.model 
Fetching core
Fetching pi2-kernel
Fetching pi3
COMMAND FAILED: sudo cp -dR --preserve=mode,timestamps,ownership /home/sean/pi3snap/root/* /tmp/tmpt0du7qwi/root-mount

cp: error writing '/tmp/tmpt0du7qwi/root-mount/system-data/var/lib/snapd/snaps/core_4020.snap': No space left on device
cp: error writing '/tmp/tmpt0du7qwi/root-mount/system-data/var/lib/snapd/snaps/pi2-kernel_22.snap': No space left on device
cp: cannot create directory '/tmp/tmpt0du7qwi/root-mount/system-data/var/lib/snapd/seed': No space left on device
cp: cannot create directory '/tmp/tmpt0du7qwi/root-mount/system-data/boot': No space left on device

Crash in state machine
Traceback (most recent call last):
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/__main__.py", line 341, in main
    list(state_machine)
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/state.py", line 82, in __next__
    step()
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/common_builder.py", line 337, in populate_filesystems
    self._populate_one_volume(name, volume)
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/common_builder.py", line 290, in _populate_one_volume
    preserve_ownership=True)
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/helpers.py", line 211, in mkfs_ext4
    preserve_flags, contents_dir, mountpoint), shell=True)
  File "/snap/ubuntu-image/98/lib/python3/site-packages/ubuntu_image/helpers.py", line 112, in run
    proc.check_returncode()
  File "/snap/ubuntu-image/98/usr/lib/python3.5/subprocess.py", line 349, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command 'sudo cp -dR --preserve=mode,timestamps,ownership /home/sean/pi3snap/root/* /tmp/tmpt0du7qwi/root-mount' returned non-zero exit status 1

This is my full df, interestingly there isn’t a listing for /tmp:

$  sudo df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           3.9G  9.7M  3.9G   1% /run
/dev/sda1       432G  246G  168G  60% /
tmpfs           7.8G  121M  7.7G   2% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/loop8       33M   33M     0 100% /snap/ubuntu-image/98
/dev/loop2       85M   85M     0 100% /snap/rocketchat-desktop/4
/dev/loop0       76M   76M     0 100% /snap/keepassxc/28
/dev/loop1       82M   82M     0 100% /snap/core/3887
/dev/loop5       77M   77M     0 100% /snap/keepassxc/27
/dev/loop4       84M   84M     0 100% /snap/core/3748
/dev/loop6       87M   87M     0 100% /snap/keepassxc/30
/dev/loop7       55M   55M     0 100% /snap/snapcraft/1094
/dev/nvme0n1p4  227G  179G   48G  79% /mnt/windows
/dev/nvme0n1p2  356M   77M  280M  22% /boot/efi
/dev/sda1       432G  246G  168G  60% /home
/dev/sda1       432G  246G  168G  60% /mnt/btrfs-ssd
tmpfs           1.6G  516K  1.6G   1% /run/user/1000
/dev/loop9       82M   82M     0 100% /snap/core/4017

I haven’t (yet) done anything to change or mess with /tmp, I only recently installed on this machine, to a btrfs filesystem. I’ll try adding a line to /etc/fstab to add /tmp as a tmpfs.

OK, success!

I added an entry to my /etc/fstab and rebooted:

none            /tmp                      tmpfs           noexec,nosuid,size=25%,mode=0755             0 0

And now when I run df I get:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           3.9G  9.7M  3.9G   1% /run
/dev/sda1       432G  246G  168G  60% /
tmpfs           7.8G  135M  7.7G   2% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
none            3.9G  4.0K  3.9G   1% /tmp
...
/dev/sda1       432G  246G  168G  60% /mnt/btrfs-ssd
/dev/sda1       432G  246G  168G  60% /home
tmpfs           1.6G   80K  1.6G   1% /run/user/1000

Interestingly, it lists /tmp as none type. But, anyway, I’ve now run the snap command again, and voilà, I have a pi3.img file. I haven’t checked if it actually works yet, but that wasn’t the problem.

I do also have a bug open on Launchpad, I’d actually opened that before asking this question, but it wasn’t getting much love.

Why should ubuntu-image care how the /tmp directory is mounted? Perhaps the mode=0755 line? If so though, especially since it’s run as sudo, it should surely be able to figure it out?

well, normally you would use 1777 for /tmp …

Hmm, I just copied that line from /lib/init/fstab. Which I didn’t really know much about, and can see doesn’t exist in 17.10, but seemed like a somewhat reasonable default to go with.

I don’t yet know the specific answer here in this case, but I suspect this
relates to classic confinement and filesystem namespacing. Snaps in general
run within a private filesystem namespace, with various bind mounts from the
underlying host filesystem as configured via snap interfaces. /tmp in
particular is handled specially by confined snaps; but ubuntu-image is a
classic snap, which means it should have access to the underlying /tmp, but
if you’re running out of space on /tmp within the snap’s namespace, that
implies that it’s not using your host /tmp.

Has there been a change then in the handling of filesystem namespace for
classic snaps?