UC20 image based on reference amd64 gadget failing to install

Hi,

I’ll start by saying I know that UC is under heavy development atm in preparation for the upcoming release, and I’m not using production yet, but we are developping against it as a target as that is what we expect to use when we finally go into production early next year.

I created a new image, with a new gadget config based ont eh recent changes in the reference amd64 gadget here: https://github.com/snapcore/pc-amd64-gadget

Using I use the edge version of snapd, it builds and boots fine. However, during setup if tails with this error:

snapd[1277]: taskrunner.go:271: [change 2 "Setup system for run mode" task] failed: cannon instsall system: gadget and /dev/sda partition table not compatible: cannon find disk partition /dev/sda2 (starting at 2097152) in gadget: on disk size larger than gadget size (and the role should not be expanded)

Parted reports this:

(parted) print
Model: ATA CT500MX500SSD1 (scsi)
Disk /dev/sda: 500107862016B
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start     End          Size         File system  Name         Flags
 1      1048576B  2097151B     1048576B                  BIOS Boot    bios_grub
 2      2097152B  1400369151B  1398272000B  fat32        ubuntu-seed  boot, esp

…and again in mib

Number  Start    End      Size     File system  Name         Flags
 1      1.00MiB  2.00MiB  1.00MiB               BIOS Boot    bios_grub
 2      2.00MiB  1335MiB  1333MiB  fat32        ubuntu-seed  boot, esp

It seems like the ubuntu-seed is bigger than expected for some reason. Is that the issue ?

If that’s the case, what’s the likely cause, and should I just bump the size ?

Cheers,
Just

I guess it comes down to this message at image build time:

DEBUG:ubuntu-image:-> [10] prepare_filesystems
WARNING:ubuntu-image:rootfs partition size (1258291200) smaller than actual rootfs contents 1398302720
WARNING:ubuntu-image:rootfs partition size (1073741824) smaller than actual rootfs contents 1398302720

The strange thing is, if I mount that part of the image, and check it’s usage with df, it shows less than a 1g , so I’m not sure why it thinks it should be bigger.

I have tried with the current/edge ubunui-image too, same thing.

I will try changing the gadget volume sizing, but it seems strange to me still.

Cheers,
Just

Hi, rather than changing the gadget size, can you pass the -i 8G argument to ubuntu-image? We still have a bug in ubuntu-image where it does not generate a correctly sized image unless you pass the image size to it like this.

Thanks @ijohnson - I tried that, but unfortunately it doesn’t help. I get the same messages at image build time, and the same issue at system setup.

I did also try bumping the site. of ubuntu-seed in the gadget to 1500M. That helped, but then boot fails on another, possibly related issue.

Now, after system setup and first reboot, it reaches this point:

           Mounting /run/mnt/ubuntu-seed...
[    56.982979]  FAT-fs (sda2): Volume was not properly unmounted. Some data may be corrupt.  Please run fsck.
[   OK   ] Mounted /run/mnt/ubuntu-seed.
[ FAILED ] Failed to start the-tool.service.
See `systemctl status the-tool.service` for details.

I tried booting off a live image and doing an fsck, removed the dirty bit, rebooted, and although there is no error regarding mounting ubuntu-seed, the-tool.service still fails.

Not sure if this is related or not now. I don’t have any detail printed on the screen regarding the reason for the failure of the-tool.service

Cheers,
Just

Can you show me the gadget.yaml you are using for this gadget snap ?

Sure, I’ve removed some comments and changed the volume name, but otherwise as is:

volumes:
  foo-model-name:
    # bootloader configuration is shipped and managed by snapd
    bootloader: grub
    structure:
      - name: mbr
        type: mbr
        size: 440
        update:
          edition: 1
        content:
          - image: pc-boot.img
      - name: BIOS Boot
        type: DA,21686148-6449-6E6F-744E-656564454649
        size: 1M
        offset: 1M
        offset-write: mbr+92
        update:
          edition: 2
        content:
          - image: pc-core.img
      - name: ubuntu-seed
        role: system-seed
        filesystem: vfat
        # UEFI will boot the ESP partition by default first
        type: EF,C12A7328-F81F-11D2-BA4B-00A0C93EC93B
        size: 1200M
        update:
          edition: 2
        content:
          - source: grubx64.efi
            target: EFI/boot/grubx64.efi
          - source: shim.efi.signed
            target: EFI/boot/bootx64.efi
      - name: ubuntu-boot
        role: system-boot
        filesystem: ext4
        type: 83,0FC63DAF-8483-4772-8E79-3D69D8477DE4
        # whats the appropriate size?
        size: 750M
        update:
          edition: 1
        content:
          - source: grubx64.efi
            target: EFI/boot/grubx64.efi
          - source: shim.efi.signed
            target: EFI/boot/bootx64.efi
      - name: ubuntu-save
        role: system-save
        filesystem: ext4
        type: 83,0FC63DAF-8483-4772-8E79-3D69D8477DE4
        size: 16M
      - name: ubuntu-data
        role: system-data
        filesystem: ext4
        type: 83,0FC63DAF-8483-4772-8E79-3D69D8477DE4
        size: 1G


defaults:
  rW4inp7UbJb1YBxWr6SVebxa3Yv7K1Vm:
    daemon: true
    cursor: none

connections:
  - plug: EeKt3E4xZajMZnpT58644owZ5JkU6gQN:network-control
  - plug: EeKt3E4xZajMZnpT58644owZ5JkU6gQN:daemon-notify
  - plug: 6eYkl5cx8jsq6G4zLej96KEve41TpZO1:snapd-control
  - plug: HMLworN5rMNJLe9aSt1N93ip9mT0BtO4:docker-executables
    slot: sLCsFAO8PKM5Z0fAKNszUOX0YASjQfeZ:docker-executables
  - plug: HMLworN5rMNJLe9aSt1N93ip9mT0BtO4:docker-cli
    slot: sLCsFAO8PKM5Z0fAKNszUOX0YASjQfeZ:docker-daemon

Forgot to say, that’s with the reference sizing for ubuntu-seed

Cheers,
Just

What version of ubuntu-image are you using? snap list ubuntu-image

You might for some reason need to use the version from edge which is from October, while stable is from September

I’m currently on edge. I was on stable initially, but switched to edge earlier in case that was some compatibility issue.

ubuntu-image       1.10+snap1          196    latest/edge      canonical*  classic

It seems to me that you have several extra snaps besides kernel/gadget/base/snapd.
Although you have tried to enlarge ubuntu-seed to 1.5G, but I think it’s worth to try a larger one. For example, 2.5G.
The reason is that when snapd is installing, it need some disk space for file operations.

Yes, that’s true. The actual space consumed is less than the 1.2G, still I guess there is some logic in ubuntu-image to give it some percentage of free overhead on top of that, but even that is not quite enough at setup time perhaps.

I will try increasing the sizes more still.

I wonder if lack of disk space is why the-tool.service is failing after reboot too. Does anyone know how I can debug that some more ?

Cheers,
Just

It’s probably unlikely that the-tool is failing due to lack of disk-space IMHO

Unfortunately not without rebuilding your kernel snap, etc. however I have just filed a PR which should make it’s way into the kernel snap soon which will give actual error message logs from “the-tool” in the initramfs so we can at least see what’s failing for you.

That sounds good @ijohnson

For the moment I have back-peddled a bit and am basing our development around non-edge versions of snapd and core20. It seems I was fighting a few issues at once, and I’m starting to get to the bottom of them all one by one. When I have solved them all I will try edge again and see if the problem persists.

Can I just ask for my own info - what is the-tool responsible for ?

Cheers,
Just

“the-tool” is a very poorly named service which runs in the initramfs which does all the snap-specific things for Ubuntu Core 20 that need to happen in the initramfs, so it’s a bit of a giant bag of various things. The main things it is responsible for are:

  • securely and correctly mounting various partitions in /run so they can be used in userspace
  • securely unlocking any encrypted partitions and mounting the decrypted versions
  • locking down the TPM if it was used to unlock any encrypted partitions so it can’t be tampered with or have secrets removed from it during userspace
  • picking which base snap to mount as the rootfs
  • picking which kernel snap to mount to use kernel modules from
  • handling base and kernel snap upgrade A/B try/trying logic
  • other miscellaneous things related to either security or robustness

Really all “the-tool” does these days is call snap-bootstrap initramfs-mounts, you can see that over at https://github.com/snapcore/core-initrd/blob/main/factory/usr/lib/systemd/system/the-tool.service and https://github.com/snapcore/core-initrd/blob/main/factory/usr/lib/the-tool. The snapd side of things then comes mainly into play in https://github.com/snapcore/snapd/blob/master/cmd/snap-bootstrap/cmd_initramfs_mounts.go.

I’ve had it on my TODO to try and rename some of these services in the initramfs such as “the-modeenv” and “the-tool” to more logical things, but I’ve been caught up in various other problems and haven’t been able to make those changes yet…

Thanks for the info - it helps to build up the picture how it all plumbs together :slight_smile:

Cheers,
Just