Extending CI for snapd to debian/fedora

Hey everyone,

as part of my cross-distro work I am also working on getting the CI infrastructure we have for snapd work on other distributions so we can run tests for any PR somebody creates against all supported distributions equally and see when one breaks. There is a lot of work towards that and I decided to start with the lowest hanging fruit which is Debian as it is naturally very similar to Ubuntu.

I have a branch (rough, not finished yet or ready for review) which adjusts multiple things in the existing spread infrastructure to allow us running all tests on a Debian image. As a base I use one of the openstack debian images for the current testing branch (upcoming Debian 9 - stretch) as that is the only one we support with snapd today next to unstable.

A next step is to get a proper base image onto our Linode infrastructure which is based on testing. @niemeyer asked me to give a summary here so we can discuss the next steps for this.

2 Likes

Itā€™s great to see this finally moving forward! Thanks for pushing it, Simon.

To cook an image for your target platform, please follow these steps:

  1. git clone https://github.com/snapcore/spread-images
  2. Edit spread.yaml and add your target system at the top
  3. Make sure the project restore section in spread.yaml makes sense to your system
  4. Run spread -reuse -shell <system>
  5. Once you get the shell, transform the whole image towards what you want
  6. Exit the shell
  7. The project restore will automatically run and ought to restore your image to a pristine state
  8. Fix the update procedure in tasks/update/task.yaml or create a new equivalent task if too different
  9. Run the update procedure with spread -reuse <system>:tasks/update
  10. Once that works, let me know the machine name and Iā€™ll snapshot the image.
  11. Propose a PR to snapcore/spread-images with your changes.

Step 2 starts off from a pre-existing image. For Debian, we can start with debian-sid:

    systems:
        ...
        debian-testing:
            kernel: GRUB 2
            image: debian-sid

Note that the very first image baked will likely be based on a pre-existing Linode image such as ā€œDebian 8ā€, and those use a custom kernel from Linode, so the ā€œkernel: GRUB 2ā€ line above needs to be temporarily commented out and after the image boots the real distribution kernel must be installed.

After the final image is baked, we drop that image line and always use GRUB 2 so that the kernel inside the image itself is used.

@niemeyer I wanted to perform these steps today but I donā€™t have a Linode auth key yet. Are you the right person to get me one?

Also Linode doesnā€™t list any other existing image than debian-7 or debian-8 as available images so where does the debian-sid one you mention come from?

Sent a new key your way.

The debian-sid image was created in our own account following a similar process about 7 months ago.

Thanks!

Ok, let me try that one.

Its

id: 2227753
label: Spread-12

https://github.com/snapcore/spread-images/pull/1

That image ended up pretty large at around 1600MB. Do you know why the increase in space needs? For comparison, the previous debian-sid was 1024MB and the ubuntu-16.04-64 one is at 610MB (!).

The PR needs to be updated so it drops the image: line since weā€™ll be baking that image with its real name, per note above at the end.

The overview gives

linode:debian-unstable-64 /# du -hs *
8.1M	bin
57M	boot
0	dev
4.3M	etc
20K	home
0	initrd.img
0	initrd.img.old
404M	lib
4.0K	lib64
16K	lost+found
8.0K	media
4.0K	mnt
4.0K	opt
0	proc
304K	root
2.9M	run
5.8M	sbin
4.0K	srv
0	sys
28K	tmp
634M	usr
30M	var
0	vmlinuz
0	vmlinuz.old

The two biggest items are

  • /lib
  • /usr

/lib covers kernel modules for multiple kernels and I removed those which are not in use and have corresponding debian packages. Two remain which seem to come from Linode but donā€™t have corresponding debian packages:

  • /lib/modules/4.4.0-x86_64-linode63
  • /lib/modules/4.5.5-x86_64-linode69

Both donā€™t cover much same (4K) so we can ignore those.

The image size should be down to ~940M now. @niemeyer Is that enough for now? I can look further tomorrow to see what we can strip out but would like to get a first snapshot to start testing my spread setup on Linode. Is that ok for you?

Yes, thatā€™s okay. Let me try to snapshot that right now.

Thatā€™s done! Thereā€™s now a debian-unstable-64 image available.

1 Like

Thanks!

Fixed that in the PR now.

@niemeyer If you want to do the same for a fedora 25 based image, can I use just image: fedora-25 and spread will automatically figure out to take the existing image provided by Linode as a base?

Just as heads-up for everyone: We now have CI tests running on debian with https://github.com/snapcore/snapd/pull/3156 being merged. Next is Fedora on the list :slight_smile:

1 Like

@morphis You can use that as a name, but then use the ā€œimage:ā€ field with the distribution name as made available by Linode. In this specific case, ā€œFedora 25ā€ might do.

Iā€™ve prepared a fedora-25-64 image now on Spread-26. @niemeyer can you snapshot that? spread-image PR is https://github.com/snapcore/spread-images/pull/2

Image is a bit large at 1800MBā€¦ can we get it down a bit somehow? Note that the restore logic in spread.yaml drops data such as packaging information and packages themselves which were downloadedā€¦ that needs tweaking for fedora and other rpm based distros. This may be part of the bloat there.

We do that already. The image seems bloated by itself as I didnā€™t do much
on the image. Only a ā€˜dnf upgrade && dnf cleal-allā€™ and it was on that size
already before. Will see tomorrow what of the installed software (most
space is consumed by /usr) can be dropped.

Iā€™ve tried to remove as many as unneeded packages as possible but rootfs size is still at 1.1G. Is that ok for you?

1 Like

Thereā€™s a couple of things you will probably want to purge:

  • /var/cache/dnf contains copies of the metadata and packages installed
    • Some of this can be purged by dnf clean all
  • Check to see what ā€œgroupsā€ are installed by doing dnf group list and remove groups that may be unnecessary using dnf group remove "<GROUP NAME>".

Yeah, we do that already. Please have a look at https://github.com/snapcore/spread-images/pull/2

Let me paste what is installed but I didnā€™t found anything I could easily remove without breaking the system: https://paste.ubuntu.com/24473107/