Extending CI for snapd to debian/fedora

[morphis]

Hey everyone,

as part of my cross-distro work I am also working on getting the CI infrastructure we have for snapd work on other distributions so we can run tests for any PR somebody creates against all supported distributions equally and see when one breaks. There is a lot of work towards that and I decided to start with the lowest hanging fruit which is Debian as it is naturally very similar to Ubuntu.

I have a branch (rough, not finished yet or ready for review) which adjusts multiple things in the existing spread infrastructure to allow us running all tests on a Debian image. As a base I use one of the openstack debian images for the current testing branch (upcoming Debian 9 - stretch) as that is the only one we support with snapd today next to unstable.

A next step is to get a proper base image onto our Linode infrastructure which is based on testing. @niemeyer asked me to give a summary here so we can discuss the next steps for this.

[niemeyer]

It’s great to see this finally moving forward! Thanks for pushing it, Simon.

To cook an image for your target platform, please follow these steps:

1. git clone https://github.com/snapcore/spread-images
2. Edit spread.yaml and add your target system at the top
3. Make sure the project restore section in spread.yaml makes sense to your system
4. Run spread -reuse -shell <system>
5. Once you get the shell, transform the whole image towards what you want
6. Exit the shell
7. The project restore will automatically run and ought to restore your image to a pristine state
8. Fix the update procedure in tasks/update/task.yaml or create a new equivalent task if too different
9. Run the update procedure with spread -reuse <system>:tasks/update
10. Once that works, let me know the machine name and I’ll snapshot the image.
11. Propose a PR to snapcore/spread-images with your changes.

Step 2 starts off from a pre-existing image. For Debian, we can start with debian-sid:

    systems:
        ...
        debian-testing:
            kernel: GRUB 2
            image: debian-sid

Note that the very first image baked will likely be based on a pre-existing Linode image such as “Debian 8”, and those use a custom kernel from Linode, so the “kernel: GRUB 2” line above needs to be temporarily commented out and after the image boots the real distribution kernel must be installed.

After the final image is baked, we drop that image line and always use GRUB 2 so that the kernel inside the image itself is used.

[morphis]

@niemeyer I wanted to perform these steps today but I don’t have a Linode auth key yet. Are you the right person to get me one?

Also Linode doesn’t list any other existing image than debian-7 or debian-8 as available images so where does the debian-sid one you mention come from?

[niemeyer]

Sent a new key your way.

The debian-sid image was created in our own account following a similar process about 7 months ago.

[morphis]

Sent a new key your way.

Thanks!

The debian-sid image was created in our own account following a similar process about 7 months ago.

Ok, let me try that one.

[morphis]

11. Once that works, let me know the machine name and I’ll snapshot the image.

Its

id: 2227753
label: Spread-12

Propose a PR to snapcore/spread-images with your changes.

Add debian-unstable-64 base image

[niemeyer]

That image ended up pretty large at around 1600MB. Do you know why the increase in space needs? For comparison, the previous debian-sid was 1024MB and the ubuntu-16.04-64 one is at 610MB (!).

The PR needs to be updated so it drops the image: line since we’ll be baking that image with its real name, per note above at the end.

[morphis]

That image ended up pretty large at around 1600MB. Do you know why the increase in space needs? For comparison, the previous debian-sid was 1024MB and the ubuntu-16.04-64 one is at 610MB (!).

The overview gives

linode:debian-unstable-64 /# du -hs *
8.1M	bin
57M	boot
0	dev
4.3M	etc
20K	home
0	initrd.img
0	initrd.img.old
404M	lib
4.0K	lib64
16K	lost+found
8.0K	media
4.0K	mnt
4.0K	opt
0	proc
304K	root
2.9M	run
5.8M	sbin
4.0K	srv
0	sys
28K	tmp
634M	usr
30M	var
0	vmlinuz
0	vmlinuz.old

The two biggest items are

• /lib
• /usr

/lib covers kernel modules for multiple kernels and I removed those which are not in use and have corresponding debian packages. Two remain which seem to come from Linode but don’t have corresponding debian packages:

• /lib/modules/4.4.0-x86_64-linode63
• /lib/modules/4.5.5-x86_64-linode69

Both don’t cover much same (4K) so we can ignore those.

The image size should be down to ~940M now. @niemeyer Is that enough for now? I can look further tomorrow to see what we can strip out but would like to get a first snapshot to start testing my spread setup on Linode. Is that ok for you?

[niemeyer]

Yes, that’s okay. Let me try to snapshot that right now.

[niemeyer]

That’s done! There’s now a debian-unstable-64 image available.

[morphis]

That’s done! There’s now a debian-unstable-64 image available.

Thanks!

The PR needs to be updated so it drops the image: line since we’ll be baking that image with its real name, per note above at the end.

Fixed that in the PR now.

[morphis]

@niemeyer If you want to do the same for a fedora 25 based image, can I use just image: fedora-25 and spread will automatically figure out to take the existing image provided by Linode as a base?

[morphis]

Just as heads-up for everyone: We now have CI tests running on debian with https://github.com/snapcore/snapd/pull/3156 being merged. Next is Fedora on the list

[niemeyer]

@morphis You can use that as a name, but then use the “image:” field with the distribution name as made available by Linode. In this specific case, “Fedora 25” might do.

[morphis]

@morphis You can use that as a name, but then use the “image:” field with the distribution name as made available by Linode. In this specific case, “Fedora 25” might do.

I’ve prepared a fedora-25-64 image now on Spread-26. @niemeyer can you snapshot that? spread-image PR is https://github.com/snapcore/spread-images/pull/2

[niemeyer]

Image is a bit large at 1800MB… can we get it down a bit somehow? Note that the restore logic in spread.yaml drops data such as packaging information and packages themselves which were downloaded… that needs tweaking for fedora and other rpm based distros. This may be part of the bloat there.

[morphis]

We do that already. The image seems bloated by itself as I didn’t do much
on the image. Only a ‘dnf upgrade && dnf cleal-all’ and it was on that size
already before. Will see tomorrow what of the installed software (most
space is consumed by /usr) can be dropped.

[morphis]

Image is a bit large at 1800MB… can we get it down a bit somehow? Note that the restore logic in spread.yaml drops data such as packaging information and packages themselves which were downloaded… that needs tweaking for fedora and other rpm based distros. This may be part of the bloat there.

I’ve tried to remove as many as unneeded packages as possible but rootfs size is still at 1.1G. Is that ok for you?

[Conan_Kudo]

There’s a couple of things you will probably want to purge:

• /var/cache/dnf contains copies of the metadata and packages installed
  • Some of this can be purged by dnf clean all
• Check to see what “groups” are installed by doing dnf group list and remove groups that may be unnecessary using dnf group remove "<GROUP NAME>".

[morphis]

/var/cache/dnf contains copies of the metadata and packages installedSome of this can be purged by dnf clean all

Yeah, we do that already. Please have a look at https://github.com/snapcore/spread-images/pull/2

Check to see what “groups” are installed by doing dnf group list and remove groups that may be unnecessary using dnf group remove “<GROUP NAME>”.

Let me paste what is installed but I didn’t found anything I could easily remove without breaking the system: https://paste.ubuntu.com/24473107/