PoC on Build VM using Fedora 27


#1

Introduction

Heads up, this is very preliminary and meant to be a base our next design conversations on how we want this to look like. It uses a fictional fedora27 base as seen in the snapcraft.yaml:

name: make-hello
version: 0.1
summary: test the make plugin
description: |
  This is a basic make snap. It just prints a hello world.
  If you want to add other functionalities to this snap, please don't.
  Make a new one.
confinement: strict
base: fedora27

build-packages: [gcc]

apps:
  make-hello:
    command: test

parts:
  make-project:
    plugin: make
    source: .

To determine it needs to use a specific build environment to build for that base,

sergiusens@mirkwood:~/source/snapcraft/tests/integration/snaps/make-hello$ snapcraft pull
Creating a build environment named 'multibranchiate-katlyn'
[sudo] password for sergiusens: 
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
Waiting for pending snap auto refreshes.
Connected (version 2.0, client OpenSSH_7.6)
Authentication (publickey) successful!
Refreshing snapcraft in 'multibranchiate-katlyn'
Connected (version 2.0, client OpenSSH_7.6)
Authentication (publickey) successful!
error: cannot refresh "snapcraft": snap "snapcraft" has "auto-refresh" change
       in progress
Connected (version 2.0, client OpenSSH_7.6)
Authentication (publickey) successful!
Connected (version 2.0, client OpenSSH_7.6)
Authentication (publickey) successful!
Last metadata expiration check: 3:52:33 ago on Thu 24 May 2018 09:55:45 PM UTC.
Package make-1:4.2.1-4.fc27.x86_64 is already installed, skipping.
Dependencies resolved.
================================================================================
 Package              Arch         Version                  Repository     Size
================================================================================
Installing:
 gcc                  x86_64       7.3.1-5.fc27             updates        21 M
Installing dependencies:
 binutils             x86_64       2.29-13.fc27             updates       5.9 M
 cpp                  x86_64       7.3.1-5.fc27             updates       9.4 M
 glibc-devel          x86_64       2.26-28.fc27             updates       990 k
 glibc-headers        x86_64       2.26-28.fc27             updates       505 k
 isl                  x86_64       0.16.1-3.fc27            fedora        835 k
 kernel-headers       x86_64       4.16.9-200.fc27          updates       1.2 M
 libmpc               x86_64       1.0.2-8.fc27             fedora         56 k

Transaction Summary
================================================================================
Install  8 Packages

Total download size: 40 M
Installed size: 111 M
Downloading Packages:
(1/8): isl-0.16.1-3.fc27.x86_64.rpm              89 kB/s | 835 kB     00:09    
(2/8): libmpc-1.0.2-8.fc27.x86_64.rpm            82 kB/s |  56 kB     00:00    
(3/8): binutils-2.29-13.fc27.x86_64.rpm         105 kB/s | 5.9 MB     00:58    
(4/8): glibc-devel-2.26-28.fc27.x86_64.rpm      115 kB/s | 990 kB     00:08    
(5/8): glibc-headers-2.26-28.fc27.x86_64.rpm     74 kB/s | 505 kB     00:06    
(6/8): kernel-headers-4.16.9-200.fc27.x86_64.rp  83 kB/s | 1.2 MB     00:15    
(7/8): cpp-7.3.1-5.fc27.x86_64.rpm               89 kB/s | 9.4 MB     01:48    
(8/8): gcc-7.3.1-5.fc27.x86_64.rpm               99 kB/s |  21 MB     03:41    
--------------------------------------------------------------------------------
Total                                           184 kB/s |  40 MB     03:45     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : libmpc-1.0.2-8.fc27.x86_64                             1/8 
  Running scriptlet: libmpc-1.0.2-8.fc27.x86_64                             1/8 
  Installing       : cpp-7.3.1-5.fc27.x86_64                                2/8 
  Running scriptlet: cpp-7.3.1-5.fc27.x86_64                                2/8 
  Installing       : kernel-headers-4.16.9-200.fc27.x86_64                  3/8 
  Running scriptlet: glibc-headers-2.26-28.fc27.x86_64                      4/8 
  Installing       : glibc-headers-2.26-28.fc27.x86_64                      4/8 
  Installing       : glibc-devel-2.26-28.fc27.x86_64                        5/8 
  Running scriptlet: glibc-devel-2.26-28.fc27.x86_64                        5/8 
  Installing       : binutils-2.29-13.fc27.x86_64                           6/8 
  Running scriptlet: binutils-2.29-13.fc27.x86_64                           6/8 
  Installing       : isl-0.16.1-3.fc27.x86_64                               7/8 
  Running scriptlet: isl-0.16.1-3.fc27.x86_64                               7/8 
  Installing       : gcc-7.3.1-5.fc27.x86_64                                8/8 
  Running scriptlet: gcc-7.3.1-5.fc27.x86_64                                8/8 
Running as unit: run-racf224037296449fb6618b0d6321ff0d.service
  Verifying        : gcc-7.3.1-5.fc27.x86_64                                1/8 
  Verifying        : cpp-7.3.1-5.fc27.x86_64                                2/8 
  Verifying        : isl-0.16.1-3.fc27.x86_64                               3/8 
  Verifying        : libmpc-1.0.2-8.fc27.x86_64                             4/8 
  Verifying        : binutils-2.29-13.fc27.x86_64                           5/8 
  Verifying        : glibc-devel-2.26-28.fc27.x86_64                        6/8 
  Verifying        : glibc-headers-2.26-28.fc27.x86_64                      7/8 
  Verifying        : kernel-headers-4.16.9-200.fc27.x86_64                  8/8 

Installed:
  gcc.x86_64 7.3.1-5.fc27                   binutils.x86_64 2.29-13.fc27       
  cpp.x86_64 7.3.1-5.fc27                   glibc-devel.x86_64 2.26-28.fc27    
  glibc-headers.x86_64 2.26-28.fc27         isl.x86_64 0.16.1-3.fc27           
  kernel-headers.x86_64 4.16.9-200.fc27     libmpc.x86_64 1.0.2-8.fc27         

Complete!
Preparing to pull make-project 
Pulling make-project 
Preparing to build make-project 
Building make-project 
make -j1
gcc -o test test.c
make install DESTDIR=/multibranchiate-katlyn/parts/make-project/install
mkdir -p /multibranchiate-katlyn/parts/make-project/install/bin
cp -a test /multibranchiate-katlyn/parts/make-project/install/bin/
Staging make-project 
Priming make-project 
Files from the build host were migrated into the snap to satisfy dependencies that would otherwise not be met. This feature will be removed in a future release. If these libraries are needed in the final snap, ensure that the following are either satisfied by a stage-packages entry or through a part:
lib64/libc.so.6
'grade' property not specified: defaulting to 'stable'
Snapping 'make-hello' \                                          
Snapped make-hello_0.1_amd64.snap

How

Build VM

The build VM is based out of a Fedora 27 cloud image, using cloud-init for some initial customization such as preinstalling snapd and snapcraft and to have a warm dnf cache.
The cloud-config is very bare bones:

#cloud-config
package_update: true
package_upgrade: true
packages:
  - nc
  - kernel-modules
  - cloud-utils-growpart
  - snapd
growpart:
  mode: auto
  devices: ['/']

After this cycle is done, I shutdown and removed the cloud seed that was created.

snapcraft

In snapcraft there’s a minimal cloud-config which sets up a build user and ssh keys for that user.
It launches qemu with the following characteristics,

  • a 9p mount point for the project.
  • a monitor using telnet to talk to the qemu control pane.
  • a port forward to port 22 for ssh.

snapcraft monitors the readiness of the ssh port and uses paramiko to send commands over to refresh, mount the project and later run snapcraft.

Once that is done, the telnet port is used to save the vm state (i.e.; qemu’s savevm) and then shutdown.

There are a couple of shortcuts taken here, one not mentioned is that there is a very minimal Repo handler for dnf which we would move to the build environment setup once that is done, more on that on the Build environment file format topic.

A keen eye can also see that the exact lifecycle command is not being passed on and a full snap is always created (that’s just a bit more of integration work I did not have time to get to yet).


Status Tracking for Build VM
Status Tracking for Build VM