The road to core18

snapd
1

The road to core18

This document outlines what needs to be done to have a working
Ubuntu Core 18 system from a fresh image and from an upgrade.

Current status (2018-08-02)

  • snapd snap in “edge”
  • bootable core18 snap in “edge”
  • ubuntu-image can create core18 images with 4.15 kernels
  • core18 images boot and can install/run all snaps from core16 and core18
  • snapd test suite runs on core18 (almost all tests that make sense)
  • kernel track policy is checked inside snapstate
  • implicit/system interfaces translated from snapd -> core internally and visible as “system” in the cli/API

Fresh image

A fresh image is created by ubuntu-image from a new model
assertion obtained via:

$ snap known --remote model series=16 brand-id=canonical model=core-amd64-18

This model assertion will contain the new entry “base: core18” that
identifies what base snap is used for booting and the new “kernel: pc-kernel=4.15” entry that specifies the kernel and the kernel track.

Partition layout

The partition layout will be:

  • (optional) EFI
  • system-boot
  • writable

The same “writable” setup [XXX: should this be document just
like the boot modes] will be used to make parts of /etc and
/var writable as for Ubuntu Core (16). Core18 will stop using
“synced” directories though and the amount of writeable dirs
will shrink as much as possible.

Snaps

The Ubuntu Core 18 amd64 system will consist of the following
snaps:

  • core18 (base snap, no snapd)
  • {pi2,pc,…}-kernel (kernel snap)
  • {pi2,pc,…} (gadget snap)
  • snapd (app snap with snapd)

The system will use the same boot robustness protocol as
before (described in Ubuntu Core boot modes)
with the “snap_core” variable set to “core18_XYZ.snap”.

The difference between this and the previous Ubuntu Core system
is that core18 will not contain snapd. This is nice because it
de-couples snapd from core so we can rebuild them independently
and also maintain them by different teams. It also provides some
challenges because now we need to solve the firstboot bootstrap
problem on how to install the snapd snap without having a snapd
available (see #### Bootstrap section below).It also means we
need to consider how to expose snapd (see #### The snapd snap).

The kernel snap has the same name as in core16 and we will
move it into a track (4.15).

Base snaps

The model assertion supports a new “base: some-base” field. If ubuntu-image seeds this field it will automatically put snapd in such images. A base snap has some minimal requirements:

  • some mount points like /usr/lib/snapd must exist
  • /snap/bin must be in the default PATH (e.g. via /etc/environment)
  • Right now a bootstrap script is required to start snapd once - this will probably change and gets integrated into ubuntu-image (i.e. ubuntu-image will pull a generic bootstrap script out of the snapd snap and into the image).

The snapd snap

The snapd snap is a special snap. It is available in the “edge” channel right now and build daily from git master.

It is currently only installable on systems that use “base: something” in their model assertion (which means only on core18 at this point). Eventually we will make it available to core16 and classic systems and it will be used for re-exec there.

We will most likely invent a new snap type for it “type: snapd” (but this is not done yet).

It has no base because it needs to work with multiple bases (i.e. it needs to work on core16,core18,etc). This means we need to make sure its as self-contained as possible.

The core18 snap

The core18 snap is build together with the ubuntu foundations team at https://github.com/snapcore/core18.

The bootable version of core18 is available in the “edge” channel and its currently around 50Mb (down from 90Mb on core16).

Right now core18 will run /snap/snapd/current/usr/lib/snapd/snapd.run-from-snap to generate all the right snapd units. However this will change so that snapd writes the right unit files directly (and /snap/bin/snap).

Today core18 contain a symlink from /usr/bin/snap to /snap/snapd/current/usr/bin/snap currently.

The snapd snap failover plan is:

  • Always write the sequence information for snaps into /var/lib/snapd/sequence/name.json so that simpler can inspect the snap install sequence without needing to read the full state
  • Use the “OnFailure=” feature of systemd that triggers a snapd.failure unit
  • The snapd.failure unit will read the sequence file and start the previous snapd
  • The previous snapd will detect if the error happend during a snapd refresh and in this case revert the refersh

This ensures that we can always go back to a snapd version that was able to do refrehes.

Bootstrap

A fresh Ubuntu Core 18 system will boot without an installed snapd.

The core18 (and other base snaps) contains a special script that will do the bootstrap if no /snap/snapd/* snap is mounted. In this case the script will search /var/lib/snapd/seed/ for a snapd snap, mount that, run snapd and it is expected that snapd will seed itself and restart into the fully installed snapd snap once the seeding is far enough to have the snapd snap.

Interfaces

All implicit interfaces (the ones that are currently put on “core”) will be put onto “snapd” on core18 systems.

The recommendations is that people should use the new “system” alias to refer to implicit interfaces.

Work items (fresh image):

  • robustness for bad snapd updates (similar logic as for the bootloader base/core/kernel rollbacks) (https://github.com/snapcore/snapd/pull/5592)

  • make core-{amd64,i386,pi2,pi3,dragonboard}-18 model assertions with base: core18 and kernel: xxx-kernel=4.15

  • hide bases by default from snap list (but show with snap list --all)

  • cleanup unused bases (if nothing needs them anymore)

  • cleanup snapd snap

  • solve system config being stored under “core” (new systems and later upgrade)

  • look into shrinking the core18 image

Upgrade from core to core18

The goal is to have a painless upgrade from existing Ubuntu Core 16 machines to Ubuntu Core 18. The upgrade will not happen automatically, it needs to be triggered via a manual process.

Because the partition layout and the writable-path handling does not change the upgrade is possible. We need to ensure any uid/gid mapping in core16 is also compatible with core18.

Work items (preliminary)

  • specify what process/command the user needs to run to upgrade
  • the existing model assertion will get a new field: supported-transition: which will allow to specify what new models this machine may become
  • snapd will need code to deal with the model change and trigger the right refreshes
  • we need to support reverts and deal with fail-overs (like the new kernel not booting etc).
3 Likes
2

$ snap known --remote model series=18 brand-id=canonical model=pc-amd64
Note the series=18 here.

we really don’t want to introduce series 18, a series is intended as a full new namespace for snaps and to allow fully incompatible assertions of some types.

Like we have core18 we will need models with 18 in their names. Using series 18 just for the models is going to be very confusing (16 is still hardcoded in various places).

3

Changing the series involves snapcraft and Snap Store work too.

4

yes, that’s why we have long agreed not to do that for 18, but changing it only in the model assertions is confusing as well.

5

Could we just change that parameter to base=core18, and I assume we could handle base being an optional and backward compatible field added to the assertion?

6

Thanks, I edited the above and removed the suggestion to use series=18 for this.

7

notice that there might be a way to not need the add base: core18 to the preexisting model assertions by revving them, it might also be a way to express the upgrade. ATM we don’t update model assertion of preexisting devices but we do need to allow for it in some controlled way.

8

We just discussed the model assertion issue (briefly) during the standup and the agreement was to have a new model assertion with a different name for “Ubuntu Core 18” for now. This model assertions will have different names from the existing:

  • pc-amd64
  • pc-i386
  • pi2
  • pi3
  • cm3
  • dragonboard

There are some options, the easiest to just append a -18 to each. However @niemeyer suggested to take the opportunity to cleanup the names and e.g. use something nicer for the pc-* names (as not all Ubuntu Core devices that run i386 code are PCs).

9

perhaps also reflect the architecture in the the names of the arm ones … (pi3-armhf-18, dragonboard-arm64-18 etc etc …)

10

In an ideal world, I think that ‘snapd’ as a ‘regular snap’ should be fine, but I’m concerned with how this will intersect with our sandboxing because snapd and snap cannot fail since if they do, we won’t be able to apply bug fixes.

Right off the bat, the ‘snap’ command could cause trouble since seccomp can only get more strict. Furthermore, if other interfaces are plugged then the device cgroup may come into effect. AppArmor should be ok with appropriate policy, but we’ve already seen issues where simply the act of not running under the ‘unconfined’ label causes changes in behavior (fd delegation, the recent unlink/linkat bug with Qt 5.10). We could specify ‘confinement: classic’ but that isn’t available on Ubuntu Core and even if we adjusted for that, it won’t solve the potential ‘not unconfined’ issue, especially when considering partial and full confinements and the myriad of kernels out there.

As such, at least for core18, I suggest the magic approach (or an equivalent where snapd and snap are run outside of any sandboxing/security labeling) for now. When I heard the idea of splitting out ‘snapd’ from ‘core’, I always figured we’d introduce a new snap type, ‘snapd’ and then we can do whatever is needed based on that.

1 Like
11

Thanks Jamie! I updated the section about snapd as a snap based on your feedback. We also discussed this today after the standup and reached the conclusion that the snapd snap will need some (simple) helper that gives us the systemd units it needs and then we will add a core18 snapd-startup.service unit that will add these units and try to start them. if starting of snapd fails it this snapd-startup.service unit will retry with the previous version(s) of snapd and fallback to the seeded snapd snap if needed. The key here is that we want a similar level of robustness as we have for the snap_mode=try on boot.

12

Strawman for the names:

  • core-amd64-18
  • core-i386-18
  • core-pi2-18
  • core-pi3-18
  • core-cm3-18
  • core-dragonboard-18

This leaves a clear path towards equivalent classic images on all these cases.

1 Like
13

Does systemd remain in the core(base) snap after the split?

14

Yes, although it’s not a general requirement for base snaps, some bases and specifically the ones used for Ubuntu Core will have systemd.

15

Yesterday and today I was working with mvo on an idea how we could handle implicit interfaces in core18 world. As you may know, today snapd adds all the interfaces that declare themselves as implicit to the core snap (or the ubuntu-core) if present. In fact, today the interfaces are added to any snap with the type “os”.

This poses issues in core18 world where there can be a core boot snap, a core18 base snap along with a fedora29 base snap. If an IRC client snap has a network plug, what would it be connected to?

The proposed idea is to create a new virtual snap that would hold all implicit slots. Since we already use the name system for virtual configuration of the core snap I decided to run with that name.

Some hacking and experiments later I have a branch at https://github.com/zyga/snapd/tree/rfc/virtual-system-snap which essentially works okay. Implicit interfaces show up on the system snap. Connections, auto-connections and disconnection work just fine. I even used a trick where the connection state is mapped back and forth so that the on-disk state only talks about the core snap, while in-memory state shows system. This is making the patch harmless to rollbacks.

What is missing? Unit tests on the interface manager need tweaks because they assume interfaces show up on core and that implicit interfaces are added to any type: os snap.

We also need new tests and other obviously missing aspects. This branch is an RFC for discussion.

1 Like
16

As a quick end-of-week update: creating a core18 image that boots/seeds and runs snapd via the snapd snap is almost working. The following PRs need to get merged:

When those are in ubuntu-image should be able to create an image that is able to boot and seed the snapd/core18/pc/pc-kernel snaps and that can install/run basic snaps. It will be an important step because it means we can start creating spread based integration tests.

Note that there is still a lot of work ahead, interfaces are not working yet (see @zyga-snapd post above about the plan here) and configuration is also not there yet. The general robustness is also lacking. With these caveats it will still be a nice milestone when we can run our first spread test :slight_smile:

2 Likes
17

Do you have an ETA for core18?

18

I think the rough ETA is 18.10 but it will be likely available sooner than that.

3 Likes
19

I updated the post with the latest status and agreements. Please let me know if anything is unclear here.

1 Like
20

Any idea when this new track will become available?