Developer sprint Sep 17th, 2018

Tame our command-zoo (help command organization)

  • Reorganize help so it’s shorter, fitting in a 25 lines terminal
  • Group commands by functionality
  • Automate creation, and make sure new commands have a group
  • snap help --all lists everything with summaries, grouped in the same categories
  • Proposal for the text of snap -h/–help, snap help, and snap on its own:
The snap command lets you install, configure, refresh and remove snaps.
Snaps are packages that work across many different Linux distributions,
enabling secure delivery and operation of the latest apps and utilities.

Usage: snap <command> [<options>...]

         Basics: find, info, install, list, remove
          …more: refresh, revert, switch, disable, enable
        History: changes, tasks, abort, watch
        Daemons: services, start, stop, restart, logs
       Commands: alias, aliases, unalias, prefer
  Configuration: get, set, wait
        Account: login, logout, whoami
    Permissions: interfaces, interface, connect, disconnect
    Development: run, pack, try, ack, known, download

For more information about a command, run 'snap help <command>'.
For a short summary of all commands, run 'snap help --all'.


  • Server side has most of the complexity, and is mostly done
  • Client side is not finished, but is very simple in comparison
  • We already accept both the shorthand notation and the extended notation in snap.yaml
  • Must make sure snapcraft already recognizes the field with both formats (cc @sergiusens)
  • Do we send to the store on refreshes already?
  • Must block updates
  • When the write epoch changes locally, a refresh needs to be scheduled shortly thereafter
  • Snapshots need to block restores to a revision with an incompatible epoch, much like snapd must block the install of a snap that cannot read the data
  • Snapshots already store the epoch
  • Documentation must get written. Here in the forum we have very good coverage of the low-level details, and some coverage of high-level ones.
  • User data is not part of this picture at the moment, because it requires users to actually run applications for the data to be updated, and there’s no registry of all the users that have ever run the application
  • We may add snapctl epoch for querying purposes
  • In the future we may also support dynamically setting the write epoch via snapctl set-epoch
  • As soon as the basic functionality of snapshots are in master, we’ll start working on the client side of epochs
  • Snapshots will be considered actually done when a refresh or a remove does a snapshot automatically
1 Like

Ubuntu Core 18 kernel and gadget schemas

  • We need the new kernel name and schema that allows upgrade to be independent for diverging platforms
  • A snap built can only be uploaded once to the store
  • That means a single kernel that works on multiple devices needs to either be built twice, even if the bits are effectively the same
  • Proposal is to use “kernel” as the snap name, and then individual tracks “18-pi2”, “18-pi3”, “18-cm3”, etc
  • Same for the gadget: a “pi” gadget, with tracks named “18-pi3”, etc.
  • The advantage of this schema is that one build may be published in multiple platforms, but when necessary they may also become independent builds
  • This schema may be used by third-parties as well
  • The model will mention something like “kernel: kernel=18-pi3” to lock down the track name for this particular platform
  • In those cases snapd won’t allow changing the track, as it would invalidate the model
  • That takes away some freedom from the track functionality, but that sounds okay for this specific context, since we don’t expect more variance than what’s already allowed by the risk levels (edge, beta, etc) plus branches, which all work within a single track
  • The “snap info” command needs to be modified so that when the model has a locked track, only that specific track is shown; that prevents people from seeing tracks for completely unrelated devices, and also is prevents encouraging switching tracks and that’ll error out on that device

Ubuntu Core 18 status and next steps

  • Dragonboard still crashing
  • Pi 2 and 3 have network issues with console-conf
  • We need the “classic” snap for core18 still
  • How do we get different environments from same snap
    • classic→Classic environment for current boot base
    • classic.16→Ubuntu 16.04
    • classic.18→Ubuntu 18.04
    • classic.fedora28, etc
  • It would be great to have that working inside classic systems as well, in addition to core devices
  • The “classic-support” interface and the driving shell script inside the snap need to be reviewed and modernized
  • Hide bases by default from snap list
  • Bases need to be removed when they become unnecessary
  • Timeframe is a few weeks ahead

note that these most likely come from the fact that the gadget in stable uses an extremely outdated boot firmware (one old enough to not have full support for all power management, thermal control and other bits that will influence the operation of peripherials).

There have been multiple approaches to make snapd allow gadget payload updates (and there are multiple forum posts about it), the last one is here i think:

this has never been implemented, with the result that the stable pi gadgets all stayed on a pre-release firmware (with the exeception of the cm3 where we had the opportunity to use a newer firmware from the beginnning, since the initial image was created a lot later than the other two), there are no networking (or other hardware) issues with the edge images .

the same goes for graphics support on all pi devices, the gadget firmware in stable pre-dates many fixes and configuration options for the pi that are all fixed in the edge gadget which could never be released due to the above missing feature.

EDIT: whoops, sorry, i guess for core18 michael actually used a newly created gadget with a newer firmware when doing his test images, in that case the above would indeed not apply (still an issue in core16 though)

Will there be any alternative on core devices to find out which revision of the rootfs snap you are running beyond snap list (this can be pretty essential info at times) ? Or is the plan to only hide the additionally installed bases there ?

We should probably print something on snap version for that.

1 Like


  • Feature has a few related but different use cases
  • Fundamentally, allows going from one model assertion to another
  • In some cases this might be a new revision of the assertion, which means it’s automatically updated
  • In other cases, it would be a manual (or driven externally to snapd) action shifting from one model assertion to a different one
  • We need to research how to integrate the remodeling procedure into the serial vault, so that the device can become a new model as soon as it requests a serial for itself.
  • The goal is allowing changing of all details of the model: base filesystem snap, kernel, gadget, store, required snaps, etc.
  • Internally in snapd, we hope to implement all those changes using the same code path. Among other things, that means being able to shift back to a previous model.
  • Change of model needs to be approved in the model itself, with language to be defined.
  • Change should be approved in both directions. That is, the target model should be able to restrict which models it can be transitioned from.
  • Wildcards allows accepting all transitions (or perhaps not specifying anything allows all transitions, TBD).
  • Restrictions might be specified per brand, maybe?
  • There might be use cases to allow the gadget to start the remodeling via snapctl.
  • Ideally serial would be obtained in the beginning of the transition so it can use it to communicate, but there’s a chicken and egg problem, because downloading the new gadget might require a conversation with the right brand store, and the new gadget might be required to acquire the new serial, and the new serial is required to talk to the right brand store. Loop must be broken somewhere.
  • We might start with a smaller use case where only some fields in the model are allowed to transition. Changing only the store, and perhaps installing non-core required snaps (not kernel, gadget, core, base) might be easier.
  • There are legacy cases which use a store but are using the generic model. This is an unsupported use case, and needs to be ported to a normal scenario where a model is involved and detailing the store.

Desktop topics

  • Discord autostart was starting it at all times, because it uses the X-GNOME-Autostart boolean extension to define whether to actually autostart. We’ll fix it so it’s respected, and will also look for equivalent ones in KDE.

  • Themes

    • gtk-common-themes is working well
    • Might add a couple more in the snap
    • Perhaps remove some unnecessary ones too
    • Regular cadence to release that so it doesn’t become stale
    • New extension system will help making everything look nice by default
  • Snapcraft build VMs

    • More consistent and isolated builds for everybody
    • Won’t fiddle with the host system anymore
    • A “destructive” mode may be supported for environments that are very locked down and want to pretend to be inside a VM already
  • Snapcraft remote parts

    • These are gone in the new world
    • Only true once “base:” is used in snapcraft.yaml
    • Otherwise remains compatible with the past
  • Snapcraft extensions

    • Allow manipulation of yaml in an organized way
    • Composable since one feeds into the other
    • Command chains finally remove the wrapper hacks and fits very well with extensions
    • Allow evolving the extension automatically
    • Adding plugs and slots to extensions should be done very carefully. We don’t want snaps depending on things unnecessarily by default.
  • Other snapcraft improvements

    • A build-environment field will allow setting environment variables to the build generally
    • Using snaps as a build dependency is coming as well
  • Portals

    • Security review done for the main portal package
    • Gtk one is still in the works; estimated for a week ago but there yet… but close.
    • Will get it working in Cosmic, and then backport as necessary.
    • It will just work.
    • As new portals are added, these should also just work.
  • Pulseaudio

    • Cosmic is already restricting recording
    • Bionic is next, and then Xenial
  • Desktop snap QA

    • Automatic checkbox testing of a series of snaps
    • Happens automatically whether people are there or not
    • Plans is to also check whether themes are applying correctly, are applications opening, etc
  • Metered connections

    • Plan is to change meaning of current setting in a compatible way
    • Only blocker at the moment is making sure that devices unattended aren’t incorrectly marking whatever device they use to communicate as metered
  • Translations of metadata

    • Something along the lines of meta/i18n/pt_BR.yaml
    • Consider translation of media as well (screenshots, etc)

Second meeting

  • Banner support

    • Somehow support the fact some screenshots are banners for the top
  • Support some sort of client identification when going through snapd

  • Some asynchronous operations should report an error kind (#1763289)

  • Notification of events from snapd

    • Something that allows a persistent UI to update itself when something happens
  • Channel maps in search results

    • Probably won’t happen… it’s way too much data for search results
    • Gnome software is currently looping and looking for that data… that’s a bad idea and shouldn’t be done
    • Instead, it’s much better to only ask for the channel map once you look at an individual snap
  • Access for hidden files

    • Read-only is okay, purpose is migration of data towards the snap
    • It has to be manually reviewed before it can even be installed, since there’s no way currently for the user to tell what specific directories the connection would allow. That’s okay for the use cases at hand.
    • We may conditionally allow writes as well, also subject to review and approval
  • Have a hook to notify the application that a refresh is available

    • Warn that a refresh is available for some applications
    • We need to force at some point though
  • Review of portal startup PR

    • Drop a marker file in an in-memory filesystem so that it only queries once for presence of doc portal
      • portals.present
      • portals.missing
    • Testing: verify what happens in all supported distributions, when the portals package is present (when it exists) and is not present

I’m guessing this is specifically people want EL7 support? Unless there are other RPM distributions that are desired? It’s not terribly difficult to extend the Fedora packaging to support other RPM distributions, as most distributions are more similar to Fedora than not…

Build images for Snapcraft

  • The goal of the snapcraft build process rework over the last several months was to have consistent builds across all platforms, sideways and back and forth (any given Linux distribution L1 version V1 can build a snap that uses a base snap from L2 version V2).
  • Some platforms are unable to run a full VM, but can use containers, so we’ll allow Snapcraft build VMs to also register a respective LXD image
  • Launchpad and will use the LXD image
  • We might use LXD inside a VM to avoid the consistency requirements between VM image and container image, but it will be slower, more complex, have a different kernel
  • Even ignoring the Snapcraft needs, having an Ubuntu image consistent across VMs and LXD is a good idea
  • The Ubuntu Cloud image isn’t an ideal target for this because it changes more than necessary for building purposes
  • Foundation will own the creation and retention of the image and the containers for Ubnutu, as usual for every other Ubuntu image
  • Images will contain build-essential, and lose documentation and man pages etc
  • Snapcraft will ask multipass for a snapcraft:<base> image, and multipass will do a lookup somewhere remote
  • The Launchpad build environment will be upgraded to LXD 3.0
  • Timeline: TBD, it’s a priority

(cc @slangasek @cjwatson @kyrofa @sergiusens)

1 Like

@Conan_Kudo The ideal case would be to have the package in EPEL indeed, but we may have to do something else before that as it’s taking some time for it to get through.

The main reason it’s not in EPEL is that right now, it functionally doesn’t work. I have an idea of what EPEL-specific patching I’ll need to do, though.

Recovery mode for core devices

  • Use cases:
    • Reset to original factory state for whatever reason (lost password, etc)
    • Reset device while retaining essential data (might be delegated to application level)
    • Soft-reset: remove the data, but keep the data? Sounds similar to prior case.
  • Primary goal is having back a working device
  • It should be extremely robust, since this is already the “get out of jail” card
  • Given the nature of snaps, we might be able to do it without duplicating the data already in the image today
  • We might make the seed directory immutable using attributes, but that’s not safe enough
  • How would the reset procedure actually work:
    • Tell grub to boot from the seed kernel and seed core, plus magic command line
    • We might get away without using the initramfs for the procedure
    • Either initramfs or very early systemd unit formats the partition and puts it back in a working state
  • There might be code duplication between ubuntu-image and snapd, but we already need to understand the description in gadget.yaml for performing updates, so probably unavoidable
  • It goes without saying that we are always assuming that the device can at least boot, so has a good partition table, etc. Otherwise there isn’t much we can do from inside out.
  • That said, we may support being able to have independent devices for recovery and for the writable space. In those cases, we might support repartitioning the writable space from the outside.
  • Proposed first phase: get a mechanism in place that once the reset “button” is pressed, it gets back into the exact same state of first boot (so no serial, no extra hooks executed, or anything else). This is a good stepping ground towards anything more advanced.
  • We might do a soft-reset system where we have a hook that allows the snap to decide what to preserve (export and import) on a reset situation, for cases where things are broken but not much. That would allow keeping a hard-to-reach device online, for example.

Snapd staging snap

  • Build a staging snapd deb in a PPA.
  • Have a spread-cron job building snapd snap with staging keys, and push to the staging store.
  • Regularly copy core18, the pc gadget to the staging store (probably on beta or candidate releases).
  • Use cases for store team to test snapd in a LXD.

Snapcraft extensions progress review

  • Only allow “extensions” in the app level or top-level for now, so it may be decided later what it would mean to declare both
  • Error if extensions are defined in conflicting order across different applications ([a, b] and then [b, a])
  • Extensions need to be able to chain/compose, so that in [ext-a, ext-b] the output of ext-a becomes the input to ext-b
  • Output human-built documentation to describe what an extension does, with snapcraft extension foo
  • Improve internal implementation so extensions can also declare plugs and slots
  • Break down the individual scripts that were originally concatenated into a single wrapper, so that they can now become independent commands in the chain
  • Extensions support specific bases, explicitly, so that people can trust their behavior; anyone that reviews the behavior of an extension on a new base can report and the list will be expanded
  • The internal implementation might be improved eventually to support a richer implementation of the snapcraft.yaml content, but that’s not critical since changes in extensions won’t break any public APIs

Snapcraft snap directory

  • Problem is that some projects (e.g. snapd :slightly_smiling_face: ) use the snap/ directory for content that is not related to snapcraft building process (reported as #1772584)
  • There’s currently no way to avoid that clash
  • Proposed solution is to allow .snap/ to be used instead of snap
  • That will only be used if a .snap/snapcraft.yaml file exists ($HOME/.snap exists)
  • New VM-based snapcraft will enforce use of either snap/snapcraft.yaml or .snap/snapcraft.yaml, and error clearly if snapcraft.yaml or .snapcraft.yaml are in the current path
  • needs to be updated to support .snap/snapcraft.yaml as well

CUPS snap is working now (tested printing from LibreOffice snap through CUPS snap):

  • CUPS snap now checks whether port 631 is busy. If it is not busy it starts CUPS on port 631, otherwise on 10631.
  • cups-browsed startup waits for CUPS startup having selected the port, to assure correct connection of the snap’s CUPS and cups-browsed
  • It is not recommended to run both the system’s (Debian-package-based) and the snapped CUPS on production systems, but for development setups this is possible.
  • If a system is only running the snap’s CUPS, this CUPS is available through port 631 and both Debian-package-based and snapped applications can print on it.
  • The snap still needs manual connection to Avahi and USB interfaces (on core also to home dir intrerface).
  • Connected to Avahi interface the snapped CUPS finds and sets up IPP (driverless) printers automatically, using its cups-browsed.
  • Users in “adm” group can do administrative tasks using the command line utilities of the snap (they use the snap’s domain socket by default), admin via localhost:631 (system’s utilities) does not work.
  • The snap uses now the up-to-date upstream versions: CUPS 2.2.8, cups-filters 1.21.2, and to fulfill the dependencies of cups-filters QPDF (8.2.1) is nowalso included.
  • Cleaned up warnings in snap build process
  • Updated

Next steps:

What will next cycle look like?

  • Probably will be completed next month:
    • Ubuntu Core 18!
    • Warnings
    • Snapshots working
    • Epochs
    • Parallel installs
    • Documentation outline polished
    • Documentation published on
    • Dynamic snap-update-ns for user mounts
    • Health checks
  • Probably won’t be completed next month:
    • Hotplug
    • Snapshots at remove and refresh
  • Candidates for next cycle:
    • First boot improvements
    • Prevent refreshes while applications are in use
    • Support for man page
    • Configuration schemas
    • Health checks
    • Support for more RPM distributions
    • Device reset and recovery
    • Bootloader/gadget updates
    • Repairs phase 2
    • Multi-user support
    • CLI support for holding refreshes
    • Interface for accessing hidden files
    • Remodeling
    • USN/CVE metadata association
    • More interfaces, as usual :slight_smile:
    • Christmas gifts
    • Renames of snaps
    • CLI support for holding refreshes
    • Cohort support
    • Improve license support
    • Interfaces and layouts in classic snaps
    • “Semi-classic” interface
    • Resource quotas
    • Snap “parts” for debug symbols and translations
    • Consider integration of libc localization support in base
    • Entitlements

Metered connections

  • Existing configuration enables usage of metered network setting to hold refreshes for up to 3 months
  • Ideally that should become the default
  • For compatibility, the refresh.metered setting will accept more clear string values, in addition to current boolean setting
  • Current boolean settings preserve the current meaning for compatibility
  • Before the default is changed we need to make sure no unattended devices will be bricked due to their only connections being marked as metered
  • Field team confirmed they will check whether there are devices being marked as metered, and then whether those can be reconfigured so they won’t be affected

(cc @awe @lool)

1 Like

@niemeyer, + entitlements?

1 Like

Kernel driver development experience

  • Pain points
    • Third-party maintains a wifi driver, but they cannot test their patches
    • Kernel snaps cannot be locally installed without signatures
    • Pushing to the store is a long roundtrip and impractical
    • Kernel snaps are always constantly being flagged for manual reviews
  • We cannot accept arbitrary kernels to simply replace actual device kernels
  • The proper solution must involve a signed document (assertion) saying that for a specific case, it’s okay to replace that kernel by that other kernel
  • The snap-declaration assertion may have data on it that is important for the kernel setup to happen properly. The system cannot simply reuse that assertion, which was reviewed and signed, in a third-party local snap that was never seen by the system.

Ordering across snaps and system

  • There’s a project trying to support dependencies with dynamic systemd device units
  • Another project would like to depend on daemons across snaps
  • Sensitive cases… we need to consider carefully how to support something equivalent to that without introducing brittle logic that is hard to understand, hard to debug, and easy to break
1 Like