Snapcraft Summit, Montreal 2019 - Day 1, 2 & 3

Wimpress · June 13, 2019, 10:58pm

Day 1

Here is a brief summary of the progress

Travis CI - Worked with advocacy to understand limitations and required workarounds for building snaps for Snapcrafters. Adding new snap features to .travis.yml to improve workflow. Not ready for demonstrating just yet. Snapcraft output log folding added to improve visual quality.
Julia - snap created for the latest release, published in the store, verified developers, integrated with build.snapcraft.io. Added two desktop files to the snap to make it easier to launch Julia and access the documentation.
Rust - prototype rustup init snap built and published to the store. Some UX issues identified in the snap creation process. Reviewed the rust Snapcraft plugin.
Godot - Rebuilt new Godot snap for publishing in the store. Investigated options for building snaps of exported games directly from Godot
AWS CLI - Working snap built from scratch
Electron Builder - working through bugs in the snap build process, including the current issue with Electron 5 bundling chromium sandbox binary. Adding support for bases.
Manjaro - Added support for AppArmor to the Manjaro kernel 5.1 series; full confinement is working. Two snapd pkgbuilds, one based on latest stable and another on git master.
Azure Storage Explorer - Working snap. Electron plus the .NET Core 2.0.9 runtime and GNOME platform. Store accounts and snap registered. Snap not published yet.
Ant - Experimental snap published in the edge channel.
R - Initial snap created, overcame issues with making the binaries relocatable.
Groovy - Snap is built and published in the Snap Store.
Jupyter - snap is complete and published as unlisted using a Conda PoC Snapcraft plugin. Working on improving their store page.
Octave - Making good progress, some dependencies missing from the snap which they are working through.
CMake - CMake snap that covers core functionality is built. Work has started on adding the CMake GUI.
Crystal - Snap created with everything bundled. Currently working on making it possible to use toolchains outside the snap.
Conda Forge - Creating snaps based on Conda recipes has turned out to be trivial. Conda Forge are discussing how to output snaps from Conda environment.
Anaconda - Working on creating a Conda snap by deriving from the Snapcraft python plugin to use Conda packages.
pipenv - Created a classic snap but encountered some issues due to classic confinement which are being worked through.

Wimpress · June 12, 2019, 11:24pm

Day 2

Manjaro - Pre-seeded snapd in the ISO for all 3 official Manjaro flavours. Seeding the snap-store snap on the XFCE flavour. Planned for adding their default theme to the gtk-common-themes snap.
Nix OS - Base snap for nix created which is an empty directory tree; awaiting approval. Tooling added to Nix that can transform nix package into snaps. Created test snaps for GNU hello, mariadb and patchelf.
AWS-CLI - The snap is buildable in AWS CodeBuild. We’re stuck currently with getting an appropriate testing setup in AWS infra. This is a blocker for now; need to go back to the team to discuss various options. Unfortunately this also blocks the name transfer in the store for now.
Rust - Working on building multiple rustup version in Travis CI for use by Travis CI. Some cross-build related issues.
Azure Storage Explorer - Verified publisher. https://snapcraft.io/storage-explorer (edge) Refactored for core16 and .NET Runtime 2.2. All known issues fixed. Snap build being integrated into Azure Pipelines. Some work to do on automated builds, dockerfile and snapcraft.yaml submitted for review by colleagues.
Crystal - snap published. Store metadata updated - Cannot get the banner right, publishing to edge directly. Working on integrating with Travis CI to use nightly builds in edge on Travis CI. Verified publisher. https://snapcraft.io/crystal (stable)
PyPA - pipenv is a working snap; can’t published due to name conflict. Had a colleague make a banner for their Snap Store page. Working on packaging pip and VirtualEnv. Adding pipenv support to the Snapcraft Python plugin.
Jupyter - Added store metadata, pushed to stable. Made a banner for the their Snap Store page. Verified publisher. https://snapcraft.io/jupyter (stable)
Anaconda - Started working on Anaconda Navigator, stuck getting Qt working in the snap.
Conda - trying to work out permission issues - potentially need to use the personal-files interface. Verified publisher. https://snapcraft.io/conda (beta)
Electron builder - working on fixing the sandbox in Electron 5.x, also taking the opportunity to make snaps considerably smaller by leveraging the GNOME platform snap.
Godot - Worked on creating a stand-alone squashfs builder to integrate directly into the static Godot Engine binary. Created a custom base for godot game snaps - to enable games to live beyond “core16/18” life time. Prototyped an empty base snap and a godot platform snap to hold up to date drivers. (glxgears)
Travis CI: Stephane Graber has worked on a Travis worker which uses LXD to start-up and running images inside LXD. Performing on par with running a full VM on GCE. Doing much more complex workflow, so now this can be used on Power, AMD and can replace the on-prem worker too. Hopefully will allow people to have custom LXD images, which we can clone and people can use that. Will be easier to provide updates. Also worked with many of the language project present to automate snap builds in Travis CI.
Groovy - Getting documentation sorted, done some work making sure Apache groovy / ant / netbeans have similar approaches so if other Apache Foundation projects come along, they can re-use their work. Verified publisher. https://snapcraft.io/groovy (edge)
Ant - Polished the snap, reviewed the ant plugin for snapcraft and provided a PR. Verified publisher. https://snapcraft.io/ant (edge)
CMake - Running Ubuntu in a VM was problematic, switched to using snapcraft and multipass on macOS. Also encountered issues. Now building the snap in Travis CI. Optimising the snap creation to extract version number to build multiple releases of cmake for use by Travis CI. Have a good snap build including CMake-GUI, desktop-menu entries and icons in-place. Aliases have been granted.
Julia - Looking into how other languages do their versioning. Using node as a template for building multiple supported releases of Julia. Looking forward to supporting parallel instances of Julia in classic snaps. Interested in ways to automate creation of tracks in the store to help automate the release process. Added videos to their snap page.
Octave - snap runs and mostly works, not published yet, looking at enabling users to build their own extensions hitting problems with fs overlay, may publish a first snap tomorrow and work separately on packages.
R - Snap built and works; installation (and compilation) of packages doesn’t just work as compiler uses hardcoded paths; Dirk was only with us for 2 days and will continue remotely.

Robotics Updates

Ruffin: Working to get to the point where we can distribute apparmor profile abstractions via Secure Robot Operating System 2 (SROS2) packaging and install into apparmor.d, so other ROS2 packages can depend upon them and include them to compose their own profiles.
Kyle and Emerson: Currently SROS2 depends upon and shells out to openssl, which doesn’t work in all situations (and is kinda yucky). We’re working to rip that out and replace is with native Python (the cryptography package). However, before we do that we’ve been working to increase test coverage around the components we need to modify (currently we’ve gone from 13% to around 80%).
Jeremy: Working on SROS2 tooling to automatically determine DDS permissions interactively (think aa-genprof for DDS permissions).
Jacob: Adding code necessary to ROS2 in order to distinguish between service servers and clients.
Ryan: Working with Jeremy on the genprof tool.
Michael: Working on distributing SROS2 policies to enable easier policy composition, and have progressed far enough in a Gazebo snap that the errors I encounter are mine and not snapcraft’s.
Daniel: Learning more of the ins and outs of ROS2 and SROS2, and assisting others.

IoT / Devices updates

Rock64Pro not using main-line kernels (yet) but a BSP kernel which runs; managed to compile rock64 core and will be testing this image shortly. Has agreed to write a guest blog post detailing how to bring up Ubuntu Core on SBCs.
Successor board to the PocketChip being enabled
Working on a snap for an IFM 3D camera to use with ROS2; not yet working, created a custom plugin to handle more complex scenarios for building ROS2 real-world applications, working on stage packages/snaps - the idea to build baseline a foundational ros2 snap; aim for tomorrow lunchtime to have the snap ready
Greengrass: implemented a solution for supporting additional language runtimes and installing arbitrary packages/files into the root filesystem of the lambda containers; also fixed a few small bugs with the snap and worked towards getting the snap built on their CI for automated releases
Robotiq - ursim snap successfully opening the simulation. Continuing to work through modifications needed to make hard coded dependency paths to be correctly referenced during the snap built.

Wimpress · June 13, 2019, 11:34pm

Day 3

@Wimpress, @ogra and @kyrofa took part in the 96Boards Office Hours

@lucyllewy : Refactored the Mycroft snap which is now working. Assisted just about everyone at some point during the day.
Manjaro: snapd is installed by default in their beta images. Snap Store snap is installed by default. Snapd patches to support the Manjaro live environment and /snap directory. Pull request to migrate snaps from /var/lib/snapd/snap to /snap. Add default GTK and icon themes to gtk-common-themes.
Ubuntu MATE: Added default GTK and icon themes to the gtk-common-themes snap.
AWS-CLI: Added a tool to perform integration tests (not using snapd) but should work in the future.
Nix: Updated the nix-base snap to support fonts and therefore graphical applications. Snaps composed from Nix packages for galculator and Firefox. Nix tooling to compose snaps is complete.
Octave: Verified publisher. Have a published their snap. Runs. Solved integration issues with Qt and OpenGL.
Godot: All infrastructure handed over. godot-bare base snap now approved, should be able to get a fully working godot engine snap working soon.
Design: Added links to distro specific pages on each store page.
Julia: Tracks for all supported releases set up and published.
Cmake: Travis CI building and publishing releases to edge;
R: Snap works. Added changes to yaml (to override hardcoded path to allow R package compilations).
Rust: Discussions with Travis CI to improve rust there. Has script which generates yamls to build multiple releases of rustup. Building in Travis CI, all architectures published in edge. Need to make a proposal to the Rust Working Group to make these officially blessed.
Ant: Tracks now available for all supported versions. Going to be calling for a vote to make these official. Worked on backend tasks to enable the snap builds automatically from supported releases. A prototype snapcraft plugin which switches to using the ant snap for building snaps. Will be proposing a common approach for Apache Foundation projects can employ when making/publishing snaps.
Travis CI: Hiro, improving build experience for developers, Josh improvements to caching, worked with Stephane to get Travis CI worker configured. Attempting to get an arm64 machine to test builds via lxd worker. Worker snap is transferred to the Travis CI account.
Jupyter: Working on adding support for multiple language VMs, so as not to ship everything in one snap. Prototyped an R snap based out of conda packages that is capable of building R dependencies.
Conda: Conda added access some dotfiles via personal-files interface. A plugin has been written for snapcraft that leverages conda packages, this plugin should be available on the next Snapcraft release.
Anaconda: Worked on building in their CI system. Worked on a conda plugin system to bundle things in a snap.
Electron builder: Having difficulty running smaller snaps due to library conflicts. Migrated to multipass for building snaps which will be cleaner for users (and for him)
Azure Storage Explorer: Integrated snap builds in their CI.
Pipenv: Waiting on current pipenv snap owner to reply before we hand that over. Pipenv snap works. Pip almost working, but seems possible to bundle pip as a snap separately - unsure if it’s desirable. Python also snapped, which will be good for Travis.
Groovy: Setup 3 supported tracks. Adding banners / videos to the store. Also going to propose to the Apache Foundation that the Groovy snaps be recognised as official.
Crystal: Finished their snap, has videos and a banner to help users figure out if they need to install development tools on the host they run Crystal on. A crystal Snapcraft plugin PR has been filed and should be making it into the next snapcraft release. Crystal are very happy to have nightly releases once again, via the edge channel.

IoT / Devices updates

IFM: Successfully got ros2 RVIZ visualization application running with IFM’s 3D camera system as a strictly confined snap. Succeeded in creating a stage-snap of the ros2 base desktop set that can be reused by ros2 application snaps.
Robotiq: ursim snap successfully snapped up and simulation working.
Rockchip: succeeded in creating UC18 images for the Rock64pro and Rock64 boards.
AWS Greengrass: fix a few bugs in the Greengrass snap, planned strategy for adding more languages. Work towards better snap integration in Amazon’s release infrastructure.

Robotics

Jacob: Finished changes to rmw layer for obtaining service clients, awaiting review. Also making great progress on building debug wheels of python’s cryptography package for Windows for the rest of the group to continue rolling that implementation of sros2 out.
Emerson: Working on unblocking cryptography in CI and writing more tests for sros2
Kyle: Completed generation of S/MIME signatures using the OpenSSL API via python3 cryptography’s hazard layer, allowing us to get rid of shelling out to the OpenSSL CLI.
Michael: Prototype PR open for sros2 which adds a macro that allows policies to import and build upon policies from other packages.
Ruffin: Ported apparmor abstractions from ROS1 to ROS2, made them more modular, and worked with @jdstrand to clean them up and audit them, ensuring they were as minimal as possible. Working to put them into sros_apparmor package to install the abstractions into the apparmor.d directory and users can choose to drop in policy files for their own packages.
Ryan and Jeremie: Working up a PR for first iteration of interactive tool (e.g. aa-genprof) to measure the ROS graph and prompt for additional access.
Dan: Learning ros2cli and `rcl and well as sros2 codebase.

mborzecki · June 14, 2019, 6:10am

I think this may be confusing to the users should they start switching between the distro and the AUR packages like they did in the past. Although the packages are named identically and pacman does not block upgrading from one to another the tooling which hardcodes snap mount location at build time will likely break. @zyga-snapd do you know why the whole switch rather than just shipping the /snap symlink?

zyga-snapd · June 14, 2019, 11:31am

We have eventually settled on the /snap symbolic link. I have implemented a script that does live migration from one to the other and, after review, Manjaro is interested in shipping the real thing instead of the symbolic link. We can surely discus this more next week

bcardiff · June 14, 2019, 5:49pm

Could we get an anchor on that section so we can link it directly?

popey · June 14, 2019, 6:22pm

For bugs on the site, I’d recommend filing them over at https://github.com/canonical-web-and-design/snapcraft.io/issues . We recently added links to each distro at the bottom of the pages.

So for example on the Julia snap page, you’ll find this at the bottom: