Feature request: ability to build a part in-place to use in the snap

This came up in discussions about getting my Snapcraft plugin for gnome-builder accepted upstream: they would like to be able to run the build for the main part themselves rather than have Snapcraft do it.

At first I was a bit reluctant to entertain this, since the last thing we want is to have some packages that can only really be built within an IDE (as could be the case with the Click SDK), but some of the arguments for it are compelling:

  • The gnome-builder devs want to add the features to programatically modify an {autotools, cmake, distututils, etc} project, so it is important that it can see the project’s main build system so we can take advantage of that.

  • The IDE wants to know what compiler flags a particular file in the project will be compiled with so that it can parse the file with libclang.

  • If you’re actively developing a code base, you want relatively quick incremental compile times. As things stand at the moment, direct changes to the source tree will require cleaning and redoing the “pull” stage for the part and a complete rebuild.

  • The devs were interested in adding the option to perform a second build pipeline to perform a standard build to give assurance that the package will build outside the IDE. They saw this as useful for Flatpak too.

So what sort of system am I envisioning? Projects that could benefit from this would need to satisfy a few constraints:

  1. The root directory of the Snapcraft project must also be a valid project for some other build system.
  2. There must be a part with source: ., which I’ll refer to as the “main part”
  3. The main part’s plugin must match up with a build system gnome-builder knows about.

It shouldn’t be a problem if the Snapcraft project has other parts (either defined directly, or cloud parts).

The actual process of building the project would look something like this:

  1. Ask Snapcraft to perform all steps up to but not including the “build” step for the main part.
  2. Determine the build configuration for the main part. This should probably involve Snapcraft so things don’t get out of sync. This would include:
    • flags passed to configure or the build system’s equivalent.
    • any environment variables that should be set.
  3. Have the IDE configure and run the build, setting the build directory to $root/parts/$partname/build
  4. Have the IDE install the project using a destination directory of $root/parts/$partname/install
  5. Tell Snapcraft to assume that the part’s build step has completed successfully.
  6. Ask Snapcraft to complete the package build.

With a system like this, we can basically progress to (3) and then let the user repeat compile step as many times as necessary before continuing on to create their package.

Now I know that there are some open questions with a system like this (e.g. how should build and install scripts in the part be handled), so I’d appreciate any ideas or suggestions.

So staging all the other parts or just the after ones for the main part?

I would need some more explanation on this, specifically, is it a read or a write operation, or instead an override? Is this only supposed to work with autotools based projects?

We can generate tooling to obtain what the execution environment is supposed to look like for a part and make it consumable by calling snapcraft.

We have inner state tracking which could be set with your plugin to get a PoC going and we can add hidden commands for this to work in a more API (through the cli) friendly way.

I suppose staging only the after ones would be fine: we basically want everything that the build step of the main part might rely on.

It would only need to be a read operation. The basic operation of the plugin would look something like this:

  1. plugin provides a BuildSystemDiscovery implementation that reads snapcraft.yaml to determine whether there is a “main part”, and what plugin it uses. It maps the part plugin name to the corresponding IDE build system plugin name.

  2. user says they want to build in snap mode (probably by picking a “runtime” registered by the plugin). This will add stages to the IDE build pipeline to run before (process parts that should be built first) and after (finish building the snap) the main build stage. It will also update the build system configuration with appropriate flags and environment variables for the part.

  3. user can now run though edit/compile/debug cycle as many times as they want before completing the snap build. These will only trigger incremental recompilation rather than rebuilding the world.

Hidden commands should be fine. Maybe we can revisit that decision in the future if someone would find them useful for writing a plugin for a different IDE.

Hi, any news on this?