Proposal: expanding scriptlets

snapcraft
1

Background

Although Snapcraft supports many build systems through its multitude of plugins, there are times that one’s build process falls outside of the norm, and none of the plugins quite do exactly what is needed. For example, maybe some of the source code of the part needs to be patched before it’s built. Maybe one must do some setup work before the autotools plugin’s build step is run.

Today, Snapcraft supports these use-cases in a pretty limited manner by way of three scriptlets: prepare, build, and install. These can be specified in the YAML and they surround the plugin’s build step, where, if specified:

  • prepare runs before the plugin’s build step
  • build replaces the plugin’s build step
  • install runs after the plugin’s build step

There are a few reasons why this approach was incomplete. First of all, these are named somewhat confusingly. Second, they don’t cover all use-cases, such as patching, as well as a larger one discovered more recently: needing to build e.g. gsettings schemas for the entire snap after it has been primed.

Proposal

New Scriptlets

The Snapcraft part lifecycle consists of several steps, in order:

  1. pull
  2. build
  3. stage
  4. prime

The proposal is that Snapcraft adds scriptlets that can replace each of these steps, named override-<step>. prepare, build, and install will be deprecated in favor of override-build. In order to get similar functionality, see the “Examples” section at the end.

This covers nearly all use-cases, except for one important one: building gsettings schemas. This type of thing needs to happen over the entire priming area after each part in the snap has been primed. Importantly, this needs to happen at the tail end of the overarching prime step so it works for snaps being installed with snap try. The proposal is to add a new global scriptlet (i.e. one that does not belong to a part, but is in the root of the snapcraft.yaml) called pre-pack. This scriptlet will be run only after every part in the snapcraft.yaml has been primed (when the snap.yaml and associated data is being generated).

Communicating with Snapcraft from within scriptlets

The proposal is to include a new command-line utility solely for use within scriptlets, called snapcraftctl. The purpose of this utility is for scriptlets to request some action or some tidbit of information from Snapcraft. To begin with, four subcommands will be supported:

  • snapcraftctl pull
    Run the pull step of the lifecycle for this part.

  • snapcraftctl build
    Run the build step of the lifecycle for this part.

  • snapcraftctl stage
    Run the stage step of the lifecycle for this part.

  • snapcraftctl prime
    Run the prime step of the lifecycle for this part.

Examples

One could achieve functionality equivalent to today’s prepare scriptlet by adding an override-build scriptlet that looks like:

override-build: |
    echo "I'm preparing to build..."
    echo "I'm doing MORE stuff to prepare..."

    # Now actually build like normal
    snapcraftctl build

Similarly, to get functionality equivalent to today’s install scriptlet:

override-build: |
    # Build like normal
    snapcraftctl build

    echo "I'm doing more stuff after the build completes"

To patch source code after pulling:

override-pull: |
    # Pull like normal
    snapcraftctl pull

    # Patch the code
    patch -p1 < my.patch
3 Likes
2

Please feel free to share your thoughts on this! Is your use-case covered? I particularly want to extend an invitation to @niemeyer, @kenvandine, and @jamesh.

3

Is install going away? that scriptlet is VERY useful, and used in a lot of places, relying on the fact that it runs after the installation of files from the build into the $SNAPCRAFT_PART_INSTALL area. Many snaps use this to modify files or fixup the installation without having to reimplement the entire installation of the build toolchain.

I don’t like this. It feels crazy to require modifications to the installed files to explicitly specify “I don’t want you to build anything. Psyche! I really do want you to build something, GOTCHA!! HAHAHAHA!”

4

install will be deprecated, but really in name only. Here’s some pseudocode of the build lifecycle step for a part regarding how this fits together today using prepare/build/install:

build the part:
  if prepare specified:
    run prepare scriptlet

  if build specified:
    run build scriptlet
  else:
    run plugin's build

  if install specified:
    run install scriptlet

This proposal changes that to:

build the part:
  if override-build specified:
    run override-build
  else:
    run plugin's build

The initial idea was to straight-up rename prepare and install to pre-build and post-build respectively, and do the same for the other lifecycle steps. However, when we looked at that idea alongside the fact that we needed to figure out a way for the YAML to communicate back to Snapcraft (e.g. some sort of set-version script) we realized we needed another tool with that purpose: snapcraftctl. If we already have that tool, we could simply give it a way to drive that part’s lifecycle (e.g. make snapcraftctl build call the plugin’s build), which eliminates the need for pre/post scriptlets: Want to run stuff before build? Do stuff before calling snapcraftctl build. Want to run stuff after build? Do stuff after calling snapcraftctl build.

This proposal is less “I don’t want you to build” and more “give me control of the build so I can make it do what I want.” The prepare/build/install scriptlets don’t really work that way either: how often does one use a build scriptlet with an exit 0 simply for the build step to do nothing? The idea is, in most cases, you’re not saying “hey plugin, don’t run this step” you’re saying “hey plugin, let me help you do the right thing for this step.”

1 Like
5

To expand on the pseudocode above, let’s say my override-build scriptlet looked like this:

override-build: |
  echo "Before build!"
  snapcraftctl build
  echo "After build!"

That would (effectively) make the pseudocode for the build step look like this:

build the part:
  echo "Before build!"
  run plugin's build
  echo "After build!"
6

I guess it’ll grow on me :smiley: I see the benefits you’ve outlined make sense…

wanders off for cake…

7

Mmmm… cake.

I’d like to try and understand your concerns a little more, though, and utilize your fresh pair of eyes. I don’t quite see the reasons behind why you don’t like this, can you try to explain that a little more? Does it just come down to “with install I can simply say ‘hey, do this when you’re done building’, and with override-build I have to actually tell it to build first.”?

8

yeah, I feel it weird to have to tell it I want it to build because I’ve explicitly overridden the build step but I didn’t want to stop it building. If I were actually overriding the build process from the plugin it would make sense that I’d want to kill the default build, but when I only want to run something in addition to the plugin’s build process it feels weird.

In my mind this is reading as:

  1. don’t do the thing, because I’m going to tell you how to do the thing
  2. do the thing I said I was going to tell you how to do. The way you were gonna do it anyway was fine, so do it that way
  3. now you’ve done the thing do my custom stuff
9

Very interesting, yeah we’re thinking about this in different ways. This proposal feels a bit like subclassing to me, where if I write a class with a method that child classes may want to modify, I don’t provide pre/post methods for when I call it, I expect child classes to override my method with their own implementation, even if part of that implementation is calling the original implementation in order to build upon it (e.g. think of this as super().build()).

1 Like
10

I’ve never quite understood why the install stanza seems to extend, rather than replace, the default install step. So this idea sounds good to me.

11

There has been a lot of soul searching with this feature and I think @kyrofa simplified it as much as can be for it to be very systematic and feel less magical.

12

We currently do not have a specific install life cycle step; installation is currently done in the build step. The names for scriptlets in retrospect were poorly chosen.

13

Is there any chance of allowing parts to provide a pre-pack scriptlet too? For desktop applications, there are a number of common tasks we’d want to implement, and ideally I’d like to be able to do this without requiring every application developer to add more boilerplate to their snapcraft.yaml files. It’d be a lot more convenient if we could push out these fixes as part of the cloud parts they’re already using.

14

Thanks for the input, @jamesh. It sounds like the root pre-pack scriptlet won’t really solve your issue very well then, huh? The per-part lifecycle architecture is such that they happen essentially in complete isolation from the other parts (save for the after keyword, which just controls order). The only way for us to support something like pre-pack for parts is to re-architect the lifecycle to add another step between priming and packing, which is a significant enough effort that we can’t fit it in right now.

With that in mind, I think we should probably shelve the root pre-pack property. It was relatively easy to add onto the general expansion of scriptlets, but it doesn’t solve the problem it was intended to solve. Instead, a solution that is imperfect but workable once the rest of this proposal is in place is to do this in the override-prime scriptlet of a part that runs after everything else. This still requires some boilerplate for snaps that consume desktop helpers, but you can still keep all the actual work in the remote parts that way.

Hopefully that unblocks this, and once we finish this cycle we can revisit a more perfect solution.

2 Likes
15

Well, a cloud part can’t know the names of the other parts in a project, so the current after: clause doesn’t really help here.

Would it be that big a problem to just concatenate the pre-pack scriptlets for each part (using the same ordering as for other steps) and run that? Or do you see this as working like the override-* scriptlets, with the ability to replace some undocumented work snapcraft currently does?

I guess the best we could do with the feature as described in the first post would be to have the part stage a script that isn’t included in the final snap, and rely on the developer calling it from their own pre-pack scriptlet. The main problems with this are:

  1. it would require each developer to make the change to their snapcraft.yaml to benefit.
  2. there would be no way for the part to ensure that the script was actually called.

So we’d either need to continue to include first run fallbacks in the desktop-launch script, or break everyone’s snaps.

16

snapcraftctl doesn’t honour customized environemnt variables, any way to solve this?