Fixing the snapcraft _cache_ (LP: #1582469)

snapcraft
1

Introduction

snapcraft performs caching of it’s steps to not repeat them over and over again, so in the case of running

$ snapcraft pull

The tool would run the pull step for all parts. If this is followed by

$ snapcraft build

Which is a step in the lifecycle that follows the pull step, the tool will not run the pull step again and just execute the build step for every part defined.

Problems and solutions

Local sources

For local sources, changing the source code does not mark the pull step of that part as dirty and rerun through the step. The tool should detect the change and re-run appropriately.

Changing definitions

Some definitions, such as after, stage-packages or build-packages need to mark the pull step as dirty and rerun through the cycle.
Changing a source entry or any of its related attributes (source- ones) also need to mark the pull step as dirty.
Others, like plugin specific ones, need to mark the relevant step they affect as dirty.

If a step is marked dirty, the step needs to automatically do the right thing when re-running the tool, which is to rerun the step.

Rerunning a step

When running

$ snapcraft pull

Followed by a subsequent

$ snapcraft pull

The tool should force the command and run it as the user requested.

To expand on the concept, if a user runs

$ snapcraft prime

This will go through these steps in the lifecycle: pull, build, stage and prime

If a user runs

$ snapcraft prime

This will rerun only the prime step, keeping the cached results for pull, build and stage.

However, if the user runs

$ snapcraft pull

The pull step will be rerun, invalidating what was run for build, stage and prime

2 Likes
2

If we can agree to this, we can push forward. That bug has many things in the comments with proposals and there are many in the forum that are slightly different. I am trying to condense them here to have a more consistent conversation around this specific caching problem.

3

That looks like a very welcome improvement. One detail to clarify:

That’s just a statement of status quo, but I guess the proposal is to change that too, right?

4

yes, we should fix it, that is correct, let me reflect that in the original post.

5

So let’s say I’m working on the pull step for a particular part. I run snapcraft pull foo over and over, and each time I do so, it automatically cleans and re-pulls, fine. However, once I’m happy with how that part has pulled, I want to go ahead and pull the rest, so I run snapcraft pull. What happens? Does it wipe the pulled foo and re-pull it with the rest? Or leave it alone? Just based on the proposal, it sounds like it should wipe and re-pull, but that’s not really what I want to happen there.

The bug mentions only automatically cleaning if a part is specified, which seems a little more intuitive to me. However, I’m not sure even that is completely intuitive to a newcomer who wants to e.g. ensure a given part is pulled, not necessarily force it to be done again (thereby cleaning all later steps as well). The bug also suggests a re<step> command, which I think may be a better direction here than adding subtle changes to how the <step> commands work depending on how they’re called and the state of the parts at that point in time. Then, if foo has already been pulled, the snapcraft CLI could say something like:

$ snapcraft pull foo
Skipping pull foo (already ran). You can run it again with 'snapcraft repull foo'
6

Or explicitly run snapcraft pull foo.

7

I’m afraid I don’t quite understand what part of what I said you’re addressing, there.

8

We (@sergiusens, @niemeyer, and myself) just met to have a quick chat about this, and agreed on the following:

  • snapcraft <step> should work the way it does today, skipping things that have already run
  • snapcraft <step> <part name> should automatically clean <step> for <part name> if it has already run, and run it again
  • If the snapcraft CLI happens to run through an entire command skipping all steps, it should suggest specifying a part name, or cleaning.
3 Likes
9

The PR that accomplishes the items above is here:

https://github.com/snapcore/snapcraft/pull/2152