Snapcraft filesets

doc
1

A fileset is used within snapcraft.yaml to represent a group of files, or a single file, when creating a snap.

Removed in core22:

Filesets are not supported when building on core22 or newer bases. See the core20 → core22 migration guide for details.

They’re useful when moving files into the stage and prime areas of the build process (see Snapcraft lifecycle for more details) as they can be referenced individually, rather than always having to construct a list of filenames.

A fileset is implemented as a YAML map between the fileset name (the key) and a list of corresponding filenames for each fileset. This list can be built from any of the following:

  • individual files, eg. [ bin/dnsmasq ]
  • subdirectory paths, eg. [ etc ]
  • wildcard globs, eg. [ usr/* ]
  • globstar globs, eg. [ lib/**/*.so* ]

The * (asterisk) wildcard glob returns all the files in that path. Conversely, adding an initial - (dash) will exclude the files in that path. For example, you could add usr/local/* then remove usr/local/man/* with the following:

filesets:
   allbutman: [ usr/local/*, -usr/local/man/* ]

Filenames are relative to the part install directory, eg. parts/<part-name>/install.

If you have used the organize keyword to rename files from your snapcraft.yaml part, your fileset will be built from filenames after they’re renamed.

Conflicting rules example

Snapcraft will attempt to aggregate conflicting rules from different filesets. For example, take the following directory and file structure:

adir
├── adirthat
├── adirthis
└── bdir
    ├── bdirthat
    └── bdirthis

The following fileset definition will not stage adir/bdir/* despite its specific inclusion under adir/*:

filesets:
  exclude-dir: [ -adir/bdir/* ]
  include-dir: [ adir/* ]
stage:
  - $include-dir
  - $exclude-dir

In the above example, using the excluding syntax adir/bdir instead of adir/bdir/* would exclude both the bdir directory and its contents, rather than excluding only the contents, retaining the empty bdir itself.

This is how the above will be staged within the snap:

adir
├── adirthat
├── adirthis
└── bdir

If, however, we remove the exclude-dir: [ -adir/bdir/* ] fileset definition, all the files and directories beneath adir will be staged in the snap:

adir
├── adirthat
├── adirthis
└── bdir
    ├── bdirthat
    └── bdirthis

Relevant Snapcraft source code

Check out the _organize_filesets function in snapcraft_legacy/internal/pluginhandler/init.py

1 Like
2

Note that Snapcraft doesn’t aggregate conflicting rules from different filesets, for example, the following recipe will still stage usr/share/alsa:

    filesets:
      exclude-alsa-files:
      - -usr/share/alsa/*
      include-usr-files:
      - usr/*
    stage:
    - $exclude-alsa-files
    - $include-usr-files

while this one works:

    filesets:
      include-usr-files:
      - usr/*
      - -usr/share/alsa/*
    stage:
    - $include-usr-files
1 Like
3

Thanks for mentioning this - it’s an important point which I’ve added to the main doc.

1 Like
4

Are you sure about that first example in “Conflicting rules”? I tried this, and it seems like the excluded files really are excluded.

5

You’re right about the first example! I’ve just tried this myself, and the excluded files were excluded for me too.

I’m not sure if this changed, or was wrong initially (I did test it when writing the doc, but that was some time ago). Either way, I’ll update the text. Thanks for finding this and for letting us know. Edit: now updated

6

I might point out that there is a difference with exclusion if you exclude “dir” versus excluding “dir/*”; the former excludes the entire directory, while the latter excludes the contents of the directory, while retaining the empty directory itself, the lesson being (I think) that it would seem to be relatively uncommon to want to explicitly exclude “dir/*” unless you’re doing something unusual where you really want to keep that empty directory.

7

Thanks for this. I’ve added your suggestion as a note rather than set the example to only usedir as this hopefully provides a little extra insight into Linux wildcard globs.

8

One apparent problem with this discussion of filesets is that it seems to assume that the reader is already familiar with the process of inclusion or inclusion using stage: and prime:, but I don’t see where that’s done. There’s that early link to “Snapcraft lifecycle”, but there’s no real explanation there that I can see.

Perhaps the right approach is to expand the scope of this page to cover that, with filesets as the concluding section.

9

Yeah, that’s a good point. It’s probably worth a separate document on where files are included and how they’re staged and primed. The whole process is still very opaque to the average snap builder. I’ll give it some thought and try to write something.

10

I did extensive testing and have confirmed how inclusion and exclusion works, especially if you try to combine them.

11

Upon further testing, I’m having trouble seeing the benefit of filesets as they seem to be more trouble than they’re worth in a number of ways. (I am willing to be corrected on any of this.)

First, since they’re defined on a per-part basis, it’s not like you can define a really useful, sophisticated fileset, then use it across multiple parts. Or can you? If you could, that would be at least one redeeming feature, but that doesn’t seem possible.

Next, this page promotes the power of the fileset’s “globstar” globbing, but it seems like you have that same ability in the regular stage: or prime: sections already, so you certainly don’t need to resort to filesets to use it. (As an aside, if you truly need to get that carried away with globstar-level globbing, it would seem simpler to just use a step override script and deal with it that way.)

Next, it seems odd that a fileset is defined as using either inclusion or exclusion right in the definition of the fileset, which makes it less than reusable. For instance, if a fileset simply defined a collection of files, and its invocation got to select whether it represented inclusion or exclusion, you could do something like this:

  stage:
    - $fileset
  prime:
     - -$fileset

which would represent that you want to selectively stage some stuff (perhaps for the benefit of other parts), but you would then subsequently exclude it from being primed. But that’s not how it works, of course. In fact, if you put the exclusion “-” in front of the fileset reference as above, snapcraft quietly ignores the entire entry and the fileset reference has no effect (which is definitely non-intuitive).

Finally, the only examples I’ve seen so far that attempt to reuse a fileset definition look like this (snapcraft/tests/spread/plugins/v1/plainbox/snaps/checkbox/snap/snapcraft.yaml):

 filesets:
     wrapper: [bin/checkbox-cli-wrapper]
 stage: [$wrapper]
 prime: [$wrapper]

which seems kind of redundant since, if you’ve staged just that one file, would not the default be to prime it? The above seems like a lot of work to say you just want to stage (and prime) that file.

Having said all that, I can envision one reasonable use case, as follows:

filesets:
  stage_only: [ included files ]
  stage_AND_prime: [ different included files ]
stage:
  - $stage_only
  - $stage_AND_prime
prime:
  - $stage_AND_prime

So, sure, this is a useful way to selectively stage some stuff, then further selectively prime a subset of what was staged. But beyond that, this sure seems like an awkward feature.

Thoughts?

Oh, wait, I might have misunderstood a small but critical point, will recheck, but most of the above still holds.

1 Like
12

Reportedly, filesets have been dropped from Snapcraft 7, and while they may be coming back in a different form, they’re not there now. At least as I see it. Perhaps this should be noted.

13

Thanks for bringing this to my attention. I just ran a few of my own tests and while it still works with Snapcraft 7 and core20, it does not work with core22. I’ll add a note to mention this.

14

Filesets were always badly designed – the idea that you needed to define them as inclusive or exclusive really reduced their value.