Chromium fails to build on !amd64: snapcraft complains that the 'pull' step of 'gtk3-locales' is out of date

I am now building the chromium snap from source for all supported architectures (amd64, i386, armhf, arm64), and I am seeing a consistent failure that affects all architectures but amd64:

Running build phase...
Skipping pull desktop-gtk3 (already ran)
Failed to reuse files from previous run: The 'pull' step of 'gtk3-locales' is out of date:
The source has changed on disk.
To continue, clean that part's 'pull' step by running `snapcraft clean gtk3-locales -s pull`.
Build failed

Recent failed builds with their logs can be found on Launchpad.

The only significant difference between amd64 and other architectures as far as the chromium snap is concerned is that the upstream project provides pre-built binaries for amd64 for clang and other build tools. On amd64 these tools are downloaded, unpacked and used as-is, whereas on all other architectures the tools have to be checked out from source (git and svn) and built before chromium can be built (and therefore those builds take significantly longer). This logic can be seen in the override-build scriptlet.

I don’t see how that would interfere with other parts though, this is supposed to happen in the chromium part’s build and install directories.

It typically takes anywhere between 24 and 48 hours to build a chromium snap, and this failure happens towards the end of the process, which makes it a bit of a pain to investigate.

Am I hitting a bug in snapcraft? Or am I doing something wrong with the way the snap is being built? Help appreciated!

I do not see anything in this particular snapcraft.yaml that would cause this. I do feel your pain on the slow build.

The problem lies in

    plugin: nil
      - apt
      - dpkg
    override-build: |
      set -eux

      # In Ubuntu compiled locales for GTK are installed by langpacks,
      # not by libgtk-3-common. Fetch all the langpacks and keep only the
      # GTK locales.
      apt download "language-pack-gnome-*-base"
      for deb in *.deb; do dpkg-deb -x $deb .; done
      cd usr/share/locale-langpack
      find . -type f -not -name "gtk30*.mo" -exec rm '{}' \;
      cd -
      mkdir -p $SNAPCRAFT_PART_INSTALL/usr/share
      cp -R usr/share/locale-langpack $SNAPCRAFT_PART_INSTALL/usr/share/

And this is indeed a bug in snapcraft, one we will fix when using bases by making a semantic change of not using an implicit source of ..

A workaround which will carry on working given this is the nil plugin is to add

override-pull: echo "no source pull"

Or something like that (empty object should work too) to that part.

I would like to ask as well that if you can you avoid using relative paths that leave the original PWD you are inserted into, that will probably break in the future as fulfillment on building within containers where all part directories are different becomes more prevalent (QUILT_PATCHES=../../../patches quilt push -a)

I’d recommend applying the same fix (override-pull: ...) to the shared-mime-info part, too, because that is possibly also going to encounter this problem of complaining about sources changing.

Indeed, and I now remember I read about it here. Thanks for the tip! That doesn’t explain why the build is failing only on !amd64 though.

Is there a $SNAPCRAFT_* variable that points to the root of the tree, where that patches/ directory lives? Or a mechanism to have it copied over to the part’s source directory before building?

Ack, I will apply the workaround there too, thanks for the suggestion.

In snapcraft, before realizing it was part of our local sources, we created a part, staged it and patched from that path. There currently is no exposed local directory and I am afraid of exposing one given that it can lead to more of these strange dirty behaviors if used incorrectly. We’ll think about it.

This didn’t work, unfortunately. I pushed this change and triggered a rebuild for i386, armhf and arm64, and it failed with the exact same error.

I tried to reproduce locally with a trimmed down example (that replaces the chromium part with something simpler that doesn’t take 24+ hours to build) in an i386 VM, but the snap builds fine there. Maybe there’s something specific to launchpad builders?

I set up a clean xenial i386 cloud instance and invoked snapcraft on the chromium snap branch, and it built successfully, so there must be something specific to the launchpad setup. My next test will be to build using lxd.

1 Like

I built again the same snap on i386 on launchpad, with snapcraft 3.0 (installed as a snap from the candidate channel), and it failed again. Here are the relevant bits of the log:

Running pull phase...
Installing build dependencies: …
Pulling chromium-build
Pulling gtk3-locales
Pulling launcher
'chromium' has dependencies that need to be staged: chromium-build gtk3-locales launcher
Skipping pull chromium-build (already ran)
Skipping pull gtk3-locales (already ran)
Skipping pull launcher (already ran)
Building chromium-build
Building gtk3-locales
Building launcher
Staging chromium-build
Staging gtk3-locales
Staging launcher
Pulling chromium
'shared-mime-info' has dependencies that need to be staged: chromium
Skipping pull chromium (already ran)
Building chromium
Staging chromium
Pulling shared-mime-info
Running build phase...
Skipping pull chromium-build (already ran)
Skipping pull gtk3-locales (already ran)
Updating pull step for launcher (source changed)
'chromium' has dependencies that need to be staged: launcher
Skipping pull launcher (already ran)
Updating build step for launcher ('pull' step changed)
Cleaning later steps and re-staging launcher
E:Failed to fetch  407  Proxy Authentication Required
E:Failed to fetch http://ftpmaster.internal/ubuntu/dists/bionic-security/restricted/binary-i386/Packages  407  Proxy Authentication Required
E:Failed to fetch http://ftpmaster.internal/ubuntu/dists/bionic/main/binary-i386/Packages  407  Proxy Authentication Required
E:Failed to fetch http://ftpmaster.internal/ubuntu/dists/bionic-updates/main/binary-i386/Packages  407  Proxy Authentication Required
E:Some index files failed to download. They have been ignored
or old ones used instead.

Check that the sources on your host are configured correctly.
Build failed

What’s interesting (and didn’t appear in the logs when building with snapcraft 2.x) is this:

Updating pull step for launcher (source changed)

The launcher part is defined like this:

  plugin: nil
  source: .
  after: [desktop-gtk3]
  override-prime: |
    set -eux
    mkdir -p bin
    cp $SNAPCRAFT_PART_SRC/chromium.launcher bin/
    cp $SNAPCRAFT_PART_SRC/chromium.desktop bin/

I moved the contents of the launcher part to a dedicated directory, and used the dump plugin, in the hope that this will stop snapcraft from thinking that the part has changed after building the chromium part.

Do you have multipas source: .?

I believe @sergiusens meant "do you have multiple source: ."

It seems Sergio has been messing too much with multipass that they’ve got the muscle memory ingrained :-p

1 Like

There was only one part with source: ..
I moved it to a subdirectory and it looks like it helped. The i386 build failed, but further down the road, with the following error:

Failed to generate snap metadata: The specified command 'desktop-launch' defined in the app 'chromium' does not exist or is not executable.
Ensure that 'desktop-launch' is relative to the prime directory.

I did that build with snapcraft 3.0 from the candidate channel, I assume this check is new. I’ll fix that and trigger a new build, results in ~30hours…

I finally figured out a workaround that allows me to build i386 snaps for chromium, with snapcraft 2.x.

It appears adding an override-pull scriptlet (as initially suggested by @sergiusens) is not enough for parts using the nil plugin. I added a dummy empty source directory and pointed the part to it, and that did the trick.

This still looks like a snapcraft bug to me (and one that made me bang my head against the walls for a long time), but I can live with the workaround, it’s not too ugly or invasive.