Techniques for iterating over a long build?

snapcraft
1

I work with Chromium. In x86, it has around 58 k build steps (and counting) and takes 8 h to build.

If simply building it directly from source, those 8 h are roughly a one-time factor. If a build has issues, I can easily iterate by modifying a few files and building again. That takes a couple of minutes at worst for each iteration, as most of the source code remains intact.

With the right commands, that procedure also works for Deb packaging.

For Snap, however, I don’t know a counterpart. If I make some change, for small as it might be, the build step starts over from scratch, so instead of a couple of build steps it has to run all the 58 k steps again.

As that is terribly inefficient, is there some trick to get Snapcraft to work with incremental builds?

3 Likes
2

Starting from scratch on launchpad (what you linked) or from your workstation? With local sources these should be synced and then it picks up where it left off. Can we see the snapcraft.yaml?

3

From a workstation (specifically Canonistack). I can follow up later with more details of what happens by doing snapcraft --debug --use-lxd, have the build fail at some step and then making a change and trying to continue it.

The snapcraft.yaml.

4

I think the root of your problem comes from using the nil plugin.

5

@nteodosio can you add more details on what are your changes after you “have the build fail at some step” ? Are you changing the line

    source: https://commondatastorage.googleapis.com/chromium-browser-official/chromium-118.0.5993.3.tar.xz

to some different url?

Are you changing the override-build implementation?

What happens (and can you post logs) if you run snapcraft --verbose twice in a row without any change to anything?

6

Thanks for the responses.

@mirkoferrati, I’m not changing that line, as that would require pulling in another version of Chromium with tons of changes and thus warrant a scratch new build. I’m also not changing override-build, normally just random .cc files in upstream’s tree.

I’ll make a build from scratch this week to address your remaining questions and get back to you.

7

Interesting, so it seems you are touching files directly inside the folder where the entire chromium…tar.gz was cloned locally and after that you re-run snapcraft with the intent of just recompiling the changed files and their dependants.

Not exactly a common flow but optimizations should be doable.

Before getting the logs I can only suggest to find places in your snapcraft.yaml where the compiled artifacts may be invalidated, for example some guesses could be line 211 (which applies patches every single time, right?) and line 262 (which seems to generate targets every single time, right?).

You could move them in the override_pull as an experiment to get more info to find what happens.

8

I apologize for the delay. I had problems with Snapcraft last week. I decided to reset the machine and submit a new build. Today I check in and find that the disk ran out of space near the end of the build (56469/58025). The maximum I can get from Canonistack is 50 GB and it has always sufficed, but not this time.

The issue seems legit though, as practically it all is in /var/snap/lxd/common/lxd/storage-pools/default/containers:

   45.1 GiB [############] /snapcraft_snapcraft-chromium-on-amd64-for-amd64-254965
    1.0 GiB [            ] /snapcraft_base-instance-snap...-base-v10--2d55678e0339df02cd48
9

I put an ad-hoc rm statement in snapcraft.yaml to solve the out of space issue (namely rm -rf /root/parts/chromium/src/third_party/rust-src/.git), but it wasn’t enough, I got again out of space in the last, linking step.

So I had the idea to remove a bit of the source (I can do this directly because I always run with --debug):

# rm -rf parts/chromium/src/third_party/rust_src

Then I quit the environment’s shell and ran snapcraft --verbosity=trace --debug --use-lxd again, but alas, it decided to start over the build from scratch:

2023-09-22 06:37:41.222 execute action va-drivers:Action(part_name='va-drivers', step=Step.BUILD, action_type=ActionType.SKIP, reason='already ran', project_vars={'version': ProjectVar(value='117.0.5938.88', updated=True), 'grade': ProjectVar(value='', updated=False)}, properties=ActionProperties(changed_files=None, changed_dirs=None))   2023-09-22 06:37:41.222 Skip execution of Action(part_name='va-drivers', step=Step.BUILD, action_type=ActionType.SKIP, reason='already ran', project_vars={'version': Proj
ectVar(value='117.0.5938.88', updated=True), 'grade': ProjectVar(value='', updated=False)}, properties=ActionProperties(changed_files=None, changed_dirs=None)) (because a
lready ran)
2023-09-22 06:37:41.223 Executed: skip build va-drivers (already ran)
2023-09-22 06:37:41.223 Executing parts lifecycle: build chromium                                                                                                         2023-09-22 06:37:41.223 Executing action
2023-09-22 06:37:41.224 execute action chromium:Action(part_name='chromium', step=Step.BUILD, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None))
2023-09-22 06:37:41.225 load state file: /root/parts/chromium/state/pull                                                                                                  2023-09-22 06:37:41.715 fix artifacts: unpack_dir='/root/parts/chromium/install'
2023-09-22 06:37:41.721 fix symlink: path='/root/parts/chromium/install/usr/share/X11/rgb.txt', unpack_dir='/root/parts/chromium/install', root='/root/parts/chromium/inst
all/usr/share/X11'
2023-09-22 06:37:41.743 fix symlink: target='/root/parts/chromium/install/etc/X11/rgb.txt'
2023-09-22 06:37:41.856 remove directory /root/parts/chromium/build - (42.2s)

If anything, I’d have expected it to restart the pull phase. The

reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None)

doesn’t help me.

Did I do something wrong there?

10

I have this same issue all the time.

Can’t find an efficient workflow.

Been asking the same question myself a few times.

11

Finally I can answer your question.

For a successful first build with snapcraft --verbosity=trace --debug --use-lxd, if I run it again, it doesn’t repeat build steps, and it simply re-creates the snap.

More surprisingly, if I then from inside the build environment change a source file, namely

parts/chromium/src/chrome/browser/web_applications/os_integration/web_app_shortcut_linux.cc

and try to rebuild, it doesn’t detect the change, no compilation is done and a same snap is generated just like last time (log).