Snapcraft prime not finishing stuck internally

I’m trying to make a snap for my application that coexists out of 3 flutter app with a dotnet dependendecy, when I try to run snapcraft prime --use-lxd it seems to hang on check if part websocket:STEP.pull is outdated (33560.4s)

To me to it looks like it’s stuck in some kind of infinite loop where it’s recursivly checking the websockets and packages sub parts. I tried adding some dummy parts just to kinda guide the order of execution but didn’t really help much.

Any thoughts?

name: test
summary: ...
description: ...
confinement: strict
base: core22
grade: stable
version: "1"
architectures:
  - build-on: amd64

apps:
  a:
    command: /usr/share/a/app
    plugs:
      - network
      - home

  b:
    command: /usr/share/b/app
    plugs:
      - network
      - home

  c:
    command: /usr/share/c/app
    plugs:
      - network
      - home

parts:

  source:
    plugin: nil
    source: https://github.com/flutter/flutter.git
    source-type: "git"
    source-depth: 1
    source-branch: "master"
    build-packages: [clang, cmake, git, ninja-build, pkg-config, libgtk-3-dev, liblzma-dev, curl, unzip]
    stage-packages: [libgtk-3-0, libblkid1, liblzma5]
    override-pull: |
      craftctl default
      git fetch --unshallow
      PATH="$CRAFT_PART_SRC/bin:$PATH" flutter doctor
      PATH="$CRAFT_PART_SRC/bin:$PATH" flutter channel master
      PATH="$CRAFT_PART_SRC/bin:$PATH" flutter precache
    override-build: |
      set -eux
      mkdir -p $CRAFT_PART_INSTALL/usr/bin
      mkdir -p $CRAFT_PART_INSTALL/usr/libexec
      cp -r $CRAFT_PART_SRC $CRAFT_PART_INSTALL/usr/libexec/flutter
      ln -sf $CRAFT_PART_INSTALL/usr/libexec/flutter/bin/flutter $CRAFT_PART_INSTALL/usr/bin/flutter
      ln -sf $CRAFT_PART_INSTALL/usr/libexec/flutter/bin/dart $CRAFT_PART_INSTALL/usr/bin/dart
      export PATH="$CRAFT_PART_INSTALL/usr/bin:$PATH"
    override-prime: ""

  packages:
    source: packages
    source-type: local
    plugin: nil
    build-packages: [rsync]
    override-pull: |
      echo "$(pwd)"
      rsync -r --exclude snap --exclude .git --exclude .vs --exclude .plugin_symlinks --exclude TestResults --exclude releases --exclude .dart_tool --exclude .vscode --exclude node_modules --exclude build $SNAPCRAFT_PROJECT_DIR/packages/* $SNAPCRAFT_PART_SRC/
    override-build: |
      rsync -r $SNAPCRAFT_PART_SRC/* $SNAPCRAFT_PART_INSTALL
    override-stage: ""
    override-prime: ""

  websocket:
    source: websocket
    source-type: local
    plugin: nil
    build-packages: [rsync]
    override-pull: |
      rsync -r --exclude snap --exclude .git --exclude .vs --exclude .plugin_symlinks --exclude TestResults --exclude releases --exclude .dart_tool --exclude .vscode --exclude node_modules --exclude build $SNAPCRAFT_PROJECT_DIR/websocket/* $SNAPCRAFT_PART_SRC/
    override-build: |
      rsync -r $SNAPCRAFT_PART_SRC/* $SNAPCRAFT_PART_INSTALL
    override-stage: ""
    override-prime: ""

  dummy:
    plugin: nil
    build-packages: [rsync]
    build-snaps: [ yq ]
    stage-packages: [dotnet-sdk-6.0, dotnet-runtime-6.0]
    after: [source, packages, websocket]
    override-pull: ""
    override-build: ""
    override-prime: ""

  a:
    source: a_src
    source-type: local
    plugin: nil
    after: [dummy]
    override-pull: |
      mkdir -p packages
      mkdir -p a_src
      mkdir -p websocket
      rsync -r ~/parts/websocket/install/* $SNAPCRAFT_PART_SRC/websocket
      rsync -r ~/parts/packages/install/* $SNAPCRAFT_PART_SRC/packages
      rsync -r --exclude snap --exclude .git --exclude .vs --exclude .plugin_symlinks --exclude TestResults --exclude releases --exclude .dart_tool --exclude .vscode --exclude node_modules --exclude build $SNAPCRAFT_PROJECT_DIR/a_src/* $SNAPCRAFT_PART_SRC/a_src
      #craftctl default
      cd a_src
      flutter clean
      flutter pub get
    override-build: |
      craftctl default
      cd a_src
      flutter build linux
      rsync --mkpath -r build/linux/x64/release/bundle/* $SNAPCRAFT_PART_INSTALL/usr/share/a
    override-stage: |
      rsync –-ignore-existing -r $SNAPCRAFT_PART_INSTALL/* $SNAPCRAFT_STAGE

  b:
    source: b_src
    source-type: local
    plugin: nil
    after: [dummy]
    override-pull: |
      mkdir -p packages
      mkdir -p b_src
      mkdir -p websocket
      rsync -r ~/parts/websocket/install/* $SNAPCRAFT_PART_SRC/websocket
      rsync -r ~/parts/packages/install/* $SNAPCRAFT_PART_SRC/packages
      rsync -r --exclude snap --exclude .git --exclude .vs --exclude .plugin_symlinks --exclude TestResults --exclude releases --exclude .dart_tool --exclude .vscode --exclude node_modules --exclude build $SNAPCRAFT_PROJECT_DIR/b_src/* $SNAPCRAFT_PART_SRC/b_src
      #craftctl default
      cd b_src
      flutter pub get
    override-build: |
      craftctl default
      cd b_src
      flutter build linux
      rsync --mkpath -r build/linux/x64/release/bundle/* $SNAPCRAFT_PART_INSTALL/usr/share/b
    override-stage: |
      rsync –-ignore-existing -r $SNAPCRAFT_PART_INSTALL/* $SNAPCRAFT_STAGE

  c:
    source: c_src
    source-type: local
    plugin: nil
    after: [dummy]
    override-pull: |
      mkdir -p packages
      mkdir -p c_src
      mkdir -p websocket
      rsync -r ~/parts/websocket/install/* $SNAPCRAFT_PART_SRC/websocket
      rsync -r ~/parts/packages/install/* $SNAPCRAFT_PART_SRC/packages
      rsync -r --exclude snap --exclude .git --exclude .vs --exclude .plugin_symlinks --exclude TestResults --exclude releases --exclude .dart_tool --exclude .vscode --exclude node_modules --exclude build $SNAPCRAFT_PROJECT_DIR/c_src/* $SNAPCRAFT_PART_SRC/c_src
      #craftctl default
      cd c_src
      flutter clean
      flutter pub get
    override-build: |
      craftctl default
      cd c_src
      flutter build linux
      rsync --mkpath -r build/linux/x64/release/bundle/* $SNAPCRAFT_PART_INSTALL/usr/share/c
    override-stage: |
      rsync –-ignore-existing -r $SNAPCRAFT_PART_INSTALL/* $SNAPCRAFT_STAGE

  test:
    after: [a, c, b]
    plugin: nil

Manually copying files that are usually handled by the lifecycle processor is usually a bad idea. Each step has a state, and if you move things across steps behind the processor’s back strange things can happen because the states and actual content will be inconsistent. This is especially true when you try to override stage to essentially stage contents by hand bypassing any internal states.

I strongly recommend to try a more traditional approach to parts building without using rsync or custom staging areas, and use the supported staging area instead. I supposed you’re not using the existing flutter and dotnet plugins because of some plugin limitation, can something be changed in those plugins to make them more suitable for your use case?

Can you point out any specific behavior that can attribute the behavior and provide context? I mean the documentation on snapcraft internals isn’t very great about the internals, they document that each stage just takes input and produces artifacts for the next stage e.g. pull: produces src, build: takes src and produces install. Is there anything in my stages that shouldn’t be there?

Yes, I can’t use use the more traditional plugins because of limitations i.e. they lack: --dart-define, navigating to a subdirectory and more importantly one of my flutter plugins directly interacts with dotnet from cmake to bundle it’s dependencies.

Also my application requires core22 which is incompatible with the flutter plugin (core20). While these might be minor things that can be addressed, another factor that played is efficiency. it just isn’t that great having to have a lot of overhead and duplicative effort from the downloading and install the dart and flutter sdk on every part individually, on a bandwidth & data-limited speed internet connection.

File migration across steps is heavily mediated by the parts processor: file contents may be patched, and the timestamp of migrated files and the state file is important to determine if a step needs to run again. Parts processing happens in two phases: in the “planning phase” the previous execution state is loaded and filesystem contents are examined to create a plan of what needs to be executed, and in the “execution phase” steps are executed and the state is updated. Moving files by hand without updating the corresponding state files will create inconsistencies there, so I wouldn’t use any manual rsyncs there.

You can (and should) use override-build to overcome limitations in plugin or if you build things in a different way, but avoid migrating files by hand. Use the prime filter when you have intermediate content that’s not intended to appear in the final payload:

intermediate-stuff:
  plugin: ...
  source: ...
  prime:
    - -*

So that any files created by this part will be filtered out and won’t appear in the final package (but other parts will see them in CRAFT_STAGE).

This is a good point. I’m not familiar with flutter building, but can we have a single part installing the sdk, and all the other parts that require the sdk using it from the stage dir? Something like:

parts:
  a:
    override-build: |
      ... installs the sdk ...
  b:
    after: [a]
    override-build: |
      ... uses the sdk from $CRAFT_STAGE ...

Right, so I just spend an day to make my snapcraft yaml more in-line of the defaults. Like you said instead of override-stage use stage directive to avoid file migration, avoid manual migration in favor for craftctl default. Truth is the same behavior pops up, and no effect at all.


Here I’m running stage command against a previously build part, it basically skips the previous stages without any problem < 3 seconds time.

root@ubnt:/mnt/c# snapcraft stage liquid --use-lxd --verbosity verbose
Starting Snapcraft 7.2.9.post33+git2afca58f
Logging execution to '/root/.cache/snapcraft/log/snapcraft-20230111-074728.621226.log'
Running on amd64 for amd64
Launching instance...
Starting Snapcraft 7.2.9
Logging execution to '/tmp/snapcraft.log'
Running on amd64 for amd64
Initializing parts lifecycle
Executing parts lifecycle...
Executing parts lifecycle: skip pull liquid (already ran)
Executing action
Executed: skip pull liquid (already ran)
Executing parts lifecycle: skip overlay liquid (already ran)
Executing action
Executed: skip overlay liquid (already ran)
Executing parts lifecycle: skip build liquid (already ran)
Executing action
Executed: skip build liquid (already ran)
Executing parts lifecycle: restage liquid (requested step)
Executing action
Executed: restage liquid (requested step)
Executed parts lifecycle

Here I’m running stage command against another previously build part, it basically skips the previous stages again without any problem.

root@ubnt:/mnt/c# snapcraft stage terminal --use-lxd --verbosity verbose
Starting Snapcraft 7.2.9.post33+git2afca58f
Logging execution to '/root/.cache/snapcraft/log/snapcraft-20230111-074858.640266.log'
Running on amd64 for amd64
Launching instance...
Starting Snapcraft 7.2.9
Logging execution to '/tmp/snapcraft.log'
Running on amd64 for amd64
Initializing parts lifecycle
Executing parts lifecycle...
Executing parts lifecycle: skip pull terminal (already ran)
Executing action
Executed: skip pull terminal (already ran)
Executing parts lifecycle: skip overlay terminal (already ran)
Executing action
Executed: skip overlay terminal (already ran)
Executing parts lifecycle: skip build terminal (already ran)
Executing action
Executed: skip build terminal (already ran)
Executing parts lifecycle: restage terminal (requested step)
Executing action
Executed: restage terminal (requested step)
Executed parts lifecycle

So when both those stages can individually be staged and skip any previous pull and build stages, what makes a combination of running the two simulataniously different?

root@ubnt:/mnt/c# date
Wed Jan 11 07:54:56 AM UTC 2023
root@ubnt:/mnt/c# snapcraft stage liquid terminal --use-lxd --verbosity verbose
Starting Snapcraft 7.2.9.post33+git2afca58f
Logging execution to '/root/.cache/snapcraft/log/snapcraft-20230111-075504.713683.log'
Running on amd64 for amd64
Launching instance...
Starting Snapcraft 7.2.9
Logging execution to '/tmp/snapcraft.log'
Running on amd64 for amd64
Initializing parts lifecycle - (817.7s)

This illustrates the problem that Im having because snapcraft stage essentionally marks all parts for staging, but multiple parts deadlocks on something internally without anything obvious.


This is exactly what I’m doing already. I’ve made some improvements to the code posted above but the general idea is the same. Download flutter from git using a shallow clone, use the flutter precache command to download the sdk and stage for use in other build steps. I don’t know if this use of $CRAFT_PART_INSTALL can be furter removed, didn’t have luck with organize.

  source:
    plugin: nil
    source: https://github.com/flutter/flutter.git
    source-type: "git"
    source-depth: 1
    source-branch: "3.7.0-15.0.pre"
    build-packages: [clang, cmake, git, ninja-build, pkg-config, libgtk-3-dev, liblzma-dev, libunwind-dev, curl, unzip, rsync]
    stage-packages: [libgtk-3-0, libblkid1, liblzma5, dotnet-sdk-6.0, dotnet-runtime-6.0]

    # use the default git source to do a shallow clone of the repository, and then
    # do an partial fetch to resolve git history & tags, after pulling precache the 
    # relevant sdks (installed in ./bin/internal/cache/) so consequative build steps
    # are preinitialized.
    override-pull: |
      craftctl default
      git fetch --unshallow --filter=tree:0 
      set -eux
      ./bin/flutter precache --linux
      ./bin/flutter doctor

    # prepare the directory structure for installing it in the staging area, complete 
    # with precached artifacts from the pull step.
    override-build: |
      craftctl default
      mkdir -p $CRAFT_PART_INSTALL/usr/bin
      mkdir -p $CRAFT_PART_INSTALL/usr/libexec
      cp -rp . $CRAFT_PART_INSTALL/usr/libexec/flutter
      ln -sf $CRAFT_PART_INSTALL/usr/libexec/flutter/bin/flutter $CRAFT_PART_INSTALL/usr/bin/flutter
      ln -sf $CRAFT_PART_INSTALL/usr/libexec/flutter/bin/dart $CRAFT_PART_INSTALL/usr/bin/dart

    # remove flutter compile time dependencies from being primed while preserving
    # other critical dependencies such as dotnet-sdks.
    prime:
      - -usr/libexec/flutter
      - -usr/bin/flutter
      - -usr/bin/dart

To avoid manual file migration I’ve simplified the packages so that they don’t use rsync but craftctl default to initiate the copy, and then I basically stage the source code in $SNAPCRAFT_STAGE/opt/ because the source code are prerequisites for the build, similiar to flutter sdks and toolchains.

  packages:
    source: packages
    source-type: local
    plugin: nil
    after: [source]
    override-pull: |
      craftctl default
      mkdir -p $CRAFT_PART_INSTALL/opt/
      cp -rp . $CRAFT_PART_INSTALL/opt/packages
    stage: [opt]
    prime: [-*, -opt/packages]

  websocket:
    source: websocket
    source-type: local
    plugin: nil
    after: [source]
    override-pull: |
      craftctl default
    override-build: |
      craftctl default
      mkdir -p $CRAFT_PART_INSTALL/opt/
      cp -rp . $CRAFT_PART_INSTALL/opt/websocket
    stage: [opt]
    prime: [-*, -opt/websocket]

The main parts follows the structure below, essentionally it just grabs the previously staged resources to reconstruct directory structure required for compilation and stages the artifacts. Yes there’s some manual file migration here to overome limitations.

  terminal:
    # use source local for snapcraft to identify if files have changed
    source: terminal_src
    source-type: local
    plugin: nil

    # pull source code using snapcraft and then reconstruct the directory structure using
    # previously staged artifacts required for building
    override-pull: |
      shopt -s extglob
      craftctl default
      mkdir -p terminal_src
      mv !(terminal_src) terminal_src
      mkdir -p .nuget
      mkdir -p .pubcache
      export PUB_CACHE="$(pwd)/.pubcache"
      export NUGET_PACKAGES="$(pwd)/.nuget"
      rsync --mkpath -avzr $CRAFT_STAGE/opt/websocket/* ./websocket
      rsync --mkpath -avzr $CRAFT_STAGE/opt/packages/* ./packages
      #craftctl default
      cd terminal_src
      flutter pub get
      flutter clean

    # copies over the files from ~/part/<part>/source into the current directory, reuse 
    # nuget and pubcache artifacts (precaching during pull step), and then build flutter
    override-build: |
      craftctl default
      export PUB_CACHE="$(pwd)/.pubcache"
      export NUGET_PACKAGES="$(pwd)/.nuget"
      cd terminal_src
      flutter build linux
      rsync --mkpath -avzr build/linux/x64/release/bundle/* $CRAFT_PART_INSTALL/usr/share/terminal
    stage: [usr/share/terminal]
    prime: [usr/share/terminal]

    # requires staged artifacts as a prerequisites
    after: [source, packages, websocket]

Can you supply a reproducer based on the yaml after your changes so I can debug it locally?

I believe I’ve been able to reproduce the problem on a small scale with the github link I’ve sent you before. Initially when I try to build repo it did went through from start to finish, albeit, with the “websocket” and “packages” pull is outdated… steps taking about 70-90 seconds for completion.

I’m not sure how the “checking for pull outdated” mechanism works but I believe spending 70-90 seconds for 270 kilobytes worth of data isn’t up to a right start.

Here comes the caveat, in my reproduction. I was also able to simulate an “indefinite hang” but it required one more step and that is building one of the flutter packages (and essentionally having a dirty tree). Once I did this I’ve had it hanging for more than 360 seconds without any progress.

Flutter build process is that uses symlinks and junctions, so my best guess is that something in that process doesn’t play nice, with snapcrafts source mechanism.

Hi, thanks for preparing a reproducer. I checked out the project and it still has rsyncs between stage and local sources in override pull, and that’s confusing the internal dependency verification. Local sources can’t be modified during parts processing, or internal states will be inconsistent. If you have a version without rsyncs and it still blocks somewhere, please update the reproducer so I can investigate.

If you want me to get rid of those rsync please give an valid alternative. The reason I put the source: "./packages" instead of source: "." is that the full project directory isn’t relavant and has too many files and it slows doing pull step to 40 minutes.

  • snapcraft pull is generally a slow due to directory recursion is handled python.
  • snapcraft doesn’t support any form of exclussions to avoid having to copy unrelevant files i.e. .git, .node, previous build folders with executables of a different platform or just stuff like README.md – which really have no business in being pulled.
  • community requests for such exclussions have been left ignored.
  • my proposed work around against technical impossed restrictions to partially reconstruct the project folder structure goes nuts internally.

So honestly said, what you’d actually expect me to do here? If override-pull can be used to support e.g. pulling from an non-default supported, why would rsync any different.

I often see textbook examples of snapcraft pull an github repository and then to apply an patch afterwards, those are also local source modification. Why doesn’t snapcraft get’s confussed about this?

Let’s ignore all potential internal inconsistencies and break down what your spec is doing to have a better understanding of how we can solve the problem. When I use the reproducer spec, the following actions are executed:

$ snapcraft clean; snapcraft                                                                                           
Cleaned build provider                                                                                                                                                                         
Launching instance...                                                                                                                                                                          
Executed: pull source                                                                                                                                                                          
Executed: pull packages                                                                                                                                                                        
Executed: pull websocket                                                                                                                                                                       
'override-pull' in part 'liquid' failed with code 23.                           

And the corresponding error in log is:

rsync: [sender] change_dir "/root/stage/opt/websocket" failed: No such file or directory (2)

So when the pull step runs, override-pull fails because there’s nothing to sync from stage to websocket/ (because no stage step ran, we’re still running pulls). But this doesn’t match the initial problem you described, how can we fix it so we can continue this investigation?

Just to be on the same page over here in the reproducer that I gave you the liquid part contains after: [source, packages, websocket]. According to https://snapcraft.io/docs/parts-lifecycle#heading--step-dependencies

In the above example, the part named grv will be built after the part named libgit2 has been successfully built and staged.

So according to the documentation, this means that prior to processing the liquid part, the parts websocket and packages should have already been staged. In that respect it isn’t really that much different from using one of the parts to download a special compiler, built and install it. Use that in consecutive steps in the compilation cyclus of another part.

The documentation is correct, the part that has a dependency on another part because it uses after: [another_part] will have its build step executed after another_part is built and staged. But the part’s pull step is executed before that, only the build step is executed after the other part is staged. So prior to building the liquid part parts websocket and packages are already staged, but the liquid part is pulled before anything else is built or staged.

So if I understand correctly, if I move those rsyncs to the build step the logical ordering of the steps will be correct. I don’t mind giving it a shot but I don’t think it’s fully statisfactory either.

The downside I see is because I can’t reconstruct the folder during the pull stage (e.g. rsync confusses snapcraft, logical odering of pull vs build steps), so it won’t be able to precache the application specific dependencies. I origionally settled for that approach due to limited bandwidth constraints, with the idea that pull is intended for network fetches.

Yes, moving the rsyncs to build restores the intended order, but I agree that it would be conceptually uglier and we should work towards a cleaner solution. Would using the flutter snap as a system-wide build-snap work in your case? Also we have a core22 flutter plugin that should be available soon for you to try, I’ll keep you informed about availability.

I think a build-snap can work equally well, sometimes I’m adapating the code to the more expirimental features so it would just depend which version ends up being distributed in the snap vs what’s on git.

I’d love give the new flutter plugin a shot. However honestly said I’m on an constrained 3mbsp line and often unreliable. If these plugins are designed in a way they waste system resources e.g. redunant network fetches I wouldn’t actually want to use them because really the additional resources it downloads crawls to completion.


I’ve adjust the reproducer example, as discussed. When I run snapcraft clean followed by snapcraft prime it compiles on the first try. When I run snapcraft prime consequatively a second time, I’d expect it to simply skip through the relevant stages but instead I’m getting this:

2023-01-16 16:19:58.483 CPU count (from process affinity): 4
2023-01-16 16:19:58.483 Invalid SNAPCRAFT_MAX_PARALLEL_BUILD_COUNT value
2023-01-16 16:19:58.484 Running on amd64 for amd64
2023-01-16 16:19:58.487 Initializing parts lifecycle
2023-01-16 16:19:58.487 is_snap: True, SNAP_NAME set to snapcraft
2023-01-16 16:19:58.493 load state file: /root/parts/source/state/pull
2023-01-16 16:19:58.503 load state file: /root/parts/packages/state/pull
2023-01-16 16:19:58.506 load state file: /root/parts/websocket/state/pull
2023-01-16 16:19:58.510 load state file: /root/parts/liquid/state/pull
2023-01-16 16:19:58.515 load state file: /root/parts/selfscan/state/pull
2023-01-16 16:19:58.519 load state file: /root/parts/terminal/state/pull
2023-01-16 16:19:58.523 load state file: /root/parts/liquid-pos/state/pull
2023-01-16 16:19:58.527 load state file: /root/parts/source/state/overlay
2023-01-16 16:19:58.534 load state file: /root/parts/packages/state/overlay
2023-01-16 16:19:58.540 load state file: /root/parts/websocket/state/overlay
2023-01-16 16:19:58.544 load state file: /root/parts/liquid/state/overlay
2023-01-16 16:19:58.550 load state file: /root/parts/selfscan/state/overlay
2023-01-16 16:19:58.555 load state file: /root/parts/terminal/state/overlay
2023-01-16 16:19:58.559 load state file: /root/parts/liquid-pos/state/overlay
2023-01-16 16:19:58.563 load state file: /root/parts/source/state/build
2023-01-16 16:19:58.586 load state file: /root/parts/source/state/stage
2023-01-16 16:20:01.070 load state file: /root/parts/packages/state/build
2023-01-16 16:20:01.094 load state file: /root/parts/websocket/state/build
2023-01-16 16:20:01.118 load state file: /root/parts/packages/state/stage
2023-01-16 16:20:01.130 load state file: /root/parts/websocket/state/stage
2023-01-16 16:20:01.139 load state file: /root/parts/liquid/state/build
2023-01-16 16:20:01.162 load state file: /root/parts/selfscan/state/build
2023-01-16 16:20:01.185 load state file: /root/parts/terminal/state/build
2023-01-16 16:20:01.207 load state file: /root/parts/liquid/state/stage
2023-01-16 16:20:01.213 load state file: /root/parts/selfscan/state/stage
2023-01-16 16:20:01.218 load state file: /root/parts/terminal/state/stage
2023-01-16 16:20:01.225 load state file: /root/parts/liquid-pos/state/build
2023-01-16 16:20:01.248 load state file: /root/parts/liquid-pos/state/stage
2023-01-16 16:20:01.252 load state file: /root/parts/source/state/prime
2023-01-16 16:20:02.719 load state file: /root/parts/packages/state/prime
2023-01-16 16:20:02.723 load state file: /root/parts/websocket/state/prime
2023-01-16 16:20:02.727 load state file: /root/parts/liquid/state/prime
2023-01-16 16:20:02.734 load state file: /root/parts/selfscan/state/prime
2023-01-16 16:20:02.739 load state file: /root/parts/terminal/state/prime
2023-01-16 16:20:02.743 load state file: /root/parts/liquid-pos/state/prime
2023-01-16 16:20:02.747 process source:Step.PULL
2023-01-16 16:20:02.748 check if Part('source'):Step.PULL is dirty
2023-01-16 16:20:02.752 check if Part('source'):Step.PULL is outdated
2023-01-16 16:20:02.752 add action source:Step.PULL(ActionType.SKIP)
2023-01-16 16:20:02.752 process packages:Step.PULL
2023-01-16 16:20:02.753 check if Part('packages'):Step.PULL is dirty
2023-01-16 16:20:02.753 check if Part('packages'):Step.PULL is outdated
2023-01-16 16:20:02.753 ignore patterns: ['*.snap']
2023-01-16 16:20:02.796 updated files: set()
2023-01-16 16:20:02.796 updated directories: set()
2023-01-16 16:20:02.796 add action packages:Step.PULL(ActionType.SKIP)
2023-01-16 16:20:02.796 process websocket:Step.PULL
2023-01-16 16:20:02.796 check if Part('websocket'):Step.PULL is dirty
2023-01-16 16:20:02.796 check if Part('websocket'):Step.PULL is outdated
2023-01-16 16:20:02.796 ignore patterns: ['*.snap']
2023-01-16 16:20:02.824 updated files: set()
2023-01-16 16:20:02.824 updated directories: set()
2023-01-16 16:20:02.824 add action websocket:Step.PULL(ActionType.SKIP)
2023-01-16 16:20:02.824 process liquid:Step.PULL
2023-01-16 16:20:02.825 check if Part('liquid'):Step.PULL is dirty
2023-01-16 16:20:02.825 check if Part('liquid'):Step.PULL is outdated
2023-01-16 16:20:02.825 ignore patterns: ['*.snap']
2023-01-16 16:20:02.904 updated files: set()
2023-01-16 16:20:02.904 updated directories: set()
2023-01-16 16:20:02.904 add action liquid:Step.PULL(ActionType.SKIP)
2023-01-16 16:20:02.904 process selfscan:Step.PULL
2023-01-16 16:20:02.905 check if Part('selfscan'):Step.PULL is dirty
2023-01-16 16:20:02.905 check if Part('selfscan'):Step.PULL is outdated
2023-01-16 16:20:02.905 ignore patterns: ['*.snap']
2023-01-16 16:20:02.989 updated files: set()
2023-01-16 16:20:02.989 updated directories: set()
2023-01-16 16:20:02.989 add action selfscan:Step.PULL(ActionType.SKIP)
2023-01-16 16:20:02.989 process terminal:Step.PULL
2023-01-16 16:20:02.989 check if Part('terminal'):Step.PULL is dirty
2023-01-16 16:20:02.990 check if Part('terminal'):Step.PULL is outdated
2023-01-16 16:20:02.990 ignore patterns: ['*.snap']
2023-01-16 16:20:03.076 updated files: set()
2023-01-16 16:20:03.077 updated directories: set()
2023-01-16 16:20:03.077 add action terminal:Step.PULL(ActionType.SKIP)
2023-01-16 16:20:03.077 process liquid-pos:Step.PULL
2023-01-16 16:20:03.077 check if Part('liquid-pos'):Step.PULL is dirty
2023-01-16 16:20:03.077 check if Part('liquid-pos'):Step.PULL is outdated
2023-01-16 16:20:03.077 add action liquid-pos:Step.PULL(ActionType.SKIP)
2023-01-16 16:20:03.077 process source:Step.OVERLAY
2023-01-16 16:20:03.077 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:20:03.077 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:20:03.077 add action source:Step.OVERLAY(ActionType.SKIP)
2023-01-16 16:20:03.077 process packages:Step.OVERLAY
2023-01-16 16:20:03.078 check if Part('packages'):Step.OVERLAY is dirty
2023-01-16 16:20:03.078 check if Part('packages'):Step.OVERLAY is outdated
2023-01-16 16:20:03.078 add action packages:Step.OVERLAY(ActionType.SKIP)
2023-01-16 16:20:03.078 process websocket:Step.OVERLAY
2023-01-16 16:20:03.078 check if Part('websocket'):Step.OVERLAY is dirty
2023-01-16 16:20:03.078 check if Part('websocket'):Step.OVERLAY is outdated
2023-01-16 16:20:03.078 add action websocket:Step.OVERLAY(ActionType.SKIP)
2023-01-16 16:20:03.078 process liquid:Step.OVERLAY
2023-01-16 16:20:03.079 check if Part('liquid'):Step.OVERLAY is dirty
2023-01-16 16:20:03.079 check if Part('liquid'):Step.OVERLAY is outdated
2023-01-16 16:20:03.079 add action liquid:Step.OVERLAY(ActionType.SKIP)
2023-01-16 16:20:03.079 process selfscan:Step.OVERLAY
2023-01-16 16:20:03.079 check if Part('selfscan'):Step.OVERLAY is dirty
2023-01-16 16:20:03.079 check if Part('selfscan'):Step.OVERLAY is outdated
2023-01-16 16:20:03.079 add action selfscan:Step.OVERLAY(ActionType.SKIP)
2023-01-16 16:20:03.079 process terminal:Step.OVERLAY
2023-01-16 16:20:03.079 check if Part('terminal'):Step.OVERLAY is dirty
2023-01-16 16:20:03.080 check if Part('terminal'):Step.OVERLAY is outdated
2023-01-16 16:20:03.081 add action terminal:Step.OVERLAY(ActionType.SKIP)
2023-01-16 16:20:03.081 process liquid-pos:Step.OVERLAY
2023-01-16 16:20:03.081 check if Part('liquid-pos'):Step.OVERLAY is dirty
2023-01-16 16:20:03.081 check if Part('liquid-pos'):Step.OVERLAY is outdated
2023-01-16 16:20:03.081 add action liquid-pos:Step.OVERLAY(ActionType.SKIP)
2023-01-16 16:20:03.081 process source:Step.BUILD
2023-01-16 16:20:03.081 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:20:03.082 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:20:03.082 add action source:Step.BUILD(ActionType.SKIP)
2023-01-16 16:20:03.082 process packages:Step.BUILD
2023-01-16 16:20:03.082 check if Part('packages'):Step.BUILD is dirty
2023-01-16 16:20:03.082 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:20:03.083 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:20:03.083 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:20:03.083 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:20:03.083 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:20:03.084 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:20:03.084 check if Part('source'):Step.PULL is outdated
2023-01-16 16:20:03.084 check if Part('source'):Step.PULL is dirty
2023-01-16 16:20:03.085 check if Part('packages'):Step.BUILD is outdated
2023-01-16 16:20:03.085 add action packages:Step.BUILD(ActionType.SKIP)
2023-01-16 16:20:03.085 process websocket:Step.BUILD
2023-01-16 16:20:03.085 check if Part('websocket'):Step.BUILD is dirty
2023-01-16 16:20:03.086 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:20:03.086 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:20:03.086 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:20:03.086 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:20:03.086 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:20:03.087 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:20:03.087 check if Part('source'):Step.PULL is outdated
2023-01-16 16:20:03.087 check if Part('source'):Step.PULL is dirty
2023-01-16 16:20:03.087 check if Part('websocket'):Step.BUILD is outdated
2023-01-16 16:20:03.087 add action websocket:Step.BUILD(ActionType.SKIP)
2023-01-16 16:20:03.087 process liquid:Step.BUILD
2023-01-16 16:20:03.087 check if Part('liquid'):Step.BUILD is dirty
2023-01-16 16:20:03.087 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:20:03.087 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:20:03.087 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:20:03.087 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:20:03.087 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:20:03.087 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:20:03.087 check if Part('source'):Step.PULL is outdated
2023-01-16 16:20:03.087 check if Part('source'):Step.PULL is dirty
2023-01-16 16:20:03.088 check if Part('packages'):Step.STAGE is outdated
2023-01-16 16:20:03.088 check if Part('packages'):Step.STAGE is dirty
2023-01-16 16:20:03.088 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:20:03.088 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:20:03.088 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:20:03.088 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:20:03.088 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:20:03.088 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:20:03.088 check if Part('source'):Step.PULL is outdated
2023-01-16 16:20:03.088 check if Part('source'):Step.PULL is dirty
2023-01-16 16:20:03.088 check if Part('packages'):Step.BUILD is outdated
2023-01-16 16:20:03.088 check if Part('packages'):Step.BUILD is dirty
2023-01-16 16:20:03.088 check if Part('packages'):Step.OVERLAY is outdated
2023-01-16 16:20:03.088 check if Part('packages'):Step.OVERLAY is dirty
2023-01-16 16:20:03.088 check if Part('packages'):Step.PULL is outdated
2023-01-16 16:20:03.168 updated files: set()
2023-01-16 16:20:03.168 updated directories: set()
2023-01-16 16:20:03.168 check if Part('packages'):Step.PULL is dirty
2023-01-16 16:20:03.168 check if Part('websocket'):Step.STAGE is outdated
2023-01-16 16:20:03.168 check if Part('websocket'):Step.STAGE is dirty
2023-01-16 16:20:03.170 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:20:03.172 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:20:03.172 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:20:03.172 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:20:03.173 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:20:03.173 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:20:03.173 check if Part('source'):Step.PULL is outdated
2023-01-16 16:20:03.173 check if Part('source'):Step.PULL is dirty
2023-01-16 16:20:03.173 check if Part('websocket'):Step.BUILD is outdated
2023-01-16 16:20:03.173 check if Part('websocket'):Step.BUILD is dirty
2023-01-16 16:20:03.173 check if Part('websocket'):Step.OVERLAY is outdated
2023-01-16 16:20:03.173 check if Part('websocket'):Step.OVERLAY is dirty
2023-01-16 16:20:03.173 check if Part('websocket'):Step.PULL is outdated
2023-01-16 16:20:03.216 updated files: set()
2023-01-16 16:20:03.216 updated directories: set()
2023-01-16 16:20:03.216 check if Part('websocket'):Step.PULL is dirty
2023-01-16 16:20:03.216 check if Part('liquid'):Step.BUILD is outdated
2023-01-16 16:20:03.217 add action liquid:Step.BUILD(ActionType.SKIP)
2023-01-16 16:20:03.217 process selfscan:Step.BUILD
2023-01-16 16:20:03.217 check if Part('selfscan'):Step.BUILD is dirty
2023-01-16 16:20:03.219 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:20:03.219 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:20:03.219 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:20:03.219 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:20:03.219 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:20:03.219 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:20:03.219 check if Part('source'):Step.PULL is outdated
2023-01-16 16:20:03.220 check if Part('source'):Step.PULL is dirty
2023-01-16 16:20:03.220 check if Part('packages'):Step.STAGE is outdated
2023-01-16 16:20:03.220 check if Part('packages'):Step.STAGE is dirty
2023-01-16 16:20:03.221 check if Part('packages'):Step.BUILD is outdated
2023-01-16 16:20:03.221 check if Part('packages'):Step.BUILD is dirty
2023-01-16 16:20:03.222 check if Part('packages'):Step.OVERLAY is outdated
2023-01-16 16:20:03.222 check if Part('packages'):Step.OVERLAY is dirty
2023-01-16 16:20:03.222 check if Part('packages'):Step.PULL is outdated
2023-01-16 16:20:04.719 updated files: set()
2023-01-16 16:20:04.719 updated directories: set()
2023-01-16 16:20:04.719 check if Part('packages'):Step.PULL is dirty
2023-01-16 16:20:04.720 check if Part('websocket'):Step.STAGE is outdated
2023-01-16 16:20:04.720 check if Part('websocket'):Step.STAGE is dirty
2023-01-16 16:20:04.721 check if Part('websocket'):Step.BUILD is outdated
2023-01-16 16:20:04.721 check if Part('websocket'):Step.BUILD is dirty
2023-01-16 16:20:04.721 check if Part('websocket'):Step.OVERLAY is outdated
2023-01-16 16:20:04.721 check if Part('websocket'):Step.OVERLAY is dirty
2023-01-16 16:20:04.721 check if Part('websocket'):Step.PULL is outdated
2023-01-16 16:20:05.383 updated files: set()
2023-01-16 16:20:05.384 updated directories: set()
2023-01-16 16:20:05.384 check if Part('websocket'):Step.PULL is dirty
2023-01-16 16:20:05.384 check if Part('selfscan'):Step.BUILD is outdated
2023-01-16 16:20:05.384 add action selfscan:Step.BUILD(ActionType.SKIP)
2023-01-16 16:20:05.384 process terminal:Step.BUILD
2023-01-16 16:20:05.384 check if Part('terminal'):Step.BUILD is dirty
2023-01-16 16:20:05.384 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:20:05.385 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:20:05.385 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:20:05.385 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:20:05.385 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:20:05.385 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:20:05.385 check if Part('source'):Step.PULL is outdated
2023-01-16 16:20:05.385 check if Part('source'):Step.PULL is dirty
2023-01-16 16:20:05.385 check if Part('packages'):Step.STAGE is outdated
2023-01-16 16:20:05.385 check if Part('packages'):Step.STAGE is dirty
2023-01-16 16:20:05.385 check if Part('packages'):Step.BUILD is outdated
2023-01-16 16:20:05.385 check if Part('packages'):Step.BUILD is dirty
2023-01-16 16:20:05.385 check if Part('packages'):Step.OVERLAY is outdated
2023-01-16 16:20:05.385 check if Part('packages'):Step.OVERLAY is dirty
2023-01-16 16:20:05.386 check if Part('packages'):Step.PULL is outdated
2023-01-16 16:20:55.555 updated files: set()
2023-01-16 16:20:55.556 updated directories: set()
2023-01-16 16:20:55.556 check if Part('packages'):Step.PULL is dirty
2023-01-16 16:20:55.556 check if Part('websocket'):Step.STAGE is outdated
2023-01-16 16:20:55.556 check if Part('websocket'):Step.STAGE is dirty
2023-01-16 16:20:55.556 check if Part('websocket'):Step.BUILD is outdated
2023-01-16 16:20:55.556 check if Part('websocket'):Step.BUILD is dirty
2023-01-16 16:20:55.556 check if Part('websocket'):Step.OVERLAY is outdated
2023-01-16 16:20:55.556 check if Part('websocket'):Step.OVERLAY is dirty
2023-01-16 16:20:55.556 check if Part('websocket'):Step.PULL is outdated
2023-01-16 16:21:12.856 updated files: set()
2023-01-16 16:21:12.856 updated directories: set()
2023-01-16 16:21:12.856 check if Part('websocket'):Step.PULL is dirty
2023-01-16 16:21:12.856 check if Part('terminal'):Step.BUILD is outdated
2023-01-16 16:21:12.856 add action terminal:Step.BUILD(ActionType.SKIP)
2023-01-16 16:21:12.857 process liquid-pos:Step.BUILD
2023-01-16 16:21:12.857 check if Part('liquid-pos'):Step.BUILD is dirty
2023-01-16 16:21:12.858 check if Part('terminal'):Step.STAGE is outdated
2023-01-16 16:21:12.859 check if Part('terminal'):Step.STAGE is dirty
2023-01-16 16:21:12.859 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:21:12.859 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:21:12.859 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:21:12.859 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:21:12.859 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:21:12.859 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:21:12.859 check if Part('source'):Step.PULL is outdated
2023-01-16 16:21:12.859 check if Part('source'):Step.PULL is dirty
2023-01-16 16:21:12.859 check if Part('packages'):Step.STAGE is outdated
2023-01-16 16:21:12.859 check if Part('packages'):Step.STAGE is dirty
2023-01-16 16:21:12.859 check if Part('packages'):Step.BUILD is outdated
2023-01-16 16:21:12.859 check if Part('packages'):Step.BUILD is dirty
2023-01-16 16:21:12.860 check if Part('packages'):Step.OVERLAY is outdated
2023-01-16 16:21:12.860 check if Part('packages'):Step.OVERLAY is dirty
2023-01-16 16:21:12.860 check if Part('packages'):Step.PULL is outdated
2023-01-16 16:51:16.041 updated files: set()
2023-01-16 16:51:16.041 updated directories: set()
2023-01-16 16:51:16.043 check if Part('packages'):Step.PULL is dirty
2023-01-16 16:51:16.043 check if Part('websocket'):Step.STAGE is outdated
2023-01-16 16:51:16.043 check if Part('websocket'):Step.STAGE is dirty
2023-01-16 16:51:16.043 check if Part('websocket'):Step.BUILD is outdated
2023-01-16 16:51:16.043 check if Part('websocket'):Step.BUILD is dirty
2023-01-16 16:51:16.043 check if Part('websocket'):Step.OVERLAY is outdated
2023-01-16 16:51:16.043 check if Part('websocket'):Step.OVERLAY is dirty
2023-01-16 16:51:16.043 check if Part('websocket'):Step.PULL is outdated
2023-01-16 16:58:34.133 updated files: set()
2023-01-16 16:58:34.134 updated directories: set()
2023-01-16 16:58:34.134 check if Part('websocket'):Step.PULL is dirty
2023-01-16 16:58:34.134 check if Part('terminal'):Step.BUILD is outdated
2023-01-16 16:58:34.135 check if Part('terminal'):Step.BUILD is dirty
2023-01-16 16:58:34.135 check if Part('terminal'):Step.OVERLAY is outdated
2023-01-16 16:58:34.135 check if Part('terminal'):Step.OVERLAY is dirty
2023-01-16 16:58:34.135 check if Part('terminal'):Step.PULL is outdated
2023-01-16 16:58:34.294 updated files: set()
2023-01-16 16:58:34.295 updated directories: set()
2023-01-16 16:58:34.295 check if Part('terminal'):Step.PULL is dirty
2023-01-16 16:58:34.295 check if Part('selfscan'):Step.STAGE is outdated
2023-01-16 16:58:34.295 check if Part('selfscan'):Step.STAGE is dirty
2023-01-16 16:58:34.295 check if Part('source'):Step.STAGE is outdated
2023-01-16 16:58:34.295 check if Part('source'):Step.STAGE is dirty
2023-01-16 16:58:34.295 check if Part('source'):Step.BUILD is outdated
2023-01-16 16:58:34.295 check if Part('source'):Step.BUILD is dirty
2023-01-16 16:58:34.295 check if Part('source'):Step.OVERLAY is outdated
2023-01-16 16:58:34.295 check if Part('source'):Step.OVERLAY is dirty
2023-01-16 16:58:34.295 check if Part('source'):Step.PULL is outdated
2023-01-16 16:58:34.295 check if Part('source'):Step.PULL is dirty
2023-01-16 16:58:34.295 check if Part('packages'):Step.STAGE is outdated
2023-01-16 16:58:34.295 check if Part('packages'):Step.STAGE is dirty
2023-01-16 16:58:34.295 check if Part('packages'):Step.BUILD is outdated
2023-01-16 16:58:34.295 check if Part('packages'):Step.BUILD is dirty
2023-01-16 16:58:34.296 check if Part('packages'):Step.OVERLAY is outdated
2023-01-16 16:58:34.296 check if Part('packages'):Step.OVERLAY is dirty
2023-01-16 16:58:34.296 check if Part('packages'):Step.PULL is outdated - (28565.3s)

So now that my snapcraft shouldn’t be confussed anymore due to the pulls and the stages compile on the first attempt, what is your thoughts on this? I don’t this is is normal behavior spending 28565.3s on 70 Files, 35 Folders, 276 KB (282.624 bytes)?

I agree that some operations seem to be taking an excessive amount of time, I’ll investigate and report back.

Update: I’m seeing the following error when running the updated test, could you verify and possibly add a working mock so I can run it locally?

:: Running "flutter pub get" in liquid_src...                                                                                       
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:37.928 :: Resolving dependencies...                                                                                                        
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:38.853 :: Could not find a file named "pubspec.yaml" in "/root/parts/liquid/build/packages/calc".                                          
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:39.397 :: pub get failed                                                                                                                   
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:39.397 :: command: "/root/parts/source/install/usr/libexec/flutter/bin/cache/dart-sdk/bin/dart __deprecated_pub --directory . get --example
"                                                                                                                                                                                      
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:39.398 :: pub env: {                                                                                                                       
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:39.398 ::   "FLUTTER_ROOT": "/root/parts/source/install/usr/libexec/flutter",                                                              
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:39.398 ::   "PUB_ENVIRONMENT": "flutter_bot:flutter_cli:get",                                                                              
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:39.398 ::   "PUB_CACHE": "/root/parts/liquid/build/.pubcache",                                                                             
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:39.398 :: }                                                                                                                                
2023-01-17 10:12:40.184 :: 2023-01-17 13:12:39.398 :: exit code: 66 

It’s a git submodule (https://github.com/phr34k/test/blob/main/.gitmodules) you should be able to just run git submodule update --init --recursive where you checked-out the project. Added that small detail, just so’d have a closer representation what my project does.

I’ve executed this and It takes about 7 minutes to complete on my system (I’m aware that this time is highly dependent on download times and the system configuration – it’s just a reference value to compare with the second run).

When I run snapcraft prime again on my system it completes in about 5 minutes, without the strange time gaps found in your logs – is your system very low in RAM to the point it could be starting to swap?

I also see that Snapcraft is re-executing steps that have already been executed in a previous run. I’ll have to check the logs and maybe instrument code to see what’s happening here, so it may take some time to investigate.

When I did a fresh clone that also happens for me with a consequative primes. As briefly touched on above, the pulls started to first behave weird after I tried to build/run the calc/example (outside of snapcraft).

I have no idea of what happens internally but I know that flutter likes use symlinks and junctions so I was thinking something along the lines that the dirty build (symlinks) files were causing some kind of cyclic cycle/references inside of snapcrafts pull/out-of-date mechanism.

Yeah I’m not really sure about the time gaps, I just let the system run it’s course over night. It’s literally just in check if Part ....PULL is outdated [/|\] for like hours. Am running it on an headless ubuntu machine (virtualbox) with 4gb assigned, I doubt it’s swapping, but I can always assign it another 4-5gb easilly.

What is the sequence of actions I could execute to reproduce the problematic behavior? I’m not familiar with flutter building and having a reproducible procedure would ensure our scenarios are consistent.