Validation set refresh deadlock

Hey, our Ubuntu Core system is based on core22, and we are using remodelling(commissioning device) as well as validation set refresh(device update).

We have a snap ubuntu-frame-osk , which has some bugfixes in 24/stable. We would like to use it in our model/validation sets, which requires adding core24 and mesa-2404 to in.

So when refreshing the validation set, it started hanging on task below of validation set refresh change

Doing   today at 11:50 UTC  -                   Ensure prerequisites for "ubuntu-frame-osk" are available

I am expecting it’s getting in a deadlock, which it checks for prerequisites for ubuntu-frame-osk, but those are planned to be installed in the following tasks(See below, core24 and mesa-2404 is planned after this stage.)

What I did before refreshing the validation set I manually installed prerequisites, and then tried to refresh the validation set, and it worked.(I also tried to remodelling, tasks were sorted in a way that prerequisites installed before that step.)

I would like to know if there is a way to avoid this issue; any other suggestions are welcome.


LOGS

An example VS description is below.(Putting ubuntu-frame-osk to the bottom of the validation set)

  "snaps": [
    {
      "name": "core22",
      "presence": "required",
      "revision": 1439
    },
    {
      "name": "core24",
      "presence": "required",
      "revision": 739
    },
....
    {
      "name": "mesa-core22",
      "presence": "required",
      "revision": 311
    },
    {
      "name": "mesa-2404",
      "presence": "required",
      "revision": 495
    },
....
    {
      "name": "ubuntu-frame-osk",
      "presence": "required",
      "revision": 1064
    }
...
  ]

Full logs of the Validation Set refresh change

Done    today at 11:50 UTC  today at 11:50 UTC  Ensure prerequisites for "salto-pc" are available
Done    today at 11:50 UTC  today at 11:50 UTC  Download snap "salto-pc" (107) from channel "latest/stable"
Done    today at 11:50 UTC  today at 11:50 UTC  Fetch and check assertions for snap "salto-pc" (107)
....
Doing   today at 11:50 UTC  -                   Ensure prerequisites for "ubuntu-frame-osk" are available
Do      today at 11:50 UTC  -                   Download snap "ubuntu-frame-osk" (1064) from channel "latest/stable"
Do      today at 11:50 UTC  -                   Fetch and check assertions for snap "ubuntu-frame-osk" (1064)
Do      today at 11:50 UTC  -                   Mount snap "ubuntu-frame-osk" (1064)
Do      today at 11:50 UTC  -                   Run pre-refresh hook of "ubuntu-frame-osk" snap if present
Do      today at 11:50 UTC  -                   Stop snap "ubuntu-frame-osk" services
Do      today at 11:50 UTC  -                   Remove aliases for snap "ubuntu-frame-osk"
Do      today at 11:50 UTC  -                   Make current revision for snap "ubuntu-frame-osk" unavailable
Do      today at 11:50 UTC  -                   Copy snap "ubuntu-frame-osk" data
Do      today at 11:50 UTC  -                   Setup snap "ubuntu-frame-osk" (1064) security profiles
Do      today at 11:50 UTC  -                   Make snap "ubuntu-frame-osk" (1064) available to the system
Do      today at 11:50 UTC  -                   Automatically connect eligible plugs and slots of snap "ubuntu-frame-osk"
Do      today at 11:50 UTC  -                   Set automatic aliases for snap "ubuntu-frame-osk"
Do      today at 11:50 UTC  -                   Setup snap "ubuntu-frame-osk" aliases
Do      today at 11:50 UTC  -                   Run post-refresh hook of "ubuntu-frame-osk" snap if present
Do      today at 11:50 UTC  -                   Start snap "ubuntu-frame-osk" (1064) services
Do      today at 11:50 UTC  -                   Clean up "ubuntu-frame-osk" (1064) install
Do      today at 11:50 UTC  -                   Run configure hook of "ubuntu-frame-osk" snap if present
Do      today at 11:50 UTC  -                   Run health check of "ubuntu-frame-osk" snap
....
Do      today at 11:50 UTC  -                   Ensure prerequisites for "mesa-2404" are available
Do      today at 11:50 UTC  -                   Download snap "mesa-2404" (495) from channel "stable"
Do      today at 11:50 UTC  -                   Fetch and check assertions for snap "mesa-2404" (495)
Do      today at 11:50 UTC  -                   Mount snap "mesa-2404" (495)
Do      today at 11:50 UTC  -                   Copy snap "mesa-2404" data
Do      today at 11:50 UTC  -                   Setup snap "mesa-2404" (495) security profiles
Do      today at 11:50 UTC  -                   Make snap "mesa-2404" (495) available to the system
Do      today at 11:50 UTC  -                   Automatically connect eligible plugs and slots of snap "mesa-2404"
Do      today at 11:50 UTC  -                   Set automatic aliases for snap "mesa-2404"
Do      today at 11:50 UTC  -                   Setup snap "mesa-2404" aliases
Do      today at 11:50 UTC  -                   Run install hook of "mesa-2404" snap if present
Do      today at 11:50 UTC  -                   Run default-configure hook of "mesa-2404" snap if present
Do      today at 11:50 UTC  -                   Start snap "mesa-2404" (495) services
Do      today at 11:50 UTC  -                   Run configure hook of "mesa-2404" snap if present
Do      today at 11:50 UTC  -                   Run health check of "mesa-2404" snap
Do      today at 11:50 UTC  -                   Ensure prerequisites for "core24" are available
Do      today at 11:50 UTC  -                   Download snap "core24" (739) from channel "stable"
Do      today at 11:50 UTC  -                   Fetch and check assertions for snap "core24" (739)
Do      today at 11:50 UTC  -                   Mount snap "core24" (739)
Do      today at 11:50 UTC  -                   Copy snap "core24" data
Do      today at 11:50 UTC  -                   Setup snap "core24" (739) security profiles
Do      today at 11:50 UTC  -                   Make snap "core24" (739) available to the system
Do      today at 11:50 UTC  -                   Automatically connect eligible plugs and slots of snap "core24"
Do      today at 11:50 UTC  -                   Set automatic aliases for snap "core24"
Do      today at 11:50 UTC  -                   Setup snap "core24" aliases
Do      today at 11:50 UTC  -                   Run install hook of "core24" snap if present
Do      today at 11:50 UTC  -                   Start snap "core24" (739) services
Do      today at 11:50 UTC  -                   Run health check of "core24" snap


Any idea @ogra or someone you can point out?

I think there were some issues with the snap store which should be resolved already. Have you tried aborting the change and starting it again?

Yes, I have aborted the change, tried on different devices/networks, and still stuck on there

Doing   today at 16:55 UTC  -                   Ensure prerequisites for "ubuntu-frame-osk" are available
Do      today at 16:55 UTC  -                   Download snap "ubuntu-frame-osk" (1064) from channel "latest/stable"
Do      today at 16:55 UTC  -                   Fetch and check assertions for snap "ubuntu-frame-osk" (1064)
Do      today at 16:55 UTC  -                   Mount snap "ubuntu-frame-osk" (1064)
Do      today at 16:55 UTC  -                   Run pre-refresh hook of "ubuntu-frame-osk" snap if present
Do      today at 16:55 UTC  -                   Stop snap "ubuntu-frame-osk" services
Do      today at 16:55 UTC  -                   Remove aliases for snap "ubuntu-frame-osk"
Do      today at 16:55 UTC  -                   Make current revision for snap "ubuntu-frame-osk" unavailable
Do      today at 16:55 UTC  -                   Copy snap "ubuntu-frame-osk" data
Do      today at 16:55 UTC  -                   Setup snap "ubuntu-frame-osk" (1064) security profiles
Do      today at 16:55 UTC  -                   Make snap "ubuntu-frame-osk" (1064) available to the system
Do      today at 16:55 UTC  -                   Automatically connect eligible plugs and slots of snap "ubuntu-frame-osk"
Do      today at 16:55 UTC  -                   Set automatic aliases for snap "ubuntu-frame-osk"
Do      today at 16:55 UTC  -                   Setup snap "ubuntu-frame-osk" aliases
Do      today at 16:55 UTC  -                   Run post-refresh hook of "ubuntu-frame-osk" snap if present
Do      today at 16:55 UTC  -                   Start snap "ubuntu-frame-osk" (1064) services
Do      today at 16:55 UTC  -                   Clean up "ubuntu-frame-osk" (1064) install
Do      today at 16:55 UTC  -                   Run configure hook of "ubuntu-frame-osk" snap if present
Do      today at 16:55 UTC  -                   Run health check of "ubuntu-frame-osk" snap
Done    today at 16:55 UTC  today at 16:55 UTC  Ensure prerequisites for "salto-conf-hardware" are available
Done    today at 16:55 UTC  today at 16:55 UTC  Download snap "salto-conf-hardware" (84) from channel "latest/stable"
Done    today at 16:55 UTC  today at 16:55 UTC  Fetch and check assertions for snap "salto-conf-hardware" (84)
Done    today at 16:55 UTC  today at 16:56 UTC  Mount snap "salto-conf-hardware" (84)
Done    today at 16:55 UTC  today at 16:56 UTC  Run pre-refresh hook of "salto-conf-hardware" snap if present
Done    today at 16:55 UTC  today at 16:56 UTC  Stop snap "salto-conf-hardware" services
Done    today at 16:55 UTC  today at 16:56 UTC  Remove aliases for snap "salto-conf-hardware"
Done    today at 16:55 UTC  today at 16:56 UTC  Make current revision for snap "salto-conf-hardware" unavailable
Done    today at 16:55 UTC  today at 16:56 UTC  Copy snap "salto-conf-hardware" data
Done    today at 16:55 UTC  today at 16:56 UTC  Setup snap "salto-conf-hardware" (84) security profiles
Done    today at 16:55 UTC  today at 16:56 UTC  Make snap "salto-conf-hardware" (84) available to the system
Done    today at 16:55 UTC  today at 16:56 UTC  Automatically connect eligible plugs and slots of snap "salto-conf-hardware"
Done    today at 16:55 UTC  today at 16:56 UTC  Set automatic aliases for snap "salto-conf-hardware"
Done    today at 16:55 UTC  today at 16:56 UTC  Setup snap "salto-conf-hardware" aliases
Done    today at 16:55 UTC  today at 16:56 UTC  Run post-refresh hook of "salto-conf-hardware" snap if present
Done    today at 16:55 UTC  today at 16:56 UTC  Start snap "salto-conf-hardware" (84) services
Done    today at 16:55 UTC  today at 16:56 UTC  Clean up "salto-conf-hardware" (84) install
Done    today at 16:55 UTC  today at 16:56 UTC  Run configure hook of "salto-conf-hardware" snap if present
Done    today at 16:55 UTC  today at 16:56 UTC  Run health check of "salto-conf-hardware" snap
Do      today at 16:55 UTC  -                   Ensure prerequisites for "mesa-2404" are available
Do      today at 16:55 UTC  -                   Download snap "mesa-2404" (495) from channel "stable"
Do      today at 16:55 UTC  -                   Fetch and check assertions for snap "mesa-2404" (495)
Do      today at 16:55 UTC  -                   Mount snap "mesa-2404" (495)
Do      today at 16:55 UTC  -                   Copy snap "mesa-2404" data
Do      today at 16:55 UTC  -                   Setup snap "mesa-2404" (495) security profiles
Do      today at 16:55 UTC  -                   Make snap "mesa-2404" (495) available to the system
Do      today at 16:55 UTC  -                   Automatically connect eligible plugs and slots of snap "mesa-2404"
Do      today at 16:55 UTC  -                   Set automatic aliases for snap "mesa-2404"
Do      today at 16:55 UTC  -                   Setup snap "mesa-2404" aliases
Do      today at 16:55 UTC  -                   Run install hook of "mesa-2404" snap if present
Do      today at 16:55 UTC  -                   Run default-configure hook of "mesa-2404" snap if present
Do      today at 16:55 UTC  -                   Start snap "mesa-2404" (495) services
Do      today at 16:55 UTC  -                   Run configure hook of "mesa-2404" snap if present
Do      today at 16:55 UTC  -                   Run health check of "mesa-2404" snap
Do      today at 16:55 UTC  -                   Ensure prerequisites for "core24" are available
Do      today at 16:55 UTC  -                   Download snap "core24" (739) from channel "stable"
Do      today at 16:55 UTC  -                   Fetch and check assertions for snap "core24" (739)
Do      today at 16:55 UTC  -                   Mount snap "core24" (739)
Do      today at 16:55 UTC  -                   Copy snap "core24" data
Do      today at 16:55 UTC  -                   Setup snap "core24" (739) security profiles
Do      today at 16:55 UTC  -                   Make snap "core24" (739) available to the system
Do      today at 16:55 UTC  -                   Automatically connect eligible plugs and slots of snap "core24"
Do      today at 16:55 UTC  -                   Set automatic aliases for snap "core24"
Do      today at 16:55 UTC  -                   Setup snap "core24" aliases
Do      today at 16:55 UTC  -                   Run install hook of "core24" snap if present
Do      today at 16:55 UTC  -                   Start snap "core24" (739) services
Do      today at 16:55 UTC  -                   Run health check of "core24" snap
Do      today at 16:55 UTC  -                   Enforce validation sets

Can you provide the /var/lib/snapd/state.json file from that host? Make sure to scrub any auth data. You can use this command:

 sudo cat /var/lib/snapd/state.json | jq '.data.auth = null' > state.json

And submit the output file. Or you can file a bug at launchpad snapd bugs. Make sure to attach the state file.

We already have a SF 00409558 case for this request raised yesterday, maybe we should work on this request at SF instead of multiple places?

Thanks, I have moved my request to the support portal. Whenever it gets resolved, I will post the result here.

1 Like

In summary, there is a bug, where the prerequisite task for a snap waits on installation of a base, but the base depends on completion of installation of the snap, which produces a deadlock, that will eventually time out.

We are working on finalizing a fix: o/snapstate: make sure that bases do not wait on snaps when enforcing validation sets by andrewphelpsj · Pull Request #15357 · canonical/snapd · GitHub.

This fix will likely become available as 2.69.1.

2 Likes