Intermittent failure to "snap refresh" becauses of Store issue?

store
21

Yes that was my idea, I could snap revert now I would need retriggers to see an impact

22

Well, keep in mind that revert only applies to local files, it will not download any former version from the store but only roll back to an existing version on disk (if it exists) …

even downloading remotely using --revision might be a bit tricky since only the store team might have the info what the former revision was that was released…

23

well I was lucky to find that 2.60.3 was revision 20092 so I can pin to this version. I had a look at the changelog but I could not see anything really related, I just find the coincidence at best troubling.

1 Like
24

@ogra so far ~50 retries on the 2.60.3 and no repro. but it’s painful to work because of this auto-refresh kicking randomly

25

100 retries on both version, and outside of the autorefresh I worry there was no error. Yesterday when I reproduced it was taking less.

26 
$ (for task in $(cat group_snapd_2.61.1.txt); do grep "cannot find required base" "${task}.log"; done;) | wc -l
459
$ (for task in $(cat group_snapd_2.60.3.txt); do grep "cannot find required base" "${task}.log"; done;) | wc -l
464

So it is not because of the snap version.

27

I had a look at the times, and it looks like there’s an interesting pattern:

  • tasks started at ~16:10 and ~16:20 for each batch of 500
  • until ~16:24 it works fine
  • starting at that time, it looks like both snapd version started to expose symptoms

is it possible there might be some active throttling at some point for something that might be considered unexpected trafic? or some bug on the path that makes it congested and triggers the problem?

The CI itself outside of that test is ~8 tasks running daily performing a few snap operations (snap refresh + snap install but the snap it install is a local one), so it should not be hammering the service, but it might be doing it all “at once”

28

I believe I found down the root cause, see update: Comment #13 : Bug #2048104 : Bugs : snapd

This suggest that the problem will only reproduce when refreshing snapd from < 2.60.3 to >= 2.60.3 and combining refreshing snapd and something else with dependencies (e.g. core22).

I do not expect any reproduction cases to contradict this, so please let me know.

This also means that its not a store issue, so perhaps this should be moved to snapd?

29

Flagged my first post to get it moved