In the managers and overlord we use the following tools to make sure operations don’t interfere with each other or produce broken state:
locking: State.Lock and State.Unlock where the latter also serializes the state to disk, so that it can be picked up across restarts, crashes etc and operation can continue
when an operation spans multiple tasks we use conflicts and checks for pre-existing pending/running tasks/changes to make sure for example we don’t have concurrent operations on the same snap (see e.g. snapstate.CheckChangeConflict and devicemgr.changeInFlight)
we also use TaskRunner.AddBlocked to add predicate functions to the single shared TaskRunner to control the serialisation of tasks, for example ifacestate conservatively serialises all of its tasks because they can touch multiple snaps
Generally we prefer correctness over concurrence, so in general tasks that are mutating global shared parts of the state should take the State lock once. Exceptions that release the lock around possibly slow operations should avoid touching shared global bits at all if possible, or use serialisation otherwise carefully and sparingly.
The snapstate.CheckChangeConflict is mechanism is pluggable through snapstate.AddAffectedSnapsByAttr (preferably) and snapstate.AddAffectedSnapsByKind.