Disabling automatic refresh for snap from store

backlog

#68

Denis,

thanks for outlining this again, but I might be a “special” user. I don’t really care for myself as I’m on the developer side, but I care for the users of my snap who complain about a “crash” (not a crash, it’s actually the restart) every time I push a new snap of my application.
The best I can do is suggest your solution to my users, but I would rather have a real solution.

I’ve outlined the above case as frequently the push back was to do better testing and slowly promoting the snap from edge to stable. In this case this would not have helped.
A workaround like you suggest would work. Downloading the snap and installing it manually is another (ugly) workaround to avoid updates for a specific snap.

  • Martin

#69

Updates happen on scheduled windows, not every time you push your snap. We’re working right now (@mborzecki is looking into this already) into pushing the scheduling to a monthly basis, which means people can schedule exactly what they want inside the month.

The primary goal is not to take control away from the user altogether, but to strongly encourage the practice of regular maintenance. Per conversation above, we’re working on multiple features that better support this idea, wider scheduling being one of them, and epochs being another, both in progress (health checks too, but not yet in progress). We also haven’t ruled out doing exactly what you suggest, and simply introducing a global refresh shutdown flag. The reason we haven’t done that yet, again, is because once we go there we cannot take it out, and that one chance of trying to establish new practices around this problem is gone.

If you can do what @denis suggests for the time being for the unbearable cases, and stick with us while trying to see if we’re able to move forward with the paradigm of automatic refreshes being the common rule, then we’d also be very happy to continue learning from your experiences (positive and negative) with those mechanisms so we can jointly understand how far we can take it.


#70

^^ This behavior would be desirable for our application. We have an auto-update built in to our application that allows us to update the app without updating the snap. We allow our users to set the maintenance window that we would like any updates to conform to. Even though we don’t have to update the snap to update our application, there may be times when an update to the snap is necessary and we’d love to be able to have the snap refresh conform to the maintenance window set for our app.

Any word on a timeline for this feature?


#71

We don’t have a timeline on that part of the feature just yet. The high-priority changes which are already in progress aim at increasing control on the user side.


#72

Maybe I’m late for this discussion, but I want the developers to know how necessary is to give the users the option to disable the autopdates. Imagine for a moment that you live in a 3rd world country where 40 people share a 1Mbps link and suddenly all of them start downloading the ubuntu-core update. That’s the case where I work. Microsoft Windows 10 is banned in our environment until we hack it and prevent it from autoupdating. The process is complicated, but doable. With snapd is hardcoded, so our only option would be to download the sources, disable the autoupdate in the code, compile, deply, etc and that’s insane. Implementing scheduling will take you time, giving the users a way to disable autoupdates will take you a couple of hours at most and will make everybody happy. Give the users freedom. Thank you for the all hard work and please think again about this issue.


#73

‘everybody happy’ except when some vulnerability which would’ve got fixed by updates gets exploited. It seems monthly refresh scheduling will be out by the end of January or so (there’s no release dates for snapd 2.31 yet but it looks like it’ll be in that) or a few weeks earlier if you refresh to the snapd 2.31 beta when it comes out. See The snapd roadmap

I agree with your sentiment though, it would be faster to implement an automatic refresh disable switch, and would respect users’ control…but it seems the snappy developers would rather go for the long haul here and instead try and meet all users’ needs without implementing that switch. Do they need to allow refreshes to be capped at a certain speed? Managed across all your devices at a higher level? Give a longer refresh window than one month? I’m sure they will seriously consider and implement suggestions that don’t involve an off switch, so this is the place to make them!


#74

By everybody I mean, everybody who wants to disable it for some reason. While I support 100% reasonable defaults, I also think people should be able to override them if necessary. Life’s entropy can prove any configuration, default or decision wrong. While updates are usually good there are a thousand edge cases where they are not desirable or can be even dangerous.

The current thread is about disabling the automatic refresh, so this should the right place to suggest a switch off. If snapd developers don’t want to listen and want to continue thinking their users are children who cannot handle the consequences of disabling automatic updates, then probably many will stop using it.

Another option is also to point api.snapcraft.io to 127.0.0.1. Sysadmins with low bandwidth on their networks could also implement it for their whole networks at the DNS servers.


#75

Looks like refresh scheduling planned to land in snapd 2.31 which will help mitigate this problem (as previously noted by Niemeyer). 2.31 is due to be released 12th February. Presumably it’ll be in edge soon (or is already), I don’t know when it’ll be in the other channels.

As previously discussed, this doesn’t fully mitigate the problem that people have here, but it’s something :slight_smile:


#76

Perhaps it is advisable to consider creation of a store proxy which allows interception of legitimate store requests and caching the binaries delivered in the response while allowing everything else to occur synchronously?


#77

RC1 is there now, RC2 will be coming tomorrow morning.


#78

Is there documentation for this? I can only find https://docs.ubuntu.com/core/en/reference/automatic-refreshes, and it doesn’t describe how to set the update time.


#79

Probably not yet, someone needs to add something under #doc ideally.

To see now how to use it, see here for the command and then use the appropriate syntax here for the end of that command (assuming there weren’t changes in the PR that deviated from that design…)


#80

There’s some documentation on it already actually, under the core configuration options. The monthly style isn’t there yet, though, because it’s only coming in this next release. We need to update it to cover that indeed.


#81

The monthly option is now available on 2.31, and documented. Have fun, and please let us know if you find any issues there.


#82

Chiming in with some real-world experience - we were just victims of an automatic update that bricked a bunch of our devices in the field (we’re using the Dell IoT 3001 gateways, and an update caused an issue that disabled cellular connectivity to our units). We cannot re-establish communication with these devices without performing a hardware reset.

This happened in the middle of the night (always fun!), so our developer team was awoken around 3AM to triage, and then our operations team spent the entire day fielding customer complaints.

Overall I’m a big fan of the philosophy of encouraging regular software updates as a default. But forcing users to accept updates, even with features like scheduling, strikes me as unbearable. We should be able to control the software that runs on our systems and not be beholden to our snap overlords.

I shudder to think about this in the context of a mission-critical / life safety application.


#83

I guess what the snappy design team are assuming is that, in theory, it’s possible for the upstream to test for most edge cases before pushing an update and it’s possible for the user to put time and effort into scheduling updates at a certain time and that they’ll be able to test for most problems before they actually happen - since this is what users do before upgrading software in any case. Even in mission-critical / life safety applications, at some point they update their software, right? How do they do so? By upgrading at a certain time and by extensively testing the upgrade. Why doesn’t the scheduling system allow this to happen?

I know there’s a deeper issue here too of:

It should be noted that snapd is open-source and can thus be forked and modified. The snap design team will hate it (so much that they may delete this post), but it’s possible we could undo the hardcoding to the snappy store and the automatic refreshes and publish the changes as a soft fork. We need someone with the time to code and maintain this though, do you know anyone? Note that said soft fork would probably be less reliable than the automatically refreshing and well-supported snapd so this would be more a proof-of-concept than anything… I realize this flies sharply into the face of much of the snappy project goals but equally, currently, snappy is flying in to the face of many user demands and deep-seated beliefs…


#84

Sounds like upstream might want to implement some health checks in a refresh hook to automatically roll back if issues occur.


#85

Hello @dpryan,

Can you share more details of what actually happened with us? If so, let’s have it in its own topic if you don’t mind, so we keep the conversation here in context.

Then, rest assured the picture is not so bad as it may look like.

Here are some details:

  • Scheduling of refreshes has been supported for a very long time. The fact it refreshed at 3AM means it was not configured to refresh at a more convenient time.

  • Starting with 2.31 snapd can also schedule on a specific window over the entire month, so you can also pick the exact day and time the team will be available to support it.

  • We’ll soon start working on health checks, which is a feature that will enable snaps to report whether an automatic update worked or not, and that will immediately revert if there are problems.

  • We’ve recently landed the repairs feature, which is an emergency system that allows delivering signed documents into systems to unbrick them. That’s made in a signed and very visible way so that neither us nor anybody else can sneak in.

  • Dell devices make use of a validation feature for the main snaps (core, kernel) that only release fundamental snaps into the system once they are approved by testing.

  • Dell device has further control of refreshes because they have their own management solution. I can hook you up with our commercial team if you’d like more details about it.

  • By all means, please talk to us if you plan to use snaps to control an airplane or a heart monitor.

So again, I’m sorry for your incident, and rest assured we’re working to make automatic updates not be painful for anyone, user, publisher, or manufacturer, but we do believe there’s important value in strongly encouraging everyone in that picture to not have to manually act to get problems fixed.

If you are interested, we are happy to work with you to improve this picture even further.


#86

5 posts were split to a new topic: Download size on refreshes


#87

I understand the problem with users holding back upgrades for very long, but there are many other usecases here. I for example is quick to deploy upgrades, but I strongly dislike forced background upgrades.

I want to learn about what’s new before deciding to upgrade. 99/100 times I decide to upgrade right away, but I really need to know what the upgrade brings to the table. Do the new software version remove compatibility with old files or hardware, then I may want to resolve it before upgrading. Is there a all new UI, then I want to learn about it before launching my app. Is the frustrating bug I’ve been avoiding fixed, then I want to learn about it immediately so I can stop avoiding it. Did a upgrade introduce a bug then I would probably have no idea about the background upgrade at all (unless something dramatically changed) as I don’t keep checking the version number every time I launch my apps, filing a regression bug against it would probably be one of my very last thoughts. Etc. Most things with snaps are fantastic and snaps are making Linux a whole lot better in many areas, but the upgrade process needs to be at least as transparent to the user as non-snap upgrades. One of the main reasons I moved from Windows to Linux was because of suprice upgrades, introducing this again will force people like me to stay away from using snaps. I hope this makes you better understand one of the reasons to why some of us are (unwillingly) avoiding snaps.