The monthly option is now available on 2.31, and documented. Have fun, and please let us know if you find any issues there.
1 Like
Chiming in with some real-world experience - we were just victims of an automatic update that bricked a bunch of our devices in the field (we’re using the Dell IoT 3001 gateways, and an update caused an issue that disabled cellular connectivity to our units). We cannot re-establish communication with these devices without performing a hardware reset.
This happened in the middle of the night (always fun!), so our developer team was awoken around 3AM to triage, and then our operations team spent the entire day fielding customer complaints.
Overall I’m a big fan of the philosophy of encouraging regular software updates as a default. But forcing users to accept updates, even with features like scheduling, strikes me as unbearable. We should be able to control the software that runs on our systems and not be beholden to our snap overlords.
I shudder to think about this in the context of a mission-critical / life safety application.
3 Likes
I guess what the snappy design team are assuming is that, in theory, it’s possible for the upstream to test for most edge cases before pushing an update and it’s possible for the user to put time and effort into scheduling updates at a certain time and that they’ll be able to test for most problems before they actually happen - since this is what users do before upgrading software in any case. Even in mission-critical / life safety applications, at some point they update their software, right? How do they do so? By upgrading at a certain time and by extensively testing the upgrade. Why doesn’t the scheduling system allow this to happen?
I know there’s a deeper issue here too of:
It should be noted that snapd is open-source and can thus be forked and modified. The snap design team will hate it (so much that they may delete this post), but it’s possible we could undo the hardcoding to the snappy store and the automatic refreshes and publish the changes as a soft fork. We need someone with the time to code and maintain this though, do you know anyone? Note that said soft fork would probably be less reliable than the automatically refreshing and well-supported snapd so this would be more a proof-of-concept than anything… I realize this flies sharply into the face of much of the snappy project goals but equally, currently, snappy is flying in to the face of many user demands and deep-seated beliefs…
2 Likes
Sounds like upstream might want to implement some health checks in a refresh hook to automatically roll back if issues occur.
1 Like
Hello @dpryan,
Can you share more details of what actually happened with us? If so, let’s have it in its own topic if you don’t mind, so we keep the conversation here in context.
Then, rest assured the picture is not so bad as it may look like.
Here are some details:
-
Scheduling of refreshes has been supported for a very long time. The fact it refreshed at 3AM means it was not configured to refresh at a more convenient time.
-
Starting with 2.31 snapd can also schedule on a specific window over the entire month, so you can also pick the exact day and time the team will be available to support it.
-
We’ll soon start working on health checks, which is a feature that will enable snaps to report whether an automatic update worked or not, and that will immediately revert if there are problems.
-
We’ve recently landed the repairs feature, which is an emergency system that allows delivering signed documents into systems to unbrick them. That’s made in a signed and very visible way so that neither us nor anybody else can sneak in.
-
Dell devices make use of a validation feature for the main snaps (core, kernel) that only release fundamental snaps into the system once they are approved by testing.
-
Dell device has further control of refreshes because they have their own management solution. I can hook you up with our commercial team if you’d like more details about it.
-
By all means, please talk to us if you plan to use snaps to control an airplane or a heart monitor.
So again, I’m sorry for your incident, and rest assured we’re working to make automatic updates not be painful for anyone, user, publisher, or manufacturer, but we do believe there’s important value in strongly encouraging everyone in that picture to not have to manually act to get problems fixed.
If you are interested, we are happy to work with you to improve this picture even further.
2 Likes
I understand the problem with users holding back upgrades for very long, but there are many other usecases here. I for example is quick to deploy upgrades, but I strongly dislike forced background upgrades.
I want to learn about what’s new before deciding to upgrade. 99/100 times I decide to upgrade right away, but I really need to know what the upgrade brings to the table. Do the new software version remove compatibility with old files or hardware, then I may want to resolve it before upgrading. Is there a all new UI, then I want to learn about it before launching my app. Is the frustrating bug I’ve been avoiding fixed, then I want to learn about it immediately so I can stop avoiding it. Did a upgrade introduce a bug then I would probably have no idea about the background upgrade at all (unless something dramatically changed) as I don’t keep checking the version number every time I launch my apps, filing a regression bug against it would probably be one of my very last thoughts. Etc. Most things with snaps are fantastic and snaps are making Linux a whole lot better in many areas, but the upgrade process needs to be at least as transparent to the user as non-snap upgrades. One of the main reasons I moved from Windows to Linux was because of suprice upgrades, introducing this again will force people like me to stay away from using snaps. I hope this makes you better understand one of the reasons to why some of us are (unwillingly) avoiding snaps.
1 Like
The epochs feature means that you’ll only be updated if the new snap is compatible with the old snap’s files (though this isn’t out yet), this should help with that concern (assuming snapcrafters use epochs properly).
I know this is a bit late in the process but I think our scenario might add to the discussion.
We run a SAAS system for a large number of clients. Each client has their own VM instance.
Many of our clients run a 24 x 7 operation.
Many of our clients are also hypersensitive to any system changes.
The result is that we need to have complete control over when any upgrades happen to a system (including any third party snaps that our stack depends on).
We also need to apply upgrades (such as emergancy patches) to specific systems as and when needed.
So whilst I understand the inclination to ensure that all snaps are always up to date, taking the control out of system administrators hands seems to be a mistake.
I completely agree that the default snap install should auto update.
This handles the most common case where the user either doesn’t know or doesn’t want to manage updates.
The second scenario where you can schedule updates (say micro updates) each Tuesday is also nice.
But I have to be honest I don’t see any way around a global dead switch for environments like ours that need to tightly manage upgrades.
I’m happy if the switch is hard to turn on and not obvious, but if I need the control then the control should be mine.
We can’t possibly predict every possible way that snaps are going to be used and the requirements for upgrade paths, so rather than dictating how things are we should provide great defaults but allow experienced people to manage their systems how they see fit.
5 Likes
If you are interested, we are happy to work with you to improve this picture even further.
Like it or not, there’re (super) users who have their own ideas about how should their devices be used, users who want to own their devices. All this political discussion could have been left out if you simply acknowledged that devices belong to the users. It’s not the team behind Snapcraft that should impose policy. If there’s some use case you didn’t anticipate, you shouldn’t be required to know. There’s cron and power users are a) capable of using it and b) understand risks of outdated software. You can keep automatic updates enabled for users you see as incompetent, but let the admins shoot themselves in the foot and don’t be evil please. We have plenty of overlords imposing restrictions on how we are supposed to use our devices. Snap is great, don’t be one of them.
2 Likes
@bsutton You have the control… what we’re encouraging is a limit on how far a system can remain stale, and giving a number of tools to help inside that window. We also have a number of features coming to improve further your control. If you have other ideas about how to control, or even just want to brainstorm, I’m happy to talk.
@marekhwd I’d be very happy to exchange ideas with those super users about how to improve the characteristics of software deployment and management. Ideally the conversation would be richer than simply “give me a global switch or you are evil”, though. We may eventually do exactly that, but right now we are interested in discussing and implementing ideas that will improve the world of software packaging.
We are not imposing a policy. We are making it very convenient to create your own policy that operates under a given threshold of staleness, with several tools already at your disposal and more on the way per notes above, and then we are indeed also making it inconvenient to adopt a policy that goes beyond that threshold. If you are a power user you know how to work around that inconvenience anyway, though.
We don’t see anyone as incompetent. What we see is actual data showing devices out of date. Many, for very long. We want to give a shot at improving the situation, because for once we can try to do that.
2 Likes
You’re making it inconvenient to the degree that users are supposed to install a firewall rule.
But you do see them as incompetent. You fail to see that there are users who may be interested in using snaps but are not at all interested in sharing their business decisions with you. You fail to acknowledge that running a periodic task is what every admin can do on her/his own. The kill switch requires zero effort. I could make the patch but I’m not interested in discussing politic merits of it with you. Stop trying to be Google/Apple please. Devices belongs to users and it’s not your business how they’re updated. Please keep the reasonable defaults but let the users decide.
3 Likes
So we should probably stop doing snaps altogether given that one of the main focuses when starting them was the desire to develop a package format that solves exactly this …
Software security can only be achieved through the ability to fix vulnerabilities, these in turn can only be fixed through updates of said software.
Mirai botnets are a reality and they exist due to non updated software …
Actual professional admins should be able to install a firewall rule, i would even expect them to actually script this to a stage where they have a txt file with a list of snaps they want to exclude from automatic updates.
…
Heck, i guess you could even create a snap that includes tinyproxy and ufw to do this for you through a “snap set” config option you hand to the snap …
Developing such a snap with the help of the forum might be a nice project … at least it would be more constrictive than insulting @niemeyer as incompetent …
Your arguments are very aggressive, confrontational, and judgemental. We cannot have a productive conversation without some minimum respect for the conversation and for the people involved on it.
@ogra Please do not embark on a rant. We are not here for that. Our time needs to be spent with people that are interested on the problem and willing to discuss ideas respectfully.
I don’t remember insulting anyone of incompetence. The requested feature is not to stop updating snaps automatically, but to give a sane way for minority of users to manage these updates, as they do it already with majority of other packaging software (apt, dnf, …). Firewall rule for Snapcraft isn’t a sane way, it’s just a workaround for a self-righteous policy.
There are snaps that run as root services, for instance LXC. I can’t see how can you see it as unreasonable to let the few admins decide when they want to update services that can basically own their devices. The decision to automatically update critical software should be left to the owners of devices and not on the team behind Snapcraft and software upstream.
2 Likes
Did you take a look yet at how snap confinement works ?
A confined snap running as root or not, can never own your system, beyond its own writable space in /var/snap and ~/snap (if running with user credentials) it can only access the parts of the system that you (the admin/owner) granted it access to via interfaces … (and this is even a lot stricter on Ubuntu Core devices than on classic installs)
The decision to update at a convenient time is still left to the admin, you just can not delay it indefinitely to not end up in an “android situation” …
Users who know what they’re doing should decide for themselves. Some users, including me, want to a) keep an older version of a program regardless of what upstream thinks I should be running, b) research an update of a program before it’s applied and there’re other use cases. Users who want this “feature” shouldn’t be forced to do it based on the Snapcraft policy, that is within period hardcoded in the snap tool. Again, no one here asked to change the defaults, but to opt out from forced updates.
1 Like
I feel this makes a workaround to the policy too prominent for Niemeyer to appreciate or even tolerate, regardless, @G.S.1 nice idea there? 
you will be pleased to hear that parallel installation of different versions of the same snap is being worked on as we speak 
if the maintainer of your snap is a responsible person she will release to the beta and candidate channel before moving the snap to stable … it is a single “snap switch” command to switch channels and test out if that snap suits you (or if not to contact the maintainer and report an issue) … you can easily do that while you delay the update for all the end users you are an admin for …
I’m trying to understand how to see if the snap refresh.timer changes actually stick. When I run sudo snap set core refresh.timer=sat,3:30, I still see:
admin@J82ZB02:~$ sudo systemctl list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Mon 2018-03-12 03:13:47 PDT 4 days left Wed 2018-03-07 13:01:42 PST 22min ago snapd.refresh.timer snapd.refresh.service