Disabling automatic refresh for snap from store

backlog

#8

You definitely aren’t alone in wanting this. We’ve had several users of the Rocket.Chat snap ask for this as well. They are just simply used to being able to say Tuesday is update day. Or be able to schedule a down time to alert their users.

I definitely understand where the snap team is coming from though. This is one of major benefits of snaps. The fact that they will be updated. That we as snap publishers can push out a critical vulnerability patch to our users and not have them ignore it. Of course the users get that same peace of mind.

I think if we had mechanisms in place to bring up the new version of the snap and do a more seamless hand off this worry of interruption / downtime would lesson.

I think there are probably several issues here. First by default snaps kill and unmount the previous snap version before mounting and then starting the new version. Secondly… if a network service. You couldn’t even start the second service because you would have a bind conflict.

Anyways… so far i’ve been just telling users that want this level of control to go the manual route because right now snaps aren’t designed for that scenario.


#9

Didn’t you try to block Canonical and Ubuntu domains inside your router with iptables? :laughing: :laughing: :laughing:
You are a network guy, you should try.
Write a script on the desktop which connects with ssh to your box and executes smth like:

sudo iptables -A OUTPUT -d search.apps.ubuntu.com -j DROP

And you’re free till the next reboot.


#10

Denis,

Me (personally) could do this. But then I rather dislike doing any [additional] filtering on a router as this could cause issues in the future… It also blocks other snap’s and potential more (not sure what all would be at the same location)
Another (in my opinion simpler) solution is to download the snap and install it locally - not from store.

But what I’m looking for is a solution for the USER of my snap. Something which is kind of automatic or can be set by the snap maintainer to accommodate a different update policy. Something which would probably download a new snap and give a warning/notice of a new snap available, but does NOT install it until a “ok” is given by the user.

We are currently getting close to release a new major version, but I haven’t updated the snap in weeks as I know I’ll cause network outages at various places just by pushing a new snap into the store.
My current (unfortunate) way is to just NOT push any new versions to the snap store until this is resolved or there
is a major security issue I need to address (but not for new major versions)


#11

Yesterday we discussed once more this topic, and after exchanges with several stakeholders there was agreement to increase the allowed window in which refreshes may be scheduled.

The agreed semantics to be implemented are the following:

  • Refreshes may be scheduled at an arbitrary weekday and time within the month (e.g. second Tuesday between 1pm and 2pm).
  • Refreshes may be deferred for up to another month so that missed windows and re-scheduling may happen without strange side effects. For example, if it was scheduled for the first day, and then gets scheduled for the end of the month just before it happens, there may effectively be a two months window without refreshes.
  • If the system remains out-of-date after the two months window, the system will start attempting to refresh out of the window.
  • That maximum window is reset every time the system is refreshed, so out-of-band updates may performed at a convenient maintenance window.

These changes should greatly improve the behavior when using third-party snaps in servers, while not giving up on the goal of encouraging systems to remain up-to-date and secured.

Please let us know if you have any comments on the topic or the proposed changes.


#12

Thanks, this is VERY WELCOME news.

So for my case (FRR snap, which is a routing daemon):
I think the timeframe is ok, still not sure how network admins react to automatic refreshes if they miss the window. I understand the desire to get people to update more frequent, but the forced push is painful. Definitely much better, but it may take some time to push network admins to a more constant upgrade cycle.

Related question:

What happens if I miss a bug? Ie, I push a new snap out, someone tries to upgrade a system and it fails for their setup. How would this be solved?

  • Can they block automatic upgrades to the version which they know has a bug affecting them?

  • If they report it to me, can I retract the bad version from the store (I may not be able to update it fast enough, so I would prefer to delete it and stop anyone else going to new version until this is fixed)?

  • If I can do this (I assume I can), what happens to other users which are already upgraded to new versions? Do they get forced to downgrade again? Or would they be able to stay on the removed version? (I assume bug might not
    affect all users, so most of them might want to stay)


#13

One more comment:

I would prefer to just ignore the DAY part, but keep the time of day. So if someone specified second Tuesday between 1pm and 2pm, and remains out-of-date, then drop the Day, but still try to stick to the hour part.

  • Martin

#14

I’d still quite like a global switch on the user side, I’m on less than 1mbps Internet (yes - megabits, not megabytes; UK Government failed to fulfil their pledge to have minimum 2mbps everywhere by Dec 2016) and automatic updates take up bandwidth and make the Internet unusable for my house, so I have to kill the snapd process… I can disable Apt updates from Software & Updates, graphically, but I can’t disable snap updates at all, graphically or via the command-line…

This is probably a separate issue sorry, has got a thread here


#15

Would allowing the publisher to define an update schedule work? The publisher could specify when creating the app that it is to stay updated automatically, the user can specify within certain windows, or the publisher has direct control over each snaps update.

To give more background. I’m working on a hardware device that will operate in a customer’s environment. My end user doesn’t have a terminal interface to change their update window, I won’t have physical access to a device in the field, I need to ensure a number of things, life the device isn’t moving or in other forms of operation when the device is updated.

I think in the IoT environment something more granular is needed, Ideally my snap needs to be able to signal when it is in a state it can be updated. Being able to access the update window or command for my snap from within the snap would be a good start. Having he ability to download the snap and then trigger when the updates occurs would be great.


#16

Great news!

I propose here one more additional option:

Devices download all updates for snaps:
a) every time they become available, or
b) following you new logic described in your post

But the real update gets deferred till the next reboot. It mean that upon next power cycle when Ubuntu Core boots it can switch snaps to newer ones. And certainly there shouldn’t be a forced reboot like when the core snap updates nowadays.
Snapd can throw a warning upon login to ssh that updates can be finalized. You type smth like “snap update” and there you go.

This gives a possibility to have devices with potentially infinite uptime. But still a user can update device unplugging it and then plugging it back when he wants. This solves issues with headless devices, devices with no user access to ssh, or just when a user doesn’t even know that there is Linux inside.

Generally speaking Canonical should decide what it whats Ubuntu Core to be: a platform to create toys for children or a platform for real stuff (industrial, medical, automotive). I can’t imagine an autopilot saying on a highway at 200km/h: “Hey there! We haven’t updated the system for two months, you deferred it 10 times and I give no more chance! I’m gonna download the core snap now over 4G, update it and reboot your vehicle. You’d better pray”.


#17

We do have customers using Ubuntu Core with digital signage … that means that a supermarket with 20000 products will have 20000 display devices running on Ubuntu Core showing the price tags on the shelves… would you really want that supermarket to hire someone who logs in once a month to check if there are updates ?

Same goes for whatever sensor and monitoring devices you have running in the woods, miles away from any admin…

Ubuntu Core is designed for completely unattended operation, that means that updates and reboots need to happen fully automatic …

That said, there undoubtly need to be ways to allow apps to delay an update, as well as there need to be ways to declare to never update on 4G if there is a possibility to reach WLAN when the autopilot sits in the garage, but these are special cases that should be handled via additional options …

What @niemeyer described above should be the out-of-the-box behaviour for bare images to keep them secure, if you build something specific on top that has additional needs (like an app delaying updates even more or some such), extra features can definitely be added.


#18

No, I don’t say so. That’s why I wrote about a user doesn’t even know that there is Linux inside.

Actually you start by defining a use case for a device. And based on that you configure an image to follow logic 1 or 2 or 3. For a supermarket it’s obvious that option with forced updates should be chosen.

I described a device that requires infinite uptime. It’s just another class of things in the world. Do you want to cover them adding option 3? Or ignore them? That’s the question. If it consists of creating a dummy snap that defers updates in an infinite loop forever, well it seems ugly. There should be a better way.

As an example you can take any network device, a router or a switch. It has no rights to reboot when it wants. Sometimes it’s required to have 99.999% availability that is 300 seconds of down time a year. This may only be sufficient to swap devices when hardware fails.

So for those cases I suppose a simple power cycle is a solution. Updates are downloaded every time a new version comes out. And a much newer version overwrites previous one if it wasn’t updated. And then these updates are applied on the next boot. That’s all. Completely unattended as you said. There is absolutely no need to maintain those devices.


#19

@mwinter Thanks for the feedback. It’s good to hear these ideas make some sense out of our echo chamber as well.

Responding to the questions:

That happens transparently when one does snap revert. The revision reverted out of is blacklisted and will not be reinstalled. When a new revision shows up inside the scheduled window, then that is installed and the problematic revision is "jumped over’.

Yes, you don’t even need to take it out: just publish another revision (new or old) and that will be the new tip. Soon we’ll also introduce live feedback over how well the roll out is taking place, and even block it automatically if there are strong hints that the new revision is disrupting clients.

They’ll refresh into the new tip. Whatever you release into a channel becomes the current target for every client.

The scheduling logic allows selecting multiple slots, so you can put in every slot that makes sense and the system will strictly respect those over the two months time frame. If after that none of the slots managed to be respected and there are updates pending, the system will disregard the slots as they’re not being exercised for some reason. There might be clock issues, for example, or perhaps the scheduling window falls overnight on a machine that is never turned on during that time.

That said, phasing out the requested window with less strict windows before going totally open is an interesting idea.


#20

@denis There’s a rich spectrum of real applications and each of them with different requirements. Many industrial devices can easily be updated and rebooted at predictable times, and some toys will in fact require strictly predictable refresh routines. In some appliances updating at reboot can make sense, in others it’s better to force a reboot and ensure safety sooner.

We’re gradually taking steps towards supporting every one of those cases.


#21

I know core images support having ssh turned disabled, is there a way within a snap to have a customer specify when these updates would be able to occur through a management interface. The snap would need a way to trigger the update itself in this instance.

Also in the case of safety concerns is the device is currently moving, my motor control service can’t be shut down. It needs a mechanism to block an update while in motion or during some other critical activity.


#22

Maybe there could have a snap auto-update disable and to update just snap refresh as always


#23

Yeah, that’s exactly the aspect that the conversation above addresses.


#24

What if I’m a desktop user who just wants to feel like I have control over my computer?

I’ve been using Arch for over a year, and I was used to updating at least once a week on that. I recently switched to KDE Neon because Arch lacks debug packages, and of course now I’m stuck with year-old versions of all non-KDE apps because I’m now using Ubuntu 16.04 repositories. One possible way to get modern versions of some apps is by installing their Snaps, but I absolutely refuse to use a packaging system that forces me to update, even though I fully plan to update anyway.

If I wanted my only options to be “update now” and “defer update,” I’d be using Windows 10. I use Linux because I want to have control over my computer, and it’s frankly insulting if anyone thinks they have to force updates on me because they don’t trust me to update my computer myself. I’d like the ability to disable automatic updates simply for the principle of the thing, and I’m sure there are others like me.

From what little information I’ve found, it looks like we either need to accept that developers know best and conform to mandatory updates or use a different packaging system. That seems like a closed-minded attitude toward end users. I can understand having auto-updates as the default behavior, but not having the ability to opt out when it’s clearly technically possible really puts me off of the whole project. Would it really be so bad to allow power users to choose when to update and still take advantage of everything else Snaps have to offer?


#25

Hi Jacob,

You’re clearly a bit frustrated about the topic, and as a consequence word the problem in a confrontational way which creates an us vs. them situation that detracts from the real point. As mentioned above, the issue that makes us resist the idea of simply disabling updates altogether is that very often it will mean never to update rather than update at someone’s discretion, and then we’re getting back to some of the problems that got us here in the first place. That’s why we’ve been resisting introducing that global switch, at least for the time being, and instead working with people to mitigate the undesired side effects of having automatic updates enabled.

On that route, a number of features have been created and others are on the pipeline, such as the transactional operation foundation, multi-revision installations which keeps N revisions around for safety, pretty much instantaneous reverts across these, tracks which enable software publishers to only force updates on critical issues, epochs which introduce stepped upgrades, carefully scheduled updates, health checks which allow ensuring that a revision is safe to update to before it is completed, and so on.

That’s the attitude being taken by this project, and that is not towards you or anyone else. This attitude is towards the problem of enabling systems to be up-to-date and safe. We may very well introduce a trivial global switch at some point, but right now the priority is in working with the community on the low sides of automatic updates, and the examples above are very concrete results of that.

Sounds like this is a problem you’re also very interested on, so your contributions and opinions around improvements would be appreciated. We’ll also understand if your personal priorities and preferences take you elsewhere, though.


#26

Just to add my $0.02 here because it can be difficult to hold the line in situations like this… Even though it’s uncomfortable for us to have auto-updates enabled without an off switch, I believe it’s the right way to go, for servers and embedded devices in particular, or anything headless - it’s the only way we’ll keep them secure over time.

One additional concern, that might not have been covered explicitly above - is around data usage on cellular connections. Many connected devices are on cellular plans with 100MB/mth data allocation.

Having some way to opt out of a very frequent release cadence (especially for large snaps) would be important for these devices.

Maybe a new ‘critical’ track that developers would only release to if a serious issue was identified? e.g. to prevent DDoS


#27

A lot of this sounds great. If possible could we get some more details on the health checks and carefully scheduled updates?

To add another case, this might fall under carefully scheduled updates, is it possible to ensure that a customer has all the same versions at a site? This might look like all devices signaling that they are ready for update before a simultaneous update. I could see a snap attempting to trigger them all by the core api for a refresh at the same time, but that wouldn’t ensure one did not update earlier.