Allow disabling system services on Ubuntu Core

[niemeyer]

If we’re disabling syslog by default, it’d be better to ship it as a snap instead of having it in core.

I’m sure tlsdate also has dozens of issues, such as the fact you need to trust the server to have a proper clock time in its handshake, and the fact internal infrastructure will not use TLS at all. Someone that has more experience than you and me in the real world of clock syncs needs to chime in on what’s the best NTP approach here.

[ogra]

If we’re disabling syslog by default, it’d be better to ship it as a snap instead of having it in core.

ok, i guess then we need to at least document that you need to create /var/log/journal to get logs across reboots (else the ringbuffer gets flushed)

[zyga]

ok, i guess then we need to at least document that you need to create /var/log/journal to get logs across reboots (else the ringbuffer gets flushed)

Sounds like a core config option “persistent journal” to me.

[ogra]

there is another issue here … i just checked, what is written into /var/log/journal is a binary blob, not a text file, this will make debugging a lot harder when you ask people to attach their logs to bugs …

[zyga]

This is a well-known behaviour. journalctl has options to pick the right log file fragment.

[ogra]

the question is if we as developers are ok with having to download the whole file locally and process it instead of being able to just click on the syslog text file in a bug to see it in the browser
i personally find that rather disturbing (but can probably adapt to it if i have to)

[niemeyer]

Instead of asking for a file, ask for a command. It can’t be simpler, really.

[renat2017]

Hi, guys, It’s Renat from Screenly.

It’s be nice to have such an option to dislable syslogd to avoid SD card degradation on RaspberryPI.

[ogra]

well, the plan is now to not ship rsyslogd at all (and simply rely on journald which does not write to disk) so you dont really need to disable anything

i got the change ready but @mvo wanted to give the idea another review, i’m waiting for feedback before landing the change.

[jdstrand]

It’s be nice to have such an option to dislable syslogd to avoid SD card degradation on RaspberryPI.

Mostly off topic for the thread-- I know a lot of people are using the classic snap to build armhf binaries on pi2/pi3 and when doing that the logging of ALLOWED messages was ridiculous. The ‘classic-support’ interface was created to address this but the classic snap was only updated very recently to plug the interface but now things are much quieter when using the classic snap.

[jdstrand]

FYI, snappy-debug currently relies on standard syslog and doesn’t have support for journald. Before rsyslog is dropped, snappy-debug should probably gain this support. We would also have to change any documentation that references looking at /var/log/… if moving away from rsyslog.

As @ogra mentioned, remote logging is important for certain use cases. Turning rsyslog off by default seems fine so long as we can turn it back on (indeed, snappy-debug could (ask the user to) do this in the short term if needed). I’m ambivalent about moving it to a snap, but if we do, can we make sure that it is functional before removing it from core?

[ogra]

i’ll look into if it is possible to have an rsyslogd snap

[jdstrand]

Another consideration is what snaps are going to break if rsyslog is gone and nothing is being logged to /var/log/syslog and friends. One way to understand this is looking at all the snaps that plugs ‘log-observe’.

Also, if rsyslog is removed then the log-observe interface should probably be redesigned. Some things to think about:

• on core, the implicit log-observe interface should only allow reading the journal and the log files in /var/log that exist in core
• on classic, the implicit (classic) interface should allow reading the journal and everything in /var/log
• where do app slot implementations write their logs? in SNAP_COMMON? /var/log? how do we expose these logs to other snaps? (in particular, if slot writes to /var/log, how do we reconcile implicit policy from the core snap and policy for the slot implementation? All within ‘log-observe’? Make app slot implementation policy in another interface (eg, log-support)?)
• how to migrate now broken snaps that need the new interface/snap/…

I’m starting to wonder if perhaps it would be better to leave rsyslog in core but disabled for series 16 and removing rsyslog for series 18.

[niemeyer]

What’s the real difference between having a disabled rsyslog and a missing one?

[ogra]

note that there was a similar discussion about dropping rsyslogd from the main distro as well at the mailing list which seems to come to the conclusion that rsyslog is really only needed for remote logging to another syslog instance (there is no actual action item coming out of that discussion sadly)

[jdstrand]

The way I was thinking about it was, a disabled syslogger is still part of core and enable-able, therefore the interfaces don’t need to change, the system need only be configured to enable traditional log file locations. System builders can set the core option and get what they need, and interested snaps could be adjusted to alert that it needs to be enabled. A mechanism could be provided to allow other snaps to enable core logging setting.

If there is no logger like rsyslog in core, then the interfaces probably do need to change because the log files in question are not provided by core (but are on classic). On core, the log files would instead be provided by another snap (eg, the rsyslog snap), so we’d need to think about if that requires more changes to log-observe or a new interface. System builders could preinstall this snap and snaps could be adjusted to alert that another snap needs to be installed. Depending on where those log files live, those snaps may need code changes for the new location. As things are currently implemented, there is no possibility for a mechanism for snaps to enable traditional logging because there is nothing to enable.

[jdstrand]

Disabling or removing, knowing what snaps plug log-observe is important to know what may break immediately. IIRC, there are commercial projects that expect the traditional locations.

[pedronis]

@jdstrand as far as I can tell we have 3 snaps with granted log-observe, one is snappy-debug, one is for screenly (which is asking for the disabling) and then there’s another one

[ogra]

what is the “other one” ?

[niemeyer]

The fact we won’t be shipping logs into the same location won’t break any software. They will stop observing new logs on disk, which is the obvious and intended consequence of disabling or removing such a service. We should of course publicly inform people that this is being done and why (in addition to this thread), but I see that change as a very positive one considering it will prevent wearing solid state devices out, and will also prevent unattended systems from crashing due to disks full of logs. That Ubuntu thread that reached an agreement on doing this exact change also seems to back this as a good choice. The point made above about there being almost no users of this interface is an excellent data point as well.

So, my vote for us to move on and remove rsyslog from core altogether, and then start working on a syslog snap, preferably together with someone that in fact depends on this feature instead of someone that would just develop it on a theoretical basis.