Change in logging behaviour on Ubuntu Core

Unfortunately this doesn’t work either because the journalctl in the series 16 core snap is accessing the journald files in the newer classic distro (in this case, 17.04) and it doesn’t understand them either. :\ Also, per https://www.freedesktop.org/wiki/Software/systemd/syslog/ you aren’t allowed to just connect to /run/systemd/journal/syslog and need integration with systemd that doesn’t currently exist in snapd.

This requires more investigation but I can say that making a snap that works on Ubuntu Core or a classic distro that uses a compatible systemd journal is not terribly difficult. Making that same snap work with a newer journald from a classic distro that is newer than the series of the core snap is problematic since the binary file format of the journal may be different (eg, from series 16 to Ubuntu 17.04).

Isn’t it just a matter of using the journalctl from the system? Also note the tool itself has provisions for tailing and filtering based on multiple factors.

One thing that came to mind is that on classic distro, rsyslog is likely already running. As such, an application could decide to use the journal or not based on if on classic distro. If not on classic distro, there is no discrepancy between the series and the log format, because the series should be the same on both. This probably won’t hold true with alternate core snaps (eg, Fedora), but there the snap should target the runtime and therefore understand the log format of that alternate core.

/bin/journalctl comes from the core snap on classic distro and it doesn’t understand the log format of the files in /run/log/journal/*. The security policy doesn’t allow executing /var/lib/snapd/hostfs/bin/journalctl, but if it did, then we have:

# /var/lib/snapd/hostfs/bin/journalctl --follow
/var/lib/snapd/hostfs/bin/journalctl: error while loading shared libraries: libsystemd-shared-232.so: cannot open shared object file: No such file or directory

so we’d then have to set LD_LIBRARY_PATH. This is possible. Eg:

With these rules:

/var/lib/snapd/hostfs/bin/journalctl ixr,
/var/lib/snapd/hostfs/lib/systemd/*.so* mr,

Can then run with:

$ LD_LIBRARY_PATH=/var/lib/snapd/hostfs/lib/systemd:$LD_LIBRARY_PATH /var/lib/snapd/hostfs/bin/journalctl --follow
...
May 18 10:12:41 iolanthe kernel: audit: type=1400 audit(1495120361.472:70824): apparmor="DENIED" operation="open" profile="snap.strict.sh" name="/etc/fstab" pid=8276 comm="cat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
...

I’ll prepare a PR for that because I think it is worthwhile regardless of anything related to rsyslog.

Note that with the above I can make snappy-debug itself work, this is a hurdle for people wanting to use log-observe since they have to know to use /var/lib/snapd/hostfs/bin/journalctl and libraries like python3-systemd still won’t work.

Here’s the initial customer feedback I received:

I always prefer to have a text log in the system due to:

  • Portability: Binary format is really a problem when you need to examine the log from a system which doesn’t have the systemd journald running. For example, sending the logs to a server and a developer trying to quickly examine the logs on a Windows system. Or, for troubleshooting , you need to boot off of a USB stick with a minimal linux environment ( without systemd ) in order to examine the logs on the drive.

  • Programmatic analysis of system logs to look for certain types of error or failures. With text logs, this can be done in an OS independent way.

I believe (just because of its foundational nature), at least, system logs should be in a format that can obtained and analyzed with minimum resistance.

Has anyone given thought about whether this could be accomplished in the short term via a build-time option vs. a core snap config option? I think forcing this change on customers via an update to an LTS release sets a bad precedent.

Also regarding the original arguments for making this change, while I agree that wear leveling is a valid use case, in theory log rotate should prevent system crashes due to out-of-control log files using all the disk space.

1 Like

https://github.com/snapcore/snapd/pull/3352

what would that be ? the initial change i was doing actually added a snap set/get option to dis/enable rsyslog completely either trhough the gadget or via the above set/get, how would a “build time option” look like here ?

yes, size is definitely not an issue we have (and keep) logrotate around to pervent any files from running over …
we do have a customer with tens of thousands of boards where a significant percentage needs fresh SD cards regulary because logging wore them out. this real life issue caused the whole change we are discussing now.

1 Like

FYI, I put forth the idea of shipping rsyslog disabled in the other thread. In this manner, SSD wear is addressed, sites can opt into text only logs and people could opt into remote logging (both via core config). A gadget could then update core config. The response (please correct me if I mispoke) was that all of this could be achieved with an rsyslog snap-- you preinstall rsyslog and configure the snap via the gadget.

That’s somewhat awkward. We should think through this and see if we need to split out these cases.

Fair points.

What’s the practical difference? The core option may be set in the gadget, assuming that’s the direction we go.

I’d misunderstood an earlier comment you made and assumed that this change couldn’t be accomplished via a core snap config item:

So if it’s possible to control which logging system is used via a core snap config item, then I’m +1, given that we don’t make journald the default value. What I’m opposed to is removing an existing feature/mechanism via an update, such that existing customers have to do extra work to restore it.

https://github.com/snapcore/core/pull/21 was the initial approach … resulting in a:

snap set/get service.rsyslog.disable=true|false

config option … i can clean it up (drop timesyncd) and re-propose it if wanted …

1 Like

Yes, given the feedback above and the previous discussions, it seems safe and sane to simply allow disabling the service conditionally. We can reevaluate that on 18.04 or once we have multiple bases.

Thanks @ogra.

FYI, there is a snappy-debug in edge that will use journalctl, but it has some issues I was unable to fix in time before going on holiday so I didn’t push it to stable. It will work on Ubuntu Core though and will prefer journalctl over syslog there, so if people need it:

$ sudo snap install snappy-debug --edge

or if upgrading:

$ sudo snap refresh snappy-debug --edge
1 Like

i have seeded rsyslog back into the edge image and have a matching PR at https://github.com/snapcore/core/pull/45

people wanting to make use of it need to set the option from their gadget…

which in 2.27 should work using the config defaults functionality there

1 Like

so following Gadget snaps
this would then look like:

defaults:
  99T7MUlRhtI3U0QFgl5mXXESAiSwt776:
    service:
      rsyslog:
        disabled: true

if i understand the doc correctly (99T7MUlRhtI3U0QFgl5mXXESAiSwt776 is the snap-id for core)
is that right ?

yes, that’s snap-id is correct

A post was split to a new topic: Status of logging in Ubuntu Core

@ogra - In putting an UC18 system into a production environment that uses remote logging, it seems that this has never happened. I understand there is an effort to make systemd-journal-remote work as a snap, but regardless of that, many sites standardize on syslog as the central log server, and core18 doesn’t provide any facilities for this natively and there doesn’t appear to be any snaps to achieve this. Is this work still planned for an rsyslog snap or a snap that is simply designed to forward logs (eg, perhaps with snap set)?

rsyslog is completely gone from core18 so a snap set command wouldnt be possible anymore, but we have two new snaps that at least allow central logging, despite not being rsyslog compatible. The setup is described at: