Unfortunately this doesn’t work either because the journalctl in the series 16 core snap is accessing the journald files in the newer classic distro (in this case, 17.04) and it doesn’t understand them either. :\ Also, per https://www.freedesktop.org/wiki/Software/systemd/syslog/ you aren’t allowed to just connect to /run/systemd/journal/syslog and need integration with systemd that doesn’t currently exist in snapd.
This requires more investigation but I can say that making a snap that works on Ubuntu Core or a classic distro that uses a compatible systemd journal is not terribly difficult. Making that same snap work with a newer journald from a classic distro that is newer than the series of the core snap is problematic since the binary file format of the journal may be different (eg, from series 16 to Ubuntu 17.04).
Isn’t it just a matter of using the journalctl from the system? Also note the tool itself has provisions for tailing and filtering based on multiple factors.
One thing that came to mind is that on classic distro, rsyslog is likely already running. As such, an application could decide to use the journal or not based on if on classic distro. If not on classic distro, there is no discrepancy between the series and the log format, because the series should be the same on both. This probably won’t hold true with alternate core snaps (eg, Fedora), but there the snap should target the runtime and therefore understand the log format of that alternate core.
/bin/journalctl comes from the core snap on classic distro and it doesn’t understand the log format of the files in /run/log/journal/*. The security policy doesn’t allow executing /var/lib/snapd/hostfs/bin/journalctl, but if it did, then we have:
# /var/lib/snapd/hostfs/bin/journalctl --follow
/var/lib/snapd/hostfs/bin/journalctl: error while loading shared libraries: libsystemd-shared-232.so: cannot open shared object file: No such file or directory
so we’d then have to set LD_LIBRARY_PATH. This is possible. Eg:
I’ll prepare a PR for that because I think it is worthwhile regardless of anything related to rsyslog.
Note that with the above I can make snappy-debug itself work, this is a hurdle for people wanting to use log-observe since they have to know to use /var/lib/snapd/hostfs/bin/journalctl and libraries like python3-systemd still won’t work.
I always prefer to have a text log in the system due to:
Portability: Binary format is really a problem when you need to examine the log from a system which doesn’t have the systemd journald running. For example, sending the logs to a server and a developer trying to quickly examine the logs on a Windows system. Or, for troubleshooting , you need to boot off of a USB stick with a minimal linux environment ( without systemd ) in order to examine the logs on the drive.
Programmatic analysis of system logs to look for certain types of error or failures. With text logs, this can be done in an OS independent way.
I believe (just because of its foundational nature), at least, system logs should be in a format that can obtained and analyzed with minimum resistance.
Has anyone given thought about whether this could be accomplished in the short term via a build-time option vs. a core snap config option? I think forcing this change on customers via an update to an LTS release sets a bad precedent.
Also regarding the original arguments for making this change, while I agree that wear leveling is a valid use case, in theory log rotate should prevent system crashes due to out-of-control log files using all the disk space.
what would that be ? the initial change i was doing actually added a snap set/get option to dis/enable rsyslog completely either trhough the gadget or via the above set/get, how would a “build time option” look like here ?
yes, size is definitely not an issue we have (and keep) logrotate around to pervent any files from running over …
we do have a customer with tens of thousands of boards where a significant percentage needs fresh SD cards regulary because logging wore them out. this real life issue caused the whole change we are discussing now.
FYI, I put forth the idea of shipping rsyslog disabled in the other thread. In this manner, SSD wear is addressed, sites can opt into text only logs and people could opt into remote logging (both via core config). A gadget could then update core config. The response (please correct me if I mispoke) was that all of this could be achieved with an rsyslog snap-- you preinstall rsyslog and configure the snap via the gadget.
I’d misunderstood an earlier comment you made and assumed that this change couldn’t be accomplished via a core snap config item:
So if it’s possible to control which logging system is used via a core snap config item, then I’m +1, given that we don’t make journald the default value. What I’m opposed to is removing an existing feature/mechanism via an update, such that existing customers have to do extra work to restore it.
Yes, given the feedback above and the previous discussions, it seems safe and sane to simply allow disabling the service conditionally. We can reevaluate that on 18.04 or once we have multiple bases.
FYI, there is a snappy-debug in edge that will use journalctl, but it has some issues I was unable to fix in time before going on holiday so I didn’t push it to stable. It will work on Ubuntu Core though and will prefer journalctl over syslog there, so if people need it:
@ogra - In putting an UC18 system into a production environment that uses remote logging, it seems that this has never happened. I understand there is an effort to make systemd-journal-remote work as a snap, but regardless of that, many sites standardize on syslog as the central log server, and core18 doesn’t provide any facilities for this natively and there doesn’t appear to be any snaps to achieve this. Is this work still planned for an rsyslog snap or a snap that is simply designed to forward logs (eg, perhaps with snap set)?
rsyslog is completely gone from core18 so a snap set command wouldnt be possible anymore, but we have two new snaps that at least allow central logging, despite not being rsyslog compatible. The setup is described at: