The logsync-james snap has been flagged for manual review as it uses the system-files interface. This is intended as an example to show brand store customers how they could sync systemd logs to a remote log server, so it would be helpful to have this available in the store.
1 Like
Looking at github, the current snapcraft.yaml uses:
plugs:
config-logsync:
interface: system-files
read:
- /etc/systemd
write:
- /etc/systemd/journal-upload.conf
Read access to /etc/systemd was not included as part of this request and provide quite a bit of additional information to the snap since system-files grants read access to the directory and everything under it.
Can you describe how this works? This seems like something that should instead be a snap set option on the core snap. cc @pedronis
This uses a built-in feature of systemd to allow logs to be sent to a remote systemd log service. The journal-upload.conf has the URL of the remote service and can also define the location of TLS certs to allow the connection to be over HTTPS.
@ogra has modified the approach in the snap so we don’t need full read access to /etc/systemd. Does that change your view on the snap?
plugs:
config-logsync:
interface: system-files
write:
- /etc/systemd/journal-upload.conf
I can request this in a separate forum post as a suggestion for snap set as well.
@jamesj Since this snap provides the systemd-journal-remote package internally via stage-packages, I wonder why it wants to use the host’s /etc/systemd/journal-upload.conf instead of it’s own private version of this file? By using the one on the host would this conflict on a classic system if systemd-journal-remote was also installed? Would it be better to have this config file for the snap live in $SNAP_DATA or similar, then there would be no need for a system-files interface?
because systemd has no way to override this without a very complex re-build and patching, but this means we can not use the existing maintained package from the archive (which we would very much like to do for security and maintenance reasons), note that /etc/systemd/journal-upload.conf typically does not exist on hosts unless you manually created it and installed the systemd-journal-remote package (which is not installed by default)
This seems reasonable for the snap, but in Change in logging behaviour on Ubuntu Core there was discussion that core itself might ship systemd-journal-remote itself and allowing the use of system-files as specified for this snap might break that functionality. @pedronis can you comment? This could potentially be avoided by disallowing the snap write access to /etc/systemd/journal-upload.conf and having the user have to write out /etc/systemd/journal-upload.conf for the snap to read (at least then, it is an overt administrative action).
Also, I’m not familiar with systemd-journal-remote specifically, but can say that journald’s binary log format is not compatible with other versions of journald (eg, a core 16 journalctl can’t read core 18’s binary log format). Does systemd-journal-remote not suffer from these problems? If so, can you detail how it will behave correctly and robustly when, say, an Ubuntu Core 18 system uses this snap to send logs to a Debian system? An Ubuntu 16.04 LTS system? An Ubuntu 20.04 LTS system? A RHEL system?
Similarly, this snap will be installable anywhere-- how will its systemd-journal-remote work with the binary log format on the host for other versions of systemd?
I’m no expert in this, so maybe @ogra can chime in if I get anything wrong. There is a binary format on disk, but the network transfer format is text-based and can even be JSON. Some details are here: https://www.freedesktop.org/wiki/Software/systemd/export/. So I imagine that it would be transferable to non-Ubuntu environments.
when using the receiver snap (that i was waiting with the upload for this request to be solved first) from
i can read the received logs just fine on xenial and bionic machines with the respecitive local journalctl pointing to the dir the logsync-receiver stores the logs … (i havent tested with other (newer) versions but it doesnt look like the different journalctl versions are a problem between xenial and bionic at least)
Thanks for the additional info, but what about this last question? Again, I’m not very familiar with systemd-journal-remote, does it need to read the binary format from the host to send it off or is it supplied the logs via some other means? In either case, can the snap read the logs to ship off on various hosts?
systemd-journal-remote is essentially the same as journalctl --output=export piped into a http/https stream, i.e. a text based format, not binary …
the re-assembly into binary data is depending on what you run on the receiving side …
here are two example lines to show what is actually transferred (you could actually use a script to manually assemble syslog lines from this and not involve journalctl at all on the receiver side):
__CURSOR=s=4559cb9ceb064f3a92cd33efaaa8fc9a;i=3;b=d02da647a9c04f8683e593fdc2a40b30;m=1d8dca;t=598df47090c83;x=8
__REALTIME_TIMESTAMP=1575459620457603
__MONOTONIC_TIMESTAMP=1936842
_BOOT_ID=d02da647a9c04f8683e593fdc2a40b30
_MACHINE_ID=9d83d97c30d14e73ae6271783c036759
_HOSTNAME=acheron
_SOURCE_MONOTONIC_TIMESTAMP=0
_TRANSPORT=kernel
SYSLOG_FACILITY=0
SYSLOG_IDENTIFIER=kernel
PRIORITY=5
MESSAGE=Linux version 4.15.0-72-generic (buildd@lcy01-amd64-023) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubu
__CURSOR=s=4559cb9ceb064f3a92cd33efaaa8fc9a;i=4;b=d02da647a9c04f8683e593fdc2a40b30;m=1d8dd1;t=598df47090c89;x=c
__REALTIME_TIMESTAMP=1575459620457609
__MONOTONIC_TIMESTAMP=1936849
_BOOT_ID=d02da647a9c04f8683e593fdc2a40b30
PRIORITY=6
_MACHINE_ID=9d83d97c30d14e73ae6271783c036759
_HOSTNAME=acheron
_SOURCE_MONOTONIC_TIMESTAMP=0
_TRANSPORT=kernel
SYSLOG_FACILITY=0
SYSLOG_IDENTIFIER=kernel
MESSAGE=Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-72-generic root=UUID=5cb98ba3-8583-42fb-90c0-e7d251ab7815
locally it just queries systemd for log events as I understand it, this should also not be binary and use a standardized API
From what I could see, systemd-journal-remote is just acting as a pipe between systemd and the external service. It’s just taking what it’s given and sending it to the external service, rather than doing any translation.
Thank you for the additional information. This seems to be sufficiently robust.
yes, it will take me a bit to form a comment though, I wasn’t involved in those discussions originally
note that none of the interface stuff would be necessary if /etc/systemd/journal-upload.conf could be re-mapped inside the snap via a layout (which is forbidden due to its location). it would also not be necessary to worry if the file exists on the host already in this case …
i wonder if the layouts code couldnt just whitelist this particular file for re-mapping and we could forget about the interface at all …
Having the remote journal export configuration as an option in snapd may be a better long-term approach. This snap may help to fill a gap in the short-term.
Well, adding the binary to core/core18/core20 and exposing the config via snap set should be able to go fast once people have decided that is the direction (it after all wouldn’t be tied to a snapd release).
well, given the fact that you need to import a bunch of certificates to gain https forwarding, i fear it wont be that fast.
plain http is surely quick though (just an on/off toggle in the config), but for https we need a configuration tool that imports the certs into the system.
I’m thinking about this as basically a request for remote syslog support, and traditional remote syslogging does not use certificates, so, IME, adding plain immediately and then adding certificate support later seems ‘ok’. This is just my opinion of course and will defer to @pedronis and the snapd team’s plans in this area.
Wouldn’t something like this side-step the issues for a while:
layout:
/etc/systemd/journal-upload.conf:
bind-file: $SNAP_DATA/etc/journal-upload.conf
(that’s one the original use cases for layouts)
Then the snap doesn’t need afaict system-files and it will have “snap set” ways to write the config.