because systemd has no way to override this without a very complex re-build and patching, but this means we can not use the existing maintained package from the archive (which we would very much like to do for security and maintenance reasons), note that /etc/systemd/journal-upload.conf typically does not exist on hosts unless you manually created it and installed the systemd-journal-remote package (which is not installed by default)
This seems reasonable for the snap, but in Change in logging behaviour on Ubuntu Core there was discussion that core itself might ship
systemd-journal-remote itself and allowing the use of system-files as specified for this snap might break that functionality. @pedronis can you comment? This could potentially be avoided by disallowing the snap write access to
/etc/systemd/journal-upload.conf and having the user have to write out
/etc/systemd/journal-upload.conf for the snap to read (at least then, it is an overt administrative action).
Also, I’m not familiar with
systemd-journal-remote specifically, but can say that journald’s binary log format is not compatible with other versions of journald (eg, a core 16 journalctl can’t read core 18’s binary log format). Does
systemd-journal-remote not suffer from these problems? If so, can you detail how it will behave correctly and robustly when, say, an Ubuntu Core 18 system uses this snap to send logs to a Debian system? An Ubuntu 16.04 LTS system? An Ubuntu 20.04 LTS system? A RHEL system?
Similarly, this snap will be installable anywhere-- how will its
systemd-journal-remote work with the binary log format on the host for other versions of systemd?
I’m no expert in this, so maybe @ogra can chime in if I get anything wrong. There is a binary format on disk, but the network transfer format is text-based and can even be JSON. Some details are here: https://www.freedesktop.org/wiki/Software/systemd/export/. So I imagine that it would be transferable to non-Ubuntu environments.
when using the receiver snap (that i was waiting with the upload for this request to be solved first) from
i can read the received logs just fine on xenial and bionic machines with the respecitive local journalctl pointing to the dir the logsync-receiver stores the logs … (i havent tested with other (newer) versions but it doesnt look like the different journalctl versions are a problem between xenial and bionic at least)
Thanks for the additional info, but what about this last question? Again, I’m not very familiar with
systemd-journal-remote, does it need to read the binary format from the host to send it off or is it supplied the logs via some other means? In either case, can the snap read the logs to ship off on various hosts?
systemd-journal-remote is essentially the same as
journalctl --output=export piped into a http/https stream, i.e. a text based format, not binary …
the re-assembly into binary data is depending on what you run on the receiving side …
here are two example lines to show what is actually transferred (you could actually use a script to manually assemble syslog lines from this and not involve journalctl at all on the receiver side):
__CURSOR=s=4559cb9ceb064f3a92cd33efaaa8fc9a;i=3;b=d02da647a9c04f8683e593fdc2a40b30;m=1d8dca;t=598df47090c83;x=8 __REALTIME_TIMESTAMP=1575459620457603 __MONOTONIC_TIMESTAMP=1936842 _BOOT_ID=d02da647a9c04f8683e593fdc2a40b30 _MACHINE_ID=9d83d97c30d14e73ae6271783c036759 _HOSTNAME=acheron _SOURCE_MONOTONIC_TIMESTAMP=0 _TRANSPORT=kernel SYSLOG_FACILITY=0 SYSLOG_IDENTIFIER=kernel PRIORITY=5 MESSAGE=Linux version 4.15.0-72-generic (buildd@lcy01-amd64-023) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubu __CURSOR=s=4559cb9ceb064f3a92cd33efaaa8fc9a;i=4;b=d02da647a9c04f8683e593fdc2a40b30;m=1d8dd1;t=598df47090c89;x=c __REALTIME_TIMESTAMP=1575459620457609 __MONOTONIC_TIMESTAMP=1936849 _BOOT_ID=d02da647a9c04f8683e593fdc2a40b30 PRIORITY=6 _MACHINE_ID=9d83d97c30d14e73ae6271783c036759 _HOSTNAME=acheron _SOURCE_MONOTONIC_TIMESTAMP=0 _TRANSPORT=kernel SYSLOG_FACILITY=0 SYSLOG_IDENTIFIER=kernel MESSAGE=Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-72-generic root=UUID=5cb98ba3-8583-42fb-90c0-e7d251ab7815
locally it just queries systemd for log events as I understand it, this should also not be binary and use a standardized API
From what I could see,
systemd-journal-remote is just acting as a pipe between systemd and the external service. It’s just taking what it’s given and sending it to the external service, rather than doing any translation.
Thank you for the additional information. This seems to be sufficiently robust.
Before reviewers vote, @pedronis, can you comment on the above?
yes, it will take me a bit to form a comment though, I wasn’t involved in those discussions originally
note that none of the interface stuff would be necessary if
/etc/systemd/journal-upload.conf could be re-mapped inside the snap via a layout (which is forbidden due to its location). it would also not be necessary to worry if the file exists on the host already in this case …
i wonder if the layouts code couldnt just whitelist this particular file for re-mapping and we could forget about the interface at all …
Having the remote journal export configuration as an option in
snapd may be a better long-term approach. This snap may help to fill a gap in the short-term.
Well, adding the binary to core/core18/core20 and exposing the config via snap set should be able to go fast once people have decided that is the direction (it after all wouldn’t be tied to a snapd release).
well, given the fact that you need to import a bunch of certificates to gain https forwarding, i fear it wont be that fast.
plain http is surely quick though (just an on/off toggle in the config), but for https we need a configuration tool that imports the certs into the system.
I’m thinking about this as basically a request for remote syslog support, and traditional remote syslogging does not use certificates, so, IME, adding plain immediately and then adding certificate support later seems ‘ok’. This is just my opinion of course and will defer to @pedronis and the snapd team’s plans in this area.
Wouldn’t something like this side-step the issues for a while:
layout: /etc/systemd/journal-upload.conf: bind-file: $SNAP_DATA/etc/journal-upload.conf
(that’s one the original use cases for layouts)
Then the snap doesn’t need afaict system-files and it will have “snap set” ways to write the config.
it would, if it was allowed, /etc/systemd is a protected path it seems, that is what i meant above regarding white-listing.
I tried and seems to work, and afaik it isn’t:
well, i got an error message at snap install time, i’ll have to re-build the snap with layout to give you the exact error message i got on the core18 system i tried it on …
will report back with the error as soon as i have that …
here is the error:
$ sudo snap install --dangerous logsync-james_0.2_amd64.snap error: cannot perform the following tasks: - Run configure hook of "logsync-james" snap if present (run hook "configure": ----- cannot update snap namespace: cannot write to "/etc/systemd/journal-upload.conf" because it would affect the host in "/etc/systemd" snap-update-ns failed with code 1: File exists -----)
using snapcraft.yaml and configure hook as in: