Vacuuming logs in /var/log/journal

snapd
1

Hi,

We enabled persistent logs with the documentation on:

The logs get persistent on the disc without a problem. The configuration for journald seems to be the default, https://manpages.debian.org/testing/systemd/journald.conf.5.en.html. Logs on /var/log/journal do grow up to 10% of the disk size.

However, journald seems not to rotate / auto-vacuum old archived logs to replace them with newer logs. This means, that as soon the quota is exceeded, new logs will not be persisted and stay transient.

I don’t know if this is configured and intended behavior on Ubuntu Core. But, journald seems to not rotate and auto-vacuum per default.

Further, we tried to run the vacuum command (journald --vacuum-size=100M) via a snap to clean the logs. This is not possible, as the log-observe-interface only provides read access to /var/log/journal. I also did not find a interface which provide delete-access to that folder.

So my questions would be,

  1. Is journald behaving correctly by not rotating and auto-vacuuming old log files.
  2. Is there a way to further configure journald on Ubuntu Core to maybe configure it to auto-vacuuming?
  3. How to get delete-permissions to /var/log/journal to manually vacuum the logs in /var/log/journal. Is there an interface I’m missing, if not might it be possible to add a log-control interface?

Regards

2

Friendly bump as this is still unanswered.

3

IMHO this is a bug in snapd itself.

if it enables persistence it should also set sane defaults for the log rotation in journald.conf and not have journald fall back to ring-buffered logs once the quota is reached … try filing a snapd bug in launchpad

4

What about the idea to introduce a log-control interface to trigger an vacuum run? This should be possible regardless whether snapd/journald is doing correct log-ration.

5

Well, i’m not an architect so i dont make such decisions, but i guess since there is already a snap setting for persistence, i’d expect to be an adjustable setting for the vaccuming to be added here … instead of an interface that would potentially interfere with that config option by giving you 100% control …

file that bug so the architects can take a look at it and move on with what they consider the right implementation

6

The bug is created here => https://bugs.launchpad.net/snapd/+bug/1991892

7

@retob Is it possible to get the journal logs from a reboot on the affected device? journalctl -b and then upload it? I’d like to see if systemd is reporting anything causing the rotation to not happen. In ubuntu core we are using standard rules for the default journald namespace, which is to limit the size to 10%, and rotation should occur every 1/8 of that space. So if the rotation is not happening, hopefully systemd is reporting some kind of error or reason as to why.