Rotating landscape-client logs

aliihsan · January 7, 2025, 7:38am

Hello,

We are using the Landscape client snap and noticed that its log files never rotate, which could lead to the device running out of storage over time. This behavior differs from the Debian package, where logrotate is used along with a SIGUSR1 signal sent to the Landscape service to trigger log rotation. (Refer to this file for details.)

We would like to request a mechanism for log rotation in the Landscape client snap.

landscape-client version

Name              Version  Rev  Tracking       Publisher   Notes
landscape-client  24.08    334  latest/stable  canonical✓  -

Example log file size(Although its on DEBUG level, we still would like to rotate)

-rw-r--r-- 1 root root 6.2G Jan  7 07:29 broker.log
-rw-r--r-- 1 root root 8.1M Jan  7 07:37 manager.log
-rw-r--r-- 1 root root  15M Jan  7 07:35 monitor.log
-rw-r--r-- 1 root root 305K Jan  2 13:35 watchdog.log

Thanks.

aliihsan · January 10, 2025, 8:11am

Hey @mikecw , do you have any idea about this?

mikecw · January 10, 2025, 9:11am

Hi @aliihsan,

Can have a look. Seems sensible a request so I’ll stick it in the backlog. Code base is the same between deb and snap so probably just need to propagate that signal into the snap somehow.

ogra · January 10, 2025, 9:29am

UbuntuCore does not ship logrotate since snaps usually log through journald which has internal ways of dealing with this.

If the snap writes its own logs to its own space you will likely need to stage logrotate into it and ship a systemd timer to have it run regularly with a specific config to rotate the logs in the snap owned space …

mikecw · January 10, 2025, 9:55am

Hi @aliihsan,

The request seems sensible - I’ll stick it in the backlog and we’ll have a look. As @ogra says we may have to look at options as we want to keep the snap and deb codebase the same as much as possible.

aliihsan · January 10, 2025, 10:17am

Thanks a lot, both of you guys, I will be waiting for it.

mikecw · January 13, 2025, 1:25pm

github.com/canonical/landscape-client

fix: log rotation

canonical:main ← st3v3nmw:fix-log-rotation

opened 01:22PM - 13 Jan 25 UTC

st3v3nmw

+25 -3

This change allows the landscape-client snap to periodically rotate log files. T…his issue was reported [here](https://forum.snapcraft.io/t/rotating-landscape-client-logs/44509). The new `landscape-client.logrotate` daemon will run once a week on Sundays some time between 2am & 3am. After rotation, the `landscape-client` service is restarted. --- I tested this with `logrotate` configured to rotate logs every hour with a 10kb max file size. The log rotation is working as expected: ```console $ ls /var/snap/landscape-client/current/var/log/landscape/ -lah total 3.9M drwxr-xr-x 2 root root 4.0K Jan 13 16:00 . drwxr-xr-x 3 root root 4.0K Jan 13 11:16 .. -rw-r--r-- 1 root root 356K Jan 13 16:01 broker.log -rw-r--r-- 1 root root 3.3M Jan 13 16:00 broker.log.1 -rw-r--r-- 1 root root 24K Jan 13 13:44 broker.log.2.gz -rw-r--r-- 1 root root 116K Jan 13 13:21 broker.log.3.gz -rw-r--r-- 1 root root 93 Jan 13 11:44 landscape-config.log -rw-r--r-- 1 root root 2.3K Jan 13 16:15 manager.log -rw-r--r-- 1 root root 25K Jan 13 16:00 manager.log.1 -rw-r--r-- 1 root root 2.0K Jan 13 13:21 manager.log.2.gz -rw-r--r-- 1 root root 3.7K Jan 13 16:00 monitor.log -rw-r--r-- 1 root root 56K Jan 13 16:00 monitor.log.1 -rw-r--r-- 1 root root 4.5K Jan 13 13:21 monitor.log.2.gz -rw-r--r-- 1 root root 77 Jan 13 16:00 watchdog.log -rw-r--r-- 1 root root 6.0K Jan 13 16:00 watchdog.log.1 $ snap logs landscape-client.logrotate # after rotation triggered by snapd timer 2025-01-13T16:20:06+03:00 systemd[1]: Started snap.landscape-client.logrotate.service - Service for snap application landscape-client.logrotate. 2025-01-13T16:20:06+03:00 systemd[1]: Stopping snap.landscape-client.logrotate.service - Service for snap application landscape-client.logrotate... 2025-01-13T16:20:06+03:00 systemd[1]: snap.landscape-client.logrotate.service: Killing process 730575 (snapctl) with signal SIGKILL. 2025-01-13T16:20:06+03:00 systemd[1]: snap.landscape-client.logrotate.service: Deactivated successfully. 2025-01-13T16:20:06+03:00 systemd[1]: Stopped snap.landscape-client.logrotate.service - Service for snap application landscape-client.logrotate. 2025-01-13T16:20:06+03:00 systemd[1]: Started snap.landscape-client.logrotate.service - Service for snap application landscape-client.logrotate. 2025-01-13T16:20:06+03:00 systemd[1]: snap.landscape-client.logrotate.service: Deactivated successfully. $ ls /var/snap/landscape-client/current/var/log/landscape/ -lah total 628K drwxr-xr-x 2 root root 4.0K Jan 13 16:20 . drwxr-xr-x 3 root root 4.0K Jan 13 11:16 .. -rw-r--r-- 1 root root 386 Jan 13 16:20 broker.log -rw-r--r-- 1 root root 432K Jan 13 16:20 broker.log.1 -rw-r--r-- 1 root root 24K Jan 13 13:44 broker.log.3.gz -rw-r--r-- 1 root root 116K Jan 13 13:21 broker.log.4.gz -rw-r--r-- 1 root root 93 Jan 13 11:44 landscape-config.log -rw-r--r-- 1 root root 1.8K Jan 13 16:20 manager.log -rw-r--r-- 1 root root 3.0K Jan 13 16:20 manager.log.1 -rw-r--r-- 1 root root 2.0K Jan 13 13:21 manager.log.3.gz -rw-r--r-- 1 root root 3.0K Jan 13 16:20 monitor.log -rw-r--r-- 1 root root 6.3K Jan 13 16:20 monitor.log.1 -rw-r--r-- 1 root root 4.5K Jan 13 13:21 monitor.log.3.gz -rw-r--r-- 1 root root 77 Jan 13 16:20 watchdog.log -rw-r--r-- 1 root root 259 Jan 13 16:20 watchdog.log.1 $ systemctl status snap.landscape-client.landscape-client.service systemctl status snap.landscape-client.landscape-client.service ● snap.landscape-client.landscape-client.service - Service for snap application landscape-client.landscape-client Loaded: loaded (/etc/systemd/system/snap.landscape-client.landscape-client.service; enabled; preset: enabled) Active: active (running) since Mon 2025-01-13 16:20:06 EAT; 31s ago [...] ```

@aliihsan - please feel free to have a look and see if it is sufficient for your requirements.

mikecw · January 14, 2025, 8:58am

This is now approved and is currently in the EDGE snap channel. I’ll get some docs done for it shortly.

aliihsan · January 14, 2025, 1:04pm

Thanks a lot for bringing it up quickly, I appreciated it.

I will test it soon.

aliihsan · April 6, 2025, 2:18pm

Hey @mikecw, I finally had a chance to test this implementation, but it seems something is not happy.

I checked the logs and sizes before and after the timer was supposed to expire. It seems log rotation didn’t happen.

I am using version 22/edge revision 388.

Although the snap logs stated that the service is started and deactivated in planned time frame.

2025-04-06T02:07:16Z systemd[1]: Started Service for snap application landscape-client.logrotate.
2025-04-06T02:07:16Z systemd[1]: snap.landscape-client.logrotate.service: Deactivated successfully.

Does it require being size limited before rotating? I have checked the config file you provided in the above commit, but couldn’t find it. Any idea?

Thanks.

mikecw · April 8, 2025, 1:39pm

Hmmm dunno.

@st3v3nmw - can you please have a look at this? Thanks

st3v3nmw · April 11, 2025, 9:47am

Yep… I’ll provide feedback soon.