Best practice for logging

Is there a best practice/pattern used for handling snapped application logs?

I was thinking of making a part for logrotate and logging to somewhere in $SNAP_DATA. Does this sound like a reasonable path?

Possibly a more snappy architecture would be a logrotate snap with a content interface?

Logging for daemons is automatically taken care of. @Chipaca is working on exposing some of this more easily from the snap command.

@sergiusens In my use case, nginx-passenger drives a rails app. The rails app logs separately from nginx (the daemon part).

if those components are separate apps (that are daemons) theyā€™ll each have their own journal; if they use syslog or they print to stdout/stderr, theyā€™ll be handled by systemdā€™s journald and the deviceā€™s configuration of it (which could include log rotation, but also logging only to volatile storage when the physical storage is precious). My work on this is so that you donā€™t need to figure out the systemd unit name in order to get at the logs, but until thatā€™s done and merged and released, you can look at what youā€™d see by doing sudo journalctl -u snap.<snap>.<app>.service (this will at some point become simply snap log <snap>.<app>).

@chipaca that is awesome! What Iā€™m talking about though is a application level logger that logs to a named file.

@chipaca take nginx for example. Nginx defaults to logging to logs/{access,error}.log

For the nginx example, we would need to tell it to log to somewhere writable then right? Or, I guess Iā€™m not seeing how the daemon/logging tie together.

Here is a gist of my snapcraft.yaml (trying to get nginx working), and I just mis-configuring the app?

for nginx, Iā€™d suggest using its ā€œlog to syslogā€ feature; that way log rotation is handled for you by the systemā€™s journal (and you donā€™t need to worry about turning off logging on constrained systems).

1 Like

Awesome. Thats exactly what I need to do then. Thanks!

@sergiusens @chipaca

(expanding on rant from irc)

17:38 bdx beating this nginx logging horse to death here
17:40 bdx I had great success with getting my logs to syslog via the snap logging proxy
17:40 bdx not sure the technical name for that mechanism
17:41 bdx a req came down the line that this application log to a separate file(s) and the logs end up in s3
17:41 bdx so I had to log to $SNAP_COMMON
17:41 bdx which works great until I try to setup logrotate
17:42 bdx I hit my head on a few things, but ended up getting logrotate in this working/broken state
17:43 bdx in short, the issue I seem to be having is that when logrotate goes to swap out the logfile, nginx needs a kick (reload) to start writing to the new logfile

The issue Iā€™m experiencing is that, Iā€™m not quite sure how to go about reloading nginx being that it is a process managed by the snap (see below).

17:44 bdx which brings me to the (logrotate) postrotate scripts
17:44 bdx which can be used to send signal or run a bin when your gets rotated and swapped out
17:45 bdx so my idea here was to create a wrapper command in my snap to facilitate running the nginx -s reload command
17:49 bdx hereā€™s whats in my wrapper http://paste.ubuntu.com/26115750/
17:50 bdx the output Iā€™m getting ^
17:50 bdx seems like the snap is preventing the process from being restarted
17:50 bdx nginx: [alert] kill(17878, 1) failed (1: Operation not permitted)
17:50 bdx I should probably write another forum post pertaining to this
17:51 bdx possibly my last one on logging wasnā€™t specific enough
17:51 bdx is this making sense?
17:51 bdx do I need to add more context somehow in a more proper write up?

Am I going about this the right way?

Possibly there is something else I can do to reload nginx (other then http://paste.ubuntu.com/26115750/)?

Thoughts?

thanks

Soā€¦ Iā€™m not sure Iā€™m seeing the whole picture.

Have you set nginx -s reload as the reload-command of the nginx app? If so, then snap restart --reload would run that for you from ā€˜outsideā€™, that is, when not in a snap.

To do the same thing from inside a snap is being worked on right now (it was going to be in 2.29 but got into trouble; @pstolowski knows more).

Right, the ability to control own services via snapctl didnā€™t make it for 2.29, but itā€™s going to be included in 2.30.

To summarize:

  • if you want to reload a service of another snap, you need to use snap restart (...).
  • if you want to restart your own service (from the ā€œinsideā€ of the snap), from version 2.30 onwards you can use snapctl restart.
1 Like

Just a reminder that in your snap yaml you can use e.g. assumes: [snapd2.30] to depend on features from a particular version of snapd onwards.

My bad, ā€˜pdl-api.nginxā€™ started out as an nginx wrapper, then I was deving around and decoupled the nginx wrapper into two separate wrappers; one to start nginx and one to reload it. I neglected to modify the wrapper name to reflect its purpose. In the pastebin example ^, ā€˜pdl-api.nginxā€™ should be ā€˜pdl-api.nginx-reloadā€™.

I must have missed this. I can set a ā€˜reloadā€™ command for my app in snapcraft.yaml?

If that^ is the case I think this would be the key to my solution.

I canā€™t seem to locate the docs on the ā€˜reloadā€™ command for the app in snapcraft.yaml though, possibly Iā€™ve misconstrued your meaning.

Awesome, thanks for pointing this out. Am I interpreting this correct that in using assumes, you can depend on features from future releases?

Or possibly you meant once 2.3 drops I can use use assumes to create a dep on it?

weā€™re slipping on the docs front. Iā€™ve quickly added it to The snap format but it might need some polish still.

1 Like

assumes is a safety net to avoid a snap installing when you know it wonā€™t work. Itā€™s not very smart (itā€™s not epochs which are still WIP), but should help.

Just realised assumes is not documented in that page either. Iā€™ll let somebody else do it though ā€” or get to it when I empty my plate a little bit.

1 Like