Best practice for logging

[jamesbeedy]

Is there a best practice/pattern used for handling snapped application logs?

I was thinking of making a part for logrotate and logging to somewhere in $SNAP_DATA. Does this sound like a reasonable path?

Possibly a more snappy architecture would be a logrotate snap with a content interface?

[sergiusens]

Logging for daemons is automatically taken care of. @Chipaca is working on exposing some of this more easily from the snap command.

[jamesbeedy]

@sergiusens In my use case, nginx-passenger drives a rails app. The rails app logs separately from nginx (the daemon part).

[chipaca]

if those components are separate apps (that are daemons) they’ll each have their own journal; if they use syslog or they print to stdout/stderr, they’ll be handled by systemd’s journald and the device’s configuration of it (which could include log rotation, but also logging only to volatile storage when the physical storage is precious). My work on this is so that you don’t need to figure out the systemd unit name in order to get at the logs, but until that’s done and merged and released, you can look at what you’d see by doing sudo journalctl -u snap.<snap>.<app>.service (this will at some point become simply snap log <snap>.<app>).

[jamesbeedy]

@chipaca that is awesome! What I’m talking about though is a application level logger that logs to a named file.

[jamesbeedy]

@chipaca take nginx for example. Nginx defaults to logging to logs/{access,error}.log

For the nginx example, we would need to tell it to log to somewhere writable then right? Or, I guess I’m not seeing how the daemon/logging tie together.

Here is a gist of my snapcraft.yaml (trying to get nginx working), and I just mis-configuring the app?

https://gist.github.com/jamesbeedy/134e1dcc0fe3fefc6936ee9fbe194153

[chipaca]

@chipaca take nginx for example. Nginx defaults to logging to logs/{access,error}.log

For the nginx example, we would need to tell it to log to somewhere writable then right? Or, I guess I’m not seeing how the daemon/logging tie together.

for nginx, I’d suggest using its “log to syslog” feature; that way log rotation is handled for you by the system’s journal (and you don’t need to worry about turning off logging on constrained systems).

[jamesbeedy]

Awesome. Thats exactly what I need to do then. Thanks!

[jamesbeedy]

@sergiusens @chipaca

(expanding on rant from irc)

17:38 bdx beating this nginx logging horse to death here
17:40 bdx I had great success with getting my logs to syslog via the snap logging proxy
17:40 bdx not sure the technical name for that mechanism
17:41 bdx a req came down the line that this application log to a separate file(s) and the logs end up in s3
17:41 bdx so I had to log to $SNAP_COMMON
17:41 bdx which works great until I try to setup logrotate
17:42 bdx I hit my head on a few things, but ended up getting logrotate in this working/broken state
17:43 bdx in short, the issue I seem to be having is that when logrotate goes to swap out the logfile, nginx needs a kick (reload) to start writing to the new logfile

The issue I’m experiencing is that, I’m not quite sure how to go about reloading nginx being that it is a process managed by the snap (see below).

17:44 bdx which brings me to the (logrotate) postrotate scripts
17:44 bdx which can be used to send signal or run a bin when your gets rotated and swapped out
17:45 bdx so my idea here was to create a wrapper command in my snap to facilitate running the nginx -s reload command
17:49 bdx here’s whats in my wrapper http://paste.ubuntu.com/26115750/
17:50 bdx the output I’m getting ^
17:50 bdx seems like the snap is preventing the process from being restarted
17:50 bdx nginx: [alert] kill(17878, 1) failed (1: Operation not permitted)
17:50 bdx I should probably write another forum post pertaining to this
17:51 bdx possibly my last one on logging wasn’t specific enough
17:51 bdx is this making sense?
17:51 bdx do I need to add more context somehow in a more proper write up?

Am I going about this the right way?

Possibly there is something else I can do to reload nginx (other then http://paste.ubuntu.com/26115750/)?

Thoughts?

thanks

[chipaca]

So… I’m not sure I’m seeing the whole picture.

Have you set nginx -s reload as the reload-command of the nginx app? If so, then snap restart --reload would run that for you from ‘outside’, that is, when not in a snap.

To do the same thing from inside a snap is being worked on right now (it was going to be in 2.29 but got into trouble; @pstolowski knows more).

[pstolowski]

Right, the ability to control own services via snapctl didn’t make it for 2.29, but it’s going to be included in 2.30.

To summarize:

• if you want to reload a service of another snap, you need to use snap restart (...).
• if you want to restart your own service (from the “inside” of the snap), from version 2.30 onwards you can use snapctl restart.

[chipaca]

Just a reminder that in your snap yaml you can use e.g. assumes: [snapd2.30] to depend on features from a particular version of snapd onwards.

[jamesbeedy]

My bad, ‘pdl-api.nginx’ started out as an nginx wrapper, then I was deving around and decoupled the nginx wrapper into two separate wrappers; one to start nginx and one to reload it. I neglected to modify the wrapper name to reflect its purpose. In the pastebin example ^, ‘pdl-api.nginx’ should be ‘pdl-api.nginx-reload’.

[jamesbeedy]

Have you set nginx -s reload as the reload-command of the nginx app?

I must have missed this. I can set a ‘reload’ command for my app in snapcraft.yaml?

If that^ is the case I think this would be the key to my solution.

I can’t seem to locate the docs on the ‘reload’ command for the app in snapcraft.yaml though, possibly I’ve misconstrued your meaning.

[jamesbeedy]

Awesome, thanks for pointing this out. Am I interpreting this correct that in using assumes, you can depend on features from future releases?

Or possibly you meant once 2.3 drops I can use use assumes to create a dep on it?

[chipaca]

we’re slipping on the docs front. I’ve quickly added it to The snap format but it might need some polish still.

[chipaca]

assumes is a safety net to avoid a snap installing when you know it won’t work. It’s not very smart (it’s not epochs which are still WIP), but should help.

Just realised assumes is not documented in that page either. I’ll let somebody else do it though — or get to it when I empty my plate a little bit.