Transform Pino HTTP logs into Common Log Format.
$ npm install -g pino-clf
$ pino-clf -h
pino-clf [-d] [-k] [-a] typetype common | combined-d | --dest | stderr | stdout or Number or Array of Numbers--destination Specify file descriptor to send log to [access, referral, agent]-a | --ancillary stderr | stdout or Number. Specify JSON logs fd-k | --keep true | false Retain transformed logs in ancillary output
Common Log Format
The Common Log Format is a frequently used log format, that has a very mature and well established ecosystem built around it.
It takes two forms: "Combined" and "Common".
When the type is "Common" this can break down into (up to) three separate log streams:
- Access log (also known as Common log)
- Referall log
- Agent log
Access log example:
127.0.0.1 - Aladdin [21/Jul/2016:17:34:52 -0060] "GET /api/activity/component HTTP/1.1" 200 -
Referral log example:
[21/Jul/2016:17:34:52 -0060] "http://localhost:20000/"
Agent log example:
[21/Jul/2016:17:34:52 -0060] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
The "Combined" format combines all three formats into one, for instance:
127.0.0.1 - Aladdin [21/Jul/2016:17:34:52 -0060] "GET /api/activity/component HTTP/1.1" 200 - "http://localhost:20000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
Spin up a server that uses a pino http logger (see the Supports section),
pipe it to
pino-clf and desribe the format in tokenized form
$ node server | pino-clf127.0.0.1 - Aladdin [21/Jul/2016:17:34:52 -0060] "GET /api/activity/component HTTP/1.1" 200 - "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
The default output is Combined. To use three log output use the
-d flag and specify
up to three file descriptors (first is access log, second referral log, third agent log):
node server | pino-clf -d [3,4,5] 3>> ./access.log 4>> ./referral.log 5>> ./agent.log
The above will append the Access, Agent, and Referral logs to each corresponding log file.
We can skip the middle log (referall) by using an file desriptor of
null as the
node server | pino-clf -d [3,0,4] 3>> ./access.log 4>> ./agent.log
By default, logs are output to STDOUT, however we can set the
--destination), flag to a a
stderr, or a number (
1 for stdout,
2 for stderr,
3 or more for custom file descriptor):
$ node server | pino-clf -d stderr
The above is equivalent to:
$ node server | pino-clf -d 2 combined
We can also direct formatted log output to custom file descriptors, but we must use bash redirection (in some form) from that file descriptor, otherwise the process will most likely immediately crash (this is to do with how unix works).
$ node server | pino-clf -d 8 8> ./combined.log
Ancillary Output (
By default, any logs which aren't an HTTP log (meaning, they don't have
properties and the
msg isn't "request complete") are filtered out.
However, we can specify an ancillary (secondary) output for other log messages, using
The following will write reformatted HTTP logs to STDOUT and original JSON logs which are not HTTP logs to STDERR.
$ node server | pino-clf -a 2 -d 1
Keep Original HTTP JSON Logs (
--ancillary) flag can be coupled with the
--keep) flag so that
raw HTTP JSON logs are also piped to the ancillary output stream, along with any
The following will pipe all formatted logs to the
4 file descriptor which is redirected to a file,
while all original JSON logs (instead of non-HTTP logs) are written to STDOUT.
$ node server | pino-clf -k -a 1 -d 4 4> ./combined.logs
clf(type?, destination?, ancillary?)
Returns a stream that we write Pino JSON logs to.
type (String | Object)
- combined (default)
If object form type is
false) - preserve original HTTP Logs, as well as other JSON logs when writing to ancillary stream
destination (Stream | Array)
Stream or streams to write output to.
If the type is
common destination can be an
up to three streams, to enable the three log
Stream to output original JSON logs to.