DEV Community

loading...

Configure Nginx Ingress Controller to use JSON log format

Bryan Sazon
8 years in the biz. Backend Engineer (Go). System Administrator (Kubernetes, Linux, Cloud).
Updated on ・1 min read

Problem

I'm running Nginx Ingress Controller deployed on GKE and I want to structure my web server logs so I can create metrics and alerts in Google Cloud Stackdriver.

References

Solution

If you are using GKE, there is already a Fluentd DaemonSet that forwards all container logs to Stackdriver.

All you need to do is ensure that Nginx Ingress Controller's ConfigMap has the following log-format-upstream settings:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-controller
  labels:
    app: nginx-ingress
data:
  log-format-escape-json: "true"
  log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
    "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
    "$upstream_status", "upstreamAddr": "$upstream_addr","httpRequest":{"requestMethod":
    "$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
    "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent",
    "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s",
    "protocol":"$server_protocol"}}'
Enter fullscreen mode Exit fullscreen mode

If you are deploying Nginx using the Helm Chart, the values file will look something like this:

controller:
  config:
    log-format-escape-json: "true"
    log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
    "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
    "$upstream_status", "upstreamAddr": "$upstream_addr","httpRequest":{"requestMethod":
    "$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
    "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent",
    "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s",
    "protocol":"$server_protocol"}}'
Enter fullscreen mode Exit fullscreen mode

Discussion (4)

Collapse
ianks profile image
Ian Ker-Seymer • Edited

Is this properly indexed by Stackdriver? From reading the documentation, it seems that Stackdriver wants the HTTP info nested in the httpRequest key, and IIRC, JSON logs end up in the jsonPayload key for k8s containers in Stackdriver. Or is Stackdriver smart enough to figure out it should be in the httpRequest key?

UPDATE:

So I think I figured this out, the JSON should be nested in an httpRequest key. Here's the working version:

log-format-escape-json: "true"
log-format-upstream: log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
    "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
    "$upstream_status", "upstreamAddr": "$upstream_addr","httpRequest":{"requestMethod":
    "$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
    "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent",
    "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s",
    "protocol":"$server_protocol"}}'
Collapse
cpandey05 profile image
Chandan Pandey

I do not see the difference ... is the original post also updated?

Collapse
bzon profile image
Bryan Sazon Author

Thanks for pointing this out.

Collapse
timbunce profile image
Tim Bunce

Here's another example JSON format I found useful:
journal.petrausch.info/post/2020/0...