NGINX Log Taming - Structured Data & Statistical Analysis

Logs are great! They provide a lot of information that assists in understanding the current environment and provide insight when trying to solve problems. It seems every application has its own logging style. Some log to a file, some log to the system journal, some log to standard out, etc. In addition to the variance in where they log, they generally have their own logging format. Some attempt to log in plain English while some log using application specific codes that must be cross referenced in order to understand what is happening.

At some point logging collided with the concept of structured data and their was much rejoicing! Suddenly we could shelve our custom parsers, as proud of them as we were, and enjoy logs that have predictable key/value pairs. The most common way applications offer logs in a structured data format is via JSON output.

Not only are JSONified logs wonderful for scripts that ingest them, when we are hacking for our lives during an outage we can pipe the log file through a utility such as jq and find inspiration to keep living during dark times.

Now, on to the namesake of this article, NGINX! A lot of readers here may admin NGINX servers that produce giga bytes of logs every day. processing large logs for meaningful information can be challenging. NGINX has an option to change the logging format to JSON versus the difficult to read/parse default format. This maneuver involves creating a custom log_format and then associating it with the access_log.

The custom log_format definition must be in the /etc/nginx/nginx.conf file, while the access_log directive that invokes it may be in the same file for global effect, or in an a specific site file for more granular control of logs.

The log_format definition belongs in the http context of the /etc/nginx/nginx.conf file. Here is an example custom log_format that is named custom_json:

http {
    <-- snip -->

    log_format custom_json escape=json '{'
        '"time_local":"$time_local",'
        '"remote_addr":"$remote_addr",'
        '"host":"$host",'
        '"request":"$request",'
        '"status": "$status",'
        '"body_bytes_sent":"$body_bytes_sent",'
        '"request_time":"$request_time",'
        '"http_referrer":"$http_referer",'
        '"http_user_agent":"$http_user_agent"'
    '}';

    <-- snip -->
}

Next we invoke the custom parser, this can be defined in the http, server, or location contexts. Here is an invocation example:

access_log /var/log/nginx/access.log custom_json;

Once these changes have been made, restart the service, and have a look at the new log format. Now a line in the log file looks like this:

root@pagoda:/var/log/nginx# tail -1 access.log
{"time_local":"25/Oct/2026:08:30:50 +0900","remote_addr":"192.168.82.1","host":"activebit.net","request":"GET / HTTP/1.0","status": "200","body_bytes_sent":"65","request_time":"0.000","http_referrer":"","http_user_agent":""}

This is arguably harder on the eyes than the default format, but if we pipe it to a utility such as jq in renders in this way:

root@pagoda:/var/log/nginx# tail -1 access.log | jq .
{
  "time_local": "25/Oct/2026:08:30:50 +0900",
  "remote_addr": "192.168.82.1",
  "host": "activebit.net",
  "request": "GET / HTTP/1.0",
  "status": "200",
  "body_bytes_sent": "65",
  "request_time": "0.000",
  "http_referrer": "",
  "http_user_agent": ""
}

This is much easier to read and quickly understand what all the parts of the log message mean. Since it is structured data we do not have to write a custom parser that's prone to errors in order to programmatically analyze the data. Here is an example using python3 of how to quickly get statistics for a given key seen in the log file:

#!/usr/bin/env python3

import json
import sys 
from pprint import pprint

subjFile = sys.argv[1]
keyName = sys.argv[2]

def getFlatFile(ourFile):
    hook = open(ourFile, errors='ignore')
    raw = hook.read()
    hook.close()
    return raw.splitlines()

logList = getFlatFile(subjFile)

statHash = {}
linesProcessed = 0 
for l in logList:
    l = json.loads(l)
    linesProcessed += 1
    if l[keyName] not in statHash:
        statHash[l[keyName]] = 0 
    statHash[l[keyName]] += 1

print('stats for key: ' + keyName)
for seenKey in sorted(statHash):
    percent = round((statHash[seenKey] / linesProcessed) * 100)
    print(seenKey + ' occurances: ' + str(statHash[seenKey]) + ' percent: ' + str(percent) + '%')

This is an example of evaluating the 'status' key value:

root@pagoda:/home/chris/dev.to# ./json_log_eval.py /var/log/nginx/access.log status
stats for key: status
200 occurances: 61 percent: 29%
400 occurances: 11 percent: 5%
403 occurances: 9 percent: 4%
404 occurances: 126 percent: 60%
405 occurances: 1 percent: 0%
444 occurances: 1 percent: 0%

Fear No Log File!