Your site is down and the logs are 4 GB: 7 tools for reading them fast

The alert fires. Your monitor says the checkout is returning 502s. You SSH in, open the access log, and it's four gigabytes of lines scrolling past faster than you can read.

The answer to "what's happening?" is almost always in there. The problem is getting it out in the first five minutes, while you're stressed and someone is asking for an update.

This is our shortlist of tools for reading logs under pressure. They're ordered from "already installed everywhere" to "worth installing before the next incident".

1. tail, grep and friends: the baseline

You know these, but under pressure the useful flags are easy to forget:

# follow the log, only show server errors
tail -F /var/log/nginx/access.log | grep --line-buffered '" 5[0-9][0-9] '

# last 10,000 lines only, so you aren't grepping a year of history
tail -n 10000 /var/log/nginx/access.log | grep -c '" 502 '

-F (capital) keeps following across log rotation, which matters if rotation happens mid-incident. --line-buffered stops grep from holding output back when piped.

Best for: "is it still happening right now?"

2. awk: instant aggregation

Most incident questions are counts: errors per minute, which paths fail, which IPs are hammering you. awk answers them without installing anything. For the common/combined log format:

# status codes per minute
awk '{split($4,t,":"); m=t[2]":"t[3]; c[m" "$9]++} END{for(k in c) print k, c[k]}' access.log | sort

# which paths are failing
awk '$9>=500{print $7}' access.log | sort | uniq -c | sort -rn | head

On a small sample log, that gives:

== status per minute
09:14 200 1
09:14 502 2
09:15 200 1
09:15 502 1
09:15 504 1
== top failing paths
      3 /checkout
      1 /api/cart

Now you know when it started and that it's the checkout, not the whole site, which is the first thing anyone will ask.

Best for: the first five minutes, on any box.

3. jq: when your logs are JSON

Structured logs are wonderful until you try to read them raw. jq turns them back into answers:

# errors only, compact view
jq -c 'select(.level=="error") | {ts: .time, msg: .msg, route: .route}' app.log

# count errors by route
jq -r 'select(.level=="error") | .route' app.log | sort | uniq -c | sort -rn

If some lines aren't valid JSON (startup banners, stack traces), add -R 'fromjson? // empty' handling or pre-filter with grep '^{'.

Best for: application logs from modern frameworks.

4. lnav: a log viewer that understands formats

lnav (the Logfile Navigator) opens one or many log files, detects common formats automatically, merges them by timestamp, and colour-codes errors. You can jump between errors with a keypress and run SQL queries over the parsed fields.

lnav /var/log/nginx/access.log /var/log/app/*.log

Merging the web server log and app log on one timeline is the killer feature: you see the 502 and the exception that caused it next to each other.

Best for: correlating several logs on one box.

5. GoAccess: a dashboard from an access log

GoAccess parses web server access logs and shows a live terminal dashboard (or generates an HTML report): requests per second, status codes, top URLs, top IPs, user agents.

goaccess /var/log/nginx/access.log --log-format=COMBINED

During an incident, it answers "is this a traffic spike or a real failure?" in seconds. A sudden surge from one IP range or one user agent looks very different from errors spread across normal traffic.

Best for: traffic questions: bots, spikes, floods.

6. journalctl: for anything under systemd

If your app runs as a systemd service, its logs may not be in a file at all:

# this service, since the incident started, following
journalctl -u myapp.service --since "10 min ago" -f

# only errors and worse, across all services
journalctl -p err --since today

--since and --until with human-friendly times are much faster than scrolling.

Best for: services, crashes, restarts, OOM kills (journalctl -k | grep -i oom).

7. A central log platform (when you have more than one box)

Once you have several servers or containers, SSH-and-grep stops working. Self-hosted options like Grafana Loki or the OpenSearch stack, or any hosted log service, let you query across everything at once.

The key incident feature is the same everywhere: filter by time window and service, then group by status or error message. If your team can't do that in under a minute, practise before the next incident, not during it.

Best for: fleets, containers, anything ephemeral.

A tiny incident log routine

Whatever tools you use, the order of questions barely changes:

1. When did it start? Errors per minute (awk, GoAccess, log platform).
2. What's affected? Top failing paths or routes.
3. Who's affected? All users, one region, one customer, bots?
4. What changed? Deploys, config, traffic, a dependency. Line up the start time with your deploy log.
5. What does the app say? The error message on the failing route (jq, lnav, journalctl).

Write the answers in the incident channel as you go. They become the timeline for your postmortem.

Logs tell you why; monitoring tells you when

Logs are where you find the cause, but they're a poor way to find out something is wrong in the first place. Nobody tails a log all day. An external check, from outside your network, is what notices the checkout failing at 3am. That's what upordown.online does; the log tools above take over once you're awake.

Related reading

• Alert on 1 failure or 3? I simulated a month of checks to see the trade-off
• How to read curl -v output to find where a request fails
• Is it down, or is it me? Debugging a site that won't load, one layer at a time
• 503 vs 504: what each error means and who has to fix it
• ECONNREFUSED vs ETIMEDOUT vs ECONNRESET: what your monitor's error is actually telling you

What's the first command you type when an alert fires? Share your favourite incident one-liner in the comments, I'll collect the best ones.