It's 3am, a service is down, and you have exactly one clue: something logged an error somewhere. This is the part of the job nobody teaches directly. You pick it up by staring at terminals for a few years, usually while half asleep, usually under pressure. Here's the version of that knowledge I wish someone had handed me earlier.
Know where to actually look first
Most people's first instinct is cat /var/log/syslog and scroll. That works, sort of, right up until the log file is 400MB and the failure happened three services ago. Before you dive in, figure out which of the two logging worlds you're in.
journald is what most modern distros use by default. Logs are binary, structured, and queried through journalctl, not read directly as text files.
# Everything from a specific service, most recent first
journalctl -u nginx.service -e
# Only the last 20 minutes
journalctl -u nginx.service --since "20 minutes ago"
# Only errors and worse (crit, alert, emerg)
journalctl -u nginx.service -p err
Plain text logs under /var/log are still very much alive, especially for anything that predates systemd conventions or writes its own log files by choice, think nginx, postgres, most application-level logging.
tail -f /var/log/nginx/error.log
The mistake people make here isn't picking the wrong tool, it's not checking which one applies before they start searching. If you grep a file that's empty because the service actually logs to journald, you'll conclude "no errors" when really you were just looking in the wrong place.
The actual diving technique
Once you know where the logs live, the skill isn't reading faster. It's narrowing faster. A few things that consistently save time:
Time-box before you grep. Don't search the whole file for a pattern. Get the failure window first, even roughly, then constrain the search to it.
journalctl -u myapp.service --since "14:00" --until "14:15"
Search around a match, not just for it. The line with the word "error" is rarely the whole story. The line before it usually explains why, and the line after usually shows what happened as a result.
grep -B 5 -A 10 "connection refused" app.log
Follow one identifier through the whole log, not the whole error message. If a request or a job has an ID, chase that ID specifically. Searching for the generic error string gets you a wall of matches from every unrelated failure that shares the same wording. Searching for the specific ID gets you the exact sequence of events for the thing you actually care about.
grep "req-8f21a3" app.log
If the logs are structured JSON, stop treating them like plain text. A lot of modern services log JSON lines instead of free text, and grepping those by eye is painful. jq turns that into something readable.
journalctl -u myapp.service -o json | jq 'select(.PRIORITY == "3")'
None of this is exotic. It's the difference between opening a 50,000 line file and reading top to bottom, versus asking three narrow questions in sequence: when did it start, what surrounded the failure, and what specific thing was involved.
Common errors, decoded by service
This part won't be exhaustive, nothing is, but these are the ones that show up constantly and get misread constantly.
nginx
| Message | What it usually means |
|---|---|
502 Bad Gateway |
The upstream (your app server) crashed, restarted, or refused the connection. nginx is fine, whatever's behind it isn't. |
504 Gateway Timeout |
The upstream is alive but too slow to respond within nginx's timeout. Different problem than 502, don't treat them the same. |
connect() failed (111: Connection refused) |
Nothing is listening on the port nginx is trying to reach. Check if the upstream process is even running. |
upstream sent too big header |
The app is returning headers nginx wasn't configured to expect. Usually shows up after adding auth tokens or cookies without bumping proxy_buffer_size. |
PostgreSQL
| Message | What it usually means |
|---|---|
FATAL: too many connections for role |
Connection pool exhaustion. Either the app isn't closing connections properly, or max_connections genuinely needs raising. |
deadlock detected |
Two transactions grabbed locks in opposite order. Postgres killed one to break the cycle. Look at the accompanying detail line, it names both queries. |
canceling statement due to statement timeout |
Not a bug by itself, it's a configured limit doing its job. The real question is why that query needed longer than expected this time. |
could not extend file... No space left on device |
Exactly what it says, but people still spend twenty minutes assuming it's a config problem before checking disk space. |
systemd / journald
| Message | What it usually means |
|---|---|
Failed with result 'exit-code' |
The process exited with a nonzero code. Check systemctl status <service> right after, it usually shows the actual exit code, which tells you far more than the generic message. |
Failed with result 'oom-kill' |
The kernel killed the process for using too much memory. This is a memory problem, not a crash to debug in the app logs. |
Start request repeated too quickly |
Systemd gave up restarting the service because it kept failing immediately. This message is a symptom, the real error is earlier in the same unit's log, right before the first failed start. |
What this actually comes down to
None of the tricks above are about memorizing every possible error string. That list is infinite and you'll never finish it. What actually matters is the order of operations: know which logging system you're dealing with before you search, narrow to a time window before you grep, follow a specific identifier instead of a generic phrase, and read the lines around a match instead of just the match itself.
The error message rarely lies. It's just usually incomplete on its own, and the missing part is almost always sitting a few lines away, waiting for you to widen the search by five lines in either direction.
Top comments (0)