DEV Community: Alex Mitu

How to Actually Read Linux Logs When Something's Broken

Alex Mitu — Wed, 12 Aug 2026 14:51:08 +0000

It's 3am, a service is down, and you have exactly one clue: something logged an error somewhere. This is the part of the job nobody teaches directly. You pick it up by staring at terminals for a few years, usually while half asleep, usually under pressure. Here's the version of that knowledge I wish someone had handed me earlier.

Know where to actually look first

Most people's first instinct is cat /var/log/syslog and scroll. That works, sort of, right up until the log file is 400MB and the failure happened three services ago. Before you dive in, figure out which of the two logging worlds you're in.

journald is what most modern distros use by default. Logs are binary, structured, and queried through journalctl, not read directly as text files.

# Everything from a specific service, most recent first
journalctl -u nginx.service -e

# Only the last 20 minutes
journalctl -u nginx.service --since "20 minutes ago"

# Only errors and worse (crit, alert, emerg)
journalctl -u nginx.service -p err

Plain text logs under /var/log are still very much alive, especially for anything that predates systemd conventions or writes its own log files by choice, think nginx, postgres, most application-level logging.

tail -f /var/log/nginx/error.log

The mistake people make here isn't picking the wrong tool, it's not checking which one applies before they start searching. If you grep a file that's empty because the service actually logs to journald, you'll conclude "no errors" when really you were just looking in the wrong place.

The actual diving technique

Once you know where the logs live, the skill isn't reading faster. It's narrowing faster. A few things that consistently save time:

Time-box before you grep. Don't search the whole file for a pattern. Get the failure window first, even roughly, then constrain the search to it.

journalctl -u myapp.service --since "14:00" --until "14:15"

Search around a match, not just for it. The line with the word "error" is rarely the whole story. The line before it usually explains why, and the line after usually shows what happened as a result.

grep -B 5 -A 10 "connection refused" app.log

Follow one identifier through the whole log, not the whole error message. If a request or a job has an ID, chase that ID specifically. Searching for the generic error string gets you a wall of matches from every unrelated failure that shares the same wording. Searching for the specific ID gets you the exact sequence of events for the thing you actually care about.

grep "req-8f21a3" app.log

If the logs are structured JSON, stop treating them like plain text. A lot of modern services log JSON lines instead of free text, and grepping those by eye is painful. jq turns that into something readable.

journalctl -u myapp.service -o json | jq 'select(.PRIORITY == "3")'

None of this is exotic. It's the difference between opening a 50,000 line file and reading top to bottom, versus asking three narrow questions in sequence: when did it start, what surrounded the failure, and what specific thing was involved.

Common errors, decoded by service

This part won't be exhaustive, nothing is, but these are the ones that show up constantly and get misread constantly.

nginx

Message	What it usually means
`502 Bad Gateway`	The upstream (your app server) crashed, restarted, or refused the connection. nginx is fine, whatever's behind it isn't.
`504 Gateway Timeout`	The upstream is alive but too slow to respond within nginx's timeout. Different problem than 502, don't treat them the same.
`connect() failed (111: Connection refused)`	Nothing is listening on the port nginx is trying to reach. Check if the upstream process is even running.
`upstream sent too big header`	The app is returning headers nginx wasn't configured to expect. Usually shows up after adding auth tokens or cookies without bumping `proxy_buffer_size`.

PostgreSQL

Message	What it usually means
`FATAL: too many connections for role`	Connection pool exhaustion. Either the app isn't closing connections properly, or `max_connections` genuinely needs raising.
`deadlock detected`	Two transactions grabbed locks in opposite order. Postgres killed one to break the cycle. Look at the accompanying detail line, it names both queries.
`canceling statement due to statement timeout`	Not a bug by itself, it's a configured limit doing its job. The real question is why that query needed longer than expected this time.
`could not extend file... No space left on device`	Exactly what it says, but people still spend twenty minutes assuming it's a config problem before checking disk space.

systemd / journald

Message	What it usually means
`Failed with result 'exit-code'`	The process exited with a nonzero code. Check `systemctl status <service>` right after, it usually shows the actual exit code, which tells you far more than the generic message.
`Failed with result 'oom-kill'`	The kernel killed the process for using too much memory. This is a memory problem, not a crash to debug in the app logs.
`Start request repeated too quickly`	Systemd gave up restarting the service because it kept failing immediately. This message is a symptom, the real error is earlier in the same unit's log, right before the first failed start.

What this actually comes down to

None of the tricks above are about memorizing every possible error string. That list is infinite and you'll never finish it. What actually matters is the order of operations: know which logging system you're dealing with before you search, narrow to a time window before you grep, follow a specific identifier instead of a generic phrase, and read the lines around a match instead of just the match itself.

The error message rarely lies. It's just usually incomplete on its own, and the missing part is almost always sitting a few lines away, waiting for you to widen the search by five lines in either direction.

The error said "Action failed." That was all it said.

Alex Mitu — Wed, 22 Jul 2026 12:57:30 +0000

A nightly sync job processes a few thousand records against a third party API. It runs fine for weeks. Then one night, one record throws Operation failed and the job stops there. No stack trace, no field name, no hint about what actually went wrong. Every other record in the batch went through without a problem.

If you've worked with any API that talks to a system you don't control, you've hit this exact wall. The error is real, the failure is real, and the message tells you almost nothing. This is not really about that one error string. It's about what to do when a vendor decides vague is safer than specific.

Vague errors are usually on purpose

It's tempting to assume the API author was just being lazy. Sometimes that's true. But a lot of the time, generic errors are a deliberate choice: security teams don't want to leak internal state through error messages, and support teams don't want users acting on assumptions that turn out to be wrong. Action failed protects the vendor from a thousand support tickets built on a misread error code.

That reasoning makes sense from where they're sitting. It doesn't make the debugging session any shorter from where you're sitting.

Step one: stop trusting the summary

Most tools that call an API don't show you the raw response. They catch it, wrap it, and print something like Job failed for item 4821. That wrapper is doing you a disservice the moment something unusual happens, because it's discarding the one thing you actually need: the body of the response the server sent back.

Before doing anything else, find where the raw HTTP response is logged, or reproduce the call yourself directly against the API:

curl -i -X POST "https://api.example.com/v1/items/4821/sync" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @item-4821-payload.json

Nine times out of ten the raw response has more in it than the tool ever showed you. A status code, a nested error.details object, sometimes even the exact field that failed validation. The wrapper just never surfaced it.

Step two: isolate down to one unit

If the failure is buried inside a batch job, don't try to debug the batch. Pull out the single item that's failing and run it on its own. This sounds obvious written down, but under time pressure it's the step people skip, because rerunning the whole job "just to see if it happens again" feels faster than isolating.

It isn't faster. A batch run gives you noise: successful items, retries, unrelated warnings, timestamps that don't line up. A single isolated call gives you a clean before and after. Cut the batch out of the picture entirely and you can iterate in seconds instead of minutes.

If you're working from logs instead of live access, the same principle applies. Grep for the specific identifier, not the general error string. Searching for Operation failed gets you a wall of matches. Searching for the one ID that's actually broken gets you a narrow, readable slice.

grep -n "4821" service.log | grep -v "heartbeat"

Step three: resolve the opaque ID before anything else

A lot of "generic error, specific cause" situations turn out to be an identity problem in disguise. The ID in your log line is often internal, an integer, a GUID, a hash, something that means nothing to a human and nothing to the API's own documentation search. Before you theorize about causes, translate that ID into something you can actually reason about: a name, a path, a resource type.

Most APIs that use opaque IDs also expose a lookup endpoint for exactly this purpose. It's easy to skip this step because it feels like a detour from the "real" debugging. It isn't. Half the time, once you know that ID 4821 is actually "a shared mailbox with no owner" or "a file with a name 400 characters long," the cause explains itself.

Step four: write the raw response down, not your interpretation of it

Once you get the real response back, resist the urge to summarize it in your own words right away. Copy the actual payload into your notes first. Interpretations drift. Two days later you'll remember "something about permissions" when the actual field said insufficient_scope: files.write, and those are not the same debugging path.

This matters even more if you're about to hand the case to someone else, or escalate it. "It's probably a permissions issue" gets deprioritized. A pasted error body with the exact scope name attached gets picked up fast, because the next person doesn't have to reproduce your work to trust it.

What this actually is

None of this is specific to any one API or platform. It's the same four moves every time: stop trusting the wrapper, isolate to one unit, resolve the opaque identifier, and preserve the raw evidence before you translate it into a theory. The tools change. The order doesn't.

The annoying part is that this process feels slow the first time you do it on a new system, and fast every time after. Once you know where a particular API hides its real error body, or which lookup endpoint turns an ID into a name, the next generic failure takes you five minutes instead of an afternoon. That's really the whole payoff: you're not getting better at guessing, you're building a small map of exactly where each system likes to hide the truth.

So next time something throws you a wall of nothing, resist the urge to just retry it and hope. The information is almost always there. It's just sitting one layer below where the error message stopped looking.

Why Your EWS Impersonation Suddenly Stopped Working (And It's Probably Not Throttling)

Alex Mitu — Fri, 10 Jul 2026 18:17:12 +0000

Two months ago I picked up a ticket that looked routine: a job that reads mailbox data from Microsoft 365 through EWS, running fine for over a year, started failing on a subset of mailboxes in one tenant. Same app registration, same code path, same service account. The error in the logs pointed at throttling, so that's where the admin had already spent three days looking. Wrong direction. The actual cause had nothing to do with throttling budgets.

This mix-up happens constantly right now, and it's worth understanding why, because the fix for one problem does nothing for the other, and chasing the wrong one wastes days.

TL;DR: if your EWS failures don't scale with request volume, stop tuning throttling and go check your Application Access Policy scope groups instead.

What EWS throttling actually looks like

Exchange Online throttles EWS the same way it always has: budget-based. Every account gets a policy (the default is EwsDefaultThrottlingPolicy, but plenty of tenants layer custom ones on top) that tracks a slowly-refilling budget rather than a simple call count. When you overspend it, you get back a 503 or 429 with an X-MS-Diagnostics header telling you which budget got exhausted, usually the connection count or the concurrent-request limit.

The tell for real throttling is consistency. It scales with load, correlates with concurrency and batch size, and clears up within minutes once you back off. If you graph failure rate against request volume, you'll see a clean relationship. If you double your batch size, failures increase. If you throttle yourself proactively (respecting Retry-After, staying under EWSFindCountLimit for FindItem calls), it mostly goes away.

That correlation is the whole diagnostic test. If your failures don't scale with volume, you're not looking at a throttling problem, no matter what the error message on the surface says.

What actually changed

Over the past year or so, Microsoft tightened enforcement in two places that both produce errors easy to mistake for throttling.

First, Application Access Policies. New-ApplicationAccessPolicy has existed for years, letting admins scope which mailboxes an app registration can touch instead of granting it impersonation rights over the entire tenant. It used to be optional, something only security-conscious tenants bothered configuring. That's changed. More tenants are applying scoped access policies as a default hardening step, often as part of a broader Conditional Access or Secure Score push, and they're doing it without touching the app's actual permissions or the service account's RBAC role. The app still has ApplicationImpersonation. It just can't reach mailboxes outside its assigned scope group anymore, and mailbox group membership drifts constantly as people get added, removed, or reassigned.

Second, the long tail of Basic Auth deprecation. Most environments moved off Basic Auth for EWS years ago, but I still see tenants with a leftover service principal or a scheduled task somewhere using a cached token flow that quietly stopped working when Microsoft closed another enforcement gap. It's rarely the primary auth path failing outright. It's usually one forgotten integration that everyone assumed was already migrated.

Both of these produce intermittent, mailbox-specific failures. Neither has anything to do with request volume.

Telling them apart

Here's the sequence I actually use now before touching any throttling knobs.

1. Check whether failures correlate with specific mailboxes, not specific times. Pull the failing mailbox list and check it against the policy's scope group:

Get-ApplicationAccessPolicy | Format-List AppId, PolicyScopeGroupId, AccessRight

Get-DistributionGroupMember -Identity "EWS-Scoped-Mailboxes" |
    Select-Object PrimarySmtpAddress

Compare the mailbox list this returns against the mailboxes that are actually failing. Anything missing from the group is out of scope, and that's your answer.

2. Read the actual error code, not just the HTTP status. These two failures look almost identical if your logging only captures the status code:

# Real throttling
HTTP/1.1 503 Service Unavailable
X-MS-Diagnostics: 3006;reason="EWSMaxConcurrency budget exhausted";error_category="transient"

# Access policy scoping
HTTP/1.1 403 Forbidden
<S:Fault>
  <S:Code><S:Value>Sender</S:Value></S:Code>
  <S:Reason><S:Text>ErrorAccessDenied</S:Text></S:Reason>
</S:Fault>

If your logging layer only captures the HTTP status and not the response body, fix that first. You're throwing away the one piece of information that tells these two apart.

3. Check the Microsoft 365 audit log around the date the failures started, not the date they were noticed:

Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-30) -EndDate (Get-Date) `
    -Operations "New-ApplicationAccessPolicy","Set-ApplicationAccessPolicy","Remove-ApplicationAccessPolicy" |
    Select-Object CreationDate, UserIds, Operations

I've found the actual change was made three weeks before anyone noticed, during a routine security review nobody connected to this app.

4. Confirm the app's permission model.

Get-ManagementRoleAssignment -RoleAssignee "svc-backup-app" |
    Where-Object { $_.Role -eq "ApplicationImpersonation" }

If this returns a result, the app can impersonate any mailbox in the org by design, unless an access policy is actively narrowing that down. Worth knowing before you assume the access policy is even the right suspect.

If none of that turns anything up and failures genuinely track with request volume and clear up after backing off, then fine, it's real throttling, go tune your batching.

Fixing the actual problem

If it's an access policy issue, the fix is usually mechanical but tedious: audit the scope group, figure out why the mailbox fell out of it, and set up something that doesn't rely on someone remembering to update a static group every time a mailbox changes hands. A dynamic distribution group tied to whatever attribute actually determines "should this app reach this mailbox" beats a manually maintained list every time:

New-DynamicDistributionGroup -Name "EWS-Scoped-Mailboxes" `
    -RecipientFilter {RecipientTypeDetails -eq "UserMailbox" -and CustomAttribute1 -eq "backup-enabled"}

Point the access policy at this group instead of a static one. Membership now updates itself whenever CustomAttribute1 changes, instead of depending on someone remembering to edit a list.

The bigger fix, though, is getting off tenant-wide impersonation entirely. Microsoft has been pushing RBAC for Applications as the replacement, and it's worth the migration even outside of this specific problem. Instead of one service account impersonating anyone, you assign granular, resource-scoped permissions directly to the app registration:

New-ServicePrincipal -AppId <your-app-id> -ObjectId <object-id> -DisplayName "Backup-App"

New-ManagementScope -Name "Backup-App-Scope" `
    -RecipientRestrictionFilter {CustomAttribute1 -eq "backup-enabled"}

New-ManagementRoleAssignment -Role "Application Mail.Read" -App <your-app-id> `
    -CustomRecipientWriteScope "Backup-App-Scope"

This replaces a single tenant-wide ApplicationImpersonation grant with a role scoped to exactly the mailboxes the management scope defines. Exact cmdlet parameters shift a bit between Exchange Online PowerShell module versions, so treat this as the shape of the migration and confirm current syntax against Microsoft's docs before running it against production. It's more setup work up front, but it means you stop being exposed to "someone tightened a policy somewhere and now half your mailboxes silently fail" as a category of incident. Given that EWS itself is on Microsoft's retirement roadmap in favor of Graph API, this is also the direction you'd be migrating toward anyway. Doing the permission model migration first, before the API migration, splits the work into two smaller, less risky changes instead of one large one.

One more thing worth doing regardless: separate your retry logic by error type. A lot of applications treat every failed EWS call the same way and retry with exponential backoff, which is correct for a 503 throttling response and actively harmful for a 403 access-denied response:

def should_retry(status_code, error_code):
    if status_code == 503 or error_code == "ErrorServerBusy":
        return True   # transient, back off and retry
    if status_code in (401, 403) or error_code == "ErrorAccessDenied":
        return False  # a permission problem, retrying won't fix it
    return False

Retrying an access-denied error just burns your throttling budget on calls that were never going to succeed, and can end up creating the throttling problem you originally thought you had.

The pattern underneath this

None of this is exotic. It's the same lesson as always: a generic error handler will happily lump together two unrelated failure modes under one log line, and the fix for each one does nothing for the other. The throttling ticket that isn't actually about throttling has become common enough in the last year that I've started assuming access policy scope first and traffic volume second, and I've been right more often than not.