DEV Community

y4u
y4u

Posted on Originally published at uvp.y42u.net

The First Hours of an Intrusion Investigation — Four Steps Through Tens of Millions of Auth Log Lines

#security #devops #logging #tools

4,812 failures. One success.

All from the same IP. It could be an attack, or someone who forgot their password. You don't know yet.
So how do you decide whether to trust that one success — or fear it?

In the first hours after a suspected intrusion, logs are the only evidence you have. And auth logs run to tens of millions of lines even in peacetime. A very small number of meaningful lines, buried in an enormous number of ordinary ones — that is the entire difficulty of triage.

This article walks four steps in order: find the trace, strip the noise, confirm who the IP was, and close the blast radius. Each has a workable answer in grep and awk. And each has a point where that answer stops — and across all four, the operations being asked for collapse into the same three.

This is about what to look at first when you already have the logs in hand. Full incident response — isolation, preservation, disclosure — follows your organization's own procedures.

Up front: UwView Pro opens a 258.68 GB, 4.5-billion-line file readable from the first moment — then drill-down and sequence search let you follow the trail (measured on one setup; details at the end)

1. Finding a pattern in tens of millions of auth lines

Situation

Look at those 4,812 attempts more closely. Something equivalent to /var/log/auth.log, tens of millions of lines. Hundreds of thousands of them are Failed password. Tens of thousands are successes. Both appear every day as part of normal operation.

What you're looking for isn't a line. It's a shape: which IP, in which hours, at what ratio of failures to successes.

Why it happens

Auth log lines are independent, and the shape of an attack only exists in the relationships between them.

Failed password for admin from 203.0.113.44 is, on its own, not an anomaly at all. At one per second from a single IP at 3am, it is. Three times during business hours, it's a typo. The same string is normal or abnormal depending entirely on its context.

And attackers know this. Password spraying — a handful of attempts per IP, spread across many IPs — looks perfectly normal line by line. The whole instinct of "find the IP with lots of failures" simply misses it.

There's a second thing. What's actually unsettling about "4,812 failures, one success" isn't the count — it's the arrangement. Failures, then a success, then a key gets added, then a connection to another host. Only when failure streak → success → key added → lateral movement appears in that order does it take the shape of an intrusion. The same four events in a scrambled order are probably nothing at all. What you're looking for isn't a set of terms; it's a sequence of them.

Command-line triage, and where it stops

Start with counts.

grep 'Failed password' auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20
grep 'Accepted publickey\|Accepted password' auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
# The other direction: few attempts per IP, but the same targeted usernames
grep 'Failed password' auth.log | awk '{print $(NF-5)}' | sort | uniq -c | sort -rn | head
Enter fullscreen mode Exit fullscreen mode

That third one earns its keep. Counting by IP makes a distributed attack disappear; counting by targeted username makes it reappear. Changing the aggregation axis changes what you can see, and that's the standard move in this domain.

Once you have a lead, you read around it.

grep -n -C 5 '203\.0\.113\.44' auth.log | less        # five lines of context either side
grep -C 5 '203\.0\.113\.44' auth.log | grep -n 'Accepted\|new key\|sudo'  # re-search inside that context
Enter fullscreen mode Exit fullscreen mode

grep -C N is the single most-used tool in triage. What one line can't settle, five lines either side usually can. And then, as in the second command, you narrow again inside that context with another term — that's the real-world pipeline.

It stops in three places. First, you can't know the right N in advance. ±3 isn't enough; ±50 is unreadable. The right N differs per investigation and changes within one investigation (tight while narrowing, wide when you finally read). And every change of N makes grep -C re-read gigabytes from the top.

Second, piping destroys the original line numbers. The grep -n in that second command is numbering lines within the context blocks flowing through the pipe, not lines of auth.log. So "let me jump back to that line" isn't available. And narrowing is never one shot — you add a term, drop a term, try again — which means rebuilding the whole pipeline and re-reading the original each time.

Third, grep cannot express order. Try to find the failure streak → success → key added → lateral movement from the previous section and grep -E 'Failed|Accepted|new key' just returns lines containing any of the three; it never looks at the arrangement. You can write a state machine in awk — but writing a throwaway script for every lead doesn't match the pace of triage. So the ordering judgment ends up being made by eye, and the "hit list versus the scene" round trip from part 2 repeats dozens of times. When the original is several gigabytes, every return costs you a wait.

2. Separating a runaway crawler from an attack

Situation

Load spikes one night. In the access log, requests to a particular path are up by orders of magnitude. Is it an attack, or a misconfigured crawler hammering the same page?

Get it wrong in one direction and you block a legitimate search engine and lose your index. Get it wrong in the other and you wave off genuine probing as "probably a bot."

Why it happens

Bots and attacks look almost identical in an access log.

Both arrive from one IP or range, at high frequency, at mechanical intervals, following URL sequences no human would. The difference is intent, and intent isn't in the log.

The User-Agent is a hint, but it's self-declared, so its trustworthiness varies. A request claiming to be Googlebot isn't necessarily from Google; confirming it takes a reverse DNS lookup followed by a forward one. Conversely, a legitimate bot on an old UA string can look suspicious for no reason.

The nastier case is when both are happening at once: a crawler is driving the load, and under cover of that noise a different IP is probing. Close the ticket on the first one and you never see the second.

Command-line triage, and where it stops

Aggregate to get your bearings.

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20   # UA distribution
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20         # IP distribution
# Verify a claimed identity (one at a time)
dig +short -x 66.249.66.1 | xargs -r dig +short
# Exclude known bots, then re-count what's left
grep -v -E 'Googlebot|bingbot|Applebot' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head
Enter fullscreen mode Exit fullscreen mode

That last step — exclude the known, then re-count — is the single most useful move in practice. Drop the noise and a second peak you couldn't see appears. And in reality you do it repeatedly: exclude by UA, then exclude the remainder by IP range, then narrow by path, then narrow by hour. Narrowing isn't one search; it's a stack of stages.

It stops in three places. First, that stack can only be expressed as one pipeline. Get to stage four, suspect stage three was too aggressive, and you rewrite the pipeline and run the whole thing again. The per-stage counts don't survive either, so you can't see where the volume dropped. Something like "94,979 → 184 → 54" tells you instantly which filter did the work — but a pipeline's intermediate results flow past and vanish.

Second, exclusion is destructive. The intermediate file grep -v produced doesn't contain the lines you removed. The moment you think "actually, put the crawler's window back," you rebuild from the original. And exclusion criteria change constantly during triage: add a UA, drop an IP range. Each change re-reads several gigabytes.

Third, a case that always arrives: you want the filtered view and the full view at the same time. Put "bots removed" next to "bots included" and check whether the peaks line up. grep -v only ever leaves you holding one of the two.

3. "Is that IP internal?" — reconciling with lease logs

Situation

An internal address, 10.20.30.44, shows up in the access log. Your manager asks whose machine it is.

A private IP identifies nobody on its own. Because it's handed out by DHCP, 10.20.30.44 on August 14th and 10.20.30.44 on August 20th may well be two different people.

Why it happens

The mapping between IP and person changes over time, and that history lives in a different log.

The information you need is split across two files. The access log knows "when, from which IP, what was done." The DHCP server log knows "when, which IP, leased to which MAC address." Neither answers the question alone. And a few seconds of clock skew between the two servers is enough to flip which host a lease boundary belongs to.

Command-line triage, and where it stops

The naive approach is to pull from both and read them side by side.

grep '10\.20\.30\.44' access.log | head -50
grep '10\.20\.30\.44' dhcpd.log | grep -E 'DHCPACK|DHCPRELEASE'
# Merge by time into a single narrative (column positions vary by environment)
sort -m -k1,2 <(grep '10\.20\.30\.44' access.log) <(grep '10\.20\.30\.44' dhcpd.log) | less
Enter fullscreen mode Exit fullscreen mode

That merge is powerful. Once it's one timeline, you can follow by eye whether the lease change came before or after the requests.

And here too, the thing you're really adjudicating is order: DHCPACK (lease granted) → the request in question → DHCPRELEASE (lease returned). In that order, the request falls inside that lease. Break the order — the request precedes the DHCPACK — and it belongs to the previous holder. That's what "one minute flips the conclusion" actually means.

It stops in three places. First, the timestamp formats don't match. Mix syslog style (Aug 31 04:12:07) with ISO 8601 (2026-08-31T04:12:07+09:00) and sort won't order them sensibly, so you preprocess — and preprocessing erases the original line numbers, which is exactly the "normalize and you can't get back" problem from part 5.

Second, the DHCP log usually lives on a different machine and is often written in a different encoding. Logs from Windows-side appliances arriving in a legacy code page is routine in many environments; the triage procedure for that is part 3.

Third, all you actually wanted was to look at two files side by side, and merging isn't that. Merging collapses them into one stream, so you lose track of which line came from which file. What you wanted was both files open at once, scrolled to the same window of time.

4. Sweeping departed-employee accounts, exhaustively

Situation

Every suspected intrusion generates the same homework: "are there accounts we forgot to disable?"

Leavers, transfers, test accounts, contractors. Dozens of names, months of logs. What's being asked for here isn't an interesting finding — it's the absence of gaps.

Why it happens

To prove that nothing turned up, you have to look at everything.

An ordinary outage investigation ends when you find one answer. An audit sweep is the inverse: you go through the whole set in order to confirm there's nothing there. Miss one name and that one might have been the one.

And the candidate list almost always lives outside the logs — an HR roster, a list of separation dates, a column of account IDs handed over in a spreadsheet. The log says jsmith; the roster says "John Smith (left 2026-05-31)." A human does that mapping.

Command-line triage, and where it stops

Given a list, you can loop.

# Last activity for each departed account
while read -r u; do
  last=$(grep -h "user=$u" auth.log* | tail -1)
  printf '%s\t%s\n' "$u" "${last:-(no record)}"
done < leavers.txt

# Only activity after the separation date
grep -F -f leavers.txt auth.log | grep 'Accepted' | awk '{print $NF, $0}' | sort | head -50
Enter fullscreen mode Exit fullscreen mode

grep -F -f listfile is the workhorse here: dozens of IDs in one pass, matched as fixed strings so nothing misfires as a regex.

It stops in two places. First, one line isn't enough to decide anything. "Successful login for jsmith on 2026-06-12" could be an intrusion, legitimate work during offboarding, or a different account that happens to share a name — and you can't tell without reading around that line. You'll want to add grep -C here too, but emitting context inside a loop over dozens of names produces thousands of lines, and now reading that is the job. What you want is to move back and forth between the hit list and the neighbourhood of a hit — not one flat text with both mashed together. So you go back to the original, dozens of times, once per name.

Second, this work spans days. A sweep doesn't finish in an afternoon. Tomorrow you reopen the same log, try to recall how far you got, and rebuild the same filters. The "investigation state lives outside the tool" problem from part 5, multiplied by the number of names on the list.

What all four steps had in common

Step How it shows up The operation you wanted Where it stops
Find a pattern across IPs The trace is in the arrangement, not the set ±N context ＋ search by order grep -C re-reads on every N; grep can't express order
Separate bots from attacks Narrowing becomes a stack of stages Stacked narrowing A pipeline keeps no intermediate state; changing one stage restarts it
Confirm who an IP was The verdict is ACK → request → RELEASE ±N context ＋ search by order You can't follow both files at once
Sweep departed accounts One line decides nothing; you must read all Moving between hit list and neighbourhood Back to the original per name, for days

None of these are blocked on analysis technique. Look at the third column and the operations being asked for collapse into just three.

  • See ±N lines of context, with a variable N. One line settles nothing. It settles once context is attached — and the right amount of context is different every time.
  • Stack the narrowing, and see the count at each stage. One search is never enough. When the drop is visible — 94,979 → 184 → 54 — you can see which condition did the work.
  • Search by order. A success after a streak of failures; a request after a lease was granted. The trace of an attack lives in the arrangement of terms, not the set of them.

All three are things anyone does without thinking on a small file. At a few gigabytes, grep -C demands a full re-read just to change N, a pipeline throws away everything between stages, and judging order falls back to a throwaway script or your own eyes. The operations aren't hard; size is what stops them being worth their cost.

Security triage feels this hardest, because you change angles constantly on the assumption that most hypotheses are wrong. Count by IP — nothing. Count by username — something. Widen the context. Drop the bots and count again. Even at tens of seconds each, that adds up over a few hours of triage. And what it adds up to is one more hypothesis you never got around to testing.

This shape has recurred throughout the series: part 1's tools that demand a full read before showing anything, part 2's round trip between the hit list and the scene, part 4's trade of size against readability. Auth logs and access logs get stuck in exactly the same places outage logs do. The only difference is what it costs to be wrong.

The tool I use

UwView (free), which I develop, is a viewer built to drop the assumption that size makes that round trip expensive. Even an auth log of tens of millions of lines displays, scrolls, and searches from the moment it opens; the index is built in the background and line numbers appear when it completes. Search for the IP your aggregate flagged, then read the lines around the hit — the work in sections 1 and 4 continues without ever closing the file. Highlights add colour without removing lines, which is exactly the "exclude it but don't delete it" of section 2. It never writes to the original — the first principle of forensics.

The three operations listed above — ±N context, stacked narrowing, and search by order — are exactly what UwView Pro adds.

  • Drill-down search: narrow a result list by another term, then another. Tabs show term (count), so section 2's "exclude by UA → exclude by IP range → narrow by path" runs with the count visible at every stage. From the second stage on, only the previous stage's ±N window is searched, so there's no wait. Changing your mind means going back to that tab — not rebuilding a pipeline. Here's the implementation write-up.
  • Sequence search: match only where w1 → w2 → w3 appear in that order. Section 1's failure streak → success → key added and section 3's DHCPACK → request → DHCPRELEASE become the query itself. A right-click "history" shows the path that actually matched — which line held which term — and you can jump to any of them. One honest note: each stage searches the body using the previous stage's position as a start point, so it takes about as long as a full-text search (unlike drill-down, where stages after the first are immediate). Here's the implementation write-up.
  • ±N is independent per stage: keep it at ±1 while narrowing, widen the final stage to ±10 to actually read — and changing N doesn't re-read the original.

It also saves the index and the compression, so from the second open onward the file opens instantly with line numbers — no more mornings that start with reopening. Compressed-cache search and ~1/9 storage come with it, so retention-mandated old logs cost you less disk (all OS, one-time or monthly). The free UwView covers single-stage search with ±1 context and result export; drill-down, sequence search, and the variable ±N are Pro features.

One honest limit: UwView is a viewer, not a SIEM and not a correlation engine. If your scale calls for automated cross-log correlation and alerting, that's a different product's job. UwView covers the step before it — looking at the raw log, as it is, with your own eyes, before anything ingests it.

Links

From the developer: My apps, Kindle books, and open-source projects are listed at GitHub: amru195704.

A note
The information in this article is provided for reference and is not guaranteed to be accurate or complete. Real incident response should follow your organization's procedures and, where appropriate, the guidance of a qualified responder. Log formats, field positions, and timestamp conventions vary widely by OS, middleware, and configuration. Command examples may need adjusting for your environment (GNU vs. BSD, your shell, the awk implementation, your log's field layout, etc.). All IP addresses shown are documentation-range examples. If you find an error or inaccuracy, please point it out in the comments and it will be corrected after verification.

Top comments (0)