DEV Community

y4u
y4u

Posted on Originally published at uvp.y42u.net

"I Ordered It and It Never Arrived" — Four Support-Desk Log Investigations

#logging #devops #sre #support

One order number. That's the whole lead.

The ticket says nothing else. The record of what happened is scattered across five systems. Can you put it back into a single line before the customer gives up on you?

The answer isn't a faster search.

Support-desk log work is a different sport from incident response. An incident ends when you know what happened. A support ticket has to end with a sequence of facts about one specific case, written so a person can read it — and someone is waiting while you assemble it.

Four situations: a delayed e-commerce shipment, a cheating report in a game, a CDR reconciliation at a telco, and a multi-tenant SaaS log with no tenant separation. Different industries; all four jam on the same single point.

Up front: UwView searches the whole file from the moment it opens, and drill-down search lets you narrow a result and then narrow that — following "order ID → payment ID → waybill number" without ever extracting from the original. With Pro, a 47.73 GB file you have already opened reopens in 0.02–0.07 s with line numbers intact (measured on one setup; results vary — details at the end)

1. E-commerce: you have the order number and nothing downstream of it

Situation

"The item I ordered last week never arrived." The only concrete thing in the message is the order number ORD-2026-0831-88213.

You grep the order API log. One line. The order was accepted. You grep the payment log for the same order number. Zero results. The payment log doesn't carry order numbers.

Why it happens

IDs change hands at every system boundary.

The order API owns the order number. Hand it to the payment provider and it becomes their transaction_id, which is what comes back. Inventory allocation works in SKUs and reservation IDs; the warehouse works in shipment lots; the carrier integration works in waybill numbers. A design that carries the order number all the way through is rarer than you'd hope.

So the investigation is inherently multi-stage. The order number gives you a payment request ID; that gives you a transaction_id; that gives you a waybill number; that finally gets you into the carrier API log. Each search's output is the next search's input.

On top of that, e-commerce logs are usually split by date. Order on 8/31, shipment on 9/2, carrier handoff on 9/3. Four searches against four different files.

What general tools do, and where they stop

Four commands, run in order.

grep 'ORD-2026-0831-88213' order-2026-08-31.log
grep 'req-8f2a11' payment-2026-08-31.log            # ID from step 1
grep 'txn_9931882' shipping-2026-09-02.log          # ID from step 2
grep 'DENPYO-4410-2299' delivery-2026-09-03.log     # ID from step 3
Enter fullscreen mode Exit fullscreen mode

Written out, it's four lines. The cost is in the time it takes to write them.

Three limits. First, it's strictly sequential. Read the output, spot the ID, copy it, compose the next command. Four times. If each pass scans several GB, the waiting stacks in series too — and it stacks while someone is waiting on you.

Second, the output scrolls away. By stage four you'll want the payment timestamp from stage two again, and you either hunt through scrollback or rerun the grep. The "two minutes to reopen" from Part 10 happens several times inside a single ticket here.

Third, none of it is in a shape you can send. The reply, or the escalation ticket, needs to read: order accepted 8/31 14:22, payment authorised 14:22, shipping instruction 9/2 09:14, no record of carrier handoff. Reassembling four terminal outputs into that by hand is a second job waiting after the first.

2. Games: the cheating is real, but it isn't on any single line

Situation

A player report comes in: "this account is obviously cheating." You have a player ID and a rough time window.

The action log runs to hundreds of millions of lines a day. You grep the player's lines out. A few thousand of them. Nothing looks wrong.

Why it happens

The evidence is in the sequence of lines, not in any one line.

A single position update looks fine. What looks wrong is the difference from the line before it — 300 metres covered in 0.2 seconds. Or attack intervals that are more uniform than a human hand can produce. Or an effective input rate, net of packet resends, above the documented ceiling. All of these are relations between adjacent lines.

And there's a second trap: extracting only that player's lines destroys your ability to judge. You need to know what everyone else on that map was doing at that moment. If every player was teleporting, it was server-side lag, not a cheat. The moment you extract, that comparison is gone.

Reports arrive by the dozen every day. Re-entering the same day's log with a different player ID, over and over, is the normal shape of the work.

What general tools do, and where they stop

Extract, and keep some context.

grep -n 'player=88213' action-2026-09-05.log > case.txt
grep -n -C 20 'player=88213' action-2026-09-05.log | less    # ±20 lines
awk '$1>="14:20:00" && $1<="14:25:00"' action-2026-09-05.log | less
Enter fullscreen mode Exit fullscreen mode

The -C in line two genuinely helps: other players' lines come along, so you keep something to compare against.

Three limits. First, you have to pick the context width before you look. Twenty lines is too few; a hundred is more than you can read. The right width is only knowable once you're inside the data, and grep wants the number up front. Changing your mind means another full scan of hundreds of millions of lines.

Second, every report costs a full scan. Thirty reports in a day means thirty passes over the same file. The log you decided to keep back in Part 4 turns out to charge you the same wait every single time you consult it.

Third, the extracted files pile up. One case.txt per report. Three days later nobody remembers which file belongs to which ticket — and each one contains other players' activity, so each one is a copy that needs handling with care.

3. Telco: all you want is to compare two enormous files

Situation

A call that appears in the billing data has no matching CDR (call detail record) from the switch. And there are calls the other way round.

Both sides run to tens of millions of records. One is a proprietary fixed-width format, the other is CSV. Neither file answers the question on its own.

Why it happens

Reconciliation is a set difference, and the keys don't match exactly.

For the same call, the switch records the start time to the second while billing rounds to the minute. The calling number is +81-90-... on one side and 090... on the other. One is UTC, the other JST. Before you have a key, you need a normalisation rule.

That rule is almost never right the first time. "Truncate the seconds and reconcile" produces 80,000 mismatches; maybe it should have been rounding; start over. Every change to the rule reruns tens of millions of records from the beginning.

And there's what always comes next: "let me see what surrounds that one record." Once a call is confirmed missing on one side, you want to read the switch log around that timestamp. But a file that has been through sort no longer has its original order.

What general tools do, and where they stop

Normalise, sort, then join or comm.

awk -F, '{gsub(/[-+]/,"",$2); print $2"_"substr($3,1,16)}' billing.csv | sort > a.key
cut -c 1-11,20-35 cdr.dat | tr -d ' ' | sort > b.key
comm -23 a.key b.key | head          # in billing, absent from CDR
join -t, -1 1 -2 1 a.key b.key | wc -l
Enter fullscreen mode Exit fullscreen mode

The approach is sound. The execution is where it bites.

Three limits. First, sort needs scratch space. Sorting tens of millions of lines wants a working area in /tmp roughly the size of the input. No space left on device partway through a reconciliation is a classic. -T moves it elsewhere, at a speed cost.

Second, retries aren't cheap. A one-character fix to the awk normalisation sends tens of millions of records through again. Ten hypotheses means ten waits — and your appetite for hypotheses runs out before the data does.

Third, the ordering is gone and you can't get back to the original. comm returns keys, not positions; nothing tells you which line of the original a key came from. Answering "what was the switch doing just before this call" means opening the original separately and searching by time. The "jump by timestamp" work from Part 7 is a mandatory second phase here.

4. SaaS: covering for a log that was never tenant-separated

Situation

A multi-tenant SaaS. Customer A reports that one screen is extremely slow.

The application log is a single stream for every tenant. To produce something you can send to A, you have to investigate a log full of B's and C's lines.

Why it happens

tenant_id isn't on every line.

It's on the application-layer lines. It is not on the layers below them — ORM slow-query logs, runtime GC logs, load-balancer access logs, connection-pool exhaustion warnings. At best those carry a request ID.

And the cause of a latency problem usually lives in exactly those lower layers. What's actually behind A's slow screen might be B's overnight batch monopolising the connection pool. Filter on tenant=A and the cause disappears from view.

Layered on top is a constraint specific to support work: you have no choice but to look at other customers' lines during the investigation, but not one of them may end up in what you send. A single leaked line is an incident of its own.

What general tools do, and where they stop

Extract, then investigate the extract.

grep 'tenant=acme' app-2026-09-05.log > acme.log       # A's lines
grep -f req_ids.txt app-2026-09-05.log > acme_full.log # pick up by request ID too
grep -c 'tenant=' app-2026-09-05.log                   # how many lines even have it?
Enter fullscreen mode Exit fullscreen mode

Run the third one first. What fraction of the file carries tenant= decides whether this approach is viable at all.

Three limits. First, the lines you drop are the lines you needed. A filter that discards everything without tenant= reliably discards the slow-query log and the GC log. It throws away the leading candidates before the investigation starts.

Second, the extract becomes a new thing to manage. acme.log can't be sent until it's verified free of other customers' data — and verifying means reading all of it. Meanwhile acme_full.log, which pulled lines back in by request ID, is more likely to contain foreign lines, not less. Extraction meant to save work has manufactured verification work.

Third, the original line numbers are gone. Line 412 of the extract is not line 412 of the original. When you escalate to engineering without a coordinate on the original, they start their search from scratch. The "once you transform it, it isn't the original any more" point from Part 9 comes back here as an internal handoff cost.

What the four had in common

Situation The lead Shape of the investigation Where general tools stop
E-commerce delay one order number four stages, changing ID each time waiting stacks in series, stage by stage
Game cheat report player ID and a time compare against neighbouring lines context width must be chosen before you look
CDR reconciliation two files normalise, then diff every retry reruns the whole thing
SaaS tenant mixing a tenant name you want to filter but can't filtering drops the causal lines

Read the third column downward and it resolves. Not one of these ends after a single search. The previous result feeds the next one; you widen the window and look again; you change the normalisation and try again; you start doubting the filter itself. Support-desk log work is decided by how many times you can afford to search.

The fourth column agrees. General tools stop at the moment you think "once more, with a different condition." Whether the first search is fast matters less than whether the second, fifth and twentieth are cheap. And since tickets arrive all day, that repetition happens not just within one case but between cases.

Three things are needed.

  • Narrow a result, then narrow that — "filter by order number, then keep only the payment-related lines" stacked on the original without extracting. No extracts means no copies to manage and nothing to verify.
  • Narrowing that preserves coordinates on the original — every line of a result should still know which line of the original it is. In an escalation or a report, having a coordinate you can point at is the difference between someone reading it and someone redoing it.
  • Cheap reopening — opening the same file dozens of times a day is the job. The smaller each wait, the more hypotheses you can afford.

Back to the order number you started with. What decides whether you can trace it in five minutes isn't the speed of one search. It's whether you can fire four in a row, and whether redoing one costs you anything worth hesitating over.

The tool I use

I build UwView (free), a viewer that displays, scrolls and searches a huge text file from the moment it opens. Indexing runs in the background and line numbers appear when it completes (most other viewers show only the head until indexing finishes). Nothing is split and nothing is extracted: the original stays one file, unmodified. It has no edit mode, which makes opening a log containing other customers' data a safe operation in itself.

  • Stack a filter on a filter: drill-down search runs a new search against the results of the previous one. Section 1's "order number → payment request ID → waybill number" and section 4's "filter by A, then pull back in by request ID" both stack on screen, with no extract file created.
  • Match on order, not just on content: sequence search takes "B follows A" as the condition. That's what section 2's uniform intervals need — an anomaly no single line can express.
  • Move between the result list and the original: results stay in their own window, and any entry jumps to that line in the original. Because you land in the original, you can widen the surroundings as much as you like — section 2's "choose the context width before you look" problem never arises. More on this in why search results live in a separate window.
  • Make reopening cheap: UwView Pro persists the index and compression, so every open after the first comes back with line numbers already there (measured at 0.02–0.07 s on a 47.73 GB text file; varies by environment). Across section 2's thirty reports in a day, that difference multiplies by thirty.

To be straight about it: UwView is not a reconciliation tool. Section 3's "normalise two files' keys and take the difference" belongs to sort and join. This tool's turns are before that (checking raw column positions so you can decide the normalisation rule) and after it (reading the surroundings of one flagged record in the original). It does not compute the diff. It also does not aggregate — section 2's "distance divided by time" needs something else. What it does is find, follow, and show you the original unchanged. And there is no cross-file search: section 1's four files are four files you open one after another.

And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly). For support work, where you reopen the same log all day under different conditions, a cheap second open converts directly into tickets closed.

Links

From the developer: A full list of my apps, Kindle books and open-source work lives at GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. The system layouts, ID schemes and log fields described are generalised examples for explanation and do not represent any specific real service's implementation. CDR formats and tenant-identifier conventions vary widely between operators and products. Measured figures come from one specific environment and are not a promise of the same result elsewhere. Command examples may need adjusting for your environment (GNU vs BSD, awk and sort implementation differences, which grep options are supported). Handling of logs containing customer or third-party data should follow your organisation's own policies. If you spot an error or something inaccurate, please leave a comment and I'll check and correct it.

Top comments (0)