Your AI Agent Is Debugging Performance Like a Drunk Man. This Skill Teaches It the Brendan Gregg Method.
How I packaged the USE Method (Utilization, Saturation, Errors) and the TSA Method (Thread State Analysis) into a vendor-agnostic skill that turns Claude Code, Kimi Code, Codex, or any LLM agent into a methodical performance investigator — complete with root-cause analysis and evidence-backed postmortems.
It's 2 a.m. Latency on your checkout API just went 20×. You open your AI coding agent — Claude Code, Codex, whatever you run — and type: "the server is slow, figure out why."
And the agent does what agents do. It runs top. It runs df -h. Maybe ps aux | grep java. Three commands it happens to know, a shrug dressed up as analysis, and a confident suggestion to "restart the service and see."
There's a name for that, and Brendan Gregg coined it: the Streetlight Anti-Method — searching for the problem only where the light happens to be, because those are the tools you know. Its cousin is the Drunk Man Anti-Method: change things at random until the problem goes away. Most AI agents debug performance exactly like this, because nothing in their context tells them not to.
Meanwhile, the actual discipline of performance analysis has been solved for over a decade. Gregg's USE Method finds roughly 80% of server issues with 5% of the effort. His TSA Method decomposes where every thread's time actually goes. These are procedures — checklists, like the emergency ones in a flight manual — and procedures are exactly what an LLM agent executes beautifully, once you write them down.
So I wrote them down. Meet brendangregg-use-tsa-skill — a vendor-agnostic, MIT-licensed agent skill that gives any AI coding agent the full methodology: systematic performance troubleshooting, root cause analysis, and the postmortem report at the end.
What the Skill Actually Teaches
The core is two complementary methods, plus the supporting toolbox:
USE — Utilization, Saturation, Errors. Resource-oriented. For every resource — CPUs, memory capacity, network interfaces, storage I/O and capacity, controllers, interconnects, and the ones everyone forgets (mutex locks, thread pools, file descriptors, and cgroup/hypervisor limits) — the agent checks three things: how busy is it, how much work is queued behind it, and is it erroring. It's deliberately question-first: instead of starting from whatever metrics a tool happens to show and working backwards, you start from the complete list of questions and find the metrics to answer them. What can't be measured becomes a documented known-unknown instead of an unknown-unknown.
TSA — Thread State Analysis. Thread-oriented, USE's mirror image. For each thread of interest, split time into six exclusive states: Executing, Runnable, Anonymous Paging, Sleeping, Lock, Idle. Then investigate the biggest states first with state-appropriate tools. The killer heuristic: if more than ~10% of time is Runnable or Anonymous Paging, fix those first — latency states can be tuned to zero, and they're often the cheapest wins in all of performance engineering.
Around that core, the skill packs the rest of Gregg's toolbox: off-CPU analysis (why threads block, with stack traces), flame graphs (where on-CPU time goes, visually), workload characterization, drill-down analysis, the RED method for microservices, the 60-second Linux triage (the famous "60,000 milliseconds" checklist), 24 methodologies, 6 anti-methodologies to refuse, and the performance mantras for deciding the fix.
The Procedure Is the Product
Here's what happens when the skill is installed and you say "prod is slow, triage it":
Phase 0 PROBLEM STATEMENT What changed? Quantify it. Who's affected?
Phase 1 60s TRIAGE uptime, dmesg, vmstat, mpstat, pidstat,
iostat, free, sar ×2, top — errors first
Phase 2 USE SWEEP every resource: utilization, saturation, errors
Phase 3 TSA SWEEP every thread: executing/runnable/paging/
sleeping/lock/idle
Phase 4 DRILL DOWN flame graphs, off-CPU stacks, time division
Phase 5 ROOT CAUSE causal chain, every link evidence-backed
Phase 6 FIX & VERIFY re-measure with the SAME instruments
Phase 7 REPORT triage note / RCA / postmortem
The golden rules baked into SKILL.md are what change agent behavior: diagnose before touching anything; installed ≠ working; rule out alternatives explicitly and keep the evidence; quantify everything; the first bottleneck you find may be a problem, not the problem; be candid about what you can't measure; absolute dates in all reports.
A Real Shape of a Real Investigation
The repo includes a worked example postmortem — a fictional-but-realistic incident that shows the expected depth. Short version:
API p99 goes 95ms → 1.9s after a 14:02 deploy. The naive agent says "the deploy broke the code, roll back." The methodical agent works the phases. The 60-second sweep exonerates the host: CPU 48%, no iowait, zero retransmits, clean dmesg. TSA on the application threads shows 61% of time in Runnable — threads ready to run, waiting for a CPU — on a half-idle host. That combination is the signature of exactly one thing: a resource-control limit. Two cat commands later: cpu.max allows 1.5 CPUs, and cpu.stat nr_throttled is climbing 54,000 times per minute. The deploy had doubled the replica count; nobody resized the cgroup quota.
Root cause: not the code, not the host — the quota. Fix: raise the limit. Verified: nr_throttled flat at zero for 72 hours, p99 back to 110ms under a 1.4× load test. And the postmortem records what was ruled out — host CPU, memory, disk, network — each with the command and output that exonerated it.
That last part is what separates an RCA from a guess. The skill's report templates — triage, RCA, and postmortem — demand an evidence table where every claim maps to a command and its output, and an investigation log where dead ends stay on the record.
An Honest Word About the Sources
Everything in the skill is an original distillation — with attribution — of Brendan Gregg's publicly published work: the USE and TSA method pages, the USE Linux checklist and Rosetta Stone, the off-CPU analysis and flame graph pages, the methodology list, and Systems Performance: Enterprise and the Cloud.
One discovery from the research worth sharing: Gregg's OS-specific TSA checklists were never actually published. His tsa-linux.html page has been an "Under construction" stub since 2014 — I verified it against a decade of Wayback Machine snapshots. So the TSA-on-Linux instructions in the repo are a labeled reconstruction, assembled from his TSA method page, the USE-Linux checklist, and his off-CPU work — and marked as such, because a skill that preaches "evidence over guessing" has to hold itself to the same standard.
Install It in 30 Seconds
The skill is a plain directory with a SKILL.md — no build, no dependencies, nothing executable. Pick your agent:
# Claude Code
git clone https://github.com/thecsdoctor/brendangregg-use-tsa-skill \
~/.claude/skills/brendangregg-use-tsa
# Kimi Code
git clone https://github.com/thecsdoctor/brendangregg-use-tsa-skill \
~/.kimi-code/skills/brendangregg-use-tsa
# OpenAI Codex
git clone https://github.com/thecsdoctor/brendangregg-use-tsa-skill \
~/.codex/skills/brendangregg-use-tsa
# Universal (agentskills-compatible agents)
git clone https://github.com/thecsdoctor/brendangregg-use-tsa-skill \
~/.agents/skills/brendangregg-use-tsa
OpenClaw discovers it from your workspace skills/ directory; for Gemini CLI, Cursor, Windsurf, and Aider the README has per-tool instructions — and for any other LLM, pasting SKILL.md into the system prompt works too. Then just talk to it: "this box feels slow, triage it and tell me what you ruled out."
Why This Matters Now
AI agents are being handed production access faster than they're being handed engineering discipline. The gap between "can run commands" and "knows how to investigate" is exactly where 3 a.m. incidents get longer, rollbacks get riskier, and postmortems get vaguer. Agent skills — reusable, versioned, shareable instruction packs — are how that gap closes: not by making models smarter in general, but by giving them the specific procedures senior engineers carry in their heads.
Performance analysis is the perfect test case, because the methodology literature is explicit that guessing is the failure mode. Gregg's line on methodology-free analysis: it "can become a fishing expedition, where metrics are examined ad hoc, until the issue is found – if it is at all." The skill's job is to make the fishing expedition structurally impossible.
Star it, fork it, tear it apart: github.com/thecsdoctor/brendangregg-use-tsa-skill. Contributions are welcome — especially per-OS checklist expansions and new worked examples. And the next time your agent suggests "restart it and see," you'll know exactly which skill it's missing.
Tags: AI Agents, Claude Code, LLM Tools, Performance Engineering, Root Cause Analysis, Brendan Gregg, USE Method, TSA Method, Linux Performance, Observability, SRE, DevOps, Incident Response, Postmortem, Flame Graphs, eBPF, Off-CPU Analysis, OpenAI Codex, Kimi Code, OpenClaw, Agent Skills, SKILL.md, Site Reliability Engineering, Debugging, Troubleshooting
Canonical links:
Top comments (0)