DEV Community

Bala Paranj
Bala Paranj

Posted on

Solve It Once: Kelsey Hightower's Talk Applied to Security Verification

✓ Human-authored analysis; AI used for formatting and proofreading.


Kelsey Hightower gave a talk at PlatformCon 2026 that was about the arc of a career, from running commands in SharePoint to writing Go tools that play music on your terminal.

The stories has an architecture principle that applies to how security verification should work. Solve the problem once, encode the solution as a reusable artifact, and never solve that problem again.

The Jira loop

He joined a company where deployments were driven by Jira tickets. Someone opens a ticket with deployment parameters. An engineer would read the ticket, copy the parameters, run the commands, paste the output back into the ticket, close it, and wait for the next one. Every hour, another ticket. Same process, commands and manual steps. The engineer became the loop.

He wrote a Puppet manifest that watches the tickets, extracts the parameters, runs the deployment, posts the output, and closes the ticket. The loop ran once as automation and then it was over. No engineer in the loop or ticket waiting for a human. The problem was solved, permanently, by encoding the solution into a reusable artifact.

Doing a repetitive manual process faster is not the right thing to do. Eliminate the loop by recognizing the abstraction hiding in the repetition and encoding it into an artifact that makes the manual steps unnecessary.

The substrate

This is the pattern that runs through every transition he describes. It's missed by most people when they talk about automation.

System administrators ran deploy.sh manually. Docker didn't automate typing apt-get install. Docker recognized that "application + dependencies + environment" was a repeatable unit. The container image became the substrate. Deployment stopped being a sequence of commands and became a declaration. The commands didn't get faster. They became unnecessary.

Operators placed workloads on servers manually. Kubernetes didn't automate SSH-ing into machines to check available memory. Kubernetes recognized that "place this workload on infrastructure meeting these constraints" was a schedulable unit of work. The pod spec became the substrate. Placement stopped being a human decision and became a computation. The human decisions became unnecessary.

Each time, an outsider saw the abstraction that insiders were too close to see. The outsider didn't build a faster version of the manual process. They built the substrate that eliminated it.

If you don't see the substrate, you'll build faster versions of the old process and call it innovation. You'll automate the Jira loop with a bot instead of writing the manifest that makes the loop unnecessary. You'll put an AI on the SSH session instead of building the scheduler that makes SSH irrelevant.

The security version of the same trap

A security engineer audits IAM policies across an AWS organization. They open each account, read each role's policy, check for wildcards, check for overpermissive managed policies, check for cross-account trust without conditions, file findings, track remediation, and do it again next week. Every week, same process, same checks, same categories of findings on different resources. The engineer is the loop.

A Reddit thread captured this:

"im actually losing my mind doing security audits this week.
looking at our internal accounts and it feels like every single
dev just slaps s3:* or literal AdministratorAccess on their
roles"

Fifty comments. Same pain from every angle. One engineer admitted to wildcarding permissions under deadline pressure. Another described spending 30 iterations at 10 minutes each trying to scope down a single role. Another asked for "an easy, out of the box, fast CI/CD PR check that tells you what permission errors you will receive before you get to prod."

Every commenter was stuck in the loop.

The current market response: "use AI to do the audit faster." Let Claude read the IAM policies. Let an AI agent review the CloudFormation templates. Let ChatGPT analyze the security group rules. Each one automates the manual review step without changing the abstraction. The AI reads the policy, infers the risks, generates prose, costs tokens. Next scan, different policy, same question, same inference, same tokens. A thousand policies, a thousand LLM calls. The same category, wildcard action with wildcard resource
is discovered a thousand times. Each discovery costs the same as the first. The LLM never remembers. The organization never accumulates. The token meter never stops running.

This is Kelsey's Jira loop at machine speed. The AI agent opens the policy, reads it, files a finding, closes the ticket, opens the next policy, reads it, files the same category of finding, closes the ticket. Faster. Same loop, cost structure and lack of accumulation.

The substrate hiding in the security process

The substrate is a formal specification evaluated against a configuration snapshot.

That's the abstraction hiding in the manual security review process. The same way the container image was hiding in the manual deployment process, the same way the pod spec was hiding in the manual placement process.

A Stave control is a solved problem compiled into a reusable artifact. CTL.IAM.ROLE.FULLACCESS.MANAGED.001 checks whether any IAM role has a full-access managed policy attached. It's a CEL predicate that is deterministic, zero tokens per evaluation, zero LLM calls, forever. The discovery that AdministratorAccess on a Lambda role is dangerous happened once. Maybe a human recognized it from a breach. Maybe an LLM helped surface it during authoring. The discovery was encoded into the control, and from that point forward, the loop is eliminated for that category.

The catalog with over 3000 controls is 3000+ loops that each ran once and became reusable artifacts. No LLM call will ever be made again to detect AdministratorAccess on a Lambda role. No human will ever manually check whether an S3 bucket policy grants cross- account access. No security engineer will lose their mind looking at the same wildcards every week. Each control makes the manual step structurally unnecessary.

The token economics are stark:

An AI agent approach: 1,000 accounts, 100 roles each, one LLM call per role per scan, weekly scans — 5.2 million LLM calls per year. Each call re-discovers what a CEL predicate already knows. The same finding, regenerated from scratch, every time, at full token cost.

Stave's approach: 3,000+ predicates evaluated against observations. Cost: milliseconds of CPU. Zero tokens or API calls. The same findings, deterministic, every run. The marginal cost of detecting the same category on the thousandth role is identical to the first: zero.

Train your own model

His closing line: "make sure you train your own model."

He means the mental model. The pattern recognition that practitioners build through decades of incidents, audits, misconfigurations, and fixes. The accumulated judgment that lets a senior engineer look at a configuration and see the attack path that a junior engineer misses. That model is the most valuable thing in the profession, and it lives in one person's head.

The control catalog is that model externalized. Each control encodes a pattern that a practitioner learned from a breach, an audit, a public incident report or a Reddit thread where someone described the pain. The sprint loop that produced the controls this session is the training loop: a gap audit discovers a new pattern, the Red-Green test proves the control catches it, a HAZOP validates the control itself. The model learns permanently. It never forgets, retires or fails to check because it's Friday afternoon.

The difference between Kelsey's trained model and the control catalog: his lives in one person's head and goes away when they retire at 42. The catalog is externalized, formalized, executable, and sharable. Every practitioner who runs stave eval inherits every pattern every contributor ever learned. The decades of experience, encoded once, applied by everyone, forever.

That's the multiplier. Not "AI reviews your policies faster." Instead: every insight, from every practitioner, compiled into a reusable artifact that runs at zero marginal cost, permanently. The model grows. The cost doesn't.

The SharePoint problem

Kelsey's other war story: deploy.sh was the wrong command. It was documented in SharePoint. He followed it, the deployment failed. The right command was this_one_works.sh. Nobody had updated the documentation.

This is the current state of cloud security knowledge. CIS benchmarks in PDFs. Best practices in Confluence. Security checklists written three years ago for services that have changed twice since. A practitioner follows the checklist, checks every box, and still has a misconfigured environment because the checklist doesn't cover the compound path between IAM and network that was discovered last month.

The substrate replaces SharePoint-as-security-knowledge with executable specifications. The control isn't a document someone might read and might follow. It's a predicate that evaluates against every configuration, every time, without anyone remembering to check. There's no this_one_works.sh because there's only one command: stave eval. The knowledge is the execution. If the knowledge is wrong, the execution is wrong, and the Red-Green test catches it before it ships. SharePoint couldn't tell you deploy.sh was wrong. The test suite tells you the control is wrong before anyone depends on it.

The Unix pipeline

Kelsey's progression: learn one command at a time → compose commands with pipes → create your own commands. He didn't jump from SharePoint to writing Go tools. Each step built on the previous one.

The same progression applies to the Stave adoption path:

# Learn one command
iam-explain role.json

# Compose with pipes
iam-explain role.json --output obs | stave apply

# Create your own
# Write a custom control for your organization's specific pattern
Enter fullscreen mode Exit fullscreen mode

Each step builds trust through demonstrated value. The practitioner who starts with iam-explain checking one policy. The same way Kelsey started with find listing files in a directory, graduates to composing it with stave apply when one policy at a time stops being enough. The same way Kelsey graduated from single commands to find | xargs rm.

The pipe character is the transition from "automate the manual step" to "operate on the substrate." iam-explain alone automates the manual policy review. iam-explain --output obs | stave apply changes the abstraction. The output is structural findings from over 3,000 specifications evaluated simultaneously across every role in the observation set, including compound chains that exist only in the conjunction across roles.

Each tool follows the Unix principle that survived 50+ years: do one thing well, compose through a universal interface, stay closed for modification. iam-explain parses IAM policies and emits observations. stave apply evaluates observations against specifications. Neither modifies the other. Neither needs to know the other's internals. The obs.v0.1 JSON format is the text stream between them. It is inspectable at the pipe junction, the same way grep | sort | uniq -c is inspectable at every stage.

The convergence

I was staring at practitioners doing manual security audits. The same wildcards every week, the same compounds nobody could see, the same remediation loop that never converged and I built what they needed.

The architecture that emerged follows his principles anyway:

Solve it once, reuse forever. Each control encodes a discovery that never needs to be re-derived. The catalog is a library of solved problems. The marginal cost of applying the library is zero.

The substrate eliminates the loop. The specification evaluated against the snapshot is the abstraction that makes manual security review structurally unnecessary.

Train your own model. The catalog externalizes practitioner judgment into an executable, sharable, permanent artifact. The model grows monotonically. It never forgets.

The Unix pipeline. Small tools, composed through a universal interface, each doing one thing well, each independently verifiable. The pipe character is the adoption path.

Kelsey arrived at these principles from system administration to Docker to Kubernetes. I arrived from the security verification side, from manual audits to formal specifications to compound chain detection. Different starting points, domains and problems. Same destination.

The destination is the substrate: a formal, reusable, composable artifact that makes the manual process structurally unnecessary. For infrastructure, it's the container image and the pod spec. For security verification, it's the control catalog and the observation schema. For both, the principle is the same: solve it once, compile it into the library, and never solve that problem again.


Stave is an open-source AWS configuration verifier. iam-explain is a single-policy IAM analyzer with Z3 formal verification. The pipe between them is the adoption path.

Top comments (0)