DEV Community: kabirnarang39

I let Claude Code build a real security tool, here's the honest writeup

kabirnarang39 — Sun, 09 Aug 2026 19:51:14 +0000

AI agents now call tools, MCP servers, and each other on their own. That is the
whole point of an agent. It is also the problem: the moment an agent is
compromised — a prompt injection, a leaked key, a poisoned tool result — it
keeps its credentials and its network path, and nothing between it and your
systems says no.

Most of the stack around agents watches and reports. It logs the calls, maybe
scores them, and pages a human. By the time the human reads the page, the agent
has made a thousand more calls.

I built Wardline to enforce instead of watch. It's a control-plane proxy: an
AI agent reaches its MCP servers, tools, and gRPC upstreams only through
Wardline, which applies identity, policy, budget, and anomaly detection
in-process and writes every decision to an audit trail. One static Go binary —
no database, no identity provider, no sidecar to start.

The part that's actually different: auto-block

Alerting is easy and everyone does it. The claim worth making is enforcement.

Wardline keeps a per-identity behavioral baseline using Welford's algorithm —
a running mean and variance over four features per time window: call rate,
distinct-tool count, deny ratio, and mean inter-arrival time. No training data,
no external model, no history to store. Each completed window is scored as a
combined z-score against that identity's own baseline.

When the score crosses a configured threshold and auto_block is on, Wardline
doesn't just write an anomaly record — it rejects that identity's calls for a
bounded TTL. The compromised agent is cut off in real time, with no rule written
for the specific attack and no human in the loop.

// The detector wires a real BlockChecker as the blocker; when the combined
// z-score exceeds the auto-block threshold, the identity is blocked.
if d.cfg.AutoBlock.Enabled && d.blocker != nil && blockScore > d.cfg.AutoBlock.ScoreThreshold {
    d.blocker.Block(e.Identity, e.Tenant, /* reason */)
}

The decision path benchmarks at ~33ns / 0 allocations on the default YAML
backend.

The part most projects won't tell you: what it does not catch

The auto-block catches abrupt deviation. It does not catch low-and-slow.

Because the baseline is self-learned and unsupervised, an attacker who ramps
activity gradually — staying within a few standard deviations of the moving
baseline each window — is never blocked. The baseline adapts upward and absorbs
the ramp. Wardline blocks the agent that suddenly does 10x its normal rate; it
does not block the agent that patiently climbs to 10x over an hour.

This isn't a threshold you can simply tighten. Tighten it and you start blocking
normal agents — the false-positive rate is regression-guarded to stay near zero
on steady traffic, and that guard is the thing keeping the feature usable. It's
an inherent tradeoff of unsupervised, per-identity baselining.

Both behaviors are pinned by tests in the repo —
TestDetector_AutoBlock_AbruptSpikeIsBlocked and
TestDetector_AutoBlock_LowAndSlowEvades — so the boundary is documented, not
marketed around.

The takeaway isn't "anomaly detection is weak." It's that anomaly detection is
the last line, not the only one. Keep explicit policy and budget limits as the
hard floor — they bound absolute behavior regardless of ramp speed — and let
auto-block catch the fast, obvious compromise that policy didn't anticipate.

Secure by default is a claim; read the defaults

Wardline fails closed on policy. It does not fail closed on identity or the
dashboard by default: identity is trusted from the X-Wardline-Identity header
(spoofable) and the dashboard's read views are unauthenticated, until you turn on
the flags that change that. The binary logs a WARN on startup for every
insecure default still in effect, so the posture is never silent.

features:
  credential_issuance: true   # verify a signed bearer token instead of trusting the header
  rbac: true                  # gate the dashboard and admin actions on real permissions

About the "built with Claude Code" part

I'll be transparent, because the honesty is the whole theme: Claude Code did the
bulk of the implementation, under my direction. Architecture, threat model, and
every design decision are mine and were reviewed by me. The workflow was
spec-driven — spec → plan → build → and on any failure, trace the root cause and
update the spec before retrying. That loop is what kept ~46k lines of Go coherent
instead of drifting into slop: ~950 tests, race + coverage in CI, golangci-lint
clean, Clean Architecture actually enforced.

The contribution I valued most wasn't code volume — it was that it kept pushing
me to document limitations (like the low-and-slow gap) and write tests proving
them, rather than ship an impressive demo that overpromised. For a security tool,
that's exactly the instinct you want.

Where it fits

Wardline is young and unproven at scale — that's the honest status. It's not
trying to replace an LLM router like LiteLLM or a managed gateway like Portkey.
It's the enforcement-first control plane for the traffic between an agent and
everything it calls, in a single self-hosted binary, Apache-2.0.

If that's the layer you're missing, the repo is here:
github.com/kabirnarang39/wardline.
Feedback on the anomaly approach and the threat model is exactly what I'm after —
poke holes.

Your Claude Skill Passed Every Review

kabirnarang39 — Wed, 29 Jul 2026 21:23:59 +0000

Here's a scenario that's already happened to someone reading this, they just don't know it yet.

You write a Claude Skill. You test it by hand — a few prompts, looks good, ship it. It works for months. Then Anthropic ships a new model. Nothing about your skill changed. But the model reads your instructions slightly differently now: it stops triggering on a phrasing that used to work, or it triggers fine but quietly drops a constraint it used to follow, or it still does the right thing but now costs three times as much to get there.

There's no error. No stack trace. No red CI badge. The skill just… drifts. You find out the way everyone finds out — a workflow breaks downstream, a review comes back wrong, a support ticket shows up, and twenty minutes into debugging you realize: it's not your code, it's the model, and it changed weeks ago.

Every other kind of software would have caught this on day one
If a dependency bump broke your app, you'd have a failing test in CI before it ever reached staging. If a database migration corrupted a query, your integration suite would go red immediately. That's not because those systems are magic — it's because someone decided, a long time ago, that "did this still work after the thing underneath it changed" deserves to be a test, not a vibe.

Skills don't have that yet. Most SKILL.md files in the wild have never been tested against more than one model, because until recently there wasn't a standard way to test them at all. Anthropic clearly sees the gap too — skill-creator now ships with evals, benchmark mode, and A/B testing between skill versions, which is a real step forward for authoring a skill well. But authoring-time testing answers "does this work today." It doesn't answer "does this still work six months from now, on a model that didn't exist when I wrote it" — and that second question is the one that actually burns teams, because it fails silently, in production, on someone else's schedule.

What a safety net for this actually looks like
Not a one-time eval you ran once and forgot about. A safety net that:

Re-runs automatically against every model in your matrix, not just the one you tested on originally
Diffs against last-known-good, so a flaky one-off doesn't cry wolf and a real regression can't hide in the noise
Turns a caught failure into permanent coverage — the exact gap that just bit you gets written down as a test case automatically, so it can never silently regress twice
Tells you which commit broke it, the same way git bisect tells you which commit broke a normal test, instead of leaving you to guess
Treats adversarial input as a real test category, not an afterthought — prompt injection, jailbreak attempts, and instruction leakage are things a skill can regress on too, and most testing setups don't check for that at all
That's the actual shape of the problem. It's not "skills need more eval prompts." It's "skills need the same CI discipline every other piece of software has had for twenty years, and nobody had built it yet."

So I built it
SkillCI is an open-source CLI that treats a Claude Skill like any other piece of software with a regression suite:

skillci check — a local, zero-API-call lint mapped to OWASP's Agentic Skills Top 10 (prompt injection risk, over-privileged access, unpinned dependencies) plus basic bloat checks. Free, no API key, catches the obvious stuff before you even run a model.
skillci regress — runs your eval suite across every model in your config, diffs against the last known-good run, and fails CI only on a new regression, not on every pre-existing flaky case.
The self-growing eval loop — when regress catches a failure with zero prior coverage, it doesn't just print red text. It writes the missing test case for you, so skillci accept turns "we got lucky this time" into permanent, tracked coverage. Catch once, covered forever.
skillci bisect — real git worktree-based binary search over your skill's own commit history, so when something breaks, you get a culprit commit instead of a guessing game.
Redteam plugins — prompt injection, jailbreak attempts (including multi-turn crescendo attacks), PII exfiltration, SSRF-bait, and more, run against your skill the same way a pentest would, with a successful attack becoming a permanent regression case exactly like a caught model drift does.
It's complementary to skill-creator's evals, not a replacement — skill-creator helps you author a good skill today; skillci watches it stay correct over time, across models you haven't tested yet and commits you haven't shipped yet.

Try it in under a minute, no API key needed
brew install kabirnarang39/skillci/skillci
skillci check path/to/your-skill
That's the free, local-only half — no ANTHROPIC_API_KEY, no network calls, just a static scan that'll tell you if your skill is carrying obvious risk. The full regression loop needs a key, but you can see whether this is worth your time before spending a single API call.

It's early — the core CLI is stable, the dashboard is newer, and I'd genuinely rather hear "here's what's missing" than silence. If you've been burned by a skill that quietly stopped working, or you're maintaining enough skills that this is starting to feel like a real problem instead of a hypothetical one, I'd like to know what broke and how you found out. That's the next thing I'm building toward.

kabirnarang39 / skillci

CI for Claude Skills — lint, eval, and regression-test SKILL.md files across a model matrix, with a self-growing eval loop that turns uncovered regressions into permanent test cases.

SkillCI

Regression testing for Claude Skills. When a model update silently changes how your skill behaves, SkillCI catches it in CI — and turns the failure into a permanent test case, automatically.

A skill fails against a model it's never been tested on → SkillCI doesn't just report red, it writes the missing test case for you (evals/_generated/...) so skillci accept turns it into permanent coverage. That loop — catch once, covered forever — is the whole point.

Why · What's inside · How this compares · Install · Quick start · GitHub Actions · Dashboard · VS Code · Compliance reports · For AI agents · MCP server · Commands · Status

Why

You write a Claude Skill. It works today. Six months from now, Anthropic ships a new model, and nobody tested your skill against it first — because until now, no tool did that automatically…

View on GitHub

If this is a problem you've hit, a star helps others find it — and tells me what to build next.