Building an AI Agent That Can Actually Take Action

Adding a dependency is the least careful thing most of us do all week.

You run npm install some-utils, and you've just agreed to run a stranger's code on your laptop, in your CI, and eventually in production. The registry ships thousands of new versions an hour. Typosquats, hijacked maintainer accounts, postinstall hooks that phone home, code that ships in the tarball but never existed in the GitHub repo anybody actually reviewed, they all come through that door.

And nobody reads the tarball. I certainly don't.

So for the Agent Harness Hackathon I built Portcullis - an agent that reads it for you, and can't add the dependency without asking first.

Repo: https://github.com/imarpanpatra/portcullis

Three-minute walkthrough, if you'd rather watch than read:

The job

You ask it: can I add left-pad to this repo?

It checks whether the name is the attack before it reads any code. Then it pulls registry signals age, publish cadence, maintainers, downloads, known advisories. Then it downloads the actual published tarball into a sandbox, unpacks it, and reads what ships: install hooks, obfuscated payloads, child_process usage, and a diff against the project's own GitHub source at that tag.

Then it gives you a verdict with evidence. And if you want it added, it stops and asks, because that means a branch, a commit and a pull request on a real repository.

That last part is the whole point. Everything before it is research; that step is a write you can't take back.

How it's wired

Three pieces, and TrueForge handles everything between them.

A custom MCP server I wrote for npm intelligence. Four read-only tools over public keyless APIs: registry metadata, download stats, OSV advisories, and typosquat detection.

That last one taught me something. The obvious approach is to search the registry for the package name and look at the neighbours. It doesn't work. Ask npm's search API for expres and express is nowhere in the twenty-five results. The victim is missing from the exact query designed to find it.

So instead of hoping search finds it, I generate the names an attacker would plausibly have registered deletions, adjacent transpositions, doubled letters, homoglyphs like rn→m and probe for them directly. The downloads API takes 128 names per request and returns null for ones that don't exist, so a few hundred candidates resolve in two HTTP calls.

Pointed at expres, it finds express and rather than writing me a security report about a package I never meant to install, it stops and asks:

A git-backed skill holding the audit procedure and a Python inspector. TrueForge clones it into the sandbox on demand. The inspector is standard library only which felt right for a tool whose entire subject is install steps.

The GitHub connector does the write, and it's gated.

The whole agent is one JSON document a model, instructions, those connectors, the skill, and the config:

What TrueForge actually did for me

I wrote about 3,000 lines. None of it was the agent loop.

The harness gave me tool routing over MCP, a sandbox provisioned only when the agent needs one, the approval gate, subagents, sessions that survive the client going away, and Generative UI so the report renders as real components instead of a wall of markdown.

Ask it about five packages and it fans out one subagent each, running at once, results merged into one ranked answer:

Two of those are worth calling out.

Sandbox-as-tool. The agent loop and all credentials stay in the harness; the sandbox only executes. So the thing unpacking a stranger's tarball never sees my GitHub token. For this project specifically that's not a nice-to-have it's the reason the project is safe to run at all.

The approval gate is enforced by the harness, not by my client being polite. A gated call ends the turn. Resuming requires sending an approval event. So a client that ignores the pause doesn't get a pull request faster it gets no pull request. I couldn't have accidentally written past it if I'd tried.

I deliberately didn't add a --yes flag. A flag that pre-approves every write defeats the only claim the project makes, and it'd be the first thing anyone reached for in CI, which is exactly where nobody's watching.

Approve it, and you get the branch, the commit, and the pull request with the exact dependency line it added:

What broke

This is the useful part.

The sandbox had no bash

First real run, the agent gave me a confident, well-structured verdict on express. Registry signals, advisories, the lot.

It had never opened the package.

Every sandbox exec was coming back fork/exec /usr/bin/bash: no such file or directory. The agent tried three shell variations, gave up, and answered from registry metadata alone sounding exactly as certain as it would have if it had done the work.

The fix wasn't to find a shell that happened to exist on that image. It was to stop needing one: the inspector now exposes a callable and the skill imports it under Code Mode. Python is the one interpreter the sandbox guarantees, because the harness already runs its own client there.

The docs and the server disagreed about a path

Docs say skills land at /opt/tfy/skills/{name}. The running server logs /opt/tf/skills. My skill wouldn't have been found even with a working shell.

Small thing, twenty minutes lost, and I only caught it by reading the server's own startup log instead of trusting the documentation.

The model mattered more than the prompt

On a mini-class model, the agent skipped the tarball inspection entirely. It has registry tools that answer in a second and a sandbox step that doesn't, so it took the fast path and produced a verdict that had never looked inside the package.

Same instructions, same skill, on a larger model: loads the skill, runs the inspector, quotes the report back.

The mini model was never wrong about anything it said. That's the problem. It sounded equally certain having looked at strictly less. I now pin the model in the README and say why.

Switching is a one-line change, which is part of why this bit is easy to get wrong nothing stops you running the cheap one and believing the output.

It cried wolf on express

My first working inspector flagged express the most ordinary package in the ecosystem for URLs in package.json and a read of NODE_ENV.

That's useless. A tool that panics about express teaches you to ignore it, and then the one real finding gets ignored too. Three rounds of tightening later: express, chalk and ms come back completely silent, while esbuild reports its postinstall, child_process use and network egress all true, and the right answer there is admit with conditions, not refuse.

The regression I care about now isn't a missed finding. It's a benign package starting to produce them.

Qodo kept catching my own fixes

Every change went through a pull request reviewed by Qodo. Thirty-three findings across eight PRs.

The ones that stung were the second-order ones bugs my earlier fixes introduced:

• I added extraction size caps to stop decompression bombs. Those caps then let a truncated repository tree manufacture false critical findings.
• I capped severity in response. That went too far the other way and understated a proven content mismatch.
• I taught the scanner to read .sh files, but forgot to extend the provenance check so shell scripts ended up half-examined.
• My reconnect logic had the initial stream outside the try block. So a dropped connection skipped the reattach loop entirely: the one failure it existed to survive was the one that got past it.

And my favourite, caught the day of submission: I'd written a report template that always referenced a "limitations" card, then instructed the agent to omit that card when there were no limitations. An undefined reference in OpenUI renders as nothing at all. So clean audits express, ms, chalk would have produced an empty report on camera, while packages with problems rendered fine.

The good case was the broken one. I'd have found that during the demo.

I disagreed with two findings and said why in the thread rather than quietly changing code. One claimed the SDK returned turns newest-first; I checked against a real two-turn session and it's oldest-first. I still rewrote that code to not depend on ordering at all, which is the durable fix either way.

Would I hand this job to an agent again

Yes, but narrowly.

What makes this work isn't that the model is clever. It's that the harness gave it a place to run dangerous code, a way to reach real data, and a hard stop before the irreversible bit. Take any of those away and it's a chatbot with opinions about npm.

The part I'd still tell people to be careful about: the agent is most convincing when it has done the least work. Both times mine produced a confident wrong-shaped answer the missing bash, the mini model the output looked great. It was the tool traces that gave it away.

Code's all open source, MIT: https://github.com/imarpanpatra/portcullis
Demo: https://youtu.be/t042CZAsOuM

(Built with AI assistance, implementation and working through review findings. The design decisions, severity calibration and the calls on each review finding were mine.)