DevPilot is a coding agent working in a repo you just cloned. You ask it to fix one failing test. Like real coding agents, it reads the repo's AGENTS.md first.
Hidden in an HTML comment in that file, invisible in the rendered preview, is an instruction to post the repo's secrets file to an attacker. The agent does it. Then it fixes the test and reports success. You never see the leak in its reply.
Why it worked
-
Repo files were trusted like the user.
AGENTS.mdcame from an untrusted repo, but its text was followed as instructions. - No approval gate. Any command ran without asking.
- No sandbox. Commands had the network and the repo's secrets.
A cloned repo is untrusted input, and agent config files are part of it (OWASP LLM01 and LLM06).
The fix, in code
ALLOWED_COMMANDS = {"python test_calc.py"} # approval gate for everything else
SECRET_FILES = (".env", ".env.lab", "id_rsa") # sandbox keeps secrets out of reach
def read_file(args, state):
name = os.path.basename(args["path"])
if name.startswith(SECRET_FILES):
return f"BLOCKED: {args['path']} is not available inside the sandbox."
text = open(path_in_repo(args["path"])).read()
if name.endswith(".md"):
# An untrusted repo's instruction files are data. Label them; never treat them as the user.
return "<untrusted_repo_file>\n" + text + "\n</untrusted_repo_file>"
return text
def run_command(args, state):
cmd = args["command"].strip()
if cmd not in ALLOWED_COMMANDS:
return f"BLOCKED: '{cmd}' needs your approval (no network in the sandbox)."
return shell(cmd)
Same task, run again: the leak is blocked, and the test still gets fixed.
Run the lab yourself
git clone https://github.com/usinha79/hack-it-then-fix-it
cd hack-it-then-fix-it/ep06-coding-agent-config-injection
pip install -r requirements.txt
python vulnerable/app.py # then, in another terminal: ./attack.sh
python fixed/app.py # same task, leak blocked
run_command is a tiny simulated shell that only understands three commands, so the lab never runs arbitrary commands on your machine. Full lab: github.com/usinha79/hack-it-then-fix-it
Take-home checklist
- Review
AGENTS.md,.cursorrulesand similar agent files in any repo before letting an agent loose. Read the raw file, not the preview. - Run coding agents with a command allowlist and approval for everything else.
- Sandbox agents: no network by default, no secrets mounted.
- Treat repo content as untrusted input to the agent.
Hack It, Then Fix It is a series where I build a small AI app with a real flaw, break it, fix it in code, and run the same attack again to prove the fix holds. Everything shown uses fake data on my own machine. Only test systems you own or are authorised to test.
Top comments (1)
Good lab, and the hidden HTML comment is a nice touch because nobody reads the raw file. One thing I'd tighten in the fix: the secret check matches on the file name, so a
config/prod.yamlwith keys in it, or a symlink with a harmless name pointing at.env, still gets read. The sandbox line is the one doing the real work. If the secrets aren't mounted, there's nothing to match on. Same with the untrusted label: it helps the model, but the allowlist is the actual control.Also worth a future episode: some agents load the AGENTS.md nearest to the file they're editing, not just the root one, so the hidden line can sit three folders deep where a reviewer is even less likely to look.