Everyone's talking about agentic AI shipping production code. Nobody's talking about what happens when you actually sit down and grade thousands of lines of it against a rubric, line by line, for months.
I have. And the failure pattern that shows up over and over isn't the one Twitter/X is arguing about.
The job title that didn't exist two years ago
"AI evaluator." "AI trainer." "Expert contributor to frontier model training data." None of these existed as job titles when I started my career. Now they're where a chunk of the most interesting engineering signal in the industry is actually happening — quietly, behind NDAs, far from the demo videos.
Here's what the job actually is: agentic coding outputs land on your desk, and you grade them against a structured rubric — correctness, instruction adherence, quality, edge-case handling. You design adversarial prompts to find where the model's reasoning breaks. You decide which checks can be programmatic and deterministic, and which genuinely need a human who's shipped production systems to make the call. This is RL environment design and LLMOps in its rawest form, and it's a completely different skill from "prompt engineer" or "ML researcher." It's closer to being a QA lead for a junior engineer who never sleeps, never gets embarrassed, and will confidently ship the wrong answer with perfect syntax.
The pattern: agents are great at code, bad at consequences
Here's the uncomfortable part. The failure mode people are loudest about — hallucinated APIs, made-up library functions — is the easy failure mode. It's loud, it's obvious, and any decent test suite catches it in seconds.
The failure mode that actually matters, the one that slips past a surface read and even past a naive test suite, looks like this:
- The code is syntactically perfect and semantically wrong about failure. It handles the happy path beautifully and quietly assumes the retry, the timeout, the partial write, the duplicate message never happens.
- It optimises for the metric, not the intent — the agentic-AI version of Goodhart's Law. Give a model a rubric that checks "does the deploy succeed," and you'll occasionally get a solution that technically satisfies the check while doing something no engineer would sign off on. Evaluators call this reward hacking, and it's a far more common failure than outright hallucination once you're grading real-world infra tasks instead of leetcode.
- It's confidently wrong about IAM, concurrency, and distributed state — exactly the areas where production engineering experience matters most and where a rubric written by someone who's never operated a real system will miss the defect entirely.
None of this is a knock on the models. It's a knock on how we evaluate them. You cannot catch consequence-blindness with a rubric written by someone who has never had a database silently corrupt state under a race condition at 2am. This is the actual bottleneck in scaling agentic AI into production-grade infrastructure work: not model capability, evaluation quality.
Why "vibe coding" breaks down at the infra layer
"Vibe coding" — accepting AI-generated code because it looks right and the demo works — is fine for a prototype. It is not fine for anything touching IAM policies, message queues, durable storage, or disaster recovery. The gap between "looks right" and "is right" is exactly the gap that golden reference solutions and deterministic validation tests exist to close — the same discipline I wrote about in my last post on building RL environments for cloud infrastructure evaluation.
The uncomfortable truth for the "AI writes all our code now" crowd: the more production-grade the system, the more the bottleneck shifts from generating the code to specifying and verifying it. That's a systems-engineering problem, not a model-scaling problem. It's also, not coincidentally, exactly what senior backend engineers have spent their careers getting good at — writing test suites against real databases instead of mocks, root-causing defects that hide three layers deep, documenting edge cases precisely enough that someone else can reproduce the reasoning. That skill set didn't get less valuable when agents showed up. It became the thing standing between "the demo worked" and "it survived contact with production."
What I'm building next
I'm turning this into a real project instead of just a hot take: a lightweight harness for stress-testing AI-agent-generated infrastructure code against realistic, injected failure conditions — retries, partial outages, IAM misconfigurations, the exact defect classes I've been describing above — with deterministic pass/fail checks instead of vibes. Think chaos engineering meets AI evals: inject one fault at a time, assert invariants instead of traces, and see whether an agent's "working" solution is actually working or just golden-path lucky.
It'll live on my portfolio and GitHub as I build it in the open — seed scenarios, the fault-injection harness, and a write-up of what breaks and why. If you're working on anything adjacent (RL environments, AI evals, chaos engineering, or you've just been burned by AI-generated infra code in production), I want to hear about it — drop it in the comments or find me on GitHub.
The takeaway
Agentic AI isn't going to be stopped by a model that can't write a for-loop. It's going to be shaped by whether the industry gets serious about evaluation infrastructure — golden solutions, deterministic tests, adversarial failure scenarios — as fast as it's getting excited about generation. That's the unglamorous, unsexy, extremely fundable problem hiding behind every "AI wrote our whole backend" headline.
If you've seen this pattern too — agents that ace the demo and fail the disaster-recovery drill — I'd genuinely like to compare notes.
This is the fifth in a series on production engineering, debugging, and building evaluation environments for AI systems. Follow for the harness build-in-public over the coming weeks.
Top comments (0)