Building Deterministic RL Environments for Cloud Infrastructure Evaluation: What Actually Transfers From Production Engineering

AI models are increasingly being asked to design, deploy, secure, and recover production-grade infrastructure — not just write functions that pass a unit test. That shift changes what a training environment has to be. It's no longer enough to check that output "looks right." You need a golden reference solution, a deterministic validation suite, and a set of intentionally broken variants that probe exactly where a model's reasoning breaks down under failure.

I've spent the last several months on the other side of this problem — evaluating agentic coding outputs against structured rubrics, designing programmatic verification checks, and writing up edge cases that rubric authors hadn't considered. Before that, I spent seven years building and debugging the kind of systems these environments are meant to simulate: multi-tenant SaaS platforms with row-level security, AWS infrastructure serving high-traffic e-commerce, and data pipelines processing tens of thousands of concurrent requests. This post is about where those two things meet — what it actually takes to build an infrastructure RL environment that's reproducible, fair to evaluate, and hard to game.

A golden solution is only as good as its ambiguity budget

The first mistake in building any evaluation environment is under-specifying the scenario and over-specifying the solution. If the task says "deploy a fault-tolerant queue consumer" without pinning down delivery semantics, retry policy, and what "fault-tolerant" means operationally, you'll end up with a golden solution that's just one valid interpretation among several — and you'll penalize a model for a decision the spec never actually made.

This is the same discipline as writing rubrics for coding evaluations: every ambiguous term is a future dispute. In practice that means, before writing a single line of infrastructure code, defining:

• The exact failure modes in scope (node loss, network partition, message duplication, clock skew)
• The invariants that must hold regardless of implementation (at-least-once vs exactly-once delivery, idempotency guarantees, RTO/RPO targets)
• What counts as "recovered" — not just "the service is up," but that state is consistent

Deterministic validation means designing against non-determinism

Distributed systems are inherently non-deterministic — that's exactly what makes them hard to evaluate. A validation suite that works by re-running the same sequence of API calls and diffing output will produce flaky, unfair results the moment retries, timeouts, or eventual consistency are involved.

The pattern that's worked for me, most recently writing a security test suite that validates row-level-security policies against a real Postgres instance rather than mocks, is to validate invariants, not traces:

# Anti-pattern: asserting on the exact sequence of events
assert events == [
    "consumer_started",
    "message_received",
    "message_processed",
    "ack_sent",
]

# Better: assert the invariant the system must uphold,
# regardless of retries, ordering, or timing
def test_no_duplicate_side_effects(env):
    env.inject_fault("redeliver_message", count=3)
    env.run_until_settled(timeout=30)
    assert env.side_effect_count("charge_customer") == 1
    assert env.final_state_is_consistent()

Testing against a real, disposable instance of the actual dependency — a real queue, a real Postgres, a real IAM policy engine — rather than a mock catches the class of bug that mocks are structurally blind to: the ones where the contract you assumed doesn't match the behaviour you get. I've root-caused production bugs (a minor dependency version bump silently collapsing TypeScript types to never, an OAuth token-refresh edge case that only reproduced under real API rate limits) that a mocked test suite would have sailed straight past. The same principle applies at the infrastructure layer, just with higher stakes per failure.

Defective variants need to fail for the right reason

The point of an intentionally broken variant isn't just "does the model notice something is wrong" — it's "does the model correctly diagnose why." A variant with a misconfigured IAM policy that happens to also have a network misconfiguration will teach a model to pattern-match on the wrong signal.

Building these well means treating each defect as a single, isolated hypothesis:

• One fault per variant. Resist the urge to combine a broken retry policy and an under-provisioned autoscaling group into one scenario "for efficiency." You'll never know which one the model actually reasoned about.
• Fail loud enough to be observable, quiet enough to require diagnosis. A defect that immediately crashes the deploy is a smoke test, not an evaluation. A defect that silently corrupts data under specific timing conditions is where reasoning gets tested — but it has to be reliably reproducible, or you're evaluating luck.
• Document the intended diagnosis path. If you can't write down, in advance, the sequence of observability signals (logs, metrics, traces) that should lead a correct reasoner to the root cause, the variant isn't ready yet.

Where this comes from, concretely

None of this is theoretical for me. A few data points from production work that map directly onto this kind of environment design:

• Wrote and maintain a 14-test security suite validating row-level-security policies against a real Postgres database — the reproducible-environment-over-mocks principle, applied.
• Root-caused a silent dependency-resolution failure where a minor version bump collapsed TypeScript types to never across a monorepo — the kind of defect class that's genuinely worth encoding as a training scenario, because it's realistic and painful to diagnose.
• Built and operated AWS infrastructure (EC2, S3, Lambda) for a high-traffic e-commerce platform, including the profiling and query-optimisation work behind a 30% latency reduction — direct experience with the observability and performance-diagnosis loop these environments are meant to simulate.
• Currently evaluate agentic coding outputs against structured rubrics professionally, including rubric construction, adversarial prompt design, and selecting which checks are actually programmatically verifiable versus which need human judgment — the exact skill set "golden reference solution + deterministic test" environment design draws on.

The uncomfortable part: most of the work is writing, not coding

The infrastructure code for a good RL environment is often the easy part. The hard part is the documentation — writing down, precisely enough that someone else could reproduce your reasoning, what each scenario is testing, what "correct" means, what edge cases were considered and rejected, and why. That's not a side task. For environments meant to train reasoning about production systems, the documentation is the specification the golden solution is graded against. Treating it as an afterthought is how you end up with an environment that's internally inconsistent and nobody notices until a model exploits the gap.

If you're building or evaluating environments in this space, I'd be glad to compare notes — particularly on validating distributed invariants without flaking, and on keeping golden solutions honest about which parts of a spec are actually unambiguous.