Quality Gates: Keeping AI Agents Productive Without Going YOLO

Originally published at getorquesta.com/blog/quality-gates-keeping-ai-agents-productive-without-going-yolo-2026-11-05

We first turned Claude loose on our repos with the kind of reckless optimism usually reserved for hackathons. It was glorious—PRs in minutes, refactors across dozens of files, bugs found and fixed while we slept. Then it merged a database migration that dropped a column still referenced in a legacy service. The incident took four hours to unwind.

That was the moment we realized: autonomous agents are incredibly fast, but they don't carry the same context as the humans who own the system. The fix wasn't to slow the agents down; it was to build a gate—a controlled checkpoint where a human reviews what the AI is about to do before it does it for real. That's what Quality Gates are in Orquesta.

The Tension: Speed vs. Control

AI coding agents are seductive because they collapse the feedback loop. You describe a change, and minutes later there's a diff. But that speed becomes a liability when the agent makes changes that are syntactically correct but semantically wrong. A human might know that the users table is replicated across three regions and that a schema change needs a specific rollout order. An AI trained on public code doesn't.

Traditional code review exists for a reason, but it's asynchronous and often slow. If you route every agent action through a human review before execution, you lose most of the speed advantage. The key is to be selective: apply gates only to high-risk actions, and make the review process as frictionless as possible.

How Quality Gates Work in Orquesta

In Orquesta, a Quality Gate is a policy that intercepts an agent's execution pipeline before it commits real changes. When a gate is active for a prompt, the agent runs in simulation mode first. It produces a diff, a list of commands it would run, and an explanation of its reasoning. That bundle is presented to a designated reviewer—usually a team lead—who can approve or reject. Only after approval does the agent execute for real.

The gate can be configured per target environment, per command type, or per risk level. For example, you might allow the agent to create new files or run read-only commands without a gate, but require approval for anything that touches infrastructure, modifies database schemas, or targets production.

The Simulation Step

Simulation is the heart of the gate. We didn't want the agent to just describe what it would do; we wanted it to produce an actual diff that could be reviewed like a normal PR. For code changes, the agent operates on a temporary Git worktree or branch, then generates a patch. For shell commands, we use dry-run flags where available (terraform plan, kubectl diff, npm audit --dry-run) or run the command in a sandboxed container with no network egress to the real environment.

The simulation runs quickly—usually a few seconds—because it's not applying changes, just computing them. The output is captured and stored as part of the execution record. We also capture the agent's chain-of-thought (if available) to give the reviewer insight into why the agent chose a particular approach.

The Review Workflow

The review interface is deliberately minimal. The reviewer sees a side-by-side diff, the list of commands with context, and a summary written by the agent. For large diffs, the agent also generates a bullet list of the most important changes, along with any risks it identified. This is crucial because a 200-file diff is useless for a human reviewer without a summary.

The reviewer can approve, reject, or request changes. If they request changes, they can add a comment and the agent will iterate in the same simulation loop—no real changes are made until final approval. Once approved, the agent applies the diff, runs the commands, and creates a commit. The commit message includes a reference to the approval and the reviewer's name, so there's a full audit trail.

Configuration and Code

Quality Gates in Orquesta are configured via YAML in the project's .orquesta/ directory. Here's an example that requires approval for any change targeting the production environment:

quality_gates:
  - name: production-guard
    description: "Require human approval for any change to production"
    when:
      target_environment: production
    simulation:
      mode: full
      diff_provider: git
    reviewers:
      - role: team-lead
    approval_required: true
    auto_reject_after: 24h

You can also define gates that apply only to commands matching a pattern, like DROP, DELETE, or migrate. For those, the simulation mode might be dry-run with the agent instructed to generate the exact rollback plan.

From the CLI, you can submit a prompt with a gate explicitly:

orquesta prompt --gate production-guard "Update the payment service to handle new tax rates"

And to check the status of a pending approval:

orquesta status --execution 8f3a2

The status output shows the current phase (simulating, awaiting_review, approved, rejected, executing) and the reviewer assigned.

Lessons Learned

We've been running quality gates in production for about six months. The biggest lesson: the human reviewer is the bottleneck, not the AI. If the review takes too long, people get frustrated and start bypassing gates—which defeats the purpose. We mitigated this with two changes: first, we made the review UI fast enough that a typical gate review takes under two minutes. Second, we added AI-generated summaries and risk assessments that highlight exactly what needs attention.

We also learned that gates work best when they're rare. If every prompt triggers a gate, you've just recreated traditional code review with extra steps. We reserve gates for actions that are irreversible or have wide blast radius: production deployments, database migrations, infrastructure changes, and anything that touches billing or auth.

Finally, we made it easy to disable gates for trusted, low-risk workflows. An agent that can't create a new test file without approval is an agent nobody will use.

Conclusion

Quality gates aren't about mistrusting the AI; they're about defining trust boundaries. The agent is great at generating and simulating changes, but it shouldn't have the final say on what enters production. By splitting the work into simulation and execution, with a human checkpoint in between, you get the speed of autonomous agents without the YOLO risk. In Orquesta, that checkpoint is built into the platform, not bolted on as an afterthought. It's the difference between an agent that's productive and an agent that's a liability.