This isn't a proof of concept. It's been running in production for seven months across a 400-person engineering organisation. Here's exactly how it works.
The 4.2-day number isn't unusual. For a SaaS company with multiple service teams, compliance requirements and a staging environment that sometimes behaves nothing like production, a PR sitting in queue for four days before it ships is normal. Not good, but normal.
The bottleneck wasn't lazy engineers. It was handoffs. PR opened → wait for reviewer availability → review completed → wait for CI → CI passes → wait for staging deployment → staging validated → wait for deployment approval → deploy. Each wait is measured in hours and each handoff introduces the possibility of context loss, miscommunication, or someone being in a meeting when their action is required.
The 400-engineer SaaS company we worked with had the additional constraint of SOC 2 compliance requirements, meaning deployment decisions needed documented rationale and "it looked fine" was not an acceptable audit trail.
The question wasn't whether they could speed up reviews. It was whether they could redesign the entire pipeline so that handoffs between automated systems happened in seconds while human judgment was reserved for the decisions that actually require it.
The Architecture
The pipeline uses five Claude Code agents, each with a specific scope. The handoffs between them are event-driven, no polling, no scheduled checks.
PR Opened
↓
[REVIEW AGENT] — Code quality, security scan, test coverage check
↓ (passes threshold)
[TEST AGENT] — Generates missing tests, validates existing coverage
↓ (coverage met)
[STAGING AGENT] — Deploys to staging, runs smoke tests
↓ (smoke tests pass)
[VALIDATION AGENT] — Performance regression check, integration tests
↓ (no regression)
[DEPLOYMENT AGENT] — Production deployment with rollback monitoring
↓
Human review required only for: threshold exceptions, new service integrations, schema changes
The key design decision: each agent has a defined pass/fail threshold. When a PR's complexity or risk score exceeds the threshold, it surfaces to a human reviewer with a pre-assembled context package rather than routing through the full automated pipeline.
Agent 1: The Review Agent
from anthropic import Anthropic
import subprocess
import json
client = Anthropic()
def review_agent(pr_diff: str, pr_metadata: dict) -> dict:
"""
Analyses PR diff for code quality, security issues,
and coverage gaps. Returns structured review with
risk score and required actions.
"""
system_prompt = """You are a senior code reviewer for a
production SaaS platform. Analyse PRs for:
1. Security vulnerabilities (SQL injection, auth bypass,
exposed secrets, injection vectors)
2. Performance regressions (N+1 queries, missing indexes,
synchronous blocking calls)
3. Test coverage gaps on modified code paths
4. API contract changes affecting downstream services
Return ONLY valid JSON with this exact schema:
{
"risk_score": 1-10,
"security_issues": [],
"performance_concerns": [],
"coverage_gaps": [],
"api_breaking_changes": [],
"auto_approvable": boolean,
"requires_human_review": boolean,
"review_rationale": "string"
}
risk_score >= 7 MUST set requires_human_review: true.
API breaking changes MUST set requires_human_review: true."""
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=2000,
system=system_prompt,
messages=[{
"role": "user",
"content": f"""PR #{pr_metadata['number']}
Author: {pr_metadata['author']}
Files changed: {pr_metadata['files_changed']}
Description: {pr_metadata['description']}
Diff:
{pr_diff}"""
}]
)
review = json.loads(response.content[0].text)
# Audit trail, every decision gets logged
log_audit_event({
"event": "review_agent_decision",
"pr_number": pr_metadata['number'],
"risk_score": review['risk_score'],
"requires_human": review['requires_human_review'],
"rationale": review['review_rationale'],
"timestamp": datetime.utcnow().isoformat(),
"agent_version": AGENT_VERSION
})
return review
The audit trail logging is not optional, it's what satisfies the SOC 2 requirement that every deployment decision is documented. Every agent decision gets written to an immutable log with the full reasoning chain.
Agent 2: The Test Generation Agent
When the review agent identifies coverage gaps, the test agent generates the missing tests before the PR can proceed.
def test_generation_agent(
source_code: str,
coverage_gaps: list[str],
existing_tests: str
) -> dict:
"""
Generates pytest tests for identified coverage gaps.
Validates generated tests actually run before returning.
"""
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=4000,
system="""Generate pytest tests for the specified
coverage gaps. Requirements:
- Tests must be runnable (no placeholder implementations)
- Include edge cases for each identified gap
- Match the style and patterns in existing_tests
- Include docstrings explaining what each test validates
- Use fixtures from existing conftest.py patterns
Return JSON: {
"tests": "complete test file content",
"coverage_targets": ["list of functions tested"],
"edge_cases_covered": ["list of edge cases"]
}""",
messages=[{
"role": "user",
"content": f"""Source code:\n{source_code}\n\n
Coverage gaps: {json.dumps(coverage_gaps)}\n\n
Existing tests (for style reference):\n{existing_tests}"""
}]
)
result = json.loads(response.content[0].text)
# Validate generated tests actually run
validation = run_generated_tests(result['tests'])
if not validation['passed']:
# Retry with failure context
return retry_test_generation(
result,
validation['failures']
)
return result
The validation step, actually running the generated tests before they get committed, was added after week two of production operation when we discovered Claude occasionally generated tests that referenced fixtures that didn't exist. The retry loop with failure context solves this in one additional pass approximately 8% of the time.
Agent 3: Staging and Validation
The staging agent handles deployment to the staging environment and runs the smoke test suite. The validation agent runs on top of that output.
def staging_agent(pr_number: int, build_artifact: str) -> dict:
deploy_result = deploy_to_staging(build_artifact)
smoke_results = run_smoke_tests(deploy_result['endpoint'])
# Collect metrics for regression comparison
perf_metrics = collect_performance_metrics(
deploy_result['endpoint'],
duration_seconds=120
)
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1500,
system="""Analyse staging deployment results.
Compare performance metrics against baselines.
Identify any regressions or anomalies.
Return JSON: {
"staging_healthy": boolean,
"regressions_detected": [],
"anomalies": [],
"performance_delta": {},
"proceed_to_production": boolean,
"reasoning": "string"
}""",
messages=[{
"role": "user",
"content": f"""Smoke test results: {json.dumps(smoke_results)}
Performance metrics: {json.dumps(perf_metrics)}
Baseline metrics: {json.dumps(get_baseline_metrics())}
PR number: {pr_number}"""
}]
)
return json.loads(response.content[0].text)
Agent 4: The Deployment Agent with Rollback Monitoring
The deployment agent is where the most thought went into the design, because production deployments with autonomous rollback decisions are where the risk is highest.
def deployment_agent(
pr_number: int,
staging_validation: dict,
deployment_config: dict
) -> dict:
# Final pre-deployment check
risk_assessment = assess_deployment_risk(
pr_number,
staging_validation,
deployment_config
)
if risk_assessment['risk_level'] == 'HIGH':
return escalate_to_human(pr_number, risk_assessment)
# Deploy with canary rollout
deploy_result = canary_deploy(
deployment_config,
initial_traffic_percent=5
)
# Monitor for 10 minutes at 5% traffic
monitoring_results = monitor_canary(
deploy_result['deployment_id'],
duration_minutes=10,
error_rate_threshold=0.5,
latency_p99_threshold_ms=800
)
if monitoring_results['thresholds_exceeded']:
# Autonomous rollback decision
rollback_result = execute_rollback(
deploy_result['deployment_id']
)
# Claude analyses why rollback was needed
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1000,
system="Analyse rollback event and generate incident report.",
messages=[{
"role": "user",
"content": f"""Deployment: {deploy_result}
Monitoring: {monitoring_results}
Rollback: {rollback_result}
Generate incident report with root cause hypothesis."""
}]
)
incident_report = response.content[0].text
notify_team(pr_number, incident_report)
log_audit_event({
"event": "autonomous_rollback",
"pr_number": pr_number,
"trigger": monitoring_results['threshold_exceeded'],
"incident_report": incident_report
})
return {"status": "rolled_back", "report": incident_report}
# Canary healthy — ramp to full traffic
return complete_deployment(deploy_result['deployment_id'])
The canary rollout at 5% traffic with autonomous rollback if error rate exceeds 0.5% or p99 latency exceeds 800ms was the design decision that made the engineering team comfortable with autonomous deployment. Not "the agent decides to deploy and hopes for the best", the agent deploys to a tiny slice of traffic, watches it carefully and reverts immediately if anything looks wrong.
What Broke During Rollout
There were three significant failure modes in the first six weeks.
The false positive review problem: The review agent was flagging approximately 34% of PRs as requiring human review in week one, far too high for the automated pipeline to deliver meaningful speedup. The issue was the system prompt was too conservative on the "security issues" classification. A logging statement that included a user ID in the message was being flagged as "potential PII exposure in logs." Tuning the system prompt with specific examples of what constitutes an actual security issue vs a style concern reduced the human escalation rate to 11%.
The test generation hallucination problem: Mentioned above, generated tests referencing non-existent fixtures. The validation loop solved this. The broader lesson: any agent that produces artifacts that will be committed to a codebase needs validation that the artifacts actually work, not just that they look plausible.
The staging environment divergence problem: The validation agent was making production deployment decisions based on staging metrics that weren't representative of production load. Staging was running on smaller instances. A PR that performed fine under staging load would show latency issues under production traffic at 5% canary. We addressed this by calibrating the staging-to-production comparison models and adding an explicit adjustment factor for known environment differences.
The Results After Seven Months
PR-to-production average: 6.4 hours (down from 4.2 days). Human review rate: 11% of PRs (up from 100%, obviously, down from the 34% false positive rate in week one). Autonomous rollback rate: 2.3% of deployments, all within the canary window. Audit finding rate in SOC 2 review: zero deployment-related findings.
The deployment agent's incident reports have been reviewed by the security team and accepted as satisfying the "documented rationale for deployment decisions" requirement in the SOC 2 controls.
The full architecture, configuration details and the prompt engineering approach for the review agent are covered in the Claude Code multi-agent DevOps pipeline case study.
This isn't a demo, it's running in production across 400 engineers. If your DevOps pipeline has similar bottlenecks, long PR-to-production cycles, compliance documentation overhead, or too many handoffs between automated systems, Dextra Labs builds these multi-agent systems for engineering organisations at scale.
Top comments (0)