Cornerstone OnDemand built Orion AI to turn database operations from reactive firefighting into proactive automation. A three-person team cut database diagnosis from 45 minutes to 10 minutes in six months using Amazon Bedrock and the Strands Agents framework. The system coordinates three specialized agents (diagnosis, query generation, remediation) without creating circular dependencies or state conflicts.
This is not a demo. It runs in production at a major SaaS provider handling real database incidents.
The Problem: Manual Database Diagnosis at Scale
Cornerstone's database operations team faced a familiar pattern:
- Alert fires
- Engineer manually queries logs and metrics
- Engineer writes diagnostic SQL
- Engineer interprets results
- Engineer decides on remediation
- Engineer executes fix
- Engineer documents incident
Each cycle took 45 minutes on average. Multiply that by dozens of alerts per day across a multi-tenant SaaS platform, and you have engineers spending most of their time on repetitive diagnosis instead of architecture work.
The team needed automation that could handle the full loop, not just alerting or monitoring.
Architecture: Three Agents with Clear Boundaries
Orion AI splits database operations into three specialized agents, each with a narrow scope:
Diagnosis Agent
- Ingests alerts and metrics
- Identifies problem patterns
- Determines which database components need inspection
- Hands off context to query agent
Query Generation Agent
- Receives diagnosis context
- Generates safe, read-only SQL queries
- Validates query syntax before execution
- Returns structured results to diagnosis agent
Remediation Agent
- Receives approved diagnosis and query results
- Proposes remediation actions
- Requires human approval for destructive operations
- Executes approved fixes
- Logs all actions for audit
The agents communicate through a state machine managed by Strands Agents, which handles handoffs and maintains context across the workflow.
Orchestration Flow
# Simplified orchestration pattern (conceptual)
class OrionWorkflow:
def __init__(self, bedrock_client, strands_orchestrator):
self.diagnosis_agent = DiagnosisAgent(bedrock_client)
self.query_agent = QueryAgent(bedrock_client)
self.remediation_agent = RemediationAgent(bedrock_client)
self.orchestrator = strands_orchestrator
def handle_alert(self, alert_data):
# Phase 1: Diagnosis
diagnosis = self.diagnosis_agent.analyze(alert_data)
# Phase 2: Query generation and execution
queries = self.query_agent.generate(diagnosis.context)
results = self.execute_safe_queries(queries)
# Phase 3: Remediation proposal
remediation_plan = self.remediation_agent.propose(
diagnosis=diagnosis,
query_results=results
)
# Human approval gate for destructive operations
if remediation_plan.requires_approval:
return self.request_approval(remediation_plan)
# Execute approved actions
return self.remediation_agent.execute(remediation_plan)
def execute_safe_queries(self, queries):
# Read-only validation before execution
validated = [q for q in queries if self.is_safe_query(q)]
return self.run_queries(validated)
The key design decision: each agent receives structured input and returns structured output. No agent directly calls another agent. The orchestrator handles all handoffs and maintains the audit trail.
State Management and Context Preservation
Orion AI uses Strands Agents to manage state across the three-agent workflow. Each phase produces artifacts that flow to the next:
| Phase | Input | Output | State Stored |
|---|---|---|---|
| Diagnosis | Alert data, metrics | Problem classification, affected components | Alert ID, timestamp, severity, component IDs |
| Query Generation | Diagnosis context | SQL queries, expected schema | Query text, validation status, execution plan |
| Remediation | Diagnosis + query results | Action plan, approval requirements | Proposed actions, approval status, execution log |
The orchestrator maintains a workflow ID that links all artifacts. If an agent fails or times out, the system can resume from the last successful phase without re-running expensive diagnosis or query operations.
Security Boundaries and Failure Modes
The team built three layers of protection against cascading errors:
Query Safety
- All generated SQL goes through a parser that rejects DML/DDL
- Query timeout limits prevent runaway operations
- Results are capped at configurable row limits
- Connection pooling prevents database overload
Remediation Gates
- Destructive operations require explicit human approval
- Approval requests include full context and predicted impact
- Approved actions execute with transaction boundaries
- Rollback procedures are generated alongside remediation plans
Observability
- Every agent call logs input, output, and latency
- Failed agent calls trigger alerts to the operations team
- Query execution plans are logged before execution
- Remediation actions are logged to a separate audit database
The most common failure mode: the diagnosis agent misclassifies a problem, leading to irrelevant queries. The system handles this by allowing the query agent to return "insufficient data" results, which triggers a fallback to manual diagnosis.
Deployment Shape
Cornerstone runs Orion AI on AWS infrastructure:
- Amazon Bedrock for LLM inference (Claude models)
- Strands Agents for orchestration and state management
- Amazon RDS for audit logs and workflow state
- AWS Lambda for agent execution
- Amazon EventBridge for alert ingestion
The team chose Lambda for agent execution because database incidents are bursty. Most of the time, the system is idle. When alerts fire, Lambda scales to handle multiple concurrent workflows without pre-provisioned capacity.
Each agent runs in its own Lambda function with separate IAM roles. The diagnosis agent has read-only access to metrics. The query agent has read-only database credentials. The remediation agent has write access but only executes after approval.
Results and Trade-offs
After six months in production:
- Diagnosis time dropped from 45 minutes to 10 minutes (78% reduction)
- 60% of incidents now resolve without human intervention
- Engineers spend less time on repetitive diagnosis
- The system handles 40+ alerts per day automatically
Trade-offs the team accepted:
Latency for Safety
The system takes 10 minutes instead of 2 because it includes validation steps, approval gates, and structured handoffs. The team chose correctness over speed.
Limited Remediation Scope
The remediation agent only handles a predefined set of actions (restart services, clear caches, adjust connection pools). Complex fixes still require human engineers.
Model Dependency
The system relies on Amazon Bedrock availability. If Bedrock has an outage, Orion AI falls back to manual diagnosis. The team monitors Bedrock health and has runbooks for manual fallback.
Technical Verdict
Use this pattern when:
- You have repetitive operational workflows that follow predictable patterns
- You can define clear boundaries between diagnosis, analysis, and remediation
- You need audit trails and approval gates for compliance
- You have a small team that needs to scale operational capacity
Avoid this pattern when:
- Your incidents are highly variable and require creative problem-solving
- You cannot tolerate 10-minute diagnosis latency
- Your database operations require complex multi-step transactions
- You lack the infrastructure to run orchestrated agent workflows
The Cornerstone case study shows that multi-agent systems work in production when you design for failure modes, maintain clear agent boundaries, and accept trade-offs between speed and safety. A three-person team built this in six months because they focused on a narrow problem domain and reused existing frameworks (Bedrock, Strands Agents) instead of building orchestration from scratch.
Top comments (1)
tr.ee/dev-to