Multi-agent prototypes ship in days. Production deployments take months. The gap is not LLM performance. It is orchestration flow, model selection per agent, secure code execution, partial failure handling, and observability.
AWS published a breakdown of five production walls that teams hit when scaling multi-agent systems, using Deep Insight (an open-source reference implementation on AWS) as the case study. The post names specific failure modes and shows architectural solutions.
The Five Production Walls
Most teams hit these in order:
- Orchestration: How do agents coordinate without a central bottleneck?
- Per-agent model selection: Does each agent pick its own model, or does the orchestrator decide?
- Reliable operation: What happens when one agent in a chain crashes or times out?
- Secure code execution: How do you sandbox agent-generated code to prevent privilege escalation or data exfiltration?
- Monitoring: What observability primitives expose multi-agent failures for debugging?
Deep Insight addresses each with specific infrastructure choices.
Orchestration Without a Central Coordinator
Deep Insight uses a task queue with agent-specific workers instead of a central orchestrator. Each agent type subscribes to a queue. When an agent completes a task, it publishes the result to a shared state store and enqueues the next task for the appropriate agent.
Flow
- User request lands in an ingress queue.
- A planner agent reads the request, breaks it into subtasks, and enqueues each subtask to the correct agent queue.
- Specialist agents (code generator, data analyst, security scanner) pull from their queues, execute, and write results to DynamoDB.
- A final aggregator agent reads from the state store and assembles the response.
This avoids a single coordinator becoming a bottleneck. Each agent scales independently. The tradeoff is eventual consistency: agents see stale state if they read before a write completes.
Per-Agent Model Selection
Deep Insight uses a model registry that maps agent roles to model endpoints. The registry is a JSON configuration file deployed with the system.
{
"planner": {
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
"temperature": 0.2,
"max_tokens": 4096
},
"code_generator": {
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
"temperature": 0.7,
"max_tokens": 8192
},
"security_scanner": {
"model": "anthropic.claude-3-haiku-20240307-v1:0",
"temperature": 0.0,
"max_tokens": 2048
}
}
Each agent reads its own configuration at startup. The planner uses a low-temperature model for deterministic task decomposition. The code generator uses higher temperature for creative solutions. The security scanner uses a fast, cheap model because it runs on every code artifact.
The orchestrator does not decide. Agents own their model choice. This decouples deployment: you can swap the code generator's model without redeploying the planner.
Reliable Operation and Partial Failure Handling
Deep Insight uses idempotent task IDs and exponential backoff retries. Each task gets a UUID. If an agent crashes mid-execution, the task remains in the queue. Another worker picks it up and checks the state store for partial results. If the task ID exists, the agent resumes from the last checkpoint.
Failure Modes
| Failure Type | Handling Strategy | Recovery Time |
|---|---|---|
| Agent crash | Task remains in queue, retry with new worker | < 30 seconds |
| Model timeout | Exponential backoff (2s, 4s, 8s), then dead-letter queue | < 60 seconds |
| Invalid output | Schema validation, retry with prompt refinement | < 45 seconds |
| Downstream service unavailable | Circuit breaker, fallback to cached result | Immediate |
Dead-letter queues capture tasks that fail after three retries. A separate monitoring agent inspects these for patterns (e.g., all code generation tasks failing on a specific model version).
Secure Code Execution
Deep Insight sandboxes agent-generated code in AWS Lambda with restricted IAM roles. Each code execution request triggers a new Lambda invocation with:
- No network access (VPC with no internet gateway).
- Read-only access to a specific S3 prefix for input data.
- Write-only access to a different S3 prefix for output.
- 15-second timeout.
- 512 MB memory limit.
The Lambda runtime is a minimal container with Python 3.11 and no additional packages. If the agent needs external libraries, it must declare them in a manifest. A separate approval agent reviews the manifest before installing dependencies.
Privilege Escalation Prevention
The Lambda execution role cannot:
- Assume other roles.
- Modify IAM policies.
- Access secrets outside its designated namespace in Secrets Manager.
- Invoke other Lambdas.
If agent-generated code attempts any of these, the Lambda runtime returns a 403 and logs the attempt to CloudWatch. A security monitoring agent flags repeated attempts as potential adversarial behavior.
Observability for Multi-Agent Failures
Deep Insight instruments every agent with AWS X-Ray traces. Each task gets a trace ID that propagates through the entire agent chain. When a user request fails, you can reconstruct the full execution path: which agents ran, which model calls succeeded, where timeouts occurred.
Key Metrics
- Agent latency: P50, P95, P99 per agent type.
- Model call success rate: Percentage of successful inference requests per model.
- Queue depth: Number of pending tasks per agent queue.
- Dead-letter queue size: Number of tasks that failed after retries.
CloudWatch dashboards show these in real time. Alarms trigger when queue depth exceeds 100 or dead-letter queue size exceeds 10.
Debugging a Failure
- User reports a failed request.
- Look up the trace ID in X-Ray.
- See that the code generator agent timed out after 15 seconds.
- Check the model call logs: the model returned a 200 but the response was incomplete (truncated at max_tokens).
- Increase max_tokens for the code generator in the model registry.
- Redeploy the agent configuration.
Architecture Summary
Deep Insight's production architecture uses:
- SQS queues for task distribution.
- DynamoDB for shared state.
- Lambda for sandboxed code execution.
- Bedrock for model inference.
- X-Ray for distributed tracing.
- CloudWatch for metrics and alarms.
Each agent is a containerized service running on ECS Fargate. Agents scale independently based on queue depth. The model registry is a versioned JSON file in S3.
Technical Verdict
Use Deep Insight's patterns when:
- You need to scale individual agents independently (e.g., code generation is 10x slower than planning).
- You require secure code execution with strict privilege boundaries.
- You want to swap models per agent without redeploying the entire system.
- You need distributed tracing to debug multi-agent failures.
Avoid this architecture when:
- Your agents need strong consistency (e.g., financial transactions). The eventual consistency model will cause race conditions.
- You have fewer than three agents. The orchestration overhead is not worth it for simple chains.
- Your agents need to share large state (> 400 KB). DynamoDB item size limits will force you to use S3, which adds latency.
- You need sub-second end-to-end latency. The queue-based design adds 200-500ms per hop.
The Deep Insight reference implementation is open source. You can inspect the orchestration flow, model selection logic, and Lambda sandbox configuration in the GitHub repository.
Top comments (0)