DEV Community

mech.app
mech.app

Posted on Originally published at mech.app on

ADOP: Multi-Agent Orchestration for Bronze-to-Gold Data Pipelines

#agents #architecture #ai #aws

AWS published ADOP (Agentic Data Operations Platform) as a production reference architecture on Bedrock. The platform uses specialized AI agents to automate the full Bronze-Silver-Gold medallion pipeline lifecycle. Teams report compressing new-source onboarding from weeks to hours while keeping governance and compliance controls inline.

This is not a chatbot wrapper. ADOP exposes coordination patterns for stateful, multi-stage agent pipelines with hard governance constraints.

Architecture: Specialized Agents with Handoff Boundaries

ADOP deploys four agent types, each with a bounded domain:

  • Ingestion Agent: Connects to source systems, generates schema mappings, writes raw data to Bronze layer (S3 or Iceberg).
  • Transformation Agent: Applies business rules, deduplication, and type coercion to produce Silver layer tables.
  • Quality Agent: Runs validation checks, flags anomalies, and blocks promotion to Gold if thresholds fail.
  • Governance Agent: Enforces PII masking, retention policies, and audit logging across all layers.

Each agent runs as a Bedrock Agent with tool access to AWS Glue, Lake Formation, and Step Functions. The agents do not share state directly. Instead, they write metadata to a DynamoDB coordination table and emit events to EventBridge.

Coordination Without a Central Orchestrator

ADOP avoids a monolithic orchestrator. Instead, agents subscribe to EventBridge rules:

  1. Ingestion Agent completes a Bronze write and emits BronzeReady event.
  2. Transformation Agent picks up the event, reads Bronze metadata from DynamoDB, and starts Silver processing.
  3. Quality Agent listens for SilverReady, runs checks, and either emits GoldPromotable or QualityFailed.
  4. Governance Agent runs inline on every layer transition, blocking writes if policies fail.

This event-driven handoff keeps agents decoupled. If the Transformation Agent retries, it reads the last Bronze checkpoint from DynamoDB and resumes without re-ingesting.

State Management: Lineage and Checkpoints

Each pipeline stage writes a state record to DynamoDB with these fields:

Field Purpose Example
pipeline_id Unique identifier for the source-to-Gold flow crm_contacts_20260821
stage Current medallion layer bronze, silver, gold
checkpoint S3 path or Iceberg snapshot ID s3://bucket/bronze/crm/snap_123
status Agent execution state in_progress, completed, failed
lineage Upstream dependencies [bronze_snap_122]
governance_pass Boolean flag from Governance Agent true

When an agent retries, it queries DynamoDB for the last completed checkpoint and resumes from that snapshot. Lineage tracking ensures Gold tables trace back to specific Bronze versions, which matters for audit and rollback.

Governance Inline: Blocking Writes, Not Auditing After

The Governance Agent runs synchronously before each layer write. It does not audit after the fact. The flow looks like this:

# Transformation Agent calls Governance Agent before Silver write
governance_result = bedrock_agent.invoke(
    agent_id="governance-agent-xyz",
    input_text=f"Validate schema and policies for {silver_table_path}",
    session_state={
        "schema": silver_schema,
        "source_classification": "PII",
        "retention_days": 365
    }
)

if governance_result["status"] != "approved":
    raise GovernanceBlockedException(governance_result["violations"])

# Only write if governance passes
glue_client.create_table(silver_table_path, silver_schema)
Enter fullscreen mode Exit fullscreen mode

This inline check prevents non-compliant data from landing in Silver or Gold. The Governance Agent uses Lake Formation tags and AWS Config rules to enforce policies. If a PII column appears without masking, the write fails immediately.

Tool Calls and Bedrock Integration

Each agent has a tool manifest that maps natural language intents to AWS SDK calls. The Ingestion Agent's manifest includes:

  • connect_to_source: Wraps AWS Glue Connection API.
  • infer_schema: Calls Glue Crawler or runs Spark schema inference.
  • write_bronze: Writes Parquet or Iceberg to S3 with partitioning.

Bedrock Agents use Claude 3.5 Sonnet for reasoning and tool selection. The agent receives a prompt like "Ingest new CRM contacts table" and generates a plan:

  1. Call connect_to_source with JDBC credentials.
  2. Call infer_schema to detect columns.
  3. Call write_bronze with partition key ingestion_date.

The agent retries failed tool calls up to three times with exponential backoff. If all retries fail, it writes a failed status to DynamoDB and emits a PipelineFailed event.

Observability: Tracing Agent Decisions

ADOP logs every agent decision to CloudWatch Logs with structured JSON. Each log entry includes:

  • agent_id: Which agent made the decision.
  • tool_call: The AWS API invoked.
  • reasoning: Claude's natural language explanation.
  • latency_ms: Time from prompt to tool execution.

Step Functions tracks the overall pipeline state machine, but individual agent reasoning lives in CloudWatch. This split keeps high-cardinality logs (agent thoughts) separate from workflow state (Step Functions execution history).

For debugging, teams query CloudWatch Insights:

fields @timestamp, agent_id, tool_call, reasoning
| filter pipeline_id = "crm_contacts_20260821"
| filter status = "failed"
| sort @timestamp desc
Enter fullscreen mode Exit fullscreen mode

This surfaces why the Quality Agent blocked a Gold promotion or why the Ingestion Agent chose a specific partition strategy.

Failure Modes and Retry Strategy

Failure Type Detection Recovery
Source unavailable Ingestion Agent connection timeout Exponential backoff, max 3 retries, then alert
Schema drift Quality Agent detects column mismatch Block Silver write, emit SchemaDriftDetected event
Governance violation Governance Agent finds unmasked PII Block write, log violation, require manual approval
Transformation crash Step Functions timeout Resume from last Bronze checkpoint in DynamoDB
EventBridge delivery failure Dead-letter queue receives event Replay from DLQ after fixing downstream agent

The most common failure is schema drift. When a source system adds a column, the Quality Agent detects the mismatch and blocks the Silver write. The platform emits a SchemaDriftDetected event, which triggers a human-in-the-loop workflow in Step Functions. A data engineer reviews the change, updates the transformation logic, and re-runs the pipeline.

Deployment Shape

ADOP runs entirely on AWS managed services:

  • Bedrock Agents: Four agent instances (ingestion, transformation, quality, governance).
  • Step Functions: Orchestrates human-in-the-loop approvals and pipeline retries.
  • DynamoDB: Stores pipeline state and checkpoints.
  • EventBridge: Routes events between agents.
  • S3 + Iceberg: Stores Bronze, Silver, and Gold layers.
  • Lake Formation: Enforces fine-grained access control.
  • CloudWatch: Logs agent reasoning and tool calls.

The reference architecture deploys via CDK. Each agent gets its own IAM role with least-privilege access to Glue, S3, and Lake Formation. The Governance Agent has read-only access to AWS Config and Lake Formation tags but cannot write data.

Cost and Latency

AWS reports these benchmarks for a typical pipeline (10 GB source, 50 columns):

  • Ingestion: 2 minutes, $0.15 (Bedrock API + Glue Crawler).
  • Transformation: 5 minutes, $0.40 (Bedrock API + Glue Spark job).
  • Quality: 1 minute, $0.10 (Bedrock API + Athena queries).
  • Governance: 30 seconds, $0.05 (Bedrock API + Lake Formation checks).

Total end-to-end latency: 8.5 minutes. Total cost: $0.70 per pipeline run.

For comparison, a manual pipeline with the same scope takes 2-4 weeks of engineering time and costs $5,000-$10,000 in labor.

Technical Verdict

Use ADOP when:

  • You onboard new data sources frequently (weekly or monthly).
  • You need inline governance checks, not post-hoc audits.
  • Your team already uses the medallion architecture (Bronze-Silver-Gold).
  • You want agent reasoning to be auditable and traceable.

Avoid ADOP when:

  • Your pipelines have complex, stateful transformations that agents cannot reason about (e.g., time-series forecasting, graph algorithms).
  • You need sub-minute latency for real-time data.
  • Your governance policies change faster than you can update agent prompts.
  • You run on-premises or in a non-AWS cloud.

The platform shines for repetitive, schema-driven ETL work. It struggles with pipelines that require deep domain knowledge or custom algorithms. If your transformation logic fits in a SQL query or a Spark DataFrame operation, ADOP will compress your onboarding time. If it requires a PhD to understand, stick with hand-coded pipelines.

Source Links

Top comments (0)