Intuit's EWOK Agent: How Production Failover Became a Plain-Language Request with Audit Trails

Disaster recovery at scale is hard. Runbooks are long, steps are interdependent, and on-call engineers need to execute them under pressure. Intuit built EWOK Agent, an agentic disaster recovery assistant on Amazon Bedrock, to let engineers trigger production failovers from a plain-language request while keeping every action audited, policy-compliant, and safe.

This is not a chatbot that summarizes runbooks. EWOK executes real infrastructure changes: DNS failover, database promotion, traffic rerouting. The agent interprets intent, maps it to approved procedures, invokes tools, and logs every decision for compliance review. The plumbing matters because mistakes are expensive and every action must be reversible.

Why Disaster Recovery Needs Agents

Traditional DR automation falls into two camps:

• Fully manual runbooks: Engineers follow checklists. Slow, error-prone, and stressful during outages.
• Hardcoded scripts: Fast but brittle. Every edge case requires a new script. No natural language interface, no context awareness.

Agents sit in the middle. They understand intent, adapt to context, and execute multi-step procedures while enforcing guardrails. For DR, this means:

• On-call engineers can say "failover QuickBooks to us-west-2" instead of running 12 CLI commands.
• The agent checks policy (is this engineer authorized? is the target region healthy?) before executing.
• Every tool invocation is logged with reasoning, timestamps, and approval chains.

The risk is obvious: an agent with production write access can cause outages if it misinterprets intent or bypasses safety checks. EWOK's architecture addresses this with policy enforcement, audit trails, and rollback mechanisms.

Architecture: Policy Boundaries and Tool Orchestration

EWOK is built on Amazon Bedrock with a custom orchestration layer. The flow looks like this:

1. Natural language input: Engineer submits a request via Slack or CLI.
2. Intent parsing: Bedrock LLM extracts intent, target service, and region.
3. Policy check: Before any tool call, EWOK validates against a policy engine (who can failover what, under what conditions).
4. Tool invocation: If approved, the agent calls AWS APIs (Route 53, RDS, ECS) via a tool registry.
5. Audit log: Every decision, tool call, and result is written to an immutable log (S3 + CloudWatch).
6. Rollback hook: If a step fails, the agent can invoke a rollback procedure or escalate to a human.

The policy engine is the critical piece. It's not just RBAC. It's context-aware:

• Time-based rules (no failovers during peak hours without VP approval).
• Health checks (don't failover to a region that's already degraded).
• Dependency graphs (if you failover service A, you must also failover service B).

EWOK doesn't let the LLM decide these rules. The LLM interprets intent and maps it to a procedure. The policy engine enforces boundaries.

Tool Registry and State Management

EWOK's tool registry is a curated set of DR operations. Each tool is:

• Idempotent: Running the same tool twice doesn't cause double failovers.
• Atomic: Tools either complete fully or roll back cleanly.
• Versioned: Every tool has a schema version so the agent knows what parameters are required.

Example tool: failover_rds_cluster

{
  "name": "failover_rds_cluster",
  "version": "v2",
  "parameters": {
    "cluster_id": "string",
    "target_region": "string",
    "force": "boolean"
  },
  "preconditions": [
    "target_region_healthy",
    "no_active_failover_in_progress"
  ],
  "rollback": "promote_original_primary"
}

The agent doesn't call arbitrary AWS APIs. It calls tools from this registry. If the LLM hallucinates a tool name or parameter, the invocation fails at the registry layer, not at the AWS API layer.

State management is handled with a workflow engine (likely Step Functions or a custom state machine). Each failover is a workflow instance with:

• Current step: Which tool is executing.
• Checkpoint: State snapshot before each tool call.
• Rollback stack: List of inverse operations to undo completed steps.

If a failover fails halfway through, the agent can either retry the failed step or execute the rollback stack to restore the original state.

Audit Trail Architecture

Every EWOK action generates an audit event. The log schema includes:

• Request ID: Unique identifier for the failover request.
• User identity: Who initiated the request (engineer, service account).
• Intent: Natural language input and parsed intent.
• Policy decision: Which rules were evaluated and whether they passed.
• Tool calls: Which tools were invoked, with parameters and results.
• Timestamps: Start, end, and duration for each step.
• Approval chain: If the request required human approval, who approved it and when.

These logs are immutable (write-once to S3 with object lock) and indexed in CloudWatch for querying. Compliance teams can run queries like:

• "Show all failovers initiated by user X in the last 30 days."
• "Which failovers bypassed the health check precondition?"
• "How many rollbacks were triggered by agent errors vs. infrastructure failures?"

The audit trail is not just for compliance. It's also a feedback loop for improving the agent. If the LLM consistently misinterprets a certain type of request, the logs show the pattern and engineers can add examples to the prompt or refine the tool schema.

Failure Modes and Rollback Safety

EWOK's failure modes fall into three categories:

Failure Type	Example	Mitigation
Intent misinterpretation	LLM thinks "failover QuickBooks" means "failover TurboTax"	Policy engine rejects unauthorized service, logs the attempt, escalates to human
Partial execution	DNS failover succeeds but database promotion fails	Workflow engine detects failure, executes rollback stack to restore DNS, alerts on-call
Policy bypass	Agent tries to skip health check precondition	Tool registry enforces preconditions before invocation, logs the violation, halts execution

Rollback safety is built into the tool design. Each tool declares its inverse operation. For example:

• failover_rds_cluster has rollback promote_original_primary.
• update_route53_record has rollback restore_previous_record.

The workflow engine maintains a rollback stack. If step 5 of a 10-step failover fails, the engine executes the inverse of steps 4, 3, 2, 1 in reverse order. This is not perfect (some operations are hard to reverse cleanly), but it's better than leaving the system in a half-failed state.

Observability and Human-in-the-Loop

EWOK exposes real-time observability through:

• Slack notifications: Each step of the failover posts to a dedicated channel with status and next action.
• Dashboard: Live view of in-progress failovers, pending approvals, and recent completions.
• Approval gates: For high-risk operations (like failing over a revenue-critical service), the agent pauses and waits for human approval before proceeding.

The approval gate is not a chatbot asking "are you sure?" It's a structured decision point with context:

• What is the agent about to do?
• What preconditions passed?
• What are the rollback options if this fails?
• Who else has been notified?

The human can approve, reject, or modify the plan. If they modify it, the agent re-runs the policy check and updates the workflow.

Code Snippet: Policy Check Before Tool Invocation

Here's a simplified example of how EWOK enforces policy before calling a tool:

def invoke_tool(tool_name, parameters, user_context):
    # Load tool definition from registry
    tool = tool_registry.get(tool_name)
    if not tool:
        raise ToolNotFoundError(f"Tool {tool_name} not in registry")

    # Evaluate policy
    policy_result = policy_engine.evaluate(
        user=user_context.user_id,
        action=tool_name,
        resource=parameters.get("cluster_id"),
        context={
            "time": datetime.utcnow(),
            "region": parameters.get("target_region"),
            "service_health": get_service_health(parameters.get("target_region"))
        }
    )

    if not policy_result.allowed:
        audit_log.write({
            "event": "policy_violation",
            "user": user_context.user_id,
            "tool": tool_name,
            "reason": policy_result.reason
        })
        raise PolicyViolationError(policy_result.reason)

    # Check preconditions
    for precondition in tool.preconditions:
        if not check_precondition(precondition, parameters):
            audit_log.write({
                "event": "precondition_failed",
                "tool": tool_name,
                "precondition": precondition
            })
            raise PreconditionFailedError(f"Precondition {precondition} not met")

    # Execute tool
    result = tool.execute(parameters)

    # Log invocation
    audit_log.write({
        "event": "tool_invocation",
        "user": user_context.user_id,
        "tool": tool_name,
        "parameters": parameters,
        "result": result,
        "timestamp": datetime.utcnow()
    })

    return result

This pattern ensures that no tool runs without passing policy and precondition checks. The audit log captures both successful invocations and rejected attempts.

Deployment Shape

EWOK runs as a set of microservices on ECS:

• Agent orchestrator: Receives requests, calls Bedrock, manages workflow state.
• Policy engine: Evaluates rules, returns allow/deny decisions.
• Tool executor: Invokes AWS APIs, handles retries and rollbacks.
• Audit logger: Writes events to S3 and CloudWatch.

The orchestrator is stateless. Workflow state lives in DynamoDB with TTL for cleanup. The policy engine caches rules in memory but reloads from S3 every 60 seconds to pick up updates.

Secrets (AWS credentials, API keys) are stored in Secrets Manager and rotated automatically. The agent assumes an IAM role with least-privilege permissions: it can only call the specific AWS APIs needed for DR operations, and only for resources tagged with ewok:managed=true.

Technical Verdict

Use EWOK's pattern when:

• You have complex, multi-step operational procedures that need to be executed under pressure.
• You need auditability and compliance for every action (regulated industries, SOC 2, ISO 27001).
• You want to reduce human error in high-stakes operations without removing human oversight.
• You can invest in building a policy engine and tool registry (this is not a weekend project).

Avoid this pattern when:

• Your DR procedures are simple enough that a bash script or Terraform module is sufficient.
• You can't tolerate the latency of LLM inference during an outage (EWOK adds seconds to each decision).
• Your team doesn't have the expertise to debug agent failures (when the LLM misinterprets intent, you need to understand prompt engineering and tool schema design).
• You need sub-second failover (agents are great for orchestration, not for real-time systems).

The key insight is that EWOK doesn't replace engineers. It gives them a safer, faster interface to execute procedures they already know. The agent handles the tedious parts (remembering the exact sequence of steps, checking preconditions, logging everything) while humans make the high-level decisions (when to failover, which region to target, whether to proceed if a precondition fails).

Source Links

• How Intuit built an agentic disaster recovery assistant with Amazon Bedrock