One domain broke. I refused to rerun the other five.

#terraform #ai #python #aws

TerraformAgent turns a plain-English infrastructure request into validated Terraform. Six nodes: an orchestrator plans the domains, a researcher pulls AWS docs, a set of domain subagents write code in parallel, an aggregator stitches it together, a reviewer inspects it with multiple LLMs, and an evaluator runs the real terraform fmt and terraform validate before anything ships.

The subagents run in parallel, one per domain. Networking writes network.tf, compute writes compute.tf, security writes iam.tf and kms.tf, and so on. On a typical request, four or five of them fire at once.

Then the evaluator finds a problem in exactly one of them. A wildcard IAM action, an invalid reference in network.tf, a validate failure that names a single file.

The easy move is to loop the whole pipeline back to the start and regenerate everything. That works. It also throws away four passing domains to fix one broken one, and every domain is a separate LLM call. On a five-domain request, a naive full rerun costs five calls to fix a one-call problem, and it does that on every retry.

I didn't want to pay for that. So the retry only reruns the domains that actually failed.

The pipeline, briefly

The graph is linear up to the evaluator, then it branches. After the evaluator runs, one conditional edge decides whether to finish or route back into the domain subagents. When it routes back, the subagents don't rerun everything. They rerun only the domains that got flagged.

The key decision: attribute the failure to a domain

The whole thing hinges on one question. When the evaluator or reviewer reports a problem, which domain owns it?

I answer that by string-matching filenames. Each domain owns a known set of files:

DOMAIN_FILES: Dict[str, List[str]] = {
    "networking": ["network.tf"],
    "compute": ["compute.tf"],
    "storage": ["storage.tf"],
    "database": ["database.tf"],
    "security": ["iam.tf", "kms.tf"],
}

When something fails, compute_domains_to_retry scans the combined text of the review feedback, the fmt output, and the validate output for any of those filenames. Every filename it finds maps back to the domain that owns it, and only those domains get rerun.

def compute_domains_to_retry(
    review_feedback: str, eval_results: Dict[str, Any], active_domains: List[str]
) -> List[str]:
    is_blocking = "BLOCKING_ERROR" in review_feedback
    fmt_failed = eval_results.get("terraform_fmt_pass") is False
    validate_failed = eval_results.get("terraform_validate_pass") is False

    if not (is_blocking or fmt_failed or validate_failed):
        return []

    combined_text = "\n".join([
        review_feedback,
        eval_results.get("terraform_fmt_output", ""),
        eval_results.get("terraform_validate_output", ""),
    ])

    known_files = [f for files in DOMAIN_FILES.values() for f in files]
    flagged_files = [f for f in known_files if f in combined_text]
    domains_to_retry = sorted({file_to_domain(f) for f in flagged_files if file_to_domain(f)})

    if not domains_to_retry:
        domains_to_retry = sorted(active_domains)

    return domains_to_retry

If validate fails with Error in network.tf: invalid reference, that returns ["networking"] and nothing else reruns. If the reviewer writes BLOCKING_ERROR: iam.tf has a wildcard action, that returns ["security"]. If both files show up, both domains come back.

There's a deliberate fallback in the last three lines. If something failed but no filename was attributable, it reruns every active domain rather than skip the retry. I would rather pay for a full rerun than silently pass broken code because I couldn't parse a blame target.

Wiring it into the graph

LangGraph does the routing with a conditional edge. The evaluator writes domains_to_retry into state, and a router function reads it:

def should_retry_or_finish(state: dict) -> str:
    if state.get("domains_to_retry"):
        return "domain_subagents"
    return "end"

workflow.add_conditional_edges(
    "evaluator",
    should_retry_or_finish,
    {"domain_subagents": "domain_subagents", "end": END},
)

When the subagents node runs on a retry, it reads that same list and only spins up workers for the flagged domains. Everything else keeps its previous output untouched:

active_domains = state.get("active_domains", [])
domains_to_retry = state.get("domains_to_retry", [])
domains_this_pass = domains_to_retry if domains_to_retry else active_domains

existing_outputs = dict(state.get("domain_outputs", {}))

with ThreadPoolExecutor(max_workers=max(len(domains_this_pass), 1)) as pool:
    futures = {
        pool.submit(_run_domain_subagent, domain, state, feedback): domain
        for domain in domains_this_pass
    }
    for future in as_completed(futures):
        domain = futures[future]
        existing_outputs[domain] = future.result()

existing_outputs starts as a copy of the previous pass, so the passing domains carry forward and only the flagged ones get overwritten. First pass runs everything, retry pass runs the subset.

The cap, and why it's there

A retry loop with no ceiling is a way to burn money on a domain the model can't fix. The evaluator increments a counter and stops routing back after three:

retry_count = state.get("retry_count", 0)
if domains_to_retry and retry_count < 3:
    next_status = AgentStatus.EVALUATING
    retry_count += 1
else:
    next_status = AgentStatus.COMPLETED
    domains_to_retry = []

Three is a guess, not a tuned number. It's enough for the model to recover from a bad reference or a wildcard it can see in the feedback, and low enough that a domain it fundamentally can't fix doesn't spin forever.

What I'd do differently

The attribution is string matching, and that's the weak point. It only works when the failure names a file that maps to a domain. A cross-file error, a validate message that doesn't mention a filename, or a provider-level error with no file in it all fall through to the "rerun everything" branch. When that happens, selective retry quietly becomes a full rerun and I've paid for the routing logic without getting the savings.

I have real test coverage on the happy paths. test_validate_failure_with_file_in_output confirms a network.tf error returns only ["networking"], and test_blocking_with_no_attributable_file_retries_all_active confirms the fallback reruns everything. What I don't have is a way to attribute a failure that doesn't name a file. The honest next step is to make the subagents emit structured findings tagged with their domain, so attribution reads a field instead of grepping a string. That would shrink the fallback-to-everything case, which is the only case where this design earns nothing.