mech.app

Posted on Nov 12 Originally published at mech.app

ReTrace: Distilling Agent Workflows into Planner-Free Code

#ai #llm #agents #performance

LLM agents can follow a Standard Operating Procedure written in natural language, but they call an inference endpoint at every decision point. That works for one-off tasks. When you run the same SOP a thousand times a day, the per-step latency and token cost becomes a production problem.

ReTrace solves this by tracing agent execution, extracting the decision logic from reasoning traces, and compiling a planner-free workflow. The workflow is deterministic code that selects the next tool call without an LLM in the loop. On production SOPs, ReTrace workflows reach 93.8% completion at 24% lower cost per task. On τ²-Bench, they run in 1.3 seconds instead of 170 seconds.

Why Workflow Distillation Matters

Agent frameworks like LangGraph and AutoGen orchestrate tool calls by asking the LLM "what next?" after every step. This pattern is flexible but expensive:

Every decision costs tokens and adds round-trip latency.
Repeated SOPs execute the same logic on different inputs.
The agent's reasoning trace already explains why it chose each step.

Workflow distillation extracts that logic once, then reuses it. The result is a state machine that calls tools directly, falling back to the agent only when it encounters an edge case.

How ReTrace Extracts Workflows

ReTrace takes three inputs: an SOP in natural language, a few logged agent runs, and the tool definitions. It produces an executable workflow in three stages.

1. Condition Extraction

The agent writes a reasoning trace before each tool call. ReTrace parses these traces to extract the condition that triggered the action:

Input: "The user requested a refund, so I will call check_refund_policy with order_id."
Extracted condition: user_intent == "refund"
Action: check_refund_policy(order_id)

This step relies on the agent's natural language explanation, not just the tool call signature. Tool calls alone do not reveal why the agent chose them.

2. Rule Merging

ReTrace merges conditions across multiple runs into a compact rule document. If three runs all call check_refund_policy when the user mentions "refund," "return," or "money back," the rule becomes:

IF user_intent IN ["refund", "return", "money_back"]
THEN check_refund_policy(order_id)

The rule document is platform-agnostic. It describes the decision logic without committing to a specific orchestration engine.

3. Code Generation

The final stage compiles the rule document into executable code for a target platform (LangGraph, Temporal, Prefect). The generated workflow is a state machine with explicit transitions:

def workflow(state):
    if state["user_intent"] in ["refund", "return", "money_back"]:
        result = check_refund_policy(state["order_id"])
        state["policy_result"] = result
        return "check_eligibility"
    elif state["user_intent"] == "track_order":
        return "fetch_tracking"
    else:
        return "fallback_to_agent"

Only this last stage is platform-specific. The same rule document can be compiled for different orchestration systems.

Fallback to Agent

The distilled workflow includes a fallback node. When the workflow encounters a condition not covered by the rule document, it hands control back to the full LLM agent. This happens when:

The input does not match any known condition.
A tool call fails in an unexpected way.
The user explicitly requests a deviation from the SOP.

The fallback is a safety valve, not a failure. The workflow logs the fallback event, and you can retrain the rule document to cover the new case.

Performance and Cost

ReTrace was tested on three benchmarks:

Benchmark	Workflow Success	Agent Success	Workflow Latency	Agent Latency	Cost Reduction
79 production SOPs	93.8%	96.8%	Not reported	Not reported	24%
τ²-Bench	75.8%	80.6%	1.3s	170s	Not reported
SOP-Bench (8 procedures)	80.6% (no repair)	85.5%	0.40s	46.6s	Not reported

With three rounds of repair on training failures, τ²-Bench workflows improved from 66.7% to 81.2% for k=2 workflows. SOP-Bench workflows reached 87.8% after repair, exceeding the agent's 85.5%.

The latency reduction is dramatic. On τ²-Bench, workflows run 130× faster. On SOP-Bench, they run 116× faster. This matters when you process thousands of tasks per day.

Repair Loop

ReTrace includes a repair mechanism. When a workflow fails on a training example, the system:

Logs the failure and the input that caused it.
Extracts a new condition from the agent's reasoning on that input.
Merges the new condition into the rule document.
Regenerates the workflow code.

Three rounds of repair on labeled training failures lifted SOP-Bench workflows from 80.6% to 87.8%. The repair loop is supervised: you provide labeled examples, and ReTrace updates the rule document.

Versioning and Testing

Distilled workflows introduce a new artifact to manage: the rule document. When the underlying SOP changes, you must:

Retrace the agent on new examples.
Merge the new conditions into the rule document.
Regenerate the workflow code.
Test the updated workflow against a regression suite.

The paper does not describe a versioning strategy. In production, you will need:

A version number for the rule document.
A mapping from rule version to workflow version.
A rollback plan when a new workflow underperforms.

The rule document is human-readable, so you can diff versions and review changes before deploying.

Failure Modes

Distilled workflows fail in predictable ways:

Condition drift: The SOP changes, but the rule document is stale. The workflow executes outdated logic.
Overfitting: The rule document captures noise from the training runs. The workflow fails on inputs that differ slightly from the examples.
Fallback thrashing: The workflow falls back to the agent too often, erasing the cost savings.

You can detect these failures by monitoring the fallback rate. If more than 10% of tasks hit the fallback, the rule document needs retraining.

Architecture

A production deployment of ReTrace requires:

Agent runtime: Executes the full LLM agent for tracing and fallback.
Trace storage: Logs agent runs, including reasoning traces and tool calls.
Rule compiler: Extracts conditions, merges rules, and generates workflow code.
Workflow runtime: Executes the distilled workflow (LangGraph, Temporal, etc.).
Fallback router: Detects when the workflow cannot proceed and hands control to the agent.
Repair pipeline: Retrains the rule document from labeled failures.

The rule compiler and repair pipeline run offline. The workflow runtime and fallback router run in the hot path.

Security Boundaries

The distilled workflow executes code generated from agent reasoning traces. This introduces a code injection risk if the agent's reasoning is not sanitized. An attacker could craft an input that causes the agent to write a malicious condition, which then gets compiled into the workflow.

Mitigations:

Validate extracted conditions against a schema before merging.
Sandbox the workflow runtime to limit the blast radius of a compromised workflow.
Review the rule document before deploying a new workflow version.

The fallback to the agent is also a security boundary. The agent has access to all tools, while the workflow may have a restricted set. Ensure the fallback does not bypass access controls.

Observability

You need visibility into:

Fallback rate: Percentage of tasks that fall back to the agent.
Workflow completion rate: Percentage of tasks that complete without fallback.
Latency distribution: P50, P95, P99 for workflow execution.
Cost per task: Token usage for workflows vs. agents.
Failure reasons: Why workflows fall back or fail.

The rule document itself is an observable artifact. Diff it across versions to understand how the workflow logic evolves.

Technical Verdict

Use ReTrace when:

You run the same SOP hundreds or thousands of times per day.
The SOP is stable enough that the decision logic does not change hourly.
Latency matters more than flexibility.
You have a few labeled examples to trace and a repair budget for edge cases.

Avoid ReTrace when:

The SOP changes frequently, requiring constant retraining.
You need the agent's full reasoning on every task for auditability.
The task is one-off or exploratory, not a repeated process.
You cannot tolerate the risk of stale or overfitted workflows.

Workflow distillation trades flexibility for speed and cost. It works when the task is repetitive and the decision logic is extractable from reasoning traces. It fails when the task is too dynamic or the agent's reasoning is too opaque.

DEV Community