Demo Scenario
This harness uses a game recommendation agent as its working scenario. The user describes preferences: relaxed pacing, short sessions, story, tactical depth, low stress, budget, or favorite genres. The harness turns that goal into an executable plan, collects data from local tools, and returns a final recommendation.
The task is small, but it has all the properties of an agentic pipeline. The data lives in independent sources: genre facts, aggregated reviews, a game catalog, and prices. The get_genre_facts, get_genre_reviews, get_games, and get_prices nodes run in parallel. Their results then converge into summarizer_node, which receives the user's preferences and the collected data, then produces the final report through Spring AI.
From there, the scenario goes through one pipeline: planning, graph validation, parallel execution, verification, and final report assembly.
The scenario data lives in GameRecommendationData: Outer Wilds, Stardew Valley, Baldur's Gate 3, Civilization VI, Hades, genre facts, reviews, and fake prices. The local data layer makes the run reproducible and keeps the focus on the key parts of the harness: planner structured output, DAG validation, parallel execution, budget pressure, recovery decisions, trace events, and the final verification gate.
The whole run can be represented as one pipeline:
User Goal
|
v
Planner ---> Plan
|
v
Validator
|
v
DAG Scheduler ---> Tools
|
v
Verifier
|
v
FinalReport
Budget / Trace / Recovery run alongside the whole pipeline:
- Budget constrains planning and execution.
- Trace records events at every stage.
- Recovery sends a failed run back to the planner when replan is possible.
Model Call
All LLM calls in the harness go through Spring AI ChatClient. For the rest of the code, the model looks like a regular runtime component: it receives a system prompt, user input, and an expected response shape, then returns a structured result or text. This keeps orchestration separate from the details of a specific LLM API.
The model is currently used in two places. SpringAiPlanner receives the user's goal and the tool catalog, then returns a structured Plan through provider structured output. summarizer_node inside GameRecommendationTools receives already collected data and produces the final recommendation text. These roles are separated at the code level: the planner is responsible for the execution shape, while the summarizer is responsible for the user-facing answer.
The key point in the planner: the model returns a domain model Plan, not arbitrary text.
ResponseEntity<ChatResponse, Plan> response = chatClient.prompt()
.system(SYSTEM_PROMPT)
.user(userPrompt(request))
.call()
.responseEntity(Plan.class, spec -> spec.useProviderStructuredOutput().validateSchema());
Plan plan = response.entity();
AiUsage usage = usageExtractor.extract(response.response());
return new PlanningResult(plan, usage);
After every LLM call, the harness extracts usage metadata through AiUsageExtractor: model, input tokens, output tokens, and total tokens. That data goes into Budget, trace events, and the final RunResult. The LLM remains a pluggable runtime component, while validation, execution, budget, and verification continue to live around it as separate layers.
Tools
Tools in the harness are declared as regular Java methods with Spring AI @Tool and @ToolParam annotations. From these annotations, Spring AI builds tool definitions: name, description, and input schema. ToolCatalog turns them into one catalog read by the planner, validator, and orchestrator.
ToolCallbackProvider is a Spring AI interface for supplying tool callbacks. The harness uses the built-in MethodToolCallbackProvider: it takes an object with @Tool methods and turns those methods into ToolCallback instances available through getToolCallbacks().
public ToolCatalog(GameRecommendationTools tools) {
this.callbackProvider = MethodToolCallbackProvider.builder()
.toolObjects(tools)
.build();
}
Each ToolCallback contains a ToolDefinition: tool name, description, and input schema. ToolCatalog extracts these definitions and turns them into a compact ToolDefinitionView understood by the harness.
public List<ToolDefinitionView> definitions() {
return Arrays.stream(callbackProvider.getToolCallbacks())
.map(ToolCallback::getToolDefinition)
.map(definition -> new ToolDefinitionView(
definition.name(),
definition.description(),
definition.inputSchema(),
roleOf(definition.name())))
.toList();
}
The catalog is used for more than the planner prompt. It becomes the source of truth for plan validation. DagValidator checks every node against real tool names, validates argument names, required parameters, and references to other node results. A wrong tool or argument name stops the run before the execution phase. ToolCatalog also assigns each tool a role, so the harness can distinguish data tools from the final synthesis node.
One important detail in this harness is explicit argument passing. deps define execution order, while arguments define data flow. LITERAL is used for values that the model extracts or formulates from the user request. NODE_RESULT is used for values that come from another node's result. The validator also checks that NODE_RESULT references an existing node and that this node is listed in deps.
This contract makes tools suitable for an agentic runtime. The planner sees real schemas, the scheduler receives already validated arguments, the budget counts every tool call, and the trace shows which tool ran, with what status and role: DATA or FINAL_SYNTHESIS.
Planner Graph
The planner returns the domain model Plan. Inside it is a set of PlanNode objects: each node contains an id, tool name, argument list, and dependencies in deps. This format turns the model response into an executable structure that can be validated before launch.
public class PlanNode {
private String id;
private String tool;
private List<ArgumentBinding> arguments;
private List<String> deps;
private NodeStatus status = NodeStatus.PENDING;
...
}
deps describe the graph structure: which node must finish before another. arguments describe data flow: which values the model extracts from the user request and which values come from previous node results. The validator and scheduler use this separation to check execution order and data flow independently.
Before execution, DagValidator checks the plan as a structure. It indexes nodes by id, checks references in deps, verifies that tools exist in ToolCatalog, rejects cycles, checks required arguments, and requires exactly one FINAL_SYNTHESIS node. After this check, the scheduler receives a valid DAG.
Parallel Graph Execution
After validation, the plan goes to DagScheduler. It starts with root nodes that have no dependencies, then launches downstream nodes when their dependencies reach a terminal state. In the demo scenario, get_genre_facts, get_genre_reviews, get_games, and get_prices can start immediately, while summarizer_node starts after the required data nodes complete successfully.
The scheduler keeps runtime state in SchedulerState. It indexes nodes by id, builds a dependents map, counts remaining dependencies for each node, and separately tracks failed dependencies. This lets the graph move forward through node completion events: a completed node either opens the path for its children or closes it through a propagated skip.
In an event-driven scheduler, independent graph branches continue as their own dependencies become ready. If one data node finishes quickly, its downstream node can start immediately while other independent tool calls are still running.
Event-driven DAG scheduling
120 ms facts ───────────────> normalize_facts ───────┐
180 ms games ───────────────> filter_games ──────────┤
700 ms prices ───────────────> attach_prices ─────────┤
900 ms reviews ───────────────> normalize_reviews ─────┘
│
v
summarizer_node
Synchronization appears only where the graph itself requires it. In the example above, summarizer_node waits for all data branches, while intermediate steps inside each branch do not wait for neighboring branches to finish.
A compressed skeleton shows the core mechanics:
for (PlanNode node : state.initialReadyNodes()) {
schedule(node);
}
state.awaitCompletion();
private void onNodeCompleted(NodeExecutionOutcome outcome) {
for (PlanNode child : state.dependentsOf(outcome.node())) {
if (state.dependencyCompleted(child, outcome.successful()) != 0) {
continue;
}
if (state.hasFailedDependencies(child)) {
skipNode(child, "dependency failed");
continue;
}
schedule(child);
}
state.nodeTerminal();
}
Before starting a scheduled node, the scheduler charges one tool call against the budget. If the budget cannot start the call, the node is marked SKIPPED with reason budget exhausted, and that outcome is propagated to dependent nodes. If the tool completes successfully, the scheduler stores result and usage on the node, charges token and cost usage, marks the node DONE, and checks which dependent nodes are ready to start.
Failure stays attached to the node that produced it. ToolExecutionException preserves its HarnessErrorCode; other unexpected exceptions become TOOL_EXECUTION_FAILED. If a dependency fails or is skipped, downstream nodes that require it are marked SKIPPED with reason dependency failed.
Parallelism is capped by harness.execution.max-concurrency. Internally, DagScheduler uses a fixed thread pool, so fan-out stays bounded. The scheduler launches ready nodes, records node.start, node.finish, node.fail, and node.skip events, updates dependent state, and waits for the graph to finish.
Execution Context
Context in the harness lives in layers. Each layer appears at its own stage of the run and passes forward only the data needed by the next phase.
Request scope starts with RunRequest: the user goal and sessionId. goal becomes the main input for the planner, while sessionId flows through trace events and the final RunResult to link the run to an external session.
Planning attempt scope appears inside each planning attempt. The planner receives goal, the current tool catalog, and failureContext. On the first attempt, failureContext is empty. After a validation failure or execution failure, the orchestrator fills it with the failure description and starts a new planning attempt. The result of this phase is the generated Plan.
Execution scope lives inside the DAG. As nodes execute, they accumulate result, usage, error, and errorCode. These values are used for passing NODE_RESULT arguments, classifying errors, making recovery decisions, and final verification. Execution context stays attached to specific nodes, so downstream logic can see the source of every result or failure.
Run scope covers the whole run. Budget tracks resources from planning through verification, Tracer writes events at every stage, and the final RunResult collects status, report, plan, verdict, budget snapshot, and trace events. From it, you can see which plan was built, which nodes executed, where a failure appeared, and what decision the recovery logic made.
Long-term memory, session history, and retrieval over previous runs are outside the scope of this article. The focus here is the context of a single run: request, planning attempt, execution state, and final result.
Executor Verification
After the execution phase, the harness runs ReportVerifier. It checks the structural correctness of the result: the plan contains exactly one final synthesis node, that node completed successfully, all of its dependencies are done, the result implements FinalReport, and the report text is present.
The check is based on tool roles from ToolCatalog. Regular data tools collect inputs, while summarizer_node is marked as FINAL_SYNTHESIS. The verifier finds the final node by tool role, so the result does not depend on the specific node id chosen by the planner.
This verification gate catches basic orchestration errors: empty reports, multiple final nodes, missing dependencies, failed data tools, skipped nodes, and wrong result types. On failure, the orchestrator returns FAILED_VERIFICATION, stores the reason in RunResult, and writes a verification.finish event to the trace.
This version uses structural verification. An LLM judge can be added as a separate layer on top of ReportVerifier when there are criteria for text quality, recommendation relevance, and completeness of the explanation.
Budget and Recovery
The harness tracks budget across several dimensions: tokens, tool calls, wall-clock time, and estimated cost. All of these limits converge into one pressure value. It reports the maximum utilization across all resources, so the run stops on the most constrained limit. This uses max, not an average or weighted score: exceeding any hard limit must stop the run regardless of the state of the other resources.
public synchronized double pressure() {
return max(
tokenPressure(),
toolCallPressure(),
wallClockPressure(),
estimatedCostPressure()
);
}
The budget is updated during planning and execution. When each node starts, the scheduler charges one tool call. After LLM calls, usage metadata turns into token and cost accounting. A budget snapshot goes into trace events and the final RunResult, so after the run you can see how many resources went into planning, execution, and final synthesis.
The second part of operational logic is error classification. The tool layer returns HarnessErrorCode, then ErrorClassifier maps it into one of four groups: VALIDATION, MISSING_INFO, TRANSIENT, or FATAL.
return switch (code) {
case UNKNOWN_TOOL, MISSING_REQUIRED_ARGUMENT, INVALID_ARGUMENT -> ErrorClass.VALIDATION;
case MISSING_INFO -> ErrorClass.MISSING_INFO;
case RATE_LIMITED, TIMEOUT -> ErrorClass.TRANSIENT;
case TOOL_EXECUTION_FAILED -> ErrorClass.FATAL;
};
After classification, RecoveryPolicy chooses an action. VALIDATION and MISSING_INFO lead to REPLAN, because the problem can be fixed with a new plan. TRANSIENT leads to RETRY for tool execution. FATAL leads to HALT.
return switch (errorClass) {
case VALIDATION, MISSING_INFO -> RecoveryAction.REPLAN;
case TRANSIENT -> RecoveryAction.RETRY;
case FATAL -> RecoveryAction.HALT;
};
Retry logic is intentionally omitted for simplicity. In this model, RETRY refers to transient tool execution failures: RATE_LIMITED and TIMEOUT. In a production version, this should be node-level retry inside DagScheduler: retrying a specific PlanNode, enforcing an attempt limit, using backoff, charging budget for every attempt, and recording separate node.retry events in the trace.
Recovery is expressed through run phases and state transitions. PlanningLoop handles validation failures, ExecutionEngine handles execution failures, and both can move the run into REPLANNING when RecoveryPolicy chooses REPLAN, budget still has room, and maxReplans has not been reached. The next planning attempt receives the original goal, the same tool catalog, and the previous failure context.
Tracing
Every run leaves behind trace events. This is an append-only event log that can reconstruct the execution path: when the run started, how planning and validation went, which nodes started, which completed, where an error appeared, which decision recovery logic made, and what budget state the run had at the end.
A trace event stores technical and operational information in one flat record.
public record TraceEvent(
Instant timestamp,
String runId,
String sessionId,
String kind,
String role,
String nodeId,
String status,
Long latencyMs,
String message,
BudgetSnapshot budget,
Map<String, Object> data
) {
}
kind describes the event type: run.start, planning.finish, validation.finish, node.start, node.finish, node.fail, node.skip, execution.finish, recovery.decide, verification.finish, run.finish. role links node events to the tool role: DATA or FINAL_SYNTHESIS. nodeId points to a concrete plan node. latencyMs shows tool call duration. budget records a resource snapshot at the time of the event.
The current implementation uses InMemoryTracer: it stores events in a list and returns a copy at the end of the run. For a local harness, that is enough: RunResult already contains the full trace, which can be printed in the CLI, asserted in tests, or serialized outward.
Trace is especially useful on failures. If execution fails, the events show the failed node, error code, recovery decision, and budget pressure at the time of the error. This turns a run from a black box into a sequence of verifiable facts.
Orchestrator
The orchestrator is implemented as a state machine. A run moves through named phases, and each phase is handled by a small transition step that receives the current RunState and returns the next one.
RunState keeps the current phase, plan, failure context, attempt counter, runtime context, and final result. Recovery uses the same mechanism: a validation or execution failure can return a REPLANNING state with updated failureContext and incremented attempt counter.
RunResult run(RunRequest request) {
RunState state = lifecycle.create(request);
while (!state.terminal()) {
state = transition(state);
}
return state.result();
}
private RunState transition(RunState state) {
return switch (state.phase()) {
case CREATED -> lifecycle.start(state);
case PLANNING -> planningLoop.plan(state);
case VALIDATING -> planningLoop.validate(state);
case EXECUTING -> executionEngine.execute(state);
case VERIFYING -> verify(state);
case REPLANNING -> planningLoop.replan(state);
case RETRYING -> executionEngine.retry(state);
case SUCCEEDED, FAILED, BUDGET_EXHAUSTED -> throw new IllegalStateException(...);
};
}
The code reads like a transition table: one phase, one handler, one returned state. Replanning and retrying are visible as regular transitions.
What Remains Offstage
The next layer is quality evaluation across many runs. A single successful CLI run only shows one specific happy path. Confidence requires eval cases: different user preferences, budget constraints, tool errors, invalid plans, and replan scenarios.
Long-term memory is also a next step. Today, context lives inside one run: goal, failureContext, node results, budget snapshot, and trace events. Sessions will need a separate history store, a policy for selecting relevant past runs, and context limits before sending anything to the planner.
Another open layer is production hardening for tools. Local data tools are safe for a demo, but real tool calls may access the network, read databases, mutate state, or call external APIs. Those tools need timeouts, idempotency, sandboxing, rate limits, audit logs, and human approval for irreversible actions.
Verification can also grow. The current ReportVerifier checks result structure. The next level is domain quality checks: whether recommendations match preferences, whether prices are not invented, whether the explanation is complete, and whether the response format is stable. These checks can be deterministic, LLM-based, or mixed.
The main idea stays the same: every new layer should plug into the harness as a separate part. Eval suite, session memory, production tool policies, and a stronger verification gate should not bloat the orchestrator. The runtime state machine keeps execution order, while new capabilities are added around it through explicit contracts.
The full demo is available at https://github.com/lbobylev/harness-demo.
Top comments (0)