Your Java LLM agent works in the demo. Friday night in production is another story.

Lago Moreno — Wed, 21 Oct 2026 08:35:16 +0000

Friday, 6:40 PM.

Picture an agent that handles customer refunds. In the demo it was perfect: it reads the request, checks the order, calls the refund tool and returns clean JSON. Everyone is happy.

Then comes the first Friday night in production.

6:40 PM. The provider starts returning 503s. Every request waits for the timeout before failing, and the queue keeps growing.
6:52 PM. The model decides to wrap its JSON in a Markdown code fence. The downstream service can't parse it anymore.
7:15 PM. A customer asks for a €4,000 refund. The agent calls refund. Nobody thought to put a check in front of an action like that.

Monday morning, two questions arrive: what did that weekend cost, and what exactly did the agent do? Nobody can answer with confidence.

The scene is made up. But if you've run an LLM agent in production, you've probably met at least one piece of it. And notice: none of these problems is about how smart the model is.

TL;DR — ARA is an open source (Apache 2.0) Java 21 runtime for LLM agents, built on LangChain4j. It handles output contracts, human approval, provider failover, spend caps and execution traces, and runs fully offline for tests. On Maven Central as io.github.xmor:ara-runtime:1.0.3.

The loop is easy. The rest is the problem.

Stripped down, an agent is a loop: send a request to the model, run the tool it picked, hand back the result, repeat until there's a final answer. You can write that in an afternoon, which is why so many tutorials stop there.

The real work starts after it:

How do you stop malformed output from reaching downstream systems?
How do you stop an agent from taking a critical action without sign-off?
What happens when the provider is down?
How do you cap cost?
How do you reconstruct a decision when something goes wrong?
How do you test all of this without a live API?

These share one trait: you shouldn't leave them to the model. An LLM can help decide what to do. It shouldn't be the only thing responsible for the rules that govern how that decision gets executed.

I work on the JVM, and I wanted these rules in one place: explicit, in Java code you can read, with tests you can repeat. That's how ARA started.

ARA in thirty seconds

ARA (Agent Runtime Architecture) is an open source runtime for LLM agents. It's written in Java 21, licensed under Apache 2.0 and published on Maven Central.

It's not a new model, and it doesn't replace the LLM integration layer. It sits on top of LangChain4j, which handles the providers (OpenAI, Anthropic, Ollama, Mistral, any OpenAI-compatible endpoint), and takes care of how agents run. It's a library, not a container: no mandatory annotations, no reflection to discover components, no Spring requirement.

The principle behind it: the model's probabilistic decisions and the runtime's deterministic rules should have separate owners.

If you want to follow along with the code, the repository is on GitHub. Let's go back to Friday night.

1. Model output is not a contract (6:52 PM)

The model wraps its JSON in a Markdown fence, or forgets a required field. The downstream service expects structured data and breaks.

A common fix is asking a second LLM to check the first one's answer. But a deterministic validation doesn't need another model call.

In ARA you declare an AgentContract with checks written in Java:

AgentContract contract = AgentContract.builder()
        // Input side: runs before the model sees anything
        .addInputProcessor(InputSanitizer.instance())     // blocks known prompt-injection patterns
        .addInputProcessor(ContentTruncator.to(4000))     // hard cap on input length
        // Declares the expected shape and appends it to the system prompt
        .outputSchema(JsonSchemaValidator.forOutput(SCHEMA))
        // Output side: plain Java, zero tokens, applied in order
        .addOutputProcessor(MarkdownFenceStripper.instance())          // strips the Markdown code fence around the JSON (6:52 PM)
        .addOutputProcessor(JsonSchemaValidator.forOutput(SCHEMA))     // rejects missing or wrong fields
        // On rejection, send the answer back with the reason, at most twice (these calls cost tokens)
        .outputRepairAttempts(2)
        .build();

// The contract is bound to the agent: every execute() goes through it
AraAgent agent = runtime.createAgent(config, contract);

Two separate mechanisms are at work.

The first is deterministic. MarkdownFenceStripper removes the 6:52 PM code fence before the downstream service ever sees it, and the schema validator catches the missing field. These checks run in plain Java and consume no tokens.

The second is recovery. With outputRepairAttempts(2), the model gets its rejected answer back along with the reason, and two chances to fix it. Those calls do cost tokens, so account for them separately.

If the answer still doesn't satisfy the contract, the task fails explicitly instead of passing invalid data along.

The point isn't to make the model infallible. It's to make sure its mistakes don't get accepted unchecked.

2. Some actions should wait for a human (7:15 PM)

An agent can have access to the right tool, understand the request and even write a convincing justification. None of that means it should execute immediately.

In ARA you configure an approval gate:

// Holds pending requests in memory (they don't survive a restart, see the limits below)
ApprovalGate gate = new InMemoryApprovalGate();

// Level 1: enable approvals on the runtime
AraRuntime runtime = AraRuntime.builder()
        .llmClient(llmClient)
        .approvalGate(gate)
        .build();

AraAgent agent = runtime.createAgent(
        AgentConfig.builder()
                .agentId("refund-agent")
                .agentType("support")
                // Level 2: opt this agent in; every tool call now waits for a human
                .humanApprovalRequired(true)
                // Tools are opt-in per agent: this one can only call "refund"
                .enabledTools(List.of("refund"))
                .build()
);

From then on, when the agent calls refund, the runtime parks the call and creates an approval request carrying the tool, its arguments and an expiry. Thanks to Java 21 virtual threads, waiting doesn't hold a platform thread, so many pending requests don't cost you resources.

The operator has three answers: approve (the refund goes through as is), reject with a reason (the agent receives it and can change course), or modify the arguments (correct the amount to €400 and the refund goes through with that).

// What the operator's UI (or your own endpoint) calls, one of these per request
gate.submit(requestId, new ApprovalDecision.Approved());                // run as proposed
gate.submit(requestId, new ApprovalDecision.Rejected("Too expensive")); // the agent sees the reason
gate.submit(requestId, new ApprovalDecision.Modified(newArgumentJson)); // run with corrected arguments, e.g. €400

ApprovalDecision is a sealed interface, so a switch over it must handle all three cases or it won't compile:

// Sealed interface: remove a case and this no longer compiles
switch (decision) {
    case ApprovalDecision.Approved a -> proceed();
    case ApprovalDecision.Rejected r -> abort(r.reason());
    case ApprovalDecision.Modified m -> proceed(m.newPayload());
}

(proceed() and abort() stand in for your own code.)

If nobody answers before the deadline, the call fails. The default is to do nothing.

One thing to know: the gate applies per agent, not per tool. With humanApprovalRequired(true), every tool call of that agent waits for a human. So keep sensitive actions in a dedicated agent, like the refund-agent above, and let read-only agents run without the gate.

The value isn't an "Approve" button. It's an explicit control point between the agent's decision and the action that changes the outside world. An agent can propose an action without having the authority to execute it.

3. When the provider goes down (6:40 PM)

At 6:40 the provider answers 503.

A retry may help, but repeating the same request against an unavailable service isn't a resilience strategy.

In ARA you configure an ordered chain of models:

// "smart", "local" and "cheap" are names of clients registered on the runtime:
// no credentials live in the agent config
AgentConfig config = AgentConfig.defaults()
        .agentType("resilient")
        .primaryLlm(LlmProfile.of("smart"))
        .fallbackLlms(List.of(              // tried in declaration order
                LlmProfile.of("local"),
                LlmProfile.of("cheap")
        ))
        // Move to the next model only on retryable errors (network, 5xx, 429);
        // auth errors and bad requests stop the chain
        .llmSelectionPolicy(LlmSelectionPolicy.FAILOVER)
        .build();

The interesting part is deciding when to switch. A network error, a 5xx or a rate limit moves on to the next model. An invalid API key or a malformed request stops the chain instead: it would fail identically on every model, and switching would only clutter your logs.

Then there's the growing queue. Each model in the chain sits behind a circuit breaker: after three consecutive failures it's skipped outright, so an outage stops costing you a timeout on every request. After thirty seconds a single probe call decides whether to reopen it. Health always comes from real traffic, never from background probes, which would cost money and burn the very rate limit you're trying to protect.

What's left is spend. In the model profile you set unit prices and a spending cap per agent. Every response reports tokens and cost, and when the cap is exceeded, execution stops. You find out from your code, not from Monday's invoice.

One caveat: a fallback model isn't equivalent to the primary one. Quality, latency and cost can all change. The chain is an explicit architectural choice, not a promise of identical results.

4. Monday morning: what did the agent do?

A reliable system doesn't just react to errors. It lets you understand what happened.

In ARA every response carries its own history. AgentResponse.steps() holds the steps the runtime recorded, including tool calls with their arguments and results, even when the execution fails. In that case you get the partial trace, which is exactly what you need to see where it broke. If you use OpenTelemetry, ARA emits a span per execution, per LLM call and per tool dispatch.

Two caveats. An operational trace records steps, calls and events, not the model's internal reasoning. And tracing doesn't remove the need to protect data: tool arguments and request contents still need to follow your own security and retention rules.

"Why not just assemble this from Resilience4j and LangChain4j?"

You can, and both are good libraries. Resilience4j gives you circuit breakers and retries; LangChain4j gives you the provider layer.

The work is in the glue: deciding that a 429 should fail over but a 401 shouldn't, wiring a spend cap into the same call path, parking a tool call on a human decision without tying up a thread, and making sure validation runs before output leaves the agent. ARA is that glue in one place, built on top of LangChain4j rather than instead of it.

If you'd rather own the glue, that's a fair choice. The four sections above are a decent checklist either way.

Try it without an API key

One thing I care about a lot: being able to develop and test agents without depending on a remote provider.

ARA includes ScriptedLlmClient, a test client that replays predefined responses. It doesn't simulate a real model's intelligence. It lets you check the runtime's behavior repeatably, under controlled conditions.

Add the dependency:

<dependency>
    <groupId>io.github.xmor</groupId>
    <artifactId>ara-runtime</artifactId>
    <version>1.0.3</version>
</dependency>

And write:

// AraRuntime is AutoCloseable: try-with-resources shuts it down cleanly
try (AraRuntime runtime = AraRuntime.builder()
        // Stub client: replays a fixed script, one entry per LLM call
        .llmClient(ScriptedLlmClient.script()
                .thenFinalAnswer("Virtual threads are lightweight JVM threads.")
                .build())
        .build()) {

    AraAgent agent = runtime.createAgent(AgentConfig.defaults()
            .agentType("assistant")
            .systemPrompt("You are a concise technical assistant.")
            .build());

    // Runs the real execution loop; only the model's answer is canned
    AgentResponse response = agent.execute(AgentTask.of("Explain virtual threads"));
    System.out.println(response.content());
}

No API key, no network, no cost. The runtime is the real one; only the answer is prepared in advance. The same approach works for tests: success paths, errors, tool calls, recovery conditions, without every test depending on an external provider. ARA tests itself this way.

When you want a real model, add ara-adapters and change one line:

// Pass this instead of ScriptedLlmClient: the rest of the code stays the same
LlmClient llm = AraLlmClientFactory.openAi()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-4o")
        .build();

You need Java 21 and Maven.

Don't take my word for it

People tend to oversell their own projects. So here's where you can check every claim in this post yourself.

Claim	Where to verify it in the repo
Output validated without spending tokens	`AgentContract`, `docs/CONTRACTS.md`
Human approval the compiler forces you to handle	`ApprovalDecision`, example `hitl/HumanInTheLoopExample`
Failover only on retryable errors	`FailoverLlmClient`, `LlmException.isRetryable()`
Per-agent spend cap	`LlmProfile` (`costBudget`)
A trace of every execution, even failed ones	`AgentResponse.steps()`
Tests without real providers	`ScriptedLlmClient` and the `src/test` folders

And one number: the sources contain 2,215 tests, all run offline by mvn verify.

What ARA is not (yet)

I'd rather tell you the limits now than have you discover them later.

It's a library, not a ready-to-install service. You integrate it into your application.
It can't resume an interrupted execution. There's no checkpoint to restart from yet, and pending approvals live in memory: if the process restarts, they're lost.
It doesn't design your security for you. Authorization, tool idempotency and data handling stay your application's decisions.
It's young. This is version 1.0.3, and the APIs will evolve.

If any of these is a blocker for you, open an issue. It's exactly the feedback that decides what comes next.

Why I built it

ARA started in the evenings, after work. By day I'm a consultant, and I've seen enough systems to know the problem rarely ends when the code works in the ideal case.

The project is built with heavy AI assistance, and I say so openly. Using AI to write code doesn't make design less important; it makes it more necessary. That's why the architectural rules are written down, dependencies are checked on every build, and every module has a test-coverage threshold that can only go up.

AI speeds up the writing. It doesn't take over responsibility for the decisions.

Next step

ARA is an independent project with no company behind it. It grows only if developers who find it useful say so. Three things you can do, from quickest to most involved:

Star the repo. Ten seconds. It's how other Java developers discover the project.
Run an example. Clone it and run hitl/HumanInTheLoopExample: it runs offline with no API key, and in five minutes you see the 7:15 PM refund handled.
Tell me what doesn't convince you. A debatable architectural choice, a case ARA handles badly: open an issue.

xmor / ara

ARA — Agent Runtime Architecture: Java 21 framework for building autonomous AI agents and multi-agent systems. LLM integration, tool calling, deterministic I/O contracts, RAG and human-in-the-loop. On Maven Central (io.github.xmor).

ARA — Agent Runtime Architecture for Java

Build autonomous AI agents and multi-agent systems in plain Java 21. LLM integration, tool calling, deterministic I/O contracts, RAG and human-in-the-loop — no annotations, no reflection, no Spring, no Kotlin.

Quick start · Documentation · Examples · Website · Discussions

ARA is a Java 21 agent runtime: a place where agents actually run, with sessions memory, execution strategies, cost budgets, approval gates and a full execution trace — not a thin wrapper around a chat completion endpoint.

It works with OpenAI, Anthropic, Ollama, Mistral, LM Studio, Groq and any OpenAI-compatible endpoint, and it is published on Maven Central as io.github.xmor:ara-runtime.

Your code ──▶ AraRuntime ──▶ Agent (strategy + contract + session)
                                │
                     ┌──────────┼───────────┐
                     ▼          ▼           ▼
                 LlmClient   Tools      Retriever
             OpenAI/Claude/  (yours,   (in-memory
             Ollama/Mistral  parallel   or Qdrant)
                             on vthreads)

Quick start — 60 seconds, no API key

1. Add the dependency…

View on GitHub

An agent can work perfectly in a demo. Taking it to production means dealing with everything that happens once ideal conditions stop existing. That's why I built ARA.

DEV Community: Lago Moreno

Hi everyone! I’m Moreno, a Java developer and consultant from Italy. In the evenings I dig into LLM agents and what breaks in production: broken JSON, provider outages, agents doing what they shouldn’t.

Welcome Thread - v 397