DEV Community: Pedro Santos

Part 3 - Agents That Diagnose, Plan, and Query a Distributed Saga

Pedro Santos — Mon, 13 Apr 2026 23:00:00 +0000

In the previous posts, I set up LangChain4j and connected AI agents to 5 microservices via MCP. The plumbing was done. Now for the actual agents, the part that made me rethink how I approach operations in distributed systems.

I built 3 agents, each with a different trigger and a different job. None of them are chatbots. They’re background workers and query interfaces that use LLMs to reason over real system data.

Agent 1: OperationsAgent (Auto-Diagnosis on Failure)

Trigger: Kafka consumer on notify-ending topic (only when status = FAIL)
Job: Figure out why a saga failed, find similar past incidents, write a diagnostic report
Storage: pgvector (embeddings) + PostgreSQL (diagnostics table)

This was the first agent I built, and it’s the one that surprised me the most.

How It Works

Every saga, whether it succeeds or fails, ends with a notify-ending event on Kafka. My agent listens to that topic:

@KafkaListener(
    topics = "${spring.kafka.topic.notify-ending}",
    groupId = "ai-agent-group")
public void onSagaEnded(String payload) {
    Event event = objectMapper.readValue(payload, Event.class);

    // Vectorize ALL events, builds the historical base
    String historyText = buildHistoryText(event);
    vectorize(event, historyText);

    // Diagnose only failures
    if (event.getStatus() == FAIL) {
        diagnose(event, historyText);
    }
}

Two things happen here. First, every event gets vectorized, converted to an embedding and stored in pgvector. This builds up a knowledge base over time. Second, failures get diagnosed.

The RAG Pipeline

The diagnosis uses RAG. Before asking the LLM anything, I search for similar past incidents:

private String findSimilarIncidents(String historyText) {
    var queryEmbedding = embeddingModel.embed(historyText).content();
    var results = embeddingStore.search(
        EmbeddingSearchRequest.builder()
            .queryEmbedding(queryEmbedding)
            .maxResults(3)
            .minScore(0.75)
            .build());

    if (results.matches().isEmpty())
        return "No similar incidents found in history.";

    return results.matches().stream()
        .map(m -> "--- Similar incident (score=" +
            String.format("%.2f", m.score()) + ") ---\n" + m.embedded().text())
        .collect(Collectors.joining("\n\n"));
}

The embedding model is Ollama’s nomic-embed-text. Runs locally and costs nothing. The vector store is pgvector on PostgreSQL. Nothing exotic.

Then I build a prompt with the saga history + RAG context and pass it to the agent:

private void diagnose(Event event, String historyText) {
    String ragContext = findSimilarIncidents(historyText);
    String prompt = """
        SAGA FAILED, DIAGNOSE
        OrderId: %s | TransactionId: %s
        Final status: %s | Total amount: R$ %.2f

        SAGA HISTORY:
        %s

        SIMILAR INCIDENTS (RAG):
        %s
        """.formatted(
            event.getOrderId(), event.getTransactionId(),
            event.getStatus(), totalAmount, historyText, ragContext);

    String diagnosis = operationsAgent.analyze(prompt);

    diagnosticRepository.save(SagaDiagnostic.builder()
        .orderId(event.getOrderId())
        .diagnosis(diagnosis)
        .createdAt(LocalDateTime.now())
        .build());
}

The Agent Definition

The agent itself is minimal, just a system prompt defining the output format:

public interface OperationsAgent {

    @SystemMessage("""
        You are a failure diagnosis specialist for distributed sagas.
        You receive the full history of a FAIL saga and similar past incidents.

        Required format:
        ROOT CAUSE: <service and reason>
        AFFECTED SERVICES: <list>
        FINANCIAL IMPACT: <based on totalAmount>
        HISTORICAL PATTERN: <if RAG found similar cases>
        RECOMMENDATION: <corrective action>

        Rules:
        1. Only use the provided context, never invent data.
        2. If no similar incidents found, say so.
        3. Be concise, consumed by a monitoring system.
        """)
    String analyze(@UserMessage String context);
}

No tools here. The OperationsAgent doesn’t need to query anything. All the data arrives via the Kafka event + RAG. It just needs to reason over the context and produce a structured report.

What It Catches

After running this for a while, it started finding patterns I hadn’t noticed. Payment failures from new customers during late hours. Inventory rollbacks always hitting the same product. Fraud scores spiking for a specific order amount range. The RAG context gets better as more events accumulate. The agent learns from your system’s history.

Agent 2: SagaComposerAgent (Dynamic Saga Planning)

Trigger: Scheduled, every 60 seconds in dev, every 30 minutes in production
Job: Decide the optimal execution order for each customer profile
Storage: Redis with TTL (saga-plan:{profile})

This is the weird one. Instead of hardcoding the saga step order, I let the AI decide it based on actual failure data and system metrics.

The Idea

My saga has a default order: Product Validation → Payment → Inventory. If Payment is failing 40% of the time, it’d be smarter to run it first. Fail fast, avoid unnecessary validation calls.

Same logic applies to fraud. A “new customer + high value order” profile with a 30% fraud block rate probably needs a Fraud Validation step before Payment.

How It Works

Every minute, the agent runs for each customer profile:

@Scheduled(fixedDelayString = "${saga.composer.interval:60000}")
public void recomputePlans() {
    for (String profile : profiles) {
        String ragContext = findHistoricalPatterns(profile);
        String metrics = queryMetrics(dataAnalystAgent);
        String stockAlerts = queryStockAlerts(dataAnalystAgent);

        String prompt = buildCompositionPrompt(profile, metrics, stockAlerts, ragContext);
        String planJson = sagaComposerAgent.compose(prompt);

        redis.opsForValue().set("saga-plan:" + profile, planJson, 35, MINUTES);
    }
}

Notice something: the SagaComposerAgent uses the DataAnalystAgent to get current metrics. Agents calling agents.

The Agent Definition

The system prompt is very specific about the output format and decision rules:

public interface SagaComposerAgent {

    @SystemMessage("""
        You are a saga plan architect. Respond ONLY with raw JSON.
        First character MUST be '{', last MUST be '}'.

        Response format:
        {
          "steps": ["PRODUCT_VALIDATION", "FRAUD_VALIDATION", "PAYMENT", "INVENTORY"],
          "reasoning": "reason for the chosen order"
        }

        Decision rules:
        1. Place high-failure services earlier to fail fast.
        2. If INVENTORY failure rate > 30%, place before PAYMENT.
        3. Include FRAUD_VALIDATION for new + high-value or high fraud rate.
        4. Skip FRAUD_VALIDATION for VIP with < 5% fraud and long positive history.
        5. If data is insufficient, use default order.
        """)
    String compose(@UserMessage String profileContext);
}

The Orchestrator Reads the Plan

On the orchestrator side, when a saga starts, it checks Redis:

public String getFirstTopicForOrder(Order order) {
    String profile = classifyProfile(order);  // e.g., "new:high-value"
    String json = redis.opsForValue().get("saga-plan:" + profile);

    if (json == null) return DEFAULT_FIRST_TOPIC;  // fallback

    var steps = objectMapper.readTree(json).get("steps");
    return resolveTopicFromStep(steps.get(0).asText());
}

If Redis has a plan, use it. If not, fall back to the default order. Redis could be down. The plan could be expired. The agent might not have run yet. Doesn’t matter. The AI layer is additive. It never breaks the existing flow.

Example Output

For a new:high-value profile with recent payment failures:

{
  "steps": ["PRODUCT_VALIDATION", "FRAUD_VALIDATION", "PAYMENT", "INVENTORY"],
  "reasoning": "New high-value customer profile. Historical fraud rate 18% warrants early fraud check. Payment placed after fraud validation to avoid unnecessary payment attempts on blocked orders."
}

For a vip:any profile with clean history:

{
  "steps": ["PRODUCT_VALIDATION", "PAYMENT", "INVENTORY"],
  "reasoning": "FRAUD_SKIP_REASON: VIP customer with 2% fraud rate and 98% success rate over 47 orders. No night order patterns detected."
}

Agent 3: DataAnalystAgent (Natural Language Queries)

Trigger: HTTP GET request to /api/agent/chat?question=...
Job: Answer operational questions by querying all microservices via MCP tools
Output: Human-readable analysis

This is the agent that uses MCP most heavily. It connects to all 4 microservices and has 12+ tools available.

The Agent Definition

public interface DataAnalystAgent {

    @SystemMessage("""
        You are a data analyst for distributed sagas. Answer using exclusively
        the available MCP tools. Never invent data.

        Workflow for finding failed sagas:
        1. Extract N from the question, default to 5.
        2. Call listRecentEvents(limit = N + 10) to get enough FAIL events.
        3. Filter where status=FAIL, take only the first N.
        4. For each failed saga:
           a. Call getOrderById(orderId) to get clientType, totalAmount.
           b. Extract hourOfDay from the event timestamp.
           c. Call getFraudRiskScore with the order data.
        5. Report only the N requested sagas.
        """)
    String analyze(@UserMessage String question);
}

The Critical Lesson: Workflow Instructions Beat Tool Descriptions

Look at the ## Workflow section in the system prompt. That’s the most important thing I learned building these agents.

At first, I just described the tools and let the model figure out the workflow. It worked… sometimes. Other times it would call tools in the wrong order, forget to filter by FAIL status, or process 15 sagas when I asked for 5.

Once I wrote explicit step-by-step instructions in the system prompt, the reliability jumped. I told the agent HOW to use the tools, not just WHAT they do. The model still decides which tools to call, but it follows the prescribed workflow.

Example Interaction

Question: “List the 5 most recent failed sagas and assess their fraud risk.”

The agent:

Calls listRecentEvents(limit=15) on order-service
Filters for FAIL status, takes first 5
For each: calls getOrderById() then getFraudRiskScore()
Returns a structured report:

Order 69b6c29f → Payment rejected (R$932.80 > R$500 limit). Fraud score: 12/100 APPROVED. No compensation needed, payment was never processed.

Lessons Learned (the Hard Way)

After building all 3 agents, here are the things I wish someone had told me:

1. MCP beats @Tool for microservices. Not even close. The decoupling alone is worth it. Any agent can connect to any service without code changes.

2. SystemMessage alignment is critical. If your system prompt mentions a tool that doesn’t exist, the agent fails silently. It tries to call it, gets no result, and gives a vague answer. I spent hours debugging this before I realized the prompt referenced getTransactionStatus but the tool was actually named getPaymentStatus.

3. JSON responses from tools win over key=value. I started with "status=SUCCESS | amount=150.00" and switched to ObjectMapper.writeValueAsString(). One line of code, zero parsing bugs on the model side.

4. maxOutputTokens matters more than you think. I set it to 1024 initially. Asking for 5 sagas + fraud scores was consistently truncated. Bumped it to 4096 and the problem disappeared.

5. Virtual threads are not optional. When an agent calls 5 MCP tools, those are HTTP calls. Without virtual threads, they’re sequential and slow. One line in application.yml:

spring:
  threads:
    virtual:
      enabled: true

Parallel MCP calls at zero cost.

6. The AI layer should always be additive. Every piece of AI in my system has a fallback. Redis plan not found? Use default saga order. Diagnosis fails? The saga still completes normally. The AI improves operations but never blocks them.

The Full Picture

Here’s what the system looks like with all 3 agents running.

An order comes in. The orchestrator checks Redis for an AI-generated plan and executes the saga. If the saga fails, the OperationsAgent diagnoses the failure using RAG and saves it to the database. Every minute, the SagaComposerAgent reads metrics and failure patterns, then writes new plans to Redis. And anytime, a developer can ask the DataAnalystAgent “why are payments failing for new customers?” and get a grounded answer.

The agents feed each other. The OperationsAgent’s vectorized events improve the SagaComposerAgent’s RAG context. The DataAnalystAgent’s metrics help the SagaComposerAgent make better plans. It’s a flywheel.

Try It Yourself

The entire project is open source with setup instructions:

Repo: github.com/pedrop3/saga-orchestration
LangChain4j: langchain4j.dev

You’ll need Docker Compose for the infrastructure (Kafka, PostgreSQL, Redis, MongoDB). You’ll also need Ollama for local embeddings and a Gemini API key (free tier works fine for testing).

The README has step-by-step instructions and a pre-configured Bruno collection with all the API requests.

If you have questions or find issues, open an issue on the repo or drop a comment here. I’m still iterating on the system prompts. They’re never really “done.”

This is part 3 of a 3-part series on integrating AI into a distributed saga system:

Part 1 - Why I Picked LangChain4j Over Spring AI
Part 2 - Connecting AI Agents to Microservices with MCP
Part 3 - Agents That Diagnose, Plan, and Query a Distributed Saga ← you are here

Part 2 - Connecting AI Agents to Microservices with MCP

Pedro Santos — Tue, 07 Apr 2026 18:32:17 +0000

Connecting AI Agents to Microservices with MCP (No Custom SDKs)

In the previous post, I showed how LangChain4j lets you build agents with a Java interface and a couple of annotations. But those agents were using @Tool, methods defined in the same JVM. Fine for a monolith, but I’m running 5 microservices.

I needed the AI agent in service A to call business logic in service B, C, D, and E. Without writing bespoke HTTP clients for each one.

That’s where MCP comes in, and it changed how I think about exposing business logic.

The Problem: @Tool Doesn’t Scale Across Services

In my saga orchestration system, I have:

order-service (port 3000): MongoDB, manages orders and events
product-validation-service (port 8090): PostgreSQL, validates catalog
payment-service (port 8091): PostgreSQL, handles payments and fraud scoring
inventory-service (port 8092): PostgreSQL, manages stock
orchestrator (port 8050): coordinates the saga via Kafka

And then there’s the ai-saga-agent (port 8099), the service that hosts my AI agents. It needs to query data from ALL other services.

With @Tool, I’d have to write HTTP clients and DTOs for each service. Error handling, retry logic, the whole nine yards. Every time a service adds a new capability, I’d update the agent’s code. Tight coupling everywhere.

MCP: One Protocol for Everything

MCP (Model Context Protocol) is basically USB for AI. Instead of writing custom integrations per service, you expose tools via a standard JSON-RPC protocol over HTTP/SSE. Any agent can connect, discover available tools, and call them.

The before/after in my codebase was dramatic.

Before (without MCP): Agent needs stock data, write InventoryHttpClient. Agent needs payment status, write PaymentHttpClient. Agent needs order details, write OrderHttpClient. New tool in inventory? Update the client, update the agent.

After (with MCP): Each service exposes an MCP server. Agent connects to http://localhost:8092/sse and automatically discovers getStockByProduct, getLowStockAlert, checkReservationExists. New tool? Just add it to the MCP server. The agent sees it on next connection.

Making a Microservice an MCP Server

Let me show you the actual code from my payment-service. It already had a PaymentService and a FraudValidationService, real business logic with database queries. I just needed to expose some of those methods as MCP tools.

Add the Dependency

implementation 'io.modelcontextprotocol.sdk:mcp:0.9.0'

Set Up the Transport

@Bean
public HttpServletSseServerTransportProvider mcpTransport() {
    return HttpServletSseServerTransportProvider.builder()
        .objectMapper(new ObjectMapper())
        .messageEndpoint("/mcp/message")
        .build();
}

@Bean
public ServletRegistrationBean<HttpServletSseServerTransportProvider> mcpServlet(
        HttpServletSseServerTransportProvider transport) {
    return new ServletRegistrationBean<>(transport, "/sse", "/mcp/message");
}

Register Your Tools

Here’s the key part. I’m reusing the same PaymentService and FraudValidationService beans that already exist:

@Bean
public McpSyncServer mcpServer(
        HttpServletSseServerTransportProvider transport,
        PaymentService paymentService,
        FraudValidationService fraudService) {

    return McpServer.sync(transport)
        .serverInfo("payment-mcp", "1.0.0")
        .capabilities(ServerCapabilities.builder().tools(true).build())
        .tools(
            getPaymentStatus(paymentService),
            getRefundRate(paymentService),
            getFraudRiskScore(fraudService)  // same business logic, now via MCP
        )
        .build();
}

Each tool needs four things. A name and description so the LLM understands what it does. A JSON schema for parameters. And a handler function that runs your actual business logic:

private SyncToolSpecification getPaymentStatus(PaymentService paymentService) {
    return tool(
        "getPaymentStatus",
        "Returns the current payment status for a given transaction. " +
        "Use to verify whether a payment was processed, pending, or refunded.",
        """
        {
          "type": "object",
          "properties": {
            "transactionId": {
              "type": "string",
              "description": "Transaction ID associated with the saga"
            }
          },
          "required": ["transactionId"]
        }
        """,
        args -> {
            String txId = (String) args.get("transactionId");
            return paymentService.findByTransactionId(txId)
                .map(p -> "status=" + p.getStatus()
                    + " | totalAmount=" + p.getTotalAmount()
                    + " | totalItems=" + p.getTotalItems())
                .orElse("No payment found for transactionId=" + txId);
        }
    );
}

Notice: no new code. The paymentService.findByTransactionId() method already existed. I’m just wrapping it with a description so the LLM knows when to call it.

What Each Service Exposes

I did this for all 4 services:

Service	MCP Tools
order-service	`getOrderById`, `listRecentEvents`, `getLastEventByOrder`
payment-service	`getPaymentStatus`, `getRefundRate`, `getFraudRiskScore`
inventory-service	`getStockByProduct`, `getLowStockAlert`, `checkReservationExists`
product-validation	`checkProductExists`, `checkValidationExists`, `listCatalog`

Each service keeps full ownership of its data. The MCP layer is just a thin exposure.

The Agent Side: Connecting as an MCP Client

Now on the ai-saga-agent, I connect to all these servers:

@Bean
public McpToolProvider mcpToolProvider() {
    return McpToolProvider.builder()
        .mcpClients(List.of(
            buildClient("http://localhost:3000/sse"),     // order
            buildClient("http://localhost:8091/sse"),     // payment
            buildClient("http://localhost:8092/sse"),     // inventory
            buildClient("http://localhost:8090/sse")      // product-validation
        ))
        .build();
}

private McpClient buildClient(String sseUrl) {
    return new DefaultMcpClient.Builder()
        .transport(new HttpMcpTransport.Builder()
            .sseUrl(sseUrl)
            .build())
        .build();
}

Then when I build an agent, I just pass the mcpToolProvider:

DataAnalystAgent agent = AiServices.builder(DataAnalystAgent.class)
    .chatModel(gemini)
    .toolProvider(mcpToolProvider)   // discovers tools from all 4 services
    .build();

That’s it. The agent now has access to 12+ tools across 4 services, without a single HTTP client written by hand.

The Saga Architecture (Quick Context)

For those not familiar with the Saga Pattern: it’s how you handle distributed transactions without two-phase commit. Instead of one big transaction, you have a chain of local transactions. If any step fails, you run compensating transactions to undo the previous steps.

My flow looks like this:

Order Service → Orchestrator → Product Validation → Payment → Inventory → Success
                                    ↑                  ↑          ↑
                                    └──── Rollback ←───┴──────────┘

Everything communicates via Kafka topics. The orchestrator listens for results and decides what to publish next. There’s a state transition table that maps (source, status) to the next topic:

Source	Status	→ Next Topic
ORCHESTRATOR	SUCCESS	product-validation-success
PRODUCT_VALIDATION	SUCCESS	payment-success
PAYMENT	SUCCESS	inventory-success
INVENTORY	SUCCESS	finish-success
INVENTORY	FAIL	payment-fail (rollback)
PAYMENT	FAIL	product-validation-fail (rollback)

The beauty of this setup is that the saga flow is deterministic and auditable. Every event is stored, every transition is logged.

@Tool vs MCP Tool: When to Use Each

After building this, my rule of thumb is simple:

Use @Tool when the logic lives in the same JVM as the agent. No network overhead, tightly coupled, only that agent can use it.

Use MCP when the logic lives in another service. Any agent can connect. The protocol is language-agnostic (just JSON-RPC), and adding new tools doesn’t require changes on the agent side.

In practice, my agents use MCP for everything. The only @Tool I still use is for utility functions that don’t belong in any microservice, like formatting helpers or date calculations.

Testing MCP Endpoints Manually

You can test MCP without an AI agent. It’s just HTTP:

# 1. Open an SSE session
curl http://localhost:8092/sse
# Returns a sessionId

# 2. List available tools
curl -X POST "http://localhost:8092/mcp/message?sessionId=YOUR_SESSION" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'

# 3. Call a tool
curl -X POST "http://localhost:8092/mcp/message?sessionId=YOUR_SESSION" \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc":"2.0","id":3,"method":"tools/call",
    "params":{"name":"getStockByProduct","arguments":{"productCode":"COMIC_BOOKS"}}
  }'

This is super useful for debugging. When an agent does something unexpected, I test the tool directly to check if it’s the tool or the prompt that’s wrong.

What’s Next

With MCP in place, the infrastructure was ready. But the interesting part is what the agents actually do with all these tools. In the next post, I’ll walk through the 3 agents I built. The OperationsAgent listens for failed sagas on Kafka and auto-diagnoses them using RAG. The SagaComposerAgent periodically rewrites the saga execution plan based on real failure data. And the DataAnalystAgent answers natural language questions like “list the 5 most recent failed sagas and assess their fraud risk.”

The code is all open source: github.com/pedrop3/sagaorchestration

This is part 2 of a 3-part series on integrating AI into a distributed saga system:

Part 1 - Why I Picked LangChain4j Over Spring AI
Part 2 - Connecting AI Agents to Microservices with MCP
Part - Agents That Diagnose, Plan, and Query a Distributed Saga

Part 1 - Why I Picked LangChain4j Over Spring AI

Pedro Santos — Tue, 31 Mar 2026 22:37:29 +0000

Distributed sagas are hard enough without AI. You're already dealing with compensating transactions, Kafka topics, state machines, and rollback chains across 5 microservices. Adding an AI layer on top sounds like a recipe for more complexity.
But that's exactly what this series covers: where AI actually helps in a saga-based architecture, and how to wire it up without making the system more fragile. The AI layer auto-diagnoses failures, dynamically reorders saga steps based on real failure data, and lets developers query the entire system in natural language.
This first post covers the foundation: why I went with LangChain4j as the Java SDK, the core concepts you need, and how to get a working agent running.

Why LangChain4j

If you're building AI-powered applications in Java, you're choosing between three options: Python's LangChain (separate stack), Spring AI (native Spring), or LangChain4j (standalone Java library). Here's how they compare on the things that matter for production:

LangChain4j took me by surprise. You define an agent as a Java interface, slap @SystemMessage on it, and you're done. No implementation class. The framework generates a proxy at runtime. It felt almost too simple, so I kept looking for the catch, but it held up in production.

Here's the actual comparison I wrote down in my notes:

	Python LangChain	Spring AI	LangChain4j
Ecosystem fit	New stack alongside your Java app	Native Spring	Zero friction in any Java project
Agent definition	Explicit chain construction	Works but needs extra wiring	Interface + annotation = done
API stability	Breaking changes between versions	0.x→1.x felt like a rewrite	Stable post-1.0, SemVer respected
MCP support	Full (Python SDK)	Full since 1.1 GA	Full, client + server out of the box

Spring AI 1.1 has caught up on MCP support, which is great. But LangChain4j's agent definition model and API stability won me over.

Core Concepts

Before jumping into code, here's the mental model. There are really just 4 things you need to understand:

Models: the AI engine. You send text, it predicts tokens back. It has no memory between calls. LangChain4j abstracts this behind a ChatModel interface, so you can swap between Gemini, Ollama, OpenAI, or Claude with one line.

Agents: a loop. The model receives a task, decides which tools to call, calls them, looks at the results, and repeats until it's done. In LangChain4j, you define this as a Java interface.

Tools: Java methods that the model can invoke. You annotate a method with @Tool, and the model sees its signature and description. It decides when to call it. You don't write if/else routing logic, the LLM figures it out.

RAG: Retrieval Augmented Generation. Before asking the model a question, you search your own database for relevant context and inject it into the prompt. This is how you get answers based on your data without retraining the model.

Setting Up Your First Chat Model

Let's start with the basics. You need a ChatModel, your connection to the LLM.

Option 1: Gemini (Cloud)

Add the dependency:

implementation "dev.langchain4j:langchain4j-google-ai-gemini:1.11.0"

Build the model:

GoogleAiGeminiChatModel gemini = GoogleAiGeminiChatModel.builder()
    .apiKey(System.getenv("GEMINI_API_KEY"))
    .modelName("gemini-2.5-flash")
    .temperature(0.0)
    .maxOutputTokens(1024)
    .build();

Option 2: Ollama (Local, Free)

Ollama runs LLMs on your machine. Llama, Mistral, Gemma, Qwen — over 100 models, one ollama pull away. No API key, no cloud account, your data stays local.
I rely on it for two things in this project. First, as a chat model during development.I don't want to burn Gemini quota every time I tweak a system prompt. Second, and more importantly, for embeddings. The nomic-embed-text model (~274MB) is what powers the entire RAG pipeline: vectorizing saga events, searching for similar past failures, feeding context into the diagnosis agent. It runs in mill

brew install ollama
ollama pull llama3
ollama pull nomic-embed-text   # for embeddings later
ollama serve

Add the dependency:

implementation "dev.langchain4j:langchain4jollama:1.11.0"

LangChain4j integration — chat and embeddings:

// Chat model — swap for Gemini in prod
OllamaChatModel ollama = OllamaChatModel.builder()
    .baseUrl("http://localhost:11434")
    .modelName("llama3")
    .temperature(0.0)
    .build();

// Embedding model — used for RAG (vectorizing saga events)
OllamaEmbeddingModel embeddingModel = OllamaEmbeddingModel.builder()
    .baseUrl("http://localhost:11434")
    .modelName("nomic-embed-text")
    .build();

In practice: Ollama handles all embeddings (even in production, it's that reliable), and I only reach for Gemini when the task needs heavier reasoning. Check the model library to see what's available.

The Beautiful Part: Swap With One Line

Both implement the same ChatModel interface:

ChatModel model = isProduction ? gemini : ollama;

Your agent code doesn't change. I use Ollama locally and Gemini in staging/production.

Your First Agent

Here's where LangChain4j shines. Define an interface:

public interface DataAnalystAgent {

    @SystemMessage("""
        You are a data analyst for distributed sagas.
        Answer operational questions using the available tools.
        Never invent data.
        """)
    String analyze(@UserMessage String question);
}

Build it:

DataAnalystAgent agent = AiServices.builder(DataAnalystAgent.class)
    .chatModel(gemini)
    .build();

String answer = agent.analyze("What's the current refund rate?");

That's it. No implementation class. LangChain4j creates a proxy that handles the conversation loop, tool calling, and response parsing.

Adding Tools

Without tools, the agent can only generate text from its training data. Tools let it access real data.

public class OrderTools {

    @Tool("Returns stock for a product. Use to check availability.")
    public int getStock(@P("Product code") String code) {
        return inventoryRepo.findByCode(code).getAvailable();
    }

    @Tool("Returns the fraud risk score for an order.")
    public String getFraudScore(double amount, String type, int hour) {
        return fraudService.calculate(amount, type, hour);
    }
}

DataAnalystAgent agent = AiServices.builder(DataAnalystAgent.class)
    .chatModel(gemini)
    .tools(new OrderTools())
    .build();

Now when you ask "Is COMIC_BOOKS in stock?", the model sees the getStock tool, decides to call it with "COMIC_BOOKS", gets the result, and formulates a response. You didn't write any routing logic.

Here's what happens under the hood:

LangChain4j sends the method signature to Gemini as a functionDeclaration
Gemini responds with a functionCall: "I want to call getStock with code=COMIC_BOOKS"
LangChain4j intercepts this, runs your Java method, gets the result
It sends the result back to Gemini as a functionResponse
Gemini generates the final answer using the real data

A Quick Note on Tokens and Cost

Every word you send and receive costs tokens. On Gemini Flash, input tokens cost about $0.075 per million and output about $0.30 per million. Pretty cheap. But thinking tokens (internal reasoning) can be $3.50 per million.

Some settings I use to keep costs predictable:

OllamaChatModel.builder()
    .numPredict(512)        // caps output tokens
    .numCtx(32768)          // context window size
    .temperature(0.0)       // deterministic, no wasted sampling
    .repeatPenalty(1.2)     // avoids loops, shorter responses
    .think(true)            // free locally, expensive on cloud
    .listeners(List.of(new TokenUsageListener()))
    .build();

That TokenUsageListener logs input/output tokens per call. I learned the hard way that think(true) on Ollama is free locally but those thinking tokens count as INPUT on cloud APIs. It can 10x your cost per call.

What's Next

In the next post, I'll show how I used all of this to build an AI layer on top of a distributed saga system: 5 microservices coordinated via Kafka, with each service exposing its business logic as MCP tools that any agent can discover and call remotely.

Everything I covered here (and in the next posts) is already implemented in a complete, working project. The repo includes the 5 microservices, the Kafka-based saga orchestration, the MCP tool layer, and the AI agents, all wired together and ready to run: github.com/pedrop3/sagaorchestration

This is part 1 of a 3-part series on building AI-powered microservices with LangChain4j:

Part 1 - Why I Picked LangChain4j Over Spring AI
Part 2 - Connecting AI Agents to Microservices with MCP
3 Agents That Diagnose, Plan, and Query a Distributed Saga