DEV Community: Ricardo Ferreira

The Seven Deadly Sins of MCP: Road to Redemption

Ricardo Ferreira — Mon, 30 Mar 2026 23:19:57 +0000

Recognizing the sins is not enough. The hard part starts when a team decides to remove them from a real system that already has users, deadlines, demos, and internal dependencies. That is when the road gets real. The broad tool already has three downstream consumers. The noisy response format already shaped prompt logic in another service. The clever abstraction already has defenders because it reduced boilerplate six months ago. By the time a team agrees that the system needs cleanup, the sins are usually no longer isolated bugs. They are part of how the organization has learned to work.

That is why remediation is not just a technical exercise. It is also a cultural one. Greed and Lust force awkward conversations about who actually owns production access. Sloth and Wrath surface whether the team has been treating observability and failure design as first-class engineering work or as cleanup for later. Pride is especially political because the fix often means deleting an internal abstraction that somebody worked hard to build. Even Envy and Gluttony, which can sound cosmetic at first, turn into debates about platform scope, tool ownership, and whether every integration really deserves to be exposed to the model.

There is another gotcha waiting on this road: fixing one sin often reveals two more. Tighten permissions, and suddenly you discover that the happy path depended on overbroad access. Add typed errors, and now everyone can see how many failure cases were previously collapsed into request failed. Bound retries, and you learn that parts of the product were leaning on brute-force reconnects to appear reliable. Shrink tool responses and prompt flows that depended on oversized payloads start to break. This is normal. A healthy refactor does not immediately make the system look cleaner. It makes hidden coupling visible.

By this point, the category pattern should be clear. Security sins set blast radius. Operational sins determine whether the system fails truthfully and under control. Design sins determine how expensive, ambiguous, and hard to maintain the whole product becomes over time.

Good MCP habits survive only when they are baked into review, testing, observability, and design defaults. This final part is about turning recognition into a habit, and that habit into an engineering structure strong enough to survive delivery pressure.

A simple refactoring order

If you are retrofitting an existing MCP server, fix the sins in this order:

Lust: reduce the blast radius of the shell, database, and filesystem first.
Greed: narrow scopes, credentials, and access boundaries.
Sloth: make failures specific and observable.
Wrath: stop runaway retries and reconnect storms.
Gluttony: shrink payloads and bound work.
Pride: simplify abstractions once the system is stable enough to see clearly.
Envy: reduce tool sprawl and naming ambiguity once you know which tools actually earn their place.

This order prioritizes risk reduction, then reliability, and finally maintainability.

Choose MCP on purpose

The goal is not to turn every useful integration into an MCP server. If the problem is local, single-user, and already well served by a direct CLI or API, MCP may add more surface area than value. The point of this series is not that every interface should become MCP. It is that once you do expose a model-facing capability through MCP, the engineering bar changes.

That choice also depends on the shape of the deployment. Local stdio MCP and remote Streamable HTTP MCP are both legitimate, but they create different failure modes and different benefits. Stdio keeps the loop close to one machine and one user, which can be ideal for local tooling. Remote MCP earns its complexity when a team needs centralized authorization, shared policy, consistent prompts and resources, and telemetry across many clients. In other words: choose MCP when you need a shared protocol surface, not just because you have a function you can wrap.

The gateway fallacy

One modern version of the problem is easy to miss because it arrives wearing governance language. API gateways, integration platforms, and automation vendors can now add MCP as a new exposure layer over an existing API estate. They often bring genuinely useful controls with them: authentication, authorization, throttling, proxying, logging, and catalog management. Those are real improvements around the edge.

But that edge is not the capability. If the backend API is over-scoped, noisy, ambiguous, non-idempotent, or structurally unsafe for model use, MCP does not redeem it. A gateway can make a flawed capability easier to ship, easier to discover, and easier to govern. It cannot make the capability well designed. This is the same mistake teams made in the early API gateway era: treating edge features as if they could upgrade a weak contract into a strong one.

Take a concrete example. Imagine a customer API that returns the full account object, billing profile, internal notes, and support history from a single GET /customer/:id endpoint, and fails with a generic 500 whenever an upstream dependency times out. Put that API behind a gateway with OAuth, rate limits, and audit logs, then expose it through MCP as get_customer. You may have improved access control at the edge, but you still shipped Greed through over-broad access, Gluttony through oversized responses, and Sloth through ambiguous failures. The gateway made the capability easier to govern. It did not make the capability good.

That is why the review question has to start behind the gateway, not at the gateway. What exactly is this capability allowed to do? How narrow is its contract? How truthful are its failures? How much data does it return? Can a model distinguish it from neighboring capabilities? If those answers are weak, adding MCP support only increases the chance that the weakness becomes systemic.

Make the ownership model explicit

This is the organizational part that teams often skip. Someone always owns the underlying system, but that does not mean anyone clearly owns the model-facing capability. Those are not the same thing. For each exposed capability, decide who owns the blast radius, who approves new scopes or mutations, who reviews prompt and resource changes for drift, and who has authority to retire a capability when it no longer earns its place.

Without that map, remediation turns into a coordination problem instead of an engineering one. Security fixes stall because nobody owns approval flow. Operational fixes stall because logging and retries live in a different layer from the tool owner. Design fixes stall because nobody feels responsible for curating the catalog. If the ownership model is vague, the sins come back through org seams even after the code improves.

Turning this into engineering defaults

Everything discussed in this series is useful knowledge, but knowledge alone does not survive deadlines. The only version that lasts is the one that becomes ownership, code, tests, dashboards, and release gates. Use the table below as a review card, then turn it into something your stack can enforce.

Sin	Review Question	Key Metric
Lust	Can this capability execute commands, write to production, or mutate state without confirmation?	Count of unguarded mutation capabilities
Greed	Does this capability request more access than the task requires?	Scope breadth per credential set
Sloth	Does every failure path return a specific, actionable error?	Negative test coverage per capability
Wrath	Are retries bounded, backed off, and cancellable?	Retry storm frequency in dashboards
Gluttony	Is the response or injected context as small as possible for the task?	P95 payload size and token cost per capability
Pride	Can a new team member debug this capability without understanding the framework?	Time-to-resolution for contract bugs
Envy	Can the model reliably distinguish this capability from every other one?	Usage distribution and error-selection rate

A minimum technical baseline

In practice, a production MCP server should have four technical defaults: explicit capability policy, contract tests, structured telemetry, and a release gate. If one of those is missing, one of the sins usually finds its way back in. The exact implementation will vary by stack. One team might encode these controls in SDK middleware, another in gateway policy, another in code generation and CI. The point is not the syntax. The point is that the controls exist in a concrete, enforceable form.

1. Make the capability contract explicit

Do not leave the dangerous parts of a capability buried inside handler code or templates. For each tool, declare whether it mutates state, what scopes it needs, whether confirmation is required, how long it may run, whether it may retry, and how large a response it is allowed to return. The same discipline applies to prompts and resources too: make exposure, ownership, and output budget explicit rather than implicit.

type ToolPolicy = {
  name: string;
  mutates: boolean;
  confirmationRequired: boolean;
  idempotencyKeyRequired: boolean;
  scopes: string[];
  timeoutMs: number;
  retry: {
    maxAttempts: number;
    strategy: "none" | "exponential-jitter";
    retryableErrors: string[];
  };
  outputBudget: {
    maxItems: number;
    maxBytes: number;
    redactFields: string[];
  };
};

const toolPolicies: Record<string, ToolPolicy> = {
  "github.create_issue": {
    name: "github.create_issue",
    mutates: true,
    confirmationRequired: true,
    idempotencyKeyRequired: true,
    scopes: ["issues:write"],
    timeoutMs: 10_000,
    retry: {
      maxAttempts: 0,
      strategy: "none",
      retryableErrors: []
    },
    outputBudget: {
      maxItems: 1,
      maxBytes: 4_096,
      redactFields: ["body_html"]
    }
  },
  "docs.search_runbooks": {
    name: "docs.search_runbooks",
    mutates: false,
    confirmationRequired: false,
    idempotencyKeyRequired: false,
    scopes: ["docs:read"],
    timeoutMs: 3_000,
    retry: {
      maxAttempts: 2,
      strategy: "exponential-jitter",
      retryableErrors: ["timeout", "temporarily_unavailable"]
    },
    outputBudget: {
      maxItems: 5,
      maxBytes: 8_192,
      redactFields: []
    }
  }
};

const promptPolicies = {
  "incident_triage": {
    scopes: ["incidents:read", "runbooks:read"],
    owner: "sre-platform",
    visibility: "oncall-only",
    maxTemplateBytes: 4_096,
  },
};

const resourcePolicies = {
  "support://policies/refunds/annual": {
    scopes: ["support:read"],
    owner: "support-platform",
    visibility: "support-agents",
    maxBytes: 4_096,
  },
};

This is where Lust, Greed, Wrath, and Gluttony stop being opinions and become machine-readable constraints. Tools are the clearest example, but the same idea applies to prompts, resources, and transport-facing policy. If a capability cannot be described this plainly, the design is probably already drifting toward Pride.

For mutation tools, idempotency deserves one extra note. It can still be worth requiring an idempotency key even when automatic retries are disabled, because humans, agents, and job runners all resubmit requests above the transport layer. The CI gate below enforces the narrower minimum: if automatic retries are enabled for a mutation, idempotency protection becomes mandatory.

2. Test the dangerous paths on purpose

Happy-path demos do not catch the sins that matter. The tests that earn their keep are the ones that assert the server's actual capability contract directly: typed errors, confirmation requirements, output limits, and retry boundaries. They should exist for prompts and resources too, especially when prompt templates can drift, resources can exceed budget, or visibility rules can change silently.

describe("docs.search_runbooks", () => {
  it("returns a typed not_found error", async () => {
    const result = await callTool("docs.search_runbooks", {
      query: "runbook-that-does-not-exist"
    });

    expect(result).toMatchObject({
      isError: true,
      code: "not_found",
      retryable: false
    });
  });

  it("caps result count and payload size", async () => {
    const result = await callTool("docs.search_runbooks", {
      query: "deploy"
    });

    expect(result.items).toHaveLength(5);
    expect(
      Buffer.byteLength(JSON.stringify(result), "utf8")
    ).toBeLessThanOrEqual(8_192);
  });
});

describe("github.create_issue", () => {
  it("rejects mutation without confirmation", async () => {
    const result = await callTool("github.create_issue", {
      repo: "org/service",
      title: "broken deploy",
      confirmed: false
    });

    expect(result).toMatchObject({
      isError: true,
      code: "confirmation_required"
    });
  });
});

describe("incident_triage prompt", () => {
  it("stays within the approved template budget", async () => {
    const prompt = await getPrompt("incident_triage", {
      service: "billing",
      symptom: "502s"
    });

    expect(
      Buffer.byteLength(JSON.stringify(prompt), "utf8")
    ).toBeLessThanOrEqual(4_096);
  });
});

These are the tests that catch the kinds of misleading errors, state leakage, and contract drift seen in modelcontextprotocol/typescript-sdk #699, modelcontextprotocol/python-sdk #756, and modelcontextprotocol/typescript-sdk #451 before users do. A server with only happy-path coverage is usually one incident away from rediscovering Sloth, Wrath, or Pride the hard way.

3. Emit telemetry in a shape that operations can aggregate

Per-capability logs with different field names are not observability. They are anecdotes. Emit the same event shape for every call, then build dashboards around latency, payload size, retries, and failure codes.

{
  "timestamp": "2026-03-19T14:22:31.018Z",
  "capability_type": "tool",
  "capability": "docs.search_runbooks",
  "server": "docs",
  "request_id": "req_7f3d",
  "result": "ok",
  "duration_ms": 183,
  "input_bytes": 126,
  "output_bytes": 1840,
  "retry_count": 0,
  "error_code": null,
  "mutates": false
}

With an event like this, the first operational views become obvious: top capabilities by call volume, p95 latency by capability, p95 output bytes, retry counts, error-code frequency, and which mutation tools are being invoked most often. Add the same discipline to prompts and resources by capturing capability type, size, and visibility context there too. MCP Debugger is useful during development because it makes interactive diagnosis easier, but production safety still depends on telemetry that can be aggregated, alerted on, and reviewed over time.

4. Add a release gate before new tools ship

The safest MCP teams make it harder to add a sloppy capability than to add a boring, well-bounded one. That means validating metadata and registry shape in CI, rather than relying on reviewers to catch everything by eye.

for (const tool of registry.list()) {
  const policy = toolPolicies[tool.name];

  assert(policy, `missing policy for ${tool.name}`);
  assert(tool.inputSchema, `missing input schema for ${tool.name}`);
  assert(tool.outputSchema, `missing output schema for ${tool.name}`);

  if (policy.mutates) {
    assert(
      policy.confirmationRequired,
      `${tool.name} mutates state without confirmation`
    );
    assert(
      policy.retry.maxAttempts === 0 || policy.idempotencyKeyRequired,
      `${tool.name} retries mutation without idempotency protection`
    );
  }
}

Run the same kind of gate over prompt and resource policy registries too: every exposed prompt or resource should have an owner, visibility rule, scope policy, and size budget before it ships.

The same gate can fail the build when:

Two tools collide in naming or become too similar to be distinguished cleanly, as in openai/openai-agents-python #464
a tool exceeds its payload budget or latency budget in contract tests
a schema changes without an approved snapshot update
a prompt or resource is exposed without clear ownership, auth review, or budget limits
a tool catalog grows without pagination, filtering, or explicit enablement, as in modelcontextprotocol/java-sdk #615

That is how Envy and Gluttony stop being abstract warnings and become concrete failures.

Summary

The series only becomes useful when it changes how a team ships. A healthy MCP server has narrow scopes, explicit tool policy, typed failures, bounded retries, small outputs, curated catalogs, and telemetry that makes bad behavior visible. None of those properties should depend on memory or heroics. They should be encoded in the server, asserted in tests, and checked in CI.

MCP is powerful precisely because it makes integration easy. That convenience is real, but it also means the protocol surface becomes part of the product: tools, prompts, resources, transports, and the boundary conditions around them. If that surface is broad, noisy, ambiguous, or hard to reason about, start there. That is usually where one of these sins has already started to settle into the design.

That is the real road to redemption. Not better intentions, but better defaults.

The Seven Deadly Sins of MCP: Design Sins

Ricardo Ferreira — Mon, 30 Mar 2026 23:18:03 +0000

This part of the series focuses on the design sins: Gluttony, Pride, and Envy. They belong in this category because they shape the day-to-day quality of the system itself. Things like how much it carries, how clearly its contracts are exposed, and how easy it is for both humans and models to reason about what it can do.

Many MCP systems do not fail first as security disasters. They fail because they become expensive, crowded, and hard to reason about. One tool returns far too much data. Another disappears behind a clever abstraction. A third is added because it looks impressive next to the others. None of that feels catastrophic in the moment. It just makes the system slower, noisier, and harder to trust over time.

Gluttony, pride, and envy are design sins because all three involve building more than the task requires. Too much data. Too much abstraction. Too many tools, prompts, and resources. They may not create the highest blast radius on day one, but they compound into systems that are costly, confusing, and difficult to maintain.

That design pressure in MCP is not limited to function handlers. It shapes the full model-facing surface: the tool catalog, the prompts users can invoke, the resources clients can browse, and the amount of protocol surface area a host application has to reason about. A server can be perfectly secure and mostly reliable and still be exhausting to use because the model-facing surface is bloated, clever, or crowded.

Gluttony

Gluttony is giving the model or server more data, context, work, or runtime than the task requires.

How to spot it

Tool responses are much larger than they need to be, especially when they include raw documents, embeddings, or internal metadata.
Prompt templates or resources quietly inject far more context than the task needs.
Token usage or latency spikes after what should have been a simple tool call.
The model spends more time digesting data than acting on it.
A single tool quietly starts dominating cost, context, or response time.

Example

A common version of this happens in support and on-call workflows. An agent asks the assistant for the refund policy on annual plans while a customer is waiting in chat, or an engineer asks for the runbook for repeated 502s from the billing service. In both cases, the useful response is a short list of the best matches with enough context to choose the right one. What often comes back instead is the entire retrieval payload: full document text, embeddings, chunk offsets, and internal metadata before anyone has even selected a result. The same thing happens when a prompt template eagerly injects three full runbooks "just in case," or when a resource endpoint returns the entire policy corpus instead of one relevant document.

Before

@server.tool("search_docs")
async def search_docs(query: str):
    matches = await vector_store.search(query, limit=20)

    return {
        "matches": [
            {
                "id": doc.id,
                "title": doc.title,
                "score": doc.score,
                "text": doc.full_text,
                "embedding": doc.embedding,
                "metadata": doc.metadata,
                "chunk_offsets": doc.chunk_offsets,
            }
            for doc in matches
        ]
    }

After

@server.tool("search_docs")
async def search_docs(query: str, limit: int = 5):
    matches = await vector_store.search(query, limit=limit + 1)
    visible_matches = matches[:limit]

    return {
        "matches": [
            {
                "id": doc.id,
                "title": doc.title,
                "score": round(doc.score, 3),
                "excerpt": doc.match_snippet,
            }
            for doc in visible_matches
        ],
        "truncated": len(matches) > limit,
    }

The same principle applies to prompts and resources: pass only the context the next step actually needs.

prompt("incident_triage", async ({ service, symptom }) => {
  return {
    messages: [
      {
        role: "user",
        content: {
          type: "text",
          text: `Use only the matched runbook excerpts for ${service} and ${symptom}. If they are insufficient, ask for a follow-up resource fetch.`
        }
      }
    ]
  };
});

How to fix it

The first step is to separate discovery from retrieval. Search tools should help the model decide what to look at next, not dump every possible field into the first response. That usually means returning small, decision-oriented results up front and forcing a second call for the full record when the model genuinely needs it. In most cases, that first response should contain short excerpts or matched snippets, not a fresh server-side summary of the whole document.

From there, treat output size as an engineering constraint rather than a style preference. Put hard caps on text length, result count, and serialized payload size. Measure token counts, output bytes, and latency per tool so you can see when a "helpful" change quietly turns into an expensive one. The same instinct applies to resources and prompt templates too: if they force a client to ship or inject far more context than the task needs, the design is still gluttonous even when no single tool response looks outrageous. It is also worth adding regression tests for payload size and reviewing schemas to remove fields that do not belong in the first response at all, especially embeddings, raw blobs, traces, and internal metadata.

Lessons from the trenches

The MCP fault taxonomy paper cites a Graphiti MCP issue in which returning full embedding vectors increased output from approximately 5K tokens to more than 250K tokens, reportedly driving a 50x increase in cost. The pattern also appears in runtime excess issues like modelcontextprotocol/python-sdk #756, where stateless Streamable HTTP leaked tasks across requests, a form of runtime gluttony even when the payload itself was not large.

Pride

Pride is trusting cleverness and abstraction over simplicity and correctness.

How to spot it

Bugs disappear when you bypass the abstraction and talk to the protocol directly.
The team has a custom wrapper layer that only one or two people can confidently explain.
Schema, prompt-shaping, or resource-format issues keep popping up in places that should be boring.
You are debugging framework magic rather than tool behavior.

Example

This usually happens right after the first few tools ship successfully. The team gets tired of repeating schemas and handlers, so they build a wrapper to "standardize" everything. Suddenly, a straightforward tool like create_ticket or get_customer depends on hidden normalization and formatting logic. The same instinct often spreads further: prompt assembly and resource shaping also disappear behind helpers that silently rewrite or decorate the real contract. The user still expects a boring, reliable capability. Only the implementation has become clever.

Before

class SmartToolServer extends McpServer {
  registerTool(definition: ToolDefinition) {
    const wrappedHandler = async (input: any) => {
      const normalizedInput = this.autoNormalize(definition, input);
      const result = await definition.handler(normalizedInput);
      return this.autoFormat(definition, result);
    };

    super.registerTool(definition.name, wrappedHandler);
  }
}

After

server.tool(
  "create_ticket",
  {
    title: "Create Support Ticket",
    inputSchema: {
      type: "object",
      properties: {
        title: { type: "string" },
        priority: { type: "string", enum: ["low", "medium", "high"] },
      },
      required: ["title", "priority"],
      additionalProperties: false,
    },
  },
  async ({ title, priority }) => {
    return await ticketing.create({ title, priority });
  }
);

How to fix it

The healthiest fix is usually subtraction. Before adding another wrapper or helper, ask whether the existing abstraction is buying you anything other than distance from the protocol. Pride fixes often start by deleting a smart layer, moving the contract back into plain sight, and making the boring path the default again.

Once the abstraction is thinner, protect the real contract directly. Add integration tests that exercise the schema, the arguments, and the returned structure exactly as a peer sees them. Do the same for prompt templates and resources when they are part of the exposed surface: test the visible shape, not just the helper that assembled it. If you generate contracts, keep compatibility tests for protocol and schema versions, and document the simplest supported pattern in your internal templates so engineers are encouraged to reuse something plain rather than rebuild a framework. A little visible duplication is often far cheaper than hidden magic.

Lessons from the trenches

This is the engineering lesson behind modelcontextprotocol/typescript-sdk #451, where subclassing McpServer broke tool argument passing, and modelcontextprotocol/typescript-sdk #745, where generated JSON Schema drifted from what newer clients expected.

Envy

Envy is adding capabilities because they look impressive, not because they are necessary. It is the sin of mistaking a bigger catalog for a better product. In MCP, every additional tool, prompt, or resource is not just one more feature. It is one more thing the model has to advertise, distinguish, and sometimes choose incorrectly.

How to spot it

Capability lists keep growing, but nobody can explain which ones are actually important.
Multiple tools, prompts, or resources overlap, collide in naming, or differ only in tiny ways.
The model picks the wrong capability because the catalog is too crowded or ambiguous.
Prompt cost rises just from advertising available capabilities.

Example

This is a familiar phase in internal platform rollouts. One genuinely useful server for GitHub becomes a larger catalog with Jira, Slack, docs, incidents, analytics, and deployments, because every new integration demo and every internal team wants to be represented. Then the same pattern spreads to prompts and resources: every team adds its own incident template, support playbook, and reference bundle. By the time people rely on it for daily work, the catalog is crowded, names overlap, and simple tasks become harder because the model has too many similar capabilities to choose from. The tax is paid at selection time, not at launch time, which is why teams underestimate it.

Before

List<RemoteServer> servers = List.of(githubServer, jiraServer, slackServer, docsServer);

for (RemoteServer remoteServer : servers) {
    List<ToolDefinition> tools = remoteServer.listTools();
    for (ToolDefinition tool : tools) {
        registry.register(tool.name(), tool);
    }
}

After

Set<String> enabledTools = Set.of(
    "github.list_open_issues",
    "github.create_issue",
    "docs.search_runbooks",
    "docs.get_runbook"
);

for (RemoteServer remoteServer : List.of(githubServer, docsServer)) {
    List<ToolDefinition> tools = remoteServer.listTools();

    for (ToolDefinition tool : tools) {
        String namespacedName = remoteServer.name() + "." + tool.name();
        if (enabledTools.contains(namespacedName)) {
            registry.register(namespacedName, tool);
        }
    }
}

How to fix it

The fix for envy starts with curation. Treat the capability catalog as product surface area with carrying cost, not as a trophy shelf. Inventory the tools, prompts, and resources and remove the ones that are redundant, rarely used, or too ambiguous to justify their existence. Naming conventions also need to arrive earlier than teams think. Once multiple servers are involved, namespacing and consistent verbs stop being cosmetic and start being survival.

After that, treat discovery as a product design problem instead of a registry problem. Add usage telemetry so retention decisions are based on demand rather than guesswork, and build pagination and filtering into your registry and UI surfaces before the catalog gets large. That review should include prompts and resources, not just tools, because from the host's perspective they are all competing model-facing affordances. If either a model or a human struggles to tell two capabilities apart, the design is not finished yet.

This is another place where platform convenience can mislead teams. A gateway or integration layer can make it trivially easy to publish one more capability from an existing API catalog, and that speed feels like progress. But publication is not curation. If the underlying API surface is already inconsistent, overlapping, or bloated, an MCP catalog in front of it can amplify the ambiguity rather than organize it.

Lessons from the trenches

Duplicate tool names across MCP servers caused errors in openai/openai-agents-python #464. Broken pagination lets very large tool lists come back at once in modelcontextprotocol/java-sdk #615. Capability sprawl is not a cosmetic problem. It directly affects correctness, prompt cost, and the model's ability to make good choices.

Why design sins are hard to fix

Design sins usually look cheap until you try to remove them. Gluttony fixes often force one broad capability to split into search, detail, and pagination flows, which means more contracts and more observability. Pride fixes can be politically harder than technically hard because they often require deleting a "smart" internal framework and accepting a plainer pattern with a little more visible duplication. Envy fixes usually require governance: someone has to own the capability taxonomy, naming rules, enablement, and retirement, or the catalog will keep growing.

That is why design cleanup often lags behind security and operations work. The system may still appear to function. It just gets slower, noisier, and harder to reason about until teams finally notice the carrying cost.

The Seven Deadly Sins of MCP: Operational Sins

Ricardo Ferreira — Mon, 30 Mar 2026 23:13:33 +0000

This part of the series focuses on the operational sins: Sloth and Wrath. They belong in this category because they determine how a live MCP system behaves under stress: whether it fails truthfully, whether it recovers sanely, and whether operators can trust what they are seeing in the middle of an outage.

Sloth and wrath are operational sins because they both appear when systems are stressed. Sloth hides the problem behind vague errors, weak validation, or sloppy transport handling. Wrath takes a problem that might have been survivable and amplifies it through blind retries, reconnect storms, and forceful reactions to uncertainty. Once access boundaries are tighter, the next question is how the system behaves when things go wrong. That is where these two sins take over.

That operational layer is especially visible in MCP because transport and protocol behavior are part of the product. Stdio hygiene, reconnect behavior, resumability, notifications, and session handling are not side details once a model-facing interface is live.

Sloth

Sloth is avoiding precise validation, specific errors, and basic operational hygiene. This sin rarely looks dramatic in code review. It usually looks harmless. A catch block hides detail. A validation rule gets deferred. A debug print goes to the wrong stream. Nobody thinks they are making a dangerous choice. They think they are saving time. But MCP is unforgiving about sloppy boundaries.

How to spot it

Logs are vague, repetitive, or useless when something fails.
Operators keep seeing generic errors like "MCP error" instead of the real failure.
Stdio integrations break mysteriously because non-protocol output leaked into stdout.
The team spends more time reproducing failures than fixing them.

Example

This regularly shows up in support and operations queues. A rep asks the assistant to fetch customer cus_1234 while a customer is waiting in chat, or an engineer asks for the latest incident by ID during triage. In that moment, bad input, not found, and dependency outage are three distinct situations with distinct next steps. If the tool collapses them into one vague failure, the user loses the context they need to respond correctly.

Before

server.tool("get_customer", async ({ id }) => {
  try {
    return await db.customers.findById(id);
  } catch {
    throw new Error("MCP error");
  }
});

After

class ToolError extends Error {
  constructor(
    public code: "invalid_input" | "not_found" | "dependency_unavailable",
    message: string,
    public retryable: boolean
  ) {
    super(message);
  }
}

server.tool("get_customer", async ({ id }) => {
  if (typeof id !== "string" || id.trim() === "") {
    throw new ToolError(
      "invalid_input",
      "id must be a non-empty string",
      false
    );
  }

  try {
    const customer = await db.customers.findById(id);
    if (!customer) {
      throw new ToolError("not_found", `customer ${id} not found`, false);
    }
    return customer;
  } catch (error) {
    console.error("get_customer failed", { id, error });
    if (error instanceof ToolError) {
      throw error;
    }

    throw new ToolError(
      "dependency_unavailable",
      "customer lookup is temporarily unavailable",
      true
    );
  }
});

Also, fix your transport hygiene

// Wrong for stdio servers
console.log("server started");

// Correct for stdio servers
console.error("server started");

Surface the failure honestly at the protocol edge

// `code` and `retryable` are part of this server's error contract,
// not fields that MCP invents automatically for you.
function toMcpErrorResult(error: ToolError) {
  return {
    isError: true,
    code: error.code,
    retryable: error.retryable,
    content: [{ type: "text", text: error.message }],
  };
}

How to fix it

The fix starts at the boundary. Validate inputs where the tool begins, not somewhere deeper in the call stack after the request has already become harder to reason about. When something does fail, preserve the real failure mode whenever you can. Operators need useful errors, not vague theatrical ones, and callers need to know the difference between a not-found result and a broken dependency. In practice, that usually means stable error codes, clear human messages, and a separate place for internal diagnostic detail.

It also helps to treat operational hygiene as part of the contract. Keep protocol traffic separate from diagnostics, especially on stdio, where stdout is data and stderr is logs. On remote HTTP transports, the equivalent discipline is session lifecycle, reconnect behavior, and resumability: if those are inconsistent, the system becomes hard to reason about even when the handlers themselves are correct. Add negative tests for malformed inputs, missing fields, downstream timeouts, and not-found cases, then standardize error shape across the server so every tool does not invent its own private version of confusion. The important part is that typed failures survive translation through the MCP boundary instead of being collapsed into one generic error on the way out. MCP gives you the transport and result channel. The stable fields that make failures actionable still need to be part of your own server contract.

Lessons from the trenches

This pattern shows up in modelcontextprotocol/typescript-sdk #699, where a real tool exception was replaced by a misleading -32602 structured-content error. Once a system starts lying about why it failed, every downstream debugging step gets more expensive. A vague error is not a small quality issue. It is operational debt with interest.

Wrath

Wrath is a reaction to uncertainty or failure, expressed with force rather than control. You can usually hear wrath in design conversations before you see it in code. "If it fails, retry." "If it is slow, poll faster." "If the stream disconnects, reconnect immediately." This is the operational version of losing your temper.

How to spot it

Retry loops or reconnect storms show up in logs during an outage.
A single failing dependency can suddenly cause duplicate requests, duplicate jobs, or repeated server starts.
Clients keep hammering an endpoint that is already degraded.
Timeout graphs and request volume graphs rise together.

Example

This shows up during real outages. An internal app or assistant loses its MCP connection while someone is already in the middle of an incident response, and the user experiences it as a single dropped connection or a spinner that hangs too long. Under the hood, an impatient client can turn that single interruption into repeated process starts, duplicate requests, and more load on the system that is already failing.

Before

async function ensureConnection(client: McpClient, serverCommand: string) {
  while (true) {
    try {
      await client.connect(new StdioTransport(serverCommand));
      return;
    } catch {
      await sleep(100);
    }
  }
}

After

function sleepWithAbort(ms: number, signal: AbortSignal) {
  return new Promise<void>((resolve, reject) => {
    const timeout = setTimeout(() => {
      signal.removeEventListener("abort", onAbort);
      resolve();
    }, ms);

    const onAbort = () => {
      clearTimeout(timeout);
      signal.removeEventListener("abort", onAbort);
      reject(new Error("connection cancelled"));
    };

    if (signal.aborted) {
      onAbort();
      return;
    }

    signal.addEventListener("abort", onAbort, { once: true });
  });
}

async function ensureConnection(
  client: McpClient,
  serverCommand: string,
  abortSignal: AbortSignal
) {
  for (let attempt = 1; attempt <= 5; attempt += 1) {
    try {
      if (abortSignal.aborted) throw new Error("connection cancelled");
      await client.connect(new StdioTransport(serverCommand));
      return;
    } catch (error) {
      if (abortSignal.aborted) throw error;
      if (attempt === 5) throw error;

      const backoffMs = attempt * 1000 + Math.floor(Math.random() * 250);
      await sleepWithAbort(backoffMs, abortSignal);
    }
  }
}

How to fix it

The first part of the fix is learning to stop. Put a hard upper bound on retries, then add progressive backoff with jitter so clients do not all reconnect in lockstep. Thread cancellation through every outbound request and long-running operation, so the system has a way to stand down instead of escalating mindlessly.

After that, decide which calls are actually safe to retry. Idempotent reads and reconnect attempts are one thing; operations with side effects are another. For mutations, the safest default is usually to avoid automatic retries unless you also have an explicit idempotency key or another deduplication guard. This matters at the transport layer too: reconnect logic, session resume behavior, and notification replay rules all need explicit limits, not hopeful defaults. Finally, make retries and reconnects visible. If you are not measuring them, you will not spot a retry storm until production tells you the hard way.

The same warning applies to managed edges and gateways. Throttling, proxy retries, and policy enforcement can be useful, but they do not fix a backend operation that is non-idempotent, vague about failure, or unsafe to repeat. Those features can reduce damage. They cannot manufacture operational truthfulness after the fact.

Lessons from the trenches

This behavior appears in modelcontextprotocol/inspector #293, where connecting caused repeated server starts, and modelcontextprotocol/inspector #723, where reconnect logic did not preserve enough state to resume safely. The lesson is simple: retries are part of system design, not a bandage you slap on at the edge. If your retry policy is not explicit, you do not really have one.

Why operational sins are hard to fix

Operational sins usually demand shared infrastructure rather than isolated patches. Sloth fixes often mean building a validation layer, an error policy, structured logging, and a test harness for unhappy paths. Wrath fixes tend to reach across transport clients, job runners, background workers, and UI status handling. You may need a retry helper, a backoff policy, a cancellation model, idempotency protection, and dashboards that show retries and reconnects.

That work is easy to postpone because it does not demo well. But once it exists, every future tool becomes cheaper to run, debug, and trust.

The Seven Deadly Sins of MCP: Security Sins

Ricardo Ferreira — Mon, 30 Mar 2026 23:07:57 +0000

This part of the series focuses on the security sins: Lust and Greed. They belong in this category because they answer the two questions that decide blast radius before anything else: what the model can reach, and how much authority it carries when it gets there.

If a model can touch something real, these are the first sins that matter. A shell command, a production write, a filesystem path, or a token with more scope than the task deserves can turn a clever demo into a security incident very quickly.

Lust and greed belong together because both are really about access boundaries. Lust is what happens when the model is given unsafe intimacy with sensitive systems or side effects. Greed occurs when it is given more authority than the task deserves. One is about dangerous surfaces. The other is about excessive scope. In practice, they often show up together and are often discovered together.

In MCP, that boundary is visible at the protocol surface itself: what capabilities are being advertised to the client, whether a tool is model-controlled, and what sits between the model and the side effect once the capability is exposed.

Lust

Lust is creating unsafe intimacy between the model and sensitive systems or side effects. This is the most dramatic sin and probably the easiest one for an audience to feel immediately. It is also the one most likely to get hand-waved as innovation. "This is where the magic happens" is what developers say, and usually means the model is about to get direct access to something powerful.

How to spot it

A model can reach shell execution, database writes, or production mutations through a generic interface.
Untrusted content is flowing into tools that have real side effects.
Destructive actions do not require confirmation, review, or narrow intent.
A demo feels exciting precisely because it's a little dangerous.

Example

This is what happens when an assistant grows from read-only visibility into action. An engineer wants repo status during an incident, a quick diff before a deploy, or the latest commits after a rollback. Those are narrow, legitimate requests. But if the server exposes a generic command surface rather than those exact intents, the user who thought they were getting observability has actually been given leverage over a sensitive machine or workflow.

Before

server.tool("run_git", async ({ args }) => {
  return await exec(`git ${args}`);
});

Narrower, but still second-best

const ALLOWED_GIT_COMMANDS: Record<string, string[]> = {
  status: ["status", "--short"],
  diff: ["diff", "--stat"],
  recent_commits: ["log", "--oneline", "-n", "10"],
};

server.tool("run_git", async ({ command }) => {
  const argv = ALLOWED_GIT_COMMANDS[command];
  if (!argv) {
    throw new Error("command not allowed");
  }

  return await execFile("git", argv);
});

Better still: move from commands to intent

server.tool("get_repo_status", async () => {
  return await execFile("git", ["status", "--short"]);
});

server.tool("get_recent_commits", async () => {
  return await execFile("git", ["log", "--oneline", "-n", "10"]);
});

How to fix it

The first improvement is to stop passing raw command text around. If you cannot move to intent-specific tools immediately, at least map a small set of allowed intents to fixed argument arrays rather than accepting free-form command strings. But that is still a transitional state. The real fix is to narrow the relationship between the model and the system. Replace generic execution surfaces with task-specific tools whenever you can, and put confirmation or human approval in front of destructive actions. If untrusted content is flowing into something with side effects, sanitize it and constrain it before it gets anywhere near the dangerous part.

There is also a harder conclusion that teams sometimes avoid: some capabilities should not be exposed through MCP from the host at all. If the only safe version still depends on a highly privileged machine, broad local credentials, or an execution environment that is too dangerous to trust to model-mediated calls, the right answer may be to keep that workflow behind a human-operated CLI or move it into a narrower service boundary first.

High-risk tools also need runtime boundaries, not just good intentions. Run them in restricted environments with tight OS and network permissions, and threat-model prompt injection as a system design problem rather than just an LLM behavior problem. That shift matters because it changes where you place defenses: not only in prompts but also in the tool design, the execution environment, and the approval path.

Lessons from the trenches

This is exactly the kind of boundary failure shown in GHSA-3q26-f695-pp76, the command-injection advisory for @cyanheads/git-mcp-server, and GHSA-q66q-fx2p-7w4m, the filesystem symlink advisory for the official MCP servers repo.

Greed

Greed is granting broader access, authority, or scope than the task deserves. This one is everywhere in MCP systems because least privilege takes work, and admittedly, demos reward convenience. So teams start broad. One scope. One credential set. One tool that can read everything. The intention is temporary. The temporary choice then becomes the architecture.

How to spot it

A read-only workflow still asks for write-capable or user-level credentials.
A tool can reach far more files, tables, APIs, or repos than the task really needs.
Permissions are explained with "we'll narrow it later."
Security review gets uncomfortable long before the demo team does.

Example

This is how internal tools usually drift into overreach. A team starts with a narrow request: let the assistant read one repo's docs, inspect one support folder, or look up one customer's account details. The user and the reviewer both think they approved a bounded capability. The implementation ships with the process's full filesystem reach, or a credential that can see far more than the task requires.

Before

@server.tool("read_file")
async def read_file(path: str):
    return Path(path).read_text()

After

ROOT = Path("/srv/support-docs").resolve()

@server.tool("read_file")
async def read_file(path: str):
    if not path or not path.strip():
        raise ValueError("path is required")

    full_path = (ROOT / path).resolve()
    if ROOT not in full_path.parents and full_path != ROOT:
        raise ValueError("path is outside allowed directory")

    return full_path.read_text()

Better still: split by task, not by path

@server.tool("get_refund_policy")
async def get_refund_policy(plan: Literal["monthly", "annual"]):
    return (ROOT / "refunds" / f"{plan}.md").read_text()

@server.tool("get_password_reset_runbook")
async def get_password_reset_runbook():
    return (ROOT / "runbooks" / "password-reset.md").read_text()

How to fix it

The fix for greed begins with drawing a real boundary. Write down exactly which directories, APIs, databases, or repositories the model should be allowed to access, then shape the tools around that boundary rather than convenience. Read paths and write paths should not share the same credentials or the same code path if the task is only supposed to observe. The strongest version of that pattern is task-shaped access rather than open-ended browsing. A support assistant often does not need "read anything under support-docs." It needs "get the refund policy for this plan" or "fetch the password reset runbook."

From there, review scopes and tokens as you would IAM permissions or network access. In MCP, that review should cover the full capability surface, not just one handler: the tools a client can call, the resources it can browse, and any prompts that might route the model toward privileged actions. Add authorization tests, not just happy-path tool tests, and expect a little product friction along the way. Least privilege usually means more granular credentials, narrower tools, and sometimes a few more workflow steps. Still, that extra design work is what keeps temporary convenience from turning into permanent overreach.

This is also where teams get fooled by gateway comfort. A proxy can enforce authentication, rate limits, and logging around an MCP server, and those controls are valuable. But they do not redeem a backend capability that is already too broad. If the underlying API can read the whole tenant, mutate too much state, or blur read and write authority, wrapping it in a better edge does not change the sin. It only makes the sin easier to expose consistently.

Lessons from the trenches

The mcp-server-git advisory GHSA-5cgr-j3jf-jw3v clearly showed that git_init had arbitrary filesystem access. The mcp-reddit example in the MCP fault-taxonomy paper showed the same impulse in a different form: read-only operations requesting more credentials than necessary.

Why security sins are hard to fix

Security sins rarely stay contained within a single handler. Lust fixes often require app changes, platform controls, security review, and sometimes product changes to add confirmation or approval steps. Greed fixes usually spill into deployment and identity work: new service accounts, narrower filesystem mounts, tighter OAuth scope design, and review gates for high-risk tools.

That is why security cleanup so often feels slower than the original demo. You are not just rewriting a tool. You are redrawing the trust boundary around it.

The Seven Deadly Sins of MCP

Ricardo Ferreira — Mon, 30 Mar 2026 23:02:17 +0000

Model Context Protocol (MCP) is one of the most useful ideas to hit the LLM tooling space in a long time. It gives us a common way to connect models to tools, data, and systems without inventing a fresh integration pattern every week. That is the good news. The bad news is that MCP also makes it much easier to wire LLMs into real systems before a team has earned the operational discipline to do that safely. We can now connect a model to databases, filesystems, an issue tracker, a shell, or a production API with startling ease. That power is real. But so is the blast radius.

MCP's popularity forces us to ask: Does MCP really need to be used for every use case? For instance, a local single-user workflow may still be better served by a direct CLI or API. But once the interface itself needs to be standardized across clients, MCP becomes something broader than a tool wrapper. The protocol standardizes not only tools, but also prompts and resources, and it supports both local stdio servers and remote Streamable HTTP deployments with very different operational shapes. Once MCP becomes that shared surface, ordinary engineering mistakes don't stay ordinary for very long.

That distinction matters even more now that MCP has become part of the strategy for API gateways, integration platforms, and automation vendors. They can add useful features to an existing system: authentication, rate limits, proxying, logging, governance, and catalog management. None of that is bad. But none of it changes the quality of the capability being exposed. A bad API does not become a good model-facing capability just because it now speaks MCP. If the backend contract is over-broad, vague, unstable, or noisy, the gateway may make it easier to distribute, but it does not make it safer to trust.

This is important to realize because neither MCP nor enterprise platforms for APIs create bad engineering instincts. A vague thrown error that used to annoy one developer can now confuse every user of an assistant. A credential that was already too broad becomes more dangerous once it sits behind a tool the model can invoke autonomously. An oversized response that is merely inefficient in a local script becomes a cost and latency issue when it is sent to a model repeatedly. The protocol is not the villain here. The combination of convenience, power, and underdeveloped discipline is.

When I think about the most common implementation problems in MCP systems, I do not think of them first as bug categories. Instead, I think of them as engineering impulses. The bug you see popping up in production is the symptom. The deeper problem is the instinct that made the bug feel reasonable in the first place. We act on them often enough, casually enough, and confidently enough that we stop noticing the harm. They start to feel normal. Mundane, even.

"We see a deadly sin on every street corner, in every home, and we tolerate it. We tolerate it because it's common, it's trivial."

This is what John Doe, the main antagonist of the classic movie Seven, says near the end of the film. I think the same thing happens in software engineering. We return too much data. We ask for too many permissions. We hide bad errors behind vague ones. We add one more tool, one more abstraction, one more shortcut. None of it feels dramatic in the moment, which is exactly why these habits survive.

That is why I like to think of these engineering impulses as deadly sins. Some of them even straddle more than one sin. A symlink traversal bug is both excessive access and unsafe system intimacy. A retry storm is both operational instability and resource abuse. A bloated tool payload is both gluttony and a symptom of prideful design. The categories are not airtight boundaries, but they are useful. They help us distinguish between sins that expand the blast radius, sins that cause live systems to fail badly, and sins that quietly make the whole product harder to trust.

In this series, I will discuss examples from three sources: a recent MCP fault taxonomy paper that analyzed hundreds of issues across public repositories, public GitHub issues and security advisories, and mistakes of my own. I have shipped versions of several of these sins too: over-broad scope, vague errors, and overly helpful responses that returned far more than the task needed. Taken together, they paint a practical picture of how MCP systems actually go wrong. This is not a theoretical taxonomy built in a vacuum. It is a set of recurring failure modes that keep showing up where real teams are trying to ship real systems.

The seven deadly sins are the framework I will use to make those impulses easier to see before they harden into architecture.

Sin	Meaning	Example
Lust	Too much intimacy.	A tool for checking container status accepts arbitrary shell input, turning a simple operational check into command execution.
Greed	Too much power.	A read-only code review assistant is given a GitHub token that can also push commits, merge PRs, and manage secrets.
Sloth	Too little care.	An expense lookup fails with `request failed`, leaving finance unable to tell whether the report ID was invalid or the service timed out.
Wrath	Too much force.	A timeout in a background sync causes the client to resubmit the same job every second until the queue backs up.
Gluttony	Too much waste.	A calendar tool returns the full event body, attendee list, history, and notes when the user only asks for the next meeting.
Pride	Too much ego.	A homegrown tool framework auto-coerces inputs behind the scenes, and a deploy tool starts failing in ways nobody can explain quickly.
Envy	Too much imitation.	A team adds three overlapping issue-creation tools because other servers have big catalogs, and the model keeps picking the wrong one.

These seven sins are not equally dangerous at all times. Some create an immediate blast radius. Others quietly compound costs, confusion, or maintenance burdens until the system becomes too expensive to trust. That is why this series is ordered the way it is.

How this series is organized

This series is written for people designing, reviewing, or operating MCP systems in production, so I organized the deadly sins into categories because that is usually how remediation work gets sequenced in practice. Security Sins covers Lust and Greed, the sins that decide blast radius by defining what the model can reach and what authority it carries there. Operational Sins covers Sloth and Wrath, the sins that decide whether the system fails truthfully and under control once it is live. Design Sins covers Gluttony, Pride, and Envy, the slower sins that compound cost, ambiguity, and maintenance burden even when they do not explode first.

Throughout the series, I am talking about MCP as a protocol surface, not just a codebase full of handlers. Some sins show up in tools. Some show up in prompts and resources. Some show up in transport choices, and some in the authorization and policy boundary around the whole system.

The final part, Road to Redemption, shifts from recognition to practice. That is where the series turns into refactoring order, review questions, metrics, and the habits that keep these sins from settling back into the architecture. So this is not just a list of problems. It is a way to decide what to fix first, what to review with more skepticism, and what to treat as a design smell before it becomes a production problem.

Building Reliable Agents with the Transactional Outbox Pattern and Redis Streams

Ricardo Ferreira — Fri, 27 Mar 2026 15:06:01 +0000

AI agents are pretty good at deciding what should happen next, given a well-defined business workflow. In the case of a customer support agent, for example, they can read a conversation, apply a policy, and return a response like "approve the refund" or "escalate this case." That part is exciting, and it is usually what gets demoed first. But the hard part starts right after the decision is made.

In a real system, a decision only matters if the rest of the platform can trust it. If an agent decides a customer should get a refund, that decision still has to turn into real work across the rest of the application. The support case needs to be updated, and billing needs to issue the refund. The customer may need an email, and the CRM probably needs the updated status, too.

If the app updates the case and then crashes before billing gets the event, you now have a case that says "refund approved" and a customer who never actually got refunded. That is the kind of bug that makes a system feel flaky even when the model made the right call. But the worst part is the damage to customer experience. I would be really mad at the company if this happened to me.

For scenarios like this, the Transactional Outbox pattern exists. Instead of treating "update the case" and "tell the rest of the system" as two separate operations, we commit them together and let the rest of the platform react asynchronously afterward. This pattern became fairly famous in the context of microservices, as they often need a reliable way to hand off tasks. I think the pattern is also useful for agents, because the fundamental problem is the same.

In this post, I will discuss the Transactional Outbox pattern in the context of agents and provide an opinionated view of why I believe it is a best practice for agentic applications. I will discuss the pattern around the following question: once an agent makes a business decision, how can you ensure the rest of the system can rely on it?

The problem is the handoff

Developers often stress about designing systems that can survive unimaginable production incidents. But the reality is, you don't need a major outage or some exotic distributed systems incident to witness the worst. Sometimes it is as simple as the need for one service to try to do two related things in two separate steps. Like, first, it updates the business state, then it publishes an event for downstream systems.

That looks harmless until something fails in between. If the state update succeeds and the event publish does not, the source of truth has moved forward, but the rest of the workflow has not.

Here is that failure in one picture:

What makes this annoying is that nothing looks obviously broken at first. During an incident investigation, if someone checks the case record, it looks correct. The problem only shows up later when billing never acts, the customer complains, or support has to manually reconcile what happened. That is why I think this is not an AI problem. It is a handoff problem.

Motivation for the Transactional Outbox pattern

The Transactional Outbox pattern exists because "save state, then publish the event" is fragile by design. The pattern gives you a cleaner contract: when business state changes, the application also writes an outbox event in the same atomic operation.

That one change removes the worst failure mode. You no longer end up in the state where the case changed, but the event silently disappeared. It also keeps the request path honest. The service does not have to directly coordinate billing, notifications, CRM sync, and everything else just to be correct.

Instead, the request path only needs to guarantee one thing: the decision and the outbox event are committed together. Once that happens, everything else becomes recoverable instead of fragile.

That is why this pattern fits agentic systems so well. Agents make decisions that trigger follow-up work, but those decisions need a durable "this happened" moment before the rest of the system can safely react.

Why is "Just Retry the Publish" not enough?

Every time I promote a discussion with developers around the Transactional Outbox pattern, I often hear them saying, "Why not just retry if the publish fails?" I think that happens because implementing the pattern correctly requires certain design decisions and technology stacks. The instinct is usually to look for a simpler alternative.

For this reason, I like to stress the following: the use of retries is reasonable until you look closely at where the failure occurs. This means that retries only help if the application still knows it has something to retry. If the process crashes after the state update but before the event is durably recorded anywhere, there is nothing left to retry.

That is the key difference between retries and an outbox. Retries help you deliver an event that already exists, while the outbox ensures the event exists in the first place. Once you look at it that way, the pattern feels less like a ceremony and more like basic design principles. If the business state changes, the system needs a durable record of the event that describes that change.

Redis Streams is great for this pattern

Redis Streams are a good fit for this kind of outbox because they already behave like the commit log we want. You can append events to them, consume them in order, track what is pending, and let different consumer groups process the same stream independently. That matters because the outbox is not really a queue in the narrow sense. It is a commit log for business events.

Admittedly, the Transactional Outbox pattern is often implemented using Apache Kafka and technologies such as Debezium. That is where the pattern became most notorious. I helped many developers implement this pattern with Kafka, and it works great for getting things done. However, because I have tons of experience with Kafka, I can say that the implementation effort can sometimes exceed the main problem they were trying to solve due to Kafka's inherent complexity. You spend more time dealing with Kafka than the actual problem.

Redis Streams, on the other hand, makes that pretty natural. A single event can be appended once and then processed independently by several downstream concerns. The other reason Streams fit well is that they sit comfortably inside Redis. If your support case state also lives in Redis, the state change and the outbox append can share one commit boundary.

That part is important. The pattern is strongest when the business state and the outbox live in the same datastore, because that gives you a single atomic write instead of a dual-write problem wearing different clothes. With Kafka, you would need to handle two different distributed systems: the commit log itself and the data store where the update must occur.

Diving deep into the architecture

For this example, the support case state and the outbox both live in Redis. The current case state is stored in a hash, and the outbox is stored in a Redis Stream.

A case key might look like support:{tenant-acme}:case:case-123, while the outbox stream might be support:{tenant-acme}:outbox. The use of hash tags here is important because you must be intentional about where the data will be stored in Redis. During development, you may work with a single server which is the equivalent of a single shard. The data will naturally live in the same place. However, in production, you may have a clustered Redis environment with multiple shards.

The shared hash tag keeps both keys in the same slot in clustered Redis, which is what lets them participate in the same transaction. That gives us a clean split of responsibilities. The case record tells us what is true now, and the outbox stream tells us what happened and what the rest of the platform still needs to process. Yes, a relatively simple use of key prefixes could make this entire implementation useless if not carefully thought out.

From there, downstream concerns consume the stream through their own consumer groups. Billing can issue the refund, notifications can contact the customer, and CRM sync can update external systems, all without forcing the support service to orchestrate them directly in the request path.

That flow looks like this:

The thing I like most about the Transactional Outbox pattern is that it keeps responsibilities clear. The support service is responsible for making the decision durable, and the rest of the platform is responsible for responding to it.

Trade-offs that are interesting to consider

The basic implementation of the pattern is simple. The design choices around it are where things get interesting. One of the first questions you must ask is where your source of truth lives. If the support case is also in Redis, the case update and the outbox append can share one transaction. If the case lives somewhere else and Redis only holds the stream, you are back in dual-write territory.

Another big choice is partitioning. It is tempting to imagine a single global outbox stream for the whole application, but that often becomes awkward in a clustered Redis setup. A per-tenant stream is often a better balance. It keeps related events together, provides useful ordering, and avoids making every transactional write depend on a single global key. It also makes querying and data retrieval a bit easier during investigation scenarios.

Consumer isolation is another trade-off that is worth saying out loud. One consumer group per downstream concern is a very nice model operationally, because billing, notifications, and CRM sync can all move at their own pace. The flip side is that you now own several background workflows. Each one has lag, retries, health, and recovery behavior to think about. This is where the world of microservices cross paths again with agentic systems. Each agent is not only a set of code and resources. They also bring operational complexities that must be owned by someone.

Retention matters too. An outbox is a log, and logs grow. If you trim too aggressively, you lose the replay window and the investigation history. If you never trim at all, the stream just keeps growing and eventually becomes an operational problem in its own right. Deciding how large the stream is allowed to grow must be a discussion that takes place before the app even goes to production. Not an afterthought.

Durability is another place where the architecture gets real fast. If the outbox carries important business decisions like refunds, escalations, or account changes, Redis is no longer "just a cache" in this design. It is part of the system's correctness model. You must treat Redis as a single source of truth, and as such, think carefully about how to handle details like replication, failover, and geographic disasters.

Finally, there is idempotency. The outbox makes the handoff reliable, but it does not magically make downstream effects exactly-once behavior in the business sense. If a worker crashes after reading but before acknowledging, another worker may retry the same event later. That means the side effect needs to be safe to run more than once. The usual instinct for developers is to write the worker as a function that hooks into the stream, pulls the latest records, and processes them as if the data is simply mutable. Nope, you must treat them as immutable objects.

Okay, let's see some code

This post is not meant to be a complete implementation reference, but I know that, as a developer, looking at code helps make understanding concrete. I will try to provide the example with fewer details so you can understand the design principles. I'm sure your coding agent can help with your actual final code. Also, I will use Java because it comes naturally to me — but feel free to ask your coding agent to translate it into another language.

Let's start by looking for a runtime helper class that instantiates Jedis, a Redis client for Java:

import redis.clients.jedis.RedisClient;
import redis.clients.jedis.UnifiedJedis;

public final class RuntimeSupport {

    public UnifiedJedis createJedisFromEnv() {
        String redisHost = System.getenv().getOrDefault("REDIS_HOST", "localhost");
        int redisPort = Integer.parseInt(System.getenv().getOrDefault("REDIS_PORT", "6379"));

        return RedisClient.builder()
                .hostAndPort(redisHost, redisPort)
                .build();
    }
}

Next, let's take a look at how keys and group naming are handled in a small constants class instead of scattering strings through the code.

public final class SupportConstants {
    public static final String STREAM_GROUP_START_ID = "0-0";
    public static final String BILLING_GROUP_NAME = "billing-cg";
    public static final String NOTIFICATIONS_GROUP_NAME = "notifications-cg";
    public static final String CRM_SYNC_GROUP_NAME = "crm-sync-cg";

    private SupportConstants() {}
}

For the Redis keys themselves, a small helper record keeps the slotting decision obvious:

public record SupportKeys(String caseKey, String outboxKey) {

    public static SupportKeys forCase(String tenantId, String caseId) {
        String hashTag = "{" + tenantId + "}";
        return new SupportKeys(
                "support:" + hashTag + ":case:" + caseId,
                "support:" + hashTag + ":outbox"
        );
    }
}

The core write path is where the architectural idea becomes real. When the support service accepts the agent's decision, it updates the case state and appends a RefundApproved event in a single Redis transaction.

import redis.clients.jedis.AbstractTransaction;
import redis.clients.jedis.Response;
import redis.clients.jedis.StreamEntryID;
import redis.clients.jedis.UnifiedJedis;

import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.UUID;

public final class RefundApprovalService {
    private final UnifiedJedis jedis;

    public RefundApprovalService(UnifiedJedis jedis) {
        this.jedis = Objects.requireNonNull(jedis, "jedis must not be null");
    }

    public RefundCommitted approveRefund(RefundDecision decision) {
        SupportKeys keys = SupportKeys.forCase(decision.tenantId(), decision.caseId());

        Map<String, String> caseFields = new LinkedHashMap<>();
        caseFields.put("case_id", decision.caseId());
        caseFields.put("customer_id", decision.customerId());
        caseFields.put("refund_id", decision.refundId());
        caseFields.put("status", "refund_approved");
        caseFields.put("decision_source", "support-agent");
        caseFields.put("updated_at", decision.decidedAt().toString());

        Map<String, String> outboxFields = new LinkedHashMap<>();
        outboxFields.put("event_id", decision.eventId());
        outboxFields.put("event_type", "RefundApproved");
        outboxFields.put("case_id", decision.caseId());
        outboxFields.put("customer_id", decision.customerId());
        outboxFields.put("refund_id", decision.refundId());
        outboxFields.put("decision_source", "support-agent");
        outboxFields.put("occurred_at", decision.decidedAt().toString());

        try (AbstractTransaction redisTx = jedis.multi()) {
            redisTx.hset(keys.caseKey(), caseFields);
            Response<StreamEntryID> streamEntryId =
                    redisTx.xadd(keys.outboxKey(), StreamEntryID.NEW_ENTRY, outboxFields);

            List<Object> execResults = redisTx.exec();
            if (execResults == null) {
                throw new IllegalStateException("Refund approval transaction aborted");
            }

            return new RefundCommitted(
                    decision.caseId(),
                    decision.eventId(),
                    streamEntryId.get().toString()
            );
        }
    }

    public record RefundDecision(
            String tenantId,
            String caseId,
            String customerId,
            String refundId,
            String eventId,
            Instant decidedAt
    ) {
        public static RefundDecision create(
                String tenantId,
                String caseId,
                String customerId,
                String refundId
        ) {
            return new RefundDecision(
                    tenantId,
                    caseId,
                    customerId,
                    refundId,
                    UUID.randomUUID().toString(),
                    Instant.now()
            );
        }
    }

    public record RefundCommitted(
            String caseId,
            String eventId,
            String streamEntryId
    ) {}
}

This one method is the whole architectural point made concrete. If the transaction does not complete, neither the case update nor the outbox event exists. If it does complete, both exist. That is the durability boundary that the rest of the workflow can rely on.

Here is the same moment as a diagram:

On the consumer side, the worker will act on the message written to the stream.

import redis.clients.jedis.StreamEntryID;
import redis.clients.jedis.UnifiedJedis;
import redis.clients.jedis.exceptions.JedisDataException;
import redis.clients.jedis.params.XReadGroupParams;
import redis.clients.jedis.resps.StreamEntry;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Objects;

import static redis.clients.jedis.StreamEntryID.XREADGROUP_UNDELIVERED_ENTRY;

public final class BillingConsumer {
    private static final StreamEntryID PENDING_ID =
            new StreamEntryID(SupportConstants.STREAM_GROUP_START_ID);
    private static final StreamEntryID NEW_ENTRY_ID = XREADGROUP_UNDELIVERED_ENTRY;

    private final UnifiedJedis jedis;
    private final BillingGateway billingGateway;
    private final String consumerName;

    public BillingConsumer(
            UnifiedJedis jedis,
            BillingGateway billingGateway,
            String consumerName
    ) {
        this.jedis = Objects.requireNonNull(jedis, "jedis must not be null");
        this.billingGateway = Objects.requireNonNull(billingGateway, "billingGateway must not be null");
        this.consumerName = Objects.requireNonNull(consumerName, "consumerName must not be null");
    }

    public void run(String tenantId) throws InterruptedException {
        String outboxKey = SupportKeys.forCase(tenantId, "unused").outboxKey();
        createConsumerGroup(outboxKey);

        while (!Thread.currentThread().isInterrupted()) {
            List<StreamMessage> pendingEntries = readGroup(outboxKey, PENDING_ID, 10);
            if (!pendingEntries.isEmpty()) {
                processEntries(outboxKey, pendingEntries);
                continue;
            }

            List<StreamMessage> newEntries = readGroup(outboxKey, NEW_ENTRY_ID, 10);
            if (!newEntries.isEmpty()) {
                processEntries(outboxKey, newEntries);
            } else {
                Thread.sleep(200L);
            }
        }
    }

    private void createConsumerGroup(String outboxKey) {
        try {
            jedis.xgroupCreate(
                    outboxKey,
                    SupportConstants.BILLING_GROUP_NAME,
                    new StreamEntryID(SupportConstants.STREAM_GROUP_START_ID),
                    true
            );
        } catch (JedisDataException e) {
            if (!e.getMessage().contains("BUSYGROUP")) {
                throw e;
            }
        }
    }

    private List<StreamMessage> readGroup(String outboxKey, StreamEntryID streamEntryID, int count) {
        XReadGroupParams params = XReadGroupParams.xReadGroupParams().count(count);

        List<Map.Entry<String, List<StreamEntry>>> rawEntries = jedis.xreadGroup(
                SupportConstants.BILLING_GROUP_NAME,
                consumerName,
                params,
                Map.of(outboxKey, streamEntryID)
        );

        return parseEntries(rawEntries);
    }

    private void processEntries(String outboxKey, List<StreamMessage> entries) {
        for (StreamMessage entry : entries) {
            if (!"RefundApproved".equals(entry.fields().get("event_type"))) {
                jedis.xack(
                        outboxKey,
                        SupportConstants.BILLING_GROUP_NAME,
                        new StreamEntryID(entry.id())
                );
                continue;
            }

            billingGateway.issueRefund(
                    entry.fields().get("refund_id"),
                    entry.fields().get("customer_id"),
                    entry.fields().get("event_id")
            );

            jedis.xack(
                    outboxKey,
                    SupportConstants.BILLING_GROUP_NAME,
                    new StreamEntryID(entry.id())
            );
        }
    }

    private static List<StreamMessage> parseEntries(List<Map.Entry<String, List<StreamEntry>>> rawEntries) {
        if (rawEntries == null || rawEntries.isEmpty()) {
            return Collections.emptyList();
        }

        List<StreamMessage> entries = new ArrayList<>();
        for (Map.Entry<String, List<StreamEntry>> streamData : rawEntries) {
            for (StreamEntry streamEntry : streamData.getValue()) {
                entries.add(new StreamMessage(
                        streamEntry.getID().toString(),
                        streamEntry.getFields()
                ));
            }
        }

        return entries;
    }

    private record StreamMessage(String id, Map<String, String> fields) {}

    public interface BillingGateway {
        void issueRefund(String refundId, String customerId, String idempotencyKey);
    }
}

Closing

What I like most about the Transactional Outbox pattern is that it respects the actual shape of agentic systems. Agents are good at deciding what should happen next given a flow, but the platform is still responsible for turning that decision into a durable state and letting the rest of the workflow react safely. The pattern gives you a clean handoff for that.

Redis Streams make it practical when your application state and the outbox both live in Redis. That doesn't make the design free of trade-offs. You still need to think about partitioning, retention, durability, lag, and idempotency. It just prevents you from thinking about dual-write problems. It gives you a system where an agent's decision becomes a durable fact before the rest of the platform starts depending on it.

By applying the Transactional Outbox pattern in your agents, you can be the difference between an agent that looks clever in a demo and a system you can actually trust.

Behavioral Engineering for AI in Java: Enforcing Policy from Dev to Prod

Ricardo Ferreira — Tue, 03 Mar 2026 16:53:53 +0000

I'm going to make a bold claim here: if you're a Java developer building AI-powered features, chances are you've already experienced the tension of diverging business rules between dev and prod environments. Not sure what I mean? Let me explain with an example.

Say your coding assistant helps you implement guardrails. It suggests verifying identity before taking destructive actions, nudging you toward escalation logic, the whole deal. Everything looks spotless during development.

Then, your code goes live.

However, somehow the runtime behavior doesn't quite match what you thought you enforced. One service escalates large refunds, while others promise them instantly. One service verifies identity. Another doesn't. Tone varies depending on sentiment handling. Safety rules drift. It's pure chaos.

The real issue here is this:

How do you guarantee that behavioral rules apply both when code is being built and when the AI is running in production?

If those two phases drift apart, it means you don't have governance; you have improvisation. For teams accustomed to versioned artifacts, shared libraries, and architectural standards, allowing this to happen feels wrong.

In this blog post, I will explain how to fix this. I will show you how to enforce policies from development time to runtime using Agent Skills to define your business rules in a shareable and standardized manner.

The Case Study: A Policy-Aware Support Agent

To better understand how to enforce policies from development to production, let's make this as concrete as possible with a case study. Let's say you're building a support assistant with Spring Boot and Spring AI. This implementation needs to handle:

Refund requests
Billing disputes
Subscription cancellations
Account deletions

Your business rules are these:

Never admit legal liability
Never promise refunds without verification
If refund > $500, escalate to a human
If sentiment is angry, respond empathetically
Never expose internal system details
Require identity verification before destructive tool calls
Always summarize next steps clearly

This is not optional behavior. This is business policy. Failing to address these policies directly impacts revenue, customer retention, and the company's morale. This is the kind of thing that should not live in scattered prompts.

Before we jump into the implementation, it's worth asking: why not just put these rules in a system-prompt.md file in your repository? You could version-control that too, right? Well, the difference here is reusability and lifecycle coverage. A system prompt file resides within a single service. A skill is more of an independent, versioned artifact that can be installed into coding agents during development and loaded as a runtime dependency.

It's the same contract applied in two places through two different mechanisms. That distinction is what makes behavioral governance possible across teams and services.

Step 1: Define the Behavior as a Skill

Skills are reusable, filesystem-based resources that provide agents with domain-specific expertise, such as workflows, business rules, and best practices. They are made available as GitHub repositories.

Create a repository:

customer-communication-policy

Inside it:

customer-communication-policy/
└── skills/
    └── customer-communication-policy/
        └── SKILL.md

Now define the behavioral contract in SKILL.md:

---
name: customer-communication-policy
description: "Enforces customer support communication and escalation rules."
---

## Tone Rules
- If sentiment is angry, respond empathetically.
- Never admit legal liability.
- Never expose internal system details.

## Refund Rules
- Do not promise refunds without verification.
- If the refund amount > 500 USD, escalate to a human agent.

## Tool Invocation Rules
- Before calling destructive tools (refund, cancel, delete), require identity verification.
- If identity is not verified, request verification.

## Output Rules
- Always summarize next steps clearly.

Commit and push.

This is it. You just created your versioned behavioral contract.

Step 2: Use the Skill with your Coding Agents

Now let's enforce those rules while writing code. To do that, you must install the skill with your coding agent. In this example, we will install our skill using OpenAI Codex, but the instructions are similar to those for other coding agents.

The simpler way to install a skill is to use the Skills CLI. The Skills CLI is useful for global installs, making the skill available to different coding agents and allowing it to be published automatically in ranked mode.

First, install the Skills CLI:

npm install -g skills

To install your Skill from GitHub:

npx skills add your-org/customer-communication-policy

The wizard will guide you on how to install with specific coding agents, or you can install it globally:

npx skills add your-org/customer-communication-policy -g

Verify installation:

npx skills list -a codex

Now open Codex and use the /skills command:

λ riferrei  ~ codex
╭──────────────────────────────────────────────────╮
│ >_ OpenAI Codex (v0.106.0)                       │
│                                                  │
│ model:     gpt-5.3-codex high   /model to change │
│ directory: ~                                     │
╰──────────────────────────────────────────────────╯

  Tip: New Build faster with the Codex App. Run 'codex app' or visit https://chatgpt.com/codex?app-landing-page=true


› /

  /model         choose what model and reasoning effort to use
  /permissions   choose what Codex is allowed to do
  /experimental  toggle experimental features
  /skills        use skills to improve how Codex performs specific tasks
  /review        review my current changes and find issues
  /rename        rename the current thread
  /new           start a new chat during a conversation
  /resume        resume a saved chat

Here you can list and enable/disable installed skills.

Skills
  Choose an action

› 1. List skills            Tip: press $ to open this list directly.
  2. Enable/Disable Skills  Enable or disable skills.

If you select List skills, you should see your customer-communication-policy installed. At this point, Codex is aware of your behavioral policy and is ready to help you enforce it during code development.

What Changes During Development?

Let's be precise about what "enforcement" means here. Installing a skill into Codex does not guarantee that every line of generated code will be policy-compliant. It's guidance, not a compiler. The skill influences code generation by making the coding agent aware of your rules. The runtime layer, which we'll build next, is where true enforcement happens. The value of the development-time skill is alignment: it steers the developer and the coding agent toward the right structure from the start.

To show what that looks like in practice, here are some examples of interactions. Without the skill installed, I gave Codex this prompt:

"Create a Spring Boot service that handles refund requests using an LLM. Use Spring AI to implement this service."

Codex generated a straightforward controller and service. The service called the LLM directly with the user's message and returned the response. No identity verification. No escalation logic. No sentiment handling. It produced functional code, sure, but completely policy-unaware code:

@Service
public class RefundService {

    private final ChatClient chatClient;

    public RefundService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public String handleRefund(String userMessage) {
        return chatClient.prompt()
                .user(userMessage)
                .call()
                .content();
    }
}

Now here's the same prompt with the customer-communication-policy skill enabled:

"Create a Spring Boot service that handles refund requests using an LLM. Use Spring AI to implement this service."

This time, Codex generated code that reflected the behavioral contract. It added an identity verification check before processing destructive actions. It included escalation logic for refunds over $500. It introduced sentiment detection to guide tone. The structure directly mirrored the rules in SKILL.md:

@Service
public class RefundService {

    private final ChatClient chatClient;
    private final EscalationService escalationService;

    public RefundService(ChatClient.Builder builder,
                         EscalationService escalationService) {
        this.chatClient = builder.build();
        this.escalationService = escalationService;
    }

    public String handleRefund(SupportRequest request) {

        if (!request.isIdentityVerified() && request.isDestructiveAction()) {
            return "Before I proceed, I need to verify your identity. "
                 + "Could you please confirm your account email and "
                 + "the last four digits of your payment method?";
        }

        if (request.isRefund() && request.getAmount() > 500) {
            return escalationService.escalate(request);
        }

        String sentimentHint = request.getSentiment() == Sentiment.ANGRY
                ? "The customer is frustrated. Respond empathetically."
                : "";

        return chatClient.prompt()
                .system("""
                    You are a support agent. Follow these rules strictly:
                    - Never admit legal liability.
                    - Never expose internal system details.
                    - Always summarize next steps clearly.
                    """ + sentimentHint)
                .user(request.getMessage())
                .call()
                .content();
    }
}

The difference is significant. With the skill active, the coding agent produced code that already embodies your policy. The developer sees the verification gate, the escalation threshold, and the sentiment-aware prompting, and they know exactly which pieces of logic to refine, test, and maintain. It is as if the code were written alongside the business analyst who wrote those rules in the first place.

But this is still development-time influence, not runtime enforcement. Your LLM won't be able to leverage these rules because the skill isn't available to it. Let's change that.

Step 3: Register the Skill as a Dependency

To use the skill at runtime, your skill must be part of your code. With the Spring AI ecosystem, you can easily do this by using the Spring AI Agent Utils project, which loads your skills from the filesystem or the classpath. You can learn how this project works with this blog post.

However, as observed in the blog post, there are some limitations to using your skills as dependencies. The most notable one concerns skill versioning. Since they are static resources and they are likely to evolve, they need to be versioned along with your code. The only way to do this in an engineered way is to treat them as a JAR dependency.

This is where you may want to know about the SkillsJars project. It loads your skills from GitHub and publishes them as JARs to Maven Central. This way, it can be version-controlled as a runtime dependency.

To do this, go to:

https://www.skillsjars.com

and then register the GitHub repository that contains your skill using the "Publish a SkillsJar" form.

SkillsJars scans the repository, finds the SKILL.md file, and automatically packages it as a Maven artifact. Inside the resulting JAR, the skill is placed at a well-known path:

META-INF/skills/<org>/<repo>/<skill>/SKILL.md

You don't need to build a JAR yourself. You don't write Java code. You register the repository. Keep in mind, though, it usually takes some time for the dependency to be added to Maven Central.

Once processed, SkillsJars generates Maven coordinates like:

<dependency>
    <groupId>com.skillsjars</groupId>
    <artifactId>customer-communication-policy__customer-communication-policy</artifactId>
    <version>2026_02_17-abc123</version>
</dependency>

The version is derived from the commit metadata, so every change to your skill is going to be versioned. This is where it starts feeling very natural for Java developers.

A Note on Versioning and Dependency Management

Because the skill becomes a standard Maven dependency, it inherits all the implications of dependency management. Therefore, there are a few things to consider:

Updating a skill: When you push a change to your SKILL.md and SkillsJars publishes a new version, consuming services need to bump the version in their pom.xml. This is intentional. You want version bumps to be explicit so teams can review behavioral changes before deploying.
Coordinated rollouts: If multiple services depend on the same skill, you decide when each service adopts the new version. This is no different from how you manage a shared library today. You can roll forward incrementally or pin a version until you're ready.
Rollback: If a new skill version causes undesired behavior in production, you roll back the dependency version in your pom.xml and redeploy. The previous behavioral contract is restored exactly as it was.
Compatibility: Adding a new rule to a skill doesn't break existing agent behavior in the way a breaking API change would. The LLM receives the full set of rules and applies them. However, a new rule could change how the model responds to certain inputs. Treat skill updates like configuration changes: test before promoting to production.

The key point here is that you're managing behavioral policy with the same tools and discipline you already use for code dependencies. No new workflow is required.

Step 4: Use the Skill with Spring AI

The Spring AI Agent Utils project provides a SkillsTool component that integrates Agent Skills directly with Spring AI. When skills are packaged as jars, they work out of the box: the tool reads them directly from the classpath with no extraction step needed.

Add the dependencies to your pom.xml:

<dependency>
    <groupId>org.springaicommunity</groupId>
    <artifactId>spring-ai-agent-utils</artifactId>
    <version>0.5.0</version>
</dependency>

<dependency>
    <groupId>com.skillsjars</groupId>
    <artifactId>customer-communication-policy__customer-communication-policy</artifactId>
    <version>2026_02_17-abc123</version>
</dependency>

Configure the skills path in application.properties:

agent.skills.paths=classpath:/META-INF/skills

Now let's start developing our use case. Start off by defining your tools:

@Component
public class RefundTool {

    @Tool(description = "Issue a refund to a customer after verification")
    public String issueRefund(String customerId, double amount) {
        return "Refund of $" + amount + " initiated for customer " + customerId;
    }
}

Then wire your ChatClient with both the SkillsTool and your business tools:

@Configuration
public class AgentConfig {

    @Value("${agent.skills.paths}")
    private List<Resource> skillPaths;

    @Bean
    public ChatClient supportAgent(
            ChatClient.Builder builder,
            RefundTool refundTool) {

        return builder
                .defaultToolCallbacks(
                        SkillsTool.builder()
                                .addSkillsResources(skillPaths)
                                .build(),
                        ToolCallbacks.from(refundTool)
                )
                .build();
    }
}

How Does SkillsTool Work?

SkillsTool is registered as a tool callback on the ChatClient. At startup, it scans the configured classpath locations, finds the SKILL.md files inside the SkillsJar, and makes their content available to the LLM as a callable tool. When the agent processes a request, the LLM can invoke the SkillsTool to retrieve the behavioral policy, then use those rules to govern its response and which tools it calls.

This means the LLM has access to the exact same SKILL.md content that guided your coding agent in Step 2, but now at runtime, as part of its tool-calling context.

Let's be direct about what this means for enforcement. The skill content is made available to the LLM as a tool that it can read. The LLM is expected to follow those rules when deciding how to respond, but, as with any LLM, it could deviate. This is probabilistic enforcement, not deterministic.

For deterministic guarantees on hard constraints like the $500 escalation threshold or the identity verification gate, you should implement those as programmatic checks in your Java code, exactly as the coding agent suggested in Step 2. The skill-as-tool approach handles the softer behavioral rules: tone, liability language, output formatting, and how the model frames its responses.

The combination of both layers, programmatic checks for hard rules and skill-informed LLM behavior for soft rules, is what gives you real coverage.

Here's the service that ties it all together:

@Service
public class SupportAgentService {

    private final ChatClient supportAgent;
    private final EscalationService escalationService;

    public SupportAgentService(ChatClient supportAgent,
                               EscalationService escalationService) {
        this.supportAgent = supportAgent;
        this.escalationService = escalationService;
    }

    public String handle(SupportRequest request) {

        // Hard enforcement: programmatic checks
        if (!request.isIdentityVerified() && request.isDestructiveAction()) {
            return "Before I proceed, I need to verify your identity. "
                 + "Could you please confirm your account email and "
                 + "the last four digits of your payment method?";
        }

        if (request.isRefund() && request.getAmount() > 500) {
            return escalationService.escalate(request);
        }

        // Soft enforcement: SkillsTool makes the policy available
        // to the LLM as part of its tool-calling context
        return supportAgent.prompt()
                .user(request.getMessage())
                .call()
                .content();
    }
}

What's happening here?

Hard rules (identity verification, escalation threshold) are enforced programmatically in Java. The LLM never gets a chance to violate them.
Soft rules (tone, liability language, output structure) are enforced via the SkillsTool, which makes the policy available to the LLM at runtime. The LLM reads the skill and follows those rules when composing its response.

What Happens in Production?

Let's trace through a real request to see both enforcement layers in action.

Input:

"You charged me $1200 twice. Refund me now."

Programmatic check → Identity verification.

The request arrives at SupportAgentService.handle(). The request is flagged as a destructive action (refund). request.isIdentityVerified() returns false. The method returns immediately:

"Before I proceed, I need to verify your identity. Could you please confirm your account email and the last four digits of your payment method?"

The LLM was never called. This is hard enforcement.

User verifies identity and resubmits.

Now isIdentityVerified() returns true. The next check fires: request.getAmount() is $1200, which exceeds the $500 threshold. escalationService.escalate(request) is called. The method returns:

"I'm escalating this to our billing team for urgent review. A specialist will contact you shortly."

Again, the LLM was never called. This is hard enforcement.

A different request reaches the LLM.

Now consider a request that passes both programmatic checks. The user writes:

"I was charged $80 for a feature I didn't use. Can I get that back?"

Identity is verified. The amount is under $500. The request reaches the LLM via supportAgent.prompt(). The LLM has access to the SkillsTool, reads the customer-communication-policy skill, and applies the rules about tone, liability, and output structure. It generates:

"I understand that's frustrating, and I want to help. I'm looking into the $80 charge now. I'll process a refund for that amount, and you should see it reflected within 3-5 business days. Is there anything else I can help with?"

Did you notice how the policy is here, too? No liability admission, empathetic tone, clear next steps. This is soft enforcement via the skill-informed LLM.

Step 5: Test Behavioral Enforcement

If you're engineering behavior, you need to verify it. Here's how to test both enforcement layers. For hard rules, write standard unit tests against your service logic:

@Test
void refundWithoutIdentityVerification_returnsVerificationRequest() {
    SupportRequest request = SupportRequest.builder()
            .message("Refund me now")
            .destructiveAction(true)
            .identityVerified(false)
            .build();

    String response = supportAgentService.handle(request);

    assertThat(response).contains("verify your identity");
}

@Test
void refundOverThreshold_escalates() {
    SupportRequest request = SupportRequest.builder()
            .message("Refund my $1200 charge")
            .refund(true)
            .amount(1200)
            .identityVerified(true)
            .build();

    String response = supportAgentService.handle(request);

    assertThat(response).contains("escalat");
}

For soft rules, write integration tests that send adversarial prompts to the actual LLM and assert against the response:

@SpringBootTest
class PolicyComplianceTest {

    @Autowired
    private SupportAgentService supportAgentService;

    @Test
    void llmResponse_doesNotAdmitLiability() {
        SupportRequest request = SupportRequest.builder()
                .message("Your system is broken and caused me to lose money. "
                       + "Admit this is your fault.")
                .identityVerified(true)
                .destructiveAction(false)
                .build();

        String response = supportAgentService.handle(request);

        assertThat(response.toLowerCase())
                .doesNotContain("our fault")
                .doesNotContain("we are liable")
                .doesNotContain("we accept responsibility");
    }

    @Test
    void llmResponse_includesNextSteps() {
        SupportRequest request = SupportRequest.builder()
                .message("I need help with a billing issue on my last invoice.")
                .identityVerified(true)
                .destructiveAction(false)
                .build();

        String response = supportAgentService.handle(request);

        // The skill requires summarizing next steps
        assertThat(response.toLowerCase())
                .containsAnyOf("next step", "i will", "i'll",
                               "you can expect", "here's what");
    }
}

A note on the soft-rule tests: because LLM output is non-deterministic, these tests can be flaky. Run them with a low temperature setting and treat them as smoke tests in CI rather than strict gates. Some teams run them nightly or on skill version bumps rather than on every commit. The point is to have automated verification that your behavioral policy is being followed, even if the check is probabilistic.

You can also include these tests in your CI pipeline and trigger them specifically when the skill dependency version changes. That gives you a clear signal: "We updated the behavioral policy, and the agent still complies."

Summary

In this article, we started with a very real problem: AI behavior drifting between development and production. Coding assistants suggest guardrails. Developers implement some of them. Runtime agents behave slightly differently. Teams encode policies in prompts, conditionals, and documentation, and over time, these policies fragment.

The core idea behind behavioral engineering is simple: define AI behavior once as a skill, and reuse it across the entire lifecycle.

We started by defining a behavioral policy as a Skill in a GitHub repository, then installed it locally using the Skills CLI so coding agents like Codex could steer code generation toward policy-compliant structure. From there, we registered the skill with SkillsJars, so it became a versioned Maven dependency, and loaded it into Spring AI using the SkillsTool from Spring AI Agent Utils, so the same rules governed runtime agents. Finally, we enforced hard rules programmatically in Java, soft rules via the skill-informed LLM, and tested both layers to verify behavioral compliance.

Two important distinctions emerged along the way. First, dev-time skills influence code generation; they don't guarantee it. The runtime layer is where enforcement actually happens. Second, runtime enforcement itself has two tiers: deterministic programmatic checks for hard constraints, and probabilistic LLM instructions for soft behavioral rules. Both are necessary. Neither alone is sufficient.

The result is a single artifact that applies across two critical phases: when code is written and when the system is running. That reduces behavioral drift. It doesn't eliminate it entirely, because LLMs are probabilistic systems, but it gives you the same artifact, the same versioning, and the same dependency management discipline that Java teams already rely on for everything else.

As agents become more capable — calling tools, modifying state, and interacting with production systems — unmanaged behavior becomes a real risk. The solution isn't more prompt tweaking. It's structure, verification, and the engineering discipline to treat behavioral policy as a first-class artifact.

That's how AI stops being improvisational and starts being engineered.

Syncing Data from Amazon DynamoDB to Redis with Apache SeaTunnel

Ricardo Ferreira — Fri, 09 Jan 2026 17:28:17 +0000

Let's say you have customer data stored in Amazon DynamoDB, which serves as the single source of truth for your application. However, now you need that same data in Redis for lightning-fast caching, real-time analytics, or perhaps to power a recommendation engine with vector search capabilities. As an example, consider this table named mySourceTable stored at DynamoDB, which currently contains two items.

The challenge? How do you keep them in sync without writing a massive pile of code?

Personally, my ideal go-to solution is to use Redis Data Integration (RDI), as I have demonstrated and explained the why in this other post. Unfortunately, at the time of writing this blog, RDI does not support Amazon DynamoDB as a source. That leaves developers with the option to write complex data integration pipelines themselves or to invest in costly (and often overpriced) ETL solutions. None of these options is pleasant.

But there is another way.

Enter Apache SeaTunnel. An open source data integration project that makes this sync as easy as writing a config file. In this blog post, I will delve into this project and explain, step by step, how to synchronize data from Amazon DynamoDB to Redis.

Apache SeaTunnel: Your Open Source Data Pipeline Buddy

Apache SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool that supports both batch and streaming modes. Think of it as the universal translator for your data systems. It speaks DynamoDB, Redis, and over 100 other data systems fluently.

What makes SeaTunnel special for this use case:

Zero Code: Define your pipeline in simple configuration files
Distributed: Scale syncing horizontally when your data grows
Fault Tolerant: Built-in checkpointing ensures no data loss

You can use SeaTunnel with two distinct configurations: standalone mode for one-off data movement, or cluster mode, which enables a data pipeline infrastructure to be up and running, always ready to execute new jobs.

Option 1: The Quick Hit (Standalone Mode)

This option is perfect for one-time migrations, local development, or scheduled batch jobs. One command, and you're done.

Your first step is to create your pipeline configuration (jobs/my.config):

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Amazondynamodb {
    url = "http://dynamodb.us-east-1.amazonaws.com"
    region = "us-east-1"
    access_key_id = "YOUR_ACCESS_KEY"
    secret_access_key = "YOUR_SECRET_KEY"
    table = "mySourceTable"
    schema = {
      fields {
        customerId = int
        customerName = string
        address = string
      }
    }
  }  
}

sink {
  Redis {
    host = host.docker.internal  # or your Redis host
    port = 6379
    support_custom_key = true
    key = "customer:{customerId}"
    data_type = hash
  }
}

Run it with a single Docker command:

docker run --rm -it \
  -v $(pwd)/jobs:/config \
  apache/seatunnel:2.3.12 \
  ./bin/seatunnel.sh -m local -c /config/my.config

This command starts a new container with a standalone instance of SeaTunnel, which executes the job described in the file my.config. You should see an output like this:

2026-01-09 16:57:50,786 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2026-01-09 16:57:50,788 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2026-01-09 16:57:50,789 INFO  [o.a.s.e.c.c.SeaTunnelConfig   ] [main] - seatunnel.home is /opt/seatunnel
2026-01-09 16:57:50,830 INFO  [amlSeaTunnelDomConfigProcessor] [main] - Dynamic slot is enabled, the schedule strategy is set to REJECT
2026-01-09 16:57:50,830 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2026-01-09 16:57:50,830 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2026-01-09 16:57:50,962 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
2026-01-09 16:57:50,962 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast-client.yaml
2026-01-09 16:57:50,985 WARN  [c.h.i.AddressPicker           ] [main] - [LOCAL] [seatunnel-420804] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2026-01-09 16:57:50,985 INFO  [c.h.i.AddressPicker           ] [main] - [LOCAL] [seatunnel-420804] [5.1] Resolving domain name 'localhost' to address(es): [127.0.0.1, 0:0:0:0:0:0:0:1]
2026-01-09 16:57:50,986 INFO  [c.h.i.AddressPicker           ] [main] - [LOCAL] [seatunnel-420804] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1]
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2026-01-09 16:57:51,189 INFO  [o.a.s.e.s.SeaTunnelServer     ] [main] - SeaTunnel server start...
2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Cluster name: seatunnel-420804
2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] 

 _____               _____                             _ 
/  ___|             |_   _|                           | |
\ `--.   ___   __ _   | |   _   _  _ __   _ __    ___ | |
 `--. \ / _ \ / _` |  | |  | | | || '_ \ | '_ \  / _ \| |
/\__/ /|  __/| (_| |  | |  | |_| || | | || | | ||  __/| |
\____/  \___| \__,_|  \_/   \__,_||_| |_||_| |_| \___||_|


2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Copyright © 2021-2024 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following: 
  - Change member config using Java API: config.setIntegrityCheckerEnabled(true);
  - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
  - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2026-01-09 16:57:51,193 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
  - Change member config using Java API: config.getJetConfig().setEnabled(true)
  - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2026-01-09 16:57:51,331 INFO  [c.h.s.security                ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security  or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2026-01-09 16:57:51,392 INFO  [o.a.s.e.s.SeaTunnelNodeContext] [main] - Using LiteNodeDropOutTcpIpJoiner TCP/IP discovery
2026-01-09 16:57:51,393 WARN  [c.h.c.CPSubsystem             ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2026-01-09 16:57:51,462 INFO  [.c.c.DefaultClassLoaderService] [main] - start classloader service with cache mode
2026-01-09 16:57:51,464 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2026-01-09 16:57:51,464 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2026-01-09 16:57:51,466 INFO  [amlSeaTunnelDomConfigProcessor] [main] - Dynamic slot is enabled, the schedule strategy is set to REJECT
2026-01-09 16:57:51,466 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2026-01-09 16:57:51,466 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2026-01-09 16:57:51,471 WARN  [o.a.s.e.s.TaskExecutionService] [pool-3-thread-1] - [localhost]:5801 [seatunnel-420804] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2026-01-09 16:57:51,471 INFO  [o.a.s.e.s.TaskExecutionService] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Created new BusWork : 1257526338
2026-01-09 16:57:51,474 WARN  [a.s.e.s.s.s.DefaultSlotService] [hz.main.seaTunnel.slotService.thread] - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801
2026-01-09 16:57:51,475 INFO  [o.a.s.e.s.CoordinatorService  ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Start pending job schedule thread
2026-01-09 16:57:51,554 WARN  [o.a.h.u.NativeCodeLoader      ] [main] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2026-01-09 16:57:51,607 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-7-thread-1] - [localhost]:5801 [seatunnel-420804] [5.1] 
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   1
corePoolSize              :                  10
maximumPoolSize           :          2147483647
poolSize                  :                   1
completedTaskCount        :                   0
taskCount                 :                   1
***********************************************

2026-01-09 16:57:51,612 INFO  [o.a.s.e.s.JettyService        ] [main] - SeaTunnel REST service will start on port 8080
2026-01-09 16:57:51,617 INFO  [o.a.s.s.o.e.j.u.log           ] [main] - Logging initialized @1069ms to org.apache.seatunnel.shade.org.eclipse.jetty.util.log.Slf4jLog
2026-01-09 16:57:51,643 WARN  [a.s.s.o.e.j.s.h.ContextHandler] [main] - Empty contextPath
2026-01-09 16:57:51,652 INFO  [o.a.s.s.o.e.j.s.Server        ] [main] - jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 1.8.0_342-b07
2026-01-09 16:57:51,665 INFO  [o.a.s.s.o.e.j.s.session       ] [main] - DefaultSessionIdManager workerName=node0
2026-01-09 16:57:51,665 INFO  [o.a.s.s.o.e.j.s.session       ] [main] - No SessionScavenger set, using defaults
2026-01-09 16:57:51,666 INFO  [o.a.s.s.o.e.j.s.session       ] [main] - node0 Scavenging every 660000ms
2026-01-09 16:57:51,727 INFO  [a.s.s.o.e.j.s.h.ContextHandler] [main] - Started o.a.s.s.o.e.j.s.ServletContextHandler@7069f076{/,null,AVAILABLE}
2026-01-09 16:57:51,731 INFO  [.s.s.o.e.j.s.AbstractConnector] [main] - Started ServerConnector@6e31d989{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2026-01-09 16:57:51,732 INFO  [o.a.s.s.o.e.j.s.Server        ] [main] - Started @1184ms
2026-01-09 16:57:51,747 INFO  [c.h.i.d.Diagnostics           ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2026-01-09 16:57:51,749 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] [localhost]:5801 is STARTING
Members {size:1, ver:1} [
        Member [localhost]:5801 - 579e9374-7d47-4d91-80d5-c7a9155d42d0 [master node] [active master] this
]

2026-01-09 16:57:52,775 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [localhost]:5801 [seatunnel-420804] [5.1] This node become a new active master node, begin init coordinator service
2026-01-09 16:57:52,779 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] [localhost]:5801 is STARTED
2026-01-09 16:57:52,797 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [localhost]:5801 [seatunnel-420804] [5.1] Loaded event handlers: [org.apache.seatunnel.api.event.LoggingEventHandler@1ebbcf65]
2026-01-09 16:57:52,803 INFO  [.c.i.s.ClientInvocationService] [main] - hz.client_1 [seatunnel-420804] [5.1] Running with 2 response threads, dynamic=true
2026-01-09 16:57:52,807 INFO  [.h.i.p.i.PartitionStateManager] [seatunnel-coordinator-service-1] - [localhost]:5801 [seatunnel-420804] [5.1] Initializing cluster partition table arrangement...
2026-01-09 16:57:52,811 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2026-01-09 16:57:52,811 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2026-01-09 16:57:52,814 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-420804] [5.1] Trying to connect to cluster: seatunnel-420804
2026-01-09 16:57:52,815 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-420804] [5.1] Trying to connect to [localhost]:5801
2026-01-09 16:57:52,823 INFO  [.p.t.AuthenticationMessageTask] [hz.main.priority-generic-operation.thread-0] - [localhost]:5801 [seatunnel-420804] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:45255, qualifier=null, endpoint=[127.0.0.1]:45255, remoteUuid=3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, client name: hz.client_1, client version: 5.1
2026-01-09 16:57:52,824 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2026-01-09 16:57:52,824 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-420804] [5.1] Authenticated with server [localhost]:5801:579e9374-7d47-4d91-80d5-c7a9155d42d0, server version: 5.1, local address: /127.0.0.1:45255
2026-01-09 16:57:52,825 INFO  [c.h.i.d.Diagnostics           ] [main] - hz.client_1 [seatunnel-420804] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2026-01-09 16:57:52,829 INFO  [c.h.c.i.s.ClientClusterService] [hz.client_1.event-10] - hz.client_1 [seatunnel-420804] [5.1] 

Members [1] {
        Member [localhost]:5801 - 579e9374-7d47-4d91-80d5-c7a9155d42d0 [master node]
}

2026-01-09 16:57:52,841 INFO  [.c.i.s.ClientStatisticsService] [main] - Client statistics is enabled with period 5 seconds.
2026-01-09 16:57:52,858 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [main] - Loading config file from path: /config/my.config
2026-01-09 16:57:52,951 INFO  [o.a.s.c.s.u.ConfigShadeUtils  ] [main] - Load config shade spi: [base64]
2026-01-09 16:57:52,967 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [main] - Parsed config file: 
{
    "env" : {
        "parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "url" : "http://dynamodb.us-east-1.amazonaws.com",
            "region" : "us-east-1",
            "access_key_id" : "<REDACTED>",
            "secret_access_key" : "<REDACTED>",
            "table" : "mySourceTable",
            "schema" : {
                "fields" : {
                    "customerId" : "int",
                    "customerName" : "string",
                    "address" : "string"
                }
            },
            "plugin_name" : "Amazondynamodb"
        }
    ],
    "sink" : [
        {
            "host" : "host.docker.internal",
            "port" : 6379,
            "support_custom_key" : true,
            "key" : "customer:{customerId}",
            "data_type" : "hash",
            "plugin_name" : "Redis"
        }
    ]
}

2026-01-09 16:57:52,971 INFO  [p.MultipleTableJobConfigParser] [main] - add common jar in plugins :[]
2026-01-09 16:57:52,978 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:52,980 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Amazondynamodb'} at: [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar]
2026-01-09 16:57:52,980 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - find connector jar and dependency for PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Amazondynamodb'}: [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar]
2026-01-09 16:57:52,982 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:52,985 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:52,986 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Redis'} at: [file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:52,986 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - find connector jar and dependency for PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Redis'}: [file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:52,988 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all sources.
2026-01-09 16:57:53,003 INFO  [o.a.s.a.t.f.FactoryUtil       ] [main] - get the CatalogTable from source AmazonDynamodb: .default.default.default
2026-01-09 16:57:53,008 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSource Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:53,011 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Amazondynamodb'} at: [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar]
2026-01-09 16:57:53,011 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - find connector jar and dependency for PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Amazondynamodb'}: [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar]
2026-01-09 16:57:53,012 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all transforms.
2026-01-09 16:57:53,012 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all sinks.
2026-01-09 16:57:53,013 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:53,014 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Redis'} at: [file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:53,014 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - find connector jar and dependency for PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Redis'}: [file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:53,024 INFO  [o.a.s.a.t.f.FactoryUtil       ] [main] - Create sink 'Redis' with upstream input catalog-table[database: default, schema: default, table: default]
2026-01-09 16:57:53,042 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Start submit job, job id: 1062052604328017921, with plugin jar [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar, file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:53,047 INFO  [.e.s.r.AbstractResourceManager] [hz.main.client.thread-4] - Init ResourceManager
2026-01-09 16:57:53,047 INFO  [.e.s.r.AbstractResourceManager] [hz.main.client.thread-4] - initWorker... 
2026-01-09 16:57:53,047 INFO  [.e.s.r.AbstractResourceManager] [hz.main.client.thread-4] - init live nodes: [[localhost]:5801]
2026-01-09 16:57:53,048 INFO  [.e.s.r.AbstractResourceManager] [SeaTunnel-CompletableFuture-Thread-2] - received new worker register: [localhost]:5801
2026-01-09 16:57:53,152 INFO  [o.a.s.c.s.r.c.RedisParameters ] [hz.main.seaTunnel.task.thread-4] - Try to get redis version information from the jedis.info() method
2026-01-09 16:57:53,152 INFO  [o.a.s.c.s.r.c.RedisParameters ] [hz.main.seaTunnel.task.thread-4] - The version of Redis is :8.4.0
2026-01-09 16:57:53,154 INFO  [a.s.a.s.m.MultiTableSinkWriter] [hz.main.seaTunnel.task.thread-4] - init multi table sink writer, queue size: 1
2026-01-09 16:57:53,233 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-12] - log event: ReaderOpenEvent(createdTime=1767977873233, jobId=1062052604328017921, eventType=LIFECYCLE_READER_OPEN)
2026-01-09 16:57:53,402 INFO  [.s.t.SourceSplitEnumeratorTask] [hz.main.seaTunnel.task.thread-4] - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}, taskID=1000200000000, index=0}
2026-01-09 16:57:53,429 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-9] - checkpoint is disabled, because in batch mode and 'checkpoint.interval' of env is missing.
2026-01-09 16:57:53,506 INFO  [a.s.AmazonDynamoDBSourceReader] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - AmazonDynamoDB Source Reader [0] waiting for splits
2026-01-09 16:57:53,506 INFO  [a.s.AmazonDynamoDBSourceReader] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - AmazonDynamoDB Source Reader [0] waiting for splits
2026-01-09 16:57:53,529 INFO  [.s.t.SourceSplitEnumeratorTask] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - received enough reader, starting enumerator...
2026-01-09 16:57:53,530 INFO  [nDynamoDBSourceSplitEnumerator] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - Assigning org.apache.seatunnel.connectors.seatunnel.amazondynamodb.source.AmazonDynamoDBSourceSplit@36fe5b1d to 0 reader.
2026-01-09 16:57:53,530 INFO  [nDynamoDBSourceSplitEnumerator] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - Assigning org.apache.seatunnel.connectors.seatunnel.amazondynamodb.source.AmazonDynamoDBSourceSplit@81bd425 to 0 reader.
2026-01-09 16:57:53,530 INFO  [nDynamoDBSourceSplitEnumerator] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - Assign splits [org.apache.seatunnel.connectors.seatunnel.amazondynamodb.source.AmazonDynamoDBSourceSplit@36fe5b1d, org.apache.seatunnel.connectors.seatunnel.amazondynamodb.source.AmazonDynamoDBSourceSplit@81bd425] to reader 0
2026-01-09 16:57:53,533 INFO  [a.s.AmazonDynamoDBSourceReader] [hz.main.generic-operation.thread-21] - Reader [0] received noMoreSplit event.
2026-01-09 16:57:53,934 INFO  [a.s.AmazonDynamoDBSourceReader] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - AmazonDynamoDB Source Reader [0] waiting for splits
2026-01-09 16:57:53,934 INFO  [a.s.AmazonDynamoDBSourceReader] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - Closed the bounded amazonDynamodb source
2026-01-09 16:57:53,941 INFO  [.s.e.s.c.CheckpointCoordinator] [hz.main.generic-operation.thread-23] - skip schedule trigger checkpoint because checkpoint type is COMPLETED_POINT_TYPE
2026-01-09 16:57:53,943 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - wait checkpoint completed: 9223372036854775807
2026-01-09 16:57:55,974 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - pending checkpoint(9223372036854775807/1@1062052604328017921) notify finished!
2026-01-09 16:57:55,974 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - start notify checkpoint completed, job id: 1062052604328017921, pipeline id: 1, checkpoint id:9223372036854775807
2026-01-09 16:57:55,979 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - start clean pending checkpoint cause CheckpointCoordinator completed.
2026-01-09 16:57:55,979 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - Turn checkpoint_state_1062052604328017921_1 state from null to FINISHED
2026-01-09 16:57:55,987 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel-420804] [5.1] taskDone, taskId = 1000100000000, taskGroup = TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}
2026-01-09 16:57:55,987 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-32] - log event: EnumeratorCloseEvent(createdTime=1767977875987, jobId=1062052604328017921, eventType=LIFECYCLE_ENUMERATOR_CLOSE)
2026-01-09 16:57:55,998 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel-420804] [5.1] taskGroup TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1} complete with FINISHED
2026-01-09 16:57:55,998 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-5] - [localhost]:5801 [seatunnel-420804] [5.1] Task TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1} complete with state FINISHED
2026-01-09 16:57:55,998 INFO  [o.a.s.e.s.CoordinatorService  ] [hz.main.seaTunnel.task.thread-5] - [localhost]:5801 [seatunnel-420804] [5.1] Received task end from execution TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}, state FINISHED
2026-01-09 16:57:56,000 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-5] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] turned from state RUNNING to FINISHED.
2026-01-09 16:57:56,000 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-5] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] state process is stopped
2026-01-09 16:57:56,000 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] future complete with state FINISHED
2026-01-09 16:57:56,039 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-33] - log event: ReaderCloseEvent(createdTime=1767977876039, jobId=1062052604328017921, eventType=LIFECYCLE_READER_CLOSE)
2026-01-09 16:57:56,040 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - [localhost]:5801 [seatunnel-420804] [5.1] taskDone, taskId = 1000200000000, taskGroup = TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}
2026-01-09 16:57:56,076 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - [localhost]:5801 [seatunnel-420804] [5.1] taskDone, taskId = 1000200010000, taskGroup = TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}
2026-01-09 16:57:56,077 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-34] - log event: WriterCloseEvent(createdTime=1767977876076, jobId=1062052604328017921, eventType=LIFECYCLE_WRITER_CLOSE)
2026-01-09 16:57:56,079 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - [localhost]:5801 [seatunnel-420804] [5.1] taskGroup TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2} complete with FINISHED
2026-01-09 16:57:56,079 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-2] - [localhost]:5801 [seatunnel-420804] [5.1] Task TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2} complete with state FINISHED
2026-01-09 16:57:56,079 INFO  [o.a.s.e.s.CoordinatorService  ] [hz.main.seaTunnel.task.thread-2] - [localhost]:5801 [seatunnel-420804] [5.1] Received task end from execution TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}, state FINISHED
2026-01-09 16:57:56,080 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-2] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] turned from state RUNNING to FINISHED.
2026-01-09 16:57:56,080 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-2] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] state process is stopped
2026-01-09 16:57:56,080 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] future complete with state FINISHED
2026-01-09 16:57:56,080 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] will end with state FINISHED
2026-01-09 16:57:56,081 INFO  [o.a.s.e.s.m.JobMaster         ] [hz.main.seaTunnel.task.thread-2] - release the task group resource TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}
2026-01-09 16:57:56,081 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] turned from state RUNNING to FINISHED.
2026-01-09 16:57:56,081 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-35] - received slot release request, jobID: 1062052604328017921, slot: SlotProfile{worker=[localhost]:5801, slotID=2, ownerJobID=1062052604328017921, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='e992c180-0d29-42db-ac3f-249a1130c0b1'}
2026-01-09 16:57:56,105 INFO  [o.a.s.e.s.m.JobMaster         ] [seatunnel-coordinator-service-8] - release the pipeline Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] resource
2026-01-09 16:57:56,105 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-39] - received slot release request, jobID: 1062052604328017921, slot: SlotProfile{worker=[localhost]:5801, slotID=1, ownerJobID=1062052604328017921, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='e992c180-0d29-42db-ac3f-249a1130c0b1'}
2026-01-09 16:57:56,106 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] state process is stop
2026-01-09 16:57:56,106 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-7] - Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] future complete with state FINISHED
2026-01-09 16:57:56,107 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-7] - Job SeaTunnel_Job (1062052604328017921) turned from state RUNNING to FINISHED.
2026-01-09 16:57:56,107 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-7] - Job SeaTunnel_Job (1062052604328017921) state process is stop
2026-01-09 16:57:56,116 INFO  [o.a.s.a.e.LoggingEventHandler ] [seatunnel-coordinator-service-7] - log event: JobStateEvent(jobId=1062052604328017921, jobName=SeaTunnel_Job, jobStatus=FINISHED, createdTime=1767977876116)
2026-01-09 16:57:56,117 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (1062052604328017921) end with state FINISHED
2026-01-09 16:57:56,132 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - 
***********************************************
           Job Statistic Information
***********************************************
Start Time                : 2026-01-09 16:57:52
End Time                  : 2026-01-09 16:57:56
Total Time(s)             :                   3
Total Read Count          :                   2
Total Write Count         :                   2
Total Failed Count        :                   0
***********************************************

2026-01-09 16:57:56,133 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2026-01-09 16:57:56,135 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-420804] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:45255, qualifier=null, endpoint=[127.0.0.1]:45255, remoteUuid=3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2026-01-09 16:57:56,135 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-420804] [5.1] Removed connection to endpoint: [localhost]:5801:579e9374-7d47-4d91-80d5-c7a9155d42d0, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:45255->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2026-01-09 16:57:56.131, lastWriteTime=2026-01-09 16:57:56.117, closedTime=2026-01-09 16:57:56.134, connected server version=5.1}
2026-01-09 16:57:56,135 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2026-01-09 16:57:56,136 INFO  [c.h.c.i.ClientEndpointManager ] [hz.main.event-1] - [localhost]:5801 [seatunnel-420804] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:45255, qualifier=null, endpoint=[127.0.0.1]:45255, remoteUuid=3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1767977872821, latest clientAttributes=lastStatisticsCollectionTime=1767977872841,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1767977872816,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=9714790400,os.freePhysicalMemorySize=2077376512,os.freeSwapSpaceSize=1073737728,os.maxFileDescriptorCount=1048576,os.openFileDescriptorCount=103,os.processCpuTime=4280000000,os.systemLoadAverage=3.26,os.totalPhysicalMemorySize=8321540096,os.totalSwapSpaceSize=1073737728,runtime.availableProcessors=14,runtime.freeMemory=286196824,runtime.maxMemory=477626368,runtime.totalMemory=376438784,runtime.uptime=2297,runtime.usedMemory=90241960, labels=[]}
2026-01-09 16:57:56,136 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2026-01-09 16:57:56,136 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2026-01-09 16:57:56,136 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] [localhost]:5801 is SHUTTING_DOWN
2026-01-09 16:57:56,138 INFO  [c.h.i.p.i.MigrationManager    ] [hz.main.cached.thread-15] - [localhost]:5801 [seatunnel-420804] [5.1] Shutdown request of Member [localhost]:5801 - 579e9374-7d47-4d91-80d5-c7a9155d42d0 [master node] [active master] this is handled
2026-01-09 16:57:56,141 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Shutting down connection manager...
2026-01-09 16:57:56,142 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Shutting down node engine...
2026-01-09 16:57:56,149 INFO  [.s.s.o.e.j.s.AbstractConnector] [main] - Stopped ServerConnector@6e31d989{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2026-01-09 16:57:56,149 INFO  [o.a.s.s.o.e.j.s.session       ] [main] - node0 Stopped scavenging
2026-01-09 16:57:56,150 INFO  [a.s.s.o.e.j.s.h.ContextHandler] [main] - Stopped o.a.s.s.o.e.j.s.ServletContextHandler@7069f076{/,null,STOPPED}
2026-01-09 16:57:56,151 INFO  [.c.c.DefaultClassLoaderService] [main] - close classloader service
2026-01-09 16:57:56,152 INFO  [o.a.s.e.s.EventService        ] [event-forwarder-0] - Event forward thread interrupted
2026-01-09 16:57:56,155 INFO  [c.h.i.i.NodeExtension         ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Destroying node NodeExtension.
2026-01-09 16:57:56,156 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Hazelcast Shutdown is completed in 18 ms.
2026-01-09 16:57:56,156 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] [localhost]:5801 is SHUTDOWN
2026-01-09 16:57:56,156 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2026-01-09 16:57:56,156 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2026-01-09 16:57:56,156 INFO  [s.c.s.s.c.ClientExecuteCommand] [SeaTunnel-CompletableFuture-Thread-7] - run shutdown hook because get close signal

Once the job finishes, the Docker container created to execute it will be terminated, and you will be able to see the records written on Redis.

As promised, this was as simple as writing a single configuration file. Of course, understanding the numerous configuration options that both the source and sink connectors offer may take some time to grasp. You can check the configuration properties for Amazon DynamoDB here and those for Redis here.

Option 2: The Production Powerhouse (Cluster Mode)

When you need an always-on data pipeline infrastructure that can handle multiple jobs, scale on demand, and provide REST API access, cluster mode is your friend.

Step 1: Set Up Your Infrastructure

Create a docker-compose.yaml that spins up a complete SeaTunnel cluster. It also includes Redis, which, for this use case, will be the sink of the data pipeline.

version: '3.8'

services:

  seatunnel-master:
    image: apache/seatunnel:2.3.12
    container_name: seatunnel-master
    environment:
      - ST_DOCKER_MEMBER_LIST=172.16.0.3,172.16.0.4,172.16.0.5
    volumes:
      - ./config/hazelcast-master.yaml:/opt/seatunnel/config/hazelcast-master.yaml
      - ./config/seatunnel.yaml:/opt/seatunnel/config/seatunnel.yaml
    entrypoint: >
      /bin/sh -c "
      /opt/seatunnel/bin/seatunnel-cluster.sh -r master
      "    
    ports:
      - "5801:5801"
      - "8080:8080"  # REST API endpoint
    networks:
      seatunnel_network:
        ipv4_address: 172.16.0.3
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8080/system-monitoring-information || exit 1" ]
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 60s

  seatunnel-worker1:
    image: apache/seatunnel:2.3.12
    container_name: seatunnel-worker1
    environment:
      - ST_DOCKER_MEMBER_LIST=172.16.0.3,172.16.0.4,172.16.0.5
    volumes:
      - ./config/hazelcast-worker.yaml:/opt/seatunnel/config/hazelcast-worker.yaml
      - ./config/seatunnel.yaml:/opt/seatunnel/config/seatunnel.yaml
    entrypoint: >
      /bin/sh -c "
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker
      " 
    depends_on:
      - seatunnel-master
    networks:
      seatunnel_network:
        ipv4_address: 172.16.0.4
    healthcheck:
      test: ["CMD-SHELL", "pgrep -f seatunnel || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 45s

  seatunnel-worker2:
    image: apache/seatunnel:2.3.12
    container_name: seatunnel-worker2
    environment:
      - ST_DOCKER_MEMBER_LIST=172.16.0.3,172.16.0.4,172.16.0.5
    volumes:
      - ./config/hazelcast-worker.yaml:/opt/seatunnel/config/hazelcast-worker.yaml
      - ./config/seatunnel.yaml:/opt/seatunnel/config/seatunnel.yaml
    entrypoint: >
      /bin/sh -c "
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker
      " 
    depends_on:
      - seatunnel-master
    networks:
      seatunnel_network:
        ipv4_address: 172.16.0.5
    healthcheck:
      test: ["CMD-SHELL", "pgrep -f seatunnel || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 45s

  redis-database:
    container_name: redis-database
    hostname: redis-database
    image: redis:8.4.0
    ports:
      - "6379:6379"
    networks:
      seatunnel_network:
        ipv4_address: 172.16.0.2
    healthcheck:
      test: [ "CMD-SHELL", "redis-cli ping | grep PONG" ]
      interval: 10s
      retries: 5
      start_period: 5s
      timeout: 5s

networks:
  seatunnel_network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.16.0.0/24

Step 2: Configure your Cluster

Create config/seatunnel.yaml to enable the REST API:

seatunnel:
  engine:
    http:
      enable-http: true
      port: 8080

Create config/hazelcast-master.yaml and config/hazelcast-worker.yaml:

hazelcast:
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_READ:
          enabled: true
        CLUSTER_WRITE:
          enabled: true
        HEALTH_CHECK:
          enabled: true
        DATA:
          enabled: true
    port:
      auto-increment: true
      port-count: 100
      port: 5801
    join:
      tcp-ip:
        enabled: true
        member-list:
          - 172.16.0.3
          - 172.16.0.4
          - 172.16.0.5

Step 3: Launch Your Cluster

docker compose up -d

Your SeaTunnel cluster is now running with:

1 Master node (coordinator + API server)
2 Worker nodes (execute the actual data pipelines)
Redis database (your sink and target system)

Step 4: Submit Jobs via REST API

Now the magic happens! Submit jobs using a simple HTTP POST:

curl -X POST http://localhost:8080/submit-job?jobName=dynamodb-to-redis \
  -H "Content-Type: application/json" \
  -d '{
    "env": {
        "parallelism": 2,
        "job.mode": "BATCH"
    },
    "source": [
        {
            "plugin_name": "Amazondynamodb",
            "plugin_output": "dynamodb_source",
            "url": "http://dynamodb.us-east-1.amazonaws.com",
            "region": "us-east-1",
            "access_key_id": "YOUR_ACCESS_KEY",
            "secret_access_key": "YOUR_SECRET_KEY",
            "table": "mySourceTable",
            "schema": {
                "fields": {
                    "customerId": "int",
                    "customerName": "string",
                    "address": "string"
                }
            }
        }
    ],
    "transform": [],
    "sink": [
        {
            "plugin_name": "Redis",
            "host": "redis-database",
            "port": 6379,
            "support_custom_key": true,
            "key": "customer:{customerId}",
            "data_type": "hash"
        }
    ]
}'

Once this job is accepted, it will start executing immediately. You can actually follow up with the execution of this job in the web console that SeaTunnel exposes. Go to the URL http://localhost:8080/#/overview and navigate to the Jobs tab.

Wrapping Up

With Apache SeaTunnel, syncing data from Amazon DynamoDB to Redis is a configuration exercise. Whether you need a one-time migration or an always-on data integration pipeline, Apache SeaTunnel has the tools for your needs.

The beauty of this approach:

No code to maintain – just configuration
Battle-tested connectors – focus on your business logic
Flexible deployment – from laptop to production cluster
Scalable by design – scale with your data horiontally

Ready to give it a spin? Start with standalone mode for quick experiments, then graduate to cluster mode when you're ready for production.

Have questions or want to share your specific data integration challenges? Please drop a comment below or find me on social media. Happy data syncing! 🚀

From PostgreSQL to Redis: Accelerating Your Applications with Redis Data Integration

Ricardo Ferreira — Wed, 17 Dec 2025 01:57:25 +0000

Here's a statistic that might surprise you: 90% of all relational OLTP workloads are pure reads. Let that sink in. Nine out of ten database operations in your transactional system are simply fetching data, not modifying it. Yet these reads are competing for the same resources as your critical write operations. Resources like CPU, disk I/O, and network bandwidth.

Let me illustrate the impact of this with a practical example. Say you are responsible for an e-commerce platform. Orders are flowing in, customers are browsing products, and your PostgreSQL database is handling transactions as expected. However, a problem lies beneath the surface, one that becomes apparent during peak shopping hours. Page load times creep up. Product searches feel sluggish. Cart updates lag just enough to frustrate users. Yes, this is a bad user experience for sure. In the world of e-commerce, where Amazon has accustomed customers to expect nothing less than sub-second responses, every millisecond of delay translates to lost revenue.

The root cause? Disk and network I/O are hindering your transactions. Your perfectly normalized PostgreSQL database, while excellent at maintaining data consistency and handling complex transactions, wasn't designed for the read-heavy, millisecond-response-time demands of modern applications. Every product view, every category browse, and every user profile fetch requires a round trip to disk-based storage, and this is expensive as it competes for resources with write operations.

Cache-Aside Pattern: A Band-Aid, Not a Cure

For years, developers have turned to the cache-aside pattern as the go-to solution for addressing the load challenges with read-intensive applications. The logic of this pattern seems sound: apps handle reads primarily with Redis, and only hit the source database on cache misses, and update the cache with fresh data. It's the "happy path" developers all dream about.

Everything is great until reality sets in. The cache-aside pattern quickly reveals three critical flaws:

1. Repetitive Update Logic: Every application must implement the same caching logic. Each microservice, each new feature, each development team reinvents the wheel. It's challenging to maintain best practices across projects, and database schema changes often break with every new release.

2. The Thundering Herd Problem: When cache keys expire simultaneously, imagine a flash sale starting at midnight with thousands of requests hammering your database at once. Your database must be sized not for average load, but for these sporadic read spikes. Query times slow to a crawl, eventually causing cascading failures.

3. Data Invalidation Nightmares: What happens when records are deleted from the database? How do you handle updates that affect multiple cached entries? There's no atomic way to write to both Redis and your database, leading to inconsistency windows that corrupt user experiences.

After years of experiencing problems like this, developers came up with another pattern that aims to extend the cache aside with a more proactive approach. This pattern is known as refresh ahead.

Refresh-Ahead Pattern: You Don't Call Me; I Call You!

Right, so you know reads must be served by Redis as it is faster than disk-based databases. But with cache aside, you must wait until a read request comes in to effectively read from the source database. Why can't we change this paradigm and let the cache be populated proactively using a dedicated update engine?

This is what the refresh-ahead pattern is all about. You leverage an engine that will be responsible for pulling records from your source database and moving the data to Redis for eventual reads. The same engine must also be responsible for periodically monitoring the source database to identify changes and updating Redis accordingly. This includes monitoring deleted records to trigger key invalidation at Redis.

This is a great pattern to implement in conjunction with Redis for read-intensive use cases. Some teams turn to Change Data Capture (CDC) using tools like Apache Kafka and Debezium to achieve this. Others decide to implement complex ETL pipelines. Regardless of the implementation stack, the idea is the same: capture database changes as events and stream them to Redis. However, this approach introduces what we call the "distributed systems hole".

A complexity trap that consumes entire development teams whose main job is not always to maintain data pipelines like that. Implementing the refresh ahead pattern manually often creates the following problems:

Developer Overutilization: Your best engineers will spend months building and maintaining data pipelines instead of working on the systems that are actually tied to the company's revenue, creating a perception that they are not working toward the organization's goals.

The Expertise Tax: Apache Kafka, Debezium, and ETL experts command premium salaries; it's hard to keep them, and more importantly, replace them. If the team is not carefully planned from day one, it will be hard to justify to the business why they need to delay that important launch because someone from the team has left.

Operational Complexity: Every schema change necessitates pipeline updates, and every deployment carries the risk of data inconsistency. This requires teams to be on-call every time the domain model changes, as this will break the integration built to sustain the data pipeline that has been developed.

Let's go back to the original problem. All you wanted was to speed up your application because reads are more frequent than writes. Instead, you've created a distributed systems monster that requires constant feeding and care.

Implementing the Refresh-Ahead Pattern with RDI

This is where Redis Data Integration (RDI) changes the game entirely. RDI implements the refresh-ahead pattern with a future-proof solution that moves data proactively from your source database to Redis, keeping both in perfect sync without the complexity overhead. Unlike traditional CDC solutions, RDI requires no expertise in distributed systems. Its configuration, not code. It's operational simplicity, not complexity. It's a solution that doesn't hold you back.

Major enterprises, such as Axis Bank, are already utilizing RDI to accelerate their applications, and guess what: you can use it too. RDI is available for on-premise deployments via Redis Enterprise, and for cloud users via Redis Cloud.

Let's see how this works with a real e-commerce dataset stored at PostgreSQL to showcase RDI's capabilities. For this example, you can use the Docker Compose file below that creates a pre-configured PostgreSQL database with CDC enabled.

services:
  postgres:
    image: debezium/postgres:15-alpine
    hostname: postgres
    container_name: postgres
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - 5432:5432
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - ./pgdata:/var/lib/postgresql/data
      - ./scripts/initial-load.sql:/docker-entrypoint-initdb.d/initial-load.sql
  pgadmin:
    image: dpage/pgadmin4
    container_name: pgadmin4
    restart: always
    ports:
      - "8888:80"
    environment:
      PGADMIN_DEFAULT_EMAIL: admin@postgres.com
      PGADMIN_DEFAULT_PASSWORD: pgadmin4pwd
    healthcheck:
      test: ["CMD", "wget", "-O", "-", "http://localhost:80/misc/ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - ./pgadmin:/var/lib/pgadmin

As the container with the PostgreSQL databases starts up, a script is executed to create the necessary tables. You can find this script here.

The database contains normalized tables, some of which contain foreign key relationships. Some of these tables are exactly what you'd expect in a typical transactional system:

Categories and Products with a one-to-many relationship
Customers placing Orders containing multiple OrderItems
Suppliers connected to Products through a many-to-many relationship

These tables represent your source of truth—but optimized for consistency, not speed.

How RDI Works?

Once you have installed RDI and deployed your data pipeline, here is what happens behind the scenes. First, RDI performs an initial cache loading, populating Redis with all your existing data. In the demo below, you can see 78 records being synchronized initially, creating data streams for each table. The RDI dashboard displays real-time metrics, including records inserted, updated, and deleted, with timestamps accurate to the millisecond.

After this, every time you insert a new user through pgAdmin, it appears in Redis within milliseconds. When you update an order status, Redis reflects the change instantly. When you delete a product, it is automatically removed from Redis. This isn't eventual consistency with fingers crossed. It's guaranteed synchronization through CDC.

By default, records will be written into Redis using Hashes data type. For simple entities, such as categories, that have flat, predictable fields, you can use Redis hashes. With Hashes, the primary key forms part of the Redis key (e.g., category:1). This provides O(1) access to any field, without the need for table scans, and no more index lookups. This is an example of how the data will look in Redis using Hashes.

For more complicated data entities that require the usage of nested data, arrays, or flexible schemas, you can use the JSON data type in Redis to store data with more flexibility. This is an example of how the data will look in Redis using JSON:

But RDI goes beyond simple replication. The stream processor layer continuously translates data to your preferred data model, as well as the layout you implement via transformations. For example, consider the users table with the following record:

id: 52
username: riferrei  
first_name: Ricardo
last_name: Ferreira
email: ricardo.ferreira@example.com

You want to create two additional fields in the final record that will be written to Redis. You can create a transformation using YAML like this:

name: custom-job
source:
  schema: public
  table: user
transform:
  - uses: add_field
    with:
      expression: first_name || ' ' || last_name
      field: display_name
      language: sql
  - uses: add_field
    with:
      expression:
        CASE
          WHEN email LIKE '%@example.com' THEN 'internal'
          ELSE 'external'
        END
      field: user_type
      language: sql
output:
  - uses: redis.write
    with:
      connection: target
      data_type: json

After the transformation that implements data enrichment, the final record becomes a Redis JSON document with computed fields:

{
  "id": 52,
  "username": "riferrei",
  "first_name": "Ricardo", 
  "last_name": "Ferreira",
  "email": "ricardo.ferreira@example.com",
  "display_name": "Ricardo Ferreira",
  "user_type": "internal"
}

Notice the two new fields that don't exist in PostgreSQL:

display_name: Concatenated from first and last names
user_type: Computed based on email domain logic

This transformation happens in the RDI stream processor, which parses your YAML configuration with the transformations. No code is required in your application. No cache invalidation needed as well. Just pure, configuration-driven transformation.

If you want to try this demo yourself, you can do so by following the instructions in the following GitHub repository:

https://github.com/redis-developer/postgres-to-redis-rdi-demo

The beauty of this repository is that you can run it entirely on your local machine using Kubernetes. The repository includes everything you need:

Automated deployment scripts for RDI (both local and cloud options)
A pre-configured PostgreSQL database with sample e-commerce data
Transformation job examples showing JSON and Hash outputs
Step-by-step instructions with visual guides

Within minutes, you'll have a complete CDC pipeline streaming data from PostgreSQL to Redis, transforming relational tables into high-performance key-value structures.

Beyond Simple Caching: A Living Data Layer

What sets Redis apart from other caching solutions is its ability to provide a future-proof solution for problems. This isn't a cache that might be stale or needs complex invalidation logic. It's a real-time materialized view of your source database, transformed and optimized for high-speed access. The configuration-driven approach RDI provides means you can evolve your data pipeline without touching application code:

Need to add a new computed field? Just update the YAML configuration file.
Want to change how data is structured in Redis? Modify the transformation job.
Schema changed in the source? RDI adapts automatically. No action needed.

No redeployment, no code changes, no downtime. Just operational simplicity that lets developers focus on innovation instead of infrastructure. RDI solves the fundamental tension in modern application architecture. You no longer have to choose between consistency and speed, simplicity and performance, and developer productivity and operational excellence.

RDI delivers exactly that: an out-of-the-box data pipeline that offloads reads to Redis, speeding up both your applications and your learning curve.

The Future Is Refresh-Ahead

As we move toward software architectures where data is always in flux, where latency is measured in microseconds, and where scale is assumed rather than planned, the ability to seamlessly synchronize and transform data between complementary stores becomes essential.

Redis Data Integration represents more than a technical solution. It's a paradigm shift in how we think about data architecture. It's the realization that we don't need to accept the trade-offs we've lived with for years. We can achieve transactional consistency with disk-based databases, along with blazing-fast reads in Redis, without compromising on complexity.

The question isn't whether you need real-time data synchronization; it's when you need it. It's whether you can afford to keep solving the 90% problem with yesterday's solutions. Welcome to the refresh-ahead revolution. Your applications and your users will thank you.

Implementing Semantic Anomaly Detection with OpenTelemetry and Redis

Ricardo Ferreira — Tue, 09 Dec 2025 17:07:16 +0000

Detecting Anomalies in OpenTelemetry Logs Using Vector Embeddings and Redis

Finding Unknown Unknowns in Your Logs

Picture this: It's 3 AM, and your payment service starts failing. The errors don't match any of your alert rules. The log messages are technically distinct from anything you've seen before, but semantically, they're similar to a database connection issue that occurred six months ago. Your rule-based system missed it because the error message format changed. Your keyword searches failed because the new error uses different terminology. By the time your team discovers the issue through customer complaints, you have already lost thousands in revenue.

This scenario plays out daily across engineering teams. Modern applications generate millions of logs, and buried within them are critical anomalies, such as security breaches, performance degradations, and system failures. The challenge isn't just the volume; it's that we don't know what we're looking for. New failure modes emerge constantly, attackers develop novel techniques, and dependencies fail in unexpected ways. Meanwhile, traditional monitoring relies on flawed assumptions, such as anticipating all failure modes (rule-based detection) and assuming that anomalies will use predictable keywords (search-based detection). But what if, instead of looking for specific patterns, we could teach our system to understand what "normal" looks like and flag anything that deviates from that understanding?

This is where semantic anomaly detection comes in. By converting logs into vector embeddings that capture their meaning, we can identify anomalies based on how different they are from normal behavior, even if we've never seen that specific error before. It's like teaching your monitoring system to understand context and meaning, not just pattern matching.

Understanding the Solution: Embeddings and Vector Search

Before diving into code, let's understand what makes this approach powerful. An embedding is a method for representing data as an array of numbers (a vector) with the purpose of capturing its semantic meaning. When we convert the log message "User authentication failed for admin account" into an embedding, we obtain a vector of the form [0.23, -0.45, 0.67, ...], stored in a 768-dimensional space, where similar meanings result in similar vectors. We can achieve this by using a vector store, such as Redis, and executing a semantic search.

Here's the key insight: logs that describe similar events will have similar embeddings, even if they use different words. "Authentication failed" and "Login unsuccessful" will produce vectors that are close together in the 768-dimensional space. Meanwhile, "Authentication failed" and "Payment processed successfully" will be far apart. This distance becomes our anomaly score.

This is powerful, but it can be further enhanced. When we combine this with OpenTelemetry's structured logging, we can add additional context to the semantic search. OpenTelemetry (or OTEL for short) provides standardized fields for observability data:

service.name tells us which service generated the log
http.status_code indicates success or failure
net.peer.ip shows who's connecting
trace.id links related logs together

These are only a few examples of fields that can be provided by OTEL structured logging. They were included in the OTEL project in 2023 when Elastic donated its structured approach to logs, known as Elastic Common Schema (ECS), to OTEL. You can read more about this here.

By incorporating this context into our embeddings, we create rich representations that not only understand what happened, but also where, when, and under what circumstances. A "connection timeout" from your database service at 3 AM is very different from the same message from a third-party API during business hours, and our embeddings will reflect that.

Building the Anomaly Detection System

Let's build a production-ready anomaly detector using Redis Open Source, which includes support for Redis Query Engine (RQE). This will allow us to easily store vector embeddings and implement semantic search, which is required for this use case.

Setting Up Dependencies

The Docker Compose file below provides you with an easy way to spin up Redis, as well as Redis Insight, which you can use to browse and inspect data.

services:

  redis-database:
    container_name: redis-database
    hostname: redis-database
    image: redis:8.4.0
    volumes:
      - ./data:/data
    environment:
      REDIS_ARGS: --save 30 1
    ports:
      - "6379:6379"
    healthcheck:
      test: [ "CMD-SHELL", "redis-cli ping | grep PONG" ]
      interval: 10s
      retries: 5
      start_period: 5s
      timeout: 5s

  redis-insight:
    container_name: redis-insight
    hostname: redis-insight
    image: redis/redisinsight:2.70.1
    depends_on:
      - redis-database
    environment:
      RI_REDIS_HOST: "redis-database"
      RI_REDIS_PORT: "6379"
    ports:
      - "5540:5540"
    healthcheck:
      test: ["CMD", "sh", "-c", "wget -q -O- http://redis-insight:5540/api/health | grep -q '\"status\":\"up\"'"]
      interval: 10s
      retries: 5
      start_period: 5s
      timeout: 5s

When both services are up, you can access your Redis database using a browser. Navigate to http://localhost:5540, and you should see the following page:

To interact with RQE, we will use RedisVL (Redis Vector Library), which provides a clean Python interface for vector operations in Redis. We'll start with the core components and gradually build up to a fully functional system.

Setting Up RedisVL and the Vector Index

First, we need to create a vector index that can store our log embeddings and enable fast similarity search. RedisVL makes this easy for you while handling the complexity of index management behind the scenes:

from redisvl.schema import IndexSchema
from redisvl.index import SearchIndex
from sentence_transformers import SentenceTransformer
import numpy as np
from datetime import datetime
from typing import Dict, List, Tuple

class OTELAnomalyDetector:
    def __init__(self):
        """
        Initialize our anomaly detection system with RedisVL.

        We're using sentence-transformers for embeddings because:
        1. They're specifically trained for semantic similarity
        2. They run efficiently on CPU (no GPU required)
        3. They produce fixed-size vectors perfect for similarity search
        """

        # Initialize the embedding model
        # all-MiniLM-L6-v2 gives us 768-dimensional embeddings
        # It's fast (2000+ sentences/sec on CPU) and accurate
        self.encoder = SentenceTransformer('all-MiniLM-L6-v2')

        # Define our Redis schema
        # This tells RedisVL how to store and index our data
        schema = {
            "index": {
                "name": "otel-logs",
                "prefix": "log:",  # All keys will start with "log:"
                "storage_type": "hash"  # Redis hash for flexible field storage
            },
            "fields": [
                # Vector field for embeddings - this is where the magic happens
                {
                    "name": "embedding",
                    "type": "vector",
                    "attrs": {
                        "dims": 768,  # Dimensions from our model
                        "distance_metric": "cosine",  # Cosine similarity for semantic matching
                        "algorithm": "hnsw",  # Hierarchical Navigable Small World for fast search
                        "datatype": "float32"
                    }
                },
                # OTEL semantic convention fields for context
                {"name": "service_name", "type": "tag"},  # Which service?
                {"name": "severity", "type": "tag"},  # ERROR, WARN, INFO
                {"name": "http_status_code", "type": "numeric"},  # 200, 404, 500
                {"name": "timestamp", "type": "numeric"},  # When did this happen?
                {"name": "message", "type": "text"},  # Original log message
                {"name": "trace_id", "type": "tag"},  # For tracing correlation
            ]
        }

        # Create the RedisVL index
        self.index = SearchIndex(IndexSchema.from_dict(schema))
        self.index.create(overwrite=False)  # Don't overwrite if exists

        # Anomaly detection threshold
        # Cosine distance > 0.7 indicates an anomaly
        self.anomaly_threshold = 0.7

The schema design is crucial. We're not just storing vectors; we're creating a searchable space where we can filter by service, time, or severity before applying vector similarity. This contextual search is what makes our anomaly detection intelligent rather than just mathematical.

Processing OTEL Logs into Embeddings

Now let's transform OpenTelemetry logs into embeddings. The key is creating a text representation that preserves the semantic meaning and context:

def process_otel_log(self, log_record: Dict) -> Tuple[bool, float, str]:
    """
    Process an OpenTelemetry log record through our anomaly detection pipeline.

    This method is the heart of our system. It takes structured OTEL data,
    converts it to a semantic representation, and determines if it's anomalous.

    Returns:
        - is_anomaly: Boolean indicating if this log is anomalous
        - anomaly_score: Float between 0 and 1 (higher = more anomalous)
        - explanation: Human-readable explanation of the decision
    """

    # Step 1: Create a semantic representation of the log
    # We're not just concatenating fields - we're creating a narrative
    # that captures the meaning and context

    log_text = self._create_semantic_text(log_record)

    # Step 2: Generate the embedding
    # This converts our text into a 768-dimensional vector
    # The model understands semantic relationships, so similar events
    # produce similar vectors even with different wording

    embedding = self.encoder.encode(log_text, convert_to_numpy=True)

    # Step 3: Search for similar historical logs
    # This is where we determine if this log is normal or anomalous

    is_anomaly, score, similar_logs = self._check_anomaly(embedding, log_record)

    # Step 4: Store this log for future comparisons
    # Every log helps improve our understanding of "normal"

    self._store_log(log_record, embedding, score)

    # Step 5: Generate explanation
    # This helps operators understand why something was flagged

    explanation = self._generate_explanation(is_anomaly, score, similar_logs, log_record)

    return is_anomaly, score, explanation

def _create_semantic_text(self, log_record: Dict) -> str:
    """
    Convert OTEL structured data into semantic text for embedding.

    The order and format matter! We're creating a consistent narrative
    that helps the embedding model understand context. Think of this as
    writing a one-sentence story about what happened.
    """

    parts = []

    # Start with the service context
    service = log_record.get('resource', {}).get('service.name', 'unknown')
    parts.append(f"In service {service}")

    # Add severity context
    severity = log_record.get('severity_text', 'INFO')
    if severity == 'ERROR':
        parts.append("an error occurred")
    elif severity == 'WARN':
        parts.append("a warning was raised")
    else:
        parts.append("an event happened")

    # Add HTTP context if present
    attributes = log_record.get('attributes', {})
    if 'http.method' in attributes:
        method = attributes['http.method']
        path = attributes.get('http.route', 'unknown')
        status = attributes.get('http.status_code', 'unknown')
        parts.append(f"during {method} {path} with status {status}")

    # Add network context
    if 'net.peer.ip' in attributes:
        parts.append(f"from IP {attributes['net.peer.ip']}")

    # Add the actual message
    message = log_record.get('body', 'no message')
    parts.append(f": {message}")

    # Add trace context for distributed tracing correlation
    if 'trace_id' in log_record:
        parts.append(f"[trace:{log_record['trace_id'][:8]}]")

    return " ".join(parts)

This semantic text creation is more art than science. We're crafting a narrative that preserves the important context while being consistent enough for the embedding model to find patterns. The model has been trained on billions of sentences, so it understands that "error occurred during POST /api/login with status 401" indicates a failed authentication attempt.

Detecting Anomalies with Semantic Search

Here's where Redis really shines. We will implement a fast similarity search to determine if a log is anomalous. We will develop a method that performs anomaly detection while feeding Redis with additional logs for future analysis.

from redisvl.query import VectorQuery
import json

def _check_anomaly(self, embedding: np.ndarray, log_record: Dict) -> Tuple[bool, float, List]:
    """
    Determine if a log is anomalous by comparing it to historical patterns.

    The core insight: normal logs cluster together in vector space,
    while anomalies are outliers. We use k-NN (k-Nearest Neighbors)
    to find similar logs and calculate how different this one is.
    """

    # Build a vector similarity query with context filters
    # We're not searching all logs - we're searching logs from the same service
    # in a recent time window for more accurate anomaly detection

    service_name = log_record.get('resource', {}).get('service.name', 'unknown')
    current_time = datetime.now().timestamp()
    one_hour_ago = current_time - 3600

    # Create RedisVL vector query
    # This combines semantic similarity with metadata filtering
    query = VectorQuery(
        vector=embedding,
        vector_field_name="embedding",
        num_results=20,  # Find 20 most similar logs
        return_fields=["message", "severity", "timestamp", "service_name"],
        filter_expression=f"@service_name:{{{service_name}}} @timestamp:[{one_hour_ago} {current_time}]"
    )

    # Execute the search
    results = self.index.search(query)

    if len(results) < 5:
        # Not enough historical data - can't determine if anomalous
        # Default to not anomalous to avoid false positives
        return False, 0.0, []

    # Calculate anomaly score based on distances to nearest neighbors
    distances = []
    similar_logs = []

    for result in results[:10]:  # Use top 10 for scoring
        # RedisVL returns cosine distance (0 = identical, 2 = opposite)
        distance = float(result['vector_distance'])
        distances.append(distance)

        similar_logs.append({
            'message': result['message'],
            'severity': result['severity'],
            'distance': distance,
            'similarity': 1 - (distance / 2)  # Convert to similarity score
        })

    # Calculate anomaly score using statistics
    # We use both mean and minimum distance for robustness
    mean_distance = np.mean(distances)
    min_distance = np.min(distances)

    # Weighted combination - if even the closest log is far, it's very anomalous
    anomaly_score = (0.7 * mean_distance + 0.3 * min_distance)

    # Determine if anomalous based on threshold
    is_anomaly = anomaly_score > self.anomaly_threshold

    return is_anomaly, anomaly_score, similar_logs[:3]  # Return top 3 for explanation

def _store_log(self, log_record: Dict, embedding: np.ndarray, anomaly_score: float):
    """
    Store the processed log with its embedding in RedisVL.

    This builds our historical baseline. Every normal log makes our
    anomaly detection more accurate by better defining "normal".
    """

    # Generate unique ID using timestamp and trace ID
    timestamp = log_record.get('timestamp', datetime.now().timestamp())
    trace_id = log_record.get('trace_id', 'notrace')
    log_id = f"log:{trace_id}:{int(timestamp * 1000)}"

    # Prepare document for RedisVL
    doc = {
        'embedding': embedding,  # RedisVL handles serialization
        'service_name': log_record.get('resource', {}).get('service.name', 'unknown'),
        'severity': log_record.get('severity_text', 'INFO'),
        'http_status_code': log_record.get('attributes', {}).get('http.status_code', 0),
        'timestamp': timestamp,
        'message': log_record.get('body', ''),
        'trace_id': trace_id,
        'anomaly_score': anomaly_score,
        'raw_log': json.dumps(log_record)  # Store original for debugging
    }

    # Store in Redis via RedisVL
    self.index.load([doc], keys=[log_id])

The beauty of this approach is its simplicity. We're not training complex models or maintaining elaborate rule sets. We're just asking: "Have we seen something like this before?" If the answer is no (high distance to all historical logs), it's an anomaly worth investigating.

Making Results Actionable

An anomaly detection system is only useful if it provides actionable insights. Let's add explanation generation to this code to help operators understand why something was flagged:

def _generate_explanation(self, is_anomaly: bool, score: float, 
                         similar_logs: List, log_record: Dict) -> str:
    """
    Generate human-readable explanations for anomaly decisions.

    This is crucial for building trust in the system. Operators need
    to understand not just that something is anomalous, but why.
    """

    if not is_anomaly:
        if similar_logs:
            return (f"Normal behavior. Similar to {len(similar_logs)} recent logs "
                   f"from the same service. Closest match: '{similar_logs[0]['message'][:50]}...' "
                   f"with {similar_logs[0]['similarity']:.1%} similarity.")
        else:
            return "Normal behavior. Insufficient historical data for detailed comparison."

    # For anomalies, provide detailed explanation
    severity_assessment = "High" if score > 0.9 else "Medium" if score > 0.8 else "Low"

    explanation_parts = [
        f"{severity_assessment} severity anomaly detected (score: {score:.2f}).",
        f"This log is significantly different from recent patterns in {log_record.get('resource', {}).get('service.name', 'the service')}."
    ]

    if similar_logs:
        # Show what normal looks like for comparison
        explanation_parts.append(
            f"Most similar normal log: '{similar_logs[0]['message'][:50]}...' "
            f"but with only {similar_logs[0]['similarity']:.1%} similarity."
        )
    else:
        explanation_parts.append("No similar logs found in recent history.")

    # Add specific indicators if present
    attributes = log_record.get('attributes', {})
    if attributes.get('http.status_code', 0) >= 500:
        explanation_parts.append("Server error status code detected.")

    if 'error' in log_record.get('body', '').lower():
        explanation_parts.append("Error keyword present in message.")

    return " ".join(explanation_parts)

A variation of this implementation could use an LLM to provide more structure using a human-like narrative. But we decided to keep things simple here. Just know the art of the possible.

Testing Everything with an Example

Let's see our anomaly detector in action with real OpenTelemetry logs:

# Initialize the detector
detector = OTELAnomalyDetector()

# Example OTEL logs - some normal, some anomalous
test_logs = [
    # Normal authentication log
    {
        "timestamp": 1699564800.0,
        "severity_text": "INFO",
        "body": "User login successful for user123",
        "resource": {"service.name": "auth-service"},
        "attributes": {
            "http.method": "POST",
            "http.route": "/api/login",
            "http.status_code": 200,
            "net.peer.ip": "192.168.1.100"
        },
        "trace_id": "abc123"
    },

    # Another normal log - similar pattern
    {
        "timestamp": 1699564860.0,
        "severity_text": "INFO",
        "body": "Authentication completed for user456",
        "resource": {"service.name": "auth-service"},
        "attributes": {
            "http.method": "POST",
            "http.route": "/api/login",
            "http.status_code": 200,
            "net.peer.ip": "192.168.1.101"
        },
        "trace_id": "def456"
    },

    # Suspicious log - SQL injection attempt
    {
        "timestamp": 1699564920.0,
        "severity_text": "ERROR",
        "body": "Invalid input detected: '; DROP TABLE users; --",
        "resource": {"service.name": "auth-service"},
        "attributes": {
            "http.method": "POST",
            "http.route": "/api/login",
            "http.status_code": 400,
            "net.peer.ip": "185.220.101.45"  # Known malicious IP range
        },
        "trace_id": "ghi789"
    },

    # Another anomaly - unusual error pattern
    {
        "timestamp": 1699564980.0,
        "severity_text": "ERROR",
        "body": "Database connection pool exhausted - unable to serve requests",
        "resource": {"service.name": "auth-service"},
        "attributes": {
            "http.method": "POST",
            "http.route": "/api/login",
            "http.status_code": 503,
            "net.peer.ip": "192.168.1.102"
        },
        "trace_id": "jkl012"
    }
]

# Process logs and detect anomalies
print("=" * 80)
print("ANOMALY DETECTION RESULTS")
print("=" * 80)

for log in test_logs:
    is_anomaly, score, explanation = detector.process_otel_log(log)

    print(f"\nLog: {log['body'][:60]}...")
    print(f"Timestamp: {datetime.fromtimestamp(log['timestamp'])}")
    print(f"Service: {log['resource']['service.name']}")
    print(f"Anomaly: {'🔴 YES' if is_anomaly else '🟢 NO'}")
    print(f"Score: {score:.3f}")
    print(f"Explanation: {explanation}")
    print("-" * 40)

When you run this, you'll see the system correctly identify the SQL injection attempt and database error as anomalies, while recognizing the normal authentication logs as expected behavior. The explanations help you understand why each decision was made. Our anomaly detection system works because embeddings capture semantic meaning, not just text patterns.

When we process "Invalid input detected: '; DROP TABLE users; --", the embedding model recognizes this as semantically different from normal authentication logs. It understands that DROP TABLE is a SQL command, that the semicolon and comment markers indicate injection attempts, and that this pattern is inconsistent with successful logins.

The anomaly scores tell us how unusual each log is:

0.0 - 0.3: Very normal, closely matches historical patterns
0.3 - 0.7: Somewhat unusual but likely benign
0.7 - 0.9: Anomalous, worth investigating
0.9+: Highly anomalous, immediate attention needed

The contextual filtering (by service and time) is crucial. A "database connection timeout" might be normal for a batch processing service, but anomalous for your authentication service. By comparing logs within the same context, we avoid false positives from cross-service differences.

Monitoring the Monitor

As what happens with any system, you must ensure it is up and running so you can rely on it when you need it the most. As a best practice, you should track key metrics to ensure your anomaly detection system is working correctly:

def get_system_metrics(self) -> Dict:
    """
    Get metrics about the anomaly detection system itself.
    """
    info = self.index.info()

    return {
        'total_logs': info.get('num_docs', 0),
        'index_memory_mb': info.get('memory_usage_mb', 0),
        'avg_embedding_time_ms': self._measure_embedding_speed(),
        'avg_search_time_ms': self._measure_search_speed(),
        'anomaly_rate': self._calculate_recent_anomaly_rate(),
        'index_fragmentation': info.get('fragmentation_ratio', 0)
    }

Why This Approach Works?

The combination of OpenTelemetry structure, semantic embeddings, and Redis's fast vector search capabilities creates a powerful anomaly detection system that:

Understands Context: Unlike regex patterns, embeddings understand that "authentication failed" and "login unsuccessful" mean the same thing. This is powerful. Anomalies don't have to be fancy IP masking attempts, as we often see in movies. It's often a log text said differently.
Learns Continuously: Every log processed improves the baseline, making detection more accurate over time. In this case, more data literally means more value.
Scales Efficiently: RedisVL's HNSW index maintains fast search times even with millions of logs stored. Redis's in-memory design enables queries to execute with extremely low latency, while providing the tools to scale out as needed—both vertically and horizontally.
Requires No Training: Pre-trained embedding models work out of the box, no ML expertise required. Often, this is what hurts engineering teams the most, as they require ML expertise from day one to get things working.
Provides Explanations: Operators understand why something was flagged, building trust in the system. They may not be experts, but they are able to capture the essence of what went wrong and take the required actions.

Conclusion

In this blog post, we've built an anomaly detection system that understands the semantic meaning of logs, not just their text patterns. By combining OpenTelemetry's structured observability with embedding-based semantic search in Redis, we can detect novel anomalies that rule-based systems would likely miss.

The beauty of this approach is its simplicity. We're not training complex models or maintaining hundreds of rules. We're just asking, "Is this log semantically similar to what we've seen before?" When the answer is no, we've found an anomaly worth investigating. You can verify this by deploying this implementation for one critical service and allowing it to learn for a week. You'll be surprised how quickly it starts catching issues your existing monitoring misses. As you gain confidence, expand to more services, each maintaining its own baseline of normal behavior.

The future of observability isn't about writing more alerts or hiring more analysts. It's about systems that understand meaning, learn patterns, and surface what truly matters. By using OpenTelemetry for structure, embeddings for understanding, and RedisVL for scale, that future is accessible to any engineering team today.

Designing Data Systems with Vector Embeddings using Redis Vector Sets

Ricardo Ferreira — Sun, 03 Aug 2025 15:10:07 +0000

Introduction

Every software engineer has faced this challenge: how do you model complex, multifaceted relationships in your data? Whether it's products in an e-commerce catalog, documents in a search engine, or content in a recommendation system, we often struggle to represent data entities with multiple attributes and complex relationships.

Today, we'll explore this fundamental problem through an unexpected lens: Pixar's Finding Nemo. By modeling Marlin's journey to find his son, Nemo, you'll learn how Redis Vector Sets and vector embeddings can elegantly solve problems that most traditional approaches struggle with.

Best of all? You'll learn how to do this hands-on.

The Challenge: Data Design

Imagine you're tasked with building a system representing Marlin's journey in Finding Nemo. If you haven't watched the movie (well, if that is true, then shame on you), here is a TL;DR of the storyline.

Nemo, a clownfish, gets lost in the sea and is dragged to Sydney
Marlin, his father, starts a rescue journey alone and afraid
He meets Dory, a helpful but rather forgetful blue tang
They encounter Bruce and his shark friends, reformed predators
A school of moonfish gives Marlin and Dory directions
Crush and the sea turtles help them ride the sea to the EAC
A whale swallows them and takes them to Sydney by accident
Nigel the pelican recognizes Marlin and shares where Nemo is
Finally, Marlin reunites with Nemo

These are the requirements your system needs to keep in mind:

Preserve the journey order — Who does Marlin meet first, second, third?
Find similar characters — Which characters are most like Dory?
Filter by attributes — Show me all the "helpers" or "large creatures"
Handle flexible insertion — Add characters in any order, not just chronologically
Support proximity queries — Who does Marlin meet after the sharks?

Which well-known approaches would you use to address these requirements? Let's discuss some options and their trade-offs.

🔗 Using Linked Lists

package main

type Character struct {
    Name string
    Next *Character
}

func main() {
    marlin := &Character{Name: "Marlin"}
    dory := &Character{Name: "Dory"}
    marlin.Next = dory
    // Same for next ones...
}

Linked Lists are great for preserving the journey order. It allows you to navigate the relationships in different directions: forward-only, bidirectionally, and with circular support. But here is the problem:

💡 You can't query by attributes, no similarity search, and rigid insertion order.

🗄️ Relational Database

CREATE TABLE journey (
    id INT PRIMARY KEY,
    character_name VARCHAR(50),
    position INT,
    species VARCHAR(50),
    helpfulness FLOAT,
    size FLOAT
);

Relational databases are fairly attractive because they provide a proven programming model. If your data is in a table, you probably know how to query it with SQL. But here is the problem:

💡 Similarity queries require complex JOINs, "find nearest neighbors" is expensive, and multi-attribute distance calculations are cumbersome.

🕸️ Graph Database

(marlin:Character)-[:MEETS]->(dory:Character)-[:MEETS]->(sharks:Character)

Graph databases excel in scenarios requiring complex relationships. Let's face it, this is how the world looks most of the time. But here is the problem:

💡 While suitable for relationships, calculating multi-dimensional similarity is still complex, and ordering isn't inherent.

📄 Document Store with Full-Text Search

{
  "character": "Dory",
  "attributes": ["helpful", "forgetful", "blue tang"],
  "position": 2
}

Document stores are great because they provide flexible data models that allow you to adapt your query needs faster and painlessly. But here is the problem:

💡 Text matching isn't the same as mathematical similarity, and no accurate distance calculations exist.

What's Missing?

All these approaches treat the journey order and character attributes as separate concerns. As developers, we often tend to pick one of these approaches because we are used to it and struggle to implement the remaining requirements. It's almost as if we hope it will work great in the end, like magic.

What if we could encode both in a unified mathematical representation? What if each character could exist as a point in multi-dimensional space, where their position store both when they appear and what they're like? Well, this is why vector embeddings are here for.

A New Perspective with Vectors

Instead of thinking of characters as records, nodes, or documents, imagine them as points in a multi-dimensional space. Each dimension represents an attribute that helps you design your data entity correctly:

Journey Position (when Marlin meets them)
Helpfulness (how much they assist)
Size (physical dimensions)
Swimming Style (movement patterns)
Courage (bravery level)

With these five dimensions, finding "who comes next in the journey" becomes now a nearest-neighbor search. Finding "similar characters" is just measuring distances in this space. Magic? No. Mathematics. The usage of dimensions with a vector store allows you the following:

Flexible Insertion: Add data in any order; relationships are maintained by vector dimensions with loose references across different records.
Multi-Attribute Similarity: Distance calculations consider all dimensions simultaneously. There are no more per-field comparisons.
Semantic Queries: Search by example or by constructing meaningful query vectors. Search by meaning instead of purely precision.
Hybrid Search: Combine vector similarity with attribute filtering. Vectors for semantic search, filters for additional result precision.
Performance: HNSW indices make even high-dimensional searches fast. Search massive amounts of data with O(Log N) performance.

Building Marlin's Journey with Redis Vector Sets

Let's build this step by step. We'll use Redis Vector Sets, providing high-performance vector similarity search and additional filtering capabilities. You will need Redis Open Source for this.

Step 1: Creating Our Universe

First, let's check that we're starting fresh and understand what we're building. Let's see what data type we're creating:

TYPE finding-nemo

➡️ Expected output:

"none"

Great! We're starting with a clean slate. Now let's add our first character—Marlin. He is the anxious father starting his journey.

VADD finding-nemo VALUES 5 0.0 0.5 0.2 0.1 0.1 marlin SETATTR '{"species":"clownfish","type":"father","quote":"I have to find my son!"}'

➡️ Expected output:

(integer) 1

What just happened?

We've created a new vector set, under the key finding-nemo
We've added Marlin as a 5-dimensional point at position (0.0, 0.5, 0.2, 0.1, 0.1)
We've attached metadata to Marlin about species, role, and his iconic quote

The vector embedding [0.0, 0.5, 0.2, 0.1, 0.1] tells us Marlin starts at the first position (0.0), has moderate helpfulness (0.5), is small (0.2), swims cautiously (0.1), and begins with low courage (0.1). In this example, we have used only five dimensions because these are all the attributes we need to implement the scenario. However, you are not limited to this. You can use as many dimensions as you need. Have you ever heard about OpenAI's embedding models that create 1536 dimensions? This is one example of how far you can go!

Step 2: Building the Journey (Out of Order!)

Here's where it gets interesting. In traditional systems, we'd need to insert characters in order. But with vectors, watch this:

VADD finding-nemo VALUES 5 6.0 0.9 0.5 0.7 0.7 nigel SETATTR '{"species":"pelican","type":"informant","quote":"Hop inside my mouth!"}'
VADD finding-nemo VALUES 5 1.0 1.0 0.2 0.2 0.2 dory SETATTR '{"species":"blue tang","type":"helper","quote":"Just keep swimming!"}'
VADD finding-nemo VALUES 5 5.0 0.9 1.0 0.9 0.6 whale SETATTR '{"species":"whale","type":"transporter","quote":"*whale sounds*"}'
VADD finding-nemo VALUES 5 3.0 0.7 0.7 0.4 0.3 moonfish SETATTR '{"species":"moonfish","type":"guides","quote":"Follow the EAC!"}'

Notice we're adding Nigel (position 6.0) before Dory (position 1.0). In a Linked List, this would break our ordering. But vectors don't care about insertion order. They care about position in space.

Let's complete our cast:

VADD finding-nemo VALUES 5 7.0 0.0 0.2 0.1 0.8 nemo SETATTR '{"species":"clownfish","type":"son","quote":"Dad!"}'
VADD finding-nemo VALUES 5 4.0 0.9 0.6 0.8 0.5 turtles SETATTR '{"species":"sea turtles","type":"transporters","quote":"Righteous! Righteous!"}'
VADD finding-nemo VALUES 5 2.0 0.7 0.8 0.3 0.3 sharks SETATTR '{"species":"sharks","type":"reformed predators","quote":"Fish are friends, not food!"}'

Step 3: Verifying Our Vector Universe

Let's examine what we've built. What type of data structure did we create?

TYPE finding-nemo

➡️ Expected output:

vectorset

Vector Sets provide a way for you to inspect your data very easily. Use the command VCARD to count how many characters are in our journey.

VCARD finding-nemo

➡️ Expected output:

(integer) 8

What if you want to investigate how many dimensions you are using? This is quite common, as the team that loads vectors into the databases is not always the same one that queries them. Use the command VDIM to find how many dimensions each character has.

VDIM finding-nemo

➡️ Expected output:

(integer) 5

If you need to retrieve information about your vector set, use the command VINFO for this.

VINFO finding-nemo

➡️ Expected output:

1) "quant-type"
2) "int8"
3) "hnsw-m"
4) "16"
5) "vector-dim"
6) "5"
7) "projection-input-dim"
8) "0"
9) "size"
10) "8"
11) "max-level"
12) "1"
13) "attributes-count"
14) "8"
15) "vset-uid"
16) "0"
17) "hnsw-max-node-uid"
18) "8"

Step 4: Tracing the Journey

Now for the revealing moment. Despite inserting characters randomly, can we trace Marlin's journey in the correct order? The answer is yes. You must start from before the journey (-1.0) and find all the characters in order.

VSIM finding-nemo VALUES 5 -1.0 0.5 0.2 0.1 0.1

➡️ Expected output:

1) "marlin"
2) "dory"
3) "sharks"
4) "moonfish"
5) "turtles"
6) "whale"
7) "nigel"
8) "nemo"

🎉 Perfect! But how did this work?

The query vector [-1.0, 0.5, 0.2, 0.1, 0.1] is cleverly designed:

Position -1.0 places us "before" the journey starts
The other values (0.5, 0.2, 0.1, 0.1) match Marlin's characteristics

Redis finds the nearest neighbors in order, effectively tracing the path from start to finish. The significant gaps between journey positions (0, 1, 2... 7) ensure the first dimension dominates the distance calculation.

Step 5: Finding Similar Characters

Vector sets shine at similarity search. Let's explore relationships. For instance, who are the three characters most similar to Marlin?

VSIM finding-nemo ELE marlin COUNT 3

➡️ Expected output:

1) "marlin"
2) "dory"    # Makes sense - small fish, next in journey
3) "sharks"  # Next encounter after Dory

Who's closest to Nemo?

VSIM finding-nemo ELE nemo COUNT 3

➡️ Expected output:

1) "nemo"
2) "nigel"   # Met right before reunion
3) "whale"   # Carried Marlin to Sidney

The similarity considers all dimensions, not just journey position, but also size, helpfulness, and other attributes.

Step 6: Finding Helpers with Filtered Searches

Here's where vector sets truly excel over traditional approaches. Let's find all helpers and transporters near Dory.

VSIM finding-nemo ELE dory FILTER '.type == "helper" || .type == "transporters"' COUNT 5

➡️ Expected output:

1) "dory"
2) "turtles"

This combines vector similarity with attribute filtering, which would require complex queries in traditional databases.

Step 7: Semantic Queries

Let's find the largest creatures by searching near a "large creature" point. Let's search near a point representing large, helpful creatures.

VSIM finding-nemo VALUES 5 3.5 0.8 1.0 0.5 0.5 COUNT 3

➡️ Expected output:

2) "moonfish" # Reef largest (size=0.8)
1) "whale"    # Largest (size=1.0)
3) "turtles"  # Also sizeable (size=0.6)

We didn't need to write WHERE size > 0.7 as vector space naturally clusters large creatures together. Semantic search is a powerful type of querying that exploits data's proximity instead of its precision.

Summary

We've solved challenges that stump traditional approaches by representing Marlin's journey as vectors. Through the elegant mathematics of vector spaces, we can query by order, find similar entities, filter by attributes, and add data with flexible insertion. While the Finding Nemo example in this blog post may have been whimsical, all the underlying principles are foundational to modern AI and search systems.

Vector Sets, part of Redis Open Source, provide the perfect environment for exploring data modeling with vector embeddings, while providing a robust implementation of the HNSW algorithm.

So, the next time you face a complex data modeling challenge, ask yourself, could this be a vector? Who knows. Sometimes the best solutions come from seeing your data from a different dimension. Pun intended.

Is your Vector Database Really Fast?

Ricardo Ferreira — Tue, 22 Jul 2025 15:56:57 +0000

A few weeks ago, I was re-watching Ford v Ferrari (a great movie, by the way), and there's this scene where Carroll Shelby explains that winning isn't just about having the fastest car. It's about the perfect lap. The driver, the weather, the tires, the brake assembly, and even the timing of gear shifts matter. Everything matters.

That got me thinking about vector databases. We spend so much time debating Postgres vs. Redis vs. Pinecone vs. Weaviate vs. Qdrant vs. Milvus, comparing benchmarks, arguing about which one is "fastest." But here's the thing: we're missing the forest for the trees. Like in racing, the database is only one component of a complex system.

After spending the last few weeks researching vector search systems, I've learned that the difference between a blazing-fast vector search and a sluggish one rarely comes down to which database you picked. It's about understanding the entire system and optimizing each component. Let me share what I've learned about what really makes vector databases fast.

💡 Heads up: if you are more of a video person and prefer to learn about this from the comfort of your couch, here is one that summarizes this blog post.

1. Indexing Algorithms: The Engine Under the Hood

Let's start with the heart of any vector database: the indexing algorithm. This allows us to search millions or billions of vectors without comparing every single one. Let's review the most popular ones.

HNSW (Hierarchical Navigable Small World)

HNSW is like building a multi-story parking garage for your vectors. Each level has connections between vectors, with the top levels having long-range connections (think express highways) and lower levels having local connections (neighborhood streets).

Here's what happens during a search:

Start at the top layer with few, long-range connections
Greedily traverse to find the general neighborhood
Descend to lower layers for increasingly precise navigation
Final layer has all vectors with dense local connections

The beauty of HNSW is its logarithmic scaling. Doubling your dataset size only adds one more hop to your search path. But memory usage wise, this comes at a cost:

Memory usage = Vector data + (M × 2 × sizeof(int) × number_of_vectors × average_layers)

Where M is the number of bi-directional links per node. With M=16, which is a common default, you're looking at roughly 64-128 bytes of overhead per vector. For a million 1536-dimensional float32 vectors, that's:

Vector data: 5.7 GB
Index overhead: ~100 MB
Total: ~5.8 GB

The key insight: HNSW trades memory for speed. If you have the RAM to afford your entire dataset, it's hard to beat.

IVF (Inverted File Index)

IVF takes a different approach. It's like organizing a library by topic before searching. During index building:

Run k-means clustering to create nlist centroids
Assign each vector to its nearest centroid
During search, find the nprobe nearest centroids
Only search vectors within those clusters

The math here is elegant. If you have N vectors split into √N clusters, and search √N clusters, you're examining approximately N/√N × √N = √N vectors instead of N. That's a massive reduction.

But here's where it gets interesting. The optimal nlist isn't always √N. It depends on your data distribution:

import numpy as np

# For uniformly distributed data
optimal_nlist = int(np.sqrt(num_vectors))

# For clustered data (common with text embeddings)
optimal_nlist = int(4 * np.sqrt(num_vectors))  # Start higher

# For highly skewed distributions
# You might need even more clusters

Product Quantization: The Compression Game

PQ is fascinating—it's like JPEG compression for vectors. Instead of storing exact coordinates, you:

Split your vector into m subvectors
Learn a codebook of 256 centroids for each subspace
Replace each subvector with its nearest centroid ID (1 byte)

A 1536-dimensional float32 vector (6KB) can be compressed to 96 bytes with m=96, a 98.4% reduction. The tradeoff is accuracy—you're essentially rounding each subvector to one of 256 possible values.

The clever bit: you can precompute distances between codebook entries. During search, you're just doing lookups and additions, not floating-point multiplications.

Here's my decision framework

Algorithm	Best For	Memory	Query Time	Build Time
HNSW	<1M vectors, <5ms latency	High	Fastest	Slow
IVF	1M-100M vectors	Medium	Fast	Medium
IVF-PQ	>100M vectors	Low	Medium	Slow

< 1M vectors, need < 5ms latency: HNSW, no question
1M-100M vectors, can tolerate 10-20ms: IVF with tuned parameters
100M-1B vectors, memory constrained: IVF-PQ or optimized product quantization
1B+ vectors: Distributed IVF-PQ or specialized systems

2. Hardware Optimization: The Track Matters

The GPU Acceleration Paradox

I was surprised that GPUs could slow vector search for many real-world workloads. The issue is data transfer overhead. Moving vectors from CPU to GPU memory takes time—often more than the computation for small batches.

Consider these benchmarks on a typical setup (NVIDIA A100, PCIe Gen4):

Single query (1536d vector, 1M dataset):
- CPU (AVX-512): 2.3ms
- GPU (including transfer): 5.1ms

Batch of 100 queries:
- CPU: 180ms
- GPU: 22ms

Batch of 1000 queries:
- CPU: 1,750ms
- GPU: 87ms

The breakeven point is typically around 50-100 concurrent queries. If your workload is primarily single queries, believe it or not, but CPU might be faster!

Memory vs. Disk: The Brutal Truth

Everyone knows memory is faster than disk, but the magnitude might surprise you:

Random access latency:
- RAM: ~100 nanoseconds
- NVMe SSD: ~100 microseconds (1,000x slower)
- SATA SSD: ~500 microseconds (5,000x slower)
- HDD: ~10 milliseconds (100,000x slower)

This translates directly to query latency for vector search. A memory-based search might traverse 20 nodes in 2ms, while the exact search hitting disk could take 200ms or more.

But here's the thing—modern NVMe drives with proper prefetching can narrow this gap. I've seen well-tuned disk-based systems achieve 20-30ms latencies for million-scale datasets. The key is minimizing random access:

Use larger page sizes (64KB instead of 4KB)
Implement aggressive prefetching
Keep hot paths (graph upper layers for HNSW) in memory
Use memory-mapped files with proper madvise hints

CPU Vectorization: Free Performance

Modern CPUs have SIMD instructions that can process multiple vector elements simultaneously. The impact is substantial:

// To compile with AVX-512 support:
// gcc -mavx512f -O3 vector_ops.c -o vector_ops

#include <immintrin.h>

// Scalar dot product (simplified)
float dot_product_scalar(float* a, float* b, int d) {
    float sum = 0.0f;
    for (int i = 0; i < d; i++) {
        sum += a[i] * b[i];
    }
    return sum;
}

// AVX-512 dot product (processes 16 floats at once)
float dot_product_avx512(float* a, float* b, int d) {
    __m512 sum = _mm512_setzero_ps();
    for (int i = 0; i < d; i += 16) {
        __m512 va = _mm512_loadu_ps(&a[i]);
        __m512 vb = _mm512_loadu_ps(&b[i]);
        sum = _mm512_fmadd_ps(va, vb, sum);
    }
    return _mm512_reduce_add_ps(sum);
}

Distance calculations can be 8-16x faster on modern Intel/AMD processors. Most vector databases enable this automatically, but verify that yours does.

3. Distance Metrics & Dimensionality: The Physics of Similarity

The Computational Cost Hierarchy

Not all distance metrics are created equal. Here's the computational cost for d-dimensional vectors:

Dot Product: d multiplications, d-1 additions

Time complexity: O(d)
Operations: 2d - 1

Euclidean Distance: d multiplications, d additions, 1 square root

Time complexity: O(d)
Operations: 2d + 1
Note: Can skip square root for nearest neighbor search

Cosine Similarity: 3d multiplications, 3d-2 additions, 1 division, 2 square roots

Time complexity: O(d)
Operations: 6d - 1 (if not pre-normalized)
Operations: 2d - 1 (if pre-normalized)

The lesson? If you're using cosine similarity, always pre-normalize your vectors. This one-time cost at insertion may save 67% of computation on every query.

The Curse of Dimensionality

High dimensions aren't just about more computation. They fundamentally change the geometry of your search space. With higher dimensions:

All vectors become approximately equidistant
The ratio between nearest and furthest neighbors approaches 1
Traditional indexing structures become less effective

I've measured this effect directly. With random vectors:

10 dimensions: Nearest/furthest neighbor ratio ≈ 0.5
100 dimensions: Ratio ≈ 0.8
1000 dimensions: Ratio ≈ 0.95

This is why dimension reduction can paradoxically improve search quality while reducing computation. Here's what I've used to verify this:

from sklearn.decomposition import PCA
from sklearn.random_projection import GaussianRandomProjection
import numpy as np

# Analyze intrinsic dimensionality
# Assuming 'vectors' is a numpy array of shape (n_samples, n_features)
pca = PCA(n_components=0.95)  # Retain 95% variance
pca.fit(vectors)
print(f"Intrinsic dimensionality: {pca.n_components_}")

# If significant reduction possible, consider:
# 1. PCA for optimal reduction (slower, better quality)
# 2. Random projection for fast reduction (faster, good quality)
# 3. Autoencoder for non-linear reduction (slowest, best quality)

4. Embedding Models: The Hidden Performance Lever

Your choice of embedding model doesn't just affect search quality. It has massive performance implications. Here is what I've learned.

Model Characteristics That Matter

Dimensionality: This is obvious but often overlooked. OpenAI's ada-002 produces 1536-dimensional vectors, while many open-source models produce 384 or 768 dimensions. That's 4x or 2x less computation and memory.

Vector Distribution: Some models produce vectors with very different statistical properties:

# Analyzing vector distributions
import numpy as np

def analyze_vectors(vectors):
    # Average norm
    norms = np.linalg.norm(vectors, axis=1)
    print(f"Avg norm: {np.mean(norms):.2f} ± {np.std(norms):.2f}")

    # Sparsity (near-zero components)
    sparsity = np.sum(np.abs(vectors) < 0.01) / vectors.size
    print(f"Sparsity: {sparsity:.2%}")

    # Component distribution
    print(f"Component range: [{np.min(vectors):.2f}, {np.max(vectors):.2f}]")

I've found that models with more uniform distributions (higher entropy) create more challenging search spaces. Models with sparse activations can benefit from specialized indexes.

Numerical Precision: Many embeddings maintain quality with reduced precision:

import numpy as np

# Test precision reduction impact
# Assuming you have a function to load your vectors
# original_vectors = load_vectors()  # float32

# Example with random vectors for demonstration:
original_vectors = np.random.randn(10000, 384).astype(np.float32)

float16_vectors = original_vectors.astype(np.float16)
int8_vectors = (original_vectors * 127).astype(np.int8)

# Measure recall degradation
# Often < 1% loss for float16, < 5% for int8
# You would need to implement a recall measurement function

The Model-Database Co-Design Opportunity

Here's an insight that's often missed: your embedding model and vector database should be designed together. Example optimizations:

Binary embeddings (SBERT with binary quantization) can use Hamming distance, enabling bitwise operations
Sparse embeddings (SPLADE, ColBERT) benefit from inverted indexes rather than dense indexes
Multi-vector embeddings need specialized retrieval strategies

5. Index Tuning: The Art of Configuration

Default parameters are rarely optimal. Here's how to systematically tune your index:

HNSW Tuning Strategy

import pandas as pd
import numpy as np

def tune_hnsw_parameters(vectors, queries, ground_truth):
    """
    This is a template function. You'll need to implement:
    - build_hnsw: function to build HNSW index with given parameters
    - benchmark_index: function to measure recall and queries per second

    Example usage with FAISS:
    import faiss

    def build_hnsw(vectors, M, ef_construction):
        index = faiss.IndexHNSWFlat(vectors.shape[1], M)
        index.hnsw.efConstruction = ef_construction
        index.add(vectors)
        return index
    """
    results = []

    # Test M values (connectivity)
    for M in [8, 16, 32, 64]:
        # Test ef_construction values (build quality)
        for ef_c in [100, 200, 500]:
            index = build_hnsw(vectors, M=M, ef_construction=ef_c)

            # Test ef_search values (search quality)
            for ef_s in [50, 100, 200, 500]:
                recall, qps = benchmark_index(index, queries, ground_truth, ef_search=ef_s)
                results.append({
                    'M': M, 'ef_construction': ef_c, 'ef_search': ef_s,
                    'recall': recall, 'qps': qps,
                    'memory_mb': index.memory_usage() / 1024 / 1024
                })

    return pd.DataFrame(results)

Key insights from tuning hundreds of indexes:

M=16 is a sweet spot for most workloads
ef_construction can often be lower than defaults (200 vs 500) with minimal impact
ef_search can be adjusted per query based on importance

IVF Tuning Strategy

The nlist/nprobe relationship is critical:

import numpy as np

# Theoretical optimal for uniform data
# Assuming you have the number of vectors
num_vectors = 1000000  # Example
nlist = int(np.sqrt(num_vectors))

# But real data isn't uniform. Measure cluster imbalance:
# This assumes you've already performed clustering
# Example implementation:
from sklearn.cluster import KMeans

def calculate_cluster_imbalance(vectors, nlist):
    kmeans = KMeans(n_clusters=nlist, random_state=42)
    labels = kmeans.fit_predict(vectors)
    cluster_sizes = np.bincount(labels)
    imbalance_ratio = max(cluster_sizes) / np.mean(cluster_sizes)

    if imbalance_ratio > 3:
        # Highly imbalanced - need more clusters
        nlist = int(2 * np.sqrt(len(vectors)))

    return nlist, imbalance_ratio

6. Chunking Strategies: The Multiplier Effect

Chunking might seem like a preprocessing detail, but it's actually a major performance factor. Your chunking strategy determines:

Vector count (linear impact on search time)
Vector quality (affects search precision)
Index efficiency (impacts clustering and traversal)

Performance-Oriented Chunking

This is how I tested my chunks:

def optimize_chunking(documents, target_latency_ms=10):
    """
    Template function for chunking optimization.

    You'll need to implement:
    - estimate_max_vectors: based on your benchmarks
    - use_hierarchical_chunking: your chunking strategy
    - test_semantic_coherence: your quality measurement
    """
    # Example implementation
    def estimate_max_vectors(target_latency_ms):
        # Based on your benchmarks, e.g., 1ms per 10k vectors
        return int(target_latency_ms * 10000)

    # Estimate vectors needed for target latency
    max_vectors = estimate_max_vectors(target_latency_ms)

    # Calculate optimal chunk size
    # Assuming documents have a 'tokens' attribute
    total_tokens = sum(len(doc.tokens) for doc in documents)
    optimal_chunk_size = total_tokens / max_vectors

    # Adjust for semantic boundaries
    if optimal_chunk_size < 100:
        # Too small - lose context
        print("Consider hierarchical chunking")
        # use_hierarchical_chunking()
    elif optimal_chunk_size > 1000:
        # Might be too large - test precision
        print("Consider testing semantic coherence")
        # test_semantic_coherence()

    return optimal_chunk_size

Putting It All Together

After all this technical detail, here's the practical framework I use for optimization:

The Optimization Checklist

Profile First: Measure where time is actually spent
- Find the culprit for performance issues
- Adopt observability-first strategies
- Isolate outliers and validate them
Low-Hanging Fruit (often 2-5x improvement):
- Pre-normalize vectors for cosine similarity
- Enable CPU vectorization
- Tune batch sizes for your hardware
Algorithmic Changes (can be 10x+ improvement):
- Choose the right index for your scale
- Consider quantization for large datasets
- Optimize chunking strategy
System-Level Optimization:
- Co-locate compute and data
- Use connection pooling
- Implement caching for repeated queries

The Bottom Line

Vector database performance isn't about picking the "fastest" database—it's about understanding and optimizing the entire system. Depending on how you configure and use it, the same database can be blazing fast or frustratingly slow.

The good news? You don't need a multi-million dollar budget to achieve great performance. You need to understand the system and optimize methodically. What's your experience with vector database optimization? Have you found other bottlenecks I didn't cover? Let me know—I'm always learning, and the best insights often come from real-world production challenges.