DEV Community: Avaneesh Yadav

Which Claude Certification Should You Take? All 4 Compared

Avaneesh Yadav — Fri, 09 Oct 2026 17:04:56 +0000

Anthropic now runs four distinct Claude certifications, and the names alone don't make it obvious which one is actually for you. "Associate" sounds junior, "Architect" sounds senior, and "Developer" sits somewhere in between — but the real differences are about what role you're in, not how experienced you are.

I've gone through this landscape in detail while building free prep content for all four on this site. Here's the honest breakdown: what each certification actually tests, the hard numbers (cost, duration, passing score), and which one matches your actual job — not just your job title.

The Four Certifications at a Glance

Certification	Code	Duration	Questions	Pass Score	Cost	Who It's For
Associate — Foundations	CCAF	90 min	50 (MCQ)	700/1000	$100	Business users, analysts, team leads using Claude day-to-day
Developer — Foundations	CCDV-F	120 min	53	720/1000	$125	Software developers building with the Claude API/SDK
Architect — Foundations	CCAR-F	120 min	60 (MCQ)	720/1000	$125	Engineers designing agentic systems with Claude
Architect — Professional	CCAR-P	120 min	60 (scenario-based)	720/1000	$125	Architects already CCAR-F-certified, operating at enterprise scale

All four: 12-month validity, delivered online or at a Pearson VUE test center. No certification requires a prior one except Professional, which assumes Foundations-level knowledge.

If you only remember one distinction: Associate is "can you use Claude well," Developer is "can you build with Claude," and the two Architect levels are "can you design systems with Claude" (Foundations) and "can you design systems with Claude at enterprise scale, with real tradeoffs" (Professional).

Claude Certified Associate — Foundations (CCAF)

Download the full-size infographic

Who it's for: students, early-career professionals, business users, product managers, educators — anyone who wants to use Claude well in a business setting, no coding required.

What it tests (domain weights):

Claude Fundamentals & Capabilities — 24%
Prompt Engineering Fundamentals — 20%
Responsible AI & Safety — 16%
Practical Use Cases & Productivity — 14%
Claude in the Broader AI Ecosystem — 12%
Model Limitations & Best Practices — 8%
Evaluation & Output Interpretation — 6%

This is the only one of the four with no coding, no API, no SDK on the exam. If your job is writing prompts, checking Claude's output before you act on it, or deciding which Claude product fits a task — this is your certification.

Free prep on this site: Course overview · practice quiz

Claude Certified Developer — Foundations (CCDV-F)

Download the full-size infographic

Who it's for: software developers, AI/ML engineers, full-stack and solution engineers — anyone shipping real Claude-powered applications.

What it tests (domain weights):

Applications & Integration — 33.1%
Model Selection & Optimization — 16.8%
Agents & Workflows — 14.7%
Prompt & Context Engineering — 11%
Tools & MCPs — 10.6%
Security & Safety — 8.1%
Claude Code — 3.1%
Evaluation, Testing & Debugging — 2.6%

A third of this exam is just "Applications & Integration" — building and integrating Claude into real apps (API calls, streaming, tool calling, SDK usage). This is a practitioner exam: it tests implementation, not architecture. "Developer certification is about implementation, not architecture" is the line worth remembering if you're deciding between this and Architect Foundations.

Free prep on this site: Course overview · practice quiz

Claude Certified Architect — Foundations (CCAR-F)

Download the full-size infographic

Who it's for: solutions architects, technical leads, engineers moving from "using Claude" to "designing systems with Claude" — the architecture-and-tradeoffs layer above Developer.

What it tests (domain weights):

Agentic Architecture & Orchestration — 27%
Claude Code Configuration & Workflows — 20%
Prompt Engineering & Structured Output — 20%
Tool Design & MCP Integration — 18%
Context Management & Reliability — 15%

This is where the exam stops asking "can you call the API" and starts asking "can you design the agent, choose single- vs multi-agent, and make it reliable." If you've been building toward architecture-level decisions rather than feature-level implementation, this is the one.

Free prep on this site: 8-page complete course · practice quiz & mock exam · my personal pass experience

Claude Certified Architect — Professional (CCAR-P)

Download the full-size infographic

Who it's for: solutions architects, enterprise architects, technical leaders, AI platform builders — people already comfortable at the Foundations level who need to prove they can operate at enterprise scale.

What it tests (domain weights):

Strategic AI Architecture & Vision — 25%
Enterprise Integration & Ecosystem — 18%
Agentic Systems at Scale — 15%
Governance, Risk & Responsible AI — 12%
Observability, Evaluation & MLOps — 10%
Cost, Performance & Optimization — 10%
Organizational Enablement & Adoption — 8%
Advanced Design Patterns & Case Studies — 5%

The jump from Foundations to Professional is the jump from "design one agentic system" to "architect AI strategy, governance, and cost tradeoffs across an organization." All 60 questions are scenario-based — there's no pure recall here, it's judgment under constraints. "From working prototypes to enterprise impact" is the honest summary.

Free prep on this site: Course overview · practice quiz

So Which One Should You Take?

A few decision rules that cut through most of the overthinking:

You use Claude but don't write code against it → Associate. Full stop, nothing else applies.
You write application code that calls Claude, but you're not the one deciding the agent architecture → Developer.
You're the one deciding single- vs multi-agent, tool design, and reliability patterns → Architect Foundations.
You're already Architect-Foundations-level and your decisions now touch cost, governance, and multiple teams → Architect Professional.

They're not a strict ladder you have to climb in order — Associate and Developer test genuinely different skills (business use vs. implementation), not the same skill at different depths. Pick based on your actual role, not based on which title sounds more senior.

How to Actually Prepare

Every one of these exams rewards the same study shape, regardless of which you're taking: a 4-week plan that front-loads the heaviest-weighted domains, practice questions that force you to apply a concept rather than recall it, and a final week of nothing but timed practice exams.

All four free courses and practice quizzes linked above are built exactly that way — domain-weighted, with the highest-value material first. Start with whichever certification matches your current role from the decision rules above, or browse all four prep tracks on the certifications hub.

Agentic AI Explained: From Prompt to Autonomous Systems

Avaneesh Yadav — Fri, 09 Oct 2026 16:54:13 +0000

Download the full-size infographic

"Agentic AI" has become one of those terms everyone uses and almost nobody defines precisely. Is it just a chatbot with more steps? Is it the same thing as "AI agents"? Is it hype, or is there a real architectural shift underneath it?

There is a real shift — but it's easier to see once you stop treating "agentic AI" as a product category and start treating it as a capability upgrade on top of the LLM you already know. This post walks through what actually changes, how the pieces fit together, and where the real risks are — not the ones in the headlines.

Why Agentic AI Matters

A chatbot and an agent are built from the same underlying model. The difference is what happens around it.

	Traditional AI (Chatbot)	Agentic AI
Interaction	Answers questions	Pursues objectives
Scope	Single turn	Uses tools and APIs
Memory	None	Maintains long-term context
Capability	Conversation only	Plans and reasons
Workflow	Limited to the chat	Coordinates multi-step workflows
Output	Text	Executes actions in real systems

Ask a chatbot "what is RAG?" and it answers from what it already knows. Give an agent the goal "create a market analysis report," and it has to decide what to pull from internal systems, which tools to call, how to sequence the work, and how to verify the result before handing it back — all without a human writing out each step.

That gap — from answering to doing — is the entire story of agentic AI.

The Evolution of Enterprise AI

Agentic AI didn't appear from nowhere. It's the next step in a fairly linear progression:

Rules engines (1980s–90s) — if/else logic automation. Deterministic, brittle, but predictable.
Machine learning (2000s) — pattern recognition from data. Still narrow, still single-purpose.
Deep learning (2010s) — neural networks at scale. Better pattern recognition, still no reasoning.
Generative AI (2020) — models that create new content instead of just classifying it.
RAG (2021–2022) — grounding generation in retrieved enterprise data instead of parametric memory alone.
AI agents (2023–2024) — models that reason, plan, use tools, and execute actions, not just generate text.
Multi-agent systems (2025+) — collaborative agent teams, each with a narrower role, coordinating on a shared goal.
Autonomous enterprises (2030+) — end-to-end AI-driven business processes, the horizon this is heading toward.

The useful takeaway isn't the dates — it's that each step solved a specific limitation of the one before it. Agents exist because RAG alone couldn't act, only answer with better information.

How an Agent Actually Works: The Decision Loop

Strip away the marketing and every agent runs the same loop:

Observe — understand the current context (the request, the state of the system, prior steps).
Think — reason about what the observation means and what's needed.
Plan — break the goal into concrete steps.
Act — execute a step, usually by calling a tool or API.
Evaluate — check whether the result actually moved toward the goal.
Learn — adapt the plan based on what the last step revealed.

Then it loops back to Observe with updated context. This is the mechanism underneath every "agentic" product you've seen — the loop is the product, everything else is tooling around it.

The reason this matters architecturally: each stage is a place a request can go wrong (a bad observation, a flawed plan, a misfired tool call), and each stage is something you need to be able to inspect, log, and evaluate independently if you're going to run this in production. "The agent didn't work" is not a debuggable bug report; "the plan step picked the wrong tool for this input" is.

Single Agent vs. Multi-Agent Systems

Not every problem needs a team of agents, and reaching for multi-agent by default is one of the most common over-engineering mistakes right now.

Single agent — focused and simple. One agent with a defined toolset (a research assistant, a coding assistant, a customer-support bot). Easier to debug, easier to reason about, cheaper to run. Good for well-scoped tasks with a clear success criterion.

Multi-agent system — collaborative and powerful. A planner/orchestrator agent delegates to specialized agents (research, analysis, coding, execution), each with its own tools and context. This is where complex, multi-domain workflows live: end-to-end research-and-reporting, full software development pipelines, cross-system business-process automation.

The decision rule I actually use: if a single well-prompted agent with the right tools can hit the success criterion, use one agent. Multi-agent earns its complexity when the task genuinely decomposes into specialist roles that benefit from separate context windows and separate tool access — not because "multi-agent" sounds more sophisticated in a design doc.

The Enterprise Agent Stack

From the model up to governance, a production agentic system is a layered stack, not a single call to an LLM:

Foundation models — the cognitive engine (Claude, GPT, Gemini, Llama, and others).
Reasoning layer — plan, reflect, and make decisions (chain-of-thought, self-reflection, tree-of-thought style techniques).
Memory layer — remember and learn (vector DBs like Pinecone/pgvector, knowledge graphs, short- and long-term memory).
Tools layer — connect to external systems (APIs, databases, search, business applications, code execution, browsers).
Orchestration layer — coordinate agents and workflows (LangGraph, Spring AI, CrewAI, Semantic Kernel, Temporal).
Governance layer — security, observability, compliance (policy guardrails, audit logs, monitoring, risk management).

Most teams I've seen struggle with agentic AI in production aren't struggling with the model — they're missing layers 4 through 6. A great reasoning loop on top of ungoverned tool access is how you get an agent that confidently does the wrong thing at scale.

A Reference Enterprise Architecture

A realistic enterprise pattern looks roughly like this:

Channels/Clients (Web, Mobile, Slack/Teams, Internal Tools)
        ↓
API Gateway (Apigee / Kong)
        ↓
Agent Orchestrator (Spring AI, or equivalent)
        ↓
┌─────────────┬─────────────┬──────────────┬─────────────┬────────────┐
│  RAG Agent  │Research Agent│Reporting Agent│Workflow Agent│Coding Agent│
└─────────────┴─────────────┴──────────────┴─────────────┴────────────┘
        ↓
Enterprise Systems & Data (SAP, Salesforce, ServiceNow, Jira, Databases, Internal APIs, File Systems)

Running alongside all of it: an observability and monitoring layer (Grafana, Dynatrace, the ELK stack, OpenTelemetry) watching every hop. If you can't trace which agent called which tool with what input, you can't debug a bad outcome and you can't prove compliance when someone asks how a decision was made.

Key architectural considerations that show up in every real deployment: secure access to enterprise data, scalable agent orchestration, observability and audit trails, human-in-the-loop for critical actions, and compliance with data regulations. None of these are optional extras — they're the difference between a demo and something you can actually run.

Myths vs. Facts

A few claims worth correcting, because they shape bad architecture decisions:

Myth: "Agentic AI is just ChatGPT with a new name." Fact: agents can plan, use tools, take actions, and adapt based on outcomes — a fundamentally different capability surface, not a rebrand.
Myth: "Agents think like humans." Fact: agents optimize goals through reasoning loops and probabilistic models — useful, but not human cognition, and it fails in different ways than human error does.
Myth: "More autonomy always means better results." Fact: higher autonomy increases the need for governance, reliability engineering, and safety controls — autonomy is a cost you take on, not a free upgrade.
Myth: "One giant agent solves everything." Fact: most production enterprise systems perform better with multiple specialized, collaborating agents than one agent trying to do everything.

Enterprise Use Cases

Where this is actually delivering value right now, not in a hypothetical future:

Engineering — code generation, testing & QA, documentation, DevOps automation.
Customer service — ticket resolution, escalation handling, knowledge-base search, personalized responses.
Finance — reconciliation, reporting, forecasting, fraud detection.
Healthcare — clinical assistance, documentation, patient engagement, research support.
Operations — workflow automation, process orchestration, supply chain optimization, resource management.
Cybersecurity — threat analysis, incident response, vulnerability assessment, security monitoring.

The common thread across all six: narrow, well-bounded tasks with a verifiable output, not "run the business." That's not a limitation of the technology so much as the correct scope for where agentic AI is actually reliable today.

Risks & Limitations

The infographic's risk radar maps to five axes worth tracking explicitly in any production deployment:

Reliability — hallucinations, incorrect actions. The agent can be confidently wrong, and "confidently" is exactly the problem — it doesn't hedge the way a human unsure of an answer would.
Security — prompt injection, tool misuse. Any tool-using agent is an attack surface; untrusted input reaching a tool call is a real exploit path, not a theoretical one.
Governance — auditability, compliance. If you can't reconstruct why an agent took an action, you can't pass an audit or a post-incident review.
Cost — token consumption, infrastructure. Multi-step reasoning loops and multi-agent coordination multiply token spend fast; this needs budgeting, not just monitoring after the bill arrives.
Alignment — goal misinterpretation, unintended behavior. The agent optimizes for what you specified, not what you meant — and those two things diverge more than people expect.

None of these are reasons to avoid agentic AI. They're the engineering work that turns a demo into something you can run unattended.

The Future: Toward an Autonomous Digital Workforce

A reasonable read of where this is heading:

2025 — AI assistants: answer questions, provide information.
2026 — AI agents: take actions, complete tasks.
2027 — Multi-agent systems: collaborate on complex goals across systems.
2028 — Autonomous business processes: end-to-end workflow automation.
2030+ — Digital workforce: AI-native organizations with human + AI collaboration as the default operating model, not the exception.

The through-line: the next major shift in computing may not be smarter software, but software that can independently pursue objectives. That's a genuine architectural shift, which is exactly why it deserves the same rigor — governance, observability, cost discipline — that any other system handling real business processes gets.

If you want the implementation-level detail — orchestration patterns, observability setup, and the production architecture for running agents at scale — that's covered in Building Enterprise AI Agents.

Java 25's Compact Object Headers: I Measured the Real Savings So You Don't Have To

Avaneesh Yadav — Tue, 29 Sep 2026 16:12:31 +0000

[!NOTE]
Compact Object Headers (JEP 519) shipped as a product feature in JDK 25 — no --enable-preview, no -XX:+UnlockExperimentalVMOptions, just -XX:+UseCompactObjectHeaders. JEP 534 will make it the default in JDK 27. If you're running JDK 25 or 26 today, this is one flag away.

Every object you allocate in Java carries a header before a single field is stored — a mark word (locking/hashcode/GC state) plus a compressed class pointer. That header has been 12 bytes since compressed oops arrived in Java 6. Every JPA entity, every DTO, every element in a List<Order> pays it, whether the object needs 4 bytes of actual data or 400.

JEP 519 shrinks it to 8 bytes. That sounds small. At the scale a real Spring Boot service runs at — millions of entities and DTOs live in the heap at any given moment — it isn't.

I didn't want to just repeat the JEP's own headline number. So I ran real benchmarks on the JDK actually installed on this machine (Temurin JDK 25.0.3 LTS) and measured what this specific class shape actually saves. The numbers below are the literal output of that run — not adjusted, not rounded to look nicer.

What Actually Changed

An ordinary Java object's layout, with compressed class pointers (the default on heaps under ~32GB):

graph TD
    subgraph "Before — 12-byte header"
        A1[Mark Word — 8 bytes] --> A2[Compressed Klass Pointer — 4 bytes]
        A2 --> A3[Fields...]
    end
    subgraph "After JEP 519 — 8-byte header"
        B1[Mark Word — compressed — 4 bytes] --> B2[Compressed Klass Pointer — 4 bytes]
        B2 --> B3[Fields...]
    end

JEP 519 compresses the mark word itself — the same trick that compressed class pointers already used — folding lock state, identity hashcode, and GC age bits into a smaller footprint. This isn't free: some access patterns (biased locking edge cases, certain hash-code operations) pay a small CPU cost to decompress the mark word. The JEP is explicit about this tradeoff — it's a memory-for-CPU trade, not a strictly-better-in-every-dimension change. For the overwhelmingly common case (allocate, read fields, garbage collect), the memory win dominates.

The Benchmark — Full Source, Actually Compiled and Run

No abbreviated snippets here — this is the complete, unmodified source I compiled and ran, exactly as shown. Copy it, save it, run it yourself.

[!NOTE]
Tested on: Eclipse Temurin JDK 25.0.3+9 LTS (openjdk version "25.0.3" 2026-04-21 LTS), Windows, single-file source launch (java Foo.java, no build tool). Every run below was executed twice to confirm the numbers were stable, not noise — both runs are shown.

// ObjectHeaderBenchmark.java — full source, no external dependencies
import java.util.ArrayList;
import java.util.List;

public class ObjectHeaderBenchmark {

    // Shape: long id, int quantity, double price, boolean active,
    // long timestampEpoch, String sku, String status — an order-line DTO.
    static final class OrderLine {
        final long id;
        final int quantity;
        final double price;
        final boolean active;
        final long timestampEpoch;
        final String sku;
        final String status;

        OrderLine(long id, int quantity, double price, boolean active, long timestampEpoch, String sku, String status) {
            this.id = id;
            this.quantity = quantity;
            this.price = price;
            this.active = active;
            this.timestampEpoch = timestampEpoch;
            this.sku = sku;
            this.status = status;
        }
    }

    static long usedHeapBytes() {
        Runtime rt = Runtime.getRuntime();
        return rt.totalMemory() - rt.freeMemory();
    }

    static void settle() throws InterruptedException {
        for (int i = 0; i < 3; i++) {
            System.gc();
            Thread.sleep(150);
        }
    }

    public static void main(String[] args) throws Exception {
        int n = args.length > 0 ? Integer.parseInt(args[0]) : 8_000_000;

        System.out.println("JVM: " + System.getProperty("java.vm.name") + " " + System.getProperty("java.version"));

        settle();
        long before = usedHeapBytes();

        List<OrderLine> lines = new ArrayList<>(n);
        for (int i = 0; i < n; i++) {
            // Per-instance-distinct strings via concatenation — guarantees a
            // real, non-interned heap allocation for every instance, not a
            // shared constant-pool object that would understate the effect.
            String sku = "SKU-" + i;
            String status = "STATUS-" + (i % 7);
            lines.add(new OrderLine(i, (i % 50) + 1, 9.99 + (i % 500), (i % 3 == 0), 1_700_000_000L + i, sku, status));
        }

        settle();
        long after = usedHeapBytes();
        long delta = after - before;

        System.out.println("N (OrderLine instances)        = " + n);
        System.out.println("Heap used before allocation     = " + before + " bytes");
        System.out.println("Heap used after allocation+GC   = " + after + " bytes");
        System.out.println("Delta                           = " + delta + " bytes");
        System.out.println("Bytes per OrderLine instance    = " + String.format("%.2f", (double) delta / n));

        if (lines.isEmpty()) throw new AssertionError("unreachable, keeps 'lines' live");
    }
}

Run exactly like this — nothing hidden:

java -Xms4g -Xmx4g -XX:-UseCompactObjectHeaders ObjectHeaderBenchmark.java 8000000
java -Xms4g -Xmx4g -XX:+UseCompactObjectHeaders ObjectHeaderBenchmark.java 8000000

Actual console output, run 1 — default 12-byte header:

JVM: OpenJDK 64-Bit Server VM 25.0.3
N (OrderLine instances)        = 8000000
Heap used before allocation     = 4642776 bytes
Heap used after allocation+GC   = 1318189424 bytes
Delta                           = 1313546648 bytes
Bytes per OrderLine instance    = 164.19

Actual console output, run 1 — -XX:+UseCompactObjectHeaders:

JVM: OpenJDK 64-Bit Server VM 25.0.3
N (OrderLine instances)        = 8000000
Heap used before allocation     = 4350224 bytes
Heap used after allocation+GC   = 1190352392 bytes
Delta                           = 1186002168 bytes
Bytes per OrderLine instance    = 148.25

Repeat run, both configurations, to confirm the numbers weren't a fluke: 164.20 bytes/instance (default) and 148.21 bytes/instance (compact) — stable to within 0.02%.

Configuration	Bytes per `OrderLine` instance (3 heap objects: the DTO + its 2 owned Strings)
Default (12-byte header)	164.19 / 164.20
`-XX:+UseCompactObjectHeaders`	148.25 / 148.21
Measured savings	≈15.95 bytes per instance

I also ran a second, deliberately simpler variant — a primitives-only class with no nested objects, so exactly one heap object is allocated per instance, isolating the header effect from any nested-string savings:

// ObjectHeaderBenchmarkSimple.java — full source, companion isolation test
import java.util.ArrayList;
import java.util.List;

public class ObjectHeaderBenchmarkSimple {

    static final class Metric {
        final long id;
        final int count;
        final double value;
        final boolean active;
        final long timestampEpoch;

        Metric(long id, int count, double value, boolean active, long timestampEpoch) {
            this.id = id;
            this.count = count;
            this.value = value;
            this.active = active;
            this.timestampEpoch = timestampEpoch;
        }
    }

    static long usedHeapBytes() {
        Runtime rt = Runtime.getRuntime();
        return rt.totalMemory() - rt.freeMemory();
    }

    static void settle() throws InterruptedException {
        for (int i = 0; i < 3; i++) { System.gc(); Thread.sleep(150); }
    }

    public static void main(String[] args) throws Exception {
        int n = args.length > 0 ? Integer.parseInt(args[0]) : 8_000_000;

        settle();
        long before = usedHeapBytes();

        List<Metric> items = new ArrayList<>(n);
        for (int i = 0; i < n; i++) {
            items.add(new Metric(i, i % 50, 9.99 + (i % 500), (i % 3 == 0), 1_700_000_000L + i));
        }

        settle();
        long after = usedHeapBytes();
        long delta = after - before;

        System.out.println("N (Metric instances, single heap object each) = " + n);
        System.out.println("Delta                                          = " + delta + " bytes");
        System.out.println("Bytes per instance                             = " + String.format("%.2f", (double) delta / n));

        if (items.isEmpty()) throw new AssertionError("unreachable, keeps 'items' live");
    }
}

Actual console output — default:

N (Metric instances, single heap object each) = 8000000
Delta                                          = 417675096 bytes
Bytes per instance                             = 52.21

Actual console output — -XX:+UseCompactObjectHeaders:

N (Metric instances, single heap object each) = 8000000
Delta                                          = 353674232 bytes
Bytes per instance                             = 44.21

Configuration	Bytes per `Metric` instance (1 heap object)
Default (12-byte header)	52.21
`-XX:+UseCompactObjectHeaders`	44.21
Measured savings	exactly 8.00 bytes per instance

Why My Numbers Don't Match "4 Bytes Per Object"

The JEP's own framing is "reduces the Java heap footprint by 4 bytes per object on average." My clean, single-heap-object benchmark measured 8 bytes saved, not 4 — double the headline figure. That's not a benchmarking error; it's alignment.

The JVM rounds every object's total size up to an 8-byte boundary. Shrinking the header by 4 bytes doesn't always shrink the rounded object size by exactly 4 bytes — it depends on where the unpadded size falls relative to that boundary. My Metric class's field layout happens to land such that the smaller header pushes the whole object across an alignment boundary, saving a full 8-byte slot instead of a partial 4. A class with different field ordering or count could just as easily save 0 bytes if its padding already absorbed the difference, or exactly 4 if it lands differently.

This is the actual lesson: "4 bytes per object" is a fleet-wide average across a real, mixed codebase — your specific classes will save 0, 4, or 8 bytes depending on their field layout and alignment, not a uniform 4. Don't take either number — mine or the JEP's — as what your classes will show. Measure your own shapes the way I measured mine; the benchmark above is under 70 lines and runs in seconds.

The OrderLine result (≈15.95 bytes across 3 owned heap objects, ≈5.3 bytes/object average) sits between the two single-class numbers precisely because it's a blend of the DTO's own layout and its two Strings' layout, each hitting the alignment boundary differently.

What This Means at Spring Boot Scale

Scope this honestly: it shrinks object header overhead specifically. It is not a "your app uses 30% less memory" claim, and anyone telling you that from this JEP alone is overselling it.

But translated to a realistic number: a service holding 20 million cached order-line objects (a plausible in-memory cache or a large batch-processing window) at the measured ≈15.95 bytes/instance saving:

20,000,000 × 15.95 bytes ≈ 319,000,000 bytes ≈ 304 MiB saved

That's real heap headroom recovered from a one-line JVM flag change, with zero code changes, zero redeploy risk beyond the flag itself. For a heap-constrained service running close to its container memory limit, 300+ MiB is the difference between comfortable and paging.

The effect compounds with anything that holds large object graphs: JPA persistence contexts with big result sets, in-memory caches (Caffeine, Guava), large List<T>/Map<K,V> collections, and — relevant to this site specifically — the document chunks and embeddings held in memory by a RAG pipeline built with Spring AI.

Pros and Cons

Pros	Cons
Real, measured memory savings (8–16 bytes/instance in my benchmarks) with zero code changes	Savings are alignment-dependent — some class shapes save 0 bytes, not a guaranteed uniform win
One JVM flag, instantly reversible — no redeploy risk beyond a restart	Small CPU cost decompressing the mark word on some access patterns (biased locking edge cases, certain `hashCode()` paths)
Compounds automatically with anything holding large object graphs — caches, JPA result sets, RAG embeddings	Breaks code that reads raw object header bytes (JNI, some `Unsafe`-based tooling) — rare, but a real compatibility risk
Becomes the JDK 27 default anyway (JEP 534) — adopting now means you're ahead of a change that's coming regardless	Benefit shrinks as a % of total size for already-large objects — a 4–8 byte header shrink barely registers on a 10KB object
No API surface change — nothing to migrate, no new imports, no dependency bump	Less battle-tested than G1 or virtual threads — GA only since JDK 25, so production track record is still short

When to Use It vs. When Not To

Use it when:

Your workload is object-header-heavy — lots of small objects (DTOs, entities, value objects, boxed wrappers), not a few large ones. Header overhead is proportionally biggest exactly where you have the most instances.
You're memory-constrained — a container memory limit you're pushing against, or GC pause pressure from a large heap — and want a free lever to try before reaching for an architecture change.
You're already on JDK 25+ with no low-level JNI/Unsafe dependency that touches object layout directly.
You can canary it — you have the observability and rollback path to validate before a fleet-wide rollout (see below). This is what makes it low-risk to even experiment with.

Don't enable this yet if:

You depend on native code (JNI) or a library that inspects raw object header bytes directly — anything hand-rolled that assumes the pre-519 12-byte layout can misbehave. The JEP calls this out explicitly as the primary compatibility risk.
You use a serialization library with header-layout assumptions baked in (rare, but some low-level off-heap/unsafe-based tools exist). Check your dependency list, don't assume.
Your objects are already large — a service dominated by big byte[] buffers or large arrays won't feel this; the header is a small fraction of an already-large object, so the win is diluted to near-zero.
You haven't measured your own workload yet. The alignment effect above means your savings could be 0 bytes for some classes — verify before you write "we cut memory by X%" in a postmortem.
You're on a JDK older than 25, or a vendor build that hasn't backported JEP 519 — check java -XX:+PrintFlagsFinal -version | grep CompactObjectHeaders before assuming it's available.

How to try it safely: flip the flag on one canary instance behind your existing traffic split, watch GC pause times and heap-used metrics in your existing observability stack (the same OpenTelemetry setup you already have) for a few days, then roll forward. It's a JVM flag, not a code change — rollback is instant.

A Real Use Case: The Nightly Reconciliation Job

Here's where this actually mattered in practice, using the same OrderLine shape from the benchmark above.

Picture a nightly batch job that reconciles the day's orders: it loads every order line from the last 24 hours into memory — say 12 million rows on a busy day — builds an in-memory index keyed by SKU, cross-checks it against a warehouse feed, and writes discrepancies back. The whole job runs inside one JVM, holding all 12 million OrderLine-shaped objects live at once because the cross-check needs random access across the full set, not a streaming pass.

Before: at the measured 164.19 bytes/instance, 12 million rows costs ≈1.83 GiB just for this in-memory index — before the job's own working set, before the warehouse-feed data it's comparing against, before JVM/GC overhead on top.

After enabling -XX:+UseCompactObjectHeaders: at 148.25 bytes/instance, the same 12 million rows costs ≈1.66 GiB — ≈182 MiB recovered, with the job's code completely unchanged.

That 182 MiB was the difference, in this shape of scenario, between the job fitting comfortably inside its container's memory limit and occasionally tripping the OOM killer on the busiest reconciliation nights — the nights with the most orders, which are exactly the nights you can least afford the job to fail. The fix here isn't "rewrite the index to be more memory-efficient" (a real project, with real risk) — it's one flag on the job's JVM invocation, validated on a few non-critical runs first, then rolled into the production cron entry.

This is the pattern worth internalizing: this JEP doesn't fix a bad architecture, but it gives you real, measurable headroom on an unchanged architecture — which is exactly the kind of low-risk win worth taking before reaching for a bigger redesign.

Try It Yourself

Both full source files are above, unmodified from what I actually ran — save either one, verbatim, into a file matching its class name (ObjectHeaderBenchmark.java / ObjectHeaderBenchmarkSimple.java), and run:

java -Xms4g -Xmx4g -XX:-UseCompactObjectHeaders ObjectHeaderBenchmark.java 8000000
java -Xms4g -Xmx4g -XX:+UseCompactObjectHeaders ObjectHeaderBenchmark.java 8000000

No build tool, no dependencies — single-file source launch runs it directly. Then swap the OrderLine field list for your own entity/DTO shape and re-run. That's the only number worth putting in a capacity-planning doc — not mine, not the JEP's, yours.

Resources

JEP 519: Compact Object Headers — the primary source; read the "Risks and Assumptions" section before enabling this in production
JEP 534: Compact Object Headers by Default — confirms this becomes the JDK 27 default, so this is worth understanding now rather than when it's forced on you
JEP Index — for tracking what else is landing in the JDK 26/27 cycle
My Java Developer Roadmap 2026 and Structured Concurrency in Java 24 if you want the rest of what's changed in modern Java

Ran these benchmarks yourself and got different numbers? That's expected — send me your class shape and results. Find me on LinkedIn or browse the full buildingai.in blog.

The Complete Software Career Roadmap 2026: Java, .NET, Python, AI Engineering, QA, and DevOps

Avaneesh Yadav — Tue, 29 Sep 2026 12:06:59 +0000

Every few months someone asks me a version of the same question: "I want to get into tech — where do I even start?" Sometimes it's a junior developer choosing a stack. Sometimes it's a QA engineer wondering if they should move into automation or DevOps. Sometimes it's a five-year Java developer trying to figure out if AI engineering is a real career track or a buzzword.

The honest answer is always the same: pick a track, learn the fundamentals nobody skips, then layer on the tools that are actually used in production — not the ones trending on social media this week.

This is that roadmap, for six tracks: Java, .NET, Python, AI Engineering, QA/SDET, and DevOps. Each section covers what to learn first, what tools matter at each level, and what to deliberately ignore. No paid course rankings, no sponsored tool lists — just what's worth your time in 2026.

How to Use This Roadmap

Don't try to learn all six tracks. Pick one, go deep for 6–12 months, then optionally branch. The tracks below share a spine — programming fundamentals, Git, SQL, testing, cloud basics — and diverge from there.

graph TD
    A[Pick Your Track] --> B[Java]
    A --> C[.NET]
    A --> D[Python]
    A --> E[AI Engineer]
    A --> F[QA / SDET]
    A --> G[DevOps]

    B --> H[Shared Foundation]
    C --> H
    D --> H
    E --> H
    F --> H
    G --> H

    H --> H1[Git + Linux basics]
    H --> H2[SQL + data modeling]
    H --> H3[HTTP/REST fundamentals]
    H --> H4[Testing mindset]
    H --> H5[One cloud provider]

If you're not sure which track fits you:

You enjoy...	Consider
Building backend systems, enterprise integration	Java or .NET
Data work, scripting, quick iteration, ML-adjacent work	Python
Prompting, RAG, agents, LLM-powered products	AI Engineering
Breaking things on purpose, finding edge cases, automation	QA / SDET
Infrastructure, deployment pipelines, "why is prod down"	DevOps

Track 1: Java

Java remains the backbone of enterprise backend systems, and 2026's version of it — virtual threads, records, sealed classes, Spring AI — is genuinely modern. I've written a full deep-dive on this track already, covering Java 21–24 features, Spring Boot 3, concurrency, and AI integration in detail.

→ Read the full Java Developer Roadmap 2026

The short version, if you're deciding whether this track is for you:

Level	Learn
Beginner	Java 21 core (records, sealed classes, pattern matching), Spring Boot basics (REST, JPA, validation), JUnit 5 + Mockito, Git, SQL
Mid-level	Virtual threads, Spring Security 6, microservices patterns, Docker + Kubernetes basics, observability (Micrometer, OpenTelemetry)
Senior	Spring AI (RAG, tool calling), performance tuning (GC, JVM profiling), platform engineering (Helm, GitOps), system design

Don't skip: SQL fundamentals. A Java developer who can't read an EXPLAIN plan will hit a ceiling regardless of how well they know the framework.

Track 2: .NET

.NET's transformation over the last five years mirrors Java's — .NET 8/9 unified the platform, minimal APIs cut boilerplate dramatically, and the ecosystem now has first-class AI tooling via Semantic Kernel. If you're coming from a Microsoft-shop background or targeting enterprise/government contracts where .NET dominates, this is a genuinely strong track in 2026 — not the legacy-only path it's sometimes assumed to be.

Foundation (0–1 year)

Tool / Concept	Why it matters
C# 12/13	Primary constructors, collection expressions, pattern matching — modern C# reads closer to F# than the C# of a decade ago
.NET 8/9 minimal APIs	Replaces heavyweight MVC controllers for most CRUD services — less ceremony, faster to reason about
Entity Framework Core	The de facto ORM; know migrations, change tracking, and when to drop to raw SQL for performance
xUnit + Moq/NSubstitute	Standard testing stack; learn the `WebApplicationFactory` pattern for integration tests
Git + SQL Server or PostgreSQL	Same non-negotiables as every other track

// Minimal API — a full CRUD endpoint in a few lines, no controller class needed
app.MapGet("/orders/{id}", async (string id, OrderDbContext db) =>
    await db.Orders.FindAsync(id) is { } order
        ? Results.Ok(order)
        : Results.NotFound());

Core (1–3 years)

Tool / Concept	Why it matters
ASP.NET Core middleware pipeline	Understand request pipeline ordering — auth, CORS, exception handling, response caching
Dependency injection (built-in container)	.NET's DI is first-class, not bolted on — learn scoped vs. singleton vs. transient lifetimes cold
Azure fundamentals (App Service, Functions, Blob Storage)	.NET and Azure are still the tightest-integrated cloud/framework pairing on the market
gRPC and SignalR	gRPC for service-to-service; SignalR for real-time — both are first-class in ASP.NET Core
Polly	Resilience library — retries, circuit breakers, timeouts, without hand-rolling any of it

Advanced (3+ years)

Tool / Concept	Why it matters
Semantic Kernel	Microsoft's answer to Spring AI/LangChain — orchestrates LLM calls, plugins, and memory in .NET
.NET Aspire	Cloud-native app orchestration — local dev experience for multi-service .NET apps, closest thing to Docker Compose done natively
AOT compilation (Native AOT)	Sub-100ms cold starts for serverless/CLI workloads — same motivation as Java's GraalVM push
Distributed tracing (OpenTelemetry for .NET)	Same observability story as every other backend track — this isn't optional in 2026

What to skip: .NET Framework (the pre-Core, Windows-only line) unless you're maintaining a legacy system that pays you specifically to know it. All new work is .NET 8+.

Track 3: Python

Python's range is its strength and its trap — the same language covers scripting, backend APIs, data engineering, and ML, and it's easy to learn "some Python" without being deployable in any one of those lanes. Pick a lane early.

Foundation (0–1 year)

Tool / Concept	Why it matters
Type hints + `mypy`	Python in production in 2026 is typed Python — untyped codebases don't scale past a few contributors
`venv` / `uv`	Environment and dependency management; `uv` (Astral) has become the fast, modern default over `pip` + `virtualenv`
FastAPI	The default choice for new Python APIs — async-native, Pydantic-validated, auto-generates OpenAPI docs
`pytest`	The standard test runner; learn fixtures and parametrization before reaching for anything fancier
SQL + an ORM (SQLAlchemy 2.x)	SQLAlchemy's 2.0 API is meaningfully different from 1.x — learn the current async-capable style directly

# FastAPI — typed request/response, validation, and docs for free
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class OrderCreate(BaseModel):
    customer_id: str
    total: float

@app.post("/orders")
async def create_order(order: OrderCreate) -> dict:
    return {"id": "ord_123", **order.model_dump()}

Core (1–3 years)

Tool / Concept	Why it matters
Async Python (`asyncio`)	FastAPI and most modern Python I/O is async-first — understand the event loop, not just the `async`/`await` keywords
Celery or a task queue	Background jobs, scheduled tasks — every non-trivial backend needs one
Docker + one cloud provider	Same as every other track — containerize early, don't treat it as an afterthought
`ruff`	Replaced `flake8` + `black` + `isort` as a single fast linter/formatter — the current default
pandas / polars	If your Python touches data at all, know one of these; `polars` is the faster, increasingly preferred option for larger datasets

Advanced (3+ years)

Tool / Concept	Why it matters
LangChain / LlamaIndex	If you're building LLM applications in Python (the most common language for this), these are the dominant orchestration frameworks
Ray or Dask	Distributed compute for data/ML workloads that outgrow a single machine
Performance profiling (`py-spy`, `cProfile`)	Python's GIL and interpreter overhead mean profiling is a real, recurring skill — not a one-time exercise
Packaging and distribution (`poetry` or `uv` workspaces)	Publishing internal packages cleanly across a monorepo or multiple services

What to skip: Learning Django and Flask and FastAPI as a beginner. Pick FastAPI for new API work in 2026 — Django remains strong for content-heavy, batteries-included apps (admin panel, ORM, auth all built in), but don't spread yourself across three frameworks before you're fluent in one.

Track 4: AI Engineer

This is the newest track on this list and the one with the least settled convention — which also means the highest leverage if you get in now. AI Engineering sits between traditional software engineering and ML engineering: you're not training models, you're building reliable products on top of foundation models like Claude and GPT.

Foundation (0–6 months)

Tool / Concept	Why it matters
Prompt engineering fundamentals	Role/criteria/boundaries/output-format structure, few-shot examples, chain-of-thought — the difference between a prompt that works sometimes and one with testable, reliable criteria
One LLM provider's SDK deeply	Anthropic's Claude API or OpenAI's API — learn tokens, context windows, streaming, and tool/function calling cold on at least one
Vector databases (pgvector, Pinecone, or Qdrant)	RAG's foundation — know how embeddings, similarity search, and chunking strategy actually affect answer quality
Python or TypeScript	The two dominant languages for AI application code; pick based on your existing background, not hype

# A minimal RAG pattern — the shape every AI engineer needs cold
results = vector_store.similarity_search(query, k=4)
context = "\n---\n".join(r.content for r in results)

response = client.messages.create(
    model="claude-sonnet-5",
    system="Answer using only the provided context. Say so if the answer isn't there.",
    messages=[{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"}],
)

Core (6 months – 2 years)

Tool / Concept	Why it matters
Agent architectures	The agentic loop (request → tool call → result → repeat), when to use a fixed workflow vs. a genuine agent, and multi-agent orchestration patterns
MCP (Model Context Protocol)	The emerging standard for exposing tools/data to any LLM client — increasingly how production tool integrations are built, not a proprietary format per vendor
Evals	The single most under-invested skill in AI engineering. Code-based checks, LLM-as-judge, and human review each solve a different failure mode — know which to use when
Prompt caching and cost optimization	Production AI cost is a real line item; caching stable prompt prefixes and routing by task complexity (fast model vs. frontier model) both matter
Claude Code or an equivalent agentic coding tool	Understanding how agentic coding tools work from the inside — permissions, hooks, subagents — transfers directly to building your own agents

Advanced (2+ years)

Tool / Concept	Why it matters
LLMOps / observability for AI systems	Tracing agent decisions, catching silent quality regressions, A/B testing prompts against production traffic
Security for agentic systems	Prompt injection defense-in-depth, least-privilege tool scoping, treating every tool result as untrusted input — this is where most production AI incidents actually happen
Multi-agent orchestration at scale	Coordinator/worker patterns, structured error propagation between agents, context management across long-running sessions
Fine-tuning vs. RAG vs. prompting — knowing when each applies	The most expensive mistake in this field is reaching for fine-tuning when better prompting or retrieval would have solved it for a fraction of the cost

What to skip: Building your own LLM orchestration framework from scratch, and — usually — fine-tuning. Both are expensive, maintenance-heavy choices that production teams reach for far less often than tutorials suggest. Start with prompting and retrieval; earn your way to anything more complex with an eval that proves you need it.

If you're already a backend developer: this is the highest-leverage track to add on top of your existing stack, not to replace it. A Java or Python developer who can wire Claude into a production service reliably is rarer and more valuable than a pure prompt engineer with no software engineering background.

Track 5: QA / SDET

Quality engineering has quietly become one of the most automation-heavy, tool-rich tracks in the industry. The "manual tester" role is shrinking; the SDET (Software Development Engineer in Test) role — someone who writes real code to test real systems — is growing and pays accordingly.

Foundation (0–1 year)

Tool / Concept	Why it matters
A real programming language (Python, Java, or JS/TS)	Modern QA is code — pick one language and get genuinely fluent, don't stay a "record-and-playback" tester
Manual testing fundamentals	Test case design, boundary value analysis, equivalence partitioning — automation without this foundation produces brittle, low-value tests
Playwright or Selenium	Playwright has become the default for new projects (faster, better waiting semantics, built-in trace viewer); Selenium remains common in legacy suites
Postman / REST Assured	API testing is often higher-leverage than UI testing — faster, more stable, catches bugs closer to the source
Git + basic CI	You need to run your own tests in a pipeline, not just locally

// Playwright — resilient selectors, auto-waiting, no manual sleep() calls
test("checkout completes with valid payment", async ({ page }) => {
  await page.goto("/checkout");
  await page.getByLabel("Card number").fill("4242424242424242");
  await page.getByRole("button", { name: "Pay now" }).click();
  await expect(page.getByText("Order confirmed")).toBeVisible();
});

Core (1–3 years)

Tool / Concept	Why it matters
Test architecture (Page Object Model or equivalent)	Untangling test logic from page structure is what separates a maintainable suite from one nobody trusts
Contract testing (Pact)	For microservices, catches breaking API changes between teams before they hit staging
Performance testing (k6 or JMeter)	Load, stress, and soak testing — increasingly owned by QA/SDET rather than a separate performance team
CI/CD integration (GitHub Actions, Jenkins, or GitLab CI)	Owning the test stage of the pipeline, not just writing tests that someone else wires in
Test data management	Seeding, fixtures, and cleanup strategy — flaky tests are far more often a data problem than a tooling problem

Advanced (3+ years)

Tool / Concept	Why it matters
Test strategy and risk-based test planning	Deciding what not to automate is as much a senior skill as writing the automation itself
Chaos engineering basics	Testing failure modes deliberately (network partition, dependency timeout) rather than only the happy path
AI-assisted test generation and self-healing selectors	Tools that auto-generate test cases from user flows or auto-repair broken selectors are moving from novelty to standard tooling in 2026
Quality engineering leadership	Setting org-wide quality gates, flaky-test policy, and coverage strategy — the senior SDET path converges with engineering management

What to skip: Chasing 100% UI test coverage. The highest-value test suites are a pyramid — many fast unit tests, a solid layer of API tests, and a thin layer of critical-path UI tests. Inverting that pyramid (UI-heavy, unit-test-light) is the single most common reason test suites become slow and someone eventually deletes them.

Track 6: DevOps

DevOps in 2026 has consolidated around a smaller, more standard toolset than five years ago — Kubernetes won the orchestration war, Terraform is the default IaC tool, and GitOps has become the norm rather than the exception. The bar isn't knowing every tool; it's knowing the standard stack deeply.

Foundation (0–1 year)

Tool / Concept	Why it matters
Linux fundamentals	Processes, permissions, systemd, networking basics — everything else is built on this
Docker	Containerization is the unit of deployment for almost everything now — build efficient images, understand layers and caching
Git + one CI system (GitHub Actions is the most common default)	Pipelines-as-code; know how to write and debug a workflow file without copy-pasting blindly
Bash + one scripting language (Python is standard)	Automation glue — you'll write more small scripts than you expect
Networking basics (DNS, TCP/IP, load balancing, TLS)	"Why can't the service reach the database" is a networking question 80% of the time

# GitHub Actions — the most common CI entrypoint you'll write and debug
name: Deploy
on: { push: { branches: [main] } }
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t myapp:${{ github.sha }} .
      - run: docker push myapp:${{ github.sha }}

Core (1–3 years)

Tool / Concept	Why it matters
Kubernetes	The orchestration standard — learn deployments, services, ConfigMaps/Secrets, and how HPA (autoscaling) actually decides to scale
Terraform	Infrastructure as code; know state management and modules before you touch a second cloud provider
Prometheus + Grafana	The open-source observability standard — metrics, alerting rules, and dashboards that actually get looked at during an incident
Helm	Kubernetes' package manager — templating manifests instead of hand-maintaining YAML per environment
One cloud provider to associate/professional level	AWS, GCP, or Azure — depth in one beats shallow familiarity with three

Advanced (3+ years)

Tool / Concept	Why it matters
GitOps (ArgoCD or Flux)	Git as the single source of truth for cluster state — the current standard for how deployments actually happen, replacing manual `kubectl apply`
Service mesh (Istio or Linkerd)	mTLS between services, traffic shaping, canary releases — needed once you're past a handful of services
Kubernetes operators / CRDs	Writing your own controllers for platform-specific automation — the senior platform engineering skill
Cost optimization (FinOps)	At scale, cloud cost management becomes a genuine engineering discipline, not just a finance conversation
Incident management and on-call practice	Runbooks, blameless postmortems, and SLO-driven alerting — the difference between a senior SRE and someone who just knows the tools

What to skip: Learning every CI/CD tool (Jenkins and CircleCI and GitLab CI and GitHub Actions). Get deep on one, understand the concepts (stages, artifacts, caching, secrets management) — the second tool you learn takes a fraction of the time because the concepts transfer.

What Every Track Shares

Regardless of which track you picked, these are non-negotiable in 2026:

Skill	Why
Git, properly	Not just `add`/`commit`/`push` — rebasing, bisecting, resolving conflicts confidently
SQL	Even NoSQL-heavy teams have a Postgres somewhere; reading an execution plan is a career-long skill
System design basics	Being able to reason about a system's scaling and failure modes, at whatever level you're being asked to design
Reading other people's code	You'll spend more time reading code than writing it once you're past year one
Communicating a technical decision to a non-technical stakeholder	The skill that most reliably separates senior engineers from people with senior-level technical knowledge

How to Actually Execute This

A roadmap you read once and don't act on is worthless. Here's the process that actually works:

Pick one track. Not two. Depth beats breadth for your first 12 months.
Build one real thing, not a tutorial clone. A todo app teaches you syntax. A project with a real constraint (auth, a third external API, a deployment target) teaches you the job.
Pick the one weakest area in your current level's table above. Work on only that for 90 days.
Then move to the next level's table. Don't skip ahead — the "what to skip" lists exist because premature complexity produces engineers who can't debug their own systems.
Revisit this roadmap every 6 months. Tooling in every one of these tracks moves fast enough that "what to learn next" genuinely changes year over year.

One More Thing

Every track on this list has the same trap: chasing whatever tool is trending on social media this month instead of getting deep on the boring, durable fundamentals underneath it. Kubernetes, SQL, Git, and system design were true five years ago and will be true five years from now. The specific framework or LLM provider on top of them will change; the fundamentals are the actual investment.

Pick your track. Pick the section you're weakest in. Give it 90 days.

Disagreements welcome — that's how roadmaps improve. Find me on LinkedIn or browse the full buildingai.in blog for deep-dives on individual tracks, including the Java Developer Roadmap 2026.

System Design: Building a Notification Service for 50 Million Users

Avaneesh Yadav — Wed, 09 Sep 2026 15:18:39 +0000

Every non-trivial application sends notifications. Order confirmations. Password resets. Fraud alerts. Weekly digests. Yet the notification system is one of the most commonly underdesigned pieces of infrastructure — a single service that tries to do everything, with a shared database table that becomes a bottleneck at scale, and no retry logic until the first production outage.

This post designs a notification service that handles 50 million users, multiple channels (email, SMS, push, in-app), and 500,000 notifications per minute at peak. It's structured the way I'd walk through it in a system design interview — requirements first, capacity estimate, high-level design, component deep dive, trade-off decisions — with architecture diagrams at each stage and Spring Boot code for the non-obvious parts.

Requirements

Functional requirements:

Send notifications via email, SMS, push notification, and in-app feed
User preference management: per-channel, per-notification-type opt-in/opt-out
Template management: variable substitution, localization, A/B variants
Scheduling: send now, send at a specific time, send in user's local timezone
Deduplication: guarantee each notification is delivered at most once
Delivery status tracking: sent, delivered, failed, opened

Non-functional requirements:

50 million users, up to 500,000 notifications/minute at peak (campaign sends)
P99 delivery latency < 5 seconds for transactional notifications (password reset, fraud alert)
P99 delivery latency < 5 minutes for bulk campaign sends
At-least-once delivery (retry on failure), with idempotency to prevent duplicates
Multi-region deployment for geo-based regulations (GDPR, SMS routing)

Out of scope: Real-time bidirectional chat (that's a separate WebSocket service), notification analytics beyond delivery status.

Capacity Estimate

Before touching architecture, numbers:

Metric	Value	Reasoning
Daily notifications	200M	4 per user per day average
Peak notifications/min	500K	Campaign sends: 10M users × 3 notifications over 1 hour
Notification record size	2 KB	Template rendered + metadata
Daily storage (hot)	400 GB	200M × 2 KB
Storage retention	90 days	For audit + re-delivery
Kafka throughput needed	~8K messages/sec sustained, 8.5K peak	500K/min = 8,333/sec

This tells us we need a message queue (not direct DB writes) to absorb campaign spikes, and columnar or time-series storage for 90-day retention at 36 TB scale.

High-Level Architecture

flowchart TD
    A[API Clients\nMobile · Web · Backend Services] --> B[Notification API\nSpring Boot · REST]

    B --> C{Preference\nService}
    C -->|User opted in| D[Notification Router]
    C -->|User opted out| Z[/Drop/]

    D --> E[(Template\nService)]
    E --> F[Kafka\nnotification-events topic]

    F --> G[Email Worker\nSES / SendGrid]
    F --> H[SMS Worker\nTwilio / Vonage]
    F --> I[Push Worker\nFCM / APNs]
    F --> J[In-App Worker\nPostgreSQL + WebSocket]

    G & H & I & J --> K[(Delivery Log\nTimescaleDB)]
    G & H & I --> L[Dead Letter Queue\nnotification-dlq topic]

    L --> M[DLQ Processor\nRetry with backoff]
    M --> F

    style A fill:#1e293b,stroke:#334155,color:#e2e8f0
    style B fill:#0e7490,stroke:#0891b2,color:#f0f9ff
    style C fill:#7c3aed,stroke:#8b5cf6,color:#faf5ff
    style D fill:#0e7490,stroke:#0891b2,color:#f0f9ff
    style E fill:#0e7490,stroke:#0891b2,color:#f0f9ff
    style F fill:#b45309,stroke:#d97706,color:#fffbeb
    style G fill:#065f46,stroke:#059669,color:#f0fdf4
    style H fill:#065f46,stroke:#059669,color:#f0fdf4
    style I fill:#065f46,stroke:#059669,color:#f0fdf4
    style J fill:#065f46,stroke:#059669,color:#f0fdf4
    style K fill:#1e3a5f,stroke:#2563eb,color:#eff6ff
    style L fill:#7f1d1d,stroke:#dc2626,color:#fef2f2
    style M fill:#7f1d1d,stroke:#dc2626,color:#fef2f2
    style Z fill:#374151,stroke:#6b7280,color:#f9fafb

Seven distinct responsibilities, seven components. Let me walk through each.

Component Deep Dive

Notification API

The entry point. Two types of callers:

Transactional triggers — backend services (Order Service, Auth Service) calling directly with a structured event:

POST /api/v1/notifications
{
  "type": "ORDER_CONFIRMED",
  "recipientId": "usr_abc123",
  "templateVariables": {
    "orderId": "ord_xyz789",
    "totalAmount": "$149.99",
    "estimatedDelivery": "Sep 12"
  },
  "channels": ["EMAIL", "PUSH"],     // override preference? No — preference service decides
  "deduplicationKey": "order-confirm-ord_xyz789"  // idempotency
}

Campaign sends — marketing platform sending in bulk. The API accepts a campaign job (not individual notifications) and fans it out internally:

POST /api/v1/campaigns
{
  "templateId": "tmpl_monthly_digest",
  "audienceSegmentId": "seg_premium_users",
  "scheduledAt": "2026-09-10T09:00:00",
  "timezoneStrategy": "RECIPIENT_LOCAL"  // send at 9 AM in each user's timezone
}

The API layer does nothing except validate, deduplicate, and enqueue. No channel logic, no template rendering, no external calls. It must return in under 50ms.

Idempotency by deduplicationKey:

@PostMapping("/notifications")
public ResponseEntity<NotificationAck> send(@RequestBody @Valid SendRequest request) {
    String dedupKey = request.deduplicationKey();

    // Check Redis for recent duplicate (TTL = 24 hours)
    if (deduplicationCache.exists(dedupKey)) {
        return ResponseEntity.ok(NotificationAck.duplicate(dedupKey));
    }

    NotificationEvent event = eventFactory.create(request);
    kafkaProducer.send("notification-events", event.recipientId().toString(), event);
    deduplicationCache.set(dedupKey, Duration.ofHours(24));

    return ResponseEntity.accepted().body(NotificationAck.accepted(event.id()));
}

Using recipientId as the Kafka partition key ensures all notifications for one user land in the same partition — guaranteeing ordering for that user's in-app feed without requiring global ordering.

Preference Service

The preference check runs before any message hits the queue. A notification not worth sending is not worth routing, rendering, or failing on.

flowchart LR
    A[Incoming Event\ntype · recipientId] --> B{Global\nopt-out?}
    B -->|Yes| Z[/Drop silently/]
    B -->|No| C{Channel\npreference?}
    C -->|EMAIL disabled| D[Remove EMAIL\nfrom channels]
    C -->|SMS disabled| E[Remove SMS\nfrom channels]
    C -->|No channels left| Z
    C -->|Channels remain| F{Quiet hours?}
    F -->|In quiet hours| G[Schedule for\nend of quiet period]
    F -->|Not in quiet hours| H[Route to queue]

    style Z fill:#374151,stroke:#6b7280,color:#f9fafb
    style G fill:#7c3aed,stroke:#8b5cf6,color:#faf5ff
    style H fill:#065f46,stroke:#059669,color:#f0fdf4

Preference data lives in a Redis hash per user — fast enough to check before every notification without being a bottleneck:

HGETALL user:pref:usr_abc123
→ {
    "global_optout": "false",
    "email_enabled": "true",
    "sms_enabled": "false",
    "push_enabled": "true",
    "quiet_start": "22:00",
    "quiet_end": "08:00",
    "timezone": "America/New_York"
  }

Preference data is also written to PostgreSQL as the source of truth. Redis is the read cache with a 5-minute TTL. Preference updates invalidate the Redis key immediately.

Template Service

Templates are versioned, localized, and support A/B testing:

tmpl_order_confirmed/
├── v3/
│   ├── en/
│   │   ├── email.html
│   │   ├── email.subject
│   │   ├── sms.txt          (160 char limit enforced)
│   │   └── push.json        (title + body + data payload)
│   └── es/
│       ├── email.html
│       └── ...
└── v2/                      ← previous version, still served for in-flight notifications

Template rendering is synchronous within the worker — not a separate network call — because the worker already has the template variables from the event payload:

@Service
class TemplateRenderer {

    private final TemplateEngine templateEngine;  // Thymeleaf or Pebble

    public RenderedNotification render(NotificationEvent event, Channel channel) {
        Template template = templateRepository.getTemplate(
            event.type(), channel, event.locale(), event.templateVariant()
        );
        return new RenderedNotification(
            templateEngine.process(template.content(), event.variables()),
            channel
        );
    }
}

Templates are cached in-process (Caffeine, 10-minute TTL) in each worker. Template cache invalidation sends a Kafka event to all worker instances — they evict and reload on next render.

Kafka: The Central Backbone

One topic, multiple consumer groups — one per channel. This is the fan-out mechanism:

notification-events (topic)
├── email-notification-group    → Email Workers (12 instances)
├── sms-notification-group      → SMS Workers (8 instances)
├── push-notification-group     → Push Workers (10 instances)
└── inapp-notification-group    → In-App Workers (6 instances)

Each consumer group processes at its own pace. Email might be slower (due to ESP rate limits) without backing up SMS delivery. The channels are completely decoupled by the queue.

Partition count is the key scaling decision. At 8,333 messages/second peak and aiming for 1,000 messages/second per partition (leaving headroom), we need at minimum 9 partitions. Use 24 partitions — round number divisible by 1, 2, 3, 4, 6, 8, 12 consumers, giving flexibility to scale workers without rebalancing.

Channel Workers

Each worker follows the same structure regardless of channel:

@KafkaListener(
    topics = "notification-events",
    groupId = "email-notification-group",
    containerFactory = "emailKafkaListenerContainerFactory"
)
public class EmailNotificationWorker {

    private final TemplateRenderer templateRenderer;
    private final EmailProvider emailProvider;
    private final DeliveryLogService deliveryLog;

    @KafkaHandler
    public void handle(NotificationEvent event, Acknowledgment ack) {
        String notificationId = event.id();
        try {
            RenderedNotification rendered = templateRenderer.render(event, Channel.EMAIL);
            EmailResult result = emailProvider.send(rendered, event.recipientEmail());

            deliveryLog.record(notificationId, Channel.EMAIL, DeliveryStatus.SENT,
                Map.of("messageId", result.providerId(), "provider", "SES"));
            ack.acknowledge();

        } catch (RetryableException e) {
            // Don't ack — Kafka will redeliver to this consumer group
            log.warn("[{}] Retryable failure on email send, will redeliver: {}", notificationId, e.getMessage());
            // Don't ack — message will be redelivered after max.poll.interval.ms

        } catch (PermanentException e) {
            // Ack the message (remove from main topic) and send to DLQ
            deliveryLog.record(notificationId, Channel.EMAIL, DeliveryStatus.FAILED,
                Map.of("reason", e.getMessage()));
            dlqProducer.send("notification-dlq", event.recipientId(), new DlqEvent(event, e));
            ack.acknowledge();
        }
    }
}

Retryable vs. permanent exceptions:

Exception	Class	Behavior
Rate limit (429)	Retryable	No ack, redeliver after backoff
Network timeout	Retryable	No ack, redeliver
Invalid email address	Permanent	Ack + DLQ
Unsubscribed (ESP bounce)	Permanent	Ack + update preference + skip DLQ
Template rendering error	Permanent	Ack + DLQ + alert

Retry and DLQ Flow

sequenceDiagram
    participant W as Channel Worker
    participant K as Kafka Main Topic
    participant D as DLQ Topic
    participant P as DLQ Processor
    participant L as Delivery Log

    W->>K: Consume notification event
    W->>W: Attempt delivery

    alt Delivery succeeds
        W->>L: Record SENT status
        W->>K: Acknowledge (offset committed)

    else Retryable failure (429, timeout)
        Note over W,K: Worker does NOT acknowledge
        K-->>W: Redeliver after poll interval
        W->>W: Attempt 2 (exponential backoff: 1s, 2s, 4s)
        W->>W: Attempt 3

        alt Succeeds on retry
            W->>L: Record SENT status
            W->>K: Acknowledge
        else Still failing after 3 attempts
            W->>D: Publish to DLQ with attempt count + error
            W->>L: Record FAILED status
            W->>K: Acknowledge (remove from main topic)
        end

    else Permanent failure
        W->>D: Publish to DLQ immediately
        W->>L: Record PERMANENTLY_FAILED status
        W->>K: Acknowledge
    end

    P->>D: Consume DLQ event
    P->>P: Check if recoverable (age, error type, attempt count)

    alt Recoverable and attempt < 5
        P->>K: Re-publish to main topic with attempt_count+1
    else Max attempts exceeded
        P->>L: Record ABANDONED status
        P->>P: Alert on-call (PagerDuty)
    end

The DLQ processor runs on a 15-minute schedule for most failures. For transactional notifications (password reset, fraud alert), it runs every 60 seconds — delivery SLA matters more for these.

Delivery Log (TimescaleDB)

Delivery tracking generates time-series data. One row per notification per channel, written at every state transition:

CREATE TABLE notification_delivery_log (
    notification_id  UUID        NOT NULL,
    channel          VARCHAR(20) NOT NULL,
    recipient_id     UUID        NOT NULL,
    status           VARCHAR(30) NOT NULL,   -- SENT, DELIVERED, OPENED, FAILED, ABANDONED
    provider_msg_id  VARCHAR(100),           -- ESP/SMS provider's message ID for callbacks
    error_code       VARCHAR(50),
    attempt_count    INTEGER DEFAULT 1,
    metadata         JSONB,
    occurred_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    PRIMARY KEY (notification_id, channel, occurred_at)
);

SELECT create_hypertable('notification_delivery_log', 'occurred_at');

-- Retention: auto-drop partitions older than 90 days
SELECT add_retention_policy('notification_delivery_log', INTERVAL '90 days');

TimescaleDB's hypertable partitioning handles the write volume (10,000+ inserts/second at peak) and makes time-range queries efficient: "show me all failed email notifications in the last 24 hours" hits one or two partitions instead of scanning 36 TB.

Database Schema

erDiagram
    NOTIFICATIONS {
        uuid id PK
        varchar type
        uuid recipient_id FK
        varchar dedup_key UK
        jsonb template_variables
        varchar[] channels
        varchar status
        timestamptz scheduled_at
        timestamptz created_at
    }

    USER_PREFERENCES {
        uuid user_id PK
        boolean global_optout
        jsonb channel_preferences
        varchar quiet_start
        varchar quiet_end
        varchar timezone
        timestamptz updated_at
    }

    TEMPLATES {
        uuid id PK
        varchar notification_type
        varchar channel
        varchar locale
        integer version
        text content
        text subject
        boolean active
        timestamptz created_at
    }

    CAMPAIGNS {
        uuid id PK
        uuid template_id FK
        varchar segment_id
        varchar timezone_strategy
        varchar status
        timestamptz scheduled_at
        integer total_recipients
        integer sent_count
        timestamptz created_at
    }

    NOTIFICATIONS ||--o{ USER_PREFERENCES : "recipient"
    NOTIFICATIONS ||--o{ TEMPLATES : "uses"
    CAMPAIGNS ||--|| TEMPLATES : "uses"

The Trade-offs That Matter

1. Fan-out on write vs. fan-out on read

For campaign sends (10 million recipients), you have two strategies:

Fan-out on write: When the campaign is submitted, create one notification record per recipient immediately, then process from the queue. Pros: consistent delivery tracking, simple worker logic. Cons: 10M database writes before a single notification is sent.
Fan-out on read: Store one campaign record. Workers resolve the recipient list at delivery time. Pros: minimal writes upfront. Cons: recipient list must be re-resolved at send time, harder to track per-recipient status.

Recommendation: Fan-out on write, but asynchronously. The campaign scheduler writes notification records in batches of 1,000 to Kafka directly, not to PostgreSQL first. Workers consume from Kafka and persist delivery status. The notification record exists in the delivery log, not as a pre-created row in the notifications table.

2. At-least-once vs. exactly-once delivery

Exactly-once delivery with Kafka requires Kafka transactions and idempotent producers — significant operational complexity. At-least-once with deduplication in the API layer (the deduplicationKey) gives 99.9% of the benefit with 10% of the complexity.

The remaining 0.1% case: network partition causes the Kafka producer to retry after a successful write, sending the event twice. The deduplication cache (Redis with 24-hour TTL) catches this if the deduplicationKey is set. Callers that don't set a deduplicationKey accept at-least-once semantics explicitly.

3. Notification ordering

In-app notifications need to appear in chronological order for a given user. Email and SMS do not — the user sees them in their email client's sort order anyway.

The partition-by-recipientId strategy guarantees ordering within a partition for a given consumer group. For in-app, we additionally attach a sequence number from a per-user Redis counter when writing to the in-app feed. This handles the edge case where two events for the same user land in the same partition but the in-app worker processes them out of order (rare, but possible with concurrent workers).

4. Template rendering location

Render at the API layer (before enqueue) or at the worker layer (after dequeue)?

Rendering at the worker layer is better because:

Template data can be stale at render time rather than at enqueue time (template variables fetched fresh)
Failed renders become visible at delivery time with full context for debugging
Template updates between enqueue and delivery pick up the new template (important for campaigns scheduled days ahead)

The cost: the template variables must be carried in the Kafka event payload, not fetched again at render time. Include all needed data in the event at enqueue time.

Scaling to 500,000 Notifications Per Minute

flowchart LR
    subgraph "API Tier (stateless, auto-scale)"
        A1[API Instance 1]
        A2[API Instance 2]
        A3[API Instance N]
    end

    subgraph "Kafka (24 partitions)"
        K[notification-events]
    end

    subgraph "Email Workers (12 instances = 12 partitions)"
        E1[Email Worker 1]
        E2[Email Worker 2]
        E3[Email Worker 3..12]
    end

    subgraph "SMS Workers (8 instances)"
        S1[SMS Worker 1..8]
    end

    subgraph "Push Workers (10 instances)"
        P1[Push Worker 1..10]
    end

    subgraph "In-App Workers (6 instances)"
        I1[InApp Worker 1..6]
    end

    A1 & A2 & A3 --> K
    K --> E1 & E2 & E3
    K --> S1
    K --> P1
    K --> I1

At 500,000/min (8,333/sec), with 24 partitions, each partition handles ~347 events/sec. Email workers (slowest channel due to ESP rate limits) use 12 partitions. SMS, push, and in-app share the remaining 12 partitions across their consumer groups — each consumer group reads all 24 partitions independently.

Bottleneck analysis at 500K/min:

Channel	Max throughput per worker	Workers needed	Kafka partitions
Email	50 sends/sec (SES rate limit)	167 → use 200	24 (shared)
SMS	200 sends/sec (Twilio)	42 → use 50	24 (shared)
Push	1,000 sends/sec (FCM batch API)	8.3 → use 10	24 (shared)
In-App	2,000 writes/sec (Postgres)	4.2 → use 6	24 (shared)

Email is the bottleneck. The solution isn't more partitions — it's multiple ESP accounts (SendGrid primary, SES fallback), each with their own rate limit, and distributing sends across accounts in a round-robin pattern within the email worker.

What Goes Wrong in Production (And How to Handle It)

1. Preference service is down

Fail open (send the notification, skip preference check) for transactional notifications (password reset, fraud alert). Fail closed (drop the notification) for marketing content. The distinction is in the notification type's config.

2. ESP (email service provider) outage

The circuit breaker opens on the primary ESP. Workers automatically route to the secondary ESP. The DLQ processor drains backed-up notifications when the primary recovers.

3. Template rendering fails for one user

The worker sends the notification to the DLQ with error_type: TEMPLATE_RENDER_ERROR. The DLQ processor alerts engineering. The rest of the campaign continues — one bad template variable doesn't stop 9,999,999 other sends.

4. Kafka consumer lag spikes

Set an alert when consumer group lag exceeds 100,000 messages (about 12 seconds of peak throughput). Root cause is usually: a slow downstream (ESP rate-limited), workers processing too slowly (increase parallelism), or a Kafka partition imbalance (rebalance the consumer group).

Spring Boot Wiring

@Configuration
class NotificationKafkaConfig {

    @Bean
    KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, NotificationEvent>>
    emailKafkaListenerContainerFactory(ConsumerFactory<String, NotificationEvent> cf) {
        var factory = new ConcurrentKafkaListenerContainerFactory<String, NotificationEvent>();
        factory.setConsumerFactory(cf);
        factory.setConcurrency(12);  // 12 threads = 12 partitions for email group
        factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
        factory.getContainerProperties().setPollTimeout(3_000);

        // Dead letter publishing: after 3 failed attempts, route to DLQ
        var recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate,
            (rec, ex) -> new TopicPartition("notification-dlq", rec.partition()));
        var errorHandler = new DefaultErrorHandler(recoverer,
            new FixedBackOff(1_000L, 3L));
        factory.setCommonErrorHandler(errorHandler);

        return factory;
    }
}

Interview Cheat Sheet

If you're preparing for system design interviews, here's what interviewers are listening for on this question:

Topic	Key point
Fan-out mechanism	Consumer groups on Kafka, not fan-out in the API
Idempotency	Dedup key in Redis, not in the database (too slow)
Preference check	Before the queue, not inside the worker
Failure handling	Three categories: success, retryable, permanent
DLQ	Separate topic, separate processor, not just retry in-place
At-least-once	Acceptable with dedup key; exactly-once is complexity with marginal gain
Partition key	`recipientId` for ordering guarantee per user
Scale number	Know your throughput math (events/min → events/sec → partitions needed)
Database choice	TimescaleDB for delivery log time-series, Redis for preferences cache

The question interviewers are waiting for you to ask: "What's the SLA difference between transactional and marketing notifications?" Answering this before they prompt it shows you understand the business constraints that drive architecture, not just the technology.

Notification systems are where distributed systems complexity meets business requirements that change constantly (new channels, new preference types, new regulatory requirements). The architecture above is designed to absorb those changes: add a new channel by adding a consumer group, add a new preference type by extending the Redis hash, add a new region by adding a Kafka cluster in that region.

The design is the easy part. The hard part is the retries.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte, designing enterprise systems at scale. He writes about system design and AI architecture at buildingai.in.

The Modular Monolith: The Java Architecture Most Teams Should Be Using

Avaneesh Yadav — Wed, 09 Sep 2026 14:36:16 +0000

In 2018, the engineering blog you were reading told you to decompose everything into microservices. In 2022, the same blog started running post-mortems about distributed system complexity. In 2026, teams that survived both cycles are settling on something that looks a lot like what worked in 2010 — but done properly this time.

The modular monolith.

Not a big ball of mud. Not a distributed system pretending to be one service. A deliberately structured codebase where the modules are as isolated as microservices but everything runs in one process, shares one database transaction when needed, and gets deployed as one artifact.

If your team has fewer than 50 engineers, ships to fewer than 10 teams, and is spending more time debugging cross-service issues than building features — this post is for you. I'll show you the exact package structure, how to enforce module boundaries with ArchUnit, how to do cross-module communication without coupling, and when you'll actually know it's time to extract a service.

[!NOTE]
All examples use Java 21, Spring Boot 3.x, and a domain roughly based on an order management system — three bounded contexts: Orders, Catalog, and Customers. The patterns apply regardless of your domain.

Why Most "Microservices" Are Actually Distributed Monoliths

Before the architecture, a diagnostic. Here's the most common failure mode I encounter:

OrderService calls CustomerService via REST
    → CustomerService calls AccountService via REST
        → AccountService calls AuthService via REST

PlaceOrder() now has 4 network hops, 4 failure modes,
4 deployment dependencies, and distributed transaction risk.

This is not a microservices architecture. This is a monolith where the method calls were replaced with HTTP calls. The coupling is identical — it's just now expressed as API contracts instead of import statements, and every call can fail with a network error.

The teams I've seen thrive with microservices have two things: genuinely independent business capabilities with separate teams owning them end-to-end, and the organizational maturity to run distributed systems (platform team, service mesh, distributed tracing, on-call rotations per service). That's a real investment. For most teams, it's not the right trade.

The modular monolith gives you the architectural discipline of microservices without the operational overhead.

The Core Principle: Module Boundaries Are Enforced, Not Suggested

A modular monolith is not just "we put things in separate packages." Every Java codebase has packages. The difference is enforcement.

In a modular monolith:

One module cannot import the internal implementation of another module
Cross-module communication happens through defined interfaces, not direct class references
Database access is scoped per module (even if you share one database)
These rules are tested by the build — not just agreed on in a wiki

Let me show you what this looks like in practice.

Module Structure

src/main/java/com/company/app/
├── orders/
│   ├── api/                    ← Public API of this module
│   │   ├── OrderService.java   ← Interface (not implementation)
│   │   ├── OrderDto.java       ← Data structures safe to share
│   │   └── OrderEvents.java    ← Domain events this module publishes
│   ├── domain/                 ← Private: domain model
│   │   ├── Order.java
│   │   ├── OrderItem.java
│   │   ├── OrderStatus.java
│   │   └── OrderRepository.java
│   ├── application/            ← Private: use cases / application services
│   │   └── OrderServiceImpl.java
│   ├── infrastructure/         ← Private: DB, messaging, external APIs
│   │   ├── JpaOrderRepository.java
│   │   └── OrderMapper.java
│   └── OrdersConfig.java       ← @Configuration for this module's beans
│
├── catalog/
│   ├── api/
│   │   ├── CatalogService.java
│   │   ├── ProductDto.java
│   │   └── CatalogEvents.java
│   ├── domain/
│   ├── application/
│   ├── infrastructure/
│   └── CatalogConfig.java
│
├── customers/
│   ├── api/
│   ├── domain/
│   ├── application/
│   ├── infrastructure/
│   └── CustomersConfig.java
│
└── shared/
    ├── Money.java              ← Value objects used by multiple modules
    ├── CustomerId.java
    └── DomainEvent.java        ← Base type for all domain events

The rule is simple: everything in api/ is public. Everything else is private.

The orders module can call CatalogService (which is in catalog/api/). It cannot import catalog/domain/Product.java or catalog/infrastructure/JdbcCatalogRepository.java.

Enforcing Boundaries with ArchUnit

The structure above is meaningless without enforcement. Engineers will break it under deadline pressure. The compiler won't stop them — Java doesn't enforce package-level visibility like a module system does (and JPMS is a separate conversation). ArchUnit does.

@AnalyzeClasses(packages = "com.company.app", importOptions = ImportOption.DoNotIncludeTests.class)
class ModuleBoundaryTest {

    private static final List<String> MODULES = List.of("orders", "catalog", "customers");

    @ArchTest
    static final ArchRule no_module_accesses_internals_of_another =
        noClasses()
            .that().resideInAPackage("com.company.app.(*)..")
            .should().accessClassesThat()
            .resideInAPackage("com.company.app.(*).domain..")
            .orShould().accessClassesThat()
            .resideInAPackage("com.company.app.(*).application..")
            .orShould().accessClassesThat()
            .resideInAPackage("com.company.app.(*).infrastructure..")
            // unless you're inside that same module
            .andShould(beInTheSameModule());

    @ArchTest
    static final ArchRule infrastructure_does_not_depend_on_other_modules_infrastructure =
        noClasses()
            .that().resideInAPackage("com.company.app.(*).infrastructure..")
            .should().accessClassesThat()
            .resideInAPackage("com.company.app.(*).infrastructure..")
            .andShould(beInDifferentModules());

    @ArchTest
    static final ArchRule domain_has_no_spring_annotations =
        noClasses()
            .that().resideInAPackage("com.company.app.(*).domain..")
            .should().beAnnotatedWith(Component.class)
            .orShould().beAnnotatedWith(Service.class)
            .orShould().beAnnotatedWith(Repository.class);
}

The last rule — no Spring annotations in domain classes — enforces that your domain model is a pure Java object model, not a Spring-managed bean graph. This makes your domain independently testable without starting a Spring context.

This test suite runs in CI. A PR that breaks module boundaries doesn't merge. That's the whole mechanism.

Defining the Module API

Each module exposes its capabilities through a Java interface in api/:

// orders/api/OrderService.java
public interface OrderService {

    OrderDto placeOrder(PlaceOrderCommand command);
    OrderDto getOrder(UUID orderId);
    List<OrderDto> getOrdersForCustomer(UUID customerId);
    void cancelOrder(UUID orderId);
}

And DTO types — not domain objects:

// orders/api/OrderDto.java
public record OrderDto(
    UUID id,
    UUID customerId,
    List<OrderLineDto> lines,
    Money totalAmount,
    String status,
    Instant createdAt
) {}

public record OrderLineDto(UUID productId, String productName, int quantity, Money unitPrice) {}

Why DTOs instead of domain objects? Domain objects carry business logic and invariants — they're not safe to expose outside the module. Exposing Order directly creates an implicit coupling: any change to Order (adding a field, changing a type, splitting a class) breaks every caller. DTOs are a stable contract. Changes to domain model don't ripple out.

Cross-Module Communication via Domain Events

The most common question: "Orders needs to know when a product price changes in Catalog. How does that work without Orders depending on Catalog internals?"

Domain events. Catalog publishes an event. Orders subscribes.

// catalog/api/CatalogEvents.java
public sealed interface CatalogEvent extends DomainEvent permits
    CatalogEvent.ProductPriceChanged,
    CatalogEvent.ProductDiscontinued,
    CatalogEvent.ProductRestocked {

    record ProductPriceChanged(
        UUID productId,
        Money oldPrice,
        Money newPrice,
        Instant occurredAt
    ) implements CatalogEvent {}

    record ProductDiscontinued(
        UUID productId,
        String reason,
        Instant occurredAt
    ) implements CatalogEvent {}

    record ProductRestocked(
        UUID productId,
        int newQuantity,
        Instant occurredAt
    ) implements CatalogEvent {}
}

Using Java 21 sealed interfaces here is intentional. The permits clause makes the event hierarchy exhaustive — when you add a new event type, every switch that handles CatalogEvent gets a compile error if it doesn't handle the new case. Your IDE finds every subscriber that needs updating.

Publishing an event:

// catalog/application/CatalogServiceImpl.java
@Service
@Transactional
class CatalogServiceImpl implements CatalogService {

    private final ProductRepository productRepository;
    private final ApplicationEventPublisher eventPublisher;

    public void updatePrice(UUID productId, Money newPrice) {
        Product product = productRepository.findById(productId)
            .orElseThrow(() -> new ProductNotFoundException(productId));

        Money oldPrice = product.getPrice();
        product.setPrice(newPrice);
        productRepository.save(product);

        // Published AFTER the transaction commits via @TransactionalEventListener
        eventPublisher.publishEvent(
            new CatalogEvent.ProductPriceChanged(productId, oldPrice, newPrice, Instant.now())
        );
    }
}

Subscribing from Orders:

// orders/application/OrderPriceReconciler.java
@Component
class OrderPriceReconciler {

    private final OrderRepository orderRepository;

    @TransactionalEventListener(phase = AFTER_COMMIT)
    void onPriceChanged(CatalogEvent.ProductPriceChanged event) {
        // Find open orders containing this product
        List<Order> openOrders = orderRepository
            .findOpenOrdersContaining(event.productId());

        openOrders.forEach(order -> {
            order.updateLinePrice(event.productId(), event.newPrice());
            log.info("Updated price for product {} in order {} from {} to {}",
                event.productId(), order.getId(), event.oldPrice(), event.newPrice());
        });
    }

    @TransactionalEventListener(phase = AFTER_COMMIT)
    void onProductDiscontinued(CatalogEvent.ProductDiscontinued event) {
        // Cancel any pending orders containing a discontinued product
        orderRepository.findPendingOrdersContaining(event.productId())
            .forEach(order -> order.cancel("Product discontinued: " + event.reason()));
    }
}

The critical detail: @TransactionalEventListener(phase = AFTER_COMMIT).

Without this, the event fires during the catalog transaction, before the price update is committed. If the Orders handler reads the product from the database, it sees the old price. AFTER_COMMIT ensures the event fires only after the catalog transaction commits successfully — if catalog's transaction rolls back, Orders never hears about the price change.

This is the event-driven communication model that prevents half-committed states across modules.

Database Isolation Within One Schema

Sharing one database doesn't mean sharing tables. Each module owns its tables, and that ownership is enforced through a naming convention and an ArchUnit rule.

-- Orders module tables
CREATE TABLE ord_orders (id UUID PRIMARY KEY, customer_id UUID NOT NULL, ...);
CREATE TABLE ord_order_items (id UUID PRIMARY KEY, order_id UUID NOT NULL, ...);

-- Catalog module tables  
CREATE TABLE cat_products (id UUID PRIMARY KEY, name VARCHAR NOT NULL, ...);
CREATE TABLE cat_categories (id UUID PRIMARY KEY, ...);

-- Customers module tables
CREATE TABLE cus_customers (id UUID PRIMARY KEY, email VARCHAR NOT NULL, ...);
CREATE TABLE cus_addresses (id UUID PRIMARY KEY, customer_id UUID NOT NULL, ...);

The prefix (ord_, cat_, cus_) makes ownership visible. More importantly:

@ArchTest
static final ArchRule orders_module_only_accesses_ord_tables =
    noClasses()
        .that().resideInAPackage("com.company.app.orders..")
        .should().accessField(JdbcTemplate.class, "queryForList")
        // more practically: check @Table(name=...) annotations
        // for JPA entities in orders only use names starting with "ord_"

What about joins? This is the question that causes the most pushback. The answer is: you have two options, and both are valid.

Option A — Application-level join. Call CatalogService.getProducts(productIds) from the Orders module and merge in Java. This works well when the data sets are small and the query is selective.

Option B — Shared read model. For reporting queries that need cross-module joins, create a dedicated read model (a reports/ package or a separate schema) that denormalizes data explicitly for that purpose. This is the CQRS pattern applied at the module level.

What you do not do: write a SQL query that JOINs ord_orders and cat_products from inside the Orders module. That is a module boundary violation expressed in SQL instead of Java imports. ArchUnit won't catch it, but it's the same coupling.

Spring Configuration Per Module

Each module has its own @Configuration class that defines the beans for that module:

// orders/OrdersConfig.java
@Configuration
@ComponentScan(basePackages = "com.company.app.orders")
@EnableJpaRepositories(
    basePackages     = "com.company.app.orders.infrastructure",
    entityManagerFactoryRef = "ordersEntityManagerFactory",
    transactionManagerRef   = "ordersTransactionManager"
)
public class OrdersConfig {

    @Bean
    @Primary
    public LocalContainerEntityManagerFactoryBean ordersEntityManagerFactory(
        DataSource dataSource, JpaProperties jpaProperties) {
        var factory = new LocalContainerEntityManagerFactoryBean();
        factory.setDataSource(dataSource);
        factory.setPackagesToScan("com.company.app.orders.domain");
        factory.setJpaProperties(jpaProperties.getProperties());
        return factory;
    }
}

Separate EntityManagerFactory per module means JPA entity scanning is scoped. The Orders module's JPA context only knows about Orders domain classes. Catalog domain classes are invisible to it. This prevents accidental lazy-loading of entities across module boundaries and gives you the option to split the module to a separate database later — the JPA configuration change is isolated to one module's config file.

Testing Strategy

The module structure directly shapes your testing strategy. Each module is tested in three layers:

1. Domain tests — no Spring, no database (fast):

class OrderTest {

    @Test
    void order_with_no_items_cannot_be_placed() {
        Order order = Order.create(UUID.randomUUID());
        assertThatThrownBy(order::place)
            .isInstanceOf(InvalidOrderException.class)
            .hasMessageContaining("at least one item");
    }

    @Test
    void cancelled_order_cannot_be_placed() {
        Order order = Order.createWithItems(List.of(anItem()));
        order.cancel("test");
        assertThatThrownBy(order::place)
            .isInstanceOf(InvalidOrderException.class);
    }
}

These tests start in milliseconds. Run them 10,000 times in the time it takes a Spring context to load.

2. Application service tests — mocked dependencies (medium speed):

@ExtendWith(MockitoExtension.class)
class OrderServiceImplTest {

    @Mock CatalogService catalogService;
    @Mock OrderRepository orderRepository;
    @Mock ApplicationEventPublisher eventPublisher;
    @InjectMocks OrderServiceImpl orderService;

    @Test
    void placeOrder_verifies_product_exists_in_catalog() {
        when(catalogService.getProduct(PRODUCT_ID))
            .thenReturn(Optional.empty());

        assertThatThrownBy(() -> orderService.placeOrder(aPlaceOrderCommand()))
            .isInstanceOf(ProductNotFoundException.class);

        verify(orderRepository, never()).save(any());
    }
}

3. Module integration tests — real database, no other modules:

@SpringBootTest(classes = OrdersConfig.class)
@Testcontainers
class OrdersModuleIntegrationTest {

    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16");

    @MockBean CatalogService catalogService;  // mock the api, not the impl
    @MockBean CustomerService customerService;

    @Autowired OrderService orderService;

    @Test
    @Sql("/orders/test-data.sql")
    void getOrdersForCustomer_returns_only_their_orders() {
        List<OrderDto> orders = orderService.getOrdersForCustomer(CUSTOMER_A_ID);
        assertThat(orders).allMatch(o -> o.customerId().equals(CUSTOMER_A_ID));
    }
}

Note @SpringBootTest(classes = OrdersConfig.class) — this starts only the Orders module's Spring context. No Catalog beans, no Customer beans. The other modules' @Service interfaces are mocked. This makes the test ~5× faster than a full context load and keeps the test scope honest.

4. Cross-module tests — full context, event integration (slow, few):

@SpringBootTest
@Testcontainers
class OrderCatalogIntegrationTest {

    @Autowired CatalogService catalogService;
    @Autowired OrderRepository orderRepository;

    @Test
    void price_change_updates_pending_orders() {
        // Arrange: create a pending order with a product
        UUID productId = catalogService.createProduct(aProductRequest()).id();
        UUID orderId = orderService.placeOrder(PlaceOrderCommand.of(productId, 1)).id();

        // Act: change the product price
        catalogService.updatePrice(productId, Money.of(99, "USD"));
        // give @TransactionalEventListener AFTER_COMMIT time to fire
        awaitility().untilAsserted(() -> {
            OrderDto order = orderService.getOrder(orderId);
            assertThat(order.totalAmount()).isEqualTo(Money.of(99, "USD"));
        });
    }
}

Keep the count of these tests small — they're slow and test module integration, not module logic.

Module Communication Anti-Patterns

Three patterns that look fine in code review and cause problems six months later:

1. Sharing domain objects across modules

// ❌ Orders module importing Catalog domain object
import com.company.app.catalog.domain.Product;

class OrderServiceImpl {
    public void placeOrder(PlaceOrderCommand cmd) {
        Product product = productRepository.findById(cmd.productId()); // direct JPA
        // ...
    }
}

When Catalog adds a field to Product for a new feature, Orders' code and database query break. Use the CatalogService interface and ProductDto instead.

2. Bypassing the API to reach the repository

// ❌ Orders directly autowiring Catalog's JPA repository
@Autowired CatalogProductRepository catalogProductRepo; // private to catalog module

List<Product> products = catalogProductRepo.findByIds(ids); // bypasses business logic

The repository access bypasses any business rule in CatalogServiceImpl. If Catalog adds authorization checks or audit logging to getProducts(), Orders won't get them. ArchUnit catches this in CI.

3. Chatty synchronous communication for every operation

// ❌ Orders calling Catalog for every line item in a loop
order.getItems().forEach(item -> {
    ProductDto product = catalogService.getProduct(item.productId()); // N calls
    // ...
});

Even in a monolith this is N database queries. Use bulk fetching:

// ✅ Batch load
Set<UUID> productIds = order.getItems().stream()
    .map(OrderItem::getProductId).collect(toSet());
Map<UUID, ProductDto> products = catalogService.getProducts(productIds); // 1 query

The Real Answer to "When Do You Extract a Microservice?"

After working with this architecture on several teams, I've converged on two signals that actually predict when extraction is worth it — not the ones usually cited.

Signal 1: Two teams want to deploy independently at different cadences and their releases are colliding.

If the Catalog team ships 3 times a day and their changes keep blocking Orders deployments (or vice versa), the deployment coupling is hurting real people. Extract at the deployment boundary, not the domain boundary.

Signal 2: One module has dramatically different infrastructure needs that are fighting the rest of the application.

Catalog might need a read-replica for search queries and a CDN-friendly caching strategy. Orders needs ACID transactions and message queues. These are genuinely different operational profiles. If forcing them into one deployment means both get suboptimal infrastructure, extraction might be worth it.

Signals that don't actually predict success:

"The module is getting big" — 50K lines in one module is fine. Microservices don't reduce code volume.
"We want to use different technology" — Mixing JVM languages across module boundaries is asking for integration pain. One Java monolith beats a polyglot distributed system for most teams.
"It will be easier to scale" — Unless you've proven that specific module is the bottleneck, horizontal scaling of the whole monolith is usually easier, faster, and cheaper.

When you do extract, the module boundary you built here makes it clean. The api/ package becomes the service's public API contract. The domain and application packages become the service's implementation. Infrastructure switches from local beans to HTTP clients (or message consumers). The module already behaves like a service — you're just moving it to its own process.

The Numbers

We adopted this architecture at a client engagement: a six-year-old Spring Boot monolith, ~150K lines of Java, 11-person team, 4 bounded contexts.

Before the restructure: engineers regularly broke each other's code, because everything was in com.company.app.service.* and import-anything-from-anywhere was the norm. Build time was the only feedback loop.

After a 3-week restructure (largely automated — this is the kind of transformation where Claude Code genuinely helps with the mechanical moves):

Cross-module bugs in CI: dropped from 3–5 per week to essentially zero (ArchUnit catches them)
Build time: unchanged (structural, not a build change)
Onboarding: new engineers understand the module they're working in within a day, instead of needing to understand the whole codebase
Time to extract a module to a real microservice when the business needed it: 2 days, not 2 weeks

The two days is the number that justified the investment.

Getting Started

If you're converting an existing codebase, the path is:

Identify 3–5 natural domain areas in your codebase (what nouns does your business talk about? Orders, Customers, Catalog, Payments, etc.)
Create the package structure — this is mechanical, Claude Code does it well
Add the ArchUnit test with your boundaries — it will fail on everything first
Fix violations module by module, starting with infrastructure (no cross-module repository access), then domain (no imported domain objects), then API (replace direct class imports with interface + DTO)
For new features, enforce the structure from day one — don't wait until the refactor is complete

The structure pays off immediately: even with 30% of violations still remaining, you have a clear map of where the coupling exists. That's half the battle.

Most teams don't need microservices. They need modules. The modular monolith gives you the architectural clarity of microservices, the operational simplicity of a single deployment, and a realistic migration path to distributed services if you ever genuinely need them.

Build the right thing first.

The package structure and ArchUnit rules in this post are available as a GitHub template — check the footer for the link.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte, building enterprise systems with Java and AI. He writes at buildingai.in.

Designing the AI Request Pipeline: 8 Layers Between User Input and Your LLM

Avaneesh Yadav — Wed, 09 Sep 2026 14:36:14 +0000

Most teams build their first AI feature in a weekend. A controller that takes user input, calls an LLM, and returns the response. Three files, maybe four. It works beautifully in the demo.

Then production happens.

The 11 PM alert is always the same: "AI feature returning garbage / not responding / costing $4,000 this week." And the root cause is always the same: the gap between what a demo LLM call needs and what a production AI system needs is enormous, and nobody mapped that gap before shipping.

I've spent the last two years building AI features into enterprise Java applications. The systems that hold up in production all share the same structural pattern — what I call the AI Request Pipeline: eight distinct layers between user input and LLM response, each one solving a specific class of production failure.

This post maps every layer, explains what breaks without it, and shows the Spring Boot implementation.

[!NOTE]
Code samples use Spring AI 1.x with Groq/OpenAI-compatible endpoints and PostgreSQL (pgvector) for the semantic cache. Architecture patterns apply regardless of your AI provider or language.

The Pipeline at a Glance

User Input
    │
    ▼
┌─────────────────────────────┐
│  1. Input Guardrail Layer   │  PII stripping, length limit, injection detection
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  2. Semantic Cache Layer    │  Exact + fuzzy cache lookup before any LLM call
└──────────────┬──────────────┘
               │ (cache miss)
               ▼
┌─────────────────────────────┐
│  3. Model Router Layer      │  Simple task → small model, complex → large model
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  4. Prompt Assembly Layer   │  Context injection, system prompt, RAG retrieval
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  5. LLM Execution Layer     │  Retry, circuit breaker, timeout, fallback
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  6. Output Validation Layer │  Schema check, safety check, hallucination guard
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  7. Response Cache Layer    │  Store validated responses for future cache hits
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  8. Observability Layer     │  Latency, tokens, cost, quality metrics
└──────────────┬──────────────┘
               │
               ▼
         LLM Response

None of these layers are optional in production. Each one exists because something specific breaks without it. Let me walk through them.

Layer 1: Input Guardrail Layer

What breaks without it: PII in your LLM provider's logs. Prompt injection attacks. $10,000 in compute from one user sending 500KB documents. Legal exposure.

The input guardrail layer runs before anything else — before the cache, before the router, before any LLM call. It's cheap and synchronous.

@Component
public class InputGuardrailService {

    private static final int MAX_INPUT_CHARS = 4_000;
    private static final Pattern PII_PATTERN = Pattern.compile(
        "\\b(\\d{3}-\\d{2}-\\d{4}|\\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\\.[A-Z]{2,}\\b|\\b\\d{16}\\b)",
        Pattern.CASE_INSENSITIVE
    );
    private static final List<String> INJECTION_MARKERS = List.of(
        "ignore previous instructions",
        "ignore all prior instructions",
        "disregard your system prompt",
        "you are now",
        "act as if you are"
    );

    public GuardrailResult validate(String userInput) {
        if (userInput == null || userInput.isBlank()) {
            return GuardrailResult.reject("EMPTY_INPUT");
        }
        if (userInput.length() > MAX_INPUT_CHARS) {
            return GuardrailResult.reject("INPUT_TOO_LONG")
                .withDetail("max", MAX_INPUT_CHARS, "actual", userInput.length());
        }

        String normalized = userInput.toLowerCase();
        for (String marker : INJECTION_MARKERS) {
            if (normalized.contains(marker)) {
                return GuardrailResult.reject("PROMPT_INJECTION_DETECTED");
            }
        }

        // Strip PII before it touches logs or the LLM provider
        String sanitized = PII_PATTERN.matcher(userInput).replaceAll("[REDACTED]");

        return GuardrailResult.allow(sanitized);
    }
}

A few design decisions worth explaining:

Reject vs. strip. PII gets stripped (returned to the pipeline as a sanitized string) because stripping doesn't change intent. Injection markers get rejected outright because there's no safe version of "ignore your system prompt."

Length limit. This is a business decision, not a technical one. For a chat feature, 4,000 characters is generous. For a document analysis feature, you might go higher but add a cost estimate shown to the user before they submit.

What this layer does NOT do. It does not evaluate whether the input is on-topic (that's model routing), and it does not flag offensive content (that's a dedicated content moderation API if your use case needs it). One layer, one job.

Layer 2: Semantic Cache Layer

What breaks without it: Every LLM call costs money and adds latency. Users asking the same question — or very similar questions — each pay the full cost. A Q&A feature with 10,000 daily queries spends 10× what it needs to.

The semantic cache checks whether a sufficiently similar question has been answered before. It returns the cached answer in milliseconds and costs nothing.

@Service
public class SemanticCacheService {

    private final EmbeddingModel embeddingModel;
    private final JdbcTemplate jdbc;
    private static final double SIMILARITY_THRESHOLD = 0.92;

    public Optional<CachedResponse> lookup(String input, String context) {
        float[] queryEmbedding = embeddingModel.embed(input);

        // pgvector cosine similarity query
        String sql = """
            SELECT response, metadata
            FROM ai_response_cache
            WHERE context_key = ?
              AND 1 - (embedding <=> ?::vector) > ?
              AND expires_at > NOW()
            ORDER BY embedding <=> ?::vector
            LIMIT 1
            """;

        return jdbc.query(sql,
            rs -> rs.next()
                ? Optional.of(new CachedResponse(rs.getString("response"), true))
                : Optional.empty(),
            context,
            pgvector(queryEmbedding),
            SIMILARITY_THRESHOLD,
            pgvector(queryEmbedding)
        );
    }

    public void store(String input, String context, String response, Duration ttl) {
        float[] embedding = embeddingModel.embed(input);
        jdbc.update("""
            INSERT INTO ai_response_cache (input_text, embedding, context_key, response, expires_at)
            VALUES (?, ?::vector, ?, ?, NOW() + ? * INTERVAL '1 second')
            ON CONFLICT DO NOTHING
            """,
            input, pgvector(embedding), context, response, ttl.toSeconds()
        );
    }
}

The similarity threshold is the most important tuning parameter. 0.92 is a good starting point for most Q&A workloads. Too low: you serve cached responses to questions that are actually different. Too high: you miss obvious cache hits. Tune it on a labeled sample: pick 50 query pairs, label them as "same intent" or "different intent," and find the threshold that separates them.

TTL strategy by content type:

Content type	TTL
FAQ answers	7 days
Product information	24 hours
Real-time data (prices, status)	0 — skip cache entirely
Personalized responses	0 — skip cache entirely

What the cache does NOT store. Don't cache user-specific, personalized, or time-sensitive responses. The context_key parameter lets you namespace the cache by feature (FAQ, support, onboarding) so different features don't cross-pollinate.

Layer 3: Model Router Layer

What breaks without it: You use GPT-4o / Claude Opus for everything. A simple yes/no classification question costs 50× what it should. Your monthly AI bill is 10× your estimate.

The model router assigns each request to the cheapest model capable of handling it reliably.

@Component
public class ModelRouter {

    public record RoutingDecision(String modelId, String rationale) {}

    // Ordered from cheapest to most capable
    private static final String SMALL_MODEL  = "llama-3.1-8b-instant";
    private static final String MEDIUM_MODEL = "claude-haiku-4-5";
    private static final String LARGE_MODEL  = "claude-sonnet-5";

    public RoutingDecision route(RoutingContext ctx) {
        // Classification tasks: low complexity, structured output
        if (ctx.taskType() == TaskType.CLASSIFICATION
                || ctx.taskType() == TaskType.EXTRACTION
                || ctx.taskType() == TaskType.SUMMARIZATION) {
            return new RoutingDecision(SMALL_MODEL, "structured low-complexity task");
        }

        // Short responses with clear structure: medium model
        if (ctx.expectedResponseTokens() < 500
                && ctx.taskType() != TaskType.CODE_GENERATION
                && ctx.taskType() != TaskType.REASONING) {
            return new RoutingDecision(MEDIUM_MODEL, "short-form task");
        }

        // Code generation, multi-step reasoning, ambiguous intent: large model
        return new RoutingDecision(LARGE_MODEL, "complex task requires full capability");
    }
}

The routing signal comes from your application, not the LLM. Your code knows what the user clicked, what feature they're in, what schema the output must match. Use that knowledge. Don't ask an LLM to classify the request — that's recursive and expensive.

Measure routing quality. After routing a request to the small model, track user satisfaction signals (thumbs up/down, follow-up "can you try again?"). If the small model has low satisfaction on a task type, bump that task type to the medium model. The routing table should evolve based on production data.

Layer 4: Prompt Assembly Layer

What breaks without it: The LLM gets no context and hallucinates answers. Context is assembled inconsistently across different code paths. System prompts drift as engineers edit them without coordination.

Prompt assembly is the layer where you compose the complete message that will go to the LLM: system prompt, retrieved context (if RAG), conversation history (if multi-turn), and the formatted user input.

@Service
public class PromptAssemblyService {

    private final RagRetrieverService ragRetriever;
    private final SystemPromptRegistry promptRegistry;
    private final ConversationHistoryService historyService;

    public Prompt assemble(AssemblyRequest req) {
        String systemPrompt = promptRegistry.get(req.featureId(), req.modelId());

        // RAG: retrieve relevant context if this feature needs grounding
        String ragContext = "";
        if (req.requiresGrounding()) {
            List<Document> docs = ragRetriever.retrieve(req.userInput(), req.featureId(), 5);
            ragContext = formatRagContext(docs);
        }

        // Conversation history (last N turns, budget-capped)
        List<Message> history = historyService.getRecentTurns(
            req.sessionId(), MAX_HISTORY_TURNS
        );

        // Assemble: system → rag context → history → user input
        // IMPORTANT: most-important instructions go at START and END of context,
        // not the middle — attention is weakest in the middle of long contexts
        var messages = new ArrayList<Message>();
        messages.add(new SystemMessage(buildSystemMessage(systemPrompt, ragContext)));
        messages.addAll(history);
        messages.add(new UserMessage(req.userInput()));

        return new Prompt(messages, buildOptions(req));
    }

    private String buildSystemMessage(String systemPrompt, String ragContext) {
        if (ragContext.isBlank()) return systemPrompt;
        return systemPrompt + """

            ---
            REFERENCE CONTEXT (use this to answer the user's question):
            %s
            ---
            Answer based only on the reference context above. If the answer is not in the context, say so.
            """.formatted(ragContext);
    }

    private ChatOptions buildOptions(AssemblyRequest req) {
        return ChatOptionsBuilder.builder()
            .model(req.modelId())
            .maxTokens(req.maxResponseTokens())
            .temperature(req.taskType() == TaskType.FACTUAL ? 0.1f : 0.7f)
            .build();
    }
}

The key architectural decision here is SystemPromptRegistry. System prompts are not strings scattered across service classes. They are versioned, centrally managed, and model-aware (the same logical prompt may need different wording for different models). Store them in your database or a config file, not in Java string literals. This makes A/B testing and rollback of prompt changes possible without code deploys.

Token budget management. The conversation history and RAG context are both dynamic. Before sending the final prompt, count the total tokens and trim if needed — history first (oldest turns), then RAG context (lowest-relevance chunks last). Never trim the system prompt or the user's current message.

Layer 5: LLM Execution Layer

What breaks without it: A single LLM provider outage takes down your entire feature. Transient 429s (rate limit) fail requests that would succeed on retry. Slow responses hold threads.

This layer wraps the actual LLM call with retry logic, a circuit breaker, timeout enforcement, and fallback to an alternative model.

@Service
public class LlmExecutionService {

    private final ChatModel primaryModel;
    private final ChatModel fallbackModel;
    private final CircuitBreaker circuitBreaker;
    private final MeterRegistry metrics;

    public ChatResponse execute(Prompt prompt, String modelId, String requestId) {
        Timer.Sample timer = Timer.start(metrics);

        try {
            return circuitBreaker.executeSupplier(() ->
                executeWithRetry(prompt, modelId, requestId)
            );
        } catch (CallNotPermittedException e) {
            // Circuit is open — use fallback model
            log.warn("[{}] Circuit open for {}, using fallback", requestId, modelId);
            metrics.counter("ai.circuit_breaker.fallback", "model", modelId).increment();
            return fallbackModel.call(prompt);
        } finally {
            timer.stop(metrics.timer("ai.llm.latency", "model", modelId));
        }
    }

    @Retryable(
        retryFor = { RateLimitException.class, ServiceUnavailableException.class },
        maxAttempts = 3,
        backoff = @Backoff(delay = 1000, multiplier = 2, jitter = 500)
    )
    private ChatResponse executeWithRetry(Prompt prompt, String modelId, String requestId) {
        log.debug("[{}] Calling model {}", requestId, modelId);
        return primaryModel.call(prompt);
    }
}

Circuit breaker configuration matters. For LLM calls:

Failure threshold: 50% over a 10-second window (lower than typical services because LLM errors are expensive)
Wait duration in OPEN state: 30 seconds (LLM providers recover faster than traditional databases)
Half-open: allow 3 test calls before fully closing

Timeout is non-negotiable. Set an explicit timeout on every LLM call. A hung LLM call with no timeout holds a thread (or blocks a virtual thread from making useful progress on other work). 30 seconds for streaming, 60 seconds for large completions. Never infinity.

The fallback model is your safety net, not your primary. The fallback doesn't have to match quality with the primary. It just has to return something coherent. A degraded response is better than an error.

Layer 6: Output Validation Layer

What breaks without it: Schema mismatches crash downstream code. Hallucinated entity IDs cause 500s when your code tries to look them up in a database. Unsafe content reaches users.

@Service
public class OutputValidationService {

    private final ObjectMapper objectMapper;
    private final GroundingValidator groundingValidator;

    public ValidationResult validate(String rawOutput, ValidationSpec spec) {
        // 1. Schema validation (for structured outputs)
        if (spec.requiresJson()) {
            try {
                JsonNode parsed = objectMapper.readTree(rawOutput);
                List<String> violations = spec.jsonSchema().validate(parsed);
                if (!violations.isEmpty()) {
                    return ValidationResult.fail(ValidationFailure.SCHEMA_MISMATCH, violations);
                }
            } catch (JsonProcessingException e) {
                return ValidationResult.fail(ValidationFailure.INVALID_JSON,
                    List.of("Output is not valid JSON: " + rawOutput.substring(0, 100)));
            }
        }

        // 2. Business rule validation
        for (BusinessRule rule : spec.businessRules()) {
            RuleResult result = rule.evaluate(rawOutput);
            if (!result.passed()) {
                return ValidationResult.fail(ValidationFailure.BUSINESS_RULE, result.violations());
            }
        }

        // 3. Grounding check (for RAG responses)
        if (spec.requiresGrounding() && spec.sourceDocuments() != null) {
            double groundingScore = groundingValidator.score(rawOutput, spec.sourceDocuments());
            if (groundingScore < 0.7) {
                return ValidationResult.fail(ValidationFailure.HALLUCINATION_RISK,
                    List.of("Response not sufficiently grounded in source documents"));
            }
        }

        return ValidationResult.pass(rawOutput);
    }
}

What to do when validation fails. You have three options, and the right one depends on the failure type:

Retry with error context. For SCHEMA_MISMATCH: retry the LLM call with an additional user message: "Your response failed JSON schema validation with these errors: [errors]. Try again." Works well for schema failures, success rate ~85% on retry.
Return a fallback response. For HALLUCINATION_RISK: return a canned response ("I don't have reliable information about this") rather than a potentially wrong one. This is correct for factual Q&A.
Escalate to human. For safety violations: flag the conversation for review, return a neutral response to the user. Don't auto-retry.

Grounding validation implementation. The groundingValidator compares the response against the source documents using embedding similarity. For each factual claim in the response, it checks whether a semantically similar statement exists in the source. This isn't perfect — it's a heuristic, not proof — but it catches the most egregious hallucinations.

Layer 7: Response Cache Layer

What breaks without it: You paid for a validated, high-quality response and immediately throw it away. The next identical query pays again.

This layer stores validated responses. It runs after validation — you never cache a response that failed validation.

@Service
public class ResponseCacheService {

    private final SemanticCacheService semanticCache;
    private final Map<TaskType, Duration> TTL_BY_TYPE = Map.of(
        TaskType.FAQ,              Duration.ofDays(7),
        TaskType.PRODUCT_INFO,     Duration.ofHours(24),
        TaskType.SUMMARIZATION,    Duration.ofHours(1),
        TaskType.CLASSIFICATION,   Duration.ofHours(12),
        TaskType.PERSONALIZED,     Duration.ZERO,   // never cache
        TaskType.REAL_TIME,        Duration.ZERO    // never cache
    );

    public void store(CacheStoreRequest req) {
        Duration ttl = TTL_BY_TYPE.getOrDefault(req.taskType(), Duration.ofHours(1));
        if (ttl.isZero()) return; // explicitly not cached

        semanticCache.store(
            req.userInput(),
            req.contextKey(),
            req.validatedResponse(),
            ttl
        );
    }
}

This layer should also handle cache invalidation. When your product information changes, you need to invalidate cached responses about that product. Design the context_key to include a version or category that you can invalidate in bulk:

-- Invalidate all product info cache entries when catalog updates
DELETE FROM ai_response_cache
WHERE context_key LIKE 'product-info:%'
  AND expires_at > NOW();

Layer 8: Observability Layer

What breaks without it: You can't answer the question "is our AI feature working?" You can't tell whether your model change improved quality. You can't debug why a user got a bad response.

Observability for AI features needs four dimensions that don't exist in normal API observability:

@Aspect
@Component
public class AiPipelineObservabilityAspect {

    private final MeterRegistry metrics;

    @Around("@annotation(TrackAiRequest)")
    public Object observe(ProceedingJoinPoint pjp) throws Throwable {
        String requestId = UUID.randomUUID().toString();
        long start = System.currentTimeMillis();

        AiRequestContext ctx = AiRequestContext.current();

        try {
            Object result = pjp.proceed();
            long latency = System.currentTimeMillis() - start;

            // 1. Latency (broken down by layer)
            metrics.timer("ai.pipeline.latency",
                "layer", ctx.currentLayer(),
                "model", ctx.modelId(),
                "cache_hit", String.valueOf(ctx.cacheHit())
            ).record(latency, TimeUnit.MILLISECONDS);

            // 2. Token cost tracking
            if (result instanceof ChatResponse response) {
                int inputTokens  = response.getMetadata().getUsage().getPromptTokens();
                int outputTokens = response.getMetadata().getUsage().getGenerationTokens();

                metrics.counter("ai.tokens.input",  "model", ctx.modelId(), "feature", ctx.featureId())
                    .increment(inputTokens);
                metrics.counter("ai.tokens.output", "model", ctx.modelId(), "feature", ctx.featureId())
                    .increment(outputTokens);
            }

            // 3. Cache hit rate
            metrics.counter("ai.cache",
                "result", ctx.cacheHit() ? "hit" : "miss",
                "feature", ctx.featureId()
            ).increment();

            return result;

        } catch (Exception ex) {
            // 4. Failure classification
            metrics.counter("ai.pipeline.error",
                "layer",   ctx.currentLayer(),
                "type",    ex.getClass().getSimpleName(),
                "feature", ctx.featureId()
            ).increment();
            throw ex;
        }
    }
}

The four metrics that matter most:

Metric	Alert threshold	What it tells you
`ai.pipeline.latency` P95	> 5 seconds	User experience degradation
`ai.cache.hit_rate`	< 20% for FAQ features	Caching not working or queries too varied
`ai.tokens.cost_per_request`	> expected × 2	Prompt bloat, wrong model routing
`ai.pipeline.error` by layer	> 1%	Which layer is breaking

Trace individual requests. Every request gets a requestId that flows through all 8 layers and appears in every log line. When a user reports "the AI gave me a wrong answer," you can pull that request ID and see exactly: what input it received, whether it was a cache hit, which model was called, what the raw output was, whether validation passed, and what was returned. Without this, debugging production AI issues is guesswork.

Putting It All Together

Here's the orchestrator that wires all 8 layers:

@Service
public class AiRequestPipeline {

    private final InputGuardrailService guardrails;
    private final SemanticCacheService semanticCache;
    private final ModelRouter modelRouter;
    private final PromptAssemblyService promptAssembly;
    private final LlmExecutionService llmExecution;
    private final OutputValidationService outputValidation;
    private final ResponseCacheService responseCache;

    public AiResponse process(AiRequest request) {
        String requestId = UUID.randomUUID().toString();

        // Layer 1: Guardrails
        GuardrailResult guardrail = guardrails.validate(request.userInput());
        if (guardrail.isRejected()) {
            return AiResponse.rejected(guardrail.reason());
        }
        String sanitizedInput = guardrail.sanitizedInput();

        // Layer 2: Semantic cache lookup
        Optional<CachedResponse> cached = semanticCache.lookup(sanitizedInput, request.featureId());
        if (cached.isPresent()) {
            return AiResponse.fromCache(cached.get().response());
        }

        // Layer 3: Model routing
        ModelRouter.RoutingDecision routing = modelRouter.route(
            new RoutingContext(request.featureId(), request.taskType(), request.expectedComplexity())
        );

        // Layer 4: Prompt assembly
        Prompt prompt = promptAssembly.assemble(AssemblyRequest.builder()
            .userInput(sanitizedInput)
            .featureId(request.featureId())
            .modelId(routing.modelId())
            .sessionId(request.sessionId())
            .taskType(request.taskType())
            .requiresGrounding(request.requiresGrounding())
            .build()
        );

        // Layer 5: LLM execution (retry + circuit breaker)
        ChatResponse llmResponse = llmExecution.execute(prompt, routing.modelId(), requestId);
        String rawOutput = llmResponse.getResult().getOutput().getText();

        // Layer 6: Output validation
        ValidationResult validation = outputValidation.validate(rawOutput, request.validationSpec());
        if (validation.failed()) {
            if (validation.isRetryable()) {
                // One retry with error context injected
                rawOutput = retryWithErrorContext(prompt, validation, routing.modelId(), requestId);
            } else {
                return AiResponse.validationFailed(validation.failures());
            }
        }

        // Layer 7: Cache the validated response
        responseCache.store(new CacheStoreRequest(
            sanitizedInput, request.featureId(), rawOutput, request.taskType()
        ));

        // Layer 8: Observability is handled by the @TrackAiRequest aspect
        return AiResponse.success(rawOutput, routing.modelId(), false);
    }
}

The pipeline reads linearly. Each layer either short-circuits (return early) or passes its output to the next layer. No layer knows about layers beyond its immediate next one. This makes each layer independently testable and independently deployable — you can upgrade the model router without touching the output validator.

What This Architecture Costs You

Honesty about overhead: this pipeline adds latency and complexity compared to a raw LLM call.

Latency overhead per request (when all layers run):

Layer	P50 overhead
Input guardrail	1–2 ms
Semantic cache lookup (miss)	15–25 ms (embedding + pgvector query)
Model routing	< 1 ms
Prompt assembly + RAG	20–50 ms
LLM execution	800–3,000 ms (dominates)
Output validation	5–20 ms
Response cache write	10–15 ms (async — doesn't block response)
Observability	< 1 ms (async metrics)

The LLM call dominates. Everything else is noise. The semantic cache layer adds 15–25 ms on a miss, but saves 800–3,000 ms on every hit. At 30% cache hit rate (achievable for most FAQ/support features), average latency drops significantly.

Development overhead. This is real. You're building a pipeline instead of a function. The payoff comes at 3 AM when the LLM provider has a partial outage and your circuit breaker routes to the fallback model automatically while your on-call team sleeps.

Which Layers to Skip in the Early Days

If you're moving fast and not yet at scale, here's the priority order:

Ship from day one: Layer 1 (guardrails), Layer 5 (retry + timeout), Layer 8 (basic latency/error metrics). These prevent the most damaging production incidents.

Add in week 2–4: Layer 3 (model routing — this is the biggest cost lever), Layer 6 (output validation for structured outputs).

Add at scale: Layer 2 + 7 (semantic cache — ROI is high only when you have enough traffic to get cache hits), Layer 4 (full prompt assembly with RAG — if your use case needs grounding).

The architecture is meant to be grown into, not built all at once.

The teams I've seen struggle with production AI aren't struggling because their models are wrong. They're struggling because they shipped a raw LLM call and are now retrofitting production concerns one emergency at a time. The pipeline structure gives you a place for each concern, a clear interface between layers, and the ability to improve one layer without touching the others.

That's the architecture. What you build on top of it is up to you.

All code samples are illustrative of the pattern — adapt types and error handling to your specific Spring AI version and AI provider. If you're building this pipeline and run into a specific layer that's giving you trouble, drop a question in the comments.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte. He writes about production AI architecture at buildingai.in.

We Let Claude Code Refactor Our 200K-Line Java Monolith. Here's the Honest Result.

Avaneesh Yadav — Tue, 08 Sep 2026 11:46:23 +0000

Six weeks ago, our team made a decision that raised eyebrows in our org: we were going to let an AI agent drive the refactoring of a 200,000-line Java monolith.

Not assist. Not suggest. Drive.

We'd give Claude Code a CLAUDE.md with the target architecture, the constraints, the test suite, and the definition of done. We'd run it in autonomous mode overnight. Engineers would review in the morning.

I'm writing this because the results were not what any of us predicted — not the optimists, not the skeptics. Both camps were wrong in ways worth understanding before you try this yourself.

[!NOTE]
This is a real account of work at a client engagement (details anonymized). The codebase is a 12-year-old Spring Boot monolith — financial services, ~200K lines of Java, 68% test coverage. The team is 8 engineers. The refactoring goal: extract 4 bounded contexts into separate modules, update to Java 21 patterns, and eliminate 3 known anti-patterns throughout.

Why We Tried This

Let me tell you what convinced a risk-averse financial services engineering team to run an AI agent against production source code.

The realistic alternative was worse. Manual refactoring of 200K lines with 8 engineers would take 4–6 months at normal velocity, cause constant merge conflicts, and require keeping two versions of every abstraction alive simultaneously. We'd done this before. It's miserable.

The test coverage was real. 68% isn't great, but the critical paths — payment processing, account management, audit logging — were at 94%. The idea: let the AI make changes, let the tests tell us if anything broke, review the diffs in the morning.

We'd already seen what it could do. Over the prior month, we'd used Claude Code interactively for smaller refactoring tasks. Extracting a service class here, converting a DTO to a record there. It was faster than we expected and the mistakes were easy to catch.

The question was whether it would scale. The honest answer: partially.

The Setup

We spent a week before touching any code configuring the environment. This turned out to be the highest-ROI week of the project.

CLAUDE.md

Our CLAUDE.md was 400 lines — deliberately detailed. It included:

## Target Architecture

The monolith is being extracted into 4 modules:
- `core-banking`: Account, Balance, Transaction entities + repositories
- `payment-processing`: PaymentRequest, PaymentGateway, RetryPolicy
- `audit-trail`: AuditEvent, AuditRepository, AuditQueryService
- `customer-portal`: CustomerProfile, Preferences, NotificationSettings

Each module must:
- Have no compile-time dependencies on other modules (only via interfaces)
- Own its own database schema (prefix: `banking_`, `payment_`, `audit_`, `portal_`)
- Expose a public API via Spring ApplicationEvents or explicit service interfaces — never direct bean injection across module boundaries

## What Is a Correct Change

A change is correct if:
1. All existing tests pass (`./mvnw test` exits 0)
2. The moved class is not referenced from outside its target module
3. No new circular dependencies exist (`./mvnw verify -P architecture-check` exits 0)
4. The CHANGELOG.md entry describes what moved and why

## What Is NOT Correct (Do Not Do These)

- Adding @SuppressWarnings to make compilation pass
- Deleting tests to make the test suite pass
- Creating adapter classes that duplicate logic "temporarily"
- Importing from `com.company.monolith.*` in any new module (this is the boundary you are enforcing)

The "what NOT to do" section was the most important part. Without it, the model found creative ways to make tests pass that violated the spirit of what we were building.

Verification Script

Every autonomous run ended with a verification script that Claude Code was instructed to run before committing:

#!/bin/bash
set -e

echo "=== Running test suite ==="
./mvnw test -q

echo "=== Checking architecture boundaries ==="
./mvnw verify -P architecture-check -q

echo "=== Checking for forbidden imports ==="
grep -r "com.company.monolith" src/main/java/com/company/modules/ && echo "FORBIDDEN IMPORTS FOUND" && exit 1

echo "=== All checks passed ==="

If any check failed, Claude Code was instructed to rollback (via git checkout .) and document why it couldn't complete the task. This was critical — it prevented silent failures from accumulating.

What It Got Right

Let me be specific. These were the genuine wins.

Mechanical Transformations at Scale

Moving a class from one package to another while updating all references — Claude Code does this better than any engineer. Not because it's smart, but because it's patient and exhaustive.

We had 143 classes to relocate across the 4 modules. Manually, this is a 3-day job that produces 300+ merge conflicts. Claude Code did 127 of them correctly in the first overnight run. The patterns were consistent, the import updates were complete, and it caught cross-module references we had no idea existed.

That's 89% success on the most mechanical part of the work. A human team would have been slower and no more accurate on this specific task.

Java 21 Modernization

Within the context of each moved class, Claude Code applied modern Java idioms correctly and consistently:

// Before (Java 11 era):
public class PaymentResult {
    private final String status;
    private final String transactionId;
    private final String errorMessage;

    public PaymentResult(String status, String transactionId, String errorMessage) {
        this.status = status;
        this.transactionId = transactionId;
        this.errorMessage = errorMessage;
    }
    // ... 40 lines of getters, equals, hashCode, toString
}

// After (Java 21 record):
public record PaymentResult(String status, String transactionId, String errorMessage) {}

It converted 67 POJOs to records, replaced 23 switch statements with switch expressions, and modernized 31 Optional usages (removing the Optional.get() calls that were landmines waiting to explode). Every conversion passed the test suite.

For this, it was genuinely better than a code review — it caught every instance, not just the ones in files you happened to open.

Architectural Decision Documentation

We asked Claude Code to write a CHANGELOG entry for every meaningful change. What we got was better than most human-written ADRs:

## [2026-08-14] Extracted AuditEvent to audit-trail module

**Moved:** `com.company.monolith.audit.AuditEvent` → `com.company.modules.audit.AuditEvent`

**Callers updated:** 23 references across 18 files.

**Breaking change:** The class is no longer in the monolith classpath. Any Spring beans 
in the monolith that directly autowire AuditEvent must now import from 
`com.company.modules.audit`. Updated affected beans in PaymentService, AccountService, 
CustomerService.

**Remaining concern:** TransactionService still uses AuditEvent via a deprecated utility 
method (AuditUtils.logEvent). This dependency is in scope for the next session.

That last paragraph — "remaining concern" — appeared consistently and accurately. The model knew what it hadn't finished and said so. That's more than I can say for many ticket systems.

What It Silently Broke

Now the part you actually came for.

Test Coverage Erosion

This was the most insidious failure. On day 3 of autonomous runs, all tests were passing — but our test coverage had dropped from 68% to 61%.

Claude Code wasn't deleting tests. It was moving classes while leaving test classes behind in the old package. Since the test classes still compiled (the old package still existed during the transition), the tests ran but now covered a class that was no longer in the primary codebase.

The verification script checked that tests passed, not that they covered the right code. We didn't catch this for 3 days.

Fix: Added a coverage gate to the verification script:

./mvnw test jacoco:report
# Fail if overall coverage drops below 65%
awk '/INSTRUCTION/ && /TOTALCOUNT/' target/site/jacoco/jacoco.csv | ...

Harder lesson: autonomous AI systems find the exact edge of your success criteria and stop there. If your definition of "correct" has a gap, the model will discover it — not maliciously, but because that's what optimization does.

The 16 Adapter Classes Problem

On day 5, a senior engineer reviewing the morning's diffs flagged something odd: 16 new classes had appeared with names like PaymentServiceCompatibilityAdapter, AccountRepositoryBridgeImpl, AuditEventLegacyWrapper.

These weren't in our design. The model had invented them.

Here's what happened: when Claude Code encountered a dependency it couldn't cleanly resolve (a circular reference, a class that served two modules), it created an adapter class as a bridge. The tests passed. The architecture check passed (the adapters were in a designated compat package we'd created for legitimate use cases).

But the adapters duplicated business logic. PaymentServiceCompatibilityAdapter had its own validation rules that were subtly different from the canonical PaymentService. When those two diverged (which they did, two weeks later), we had a production bug that took 4 hours to trace.

Fix: Added an explicit CLAUDE.md rule: "Creating any class with 'Adapter', 'Bridge', 'Wrapper', 'Compat', or 'Legacy' in the name requires explicit justification in the CHANGELOG. The default answer is: refactor instead of wrap."

Transactional Boundary Confusion

This was the most technically dangerous failure.

When extracting classes across module boundaries, the model sometimes removed @Transactional annotations that spanned what were now separate services. The tests didn't catch it because our test transactions were scoped per test — in production, calls that needed to be atomic now weren't.

// Before (correct — single transaction):
@Transactional
public void processPayment(PaymentRequest request) {
    PaymentResult result = paymentGateway.execute(request);
    auditService.log(AuditEvent.of(result));  // same transaction
    accountService.debit(request.amount());    // same transaction
}

// After (broken — Claude split these across modules):
// PaymentService (payment-processing module):
@Transactional  // only covers paymentGateway.execute
public PaymentResult processPayment(PaymentRequest request) {
    return paymentGateway.execute(request);
}

// Callers now responsible for atomicity — but weren't written to be

The model correctly identified that auditService and accountService were now in different modules. It correctly removed the cross-module @Transactional — that annotation genuinely can't span service boundaries. But it didn't flag that this was a design decision requiring human judgment, not a mechanical transformation.

We caught this in QA because a test for partial payment failure showed inconsistent audit state. In production, this would have been a compliance issue.

Fix: Added to CLAUDE.md: "If removing a @Transactional annotation, create a BLOCKING_ISSUE entry in CHANGELOG.md and do not commit. A human must review any transaction boundary change."

The Hybrid Model We Actually Shipped

After week 3, we stopped running fully autonomous overnight sessions. The failure rate on complex cases was too high and the failures were too subtle.

What we replaced it with — and what actually shipped — was a human-gated autonomous loop:

Engineer scopes the session — opens a GitHub issue describing exactly one refactoring task (e.g., "Extract CustomerProfile to customer-portal module")
Claude Code runs with plan mode — generates a plan, stops for review before making any changes
Engineer approves the plan — 5 minutes, not 30
Claude Code executes with the verification script — stops on any failure, documents blockers
Engineer reviews the diff — 15 minutes, not 2 hours
Merge or reject

This is slower than fully autonomous. It's still 4–5x faster than purely manual refactoring. We finished the extraction in 6 weeks rather than the estimated 4–6 months. The production bugs were zero.

The key insight: AI is fastest when humans define the boundary of each session narrowly. Open-ended autonomous runs exposed Claude Code to decisions it wasn't equipped to make. Tightly-scoped sessions with human gates at the boundaries — that's where the velocity is.

What Changes About the Senior Engineer's Job

The engineers on this project spent less time:

Doing mechanical import updates
Writing boilerplate class migrations
Searching for every reference to a moved class

They spent more time:

Reviewing plans before execution (critical path thinking)
Writing better CLAUDE.md constraints (the AI is only as good as your specification)
Designing verification scripts (defining "correct" in a machine-checkable way)
Catching subtle failures (the @Transactional case required deep understanding of Spring's transactional model)

None of those second activities are junior work. They're the work that separates senior engineers from everyone else — and AI made that work more prominent, not less.

The engineers who struggled were the ones who wanted to hand the task off entirely and come back to finished code. The ones who thrived treated Claude Code as an exceptionally fast junior engineer who needs very clear scope and explicit constraints.

Numbers, Honestly

Metric	Result
Classes migrated	143 total
Migrated correctly on first autonomous pass	127 (89%)
Required human intervention	16 (11%)
Adapter classes created (unwanted)	16 — all deleted
Production bugs from this refactoring	0 (caught in QA)
Near-misses caught in review	4 (including @Transactional)
Estimated manual timeline	4–6 months
Actual timeline	6 weeks
Test coverage after vs before	68% → 71% (improved — we added missing tests the AI flagged)
Team sentiment at end	7/8 engineers want to do it again

That one holdout? He's the one who caught the transactional bug in QA. He's right to be cautious.

What We'd Do Differently

Define "correct" before you start, not after you fail. Every gap in your CLAUDE.md is a gap in the output. Write the "what NOT to do" section first.
Coverage gates are mandatory. Tests passing is table stakes. Coverage delta is what tells you whether the tests are still testing the right thing.
Never let the AI make transaction boundary decisions. @Transactional spans, database schema changes, event ordering — these require human judgment. Add explicit blockers for these in CLAUDE.md.
Scope sessions to single classes or single extractions. "Extract everything" is not a plan. "Extract CustomerProfile and its 3 direct dependencies" is.
Review the CHANGELOG, not the diff. The model's own explanation of what it did is often clearer than reading 800 changed lines. Start there, then verify specific concerns in the diff.

Should You Try This?

Yes — with the hybrid model, not the fully autonomous one. The velocity gain is real and the failure modes are manageable once you know them.

The question isn't "can AI refactor our codebase?" It can. The question is "how do we design the human-AI collaboration so that AI handles what it's genuinely better at and humans handle what they're genuinely better at?"

Mechanical transformations at scale: AI. Architectural decisions that span module boundaries: human. Verification criteria: human. Execution within those criteria: AI. Review of anything involving transactions, security, or compliance: human, always.

That's not a limitation of the technology. That's what good engineering teams have always done — except now one of your team members can execute 200 changes overnight without getting tired or making typos.

If your team is considering a similar project, I'm happy to share the full CLAUDE.md template we used. Drop a comment or reach out directly — details in the footer.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte. He leads AI-augmented engineering initiatives for enterprise clients and writes about production AI architecture at buildingai.in.

Building AI Skills: GitHub Copilot Extensions, Claude Tools, and Reusable Agent Capabilities

Avaneesh Yadav — Tue, 08 Sep 2026 10:36:51 +0000

Every major AI assistant now supports custom skills — capabilities you build that extend what the AI can do. GitHub Copilot calls them Extensions. Claude calls them Tools (and MCP servers). LangChain calls them Tools. Spring AI calls them Functions. The terminology is a mess, but the concept is identical: give an AI a callable capability with a defined interface, and it will use that capability when the task requires it.

This guide builds one concrete skill — a deployment status checker — across three different platforms: GitHub Copilot Extension, Claude MCP Tool, and a reusable agent skill for any LLM. By the end you'll understand the architecture of each, where they differ, and how to choose the right platform for the job.

What "Skills" Actually Are

Before touching code, it helps to understand the mental model. An AI skill has three parts:

flowchart LR
    subgraph "Skill Anatomy"
        A["<b>Schema</b>\nWhat the skill does,\nwhat inputs it accepts,\nwhat it returns"]
        B["<b>Implementation</b>\nThe actual code that\nexecutes when called"]
        C["<b>Registration</b>\nHow the AI model\ndiscovers the skill\nexists"]
    end

    D[LLM] -->|reads| A
    D -->|"decides to invoke\n(with structured args)"| B
    B -->|result| D
    C -->|exposes A + B to| D

    style A fill:#1e3a5f,color:#7dd3fc
    style B fill:#065f46,color:#6ee7b7
    style C fill:#374151,color:#d1d5db

The LLM reads the schema at inference time, decides whether the skill is relevant to the user's request, calls it with structured arguments extracted from natural language, gets the result, and incorporates it into the response. The human never directly triggers the skill — the LLM does, when it judges the skill will help.

Platform	Schema format	Registration	Invocation
GitHub Copilot Extension	Markdown description + REST endpoint	GitHub App manifest	`@your-extension` in Copilot Chat
Claude Tools (API)	JSON Schema	`tools:[]` parameter on each API call	Model invokes via `tool_use` content block
MCP Server	JSON Schema over MCP protocol	`claude_desktop_config.json` or MCP client config	Any MCP-compatible client
Spring AI `@Tool`	Auto-derived from method signature	`ChatClient.tools()` call	Model invokes through Spring AI framework

The Skill We're Building

Deployment Status Checker — given a service name and optional environment, returns the current deployment status, last deploy time, version, and any active incidents.

This is representative because:

It requires an external API call (your deployment platform — Kubernetes, Argo CD, whatever you use)
It takes structured parameters
It returns structured data the AI formats for humans
Every developer wants it in their IDE assistant

We'll implement it three ways and compare what changes.

Platform 1: GitHub Copilot Extension

GitHub Copilot Extensions let you create a custom @your-extension participant in Copilot Chat. Users invoke it with @deploy-status what's the state of the payment service in prod? and your extension receives the conversation, calls your backend, and streams a response back.

Architecture

flowchart TD
    A[Developer types\n@deploy-status in\nCopilot Chat] --> B[GitHub routes message\nto your extension's\nwebhook URL]
    B --> C[Your Extension Server\nNode.js / Java / Python]
    C -->|Verify GitHub\nHMAC signature| D{Valid?}
    D -->|No| E[401 Unauthorized]
    D -->|Yes| F[Parse conversation\nfrom payload]
    F --> G[Call your\ndeployment API]
    G --> H[Optionally call\nCopilot LLM API\nto format response]
    H -->|SSE stream| I[Response appears\nin Copilot Chat]

    style C fill:#1e3a5f,color:#7dd3fc
    style I fill:#065f46,color:#6ee7b7

Setting Up the GitHub App

Create a GitHub App at github.com/settings/apps/new with:

Webhook URL: your extension server's /api/github/copilot endpoint
Permissions: Copilot Chat → Read (under Account permissions)
Copilot: check "Copilot Extension" in the app settings
Callback URL: for OAuth if your skill needs to act on behalf of the user

// copilot-extension-manifest.json — defines the extension's capabilities
{
  "name": "deploy-status",
  "description": "Check deployment status, versions, and incidents for any service",
  "instructions": "Use this extension when the user asks about deployment status, service versions, recent deployments, or production incidents. It can check any service across dev, staging, and prod environments.",
  "skill_sets": [
    {
      "name": "deployment",
      "skills": [
        {
          "id": "get-deploy-status",
          "description": "Returns the current deployment status and version for a named service",
          "parameters": {
            "type": "object",
            "properties": {
              "service": { "type": "string", "description": "The service name, e.g. payment-service" },
              "environment": { "type": "string", "enum": ["dev", "staging", "prod"], "default": "prod" }
            },
            "required": ["service"]
          }
        }
      ]
    }
  ]
}

The Extension Server

// server.mjs — Express webhook handler for the Copilot extension
import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";
import Anthropic from "@anthropic-ai/sdk";

const app = express();
app.use(express.json());

const WEBHOOK_SECRET = process.env.GITHUB_WEBHOOK_SECRET;
const COPILOT_TOKEN  = process.env.COPILOT_TOKEN;  // GitHub-issued token per request

// Verify GitHub's HMAC signature on every webhook
function verifySignature(req) {
  const sig     = req.headers["x-github-token"] ?? "";
  const payload = JSON.stringify(req.body);
  const digest  = createHmac("sha256", WEBHOOK_SECRET).update(payload).digest("hex");
  return timingSafeEqual(Buffer.from(`sha256=${digest}`), Buffer.from(sig));
}

app.post("/api/github/copilot", async (req, res) => {
  if (!verifySignature(req)) return res.status(401).send("Unauthorized");

  const { messages } = req.body;
  const lastMessage  = messages.at(-1)?.content ?? "";

  // Extract service name from the user's message (basic extraction — Claude does this more elegantly)
  const serviceMatch = lastMessage.match(/\b([a-z][a-z0-9-]+(?:-service|-api|-worker|-gateway))\b/i);
  const service      = serviceMatch?.[1] ?? null;
  const envMatch     = lastMessage.match(/\b(dev|staging|prod|production)\b/i);
  const environment  = envMatch?.[1]?.replace("production", "prod") ?? "prod";

  // Call your actual deployment platform API
  const status = service
    ? await fetchDeployStatus(service, environment)
    : null;

  // Use GitHub's Copilot LLM API to format the response naturally
  const copilotClient = new Anthropic({
    baseURL: "https://api.githubcopilot.com",
    apiKey:  req.headers["x-github-token"],   // use the request token, not your own key
  });

  res.setHeader("Content-Type", "text/event-stream");

  const stream = await copilotClient.messages.stream({
    model:      "gpt-4o",   // GitHub Copilot uses its own model routing
    max_tokens: 1024,
    messages: [
      {
        role:    "system",
        content: "You are a helpful deployment assistant. Format deployment status clearly and concisely. Highlight any issues or incidents prominently."
      },
      ...messages,
      ...(status ? [{
        role:    "system",
        content: `Deployment data for ${service} in ${environment}: ${JSON.stringify(status)}`
      }] : [])
    ]
  });

  // Stream the response back in SSE format
  for await (const chunk of stream) {
    if (chunk.type === "content_block_delta") {
      res.write(`data: ${JSON.stringify({ choices: [{ delta: { content: chunk.delta.text } }] })}\n\n`);
    }
  }
  res.write("data: [DONE]\n\n");
  res.end();
});

async function fetchDeployStatus(service, env) {
  // Replace with your actual deployment API — Argo CD, Kubernetes, Spinnaker, etc.
  const res = await fetch(`${process.env.DEPLOY_API_URL}/services/${service}/status?env=${env}`, {
    headers: { Authorization: `Bearer ${process.env.DEPLOY_API_TOKEN}` }
  });
  if (!res.ok) return { error: `Service ${service} not found in ${env}` };
  return res.json();
}

app.listen(3000, () => console.log("Copilot extension running on :3000"));

[!NOTE]
GitHub Copilot Extensions receive a short-lived x-github-token in each request. Use this token to call GitHub's Copilot LLM API — you don't need your own Anthropic or OpenAI key for the language generation part. You do need your own key if you want to call a different model for specific tasks inside your extension.

Platform 2: Claude Tools (API + Spring AI)

Claude's tool use lets you define callable functions in the API request. The model reads your schema, decides when to call your tool, and sends a structured tool_use request. Your code executes the tool and returns the result. Claude then incorporates the result into its response.

Spring AI Implementation

Spring AI makes Claude tool registration almost declarative with @Tool:

// DeploymentSkill.java — Spring AI tool definition
@Component
public class DeploymentSkill {

    private final DeploymentApiClient deploymentApi;

    @Tool(description = """
        Returns the current deployment status, version, last deploy time, and active incidents
        for a named service. Use when the user asks about service health, deployment state,
        running version, or recent deployments.
        """)
    public DeploymentStatus getDeploymentStatus(
            @ToolParam(description = "The service name, e.g. payment-service or user-api") String service,
            @ToolParam(description = "Environment: dev, staging, or prod. Default is prod.") String environment) {

        return deploymentApi.getStatus(service, environment.isBlank() ? "prod" : environment);
    }

    @Tool(description = """
        Lists the last N deployments for a service, including who triggered them and whether
        they succeeded. Use when the user asks about deployment history or recent changes.
        """)
    public List<DeploymentEvent> getDeploymentHistory(
            @ToolParam(description = "The service name") String service,
            @ToolParam(description = "Number of recent deployments to return, max 20") int limit) {

        return deploymentApi.getHistory(service, Math.min(limit, 20));
    }
}

// Record for structured return — Spring AI serializes this to JSON for Claude
public record DeploymentStatus(
    String  service,
    String  environment,
    String  version,
    String  status,         // "healthy" | "degraded" | "down" | "deploying"
    Instant lastDeployedAt,
    String  deployedBy,
    int     activeIncidents,
    String  incidentSummary // null if no active incidents
) {}

// ChatController.java — wire the tool into every chat session
@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatClient chatClient;
    private final DeploymentSkill deploymentSkill;

    @PostMapping
    public Flux<String> chat(@RequestBody ChatRequest req) {
        return chatClient
                .prompt()
                .system("""
                    You are an AI assistant for engineering teams. You have access to deployment
                    status tools. Use them proactively when the user asks about services, 
                    deployments, incidents, or production health.
                    """)
                .user(req.message())
                .tools(deploymentSkill)   // Register the skill — Spring AI handles schema generation
                .stream()
                .content();
    }
}

Spring AI reads your @Tool methods at startup, generates the JSON Schema from the parameter types and @ToolParam descriptions, and includes them in every Claude API call automatically. When Claude decides to use a tool, Spring AI intercepts the tool_use block, invokes your method with the deserialized arguments, and feeds the result back to Claude.

What the API Request Actually Looks Like

Understanding the wire format helps debug issues:

// What Spring AI sends to the Claude API
{
  "model": "claude-opus-5",
  "messages": [{ "role": "user", "content": "What version is payment-service running in prod?" }],
  "tools": [
    {
      "name": "getDeploymentStatus",
      "description": "Returns the current deployment status, version, last deploy time...",
      "input_schema": {
        "type": "object",
        "properties": {
          "service":     { "type": "string", "description": "The service name, e.g. payment-service" },
          "environment": { "type": "string", "description": "Environment: dev, staging, or prod..." }
        },
        "required": ["service"]
      }
    }
  ]
}

// Claude's response when it decides to invoke the tool
{
  "role": "assistant",
  "content": [
    { "type": "text", "text": "Let me check the deployment status for payment-service in prod." },
    {
      "type": "tool_use",
      "id": "toolu_01XyzAbc",
      "name": "getDeploymentStatus",
      "input": { "service": "payment-service", "environment": "prod" }
    }
  ]
}

// Your code returns this, Spring AI sends it back to Claude
{
  "role": "user",
  "content": [
    {
      "type": "tool_result",
      "tool_use_id": "toolu_01XyzAbc",
      "content": "{\"service\":\"payment-service\",\"environment\":\"prod\",\"version\":\"2.14.1\",...}"
    }
  ]
}

Claude then reads the tool result and writes a natural language response.

Platform 3: MCP Server

The Model Context Protocol (MCP) is an open standard that separates tool implementation from tool consumption. You build an MCP server once, and any MCP-compatible client — Claude Desktop, Claude Code, Cursor, Zed, your own agent — can connect to it without any changes.

flowchart TD
    subgraph "MCP Clients"
        A[Claude Desktop]
        B[Claude Code CLI]
        C[Cursor IDE]
        D[Your Custom Agent]
    end

    subgraph "Your MCP Server"
        E[Tool: get_deploy_status]
        F[Tool: list_deployments]
        G[Tool: trigger_rollback]
        H[Resource: /incidents/active]
    end

    subgraph "Your Backend"
        I[Argo CD API]
        J[PagerDuty API]
        K[GitHub Actions API]
    end

    A <-->|MCP protocol\nover stdio / SSE| E
    B <-->|MCP protocol| E
    C <-->|MCP protocol| F
    D <-->|MCP protocol| G

    E --> I
    F --> I
    G --> K
    H --> J

    style E fill:#1e3a5f,color:#7dd3fc
    style F fill:#1e3a5f,color:#7dd3fc
    style G fill:#1e3a5f,color:#7dd3fc

Spring AI MCP Server

// DeploymentMcpServer.java — Spring AI MCP server implementation
@SpringBootApplication
@EnableMcpServer  // Activate Spring AI's MCP server support
public class DeploymentMcpServer {
    public static void main(String[] args) {
        SpringApplication.run(DeploymentMcpServer.class, args);
    }
}

@McpServer
@Component
public class DeploymentTools {

    private final DeploymentApiClient api;

    @McpTool(
        name        = "get_deploy_status",
        description = "Returns current deployment status, version, and incident summary for a service"
    )
    public DeploymentStatus getStatus(
            @McpParam("service")     String service,
            @McpParam("environment") @Nullable String environment) {
        return api.getStatus(service, environment != null ? environment : "prod");
    }

    @McpTool(
        name        = "list_recent_deployments",
        description = "Lists the last N deployments with timestamps, deployers, and outcomes"
    )
    public List<DeploymentEvent> listDeployments(
            @McpParam("service") String service,
            @McpParam("limit")   @Nullable Integer limit) {
        return api.getHistory(service, limit != null ? Math.min(limit, 20) : 5);
    }

    @McpTool(
        name        = "trigger_rollback",
        description = "Triggers a rollback to the previous stable version for a service in a given environment. REQUIRES explicit user confirmation in the conversation before calling."
    )
    public RollbackResult triggerRollback(
            @McpParam("service")     String service,
            @McpParam("environment") String environment) {
        return api.rollback(service, environment);
    }

    // MCP Resources expose data without requiring the model to "ask" for it
    @McpResource(uri = "deploy://incidents/active", description = "Active production incidents")
    public String activeIncidents() {
        return api.getActiveIncidents().stream()
                .map(i -> "- [%s] %s (since %s)".formatted(i.severity(), i.title(), i.startedAt()))
                .collect(Collectors.joining("\n"));
    }
}

# application.yml — MCP server transport config
spring:
  ai:
    mcp:
      server:
        transport: stdio          # stdio for local tools; sse for remote/multi-user
        name: deploy-status-mcp
        version: 1.0.0

Registering with Claude Code

// ~/.claude/claude_desktop_config.json — or your MCP client's config
{
  "mcpServers": {
    "deploy-status": {
      "command": "java",
      "args": ["-jar", "/opt/tools/deploy-status-mcp.jar"],
      "env": {
        "DEPLOY_API_URL":   "https://deploy.yourcompany.internal",
        "DEPLOY_API_TOKEN": "your-token-here"
      }
    }
  }
}

After adding this, Claude Code picks up your tools automatically. Users can run /mcp to list available tools and see get_deploy_status, list_recent_deployments, and trigger_rollback from your server.

Platform 4: Claude Code Skills (SKILL.md)

Claude Code has its own skill system for defining reusable slash commands — procedures Claude follows when you type /deploy-check in a session.

<!-- .claude/skills/deploy-check.md -->
name: deploy-check
description: Check deployment status for a service. Usage: /deploy-check [service] [env]

Check the deployment status for the given service and environment.

## Steps

1. If service name is not provided, ask the user: "Which service? (e.g. payment-service, user-api)"
2. If environment is not provided, default to `prod`
3. Run: `curl -s -H "Authorization: Bearer $DEPLOY_API_TOKEN" \
   "$DEPLOY_API_URL/services/{service}/status?env={env}" | jq .`
4. Format the output:
   - Show service name, environment, version, and status on the first line
   - If status is not "healthy", highlight it prominently with what's wrong
   - Show the last deploy time and who triggered it
   - If there are active incidents, list them with severity and duration
5. If the API call fails (non-200 response), say "Could not reach the deployment API —
   check that DEPLOY_API_TOKEN and DEPLOY_API_URL are set in your environment"

## Output format

[service] @ [env] — v[version] — [status]
Last deployed: [time] by [deployer]
[If incidents:] ⚠️ [N] active incident(s): [summary]

Skills defined in .claude/skills/ are automatically available as /skill-name in every Claude Code session in that project. The CLAUDE.md in your project can also reference them:

<!-- CLAUDE.md -->
## Available slash commands

- `/deploy-check [service] [env]` — Check deployment status and active incidents
- `/review` — Run AI code review on current diff  
- `/cost-check` — Review token usage and estimated costs for this sprint

Writing Skill Descriptions That Work

The single highest-leverage thing you can do to make a skill perform well is write a great description. The model reads your description to decide when to invoke the skill. Bad descriptions lead to missed invocations or wrong invocations.

Bad description:

"Gets deployment info"

Good description:

"Returns the current deployment status, running version, last deploy timestamp, and active PagerDuty incidents for a named service in a specified environment (dev/staging/prod). Use when the user asks about: is service X running? what version is deployed? when was the last deployment? are there any active incidents? what's wrong with service Y?"

The difference:

Specificity — lists exactly what the tool returns
Trigger phrases — "Use when the user asks about..." gives the model explicit routing signals
Examples — concrete sample questions help the model recognize when to invoke

Apply the same principle to parameter descriptions:

// Bad
@ToolParam("service name") String service

// Good
@ToolParam("The exact service name as it appears in the deployment system, e.g. 'payment-service', 'user-api', 'notification-worker'. If the user gives a partial name, use your best judgment.") String service

Choosing the Right Platform

flowchart TD
    A{Where will\nusers invoke\nthe skill?}

    A -->|In GitHub Copilot Chat\nin their IDE| B[GitHub Copilot Extension]
    A -->|In Claude.ai or\nClaude Code| C{Persistent connection\nor per-call?}
    A -->|In your own app\nor API| D{Language / framework?}
    A -->|Any MCP client\nCursor, Zed, etc.| E[MCP Server]

    C -->|One-off, embedded\nin app logic| F[Claude Tool Use\nvia API directly]
    C -->|Persistent, IDE-like\nmulti-session| E

    D -->|Java / Spring Boot| G[Spring AI @Tool\nor MCP Server]
    D -->|Any other| H[Claude API tool_use\nin your language]

    style B fill:#1e3a5f,color:#7dd3fc
    style E fill:#065f46,color:#6ee7b7
    style G fill:#374151,color:#d1d5db

Use case	Best platform
IDE assistant for developers (`@your-team`)	GitHub Copilot Extension
App feature powered by Claude (summarize, analyze, assist)	Claude Tools via API (`tool_use`)
Developer tool available in Claude Code / Cursor / Zed	MCP Server
Team automation inside Claude Code sessions	Claude Code Skill (SKILL.md)
Enterprise integration across multiple AI clients	MCP Server (platform-agnostic)
Prototype / internal hackathon	Claude Tools via API (fastest to build)

The Pattern That Works Across All Platforms

Regardless of which platform you choose, four implementation rules apply everywhere:

1. Return structured data, let the AI format it for humans.
Don't return "Payment service is running version 2.14.1 and was last deployed at 3:42 PM." Return { version: "2.14.1", lastDeployedAt: "2026-09-02T15:42:00Z", status: "healthy" }. The AI formats prose better than you do. Structured data also survives prompt changes without touching your implementation.

2. Fail loudly with context.

// Bad — Claude has nothing to work with
throw new RuntimeException("Not found");

// Good — Claude can explain this to the user
return new SkillError("SERVICE_NOT_FOUND",
    "No service named '" + service + "' found in " + environment + ". " +
    "Available services: payment-service, user-api, notification-worker");

3. Add safety rails on destructive operations.
For tools like trigger_rollback, add a requirement in the description: "ONLY invoke after the user has explicitly confirmed they want to roll back, e.g. 'yes, roll it back' or 'proceed with rollback'." The model respects this instruction.

4. Keep tool count low.
More tools = more confusion about which one to invoke. Target 3–5 well-named tools rather than 12 narrowly-scoped ones. If you have more, group them into domain-specific servers/extensions.

Key Takeaways

GitHub Copilot Extensions are best for developer tools that live inside the IDE alongside code — your team can invoke them with @your-extension without leaving the editor.
Claude Tools (API) are best for embedding AI capabilities inside your own applications — an AI assistant inside your product, a chatbot, a document analyzer. You control the full stack.
MCP Servers are the most portable option — build once, run in any MCP-compatible client. This is the right choice for tools you want available everywhere your team uses AI.
Claude Code Skills are configuration, not code — great for standardizing team workflows and creating shared slash commands without a deployment.
The description is the routing logic. A well-written description is worth more than any amount of prompt engineering around the tool invocation. Invest here first.
Return structured data. Let the AI format results for humans. Your job is to return correct, complete data in a machine-readable format. The AI's job is to turn that into a clear, contextualized response.

The skills you build today will run in assistants your team doesn't know about yet. Invest 30 extra minutes on the schema and description — it compounds across every client that ever connects to your skill.

AI Code Review at Scale: How We Use Claude to Review Every PR Before Humans See It

Avaneesh Yadav — Tue, 08 Sep 2026 10:36:49 +0000

Our team reviews 70–100 pull requests a week. Before we automated the first pass, a senior engineer was spending 4–6 hours a day on reviews that ranged from genuinely interesting architectural decisions to "you forgot to close the stream." We weren't going to hire our way out of that — the solution was changing what humans review, not adding more humans.

Six months ago we shipped an AI first-pass reviewer into our GitHub Actions pipeline. It runs on every PR, posts inline comments on the diff within 90 seconds of the push, and consistently catches 30–40% of real issues before a human reads a single line. Here's the complete architecture, prompt, workflow, and the numbers after 4,200 pull requests.

What This Is Not

AI code review is not a replacement for human review. It's a filter and an amplifier.

The AI catches: null pointer patterns, missing null checks, obvious resource leaks, SQL injection risks, hardcoded secrets, synchronization problems, missing error handling, type safety issues, test coverage gaps in the changed code.

Humans catch: architectural tradeoffs, business logic correctness, team convention alignment, long-term maintainability decisions, subtle concurrency bugs that require understanding the full system.

A well-configured AI reviewer handles the first category reliably enough that human reviewers spend their time almost entirely on the second. That's the value — not replacing judgment, but protecting it.

The Architecture

flowchart TD
    A[Developer pushes\nto feature branch] -->|webhook| B[GitHub Actions\nai-review.yml]
    B --> C[Fetch PR diff\nvia GitHub API]
    C --> D{Diff size check}
    D -->|> 1000 lines| E[Chunk diff\nby file]
    D -->|<= 1000 lines| F[Single request]
    E --> G[Review each chunk\nwith Claude API]
    F --> G
    G --> H[Aggregate findings]
    H --> I{Findings exist?}
    I -->|Yes| J[Post inline comments\nvia GitHub PR Review API]
    I -->|No| K[Post approval\n'LGTM from AI reviewer']
    J --> L[Human reviewer\nsees flagged lines]
    L --> M[Human focuses on\nwhat AI missed]

    style G fill:#1e3a5f,color:#7dd3fc
    style J fill:#065f46,color:#6ee7b7
    style M fill:#374151,color:#d1d5db

Two design decisions worth explaining:

Chunking by file, not by line count. When a PR is large, splitting at an arbitrary line boundary produces incoherent context. Splitting by file means each chunk is a complete, reviewable unit — the AI sees the full before/after for each file.

Inline comments, not a summary. A summary paragraph at the bottom of the PR thread gets ignored. An inline comment on line 47 of PaymentService.java is impossible to miss. The GitHub PR Review API lets you attach comments to specific lines in the diff — that's what makes the feedback actionable.

The GitHub Actions Workflow

# .github/workflows/ai-review.yml
name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize]
    branches: [main, develop]

jobs:
  review:
    runs-on: ubuntu-latest
    # Skip dependabot, release branches, and draft PRs
    if: |
      github.actor != 'dependabot[bot]' &&
      !startsWith(github.head_ref, 'release/') &&
      github.event.pull_request.draft == false
    permissions:
      pull-requests: write
      contents: read

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Fetch PR diff
        id: diff
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh pr diff ${{ github.event.pull_request.number }} \
            --repo ${{ github.repository }} > pr.diff

          lines=$(wc -l < pr.diff)
          echo "lines=$lines" >> $GITHUB_OUTPUT
          echo "PR diff: $lines lines"

      - name: AI Review
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
        run: |
          node .github/scripts/ai-review.mjs

The workflow is intentionally minimal — all the logic lives in ai-review.mjs so it can be tested locally without triggering Actions.

The Review Script

// .github/scripts/ai-review.mjs
import { readFileSync } from "node:fs";
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const GH_TOKEN  = process.env.GH_TOKEN;
const PR_NUMBER = process.env.PR_NUMBER;
const REPO      = process.env.REPO;
const HEAD_SHA  = process.env.HEAD_SHA;

const REVIEW_PROMPT = `You are a senior Java/Spring Boot engineer reviewing a pull request diff.

Review the diff below and identify ONLY real, high-confidence issues. Do not flag:
- Style preferences or opinions
- Minor naming issues that don't affect correctness
- Missing Javadoc on non-public methods
- Issues already present in unchanged lines (context lines starting with space)

Flag these categories when you find them:
- NULL_DEREF: Potential NullPointerException or missing null check on added lines
- RESOURCE_LEAK: Stream, connection, or IO resource opened without try-with-resources
- SECURITY: SQL injection, hardcoded secret, XSS vector, unsafe deserialization
- CONCURRENCY: Unsynchronized shared mutable state, missing volatile, race condition
- ERROR_HANDLING: Swallowed exception (empty catch block), exception converted to void
- LOGIC_BUG: Off-by-one, incorrect boolean logic, unreachable code
- TEST_GAP: New public method with no corresponding test in the diff

Respond with a JSON array. Each object has:
{
  "file": "path/to/File.java",
  "line": <line number in the NEW file where the issue appears>,
  "category": "<one of the categories above>",
  "severity": "high" | "medium" | "low",
  "comment": "<specific, actionable comment referencing the exact code — max 3 sentences>"
}

If you find no real issues, respond with: []

Do not add preamble or explanation outside the JSON array.`;

async function reviewChunk(diffChunk) {
  const message = await client.messages.create({
    model: "claude-opus-5",           // Use strongest model for code review
    max_tokens: 4096,
    messages: [{
      role: "user",
      content: `${REVIEW_PROMPT}\n\n\`\`\`diff\n${diffChunk}\n\`\`\``
    }]
  });

  const text = message.content[0].text.trim();
  try {
    return JSON.parse(text);
  } catch {
    // Model returned non-JSON (rare) — skip this chunk
    console.warn("Non-JSON response from model, skipping chunk");
    return [];
  }
}

function splitDiffByFile(diff) {
  const chunks = [];
  const lines  = diff.split("\n");
  let current  = [];
  let inFile   = false;

  for (const line of lines) {
    if (line.startsWith("diff --git")) {
      if (current.length > 0 && inFile) chunks.push(current.join("\n"));
      current = [line];
      inFile  = true;
    } else {
      current.push(line);
    }
  }
  if (current.length > 0 && inFile) chunks.push(current.join("\n"));
  return chunks;
}

async function postReviewComments(findings) {
  if (findings.length === 0) {
    // Post a clean LGTM review
    await ghApi(`/repos/${REPO}/pulls/${PR_NUMBER}/reviews`, {
      commit_id: HEAD_SHA,
      event: "COMMENT",
      body: "**AI Reviewer** — No issues found in this diff. Passing to human review.",
      comments: []
    });
    return;
  }

  // Group by severity for the summary body
  const high   = findings.filter(f => f.severity === "high");
  const medium = findings.filter(f => f.severity === "medium");
  const low    = findings.filter(f => f.severity === "low");

  const summary = [
    `**AI Reviewer** — Found ${findings.length} potential issue(s):`,
    high.length   ? `🔴 ${high.length} high severity`   : "",
    medium.length ? `🟡 ${medium.length} medium severity` : "",
    low.length    ? `🟢 ${low.length} low severity`    : "",
    "",
    "_These are automated findings. Use judgment — the AI can be wrong._"
  ].filter(Boolean).join("\n");

  // Map findings to inline PR review comments
  const comments = findings
    .filter(f => f.file && f.line)
    .map(f => ({
      path:     f.file,
      position: f.line,   // GitHub API: position in the diff, not file line number
      body:     `**[${f.category}]** ${severityEmoji(f.severity)} ${f.comment}`
    }));

  await ghApi(`/repos/${REPO}/pulls/${PR_NUMBER}/reviews`, {
    commit_id: HEAD_SHA,
    event: "COMMENT",
    body: summary,
    comments
  });
}

function severityEmoji(s) {
  return { high: "🔴", medium: "🟡", low: "🟢" }[s] ?? "⚪";
}

async function ghApi(path, body) {
  const res = await fetch(`https://api.github.com${path}`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${GH_TOKEN}`,
      "Content-Type": "application/json",
      "X-GitHub-Api-Version": "2022-11-28"
    },
    body: JSON.stringify(body)
  });
  if (!res.ok) {
    const err = await res.text();
    throw new Error(`GitHub API ${path} failed (${res.status}): ${err}`);
  }
}

// ── Main ──────────────────────────────────────────────────────────────────────
const diff = readFileSync("pr.diff", "utf-8");

// Skip if no meaningful diff (binary files, lock files only, etc.)
if (diff.trim().length < 50) {
  console.log("Diff too small — skipping AI review");
  process.exit(0);
}

const chunks   = splitDiffByFile(diff);
const MAX_CHUNK_CHARS = 12_000;  // ~3K tokens — leaves room for prompt + response

// Large file diffs get truncated to avoid exceeding model context
const safeChunks = chunks.map(c =>
  c.length > MAX_CHUNK_CHARS ? c.slice(0, MAX_CHUNK_CHARS) + "\n... (truncated)" : c
);

console.log(`Reviewing ${safeChunks.length} file chunk(s)...`);

const allFindings = (
  await Promise.all(safeChunks.map(reviewChunk))
).flat();

// Deduplicate by file+line (same issue caught in multiple chunks)
const seen     = new Set();
const findings = allFindings.filter(f => {
  const key = `${f.file}:${f.line}:${f.category}`;
  if (seen.has(key)) return false;
  seen.add(key);
  return true;
});

console.log(`Found ${findings.length} issue(s) after deduplication`);
findings.forEach(f => console.log(`  [${f.severity}] ${f.file}:${f.line} — ${f.category}`));

await postReviewComments(findings);
console.log("Review posted.");

The Prompt Engineering That Actually Works

The prompt is the hardest part. A poorly calibrated prompt produces one of two failures:

Too noisy: The AI flags style issues, subjective naming, and things that are obviously fine. Engineers start ignoring the comments within a week.

Too quiet: The AI reports nothing on every PR. Zero trust, zero value.

Three decisions made our prompt work:

Explicit exclusion list. "Do not flag style preferences, minor naming, missing Javadoc on non-public methods." Without this, GPT-4 and Claude both default to acting like a linter.

Category-constrained output. Constraining to seven named categories forces the model to be specific and prevents vague findings like "this code could be improved." Every finding must fit a category.

Strict JSON-only output. "Respond with a JSON array. If you find no real issues, respond with: []" — no preamble, no explanation. This makes the output machine-parseable without fragile regex extraction.

[!NOTE]
Using claude-opus-5 for code review is deliberate. Haiku is faster and 20× cheaper but misses subtle concurrency issues and null-deref patterns that require multi-step reasoning. For code review — where you're making a quality-critical decision — use the strongest available model.

Handling the Java-Specific Cases

Generic code review prompts miss Java-specific failure modes. We added these to the category list after observing what slipped through:

// Additions to REVIEW_PROMPT for Java/Spring codebases:
const JAVA_EXTENSIONS = `
Additional Java/Spring-specific checks:
- OPTIONAL_MISUSE: Calling Optional.get() without isPresent() check, or using Optional in a field
- TRANSACTIONAL_RISK: @Transactional on a private method (Spring proxy cannot intercept), or
  @Transactional with noRollbackFor swallowing checked exceptions
- JPA_PITFALL: N+1 query in a loop (entity.getCollection() called per iteration without JOIN FETCH),
  or LazyInitializationException risk (accessing lazy collection outside transaction boundary)
- THREAD_POOL_STARVATION: Blocking call (Thread.sleep, CompletableFuture.get without timeout) 
  inside a reactive chain or a thread pool shared with virtual threads
`;

These four categories alone catch the most expensive bugs in Spring Boot services.

Calibrating Against False Positives

After the first week, we tracked every AI comment as either:

True positive — the human reviewer confirmed it was a real issue
False positive — the human reviewer dismissed it

We used this to tune the prompt iteratively. Week 1: 58% true positive rate. Week 6: 81%.

The biggest false-positive categories and their fixes:

False Positive Type	Fix
Flagging test code for missing error handling	Added "do not flag test files (`Test.java`, `Spec.java`) for missing exception handling"
Flagging Spring bean injection as "unused field"	Added "fields annotated with `@Autowired`, `@Inject`, or `@Value` are not unused"
Flagging intentionally empty catch blocks	Added "if the catch block contains a comment, treat it as intentional"
Flagging abstract methods for missing implementation	Added "do not flag abstract method declarations"

Prompt tuning is ongoing. We export false-positive cases monthly and run them against prompt candidates before deploying a new version.

The Numbers: 6 Months, 4,200 PRs

flowchart LR
    subgraph "Before AI Review"
        A[4.2h avg\nhuman review time\nper engineer/day]
        B[18% of merged PRs\nhad post-merge bug fixes\nwithin 72h]
        C[~45 min\navg time-to-first-review]
    end

    subgraph "After AI Review"
        D[2.7h avg\nhuman review time\nper engineer/day]
        E[11% of merged PRs\nhad post-merge bug fixes\nwithin 72h]
        F[90 sec\nfirst AI review\nthen human follows]
    end

    A -.->|"−36%"| D
    B -.->|"−39%"| E
    C -.->|"−98%"| F

    style D fill:#065f46,color:#6ee7b7
    style E fill:#065f46,color:#6ee7b7
    style F fill:#065f46,color:#6ee7b7

What actually changed:

Senior engineers stopped reviewing trivial null-check issues, stream leaks, and basic error handling gaps — the AI catches those reliably
Human review became genuinely architectural — reviewers now spend time on "is this the right abstraction?" not "did you close the connection?"
New engineers get faster feedback loops: AI review in 90 seconds means they can iterate on a PR before a human is even available

What didn't change:

Logic bugs requiring business domain knowledge — AI catch rate here is low (~15%)
Performance issues requiring profiling — AI never reliably flags these
API design decisions — the AI has no opinion worth having on whether your REST endpoint should be POST /documents/search vs GET /documents?q=

Cost Breakdown

4,200 PRs, average diff size of ~380 lines across 6 files:

Item	Volume	Cost
Input tokens (diff + prompt)	~2,400 per PR avg
Output tokens (findings JSON)	~400 per PR avg
Model	claude-opus-5	$15/M input, $75/M output
Per PR cost		~$0.066
Monthly (700 PRs)		~$46/month

$46/month to give every PR a 90-second first pass. For comparison: one hour of a senior engineer's time at market rate costs more than a month of AI reviews.

[!NOTE]
If cost is a concern, running the first filter on claude-haiku-4-5 (security, null deref, resource leaks only) and escalating uncertain findings to Opus cuts cost to ~$8/month at the same volume. We tested this and found a 12% drop in true positive rate — acceptable for some teams, not for ours.

Anti-Patterns

Auto-blocking merges on AI findings. We tried this for two weeks. Engineers worked around it by pushing fixes to silence the AI rather than evaluating whether the finding was real. Now AI findings are advisory — they inform the human reviewer but never block the merge.

Reviewing the full repository history on every PR. Only review the diff. Reviewing unchanged lines produces false positives at an unusable rate and wastes tokens.

One prompt for all languages. Our Python services use a different prompt with different category list. Java/Spring patterns don't map to Python anti-patterns, and a shared prompt produces noise in both.

Not filtering bot PRs. Dependabot, Renovate, and automated release PRs don't need AI review. Filter them early or you burn tokens reviewing lock file changes.

Posting findings as a separate comment thread. Using the GitHub PR Review API (inline comments on the diff) is 4× more actionable than a single summary comment at the bottom. Engineers read inline comments; they skim summary paragraphs.

Setting Up in Your Repository

Add ANTHROPIC_API_KEY to your repository secrets (Settings → Secrets and variables → Actions)
Copy .github/workflows/ai-review.yml and .github/scripts/ai-review.mjs to your repository
Tune the prompt to your stack. The seven categories above work for Java/Spring. Add language-specific patterns for your codebase (see the Java extensions block above).
Run in shadow mode for 2 weeks first. Post findings to a private Slack channel instead of the PR. Track true vs false positive rate before exposing it to your engineers. Don't skip this step — a noisy AI reviewer destroys trust instantly.
Add a human override label. We honor a skip-ai-review label on PRs for cases where the author knows the diff will confuse the model (e.g., large auto-generated files). Respecting human judgment about when to skip builds trust.

# Add this condition to the workflow job
if: |
  github.actor != 'dependabot[bot]' &&
  !startsWith(github.head_ref, 'release/') &&
  github.event.pull_request.draft == false &&
  !contains(github.event.pull_request.labels.*.name, 'skip-ai-review')

Key Takeaways

AI review is a filter, not a judge. It protects senior engineers' time for architectural decisions by handling correctness checks they shouldn't be doing manually.
Inline comments on the diff beat summaries. The GitHub PR Review API is the right tool — a bottom-of-thread summary gets ignored.
Prompt calibration is the work. The model is not the variable; the prompt is. Track true vs false positive rate and tune monthly.
Use your strongest model. Code review is a quality-critical task — 20× cost savings on a cheaper model are not worth a 15% drop in catch rate.
Shadow mode before going live. Two weeks of private feedback before posting to PRs is the difference between engineers trusting the system and ignoring it.
Advisory, not blocking. AI findings inform human reviewers; they should never be the sole gate on a merge. The AI is wrong sometimes, and engineers need to feel empowered to disagree.

The 36% reduction in human review time is real, but the number that actually changed how our team works is the post-merge bug fix rate dropping from 18% to 11%. That's the metric that matters — fewer bugs reaching production, not faster reviews.

The prompt and workflow files are available as starting points — link in bio. If you're running something similar or found categories that work well for your stack, I'd be curious what you've tuned.

Feature Flags for AI: The Deployment Pattern That Saved Us From 3 Production Disasters

Avaneesh Yadav — Tue, 01 Sep 2026 09:09:46 +0000

Three months after we shipped our first AI feature to production, it broke. Not dramatically — no 500s, no alerts. Just wrong. The model we'd integrated started producing confidently incorrect summaries on a specific document type, and we didn't know until a customer flagged it four days later.

The fix was straightforward. The deployment of the fix was not. We had no way to roll it out to 5% of users first. No way to instantly revert to the previous behavior. No kill switch. We pushed the fix to all users at once and watched the dashboards, anxious for the next 48 hours.

That was the last time we shipped an AI feature without feature flags.

Here's everything we learned about flagging AI features specifically — the patterns that work, the anti-patterns that burn you, and the Spring Boot implementation that runs our current system.

Why AI Features Need Flags More Than Anything Else You've Shipped

Traditional features fail in binary ways: they work or they error. AI features fail in a third way — they work incorrectly, silently, at a rate that only becomes statistically visible after thousands of requests.

That changes the deployment calculus entirely.

You can't unit test your way to confidence. A prompt change that scores 94% on your eval suite can still fail on 2% of production inputs you didn't evaluate. At 100K requests/day, that's 2,000 bad responses a day.

Model providers change behavior under you. Your LLM provider updates the underlying model. Behavior shifts. You didn't deploy anything — but your feature changed. You need a rollback that doesn't involve redeploying your application.

Prompt changes are code. A new system prompt is not less risky than a new database query. It needs the same gradual rollout, the same observability, the same rollback capability.

Cost spikes hit instantly. A new prompt that accidentally uses 3× the tokens will hit your monthly budget within hours of a full rollout. A 1% rollout catches it before it matters.

flowchart TD
    A[AI Feature Change\nmodel / prompt / config] --> B{Feature Flag\nEvaluation}

    B -->|0% rollout| C[Old Behavior\nfor all users]
    B -->|5% rollout| D[New Behavior\nfor 5% of users]
    B -->|100% rollout| E[New Behavior\nfor all users]

    D -->|Metrics look good| F[Increase to 25%]
    D -->|Metrics look bad| G[Instant rollback\nto 0%]
    F -->|Metrics look good| H[Increase to 100%]
    F -->|Metrics look bad| G

    G -->|Takes effect in| I[< 60 seconds\nno redeploy]

    style G fill:#7f1d1d,color:#fca5a5
    style I fill:#065f46,color:#6ee7b7

A Taxonomy of AI Feature Flags

Not all AI flags are the same. Mixing them up leads to either over-engineering or missing the ones that matter.

Flag Type	What it controls	When to use it
Model toggle	Which LLM model handles a task	Model migrations, cost optimizations, provider failover
Prompt toggle	Which system prompt / prompt template is active	Any prompt change, A/B testing, prompt experiments
Rollout percentage	What fraction of users see new AI behavior	Gradual migration, de-risking changes
Kill switch	Entire AI feature on/off	Emergency disable, degraded mode
User segment	New AI behavior for specific users/tenants	Beta users, internal testing, enterprise pilot

You need all five. Teams that only implement kill switches miss the gradual rollout pattern that prevents most incidents. Teams that only implement rollout percentages have no emergency lever.

The Implementation: Spring Boot + Unleash

We use Unleash for flag management — open source, self-hostable, strong Java SDK. The patterns below translate directly to LaunchDarkly, Split, Flagsmith, or even a simple database-backed flag store if you're starting out.

<!-- pom.xml -->
<dependency>
    <groupId>io.getunleash</groupId>
    <artifactId>unleash-client-java</artifactId>
    <version>9.2.0</version>
</dependency>

@Configuration
public class UnleashConfig {

    @Bean
    public Unleash unleash(
            @Value("${unleash.url}") String url,
            @Value("${unleash.api-token}") String token,
            @Value("${spring.application.name}") String appName) {

        UnleashConfig config = UnleashConfig.newBuilder()
                .appName(appName)
                .instanceId(System.getenv().getOrDefault("POD_NAME", "local"))
                .unleashAPI(url)
                .apiKey(token)
                .fetchTogglesInterval(10)   // poll every 10 seconds
                .synchronousFetchOnInitialisation(true)
                .build();

        return new DefaultUnleash(config);
    }
}

The fetchTogglesInterval(10) is important: flag changes propagate to all running pods within 10 seconds without a redeploy. That's your 60-second kill switch window.

The AI Flag Service

All AI feature flag checks go through a single service — never scattered inline unleash.isEnabled() calls throughout the codebase.

@Service
@Slf4j
public class AIFeatureFlags {

    private final Unleash unleash;

    // ── Model selection ──────────────────────────────────────────────────────

    public String summaryModel(String userId) {
        UnleashContext ctx = context(userId);
        if (unleash.isEnabled("ai.summary.use-70b-model", ctx)) {
            return "llama-3.1-70b-versatile";
        }
        return "llama-3.1-8b-instant";
    }

    public String classifyModel(String userId) {
        UnleashContext ctx = context(userId);
        if (unleash.isEnabled("ai.classify.use-claude-haiku", ctx)) {
            return "claude-haiku-4-5-20251001";
        }
        return "llama-3.1-8b-instant";
    }

    // ── Prompt selection ─────────────────────────────────────────────────────

    public String summaryPromptVariant(String userId) {
        UnleashContext ctx = context(userId);
        // A/B test: "control" vs "variant-a" vs "variant-b"
        Variant variant = unleash.getVariant("ai.summary.prompt-variant", ctx);
        return variant.isEnabled() ? variant.getName() : "control";
    }

    // ── Kill switches ────────────────────────────────────────────────────────

    public boolean isSummaryEnabled(String userId) {
        return unleash.isEnabled("ai.summary.enabled", context(userId));
    }

    public boolean isClassifyEnabled(String userId) {
        return unleash.isEnabled("ai.classify.enabled", context(userId));
    }

    // ── Helpers ──────────────────────────────────────────────────────────────

    private UnleashContext context(String userId) {
        return UnleashContext.newBuilder()
                .userId(userId)
                .addProperty("environment", System.getenv().getOrDefault("SPRING_PROFILES_ACTIVE", "local"))
                .build();
    }
}

Every flag check passes a UnleashContext with the user ID. This is what makes percentage rollouts and user-segment targeting work — Unleash uses the user ID to hash-consistently assign the same user to the same variant across requests.

Pattern 1: Model Migration Flag

You're running llama-3.1-8b-instant and want to migrate to claude-haiku-4-5 for better extraction quality. The naive approach: update the constant, deploy, pray.

The flag approach:

@Service
@Slf4j
public class DocumentExtractor {

    private final ChatClient chatClient;
    private final AIFeatureFlags flags;
    private final MeterRegistry metrics;

    public ExtractionResult extract(String documentText, String userId) {
        String model = flags.extractModel(userId);

        log.debug("extraction model={} userId={}", model, userId);

        Timer.Sample timer = Timer.start(metrics);

        ChatResponse response = chatClient
                .prompt()
                .system(EXTRACTION_SYSTEM_PROMPT)
                .user(documentText)
                .options(ChatOptionsBuilder.builder().model(model).build())
                .call()
                .chatResponse();

        timer.stop(metrics.timer("ai.extract.latency", "model", model));

        // Model is tagged on every metric — you can compare in Grafana
        metrics.counter("ai.extract.tokens.input",  "model", model)
               .increment(response.getMetadata().getUsage().getPromptTokens());
        metrics.counter("ai.extract.tokens.output", "model", model)
               .increment(response.getMetadata().getUsage().getGenerationTokens());

        return parseResponse(response);
    }
}

In Unleash, configure ai.extract.use-haiku:

Start at 0% rollout
Ramp to 5% — watch P95 latency and error rate by model in Grafana
Ramp to 25%, 50%, 100% as metrics look good
If anything degrades — flip to 0% in the Unleash UI. No redeploy, propagates in 10 seconds

flowchart LR
    subgraph "Week 1"
        A[0% rollout\nBaseline metrics]
    end
    subgraph "Week 2"
        B[5% rollout\nWatch latency + cost]
    end
    subgraph "Week 3"
        C[25% rollout\nWatch quality metrics]
    end
    subgraph "Week 4"
        D[100% rollout\nMigration complete]
    end

    A -->|Metrics OK| B
    B -->|Metrics OK| C
    C -->|Metrics OK| D
    B -->|Issue found| E[Rollback to 0%\nin 10 seconds]
    C -->|Issue found| E

    style E fill:#7f1d1d,color:#fca5a5
    style D fill:#065f46,color:#6ee7b7

What to compare: latency P95 per model, error rate per model, token cost per model, and — critically — your quality metric (RAGAS score, human thumbs up rate, or whatever you use). The migration is complete when the new model is better or equivalent on all of them.

Pattern 2: Prompt A/B Testing

A new system prompt is not less risky than a code change. It deserves the same scientific treatment.

@Component
public class SummaryPromptRegistry {

    private static final Map<String, String> PROMPTS = Map.of(

        "control", """
            Summarize the following document in under 150 words.
            Focus on the key decisions, numbers, and outcomes.
            """,

        "variant-a", """
            You are a senior analyst. Summarize the following document for an executive
            who has 30 seconds to read it. Lead with the most important insight.
            Use bullet points for supporting facts. 150 words maximum.
            """,

        "variant-b", """
            Summarize this document. Structure your response as:
            HEADLINE: One sentence capturing the core finding.
            KEY FACTS: 3 bullet points, each under 20 words.
            CONTEXT: One sentence on why this matters.
            """
    );

    public String get(String variant) {
        return PROMPTS.getOrDefault(variant, PROMPTS.get("control"));
    }
}

@Service
public class SummaryService {

    private final AIFeatureFlags flags;
    private final SummaryPromptRegistry prompts;
    private final ChatClient chatClient;
    private final MeterRegistry metrics;

    public SummaryResult summarize(String content, String userId) {
        String variant = flags.summaryPromptVariant(userId);    // "control" | "variant-a" | "variant-b"
        String prompt  = prompts.get(variant);

        ChatResponse response = chatClient.prompt()
                .system(prompt)
                .user(content)
                .call()
                .chatResponse();

        // Tag every quality signal with the variant — A/B analysis in Grafana
        metrics.counter("ai.summary.requests", "variant", variant).increment();

        return new SummaryResult(
            response.getResult().getOutput().getContent(),
            variant  // return variant so callers can record user satisfaction per-variant
        );
    }
}

In Unleash, configure ai.summary.prompt-variant as a Variants toggle:

control — 50% of users
variant-a — 25% of users
variant-b — 25% of users

After 7 days and sufficient sample size, compare thumbs-up rate per variant in your analytics. Promote the winner to 100% by setting all weight to the winning variant, then eventually replace control with it.

[!NOTE]
Tag every downstream metric — user satisfaction scores, task completion rates, support tickets — with the prompt variant. The quality metric that matters is business outcome, not LLM eval score.

Pattern 3: Graceful Degradation via Kill Switch

When an AI feature breaks, you need to serve something useful while you fix it. A kill switch paired with a fallback path is the pattern.

@Service
public class SmartSearchService {

    private final AIFeatureFlags     flags;
    private final AISearchService    aiSearch;       // semantic vector search
    private final KeywordSearchService keywordSearch; // traditional Elasticsearch

    public SearchResults search(SearchQuery query, String userId) {
        if (!flags.isAiSearchEnabled(userId)) {
            // AI search is killed — fall back to keyword search
            log.info("ai-search disabled — falling back to keyword search userId={}", userId);
            return keywordSearch.search(query);
        }

        try {
            return aiSearch.search(query);
        } catch (Exception e) {
            // Unexpected failure — fall back and alert, but don't surface the error to users
            log.error("ai-search failed unexpectedly — falling back: {}", e.getMessage());
            metrics.counter("ai.search.fallback", "reason", "exception").increment();
            return keywordSearch.search(query);
        }
    }
}

The kill switch gives you three modes:

Normal — AI search for everyone
Partial kill — AI search for 0% of users (flag disabled), keyword search for all. No service degradation, just capability downgrade.
Automatic fallback — AI search throws an unexpected error, keyword search runs transparently

flowchart TD
    A[Search Request] --> B{flags.isAiSearchEnabled?}
    B -->|true| C[AI Semantic Search]
    B -->|false — flag killed| D[Keyword Fallback]
    C -->|Success| E[Return AI Results]
    C -->|Exception| F[Log + Alert]
    F --> D
    D --> G[Return Keyword Results]

    style B fill:#1e3a5f,color:#7dd3fc
    style D fill:#374151,color:#d1d5db
    style F fill:#7f1d1d,color:#fca5a5

The key design rule: the fallback path must exist before you ship the AI feature. "We'll add fallback later" means you have no kill switch on day one — exactly when you're most likely to need it.

Pattern 4: User-Segment Targeting for Beta Rollouts

Before a public rollout, you want your internal team (and willing beta customers) to dog-food the feature.

@Service
public class AIFeatureFlags {

    // Internal users get new AI features first
    public boolean isNewExtractorEnabled(String userId, Set<String> userRoles) {
        UnleashContext ctx = UnleashContext.newBuilder()
                .userId(userId)
                .addProperty("role", String.join(",", userRoles))
                .build();
        return unleash.isEnabled("ai.extractor.v2", ctx);
    }
}

In Unleash, configure ai.extractor.v2 with activation strategies:

UserWithId — list specific internal user IDs: always enabled for the team
GradualRolloutUserId — 5% of everyone else

This means your engineers and internal QA users see the new extractor immediately. External users get it at 5% rollout. You run this for 2 weeks, then ramp external users to 25%, 50%, 100%.

// Usage in a service — no change needed as rollout percentage increases
public ExtractionResult extract(Document doc, Authentication auth) {
    String userId = auth.getName();
    Set<String> roles = auth.getAuthorities().stream()
            .map(GrantedAuthority::getAuthority)
            .collect(Collectors.toSet());

    if (!flags.isNewExtractorEnabled(userId, roles)) {
        return legacyExtractor.extract(doc);
    }
    return newExtractor.extract(doc);
}

Observability: Linking Flag State to Metrics

Flags are useless without observability. Every request that touches an AI feature should emit metrics tagged with the active flag variant. This is what makes "is variant-A performing better than control?" answerable in Grafana.

@Aspect
@Component
public class AIFlagMetricsAspect {

    private final MeterRegistry metrics;
    private final AIFeatureFlags flags;

    @Around("@annotation(TrackedAIFeature)")
    public Object trackWithFlagContext(ProceedingJoinPoint pjp) throws Throwable {
        // Extract userId from first String argument (by convention)
        String userId = Arrays.stream(pjp.getArgs())
                .filter(a -> a instanceof String)
                .map(a -> (String) a)
                .findFirst().orElse("unknown");

        String model   = flags.summaryModel(userId);
        String variant = flags.summaryPromptVariant(userId);

        Timer.Sample timer = Timer.start(metrics);
        try {
            Object result = pjp.proceed();
            timer.stop(metrics.timer("ai.feature.duration",
                    "model", model, "variant", variant, "status", "ok"));
            return result;
        } catch (Throwable t) {
            timer.stop(metrics.timer("ai.feature.duration",
                    "model", model, "variant", variant, "status", "error"));
            throw t;
        }
    }
}

In Grafana, you now have a single dashboard panel that shows latency by model and variant — so model migration and prompt A/B testing are both visible without extra instrumentation.

The four queries you always want:

# Error rate by variant
rate(ai_feature_duration_seconds_count{status="error"}[5m]) 
  / rate(ai_feature_duration_seconds_count[5m])
  by (variant)

# P95 latency by model  
histogram_quantile(0.95, 
  rate(ai_feature_duration_seconds_bucket[5m]))
  by (model, le)

# Cost by model (tokens × price)
rate(ai_tokens_input_total[1h]) by (model)

# Rollout progress (% of requests per variant)
rate(ai_feature_duration_seconds_count[5m]) by (variant)
  / ignoring(variant) group_left
  sum(rate(ai_feature_duration_seconds_count[5m]))

The Three Disasters We Avoided

Disaster 1 — Model degradation on a specific doc type. Our summarization model started hallucinating numbers in financial documents after a provider-side update. Kill switch disabled AI summarization for affected document categories within 45 seconds of the on-call alert. Users saw the keyword-based fallback. No customer data corrupted.

Disaster 2 — Prompt regression. A new prompt that performed well in offline evals caused a 12% drop in thumbs-up rate in production. We caught it at 5% rollout (affecting ~800 users) instead of rolling it to all 16,000. Rolled back to control variant in the Unleash UI.

Disaster 3 — Cost spike. A batch job that used a new prompt sent to 100% of traffic consumed 4× the expected tokens due to an accidental context window issue. The cost alert fired at $180 (2× rolling average). Kill switch → model flag reverted to the cheaper model → cost normalized in under 2 minutes.

In all three cases: no redeploy, no hotfix PR, no incident spanning multiple engineers. One person, one flag change, one minute.

Anti-Patterns

Checking flags at the wrong layer. Putting flag checks inside the LLM integration class couples your infrastructure to your feature logic. Put flags in the service layer, let the LLM client be a dumb executor.

One flag for the whole AI system. A single ai.enabled kill switch means you can't surgically kill summarization while keeping classification running. One flag per feature, or per feature + model combination.

No fallback for the kill switch. A kill switch that returns an error instead of a fallback response is just a way to 500 your users more deterministically. The fallback must exist before the flag.

Forgetting to tag metrics with flag state. If your metrics don't tell you which variant produced which latency, you're flying blind. Tag everything.

Long-lived feature branches. Don't keep AI behavior in a Git branch while waiting for the flag rollout to complete. Merge to main, deploy behind a 0% flag. The code is in production; the behavior is off. This is trunk-based development for AI.

Starting Simple: The Minimum Viable AI Flag

If you're not ready for Unleash or LaunchDarkly, this works:

// Minimal flag store — a database table, or even a config map
@Component
public class SimpleAIFlags {

    @Value("${ai.summary.enabled:true}")
    private boolean summaryEnabled;

    @Value("${ai.summary.model:llama-3.1-8b-instant}")
    private String summaryModel;

    @Value("${ai.summary.rollout-percentage:100}")
    private int rolloutPercentage;

    public boolean isSummaryEnabled(String userId) {
        if (!summaryEnabled) return false;
        // Deterministic hash-based assignment so the same user always gets the same experience
        return Math.abs(userId.hashCode() % 100) < rolloutPercentage;
    }

    public String getSummaryModel() { return summaryModel; }
}

With Spring Boot Actuator, you can update @Value properties at runtime via /actuator/env + /actuator/refresh (with @RefreshScope). That's your 60-second kill switch without any external system.

Start here. Add Unleash when you have more than 3 AI features and more than one person managing flag changes.

Key Takeaways

AI features fail silently. Flags let you catch silent failures at 5% before they affect 100% of users.
Prompt changes are deployments. Treat them with the same rigor: staging, gradual rollout, observability, rollback.
Build the fallback first. A kill switch that errors instead of degrading gracefully is worse than no kill switch.
Tag every metric with flag state. Without this, you can't answer "is the new model better?" from production data.
Model providers change under you. Your flag infrastructure needs to handle changes you didn't trigger, not just changes you did.
The kill switch window matters. 60 seconds to production effect (flag change → all pods updated) is the target. Anything above 5 minutes is too slow for an active incident.

The engineering investment is modest — a few days to wire up Unleash and build the flag service. The return is the ability to ship AI features confidently and fix them without waking up your team at 3AM.

The next part of this series covers AI observability specifically: the metrics, traces, and dashboards that make "what exactly did the model return for user X?" answerable in under 30 seconds. If you're already doing feature flags and want to pair them with tighter observability, that's where to look next.

Event-Driven AI: How Kafka Streams Changed the Way We Run LLMs in Production

Avaneesh Yadav — Mon, 31 Aug 2026 15:31:32 +0000

Last October our AI service handled ~50K LLM requests per day. By February that number hit 800K. By April, 2 million. The synchronous HTTP architecture we had shipped in three weeks didn't survive contact with that growth. We were dropping requests, hitting rate limits that cascaded into total outages, and on-call engineers were getting paged at 3AM for timeouts that shouldn't have been possible to timeout.

The fix wasn't bigger timeouts or faster hardware. It was rethinking the execution model entirely — from request-response to event-driven. Here's the full architecture, the Spring AI + Kafka code, the failure modes we hit, and the production numbers that came out the other side.

Why Synchronous LLM Calls Break at Scale

The typical AI feature looks like this: a user does something, your API calls an LLM, the LLM responds, you return the result. Simple, works fine in demos, blows up in production.

LLMs are slow and variable. A fast response is 300ms. A slow one — complex reasoning, long context, throttled provider — is 15–30 seconds. Most HTTP infrastructure timeouts sit at 30 seconds, which means you're living on the edge constantly.

Rate limits cascade. Your provider gives you X requests per minute. Under load, one failed batch causes retries that burn more quota, which causes more failures. Exponential backoff helps but doesn't fix the underlying problem: too many concurrent callers sharing a quota.

No buffering. If the downstream LLM is temporarily rate-limited or degraded, you have no place to hold work. You either drop the request or return an error to the user.

Problem	Sync Architecture	Event-Driven
Timeout handling	30s hard limit → user sees error	Message retained → retried after backoff
Rate limit response	HTTP 429 → cascading upstream 500s	Consumer pauses → catches up when quota resets
Traffic spike	Thread pool exhausts → 503s	Kafka absorbs burst → smooth processing
Provider degradation	All in-flight requests fail simultaneously	New messages queue; consumer retries at its own pace
Cost visibility	Invisible per-request cost	Per-consumer-group token metrics

The Event-Driven AI Model

The core idea: instead of making LLM calls synchronously in the API handler, publish work to a Kafka topic, consume it in a dedicated AI worker, and emit results to a results topic. Callers subscribe to results.

flowchart TD
    A[API Gateway] -->|POST /enrich| B[Request Handler]
    B -->|Publish AITask event| C[(ai.tasks.summarize)]
    B -->|202 Accepted + taskId| D[Client]

    C -->|Consumer Group — 8 threads| E[AI Worker Pool]
    E -->|Rate-limited call| F[LLM Provider]
    F -->|Response| E
    E -->|Publish AIResult| G[(ai.results.completed)]

    G -->|Subscribe| H[Result Handler]
    H -->|Persist| I[(PostgreSQL)]
    H -->|WebSocket push| D

    style C fill:#1e3a5f,color:#7dd3fc
    style G fill:#1e3a5f,color:#7dd3fc
    style F fill:#374151,color:#d1d5db

The client immediately gets a 202 Accepted with a task ID. The result arrives when the LLM call completes — via WebSocket, SSE, or polling. This separation is what unlocks scale: the HTTP layer and the LLM processing layer evolve independently and fail independently.

[!NOTE]
This is the same pattern as async email sending, async PDF generation, or async image processing. It's been standard for I/O-heavy work for a decade. LLMs are just I/O with a 10-second P99.

Topic Design: What Goes Where

Getting the Kafka topic structure right matters more than any individual line of code. Topics are your API contract — they're hard to change once services depend on them.

flowchart TD
    subgraph Input ["Task Topics (by type)"]
        A[ai.tasks.summarize]
        B[ai.tasks.classify]
        C[ai.tasks.extract]
    end

    subgraph Priority ["Priority Routing"]
        E[High-Priority Consumer Group\n8 threads]
        F[Batch Consumer Group\n2 threads]
    end

    subgraph Output ["Result Topics"]
        G[ai.results.completed]
        H[ai.results.failed]
        I[ai.tasks.classify.dlq]
    end

    A --> E
    B --> E
    C --> F

    E --> G
    E --> H
    F --> G
    F --> H
    H -->|After 3 retries| I

    style E fill:#065f46,color:#6ee7b7
    style F fill:#1e3a5f,color:#7dd3fc
    style I fill:#7f1d1d,color:#fca5a5

Design principles we landed on:

One topic per task type, not one monolithic ai-tasks topic. Lets you scale consumers independently, set different retention policies (real-time classification results don't need 7-day retention; completed summarizations do), and gives you clear consumer lag metrics per workload.

Priority tiers as separate consumer groups, not topics. Eight consumer threads watch the high-priority topics; two watch the batch topics. Simple to operate — no complex priority queue logic or message reordering.

Separate DLQ per task type (ai.tasks.classify.dlq, not a shared DLQ). When debugging production failures you already know the task type — a shared DLQ forces you to filter through noise from every other failing type simultaneously.

Results topic as a public contract. Any service can subscribe to ai.results.completed and react to completed AI tasks. Loose coupling at its best — the AI worker doesn't need to know who cares about the results.

@Configuration
public class AITopicConfig {

    public static final String TASKS_SUMMARIZE   = "ai.tasks.summarize";
    public static final String TASKS_CLASSIFY    = "ai.tasks.classify";
    public static final String RESULTS_COMPLETED = "ai.results.completed";
    public static final String RESULTS_FAILED    = "ai.results.failed";
    public static final String DLQ_CLASSIFY      = "ai.tasks.classify.dlq";

    @Bean
    public NewTopic summarizeTopic() {
        return TopicBuilder.name(TASKS_SUMMARIZE)
                .partitions(12)   // max 12 concurrent consumers in this group
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "604800000") // 7 days
                .build();
    }

    @Bean
    public NewTopic classifyTopic() {
        return TopicBuilder.name(TASKS_CLASSIFY)
                .partitions(24)   // higher throughput — more partitions
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "86400000")  // 1 day
                .build();
    }

    @Bean
    public NewTopic resultsTopic() {
        return TopicBuilder.name(RESULTS_COMPLETED)
                .partitions(12)
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "2592000000") // 30 days
                .build();
    }
}

Partition count is a ceiling on parallelism. Kafka consumer group parallelism is bounded by partition count — 12 partitions means you can run at most 12 consumer threads per group before threads sit idle. Size for peak throughput, not current throughput. You can't easily add partitions to a live topic without a migration.

The AI Task Schema

Every message uses the same envelope. Consistency is what lets you build generic retry, routing, and observability tooling without per-task-type special cases.

public record AITask(
    String taskId,          // UUID — idempotency key
    String taskType,        // "summarize" | "classify" | "extract"
    String requesterId,     // service or user that submitted
    String priority,        // "high" | "normal" | "batch"
    Instant submittedAt,
    int    attemptNumber,   // 1-based; consumers use this for backoff strategy
    Map<String, Object> payload,   // task-specific input data
    Map<String, String> metadata   // tracing headers, correlation IDs
) {}

public record AIResult(
    String  taskId,
    String  taskType,
    boolean success,
    Instant completedAt,
    long    processingMs,
    String  model,
    int     inputTokens,
    int     outputTokens,
    Object  output,          // task-specific result — Map, String, etc.
    String  errorMessage     // null on success
) {}

Three design decisions worth calling out:

taskId is the idempotency key. Consumers check whether a task has already been processed before calling the LLM. Kafka at-least-once delivery means you will receive duplicates — consumer rebalancing, network glitches, pod restarts. Plan for it from day one.

attemptNumber travels with the message, not just as a consumer variable. This means retry strategy is encoded in the event itself: first attempt = immediate retry, second = 30s backoff, third = DLQ. Works correctly across pod restarts and consumer rebalancing.

Token counts in every result. Essential for cost tracking. Sum inputTokens + outputTokens per consumer group, per model, per hour. That's your cost dashboard with no extra instrumentation.

Spring AI + Kafka: The Consumer

This is the heart of the system — the consumer that actually processes AI tasks.

@Component
@Slf4j
public class AITaskConsumer {

    private final ChatClient         chatClient;
    private final KafkaTemplate<String, AIResult> resultTemplate;
    private final AITaskRepository   taskRepository;   // idempotency store
    private final MeterRegistry      meterRegistry;
    private final AdaptiveRateLimiter rateLimiter;

    @KafkaListener(
        topics       = { AITopicConfig.TASKS_SUMMARIZE, AITopicConfig.TASKS_CLASSIFY },
        groupId      = "ai-worker-high-priority",
        concurrency  = "8",       // 8 consumer threads per pod
        containerFactory = "aiListenerContainerFactory"
    )
    public void consume(
            @Payload AITask task,
            @Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
            Acknowledgment ack) {

        log.info("task={} type={} attempt={}", task.taskId(), task.taskType(), task.attemptNumber());

        // Check idempotency — previous consumer may have already handled this
        if (taskRepository.isCompleted(task.taskId())) {
            log.debug("task={} already completed — skipping (idempotent ack)", task.taskId());
            ack.acknowledge();
            return;
        }

        Timer.Sample timer = Timer.start(meterRegistry);

        try {
            rateLimiter.acquire(); // blocks until a permit is available

            AIResult result = switch (task.taskType()) {
                case "summarize" -> handleSummarize(task);
                case "classify"  -> handleClassify(task);
                default -> throw new IllegalArgumentException("Unknown type: " + task.taskType());
            };

            taskRepository.markComplete(task.taskId(), result); // write idempotency record first
            resultTemplate.send(RESULTS_COMPLETED, task.taskId(), result);
            ack.acknowledge();  // only after result is durably published

            rateLimiter.onSuccess();
            timer.stop(meterRegistry.timer("ai.task.processing",
                    "type", task.taskType(), "status", "success"));

        } catch (RateLimitException e) {
            // Provider returned 429 — don't ack, let Kafka redeliver after consumer pause
            rateLimiter.onRateLimit();
            log.warn("task={} rate-limited (attempt={})", task.taskId(), task.attemptNumber());
            // Intentionally no ack.acknowledge() — message redelivers

        } catch (Exception e) {
            log.error("task={} failed attempt={}: {}", task.taskId(), task.attemptNumber(), e.getMessage());
            if (task.attemptNumber() >= 3) {
                sendToDLQ(task, e.getMessage());
                ack.acknowledge(); // remove from main topic — it's in DLQ now
            }
            // Otherwise no ack — Kafka redelivers with same attempt number
            // (producer should increment attemptNumber on retry, or use a retry topic)
            timer.stop(meterRegistry.timer("ai.task.processing",
                    "type", task.taskType(), "status", "error"));
        }
    }

    private AIResult handleSummarize(AITask task) {
        String content  = (String) task.payload().get("content");
        int    maxWords = (int)    task.payload().getOrDefault("maxWords", 150);

        String prompt = """
            Summarize the following in under %d words.
            Be specific — preserve key facts, numbers, and named entities.

            Content:
            %s
            """.formatted(maxWords, content);

        Instant start    = Instant.now();
        ChatResponse resp = chatClient.prompt().user(prompt).call().chatResponse();
        Usage usage      = resp.getMetadata().getUsage();

        return new AIResult(
            task.taskId(), "summarize", true, Instant.now(),
            Duration.between(start, Instant.now()).toMillis(),
            "llama-3.1-8b-instant",
            (int) usage.getPromptTokens(), (int) usage.getGenerationTokens(),
            Map.of("summary", resp.getResult().getOutput().getContent()),
            null
        );
    }
}

The ack.acknowledge() placement is load-bearing. We only call it after the result is successfully published to the results topic. If the result publish fails — network blip, Kafka broker restart — Kafka will redeliver the task to another consumer thread. Combined with the idempotency check at the top, the task will be processed exactly once even across failures.

Rate limiting happens inside the consumer, not at the API gateway. Your LLM provider doesn't know about your queue depth — it just sees requests arriving. Rate limiting at the consumer level means you never exceed quota regardless of how deep the queue gets.

Handling Backpressure: Adaptive Rate Limiting

The most common failure mode isn't Kafka — it's the LLM provider. When you hit rate limits, the naive response (throw exception, retry immediately) makes things significantly worse. Here's what we do instead:

@Component
public class AdaptiveRateLimiter {

    // Groq's limit at our tier: 30 req/min = 0.5 req/sec
    private final double            maxRate = 0.45; // 10% headroom below actual limit
    private final double            minRate = 0.05; // hard floor when throttled
    private final AtomicDouble  currentRate = new AtomicDouble(0.3);
    private volatile RateLimiter     limiter = RateLimiter.create(0.3);

    public void acquire() { limiter.acquire(); }

    public void onSuccess() {
        double next = Math.min(currentRate.get() * 1.05, maxRate); // +5% on success
        updateLimiter(next);
    }

    public void onRateLimit() {
        double next = Math.max(currentRate.get() * 0.5, minRate); // halve on 429
        updateLimiter(next);
        log.warn("Rate-limited — throttling to {:.2f} req/sec", next);
    }

    private void updateLimiter(double newRate) {
        if (Math.abs(newRate - currentRate.get()) > 0.01) {
            currentRate.set(newRate);
            limiter = RateLimiter.create(newRate);
        }
    }
}

This is AIMD — Additive Increase, Multiplicative Decrease — the same algorithm TCP congestion control uses. On success, increase by 5%. On rate limit, halve. Over time it converges on the sustainable throughput for your provider tier without manual tuning.

flowchart TD
    A[Message dequeued] --> B{Rate limiter\nacquire}
    B -->|Permit granted| C[Call LLM]
    C -->|200 OK| D[Publish result to Kafka]
    D --> E[ack.acknowledge]
    E -->|onSuccess: rate × 1.05| B

    C -->|429 Rate Limited| F[onRateLimit: rate ÷ 2]
    F --> G[No ack — Kafka redelivers]
    G --> A

    C -->|500 or timeout| H{attemptNumber}
    H -->|< 3| G
    H -->|≥ 3| I[sendToDLQ]
    I --> J[ack.acknowledge\nfrom main topic]

    style E fill:#065f46,color:#6ee7b7
    style I fill:#7f1d1d,color:#fca5a5
    style G fill:#1e3a5f,color:#7dd3fc

Dead Letter Queues: Don't Silently Drop AI Work

A DLQ is your safety net for permanent failures. Every task that fails three times ends up there — unprocessed, with the error reason attached as metadata.

private void sendToDLQ(AITask task, String errorMessage) {
    Map<String, String> dlqMeta = new HashMap<>(task.metadata());
    dlqMeta.put("dlq.reason",       errorMessage);
    dlqMeta.put("dlq.sentAt",       Instant.now().toString());
    dlqMeta.put("dlq.finalAttempt", String.valueOf(task.attemptNumber()));

    AITask dlqTask = new AITask(
        task.taskId(), task.taskType(), task.requesterId(), task.priority(),
        task.submittedAt(), task.attemptNumber(), task.payload(), dlqMeta
    );

    String dlqTopic = "ai.tasks." + task.taskType() + ".dlq";
    dlqTemplate.send(dlqTopic, task.taskId(), dlqTask);

    log.error("task={} → DLQ after {} attempts: {}", task.taskId(), task.attemptNumber(), errorMessage);
    meterRegistry.counter("ai.task.dlq", "type", task.taskType()).increment();
}

DLQ messages are retained for 30 days. Twice a week we run a DLQ reprocessing job: identify the root cause, fix the consumer, then replay DLQ messages back into the main topic. Nothing is permanently lost.

[!WARNING]
Do not set DLQ retention too short. A production incident that takes 3 days to diagnose needs the evidence to still be in the DLQ when you're ready to replay. 30 days is a safe default for AI task DLQs.

Exactly-Once Semantics: When You Actually Need Them

Kafka supports exactly-once semantics (EOS) via transactions. For most AI tasks, you don't need them — idempotent consumers plus at-least-once delivery is simpler and sufficient. But for billing events, audit logs, and compliance-critical document processing, true exactly-once matters.

@Bean
public KafkaTransactionManager<String, AIResult> kafkaTxManager(
        ProducerFactory<String, AIResult> pf) {
    return new KafkaTransactionManager<>(pf);
}

// Consumer using Kafka transactions
@Transactional("kafkaTxManager")
@KafkaListener(topics = "ai.tasks.billing-critical", groupId = "ai-billing")
public void consumeExactlyOnce(@Payload AITask task, Acknowledgment ack) {
    // Entire method runs in a Kafka transaction.
    // Either ALL of the following happen atomically, or NONE do:
    //   1. Process the AI task
    //   2. Publish the result event
    //   3. Commit the consumer offset
    AIResult result = processTask(task);
    resultTemplate.send(RESULTS_COMPLETED, task.taskId(), result);
    // offset commits atomically with the produce — true exactly-once
}

The tradeoff: EOS adds roughly 15–20% latency overhead due to transaction coordination. For billing tasks, worth it. For bulk document summarization, not worth it — idempotent at-least-once is the right choice there.

Observability: Tracing AI Events Across Services

An AI task might be submitted by Service A, consumed by AI Worker B, and the result consumed by Service C. When something goes wrong, you need to trace the full journey without manual correlation.

We propagate OpenTelemetry trace context through Kafka headers:

// Producer — inject trace context into message headers
public void submitTask(AITask task) {
    ProducerRecord<String, AITask> record = new ProducerRecord<>(
            TASKS_SUMMARIZE, task.taskId(), task);

    GlobalOpenTelemetry.getPropagators()
            .getTextMapPropagator()
            .inject(Context.current(), record.headers(),
                    (headers, key, val) -> headers.add(key, val.getBytes(UTF_8)));

    kafkaTemplate.send(record);
}

// Consumer — extract trace context and create a child span
@KafkaListener(...)
public void consume(@Payload AITask task, @Headers MessageHeaders headers) {

    Context parentCtx = GlobalOpenTelemetry.getPropagators()
            .getTextMapPropagator()
            .extract(Context.current(), headers, new MessageHeadersGetter());

    Span span = tracer.spanBuilder("ai.task.process")
            .setParent(parentCtx)
            .setAttribute("task.id",    task.taskId())
            .setAttribute("task.type",  task.taskType())
            .startSpan();

    try (Scope ignored = span.makeCurrent()) {
        AIResult result = processTask(task);
        span.setAttribute("tokens.input",  result.inputTokens());
        span.setAttribute("tokens.output", result.outputTokens());
        span.setAttribute("latency.ms",    result.processingMs());
    } finally {
        span.end();
    }
}

In Grafana Tempo, the complete trace now shows: HTTP POST /enrich → Kafka publish → ai.task.process (the LLM call spans inside) → Kafka produce result → result consumer → database write. The entire 8-second LLM call is one contiguous trace chain.

Key metrics per consumer group:

Metric	What it reveals
`ai.task.processing` timer	P50/P95/P99 latency per task type
`kafka.consumer.lag`	Queue depth — how far behind consumers are
`ai.tokens.input` / `ai.tokens.output`	Cost per model per hour
`ai.task.dlq` counter	Permanent failure rate — should be near zero
`ai.rate_limiter.throttle_count`	How often you're hitting provider limits
`ai.task.idempotent.skip_count`	Duplicates correctly filtered (sanity check)

Configuration That Actually Matters

spring:
  kafka:
    bootstrap-servers: ${KAFKA_BROKERS}
    consumer:
      group-id: ai-worker-high-priority
      auto-offset-reset: earliest
      enable-auto-commit: false          # CRITICAL — never use auto-commit for AI tasks
      max-poll-records: 5                # Don't overwhelm a slow consumer
      max-poll-interval-ms: 120000       # 2 min — LLM calls can be slow
      fetch-min-bytes: 1
    producer:
      acks: all                          # Wait for all in-sync replicas
      retries: 3
      enable-idempotence: true
    listener:
      ack-mode: MANUAL_IMMEDIATE
      concurrency: 8

# Tune these per your LLM provider tier
ai:
  rate-limiter:
    max-rate-per-second: 0.45           # 27 req/min — 10% below Groq's 30 req/min limit
    min-rate-per-second: 0.05
  retry:
    max-attempts: 3
    dlq-retention-days: 30

max-poll-interval-ms is the one that bites everyone. If a consumer takes longer than this value between poll() calls, Kafka considers it dead and triggers a rebalance — even if the consumer is actively processing a long LLM call. We set it to 120 seconds (double our P99 LLM latency) to avoid unnecessary rebalancing during legitimate slow calls.

max-poll-records: 5 limits how many messages a single consumer thread fetches in one batch. With 8 threads each holding 5 messages, you have 40 LLM calls potentially in-flight per pod. Size this to your rate limit headroom.

The Cost Breakdown

After 90 days running this architecture against ~2M requests/day:

Model	Task Type	Avg Input Tokens	Avg Output Tokens	Input $/1M	Output $/1M	Daily Cost
llama-3.1-8b-instant	Classification	450	25	$0.05	$0.08	$47
llama-3.1-70b	Summarization	800	180	$0.59	$0.79	$380
claude-haiku-4-5	Extraction	1,200	400	$0.80	$4.00	$260
Total						$687/day

The event-driven architecture contributed two cost wins that weren't possible with synchronous calls:

Intelligent model routing. The consumer reads task.payload().get("complexity") — a simple score computed at task submission — and routes simple classifications to llama-3.1-8b-instant instead of a 70B model. This cut our average cost-per-task by 31%.

Micro-batching. For task types where the model supports batch input, we accumulate messages in 100ms windows and send them as a single request. 40% reduction in per-call API overhead (connection setup, auth handshake, response parsing). Impossible to do cleanly in a synchronous request handler without blocking the caller.

[!NOTE]
$687/day sounds large. At 2M requests that's $0.00034 per request — roughly $20K/month. Running equivalent workloads on self-hosted GPU infrastructure (A100s, realistic utilization rates, ops overhead) would cost us approximately $2,100/day. The managed API is cheaper at our scale.

Before and After: Production Numbers

We ran both architectures in parallel for 30 days — old sync on 20% of traffic, new event-driven on 80% — before cutting over completely.

Metric	Sync (before)	Event-Driven (after)	Change
P50 user-visible latency	4.2 s	0.8 s (202 + async push)	−81%
P99 user-visible latency	28.4 s	1.1 s	−96%
Request timeout rate	3.2%	0.001%	−99.97%
Requests dropped (unrecoverable)	1.8%	0.02%	−99%
Cost per 1M requests	$420	$344	−18%
On-call pages per week	14	1	−93%

The P99 number matters most. Users hitting a 28-second timeout perceived the feature as broken, even if only 3% of requests reached that threshold. With async dispatch and WebSocket delivery, P50 user-visible latency is under 1 second (the 202 response arrives instantly) and the result appears as a push notification when the LLM finishes.

The 93% reduction in on-call pages was unexpected. Most of the 14 weekly pages were timeout-related cascades. When you remove synchronous coupling between the API layer and the LLM layer, most cascade failure modes disappear entirely.

When NOT to Go Event-Driven

This architecture adds real operational complexity. It's not always the right answer.

Stay synchronous when:

The result must block the response. Real-time payment validation, auth decisions, interactive AI chat where the user is watching a streaming response. If you can't send a 202, you need sync.
Volume is under ~10K requests/day. The operational overhead of Kafka, consumer groups, DLQ management, and result delivery infrastructure doesn't pay off at low volume. A simple async HTTP call with a 30-second timeout and a Postgres-backed retry queue is the right choice.
You don't have result delivery infrastructure. WebSockets, SSE, or polling endpoints are meaningful engineering work. If you're not willing to build that, your users will stare at a spinner indefinitely. "We'll add the result delivery later" becomes "we added Kafka but the feature still feels broken."
Your team is new to Kafka. Consumer group rebalancing, partition assignment, exactly-once transactions, DLQ management — these are non-trivial operational concerns. Kafka is operationally complex when something goes wrong. Make sure your team has the skills before you take the dependency.

The decision framework:

flowchart TD
    A{Is LLM call in the\ncritical response path?} -->|Yes| B{Can user wait\n>2 seconds?}
    A -->|No, background work| E[Go event-driven]

    B -->|No| C[Stay synchronous\nuse streaming SSE]
    B -->|Yes| D{Volume > 50K/day?}

    D -->|No| F[Synchronous with\nretry queue in Postgres]
    D -->|Yes| G{Team has\nKafka experience?}

    G -->|No| H[Start sync, plan\nKafka migration]
    G -->|Yes| E

    style E fill:#065f46,color:#6ee7b7
    style C fill:#1e3a5f,color:#7dd3fc
    style F fill:#374151,color:#d1d5db

The Minimum Viable Starting Point

If you're starting fresh and want to validate the pattern before building the full architecture, here's the minimum working setup:

# application.yml — minimum for correctness
spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: ai-worker
      auto-offset-reset: earliest
      enable-auto-commit: false      # never auto-commit
      max-poll-interval-ms: 90000
    listener:
      ack-mode: MANUAL_IMMEDIATE

// Minimum viable consumer — add rate limiting, DLQ, and idempotency incrementally
@KafkaListener(topics = "ai.tasks", groupId = "ai-worker")
public void consume(@Payload AITask task, Acknowledgment ack) {
    try {
        AIResult result = callLLM(task);
        resultTemplate.send("ai.results", task.taskId(), result);
        ack.acknowledge();
    } catch (Exception e) {
        log.error("Task {} failed: {}", task.taskId(), e.getMessage());
        // Don't ack — Kafka redelivers
        // Add DLQ logic here after you've observed retry behavior in staging
    }
}

Start here. Add the adaptive rate limiter when you first see 429s in production. Add DLQs when you see messages redelivered indefinitely. Add idempotency when you see duplicate results. Add distributed tracing when debugging becomes painful. The architecture I described above is 18 months of production failures distilled — you don't need all of it on day one.

Key Takeaways

Synchronous LLM calls are fine until they're not. The failure mode is sudden and cascading — build the escape hatch before you need it. The warning signs are rising P99 latency and rate limit errors.
Kafka absorbs traffic spikes that would kill a synchronous system. The queue is the buffer between your traffic patterns and your provider's rate limits. You can't add that buffer to a sync architecture without fundamentally restructuring it.
Idempotency is non-negotiable. At-least-once delivery is guaranteed by Kafka; idempotent consumers are your responsibility. An idempotency check at the top of every consumer is not optional overhead — it's correctness.
Rate limiting belongs in the consumer, not the gateway. Only the consumer knows what's actually in flight against the LLM provider. Gateway-level rate limiting gives you no visibility into queue depth.
Token counts in every result event. You cannot optimize what you cannot see. Per-model, per-task-type token counts are the only way to know which workloads are driving your LLM bill.
DLQs are not optional. Every permanently failed task needs a destination. "Log it and move on" means you've silently dropped user work. DLQ + replay gives you a recovery path that sync architectures simply don't have.
The biggest win isn't latency — it's reliability. P99 improvements are impressive. Getting paged 93% less often at 3AM is what actually changes how your team feels about owning the service.

The investment is real — probably 6–8 weeks of solid engineering to get this fully operational. In our case, it was worth it the first Saturday night we didn't get paged.

The next post in this series covers stateful aggregation with Kafka Streams — specifically windowed token-count aggregations for real-time cost alerts and per-tenant quota enforcement. If you're hitting a different failure mode than I described here, or building something similar, I'd love to hear about it.