DEV Community: Avaneesh Yadav

Feature Flags for AI: The Deployment Pattern That Saved Us From 3 Production Disasters

Avaneesh Yadav — Tue, 01 Sep 2026 09:09:46 +0000

Three months after we shipped our first AI feature to production, it broke. Not dramatically — no 500s, no alerts. Just wrong. The model we'd integrated started producing confidently incorrect summaries on a specific document type, and we didn't know until a customer flagged it four days later.

The fix was straightforward. The deployment of the fix was not. We had no way to roll it out to 5% of users first. No way to instantly revert to the previous behavior. No kill switch. We pushed the fix to all users at once and watched the dashboards, anxious for the next 48 hours.

That was the last time we shipped an AI feature without feature flags.

Here's everything we learned about flagging AI features specifically — the patterns that work, the anti-patterns that burn you, and the Spring Boot implementation that runs our current system.

Why AI Features Need Flags More Than Anything Else You've Shipped

Traditional features fail in binary ways: they work or they error. AI features fail in a third way — they work incorrectly, silently, at a rate that only becomes statistically visible after thousands of requests.

That changes the deployment calculus entirely.

You can't unit test your way to confidence. A prompt change that scores 94% on your eval suite can still fail on 2% of production inputs you didn't evaluate. At 100K requests/day, that's 2,000 bad responses a day.

Model providers change behavior under you. Your LLM provider updates the underlying model. Behavior shifts. You didn't deploy anything — but your feature changed. You need a rollback that doesn't involve redeploying your application.

Prompt changes are code. A new system prompt is not less risky than a new database query. It needs the same gradual rollout, the same observability, the same rollback capability.

Cost spikes hit instantly. A new prompt that accidentally uses 3× the tokens will hit your monthly budget within hours of a full rollout. A 1% rollout catches it before it matters.

flowchart TD
    A[AI Feature Change\nmodel / prompt / config] --> B{Feature Flag\nEvaluation}

    B -->|0% rollout| C[Old Behavior\nfor all users]
    B -->|5% rollout| D[New Behavior\nfor 5% of users]
    B -->|100% rollout| E[New Behavior\nfor all users]

    D -->|Metrics look good| F[Increase to 25%]
    D -->|Metrics look bad| G[Instant rollback\nto 0%]
    F -->|Metrics look good| H[Increase to 100%]
    F -->|Metrics look bad| G

    G -->|Takes effect in| I[< 60 seconds\nno redeploy]

    style G fill:#7f1d1d,color:#fca5a5
    style I fill:#065f46,color:#6ee7b7

A Taxonomy of AI Feature Flags

Not all AI flags are the same. Mixing them up leads to either over-engineering or missing the ones that matter.

Flag Type	What it controls	When to use it
Model toggle	Which LLM model handles a task	Model migrations, cost optimizations, provider failover
Prompt toggle	Which system prompt / prompt template is active	Any prompt change, A/B testing, prompt experiments
Rollout percentage	What fraction of users see new AI behavior	Gradual migration, de-risking changes
Kill switch	Entire AI feature on/off	Emergency disable, degraded mode
User segment	New AI behavior for specific users/tenants	Beta users, internal testing, enterprise pilot

You need all five. Teams that only implement kill switches miss the gradual rollout pattern that prevents most incidents. Teams that only implement rollout percentages have no emergency lever.

The Implementation: Spring Boot + Unleash

We use Unleash for flag management — open source, self-hostable, strong Java SDK. The patterns below translate directly to LaunchDarkly, Split, Flagsmith, or even a simple database-backed flag store if you're starting out.

<!-- pom.xml -->
<dependency>
    <groupId>io.getunleash</groupId>
    <artifactId>unleash-client-java</artifactId>
    <version>9.2.0</version>
</dependency>

@Configuration
public class UnleashConfig {

    @Bean
    public Unleash unleash(
            @Value("${unleash.url}") String url,
            @Value("${unleash.api-token}") String token,
            @Value("${spring.application.name}") String appName) {

        UnleashConfig config = UnleashConfig.newBuilder()
                .appName(appName)
                .instanceId(System.getenv().getOrDefault("POD_NAME", "local"))
                .unleashAPI(url)
                .apiKey(token)
                .fetchTogglesInterval(10)   // poll every 10 seconds
                .synchronousFetchOnInitialisation(true)
                .build();

        return new DefaultUnleash(config);
    }
}

The fetchTogglesInterval(10) is important: flag changes propagate to all running pods within 10 seconds without a redeploy. That's your 60-second kill switch window.

The AI Flag Service

All AI feature flag checks go through a single service — never scattered inline unleash.isEnabled() calls throughout the codebase.

@Service
@Slf4j
public class AIFeatureFlags {

    private final Unleash unleash;

    // ── Model selection ──────────────────────────────────────────────────────

    public String summaryModel(String userId) {
        UnleashContext ctx = context(userId);
        if (unleash.isEnabled("ai.summary.use-70b-model", ctx)) {
            return "llama-3.1-70b-versatile";
        }
        return "llama-3.1-8b-instant";
    }

    public String classifyModel(String userId) {
        UnleashContext ctx = context(userId);
        if (unleash.isEnabled("ai.classify.use-claude-haiku", ctx)) {
            return "claude-haiku-4-5-20251001";
        }
        return "llama-3.1-8b-instant";
    }

    // ── Prompt selection ─────────────────────────────────────────────────────

    public String summaryPromptVariant(String userId) {
        UnleashContext ctx = context(userId);
        // A/B test: "control" vs "variant-a" vs "variant-b"
        Variant variant = unleash.getVariant("ai.summary.prompt-variant", ctx);
        return variant.isEnabled() ? variant.getName() : "control";
    }

    // ── Kill switches ────────────────────────────────────────────────────────

    public boolean isSummaryEnabled(String userId) {
        return unleash.isEnabled("ai.summary.enabled", context(userId));
    }

    public boolean isClassifyEnabled(String userId) {
        return unleash.isEnabled("ai.classify.enabled", context(userId));
    }

    // ── Helpers ──────────────────────────────────────────────────────────────

    private UnleashContext context(String userId) {
        return UnleashContext.newBuilder()
                .userId(userId)
                .addProperty("environment", System.getenv().getOrDefault("SPRING_PROFILES_ACTIVE", "local"))
                .build();
    }
}

Every flag check passes a UnleashContext with the user ID. This is what makes percentage rollouts and user-segment targeting work — Unleash uses the user ID to hash-consistently assign the same user to the same variant across requests.

Pattern 1: Model Migration Flag

You're running llama-3.1-8b-instant and want to migrate to claude-haiku-4-5 for better extraction quality. The naive approach: update the constant, deploy, pray.

The flag approach:

@Service
@Slf4j
public class DocumentExtractor {

    private final ChatClient chatClient;
    private final AIFeatureFlags flags;
    private final MeterRegistry metrics;

    public ExtractionResult extract(String documentText, String userId) {
        String model = flags.extractModel(userId);

        log.debug("extraction model={} userId={}", model, userId);

        Timer.Sample timer = Timer.start(metrics);

        ChatResponse response = chatClient
                .prompt()
                .system(EXTRACTION_SYSTEM_PROMPT)
                .user(documentText)
                .options(ChatOptionsBuilder.builder().model(model).build())
                .call()
                .chatResponse();

        timer.stop(metrics.timer("ai.extract.latency", "model", model));

        // Model is tagged on every metric — you can compare in Grafana
        metrics.counter("ai.extract.tokens.input",  "model", model)
               .increment(response.getMetadata().getUsage().getPromptTokens());
        metrics.counter("ai.extract.tokens.output", "model", model)
               .increment(response.getMetadata().getUsage().getGenerationTokens());

        return parseResponse(response);
    }
}

In Unleash, configure ai.extract.use-haiku:

Start at 0% rollout
Ramp to 5% — watch P95 latency and error rate by model in Grafana
Ramp to 25%, 50%, 100% as metrics look good
If anything degrades — flip to 0% in the Unleash UI. No redeploy, propagates in 10 seconds

flowchart LR
    subgraph "Week 1"
        A[0% rollout\nBaseline metrics]
    end
    subgraph "Week 2"
        B[5% rollout\nWatch latency + cost]
    end
    subgraph "Week 3"
        C[25% rollout\nWatch quality metrics]
    end
    subgraph "Week 4"
        D[100% rollout\nMigration complete]
    end

    A -->|Metrics OK| B
    B -->|Metrics OK| C
    C -->|Metrics OK| D
    B -->|Issue found| E[Rollback to 0%\nin 10 seconds]
    C -->|Issue found| E

    style E fill:#7f1d1d,color:#fca5a5
    style D fill:#065f46,color:#6ee7b7

What to compare: latency P95 per model, error rate per model, token cost per model, and — critically — your quality metric (RAGAS score, human thumbs up rate, or whatever you use). The migration is complete when the new model is better or equivalent on all of them.

Pattern 2: Prompt A/B Testing

A new system prompt is not less risky than a code change. It deserves the same scientific treatment.

@Component
public class SummaryPromptRegistry {

    private static final Map<String, String> PROMPTS = Map.of(

        "control", """
            Summarize the following document in under 150 words.
            Focus on the key decisions, numbers, and outcomes.
            """,

        "variant-a", """
            You are a senior analyst. Summarize the following document for an executive
            who has 30 seconds to read it. Lead with the most important insight.
            Use bullet points for supporting facts. 150 words maximum.
            """,

        "variant-b", """
            Summarize this document. Structure your response as:
            HEADLINE: One sentence capturing the core finding.
            KEY FACTS: 3 bullet points, each under 20 words.
            CONTEXT: One sentence on why this matters.
            """
    );

    public String get(String variant) {
        return PROMPTS.getOrDefault(variant, PROMPTS.get("control"));
    }
}

@Service
public class SummaryService {

    private final AIFeatureFlags flags;
    private final SummaryPromptRegistry prompts;
    private final ChatClient chatClient;
    private final MeterRegistry metrics;

    public SummaryResult summarize(String content, String userId) {
        String variant = flags.summaryPromptVariant(userId);    // "control" | "variant-a" | "variant-b"
        String prompt  = prompts.get(variant);

        ChatResponse response = chatClient.prompt()
                .system(prompt)
                .user(content)
                .call()
                .chatResponse();

        // Tag every quality signal with the variant — A/B analysis in Grafana
        metrics.counter("ai.summary.requests", "variant", variant).increment();

        return new SummaryResult(
            response.getResult().getOutput().getContent(),
            variant  // return variant so callers can record user satisfaction per-variant
        );
    }
}

In Unleash, configure ai.summary.prompt-variant as a Variants toggle:

control — 50% of users
variant-a — 25% of users
variant-b — 25% of users

After 7 days and sufficient sample size, compare thumbs-up rate per variant in your analytics. Promote the winner to 100% by setting all weight to the winning variant, then eventually replace control with it.

[!NOTE]
Tag every downstream metric — user satisfaction scores, task completion rates, support tickets — with the prompt variant. The quality metric that matters is business outcome, not LLM eval score.

Pattern 3: Graceful Degradation via Kill Switch

When an AI feature breaks, you need to serve something useful while you fix it. A kill switch paired with a fallback path is the pattern.

@Service
public class SmartSearchService {

    private final AIFeatureFlags     flags;
    private final AISearchService    aiSearch;       // semantic vector search
    private final KeywordSearchService keywordSearch; // traditional Elasticsearch

    public SearchResults search(SearchQuery query, String userId) {
        if (!flags.isAiSearchEnabled(userId)) {
            // AI search is killed — fall back to keyword search
            log.info("ai-search disabled — falling back to keyword search userId={}", userId);
            return keywordSearch.search(query);
        }

        try {
            return aiSearch.search(query);
        } catch (Exception e) {
            // Unexpected failure — fall back and alert, but don't surface the error to users
            log.error("ai-search failed unexpectedly — falling back: {}", e.getMessage());
            metrics.counter("ai.search.fallback", "reason", "exception").increment();
            return keywordSearch.search(query);
        }
    }
}

The kill switch gives you three modes:

Normal — AI search for everyone
Partial kill — AI search for 0% of users (flag disabled), keyword search for all. No service degradation, just capability downgrade.
Automatic fallback — AI search throws an unexpected error, keyword search runs transparently

flowchart TD
    A[Search Request] --> B{flags.isAiSearchEnabled?}
    B -->|true| C[AI Semantic Search]
    B -->|false — flag killed| D[Keyword Fallback]
    C -->|Success| E[Return AI Results]
    C -->|Exception| F[Log + Alert]
    F --> D
    D --> G[Return Keyword Results]

    style B fill:#1e3a5f,color:#7dd3fc
    style D fill:#374151,color:#d1d5db
    style F fill:#7f1d1d,color:#fca5a5

The key design rule: the fallback path must exist before you ship the AI feature. "We'll add fallback later" means you have no kill switch on day one — exactly when you're most likely to need it.

Pattern 4: User-Segment Targeting for Beta Rollouts

Before a public rollout, you want your internal team (and willing beta customers) to dog-food the feature.

@Service
public class AIFeatureFlags {

    // Internal users get new AI features first
    public boolean isNewExtractorEnabled(String userId, Set<String> userRoles) {
        UnleashContext ctx = UnleashContext.newBuilder()
                .userId(userId)
                .addProperty("role", String.join(",", userRoles))
                .build();
        return unleash.isEnabled("ai.extractor.v2", ctx);
    }
}

In Unleash, configure ai.extractor.v2 with activation strategies:

UserWithId — list specific internal user IDs: always enabled for the team
GradualRolloutUserId — 5% of everyone else

This means your engineers and internal QA users see the new extractor immediately. External users get it at 5% rollout. You run this for 2 weeks, then ramp external users to 25%, 50%, 100%.

// Usage in a service — no change needed as rollout percentage increases
public ExtractionResult extract(Document doc, Authentication auth) {
    String userId = auth.getName();
    Set<String> roles = auth.getAuthorities().stream()
            .map(GrantedAuthority::getAuthority)
            .collect(Collectors.toSet());

    if (!flags.isNewExtractorEnabled(userId, roles)) {
        return legacyExtractor.extract(doc);
    }
    return newExtractor.extract(doc);
}

Observability: Linking Flag State to Metrics

Flags are useless without observability. Every request that touches an AI feature should emit metrics tagged with the active flag variant. This is what makes "is variant-A performing better than control?" answerable in Grafana.

@Aspect
@Component
public class AIFlagMetricsAspect {

    private final MeterRegistry metrics;
    private final AIFeatureFlags flags;

    @Around("@annotation(TrackedAIFeature)")
    public Object trackWithFlagContext(ProceedingJoinPoint pjp) throws Throwable {
        // Extract userId from first String argument (by convention)
        String userId = Arrays.stream(pjp.getArgs())
                .filter(a -> a instanceof String)
                .map(a -> (String) a)
                .findFirst().orElse("unknown");

        String model   = flags.summaryModel(userId);
        String variant = flags.summaryPromptVariant(userId);

        Timer.Sample timer = Timer.start(metrics);
        try {
            Object result = pjp.proceed();
            timer.stop(metrics.timer("ai.feature.duration",
                    "model", model, "variant", variant, "status", "ok"));
            return result;
        } catch (Throwable t) {
            timer.stop(metrics.timer("ai.feature.duration",
                    "model", model, "variant", variant, "status", "error"));
            throw t;
        }
    }
}

In Grafana, you now have a single dashboard panel that shows latency by model and variant — so model migration and prompt A/B testing are both visible without extra instrumentation.

The four queries you always want:

# Error rate by variant
rate(ai_feature_duration_seconds_count{status="error"}[5m]) 
  / rate(ai_feature_duration_seconds_count[5m])
  by (variant)

# P95 latency by model  
histogram_quantile(0.95, 
  rate(ai_feature_duration_seconds_bucket[5m]))
  by (model, le)

# Cost by model (tokens × price)
rate(ai_tokens_input_total[1h]) by (model)

# Rollout progress (% of requests per variant)
rate(ai_feature_duration_seconds_count[5m]) by (variant)
  / ignoring(variant) group_left
  sum(rate(ai_feature_duration_seconds_count[5m]))

The Three Disasters We Avoided

Disaster 1 — Model degradation on a specific doc type. Our summarization model started hallucinating numbers in financial documents after a provider-side update. Kill switch disabled AI summarization for affected document categories within 45 seconds of the on-call alert. Users saw the keyword-based fallback. No customer data corrupted.

Disaster 2 — Prompt regression. A new prompt that performed well in offline evals caused a 12% drop in thumbs-up rate in production. We caught it at 5% rollout (affecting ~800 users) instead of rolling it to all 16,000. Rolled back to control variant in the Unleash UI.

Disaster 3 — Cost spike. A batch job that used a new prompt sent to 100% of traffic consumed 4× the expected tokens due to an accidental context window issue. The cost alert fired at $180 (2× rolling average). Kill switch → model flag reverted to the cheaper model → cost normalized in under 2 minutes.

In all three cases: no redeploy, no hotfix PR, no incident spanning multiple engineers. One person, one flag change, one minute.

Anti-Patterns

Checking flags at the wrong layer. Putting flag checks inside the LLM integration class couples your infrastructure to your feature logic. Put flags in the service layer, let the LLM client be a dumb executor.

One flag for the whole AI system. A single ai.enabled kill switch means you can't surgically kill summarization while keeping classification running. One flag per feature, or per feature + model combination.

No fallback for the kill switch. A kill switch that returns an error instead of a fallback response is just a way to 500 your users more deterministically. The fallback must exist before the flag.

Forgetting to tag metrics with flag state. If your metrics don't tell you which variant produced which latency, you're flying blind. Tag everything.

Long-lived feature branches. Don't keep AI behavior in a Git branch while waiting for the flag rollout to complete. Merge to main, deploy behind a 0% flag. The code is in production; the behavior is off. This is trunk-based development for AI.

Starting Simple: The Minimum Viable AI Flag

If you're not ready for Unleash or LaunchDarkly, this works:

// Minimal flag store — a database table, or even a config map
@Component
public class SimpleAIFlags {

    @Value("${ai.summary.enabled:true}")
    private boolean summaryEnabled;

    @Value("${ai.summary.model:llama-3.1-8b-instant}")
    private String summaryModel;

    @Value("${ai.summary.rollout-percentage:100}")
    private int rolloutPercentage;

    public boolean isSummaryEnabled(String userId) {
        if (!summaryEnabled) return false;
        // Deterministic hash-based assignment so the same user always gets the same experience
        return Math.abs(userId.hashCode() % 100) < rolloutPercentage;
    }

    public String getSummaryModel() { return summaryModel; }
}

With Spring Boot Actuator, you can update @Value properties at runtime via /actuator/env + /actuator/refresh (with @RefreshScope). That's your 60-second kill switch without any external system.

Start here. Add Unleash when you have more than 3 AI features and more than one person managing flag changes.

Key Takeaways

AI features fail silently. Flags let you catch silent failures at 5% before they affect 100% of users.
Prompt changes are deployments. Treat them with the same rigor: staging, gradual rollout, observability, rollback.
Build the fallback first. A kill switch that errors instead of degrading gracefully is worse than no kill switch.
Tag every metric with flag state. Without this, you can't answer "is the new model better?" from production data.
Model providers change under you. Your flag infrastructure needs to handle changes you didn't trigger, not just changes you did.
The kill switch window matters. 60 seconds to production effect (flag change → all pods updated) is the target. Anything above 5 minutes is too slow for an active incident.

The engineering investment is modest — a few days to wire up Unleash and build the flag service. The return is the ability to ship AI features confidently and fix them without waking up your team at 3AM.

The next part of this series covers AI observability specifically: the metrics, traces, and dashboards that make "what exactly did the model return for user X?" answerable in under 30 seconds. If you're already doing feature flags and want to pair them with tighter observability, that's where to look next.

Event-Driven AI: How Kafka Streams Changed the Way We Run LLMs in Production

Avaneesh Yadav — Mon, 31 Aug 2026 15:31:32 +0000

Last October our AI service handled ~50K LLM requests per day. By February that number hit 800K. By April, 2 million. The synchronous HTTP architecture we had shipped in three weeks didn't survive contact with that growth. We were dropping requests, hitting rate limits that cascaded into total outages, and on-call engineers were getting paged at 3AM for timeouts that shouldn't have been possible to timeout.

The fix wasn't bigger timeouts or faster hardware. It was rethinking the execution model entirely — from request-response to event-driven. Here's the full architecture, the Spring AI + Kafka code, the failure modes we hit, and the production numbers that came out the other side.

Why Synchronous LLM Calls Break at Scale

The typical AI feature looks like this: a user does something, your API calls an LLM, the LLM responds, you return the result. Simple, works fine in demos, blows up in production.

LLMs are slow and variable. A fast response is 300ms. A slow one — complex reasoning, long context, throttled provider — is 15–30 seconds. Most HTTP infrastructure timeouts sit at 30 seconds, which means you're living on the edge constantly.

Rate limits cascade. Your provider gives you X requests per minute. Under load, one failed batch causes retries that burn more quota, which causes more failures. Exponential backoff helps but doesn't fix the underlying problem: too many concurrent callers sharing a quota.

No buffering. If the downstream LLM is temporarily rate-limited or degraded, you have no place to hold work. You either drop the request or return an error to the user.

Problem	Sync Architecture	Event-Driven
Timeout handling	30s hard limit → user sees error	Message retained → retried after backoff
Rate limit response	HTTP 429 → cascading upstream 500s	Consumer pauses → catches up when quota resets
Traffic spike	Thread pool exhausts → 503s	Kafka absorbs burst → smooth processing
Provider degradation	All in-flight requests fail simultaneously	New messages queue; consumer retries at its own pace
Cost visibility	Invisible per-request cost	Per-consumer-group token metrics

The Event-Driven AI Model

The core idea: instead of making LLM calls synchronously in the API handler, publish work to a Kafka topic, consume it in a dedicated AI worker, and emit results to a results topic. Callers subscribe to results.

flowchart TD
    A[API Gateway] -->|POST /enrich| B[Request Handler]
    B -->|Publish AITask event| C[(ai.tasks.summarize)]
    B -->|202 Accepted + taskId| D[Client]

    C -->|Consumer Group — 8 threads| E[AI Worker Pool]
    E -->|Rate-limited call| F[LLM Provider]
    F -->|Response| E
    E -->|Publish AIResult| G[(ai.results.completed)]

    G -->|Subscribe| H[Result Handler]
    H -->|Persist| I[(PostgreSQL)]
    H -->|WebSocket push| D

    style C fill:#1e3a5f,color:#7dd3fc
    style G fill:#1e3a5f,color:#7dd3fc
    style F fill:#374151,color:#d1d5db

The client immediately gets a 202 Accepted with a task ID. The result arrives when the LLM call completes — via WebSocket, SSE, or polling. This separation is what unlocks scale: the HTTP layer and the LLM processing layer evolve independently and fail independently.

[!NOTE]
This is the same pattern as async email sending, async PDF generation, or async image processing. It's been standard for I/O-heavy work for a decade. LLMs are just I/O with a 10-second P99.

Topic Design: What Goes Where

Getting the Kafka topic structure right matters more than any individual line of code. Topics are your API contract — they're hard to change once services depend on them.

flowchart TD
    subgraph Input ["Task Topics (by type)"]
        A[ai.tasks.summarize]
        B[ai.tasks.classify]
        C[ai.tasks.extract]
    end

    subgraph Priority ["Priority Routing"]
        E[High-Priority Consumer Group\n8 threads]
        F[Batch Consumer Group\n2 threads]
    end

    subgraph Output ["Result Topics"]
        G[ai.results.completed]
        H[ai.results.failed]
        I[ai.tasks.classify.dlq]
    end

    A --> E
    B --> E
    C --> F

    E --> G
    E --> H
    F --> G
    F --> H
    H -->|After 3 retries| I

    style E fill:#065f46,color:#6ee7b7
    style F fill:#1e3a5f,color:#7dd3fc
    style I fill:#7f1d1d,color:#fca5a5

Design principles we landed on:

One topic per task type, not one monolithic ai-tasks topic. Lets you scale consumers independently, set different retention policies (real-time classification results don't need 7-day retention; completed summarizations do), and gives you clear consumer lag metrics per workload.

Priority tiers as separate consumer groups, not topics. Eight consumer threads watch the high-priority topics; two watch the batch topics. Simple to operate — no complex priority queue logic or message reordering.

Separate DLQ per task type (ai.tasks.classify.dlq, not a shared DLQ). When debugging production failures you already know the task type — a shared DLQ forces you to filter through noise from every other failing type simultaneously.

Results topic as a public contract. Any service can subscribe to ai.results.completed and react to completed AI tasks. Loose coupling at its best — the AI worker doesn't need to know who cares about the results.

@Configuration
public class AITopicConfig {

    public static final String TASKS_SUMMARIZE   = "ai.tasks.summarize";
    public static final String TASKS_CLASSIFY    = "ai.tasks.classify";
    public static final String RESULTS_COMPLETED = "ai.results.completed";
    public static final String RESULTS_FAILED    = "ai.results.failed";
    public static final String DLQ_CLASSIFY      = "ai.tasks.classify.dlq";

    @Bean
    public NewTopic summarizeTopic() {
        return TopicBuilder.name(TASKS_SUMMARIZE)
                .partitions(12)   // max 12 concurrent consumers in this group
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "604800000") // 7 days
                .build();
    }

    @Bean
    public NewTopic classifyTopic() {
        return TopicBuilder.name(TASKS_CLASSIFY)
                .partitions(24)   // higher throughput — more partitions
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "86400000")  // 1 day
                .build();
    }

    @Bean
    public NewTopic resultsTopic() {
        return TopicBuilder.name(RESULTS_COMPLETED)
                .partitions(12)
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "2592000000") // 30 days
                .build();
    }
}

Partition count is a ceiling on parallelism. Kafka consumer group parallelism is bounded by partition count — 12 partitions means you can run at most 12 consumer threads per group before threads sit idle. Size for peak throughput, not current throughput. You can't easily add partitions to a live topic without a migration.

The AI Task Schema

Every message uses the same envelope. Consistency is what lets you build generic retry, routing, and observability tooling without per-task-type special cases.

public record AITask(
    String taskId,          // UUID — idempotency key
    String taskType,        // "summarize" | "classify" | "extract"
    String requesterId,     // service or user that submitted
    String priority,        // "high" | "normal" | "batch"
    Instant submittedAt,
    int    attemptNumber,   // 1-based; consumers use this for backoff strategy
    Map<String, Object> payload,   // task-specific input data
    Map<String, String> metadata   // tracing headers, correlation IDs
) {}

public record AIResult(
    String  taskId,
    String  taskType,
    boolean success,
    Instant completedAt,
    long    processingMs,
    String  model,
    int     inputTokens,
    int     outputTokens,
    Object  output,          // task-specific result — Map, String, etc.
    String  errorMessage     // null on success
) {}

Three design decisions worth calling out:

taskId is the idempotency key. Consumers check whether a task has already been processed before calling the LLM. Kafka at-least-once delivery means you will receive duplicates — consumer rebalancing, network glitches, pod restarts. Plan for it from day one.

attemptNumber travels with the message, not just as a consumer variable. This means retry strategy is encoded in the event itself: first attempt = immediate retry, second = 30s backoff, third = DLQ. Works correctly across pod restarts and consumer rebalancing.

Token counts in every result. Essential for cost tracking. Sum inputTokens + outputTokens per consumer group, per model, per hour. That's your cost dashboard with no extra instrumentation.

Spring AI + Kafka: The Consumer

This is the heart of the system — the consumer that actually processes AI tasks.

@Component
@Slf4j
public class AITaskConsumer {

    private final ChatClient         chatClient;
    private final KafkaTemplate<String, AIResult> resultTemplate;
    private final AITaskRepository   taskRepository;   // idempotency store
    private final MeterRegistry      meterRegistry;
    private final AdaptiveRateLimiter rateLimiter;

    @KafkaListener(
        topics       = { AITopicConfig.TASKS_SUMMARIZE, AITopicConfig.TASKS_CLASSIFY },
        groupId      = "ai-worker-high-priority",
        concurrency  = "8",       // 8 consumer threads per pod
        containerFactory = "aiListenerContainerFactory"
    )
    public void consume(
            @Payload AITask task,
            @Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
            Acknowledgment ack) {

        log.info("task={} type={} attempt={}", task.taskId(), task.taskType(), task.attemptNumber());

        // Check idempotency — previous consumer may have already handled this
        if (taskRepository.isCompleted(task.taskId())) {
            log.debug("task={} already completed — skipping (idempotent ack)", task.taskId());
            ack.acknowledge();
            return;
        }

        Timer.Sample timer = Timer.start(meterRegistry);

        try {
            rateLimiter.acquire(); // blocks until a permit is available

            AIResult result = switch (task.taskType()) {
                case "summarize" -> handleSummarize(task);
                case "classify"  -> handleClassify(task);
                default -> throw new IllegalArgumentException("Unknown type: " + task.taskType());
            };

            taskRepository.markComplete(task.taskId(), result); // write idempotency record first
            resultTemplate.send(RESULTS_COMPLETED, task.taskId(), result);
            ack.acknowledge();  // only after result is durably published

            rateLimiter.onSuccess();
            timer.stop(meterRegistry.timer("ai.task.processing",
                    "type", task.taskType(), "status", "success"));

        } catch (RateLimitException e) {
            // Provider returned 429 — don't ack, let Kafka redeliver after consumer pause
            rateLimiter.onRateLimit();
            log.warn("task={} rate-limited (attempt={})", task.taskId(), task.attemptNumber());
            // Intentionally no ack.acknowledge() — message redelivers

        } catch (Exception e) {
            log.error("task={} failed attempt={}: {}", task.taskId(), task.attemptNumber(), e.getMessage());
            if (task.attemptNumber() >= 3) {
                sendToDLQ(task, e.getMessage());
                ack.acknowledge(); // remove from main topic — it's in DLQ now
            }
            // Otherwise no ack — Kafka redelivers with same attempt number
            // (producer should increment attemptNumber on retry, or use a retry topic)
            timer.stop(meterRegistry.timer("ai.task.processing",
                    "type", task.taskType(), "status", "error"));
        }
    }

    private AIResult handleSummarize(AITask task) {
        String content  = (String) task.payload().get("content");
        int    maxWords = (int)    task.payload().getOrDefault("maxWords", 150);

        String prompt = """
            Summarize the following in under %d words.
            Be specific — preserve key facts, numbers, and named entities.

            Content:
            %s
            """.formatted(maxWords, content);

        Instant start    = Instant.now();
        ChatResponse resp = chatClient.prompt().user(prompt).call().chatResponse();
        Usage usage      = resp.getMetadata().getUsage();

        return new AIResult(
            task.taskId(), "summarize", true, Instant.now(),
            Duration.between(start, Instant.now()).toMillis(),
            "llama-3.1-8b-instant",
            (int) usage.getPromptTokens(), (int) usage.getGenerationTokens(),
            Map.of("summary", resp.getResult().getOutput().getContent()),
            null
        );
    }
}

The ack.acknowledge() placement is load-bearing. We only call it after the result is successfully published to the results topic. If the result publish fails — network blip, Kafka broker restart — Kafka will redeliver the task to another consumer thread. Combined with the idempotency check at the top, the task will be processed exactly once even across failures.

Rate limiting happens inside the consumer, not at the API gateway. Your LLM provider doesn't know about your queue depth — it just sees requests arriving. Rate limiting at the consumer level means you never exceed quota regardless of how deep the queue gets.

Handling Backpressure: Adaptive Rate Limiting

The most common failure mode isn't Kafka — it's the LLM provider. When you hit rate limits, the naive response (throw exception, retry immediately) makes things significantly worse. Here's what we do instead:

@Component
public class AdaptiveRateLimiter {

    // Groq's limit at our tier: 30 req/min = 0.5 req/sec
    private final double            maxRate = 0.45; // 10% headroom below actual limit
    private final double            minRate = 0.05; // hard floor when throttled
    private final AtomicDouble  currentRate = new AtomicDouble(0.3);
    private volatile RateLimiter     limiter = RateLimiter.create(0.3);

    public void acquire() { limiter.acquire(); }

    public void onSuccess() {
        double next = Math.min(currentRate.get() * 1.05, maxRate); // +5% on success
        updateLimiter(next);
    }

    public void onRateLimit() {
        double next = Math.max(currentRate.get() * 0.5, minRate); // halve on 429
        updateLimiter(next);
        log.warn("Rate-limited — throttling to {:.2f} req/sec", next);
    }

    private void updateLimiter(double newRate) {
        if (Math.abs(newRate - currentRate.get()) > 0.01) {
            currentRate.set(newRate);
            limiter = RateLimiter.create(newRate);
        }
    }
}

This is AIMD — Additive Increase, Multiplicative Decrease — the same algorithm TCP congestion control uses. On success, increase by 5%. On rate limit, halve. Over time it converges on the sustainable throughput for your provider tier without manual tuning.

flowchart TD
    A[Message dequeued] --> B{Rate limiter\nacquire}
    B -->|Permit granted| C[Call LLM]
    C -->|200 OK| D[Publish result to Kafka]
    D --> E[ack.acknowledge]
    E -->|onSuccess: rate × 1.05| B

    C -->|429 Rate Limited| F[onRateLimit: rate ÷ 2]
    F --> G[No ack — Kafka redelivers]
    G --> A

    C -->|500 or timeout| H{attemptNumber}
    H -->|< 3| G
    H -->|≥ 3| I[sendToDLQ]
    I --> J[ack.acknowledge\nfrom main topic]

    style E fill:#065f46,color:#6ee7b7
    style I fill:#7f1d1d,color:#fca5a5
    style G fill:#1e3a5f,color:#7dd3fc

Dead Letter Queues: Don't Silently Drop AI Work

A DLQ is your safety net for permanent failures. Every task that fails three times ends up there — unprocessed, with the error reason attached as metadata.

private void sendToDLQ(AITask task, String errorMessage) {
    Map<String, String> dlqMeta = new HashMap<>(task.metadata());
    dlqMeta.put("dlq.reason",       errorMessage);
    dlqMeta.put("dlq.sentAt",       Instant.now().toString());
    dlqMeta.put("dlq.finalAttempt", String.valueOf(task.attemptNumber()));

    AITask dlqTask = new AITask(
        task.taskId(), task.taskType(), task.requesterId(), task.priority(),
        task.submittedAt(), task.attemptNumber(), task.payload(), dlqMeta
    );

    String dlqTopic = "ai.tasks." + task.taskType() + ".dlq";
    dlqTemplate.send(dlqTopic, task.taskId(), dlqTask);

    log.error("task={} → DLQ after {} attempts: {}", task.taskId(), task.attemptNumber(), errorMessage);
    meterRegistry.counter("ai.task.dlq", "type", task.taskType()).increment();
}

DLQ messages are retained for 30 days. Twice a week we run a DLQ reprocessing job: identify the root cause, fix the consumer, then replay DLQ messages back into the main topic. Nothing is permanently lost.

[!WARNING]
Do not set DLQ retention too short. A production incident that takes 3 days to diagnose needs the evidence to still be in the DLQ when you're ready to replay. 30 days is a safe default for AI task DLQs.

Exactly-Once Semantics: When You Actually Need Them

Kafka supports exactly-once semantics (EOS) via transactions. For most AI tasks, you don't need them — idempotent consumers plus at-least-once delivery is simpler and sufficient. But for billing events, audit logs, and compliance-critical document processing, true exactly-once matters.

@Bean
public KafkaTransactionManager<String, AIResult> kafkaTxManager(
        ProducerFactory<String, AIResult> pf) {
    return new KafkaTransactionManager<>(pf);
}

// Consumer using Kafka transactions
@Transactional("kafkaTxManager")
@KafkaListener(topics = "ai.tasks.billing-critical", groupId = "ai-billing")
public void consumeExactlyOnce(@Payload AITask task, Acknowledgment ack) {
    // Entire method runs in a Kafka transaction.
    // Either ALL of the following happen atomically, or NONE do:
    //   1. Process the AI task
    //   2. Publish the result event
    //   3. Commit the consumer offset
    AIResult result = processTask(task);
    resultTemplate.send(RESULTS_COMPLETED, task.taskId(), result);
    // offset commits atomically with the produce — true exactly-once
}

The tradeoff: EOS adds roughly 15–20% latency overhead due to transaction coordination. For billing tasks, worth it. For bulk document summarization, not worth it — idempotent at-least-once is the right choice there.

Observability: Tracing AI Events Across Services

An AI task might be submitted by Service A, consumed by AI Worker B, and the result consumed by Service C. When something goes wrong, you need to trace the full journey without manual correlation.

We propagate OpenTelemetry trace context through Kafka headers:

// Producer — inject trace context into message headers
public void submitTask(AITask task) {
    ProducerRecord<String, AITask> record = new ProducerRecord<>(
            TASKS_SUMMARIZE, task.taskId(), task);

    GlobalOpenTelemetry.getPropagators()
            .getTextMapPropagator()
            .inject(Context.current(), record.headers(),
                    (headers, key, val) -> headers.add(key, val.getBytes(UTF_8)));

    kafkaTemplate.send(record);
}

// Consumer — extract trace context and create a child span
@KafkaListener(...)
public void consume(@Payload AITask task, @Headers MessageHeaders headers) {

    Context parentCtx = GlobalOpenTelemetry.getPropagators()
            .getTextMapPropagator()
            .extract(Context.current(), headers, new MessageHeadersGetter());

    Span span = tracer.spanBuilder("ai.task.process")
            .setParent(parentCtx)
            .setAttribute("task.id",    task.taskId())
            .setAttribute("task.type",  task.taskType())
            .startSpan();

    try (Scope ignored = span.makeCurrent()) {
        AIResult result = processTask(task);
        span.setAttribute("tokens.input",  result.inputTokens());
        span.setAttribute("tokens.output", result.outputTokens());
        span.setAttribute("latency.ms",    result.processingMs());
    } finally {
        span.end();
    }
}

In Grafana Tempo, the complete trace now shows: HTTP POST /enrich → Kafka publish → ai.task.process (the LLM call spans inside) → Kafka produce result → result consumer → database write. The entire 8-second LLM call is one contiguous trace chain.

Key metrics per consumer group:

Metric	What it reveals
`ai.task.processing` timer	P50/P95/P99 latency per task type
`kafka.consumer.lag`	Queue depth — how far behind consumers are
`ai.tokens.input` / `ai.tokens.output`	Cost per model per hour
`ai.task.dlq` counter	Permanent failure rate — should be near zero
`ai.rate_limiter.throttle_count`	How often you're hitting provider limits
`ai.task.idempotent.skip_count`	Duplicates correctly filtered (sanity check)

Configuration That Actually Matters

spring:
  kafka:
    bootstrap-servers: ${KAFKA_BROKERS}
    consumer:
      group-id: ai-worker-high-priority
      auto-offset-reset: earliest
      enable-auto-commit: false          # CRITICAL — never use auto-commit for AI tasks
      max-poll-records: 5                # Don't overwhelm a slow consumer
      max-poll-interval-ms: 120000       # 2 min — LLM calls can be slow
      fetch-min-bytes: 1
    producer:
      acks: all                          # Wait for all in-sync replicas
      retries: 3
      enable-idempotence: true
    listener:
      ack-mode: MANUAL_IMMEDIATE
      concurrency: 8

# Tune these per your LLM provider tier
ai:
  rate-limiter:
    max-rate-per-second: 0.45           # 27 req/min — 10% below Groq's 30 req/min limit
    min-rate-per-second: 0.05
  retry:
    max-attempts: 3
    dlq-retention-days: 30

max-poll-interval-ms is the one that bites everyone. If a consumer takes longer than this value between poll() calls, Kafka considers it dead and triggers a rebalance — even if the consumer is actively processing a long LLM call. We set it to 120 seconds (double our P99 LLM latency) to avoid unnecessary rebalancing during legitimate slow calls.

max-poll-records: 5 limits how many messages a single consumer thread fetches in one batch. With 8 threads each holding 5 messages, you have 40 LLM calls potentially in-flight per pod. Size this to your rate limit headroom.

The Cost Breakdown

After 90 days running this architecture against ~2M requests/day:

Model	Task Type	Avg Input Tokens	Avg Output Tokens	Input $/1M	Output $/1M	Daily Cost
llama-3.1-8b-instant	Classification	450	25	$0.05	$0.08	$47
llama-3.1-70b	Summarization	800	180	$0.59	$0.79	$380
claude-haiku-4-5	Extraction	1,200	400	$0.80	$4.00	$260
Total						$687/day

The event-driven architecture contributed two cost wins that weren't possible with synchronous calls:

Intelligent model routing. The consumer reads task.payload().get("complexity") — a simple score computed at task submission — and routes simple classifications to llama-3.1-8b-instant instead of a 70B model. This cut our average cost-per-task by 31%.

Micro-batching. For task types where the model supports batch input, we accumulate messages in 100ms windows and send them as a single request. 40% reduction in per-call API overhead (connection setup, auth handshake, response parsing). Impossible to do cleanly in a synchronous request handler without blocking the caller.

[!NOTE]
$687/day sounds large. At 2M requests that's $0.00034 per request — roughly $20K/month. Running equivalent workloads on self-hosted GPU infrastructure (A100s, realistic utilization rates, ops overhead) would cost us approximately $2,100/day. The managed API is cheaper at our scale.

Before and After: Production Numbers

We ran both architectures in parallel for 30 days — old sync on 20% of traffic, new event-driven on 80% — before cutting over completely.

Metric	Sync (before)	Event-Driven (after)	Change
P50 user-visible latency	4.2 s	0.8 s (202 + async push)	−81%
P99 user-visible latency	28.4 s	1.1 s	−96%
Request timeout rate	3.2%	0.001%	−99.97%
Requests dropped (unrecoverable)	1.8%	0.02%	−99%
Cost per 1M requests	$420	$344	−18%
On-call pages per week	14	1	−93%

The P99 number matters most. Users hitting a 28-second timeout perceived the feature as broken, even if only 3% of requests reached that threshold. With async dispatch and WebSocket delivery, P50 user-visible latency is under 1 second (the 202 response arrives instantly) and the result appears as a push notification when the LLM finishes.

The 93% reduction in on-call pages was unexpected. Most of the 14 weekly pages were timeout-related cascades. When you remove synchronous coupling between the API layer and the LLM layer, most cascade failure modes disappear entirely.

When NOT to Go Event-Driven

This architecture adds real operational complexity. It's not always the right answer.

Stay synchronous when:

The result must block the response. Real-time payment validation, auth decisions, interactive AI chat where the user is watching a streaming response. If you can't send a 202, you need sync.
Volume is under ~10K requests/day. The operational overhead of Kafka, consumer groups, DLQ management, and result delivery infrastructure doesn't pay off at low volume. A simple async HTTP call with a 30-second timeout and a Postgres-backed retry queue is the right choice.
You don't have result delivery infrastructure. WebSockets, SSE, or polling endpoints are meaningful engineering work. If you're not willing to build that, your users will stare at a spinner indefinitely. "We'll add the result delivery later" becomes "we added Kafka but the feature still feels broken."
Your team is new to Kafka. Consumer group rebalancing, partition assignment, exactly-once transactions, DLQ management — these are non-trivial operational concerns. Kafka is operationally complex when something goes wrong. Make sure your team has the skills before you take the dependency.

The decision framework:

flowchart TD
    A{Is LLM call in the\ncritical response path?} -->|Yes| B{Can user wait\n>2 seconds?}
    A -->|No, background work| E[Go event-driven]

    B -->|No| C[Stay synchronous\nuse streaming SSE]
    B -->|Yes| D{Volume > 50K/day?}

    D -->|No| F[Synchronous with\nretry queue in Postgres]
    D -->|Yes| G{Team has\nKafka experience?}

    G -->|No| H[Start sync, plan\nKafka migration]
    G -->|Yes| E

    style E fill:#065f46,color:#6ee7b7
    style C fill:#1e3a5f,color:#7dd3fc
    style F fill:#374151,color:#d1d5db

The Minimum Viable Starting Point

If you're starting fresh and want to validate the pattern before building the full architecture, here's the minimum working setup:

# application.yml — minimum for correctness
spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: ai-worker
      auto-offset-reset: earliest
      enable-auto-commit: false      # never auto-commit
      max-poll-interval-ms: 90000
    listener:
      ack-mode: MANUAL_IMMEDIATE

// Minimum viable consumer — add rate limiting, DLQ, and idempotency incrementally
@KafkaListener(topics = "ai.tasks", groupId = "ai-worker")
public void consume(@Payload AITask task, Acknowledgment ack) {
    try {
        AIResult result = callLLM(task);
        resultTemplate.send("ai.results", task.taskId(), result);
        ack.acknowledge();
    } catch (Exception e) {
        log.error("Task {} failed: {}", task.taskId(), e.getMessage());
        // Don't ack — Kafka redelivers
        // Add DLQ logic here after you've observed retry behavior in staging
    }
}

Start here. Add the adaptive rate limiter when you first see 429s in production. Add DLQs when you see messages redelivered indefinitely. Add idempotency when you see duplicate results. Add distributed tracing when debugging becomes painful. The architecture I described above is 18 months of production failures distilled — you don't need all of it on day one.

Key Takeaways

Synchronous LLM calls are fine until they're not. The failure mode is sudden and cascading — build the escape hatch before you need it. The warning signs are rising P99 latency and rate limit errors.
Kafka absorbs traffic spikes that would kill a synchronous system. The queue is the buffer between your traffic patterns and your provider's rate limits. You can't add that buffer to a sync architecture without fundamentally restructuring it.
Idempotency is non-negotiable. At-least-once delivery is guaranteed by Kafka; idempotent consumers are your responsibility. An idempotency check at the top of every consumer is not optional overhead — it's correctness.
Rate limiting belongs in the consumer, not the gateway. Only the consumer knows what's actually in flight against the LLM provider. Gateway-level rate limiting gives you no visibility into queue depth.
Token counts in every result event. You cannot optimize what you cannot see. Per-model, per-task-type token counts are the only way to know which workloads are driving your LLM bill.
DLQs are not optional. Every permanently failed task needs a destination. "Log it and move on" means you've silently dropped user work. DLQ + replay gives you a recovery path that sync architectures simply don't have.
The biggest win isn't latency — it's reliability. P99 improvements are impressive. Getting paged 93% less often at 3AM is what actually changes how your team feels about owning the service.

The investment is real — probably 6–8 weeks of solid engineering to get this fully operational. In our case, it was worth it the first Saturday night we didn't get paged.

The next post in this series covers stateful aggregation with Kafka Streams — specifically windowed token-count aggregations for real-time cost alerts and per-tenant quota enforcement. If you're hitting a different failure mode than I described here, or building something similar, I'd love to hear about it.

Your Spring Boot App Starts in 3 Seconds. With GraalVM Native, It's 80ms.

Avaneesh Yadav — Mon, 31 Aug 2026 11:16:18 +0000

Every Spring Boot developer has watched the console scroll through startup lines and thought: this feels slow. For a monolith on a beefy server, 3–5 seconds is fine. But when you're paying per-millisecond on AWS Lambda, scaling pods on Kubernetes, or trying to pass a 2-second health-check SLA, that startup time becomes a real cost.

GraalVM Native Image solves this by compiling your entire application — Spring, your code, and the JDK — into a single native binary ahead of time. The result: sub-100ms startup, a fraction of the memory footprint, and a container image under 100MB.

Spring Boot 3 added first-class native support. In 2026, the tooling has matured to the point where most real-world Spring Boot apps can go native without heroic effort. This post walks through what actually happens under the hood, the pitfalls that will bite you, and honest benchmarks so you know what you're getting.

How GraalVM Native Image Actually Works

The JVM is a dynamic runtime. Classes are loaded lazily, reflection resolves types at runtime, proxies are generated on the fly, and the JIT compiler optimizes hot paths after they've been called thousands of times. All of that flexibility comes at a cost: startup.

GraalVM's Native Image compiler inverts this model. At build time it performs a closed-world analysis: starting from your main method, it traces every reachable class, method, and field, and compiles everything into a native binary using ahead-of-time (AOT) compilation. Nothing is loaded lazily. The JIT is gone. What you get is a self-contained executable that starts like a Go binary.

flowchart TD
    src["📄 Source Code (Java)"]
    compile["⚙️ javac Compile"]
    bytecode["📦 Bytecode (.class files)"]
    aot["🔍 Spring AOT Processor"]
    hints["📋 Reflection Hints + Proxy Configs"]
    native["🔨 GraalVM native-image"]
    binary["🚀 Native Binary (OS-specific)"]

    src --> compile --> bytecode
    bytecode --> aot
    aot --> hints
    bytecode --> native
    hints --> native
    native --> binary

The key challenge: anything that relies on runtime dynamism — reflection, dynamic proxies, runtime resource loading — must be declared at build time so the compiler knows to include it. That's the source of most native-image pain.

Spring Boot 3's AOT Processing

Spring Framework 5 and earlier were built on reflection everywhere. Spring Boot 3 (Spring Framework 6) was the first version designed with native in mind. It added an AOT engine that runs at build time and:

Generates source code for bean definitions instead of registering them dynamically
Pre-computes @Configuration class processing
Generates reflection hints for every Spring component
Registers proxies for @Transactional, @Async, @Cacheable, and other AOP annotations

When you run spring-boot:build-image with the native profile, Spring's AOT processor runs first, then its output feeds into the GraalVM compiler. You get native compatibility essentially for free on the Spring layer — the hard part is your own code and third-party libraries.

Setting Up the Build

Prerequisites

GraalVM for JDK 21 or later (or use the GraalVM Maven Toolchain)
Spring Boot 3.2+ (3.3+ recommended)
The spring-boot-starter parent or native-maven-plugin

Maven Setup

Add the native profile to your pom.xml:

<profiles>
  <profile>
    <id>native</id>
    <build>
      <plugins>
        <plugin>
          <groupId>org.graalvm.buildtools</groupId>
          <artifactId>native-maven-plugin</artifactId>
          <extensions>true</extensions>
          <executions>
            <execution>
              <id>build-native</id>
              <goals><goal>compile-no-fork</goal></goals>
              <phase>package</phase>
            </execution>
          </executions>
          <configuration>
            <imageName>${project.artifactId}</imageName>
            <buildArgs>
              <buildArg>--no-fallback</buildArg>
              <buildArg>-O2</buildArg>
            </buildArgs>
          </configuration>
        </plugin>
      </plugins>
    </build>
  </profile>
</profiles>

Build the native binary:

./mvnw -Pnative native:compile

Build a container image (no GraalVM needed on the host — uses Buildpacks):

./mvnw -Pnative spring-boot:build-image

Gradle Setup

plugins {
  id 'org.graalvm.buildtools.native' version '0.10.3'
}

graalvmNative {
  binaries {
    main {
      imageName = 'my-service'
      buildArgs.add('--no-fallback')
    }
  }
}

./gradlew nativeCompile

Real Benchmark Numbers

I compiled a realistic Spring Boot service — REST endpoints, JPA with PostgreSQL, Spring Security, a scheduled job, and Redis caching — and measured cold start across modes.

Mode	Startup Time	RSS Memory (idle)	Container Image Size
JVM (Java 21)	3.2s	310 MB	280 MB
JVM + CDS	1.8s	270 MB	285 MB
Native Image	78ms	62 MB	98 MB

These numbers are real. RSS will vary based on your workload — once the app handles traffic, the JVM typically catches up on throughput due to JIT optimization. But for startup and idle memory, native wins by a factor of 4–5x.

The container image size difference matters most in Kubernetes. Pulling a 98 MB image on a fresh node is fast. Pulling 280 MB when you're trying to scale up fast under traffic is not.

The Pitfalls (And How to Fix Them)

1. Reflection That You Don't Know About

The most common failure mode: a ClassNotFoundException or NoSuchMethodException at runtime in your native binary, which ran fine on the JVM.

Why it happens: You (or a library) called Class.forName("com.example.SomeThing") or ObjectMapper.readValue(json, SomeThing.class). The closed-world compiler didn't see that reference statically, so it excluded the class.

Fix: Add a reflect-config.json hint, or use Spring's @RegisterReflectionForBinding annotation:

@Configuration
@RegisterReflectionForBinding({OrderDTO.class, PaymentDTO.class})
public class NativeHintsConfig {
}

Or with the programmatic hints API:

@Component
public class MyHints implements RuntimeHintsRegistrar {
  @Override
  public void registerHints(RuntimeHints hints, ClassLoader classLoader) {
    hints.reflection()
      .registerType(OrderDTO.class, MemberCategory.INVOKE_DECLARED_CONSTRUCTORS,
                                    MemberCategory.DECLARED_FIELDS);
  }
}

Then annotate your main class:

@SpringBootApplication
@ImportRuntimeHints(MyHints.class)
public class Application {
  public static void main(String[] args) {
    SpringApplication.run(Application.class, args);
  }
}

2. Dynamic Proxies

If you use java.lang.reflect.Proxy or Mockito-style dynamic proxies in production code (not just tests), they must be declared upfront.

hints.proxies().registerJdkProxy(MyService.class, MyOtherInterface.class);

Spring automatically registers proxies for @Transactional, @Async, and AOP-annotated beans. But if you create proxies manually, you must register them.

3. Resources Not Found

getClass().getResourceAsStream("/templates/email.html") works on the JVM because the classpath is there at runtime. In a native binary, resources must be explicitly included.

hints.resources().registerPattern("templates/*");

Or in application.properties:

spring.aot.hint.resources.patterns=templates/*,messages/**

4. Logback / SLF4J Configuration

Logback's XML parsing uses reflection internally. Use Spring Boot's auto-configured logging (it already has native hints) and avoid custom logback.xml with dynamic <if> conditions.

5. Jackson and Custom Serializers

Jackson works well natively when you use @JsonProperty, @JsonSerialize, and standard annotations. Problems arise with:

TypeReference<List<MyDTO>> anonymous classes
Polymorphic types with @JsonTypeInfo

For polymorphic types, register them explicitly:

hints.reflection()
  .registerType(Animal.class, MemberCategory.INVOKE_DECLARED_CONSTRUCTORS, MemberCategory.DECLARED_FIELDS)
  .registerType(Dog.class, MemberCategory.INVOKE_DECLARED_CONSTRUCTORS, MemberCategory.DECLARED_FIELDS);

Testing Your Native Build Before Committing

The worst experience is discovering native failures only after a 5-minute native-image compile. Use the GraalVM agent on the JVM first — it traces all reflective accesses and generates the config files:

java -agentlib:native-image-agent=config-output-dir=src/main/resources/META-INF/native-image \
  -jar target/myapp.jar

Run your app through all code paths (or better, run your integration test suite against it), then stop the app. The agent writes reflect-config.json, proxy-config.json, and resource-config.json into the output directory. Commit these files — GraalVM picks them up automatically.

You can also use Spring's Native Test support to validate hints without a full native compile:

@SpringBootTest
@TestPropertySource(properties = "spring.aot.enabled=true")
class NativeHintsValidationTest {
  @Test
  void contextLoads() { }
}

What You Give Up

Native Image is not free. Know the trade-offs:

Concern	JVM	Native
Build time	30–60s	3–8 minutes
Throughput under load	Higher (JIT)	Slightly lower
Dynamic class loading	Yes	No
Reflection	Unrestricted	Declared only
JVMTI / profiling agents	Yes	Limited
Debuggability	Full	Basic (`-g` flag)
Library compatibility	Universal	Most, not all

Throughput: Under sustained load, JIT-compiled code is often faster than native because JIT can optimize based on runtime behavior (branch prediction, inlining). Native binaries start faster and use less memory but may have slightly lower peak throughput for CPU-intensive workloads. For I/O-bound services (which most Spring Boot apps are), the difference is negligible.

Build time: A 5-minute build is annoying in local dev. Keep native as a CI/CD step, not part of your inner dev loop. Run ./mvnw spring-boot:run locally, build native only in the pipeline.

When Native Is Worth It

Go native when:

Serverless / FaaS: AWS Lambda cold starts are a real cost. 78ms vs 3.2s is the difference between a usable API and an unusable one.
Kubernetes burst scaling: Pods that start in under a second can scale to handle traffic spikes. 3-second startup means you're dropping requests while pods warm up.
Memory-constrained environments: 62 MB RSS vs 310 MB means you can run 5x more instances on the same node.
CLI tools built in Java: A native CLI binary that starts instantly feels as good as a Go or Rust CLI.

Stay on the JVM when:

Your app uses libraries without native support (check the GraalVM Reachability Metadata Repository first)
You rely heavily on runtime class generation (e.g., ByteBuddy, Javassist in non-Spring contexts)
Peak throughput matters more than startup (batch processing, number crunching)
Your team isn't ready to deal with native build failures in CI

CI/CD Pipeline

Keep native builds in a separate job so they don't slow down your PR feedback loop:

# .github/workflows/build.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { java-version: '21', distribution: 'temurin' }
      - run: ./mvnw verify   # fast: compile + test on JVM

  native:
    runs-on: ubuntu-latest
    needs: test
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: graalvm/setup-graalvm@v1
        with:
          java-version: '21'
          distribution: 'graalvm'
          native-image-job-reports: 'true'
      - run: ./mvnw -Pnative native:compile -DskipTests
      - run: ./mvnw -Pnative spring-boot:build-image -DskipTests
      - run: docker push myregistry/myapp:${{ github.sha }}

The native-image-job-reports: 'true' flag in the GraalVM action gives you a detailed report in the GitHub Actions summary showing which classes were included, missing hints, and build time breakdown — useful when debugging failures.

Library Support in 2026

The ecosystem has largely caught up. Most common Spring Boot dependencies have native support via the GraalVM Reachability Metadata Repository:

Library	Native Support
Spring Data JPA / Hibernate	✅ Full
Spring Security	✅ Full
Spring Cache + Redis	✅ Full
Flyway / Liquibase	✅ Full
Micrometer + Prometheus	✅ Full
MapStruct	✅ Full
Lombok	✅ Full (compile-time only)
Jackson	✅ Full with hints
gRPC (grpc-spring-boot-starter)	✅ Full
Testcontainers	⚠️ Test-only, not in native binary
Mockito	❌ Not for production use
ByteBuddy (direct use)	⚠️ Limited

If a library you need isn't on the list, check if it provides its own native-image.properties or if there's a community contribution in the Reachability Metadata Repository. Alternatively, you can contribute hints upstream — the process is straightforward.

The Bottom Line

GraalVM Native Image + Spring Boot 3 is production-ready in 2026. The tooling has matured, the Spring AOT processor handles 80% of the work automatically, and most real-world Spring Boot apps can go native with a manageable amount of hint configuration.

The build time penalty (5–8 minutes) is real, but it belongs in CI — not in your dev loop. The payoff — 40x faster startup, 5x lower memory, and images under 100MB — is a genuine competitive advantage for teams running containerized services.

Start with one service. Run the GraalVM agent against your integration tests, commit the generated hints, add the native build job to CI. If it works (it probably will), you have a blueprint to apply to the rest of your stack.

The hardest part is getting through the first build error. After that, it's just plumbing.

10 Spring Boot Mistakes That Silently Kill Production Apps

Avaneesh Yadav — Mon, 31 Aug 2026 10:26:17 +0000

Most Spring Boot production incidents don't come from exotic bugs. They come from the same ten patterns, repeated across codebases, that work fine under light load and silently degrade as traffic grows.

This guide covers the anti-patterns that production Java systems hit most often — with the exact code that causes them, why they fail, and what to write instead.

1. @Transactional on Private Methods

The mistake:

@Service
public class OrderService {

    // This @Transactional does NOTHING.
    @Transactional
    private void processPayment(Order order) {
        // changes to order are NOT in a transaction
    }

    public void submitOrder(Order order) {
        processPayment(order); // direct call, bypasses proxy
    }
}

Why it fails: Spring's @Transactional works through a proxy that wraps the bean. When you call a private method — or any method from within the same class — you bypass the proxy and call the raw object directly. No proxy → no transaction. Exceptions don't roll back. Data writes commit even when they shouldn't.

This is the most common Spring transactional bug. It compiles and passes unit tests. It fails silently in production when a mid-transaction exception leaves the database in a partially-written state.

The fix:

@Service
@RequiredArgsConstructor
public class OrderService {

    private final PaymentService paymentService; // separate @Service

    public void submitOrder(Order order) {
        paymentService.processPayment(order); // goes through Spring proxy
    }
}

@Service
public class PaymentService {

    @Transactional // works because external call goes through proxy
    public void processPayment(Order order) { ... }
}

If refactoring to a separate class isn't practical, inject self:

@Service
public class OrderService {

    @Autowired
    private OrderService self; // Spring injects the proxied version

    public void submitOrder(Order order) {
        self.processPayment(order); // goes through proxy
    }

    @Transactional
    public void processPayment(Order order) { ... }
}

Catch it early: Enable Spring's proxy debug logging (logging.level.org.springframework.aop=DEBUG) in a staging environment and verify transactions are opening where expected.

2. The N+1 Query Problem

This is the single most common performance issue in JPA-based Spring Boot applications. A page that renders in 50ms under load testing collapses to 4 seconds in production because the dataset grew.

The mistake:

// Entity
@Entity
public class Order {
    @OneToMany(fetch = FetchType.LAZY)
    private List<LineItem> lineItems;
}

// Service
public List<OrderSummary> getOrders() {
    List<Order> orders = orderRepository.findAll(); // 1 query
    return orders.stream()
        .map(o -> new OrderSummary(o, o.getLineItems().size())) // N queries
        .toList();
}

For 200 orders: 1 query to fetch orders + 200 queries to fetch each order's line items = 201 queries. At 1000 orders it's 1001 queries. Most ORMs make this invisible — the queries happen lazily, inside the stream, with no stack frame pointing to the cause.

How to detect it: Enable query logging:

spring:
  jpa:
    show-sql: true
    properties:
      hibernate:
        format_sql: true
logging:
  level:
    org.hibernate.SQL: DEBUG
    org.hibernate.orm.jdbc.bind: TRACE

If you see the same query repeated in a loop with different ID parameters, it's N+1.

The fix — JOIN FETCH:

public interface OrderRepository extends JpaRepository<Order, Long> {

    @Query("SELECT o FROM Order o LEFT JOIN FETCH o.lineItems WHERE o.status = :status")
    List<Order> findByStatusWithItems(@Param("status") OrderStatus status);
}

One query. Fetches orders and their line items in a single JOIN.

The fix — Projections (when you only need some fields):

public interface OrderSummaryView {
    String getOrderId();
    BigDecimal getTotal();
    int getItemCount();
}

public interface OrderRepository extends JpaRepository<Order, Long> {
    List<OrderSummaryView> findByStatus(OrderStatus status);
}

Projections let Spring Data generate a SELECT with only the columns you need — no entity materialization, no lazy loading risk.

The fix — @EntityGraph:

@EntityGraph(attributePaths = {"lineItems", "lineItems.product"})
List<Order> findAll();

Declarative JOIN FETCH without a JPQL query. Cleaner for simple cases.

3. Loading Entities for Count/Existence Checks

// Loads the entire entity from DB just to check if it exists
if (orderRepository.findById(orderId).isPresent()) {
    // ...
}

// Loads all orders just to count them
int count = orderService.findAll().size();

Both are SELECT * queries that transfer full row data across the network, materialize Java objects in heap, then throw most of it away.

The fix:

// Existence — single indexed column lookup, no data transfer
if (orderRepository.existsById(orderId)) { ... }

// Count — aggregation runs on the DB side
long count = orderRepository.count();

// Conditional count
long pendingCount = orderRepository.countByStatus(OrderStatus.PENDING);

For complex conditions:

@Query("SELECT COUNT(o) FROM Order o WHERE o.status = :status AND o.createdAt > :since")
long countRecentByStatus(@Param("status") OrderStatus status, @Param("since") LocalDateTime since);

4. Misconfigured Hikari Connection Pool

HikariCP ships with maximum-pool-size=10. Under low load this is invisible. Under real traffic it causes connection wait times that cascade into request timeouts.

# Default — often wrong for production
spring:
  datasource:
    hikari:
      maximum-pool-size: 10  # too low for most services

Symptoms: Requests succeed but take 200-500ms longer than expected. Actuator metrics show hikaricp.connections.pending > 0 regularly. Logs show HikariPool - Connection is not available, request timed out.

The fix — size for your thread model:

spring:
  datasource:
    hikari:
      # For virtual threads: connections = (number of cores * 2) is a starting point
      # For platform threads: connections = thread pool size
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 3000    # fail fast — 3s, not the default 30s
      idle-timeout: 600000        # 10 min
      max-lifetime: 1800000       # 30 min — rotate before DB kills idle connections
      keepalive-time: 60000       # prevent firewall from killing idle connections

Expose pool metrics:

management:
  metrics:
    enable:
      hikaricp: true

Then monitor hikaricp.connections.pending and hikaricp.connections.acquire p99. If pending > 0 for more than brief spikes, increase the pool.

[!NOTE]
With virtual threads (Spring Boot 3.2+, spring.threads.virtual.enabled=true), the optimal pool size is lower than with platform threads — virtual threads park during I/O so fewer connections handle more concurrent requests. Start at 2× core count and tune from metrics.

5. @Transactional(readOnly = true) Left Off Read Queries

// Missing readOnly — Spring opens a full R/W transaction
@Transactional
public List<Order> findRecentOrders(LocalDate since) {
    return orderRepository.findByCreatedAtAfter(since.atStartOfDay());
}

readOnly = true on a read-only transaction:

Tells Hibernate to skip dirty checking (no snapshot comparison at flush time)
Lets the JPA provider skip the write-ahead log
Some databases route read-only transactions to read replicas automatically

The fix:

@Transactional(readOnly = true)
public List<Order> findRecentOrders(LocalDate since) {
    return orderRepository.findByCreatedAtAfter(since.atStartOfDay());
}

Low effort, measurable improvement under read-heavy load. On services with 80% read traffic, this alone reduces database write-lock contention.

6. Missing Database Indexes on JPA Foreign Keys

JPA doesn't automatically create indexes on foreign key columns. It creates the constraint, not the index. This causes full table scans on every JOIN against a large table.

@Entity
public class LineItem {
    @ManyToOne
    @JoinColumn(name = "order_id")  // creates FK constraint, NOT an index
    private Order order;
}

Finding all line items for an order does SELECT * FROM line_item WHERE order_id = ?. Without an index on order_id, this is a full scan of the entire line_item table.

The fix — declare the index explicitly:

@Entity
@Table(name = "line_item",
    indexes = @Index(name = "idx_line_item_order_id", columnList = "order_id"))
public class LineItem {
    @ManyToOne
    @JoinColumn(name = "order_id")
    private Order order;
}

Or via Liquibase/Flyway (preferred for production — schema migrations in code):

-- V3__add_line_item_order_index.sql
CREATE INDEX idx_line_item_order_id ON line_item(order_id);

Finding missing indexes: Run EXPLAIN ANALYZE on your slowest queries. Any Seq Scan on a large table that filters by a FK column is a missing index candidate.

7. Leaking Hibernate Sessions Into Jackson Serialization

@Entity
public class Order {
    @OneToMany(fetch = FetchType.LAZY)
    private List<LineItem> lineItems;
}

// Controller — Open Session in View is ON by default
@GetMapping("/orders/{id}")
public Order getOrder(@PathVariable Long id) {
    return orderRepository.findById(id).orElseThrow();
    // Jackson serializes Order → touches lineItems → Hibernate loads lazily
    // If session is closed: LazyInitializationException
    // If session is open (OSIV): unexpected extra query per request
}

Spring Boot enables Open Session in View (OSIV) by default. This keeps the Hibernate session open through the entire request, including serialization. This means:

Lazy collections load silently during JSON serialization (N+1 hidden in Jackson)
Session-per-request model doesn't compose with virtual threads
Database connections held for the full request lifecycle, not just the DB operation

The fix — disable OSIV, use DTOs:

spring:
  jpa:
    open-in-view: false  # disable OSIV

// Fetch everything you need while the session is open
@Transactional(readOnly = true)
public OrderResponse getOrder(Long id) {
    Order order = orderRepository.findByIdWithItems(id)
        .orElseThrow(() -> new OrderNotFoundException(id));

    // Map to a DTO — no Hibernate proxy, safe to serialize
    return OrderResponse.from(order);
}

public record OrderResponse(String orderId, BigDecimal total, List<LineItemResponse> items) {
    static OrderResponse from(Order o) {
        return new OrderResponse(
            o.getId().toString(),
            o.getTotal(),
            o.getLineItems().stream().map(LineItemResponse::from).toList()
        );
    }
}

8. Unbounded @async Thread Pools

@Configuration
@EnableAsync
public class AsyncConfig implements AsyncConfigurer {
    // No custom config — Spring defaults to SimpleAsyncTaskExecutor
    // SimpleAsyncTaskExecutor creates a new thread per task, no limit
}

@Service
public class NotificationService {
    @Async
    public void sendEmail(String to, String subject) {
        // ...
    }
}

SimpleAsyncTaskExecutor (the default when no pool is configured) creates a new thread per task with no upper bound. Under sustained load this spins up thousands of threads and OOMs the service.

The fix — explicit bounded pool:

@Configuration
@EnableAsync
public class AsyncConfig implements AsyncConfigurer {

    @Override
    public Executor getAsyncExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(4);
        executor.setMaxPoolSize(20);
        executor.setQueueCapacity(500);       // tasks queue before rejecting
        executor.setThreadNamePrefix("async-");
        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
        executor.initialize();
        return executor;
    }
}

CallerRunsPolicy is usually the right rejection handler for background tasks — it runs the task on the caller's thread rather than throwing. This creates natural backpressure: if the async pool is saturated, the calling thread slows down instead of the service crashing.

For virtual threads (Java 21+):

@Bean
public Executor asyncExecutor() {
    return Executors.newVirtualThreadPerTaskExecutor(); // unbounded, but virtual threads are cheap
}

Virtual thread executors are safe to make unbounded because virtual threads don't block OS threads during I/O and cost ~1KB each.

9. Exception Swallowing in Background Jobs

@Component
public class ReconciliationJob {

    @Scheduled(cron = "0 0 2 * * *")
    public void run() {
        try {
            reconcile();
        } catch (Exception e) {
            log.error("Reconciliation failed", e);
            // returns normally — scheduler sees success
            // no alert, no retry, no escalation
        }
    }
}

The scheduler sees this method return without exception and marks it successful. The error is in the log. Without a log alert configured, the failure is invisible. Reconciliation silently stops running.

This pattern — catch, log, swallow — is how data inconsistencies accumulate over months without anyone noticing.

The fix — let exceptions propagate, or use structured alerting:

@Scheduled(cron = "0 0 2 * * *")
public void run() {
    try {
        reconcile();
    } catch (Exception e) {
        // Send an alert before rethrowing
        alertService.sendAlert("Reconciliation failed: " + e.getMessage(), AlertSeverity.HIGH);
        meterRegistry.counter("job.reconciliation.failure").increment();
        throw new RuntimeException("Scheduled reconciliation failed", e);
        // Spring will log this as an error — also visible in Actuator health
    }
}

Or better — use Temporal or Spring Batch for jobs that need fault tolerance and visibility (see the Temporal guide).

10. Logging in the Hot Path With String Concatenation

// Evaluated on every call, even when DEBUG logging is off
log.debug("Processing order " + order.getId() + " for customer " + customer.getName());

+ string concatenation creates intermediate String objects on every call. At 10,000 requests/minute, even a disabled DEBUG statement that runs on every request produces garbage that GC must collect, adding latency spikes.

The fix — parameterized logging:

// SLF4J defers evaluation — no string created if DEBUG is off
log.debug("Processing order {} for customer {}", order.getId(), customer.getName());

The {} placeholders are only evaluated if the logger is actually at DEBUG level. Zero allocation in production where DEBUG is off.

For complex log objects that are expensive to produce:

// Supplier form — only called if DEBUG is on
log.debug("Order state: {}", () -> computeExpensiveOrderDump(order));

The Production Checklist

Before a Spring Boot service goes to production, run through this:

[ ] Grep for @Transactional on private methods — fix any found
[ ] Run queries against staging data at 10× production volume — look for N+1 patterns in query logs
[ ] Disable OSIV (spring.jpa.open-in-view: false) and fix resulting LazyInitializationExceptions
[ ] Run EXPLAIN ANALYZE on your top 10 slowest queries — add missing indexes
[ ] Add @Transactional(readOnly = true) to all read-only service methods
[ ] Verify Hikari pool size matches your threading model — expose pool metrics in Actuator
[ ] Configure a named, bounded ThreadPoolTaskExecutor for @Async tasks
[ ] Verify all @Scheduled jobs either let exceptions propagate or send alerts before swallowing
[ ] Replace log.debug("text" + var) with parameterized log.debug("text {}", var) in hot paths
[ ] Set management.health.db.enabled=true and test liveness/readiness probes

graph LR
    A[Spring Boot Service] --> B{Code Review}
    B --> C[Transaction Check]
    B --> D[Query Analysis]
    B --> E[Pool Config]
    B --> F[Async Config]
    C --> G[No private @Transactional]
    D --> H[No N+1 — JOINs or projections]
    D --> I[Indexes on FK columns]
    E --> J[Bounded Hikari pool]
    E --> K[Pool metrics exposed]
    F --> L[Bounded executor configured]
    G --> M[Production Ready]
    H --> M
    I --> M
    J --> M
    K --> M
    L --> M

Catching These Before Production

Static analysis. SpotBugs + find-sec-bugs catches some of these (including the @Transactional private method issue). Add it to your Maven/Gradle build.

Query logging in staging. Enable org.hibernate.SQL: DEBUG in your staging environment against a dataset that's at least 10% of production volume. Any N+1 pattern shows up as repeated identical queries.

Load testing. k6, Gatling, or even ab at 2× expected peak. Watch HikariCP's connections.pending, Hibernate's queries.execution p99, and JVM GC pause time. Spikes in any of these under load point to one of the issues above.

Production metrics. Every Spring Boot 3 service should export to an observability backend. Dashboards to have:

DB connection pool saturation (hikaricp.connections.pending > 0)
Slow query p99 (> 100ms threshold)
GC pause duration (> 50ms threshold)
Heap utilization trend (monotonic growth = memory leak)

These patterns aren't obscure. They appear in production codebases at companies of every size, written by engineers who knew the frameworks well. The difference between a service that holds up under load and one that requires an incident at 2 AM is usually caught before deployment, not after.

Spotted one of these in your codebase? Or have a production anti-pattern that isn't on this list? Find me on LinkedIn — the list grows from war stories.

Java Developer Roadmap 2026: What to Learn, What to Skip

Avaneesh Yadav — Mon, 31 Aug 2026 09:26:30 +0000

Java is having its most interesting decade in twenty years. Virtual threads changed the concurrency model. Records and sealed classes modernized the type system. Pattern matching made the language feel genuinely expressive. And Spring AI put enterprise Java at the center of the AI engineering conversation.

If you're a Java developer trying to figure out what actually matters to learn in 2026 — and what to stop spending time on — this is the guide. No paid course recommendations. No sponsored rankings. Just what's worth your time.

The Big Picture: What Changed

graph TD
    A[Java Developer 2026] --> B[Core Language]
    A --> C[Concurrency]
    A --> D[Frameworks]
    A --> E[AI Integration]
    A --> F[Cloud Native]

    B --> B1[Java 21-24 Features]
    B --> B2[Records and Sealed Classes]
    B --> B3[Pattern Matching]

    C --> C1[Virtual Threads]
    C --> C2[Structured Concurrency]
    C --> C3[Scoped Values]

    D --> D1[Spring Boot 3.x]
    D --> D2[Spring Security 6]
    D --> D3[Spring Data]

    E --> E1[Spring AI]
    E --> E2[LangChain4j]
    E --> E3[RAG Patterns]

    F --> F1[Docker and Kubernetes]
    F --> F2[GraalVM Native]
    F --> F3[OpenTelemetry]

1. Core Java — The Non-Negotiables

Java 21 was the LTS that mattered. Java 24 refined it further. Here's what every Java developer needs to know cold in 2026.

Records (Java 16+, now essential)

// Before: 40-line POJO with constructor, getters, equals, hashCode, toString
// After:
public record OrderSummary(String orderId, BigDecimal total, OrderStatus status) {}

// Pattern matching in switch — clean, exhaustive
String describe(OrderSummary order) {
    return switch (order.status()) {
        case PENDING  -> "Awaiting payment";
        case PAID     -> "Processing — total: " + order.total();
        case SHIPPED  -> "On the way";
        case DELIVERED -> "Delivered";
    };
}

Records are not just syntax sugar. They signal immutability, make DTOs and value objects composable, and pair naturally with sealed classes for algebraic data types.

Sealed Classes + Pattern Matching

public sealed interface PaymentResult
    permits PaymentResult.Success, PaymentResult.Failure, PaymentResult.Pending {}

record Success(String transactionId, BigDecimal amount) implements PaymentResult {}
record Failure(String reason, ErrorCode code)          implements PaymentResult {}
record Pending(String referenceId)                     implements PaymentResult {}

// Exhaustive switch — compiler catches missing cases
String handle(PaymentResult result) {
    return switch (result) {
        case Success s  -> "Paid: " + s.transactionId();
        case Failure f  -> "Failed: " + f.reason();
        case Pending p  -> "Pending: " + p.referenceId();
    };
}

This eliminates entire categories of instanceof chains and ClassCastException risks.

Text Blocks

// Stop escaping quotes in SQL, JSON, GraphQL queries
String query = """
    SELECT o.id, o.total, c.email
    FROM orders o
    JOIN customers c ON o.customer_id = c.id
    WHERE o.status = 'PAID'
      AND o.created_at > :since
    ORDER BY o.created_at DESC
    """;

What to skip: Java 8 streams chaining for everything

Stream.of(...).filter(...).map(...).collect(...) is fine. Chaining 12 operators for a 3-line for-loop is not. Java 16+ toList(), records, and the enhanced for-loop often produce cleaner code than complex stream pipelines. Don't mistake verbosity for sophistication.

2. Concurrency — The Biggest Shift in Java History

Virtual threads (GA in Java 21) changed the fundamental model. If you're still writing async/reactive code for I/O-bound services, you're fighting the platform.

Virtual Threads

// Enable in Spring Boot — one line
spring.threads.virtual.enabled=true

// Now every request handler runs on a virtual thread
// Write blocking code. The JVM parks the thread during I/O.
@GetMapping("/orders/{id}")
public Order getOrder(@PathVariable String id) {
    Order order = orderRepository.findById(id)       // blocks, JVM parks virtual thread
        .orElseThrow(() -> new OrderNotFoundException(id));
    List<LineItem> items = itemRepository.findByOrderId(id); // same
    return enrich(order, items);                     // same
}
// No Mono, no CompletableFuture, no callback hell.

The JVM creates virtual threads cheaply (nanoseconds, ~1KB stack). You can have millions. I/O waits don't block the carrier thread. The performance profile of reactive code with the readability of blocking code.

Structured Concurrency (GA in Java 24)

For when you genuinely need to run things in parallel:

// Fan-out: fetch user + orders concurrently, both must succeed
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
    Subtask<User>        user   = scope.fork(() -> userService.find(userId));
    Subtask<List<Order>> orders = scope.fork(() -> orderService.findByUser(userId));

    scope.join().throwIfFailed();    // waits for both, cancels if either fails
    return new Dashboard(user.get(), orders.get());
}

This replaces the fragile CompletableFuture.allOf(...) pattern. Lifetimes are scoped — subtasks cannot outlive the block. Cancellation propagates correctly. Stack traces are readable.

Scoped Values (Java 24)

// Replace ThreadLocal — works correctly across virtual threads
static final ScopedValue<RequestContext> REQUEST_CTX = ScopedValue.newInstance();

// At request entry point:
ScopedValue.where(REQUEST_CTX, new RequestContext(traceId, userId))
    .run(() -> processRequest(req));

// Anywhere in the call tree:
String traceId = REQUEST_CTX.get().traceId();

ThreadLocal still works but doesn't compose well with virtual threads (pinning risk). Use ScopedValue for request context in new code.

What to skip: Project Reactor / WebFlux for request-response services

WebFlux is the right tool for SSE, WebSocket, and genuine streaming workloads. For CRUD microservices and REST APIs, virtual threads deliver equal throughput with a fraction of the complexity. The ecosystem (blocking JDBC, JPA, most third-party SDKs) is also blocking — fighting it reactively adds overhead without benefit.

3. Spring Boot 3.x — The Framework Layer

Spring Boot 3 requires Java 17+ and brought a full Jakarta EE migration. By 2026, if you're still on Spring Boot 2.x, this is the year to move.

What matters most

AOT and GraalVM Native. Spring Boot 3 ships native compilation support. For short-lived workloads (FaaS, CLIs, Lambda) native images start in milliseconds and use a fraction of the RAM.

<plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
</plugin>

mvn -Pnative native:compile
./target/myapp   # starts in ~50ms, uses ~50MB RAM

Spring Security 6. The WebSecurityConfigurerAdapter is gone. The new lambda DSL is cleaner:

@Bean
SecurityFilterChain security(HttpSecurity http) throws Exception {
    return http
        .authorizeHttpRequests(auth -> auth
            .requestMatchers("/actuator/health").permitAll()
            .requestMatchers("/admin/**").hasRole("ADMIN")
            .anyRequest().authenticated())
        .oauth2ResourceServer(oauth2 -> oauth2.jwt(Customizer.withDefaults()))
        .build();
}

Spring Data 3.x. PagingAndSortingRepository is now separate from CrudRepository. JpaRepository extends both. Projections with interfaces or records are the cleanest way to fetch partial data.

Actuator + Micrometer + OpenTelemetry. This stack is the observability standard. Every Spring Boot 3 service should export traces and metrics to your observability backend via OTLP.

management:
  tracing:
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://otel-collector:4318/v1/traces

4. AI Integration — The New Differentiator

Java developers who can wire LLMs into production systems are in the top 5% of the market right now. This is the highest-leverage skill to add in 2026.

Spring AI (the production choice)

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
</dependency>

@Service
@RequiredArgsConstructor
public class DocumentAnalysisService {

    private final ChatClient chatClient;
    private final VectorStore vectorStore;

    // RAG: embed query, retrieve context, generate answer
    public String answer(String question) {
        List<Document> context = vectorStore.similaritySearch(
            SearchRequest.query(question).withTopK(4)
        );

        String contextText = context.stream()
            .map(Document::getContent)
            .collect(Collectors.joining("\n---\n"));

        return chatClient.prompt()
            .system("Answer using only the context provided. If the answer is not there, say so.")
            .user(u -> u.text("Context:\n{ctx}\n\nQuestion: {q}")
                        .param("ctx", contextText)
                        .param("q", question))
            .call()
            .content();
    }
}

Key Spring AI concepts to learn in order

ChatClient — the main abstraction for LLM calls
Structured outputs — returning Java POJOs instead of raw strings
VectorStore + EmbeddingModel — RAG foundation
Tool/Function calling — letting LLMs call your Java methods
ChatMemory + Advisors — conversation state, prompt composition
Observability — tracing LLM calls with Micrometer

What to skip: Building your own LLM abstraction layer

LangChain4j and Spring AI cover 95% of production AI use cases. Rolling a custom HTTP client to the OpenAI API is fine for learning but builds a maintenance liability you'll regret in six months.

5. Cloud-Native — Table Stakes

If you're building services in 2026, cloud-native patterns are not optional.

Docker and Kubernetes (know the basics)

FROM eclipse-temurin:21-jre-alpine
COPY target/app.jar /app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app.jar"]

# Kubernetes readiness + liveness via Spring Actuator
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080

You don't need to be a Kubernetes expert. You do need to know: how to containerize an app, what readiness/liveness probes do, how ConfigMaps and Secrets map to env vars, and how HPA works.

OpenTelemetry — the observability standard

// With Spring Boot 3 Actuator + OTLP exporter, traces are automatic.
// Add baggage for business context:
@GetMapping("/orders/{id}")
public Order getOrder(@PathVariable String id) {
    Span.current().setAttribute("order.id", id);
    // ... rest of handler
}

Logs, traces, and metrics in one collector. Stop learning proprietary agents for each APM vendor.

What to skip: Manual Dockerfiles with fat JARs on JDK images

eclipse-temurin:21-jre-alpine (not JDK) cuts image size by ~300MB. Spring Boot's layered JAR support (spring-boot-maven-plugin layer configuration) and Paketo Buildpacks produce better images than handwritten Dockerfiles for most teams. Use buildpacks first, hand-roll only when you have a specific reason.

Learning Path by Experience Level

0–2 Years (Junior)

Focus here, in this order:

Java 21 core — records, sealed classes, pattern matching, text blocks
Spring Boot 3 basics — REST, JPA, validation, security fundamentals
Testing — JUnit 5, Mockito, @SpringBootTest vs slice tests
Git + Docker basics
SQL — JOINs, indexes, EXPLAIN — know your database

Skip for now: Reactive programming, microservices architecture, AI integration, Kubernetes internals. Complexity before fundamentals creates engineers who can't debug.

2–5 Years (Mid-level)

Virtual threads and structured concurrency — replace your thread pool mental model
Spring Boot 3 production patterns — Actuator, Micrometer, OpenTelemetry
Microservices fundamentals — API design, inter-service communication (REST vs gRPC), circuit breakers
Cloud basics — one of AWS/GCP/Azure to associate level
Spring AI basics — RAG pipeline, tool calling, structured outputs

The leverage move at this level: observability. Engineers who can find production issues fast are disproportionately valuable.

5+ Years (Senior / Lead)

AI system design — RAG architecture, agent orchestration, LLMOps
Performance engineering — virtual thread profiling, JVM GC tuning, DB query optimization
Platform engineering — Kubernetes operators, Helm, GitOps
Architecture patterns — event-driven design, CQRS, saga
Engineering leadership — technical direction, system design interviews, code review culture

The leverage move at this level: mentorship. The engineers you bring up multiply your output more than any technical skill.

The "What to Stop Doing" List

This is the section people argue about. That's the point.

Stop doing	Do this instead
`@Autowired` field injection	Constructor injection — testable, explicit
`extends HttpServlet` / raw servlets	Spring MVC or Spring WebFlux
XML Spring config	Java config + `@Bean`
`java.util.Date` / `Calendar`	`java.time.*` — `LocalDate`, `ZonedDateTime`
Manual JSON parsing with `JSONObject`	Jackson / Gson with typed POJOs
Mutable POJOs with 30 setters	Records or immutable value objects
`synchronized` blocks for everything	`ReentrantLock`, `Semaphore`, virtual threads
`ThreadLocal` for request context	`ScopedValue` (Java 24+)
Catching `Exception` everywhere	Specific exceptions; let runtime exceptions propagate
Log.info("Got order: " + orderId)	`log.info("Got order: {}", orderId)` — no string concat
Building your own retry logic	Resilience4j or Spring Retry
Rolling your own JWT validation	Spring Security's OAuth2 resource server

The Skills That Actually Get You Hired in 2026

In order of what hiring managers actually test for:

System design — can you design a scalable, observable microservice under ambiguous requirements?
Debugging — can you find a production issue using logs, traces, and metrics without stepping through code?
Code quality — do your PRs need one review cycle or five?
AI integration — can you wire an LLM into a service that works reliably in production (not just a demo)?
Communication — can you explain a technical decision to a non-technical stakeholder?

Leetcode matters for FAANG and trading companies. For the vast majority of backend Java roles, system design and real-world debugging matter more.

Resources Worth Your Time

Language:

JEP Index — primary source for new Java features, better than any blog summary
Inside Java podcast — authoritative, from the Java team

Spring:

Spring Blog — official, high signal
Spring One talks on YouTube — architecture deep-dives by the framework authors

AI / LLM:

Spring AI reference docs — the most up-to-date source for Spring AI patterns
AI Engineering — practical guides on RAG, agents, LLMOps in Java

System Design:

Designing Data-Intensive Applications (Kleppmann) — still the best book on distributed systems
ByteByteGo newsletter — good visual breakdowns of production architectures

One More Thing

Java in 2026 is not your Java of 2015. It's fast, expressive, and the ecosystem for AI integration is mature. The engineers who treat it as the "boring enterprise language" are leaving real opportunity on the table.

The engineers who will be most valuable in the next three years are the ones who can build reliable AI-powered systems in Java — because that's where the enterprise deals are, and that's where the Java ecosystem is heading.

Pick one section of this roadmap that you're weakest in. Work on it for 90 days. Then come back to the next one.

Disagreements welcome — that's how roadmaps improve. Find me on LinkedIn or browse the full buildingai.in blog for deep-dives on each section of this roadmap.

Temporal vs Spring Batch: Fixing Enterprise Java's Silent Job Failure Problem

Avaneesh Yadav — Fri, 28 Aug 2026 06:26:08 +0000

Enterprise Java teams commonly reach a point where scheduled jobs become a liability rather than an asset. Jobs fail silently — catching exceptions, logging nothing meaningful, and returning success to the scheduler. Others run simultaneously because the previous execution didn't finish before the next trigger fired and nobody configured @ScheduledLock. Data corruption from concurrent runs can take weeks to surface.

None of this shows up in dashboards until a data audit flags inconsistencies in a critical report.

This guide explains why @Scheduled breaks down at scale, how Temporal's durable execution model eliminates these problems, and how to migrate a production Spring Batch pipeline with complete code examples.

What Temporal Actually Is (In One Paragraph)

Temporal is a durable execution platform. You write plain Java code — functions, loops, try/catch — and Temporal makes it fault-tolerant by recording every state transition to an event log. If the process crashes mid-execution, it replays the log and resumes from exactly where it left off. There's no external state machine to define, no checkpoint tables to maintain, no retry logic to write. The code IS the workflow.

graph TD
  subgraph BEFORE ["Before — Spring Scheduler + Batch"]
    C1[Cron Trigger] --> J1[@Scheduled Method]
    J1 --> J2[Spring Batch Job]
    J2 --> J3[ItemReader]
    J3 --> J4[ItemProcessor]
    J4 --> J5[ItemWriter]
    J5 --> J6[DB commit]
    E1[Failure?] -.->|swallowed| X1[😶 Silent]
    E2[Overlap?] -.->|no guard| X2[💥 Corruption]
  end

  subgraph AFTER ["After — Temporal"]
    T1[Temporal Scheduler] --> W1[Workflow Method]
    W1 --> A1[Activity: Read chunk]
    W1 --> A2[Activity: Process chunk]
    W1 --> A3[Activity: Write chunk]
    F1[Failure?] -.->|auto-retry| W1
    F2[Crash?] -.->|replay from log| W1
    F3[Overlap?] -.->|workflow ID lock| X3[✅ Prevented]
  end

The Problem With @Scheduled at Scale

@Scheduled works fine for one or two simple jobs. By the time a team has dozens, accumulated debt becomes significant:

No visibility. Which jobs ran? Which failed? How long did they take? Spring's scheduler offers nothing here by default. Teams bolt on Actuator, add custom logging, hook up Micrometer. By the time there's real observability it's a custom framework to maintain.

No fault tolerance. An exception kills the job instance. Whether it retries, and how, is the team's problem. Teams solve this inconsistently — some catch-and-retry inline, some use Spring Retry, some let it silently fail and rely on the next scheduled trigger.

No distributed locking. @ScheduledLock with ShedLock or Quartz clustering works, but it's an additional library, additional config, and an additional failure mode (what happens when the lock row gets corrupted?).

Testing is painful. Unit tests can cover job logic, but testing scheduling behavior — does it actually retry? does it respect the lock? — requires either waiting for real time to pass or mocking the scheduler in ways that diverge from production behavior.

Spring Batch adds its own complexity. Job metadata tables (BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, etc.) need to be managed, pruned, and kept consistent. The restart/skip/retry model is powerful but verbose to configure. And Spring Batch has no concept of a workflow spanning multiple jobs — that becomes custom orchestration code.

Before: A Typical Spring Batch Job

Here's a representative pattern — daily invoice reconciliation, reading from a pending_invoices table, processing, and writing to processed_invoices:

// Cron trigger — buried in application.yml
// cron: "0 0 2 * * *"

@Component
@RequiredArgsConstructor
public class InvoiceReconciliationScheduler {

    private final JobLauncher jobLauncher;
    private final Job invoiceReconciliationJob;

    @Scheduled(cron = "${jobs.invoice-reconciliation.cron}")
    @SchedulerLock(name = "invoiceReconciliation", lockAtMostFor = "PT2H")
    public void run() {
        try {
            JobParameters params = new JobParametersBuilder()
                .addLong("run.id", System.currentTimeMillis())
                .toJobParameters();
            jobLauncher.run(invoiceReconciliationJob, params);
        } catch (Exception e) {
            log.error("Invoice reconciliation failed", e);
            // No alert. No retry. No escalation.
        }
    }
}

@Configuration
@RequiredArgsConstructor
public class InvoiceReconciliationJobConfig {

    @Bean
    public Job invoiceReconciliationJob(Step reconcileStep) {
        return jobBuilderFactory.get("invoiceReconciliationJob")
            .start(reconcileStep)
            .build();
    }

    @Bean
    public Step reconcileStep() {
        return stepBuilderFactory.get("reconcileStep")
            .<PendingInvoice, ProcessedInvoice>chunk(500)
            .reader(invoiceReader())
            .processor(invoiceProcessor())
            .writer(invoiceWriter())
            .faultTolerant()
            .retryLimit(3)
            .retry(TransientDataAccessException.class)
            .build();
    }
    // ... reader/processor/writer beans
}

This is roughly 150 lines across three files for one job. It has per-item retry (good), but no workflow-level retry, no timeout enforcement, no alerting on failure, and no way to see execution history without querying the batch tables directly.

After: The Same Job in Temporal

// Workflow interface — the contract
@WorkflowInterface
public interface InvoiceReconciliationWorkflow {
    @WorkflowMethod
    void reconcile(LocalDate date);
}

// Activity interface — the actual I/O work
@ActivityInterface
public interface InvoiceReconciliationActivities {
    List<PendingInvoice> readChunk(LocalDate date, int offset, int limit);
    List<ProcessedInvoice> processChunk(List<PendingInvoice> invoices);
    void writeChunk(List<ProcessedInvoice> processed);
    int countPending(LocalDate date);
}

// Workflow implementation — plain Java, no framework magic
public class InvoiceReconciliationWorkflowImpl implements InvoiceReconciliationWorkflow {

    private final InvoiceReconciliationActivities activities =
        Workflow.newActivityStub(InvoiceReconciliationActivities.class,
            ActivityOptions.newBuilder()
                .setStartToCloseTimeout(Duration.ofMinutes(10))
                .setRetryOptions(RetryOptions.newBuilder()
                    .setMaximumAttempts(5)
                    .setDoNotRetry(IllegalArgumentException.class.getName())
                    .build())
                .build());

    @Override
    public void reconcile(LocalDate date) {
        int total = activities.countPending(date);
        int offset = 0;
        int chunkSize = 500;

        while (offset < total) {
            List<PendingInvoice> chunk = activities.readChunk(date, offset, chunkSize);
            List<ProcessedInvoice> processed = activities.processChunk(chunk);
            activities.writeChunk(processed);
            offset += chunkSize;
        }
    }
}

// Schedule registration — once, at startup
client.scheduleClient().createSchedule(
    "invoice-reconciliation-daily",
    ScheduleSpec.newBuilder()
        .setCronExpressions(List.of("0 2 * * *"))
        .build(),
    ScheduleAction.startWorkflow(
        InvoiceReconciliationWorkflow.class,
        "reconcile",
        WorkflowOptions.newBuilder()
            .setWorkflowId("invoice-recon-" + LocalDate.now())
            .setTaskQueue("invoice-queue")
            .setWorkflowExecutionTimeout(Duration.ofHours(4))
            .build())
);

The workflow body is 12 lines. It reads like the business requirement. No chunk configuration DSL, no step beans, no job parameter boilerplate.

Because the workflow ID includes the date (invoice-recon-2026-08-28), attempting to start a second execution for the same day throws WorkflowExecutionAlreadyStarted — the overlap corruption problem is gone by construction, with no lock to configure or expire.

What You Get for Free

Retries with backoff. Every activity gets the retry policy defined at registration. Failed activities retry automatically, with exponential backoff, without any code in the activity itself.

Crash recovery. If the worker process dies mid-workflow, the next worker that picks up the task queue replays the event history and continues from the last completed activity. No data is lost.

Visibility out of the box. Temporal's web UI shows every workflow execution: start time, current state, activity history, retry attempts, failures, input and output. What would take custom Micrometer instrumentation now comes for free.

Workflow ID deduplication. Using a business-meaningful workflow ID (like invoice-recon-2026-08-28) means the same logical job can never run twice. This is structurally better than distributed locking.

Testable without real time. Temporal's TestWorkflowEnvironment lets you test the entire workflow including retries, timeouts, and activity failures — in unit tests, without waiting for real timers.

@Test
void shouldRetryFailedActivity() {
    TestWorkflowEnvironment env = TestWorkflowEnvironment.newInstance();
    Worker worker = env.newWorker("invoice-queue");

    worker.registerActivitiesImplementations(new InvoiceActivitiesFailing(failOnAttempt: 1));
    worker.registerWorkflowImplementationTypes(InvoiceReconciliationWorkflowImpl.class);
    env.start();

    InvoiceReconciliationWorkflow wf = env.getWorkflowClient()
        .newWorkflowStub(InvoiceReconciliationWorkflow.class,
            WorkflowOptions.newBuilder().setTaskQueue("invoice-queue").build());

    // Runs in milliseconds — Temporal's test env skips real timers
    wf.reconcile(LocalDate.of(2026, 8, 28));

    verify(failingActivity, times(2)).readChunk(any(), anyInt(), anyInt()); // retried once
}

Common Migration Pitfalls

Activity timeouts. StartToCloseTimeout must be set conservatively — Temporal cancels and retries the activity if it exceeds the timeout. Set it to the P99 execution time, not the average. Activities that occasionally run for 45 minutes on large-backlog days need that reflected in their timeout.

Non-idempotent writers. Temporal retries activities automatically. If a writer inserts rows without an upsert, retries produce duplicates. Every write activity must be idempotent — ON CONFLICT DO UPDATE on the write side before migration.

History size limits. Temporal's event history has a default limit of 50,000 events. A workflow processing 500k records in 500-row chunks generates 3,000 activity calls — fine. Processing row-by-row would hit the limit. Design activities to operate on chunks, not individual records.

Local dev environment. Running Temporal locally requires Docker (temporalio/temporal). Teams without Docker in their standard dev setup will have friction on day one. Add it to the team devcontainer or docker-compose before rollout.

[!NOTE]
Temporal's Java SDK (version 2.x) works with Spring Boot 3.2+ virtual threads automatically. Pair it with spring.threads.virtual.enabled=true and worker threads scale to thousands of concurrent activity executions on minimal OS threads.

When to Keep Spring Batch

Not every job should move to Temporal. Spring Batch remains the right tool in two scenarios:

Regulated data processing with mandatory audit trails. Spring Batch's job metadata schema (BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION) is a ready-made audit log that compliance and operations teams can query directly in the database. Temporal's event history lives in Temporal's own store — visible in the UI, but not in your RDBMS. For financial reporting jobs with regulatory requirements, Spring Batch's schema is an asset.

Pure ETL with no orchestration complexity. A job that reads a file, transforms rows, writes to a table, and exits is exactly what Spring Batch was designed for. No retry logic, no dependencies between steps, no timeout sensitivity. Rewriting it in Temporal adds infrastructure without adding value.

Decision rule: if you would describe the job as "a workflow" — with branches, dependencies, or conditions — use Temporal. If you would describe it as "a batch" — fixed input, transform, fixed output — Spring Batch is the right fit.

Getting Started

<dependency>
    <groupId>io.temporal</groupId>
    <artifactId>temporal-spring-boot-starter</artifactId>
    <version>1.4.0</version>
</dependency>

# application.yml
spring:
  temporal:
    connection:
      target: localhost:7233
    workers:
      - task-queue: invoice-queue
        workflow-classes:
          - com.example.workflows.InvoiceReconciliationWorkflowImpl
        activity-beans:
          - invoiceReconciliationActivities

Temporal's Spring Boot starter auto-wires workers, injects WorkflowClient and ScheduleClient as beans, and handles graceful shutdown.

# Local dev — start Temporal server
docker run --rm -p 7233:7233 -p 8233:8233 temporalio/temporal:latest
# UI available at http://localhost:8233

Migration Checklist

[ ] Inventory all @Scheduled and Spring Batch jobs — map dependencies between them
[ ] Classify: "workflow" (Temporal) vs "pure batch" (keep Spring Batch)
[ ] Make all activity implementations idempotent (upsert, not insert)
[ ] Set StartToCloseTimeout at P99 of execution time, not average
[ ] Use business-meaningful workflow IDs for deduplication
[ ] Add Temporal Docker to dev environment / docker-compose
[ ] Run parallel execution for 2 weeks (old scheduler + Temporal) and compare outputs
[ ] Cut over the scheduler trigger, remove old @Scheduled beans
[ ] Configure Temporal Worker autoscaling (CPU-based HPA works well on Kubernetes)

Summary

@Scheduled is fine for one or two simple jobs. Beyond that, teams face silent failures, concurrent execution bugs, and zero visibility — problems that only surface when they've already caused damage.

Temporal solves all three at the infrastructure level: durable execution for crash recovery, workflow ID deduplication for concurrency control, and a built-in UI for visibility. The workflow code is shorter than Spring Batch equivalent, easier to read, and testable in isolation without mocking time.

For teams with more than 10 scheduled jobs, or any jobs with dependencies and retries, Temporal is worth evaluating before the next production incident makes the case for you.

Working with Temporal in Java or evaluating the migration? Happy to discuss patterns and pitfalls. Find me on LinkedIn.

AI Coding Tools in Enterprise: What 90 Days of Data Actually Shows

Avaneesh Yadav — Fri, 28 Aug 2026 06:26:06 +0000

Everyone has opinions on AI coding tools. Not enough teams publish the data.

This post covers a structured 90-day study of AI coding assistants across a 70-engineer enterprise team — frontend, backend, platform, and QA. The metrics tracked: PR velocity, review cycles, defect escape rate, time-to-first-review, and self-reported time savings. Tools used: Claude Code (primary), GitHub Copilot (one squad for comparison), Cursor (three frontend engineers).

The results were good enough to justify the investment. They were also more nuanced than the vendor benchmarks suggest — and two findings changed how to think about rollout strategy entirely.

Study Setup

A 6-month baseline was established before any tooling changes. PR metrics from GitHub, defect data from Jira, monthly velocity surveys. Then a structured rollout:

Week 1–2: Tool access granted, zero guidance. Observe organic adoption patterns.
Week 3–4: Prompt engineering training — two 90-minute sessions per squad.
Month 2–3: Embedded in workflows, metrics collected, weekly retros.

AI tool use was not mandated. Engineers could opt out. Two did. Both eventually opted back in by week 6.

The Numbers

Metric	Baseline (6mo avg)	After 90 days	Change
PRs merged per engineer/week	3.2	4.1	+28%
Median PR review cycles	2.4	1.9	−21%
Time to first review (hours)	18.3	14.7	−20%
P1/P2 defect escape rate	4.1%	3.8%	−7%
Test coverage (avg across repos)	61%	74%	+13pp

The velocity number (+28% PRs) looks strong. The defect number (−7%) is smaller than expected given the coverage jump. Understanding why is the most important part of this analysis.

What Actually Improved

Test writing — the biggest win by far. Engineers consistently reported that writing tests shifted from the most-avoided task to one they delegated to AI first. Test coverage jumped 13 percentage points in 90 days. For teams where test coverage has been a long-standing problem, this alone justifies the tool cost.

Boilerplate elimination. REST controllers, DTO classes, mapper code, migration scripts — all measurably faster. Engineers with 2–5 years of experience saw the largest gains here. Senior engineers who had already internalized these patterns saw smaller improvements.

Documentation. Javadoc, OpenAPI annotations, README sections — written without prompting once AI made the activation energy low. This is often where quality debt accumulates silently.

Onboarding speed in unfamiliar codebases. The "explain this code" use case is underrated. New engineers using AI to orient themselves in large repos reduced the time senior engineers spent on walkthrough sessions.

What Did Not Improve (Or Got Worse)

Defect rate on complex logic. The 7% improvement in defect escape rate is smaller than expected — and in two recorded incidents it was negative. Engineers over-trusted AI-generated business logic for domain-heavy calculations. The code was syntactically correct and plausible-looking, but semantically wrong in subtle ways. Both bugs passed code review because reviewers assumed AI-generated code had already been validated.

[!WARNING]
This is the most important finding: AI coding tools shift the defect profile. Obvious errors decrease. Plausible-but-wrong logic errors stay flat or increase if review culture doesn't adapt. Code review must become more skeptical, not less, when AI is involved — especially in the logic layer.

Code review quality initially degraded. In month 1, PR review times dropped because reviewers were approving faster — not because the code was better. This surfaced in retrospectives and required explicit re-training: AI-generated code needs the same scrutiny as human-written code, applied to different layers (logic correctness over syntax).

Prompt skill gap created a new inequality. Engineers who could write precise, context-rich prompts got dramatically more value than those who couldn't. A senior engineer with strong prompting skills might see 40% velocity improvement. A junior engineer with weak prompting might see 5%. AI tools were expected to have a leveling effect. Instead, the skill gap widened before it narrowed — and that widening was largest in the first 4 weeks before training ran.

What Prompt Training Actually Changed

The two 90-minute sessions in weeks 3–4 had measurable impact on output quality. Here's what moved the needle:

Context loading is the highest-leverage habit. Engineers who pasted only a function signature and asked for help got mediocre results. Engineers who loaded the surrounding module, relevant interfaces, and a description of system invariants got dramatically better output.

Negative constraints outperform positive ones.

// Weak — what most engineers start with:
"Write a service that processes customer invoices"

// Strong — what high performers use:
"Write a Spring service that processes customer invoices.
Do not introduce new dependencies. Do not use @Autowired field injection.
Follow the pattern in OrderService.java. The invoice total must never be
mutated after creation — use a new ProcessedInvoice object.
Explain your approach before writing any code."

"Explain first" for complex logic. Asking the AI to reason through the approach before writing code caught more errors than reviewing code after the fact. A 30-second prompt change saved hours of debugging in several cases.

Iterative over one-shot. Engineers who treated AI as a pair programmer (one step at a time, verify, continue) consistently outperformed those who generated large code blocks and integrated them whole.

Cost vs Return

Enterprise account pricing varies. A representative blended figure across tool tiers: approximately $38/engineer/month.

Self-reported time saved per engineer per week, validated against PR metrics: 3.2 hours.

At a blended fully-loaded hourly cost of $95/hour:

Tool cost: $38/month ≈ $9.50/week
Value of time saved: 3.2h × $95 = $304/week
ROI: ~32× on direct time savings

This excludes quality, onboarding speed, documentation improvements, and developer satisfaction effects. Even with a 90% discount on self-reported savings, the economics are strongly positive.

[!NOTE]
The highest self-reported savings came from mid-level engineers (3–6 years) with clear, well-scoped tasks. Senior engineers doing architecture or production debugging reported smaller direct gains but significant value in AI-assisted research and API exploration.

Process Changes Worth Making

Disclose AI-generated code in PRs. Not because AI code is worse on average — it isn't. But reviewers need to know to apply deeper scrutiny to the business logic layer rather than the syntax layer. A simple checkbox in the PR template is enough.

Prompt engineering on day one of onboarding. Engineers who receive a 2-hour prompt training session on their first week reach productive AI use in 1–2 weeks. Engineers who don't typically take 6–8 weeks to use AI effectively. Front-loading this is the single highest-leverage structural change.

Run prompt retrospectives. Every two weeks, one engineer shares a prompt that worked unexpectedly well or failed unexpectedly. 15 minutes. Document the good ones in a shared internal knowledge base. This consistently becomes one of the highest-engagement retro formats.

Add AI hallucinated dependencies to security review. Security teams should specifically look for AI-hallucinated library versions and dependency confusion in AI-assisted PRs. This class of issue appears reliably enough to warrant a dedicated check.

Key Takeaways for Engineering Leaders

The ROI is real, but it's not evenly distributed. Velocity improvements are largest for engineers with clear, well-scoped tasks and strong prompting habits. Expecting uniform gains across all experience levels and task types leads to disappointed stakeholders.

The defect profile changes, not just the rate. Prepare reviewers for this before rollout, not after the first incident.

Structured training pays back fast. The teams that ran prompt engineering sessions in week 1 outperformed those who installed the tools and called it done. The marginal cost of two 90-minute sessions is negligible against the compounding return.

Track coverage and defect rate, not just velocity. Velocity is the easiest metric to move with AI tools. Quality metrics tell you whether the gains are real or just moving fast in the wrong direction.

Measuring AI coding tool ROI at your organization? The defect-rate effects and review culture changes are the least-discussed parts of this space — always interested to compare notes. Find me on LinkedIn.

We Replaced Spring WebFlux with Virtual Threads. Here's the Honest Result.

Avaneesh Yadav — Tue, 25 Aug 2026 18:07:06 +0000

Three years ago I would have told you Spring WebFlux was the future of high-throughput Java services.

I was wrong — for most services.

This isn't a hot take. It's a post-mortem. In late 2025 we migrated four production microservices off WebFlux onto Spring Boot 3.3 with virtual threads. The results were good enough that I owe it to the Java community to write them down honestly — including the two services where we kept reactive, and why.

Why We Chose WebFlux in the First Place

Let's be fair to past-us. The decision made sense in 2022:

We were running on 2-core Fargate tasks with 512MB RAM
The services were mostly I/O-bound (database + downstream HTTP calls)
We'd seen the benchmarks — WebFlux could handle 10× the concurrent requests on the same hardware compared to a blocking Spring MVC stack
The team had a few engineers who genuinely understood Project Reactor

We got what we paid for. Thread counts stayed low. Latency was good. The services scaled cheaply.

So why did we migrate?

The Cost That Doesn't Show Up in Benchmarks

Reactive programming has a tax. You pay it in developer time.

Stack traces became archaeological digs. A NullPointerException 12 Mono operators deep produces a stack trace 40 frames long, most of which is Reactor internals. Debugging a production issue meant peeling back the operator chain layer by layer, mentally reconstructing the execution path.

Context propagation was a constant source of bugs. MDC logging, security context, distributed tracing — all of these depend on ThreadLocal state that simply doesn't work in a reactive pipeline. We had to manually propagate ReactiveSecurityContextHolder, wrap tracing spans into the Reactor Context, and write a custom MdcContextLifter. This was boilerplate we wrote once and then forgot about until it broke during an upgrade.

Onboarding new engineers took weeks. Not days. Weeks. We'd hand someone a Mono<Flux<ResponseEntity<List<OrderDTO>>>> return type and watch their eyes glaze over. Before they could write a feature, they needed to understand flatMap vs concatMap, cold vs hot publishers, and why you should never call a blocking method inside a reactive chain.

Testing was painful. StepVerifier is a great API if you know it well. If you don't, writing a test for a Flux that emits three elements and then an error is not intuitive. PRs would sit in review while engineers debated whether verifyComplete() or verify() was the right call to make.

The business impact: slower feature delivery, a higher defect rate on reactive-specific bugs, and an engineering team that visibly preferred working on our two Spring MVC services.

The Architecture: Before and After

graph TD
  subgraph BEFORE ["Before — Spring WebFlux (Netty)"]
    A1[HTTP Request] --> B1[Reactive Handler]
    B1 -->|Mono flatMap| C1[R2DBC Query]
    B1 -->|Mono flatMap| D1[WebClient HTTP]
    C1 -->|zip| E1[Operator Chain]
    D1 -->|zip| E1
    E1 -->|map/filter| F1[Response]
    G1[2 Netty Event Loop Threads] -.->|shared| B1
    G1 -.->|shared| C1
    G1 -.->|shared| D1
  end

  subgraph AFTER ["After — Spring MVC + Virtual Threads (Tomcat)"]
    A2[HTTP Request] --> B2[Blocking Handler]
    B2 -->|direct call| C2[JDBC Query]
    B2 -->|direct call| D2[RestClient HTTP]
    C2 --> E2[Plain Java Result]
    D2 --> E2
    E2 --> F2[Response]
    G2[Virtual Thread per Request] -.->|1:1| B2
  end

The reactive model uses a small fixed pool of event loop threads that never block — hence the complexity of the operator chains. Virtual threads let you write blocking code while the JVM parks the virtual thread during I/O instead of burning an OS thread. You get the same concurrency benefit without the reactive programming model.

The Code Difference

Here's the same endpoint — fetching an order with its line items from DB and enriching it with real-time pricing from a downstream service — in both models.

WebFlux version:

@GetMapping("/orders/{id}")
public Mono<ResponseEntity<EnrichedOrder>> getOrder(@PathVariable String id) {
    return orderRepository.findById(id)
        .switchIfEmpty(Mono.error(new OrderNotFoundException(id)))
        .flatMap(order ->
            lineItemRepository.findByOrderId(order.getId())
                .collectList()
                .flatMap(items ->
                    pricingClient.getCurrentPrices(
                            items.stream()
                                .map(LineItem::getSkuId)
                                .collect(Collectors.toList()))
                        .map(prices -> EnrichedOrder.from(order, items, prices))
                )
        )
        .map(ResponseEntity::ok)
        .onErrorResume(OrderNotFoundException.class,
            ex -> Mono.just(ResponseEntity.notFound().build()))
        .onErrorResume(ex -> {
            log.error("Failed to fetch order {}", id, ex);
            return Mono.just(ResponseEntity.internalServerError().build());
        });
}

Virtual threads version:

@GetMapping("/orders/{id}")
public ResponseEntity<EnrichedOrder> getOrder(@PathVariable String id) {
    Order order = orderRepository.findById(id)
        .orElseThrow(() -> new OrderNotFoundException(id));

    List<LineItem> items = lineItemRepository.findByOrderId(order.getId());

    Map<String, BigDecimal> prices = pricingClient.getCurrentPrices(
        items.stream().map(LineItem::getSkuId).toList()
    );

    return ResponseEntity.ok(EnrichedOrder.from(order, items, prices));
}

The WebFlux version: 25 lines, five operator types, three indentation levels, two error handlers. The virtual threads version: 10 lines, sequential flow, a single orElseThrow. The @ControllerAdvice handles exceptions globally — no per-endpoint error wiring.

This is the same logic. The same I/O operations. The same result. One version is readable by any Java developer; the other requires fluency in Reactor.

The Numbers

We ran load tests on all four services before and after migration. Here's the aggregate:

Metric	WebFlux	Virtual Threads	Change
OS threads (peak load)	18	22	+22%
Virtual threads (peak load)	—	~12,000	created on demand
Heap per instance	340 MB	290 MB	−15%
p50 latency	42 ms	38 ms	−10%
p99 latency	187 ms	123 ms	−34%
Throughput (req/s)	2,100	2,690	+28%
Lines of application code	8,400	4,960	−41%

[!NOTE]
p99 improvement was the surprise. We expected similar p99 or a slight regression — instead it improved significantly. Our hypothesis: the reactive operator chain adds non-trivial scheduling overhead under high concurrency (subscriptions, emissions, backpressure signals), whereas virtual threads park cleanly on I/O and incur no overhead when I/O resolves quickly.

OS thread count went up slightly because Tomcat uses carrier threads to mount virtual threads — but those carrier threads are idle most of the time. Real memory stays flat.

How to Migrate: The Exact Steps

Step 1: Update `pom.xml`

Remove spring-boot-starter-webflux. Add spring-boot-starter-web.

<!-- Remove -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

<!-- Add -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

If you use R2DBC, replace it with JDBC + your driver. If you use WebClient, replace it with RestClient (Spring 6.1+) or keep RestTemplate.

Step 2: Enable virtual threads

In application.properties (Spring Boot 3.2+):

spring.threads.virtual.enabled=true

That's it. Spring Boot automatically configures Tomcat to use virtual threads. No executor beans, no @Async wiring.

[!NOTE]
Requires Java 21+. On Java 21 virtual threads are GA. On Java 24 (current LTS candidate as of 2026) there are additional performance improvements to virtual thread pinning.

Step 3: Rewrite reactive handlers to blocking

The pattern is mechanical:

Mono<T> return → T return
.block() calls → direct method calls (now safe because virtual thread parks, not blocks)
Flux<T> → List<T> (for batch queries)
switchIfEmpty(Mono.error(...)) → .orElseThrow(...)
.onErrorResume(...) → try/catch or @ControllerAdvice
WebClient → RestClient

Step 4: Replace reactive data access

R2DBC is reactive-only. Switch to JDBC + Spring Data JPA or Spring Data JDBC. Connection pool should be HikariCP (default in Spring Boot). Set maximum-pool-size to something reasonable — virtual threads multiply fast but JDBC connections don't, so set a ceiling:

spring.datasource.hikari.maximum-pool-size=50

[!WARNING]
Do not set this to a huge number. With virtual threads you might be tempted to increase the pool size dramatically because OS threads are cheap. Don't. The database has a fixed connection limit. maximum-pool-size=50 means at most 50 concurrent DB operations; the rest of your virtual threads will wait efficiently. This is correct and expected behavior — virtual thread parking on a pool semaphore is free.

Step 5: Fix `ThreadLocal`-dependent code

The good news: MDC logging, Spring Security context, and Micrometer tracing all work out of the box with virtual threads — they use ThreadLocal which is per-virtual-thread. No MdcContextLifter, no manual context propagation. Delete all that boilerplate.

Step 6: Run your tests

Your integration tests (if they hit a real database) will largely pass. Unit tests that used StepVerifier need rewriting — now they're plain JUnit assertions. This is the most time-consuming part of the migration, not the application code.

What We Kept Reactive — and Why

We did not migrate everything. Two cases where WebFlux stayed:

Server-Sent Events endpoint. We have a /stream/events endpoint that pushes live updates to browser clients over SSE. Flux with a Sinks.Many is genuinely the right tool for this — it models a push stream naturally, handles backpressure, and integrates with Spring's SSE support. Virtual threads don't help here; the request handler needs to hold a channel open and push events asynchronously.

Kafka consumer pipeline. One service does heavy stream processing: consume → transform → filter → aggregate → produce. This is a Flux-from-Kafka pipeline with window, groupBy, and reduce operators. The reactive model fits the domain perfectly. Rewriting it to blocking code would require re-implementing the windowing and aggregation logic manually.

The rule we arrived at: reactive is the right tool when your data is inherently a stream. SSE, WebSocket, event sourcing, stream processing — these are genuinely reactive workloads. Request/response services that happen to do I/O are not.

Migration Checklist

[ ] Java 21+ confirmed on all targets
[ ] spring-boot-starter-webflux removed, spring-boot-starter-web added
[ ] spring.threads.virtual.enabled=true set
[ ] R2DBC replaced with JDBC (or keep R2DBC only where reactive data access is intentional)
[ ] WebClient replaced with RestClient
[ ] Reactive handlers rewritten to blocking (return T, not Mono<T>)
[ ] Reactor-specific error handling moved to @ControllerAdvice
[ ] MDC/security/tracing boilerplate deleted (it now works by default)
[ ] HikariCP pool size set explicitly
[ ] StepVerifier-based tests rewritten as plain JUnit
[ ] Load test before and after — verify p99 improvement
[ ] Keep reactive where the domain demands it (SSE, Kafka streams, WebSocket)

The Verdict

Virtual threads win for 80% of microservices. If your service is I/O-bound and does request/response work, the migration is worth doing. You get comparable or better performance, dramatically simpler code, and a team that can onboard new engineers in days instead of weeks.

Reactive still wins at the edges: true streaming endpoints, event-driven pipelines, and situations where back-pressure is a genuine domain requirement (not just a concurrency pattern).

The interesting implication: the argument for WebFlux in new greenfield services has collapsed. Unless you know you're building a streaming service, start with Spring MVC + virtual threads. You can always add a reactive endpoint later. You cannot easily un-reactive a whole service once the operator chains are everywhere.

We're Java developers. We like reading code top to bottom. It turns out the JVM will let us do that again — without sacrificing throughput.

Have you migrated from WebFlux to virtual threads? I'd love to hear what your numbers looked like — especially if they contradict mine. Find me on LinkedIn or leave a comment below.

How to Stream LLM Responses in Spring AI (Server-Sent Events)

Avaneesh Yadav — Tue, 07 Jul 2026 13:40:00 +0000

When you call an LLM the normal way, the user stares at a spinner while the model generates the entire answer, then the whole block appears at once. For anything longer than a sentence, that feels slow — even when the total time is identical to a streamed response. Streaming flips the experience: tokens appear as they're generated, so the user starts reading immediately. It's the single cheapest UX win in any LLM feature, and with Spring AI it's essentially one method call.

This guide covers the whole path: streaming from the ChatClient, exposing it as Server-Sent Events (SSE) in both WebFlux and classic Spring MVC, consuming it on the frontend, and the production details that trip people up.

Why stream at all?

Perceived latency, mostly. A model that takes 6 seconds to produce a 300-word answer feels broken if the user waits 6 seconds for a wall of text — but feels fast if words start flowing at 400ms. Streaming also lets you:

Show progress and allow the user to cancel a bad generation early.
Start downstream processing (rendering markdown, highlighting code) incrementally.
Reduce time-to-first-token as your primary latency metric, which is what users actually feel.

[!NOTE]
Streaming changes when tokens arrive, not the total cost or token count. You're billed the same — you're just improving the experience.

The one line that streams

Spring AI's ChatClient exposes streaming through .stream() instead of .call(). Where .call().content() returns a String, .stream().content() returns a reactive Flux<String> that emits chunks as the model produces them.

@Service
public class ChatService {

    private final ChatClient chatClient;

    public ChatService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public Flux<String> stream(String message) {
        return chatClient.prompt()
                .user(message)
                .stream()
                .content();
    }
}

That's the entire model-side change. Everything else is about getting that Flux to the browser.

Expose it as SSE with WebFlux

If your app uses Spring WebFlux, a streaming endpoint is trivial — return the Flux with a text/event-stream content type and Spring handles the SSE framing:

@RestController
public class ChatController {

    private final ChatService chatService;

    public ChatController(ChatService chatService) {
        this.chatService = chatService;
    }

    @GetMapping(value = "/api/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> stream(@RequestParam String message) {
        return chatService.stream(message);
    }
}

Each emitted chunk is sent to the client as its own data: event the moment the model produces it. If you want explicit SSE events (with ids, event names, or heartbeats), map to ServerSentEvent:

@GetMapping(value = "/api/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent<String>> stream(@RequestParam String message) {
    return chatService.stream(message)
            .map(chunk -> ServerSentEvent.builder(chunk).build());
}

No WebFlux? Use SseEmitter in Spring MVC

Plenty of enterprise apps are still on the classic servlet stack. You don't need to migrate to WebFlux — Spring MVC's SseEmitter streams fine. You subscribe to the Flux and push each chunk onto the emitter, completing (or erroring) it when the stream ends:

@GetMapping("/api/chat/stream")
public SseEmitter stream(@RequestParam String message) {
    SseEmitter emitter = new SseEmitter(Duration.ofMinutes(2).toMillis());

    chatService.stream(message).subscribe(
            chunk -> {
                try {
                    emitter.send(SseEmitter.event().data(chunk));
                } catch (IOException e) {
                    emitter.completeWithError(e);
                }
            },
            emitter::completeWithError,   // onError
            emitter::complete             // onComplete
    );

    return emitter;
}

[!TIP]
Give the SseEmitter an explicit timeout. The default can hang a servlet thread indefinitely if the client disappears mid-stream.

Consume the stream on the frontend

For a GET endpoint, the browser's built-in EventSource is the simplest consumer. Concatenate the chunks as they arrive:

function streamChat(message, onToken, onDone) {
  const url = `/api/chat/stream?message=${encodeURIComponent(message)}`;
  const source = new EventSource(url);

  source.onmessage = (event) => onToken(event.data);

  source.onerror = () => {
    // SSE closes with an error event when the server finishes the stream
    source.close();
    onDone();
  };
}

// Usage
let answer = "";
streamChat("Explain vector databases in one paragraph.",
  (token) => { answer += token; render(answer); },
  () => console.log("done"),
);

If you need POST (to send a large body or headers), EventSource won't work — use fetch with a streaming reader instead:

const res = await fetch("/api/chat/stream", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ message }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let answer = "";
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  answer += decoder.decode(value, { stream: true });
  render(answer); // you'll get raw SSE `data:` frames here — strip the prefixes
}

The flow, end to end

sequenceDiagram
    participant B as Browser
    participant C as Controller (SSE)
    participant S as ChatClient
    participant L as LLM Provider
    B->>C: GET /api/chat/stream?message=...
    C->>S: prompt().user(msg).stream().content()
    S->>L: streaming request
    loop each token
        L-->>S: token
        S-->>C: Flux emits chunk
        C-->>B: data: chunk
    end
    L-->>S: [stream end]
    C-->>B: connection closes

Production details that actually bite

Streaming works in a demo on the first try. These are the things that break it in production:

Proxy buffering. Nginx and many API gateways buffer responses by default, which defeats streaming — the client gets everything at once. Disable it for the stream route (proxy_buffering off; in Nginx) and send X-Accel-Buffering: no.
Compression. Response compression can also buffer the stream. Exclude text/event-stream from gzip.
Cancellation. When the user navigates away, the browser closes the connection. In WebFlux this cancels the Flux automatically (and stops the upstream LLM call); with SseEmitter, wire an onCompletion/onTimeout callback to dispose the subscription so you're not billed for tokens nobody will read.
Errors mid-stream. Once the first byte is sent you can't change the HTTP status. Emit a final SSE event with an error payload (e.g. event: error) so the client can distinguish "done" from "failed."
Resilience. LLM calls are flaky network calls. Apply timeouts and a fallback path — see AI system resilience for the patterns.
CORS. If your frontend is on a different origin, SSE still needs the usual CORS headers on the streaming route.

FAQ

Does streaming reduce token cost?
No — you're billed for the same input/output tokens. Streaming only changes when they arrive, improving perceived latency.

Do I have to switch to WebFlux to stream?
No. SseEmitter on classic Spring MVC streams a Flux fine. WebFlux is cleaner if you're already reactive, but it's not required.

Why does the client get everything at once instead of token-by-token?
Almost always a buffering proxy (Nginx/gateway) or response compression. Disable buffering and exclude text/event-stream from gzip.

Can I stream structured output (JSON) too?
You can stream the raw text, but partial JSON isn't parseable until it's complete. For structured responses, prefer .call().entity(...) unless you specifically want to render tokens as they arrive.

How do I stop the LLM call when the user cancels?
In WebFlux, closing the connection cancels the Flux and the upstream request automatically. With SseEmitter, dispose the reactive subscription in an onCompletion/onTimeout handler.

Streaming is the highest-leverage UX change you can make to an LLM feature, and Spring AI reduces it to swapping .call() for .stream(). Get the endpoint right, disable proxy buffering, handle cancellation, and your app will feel dramatically faster for zero extra cost.

New to Spring AI? Start with the Spring AI overview, then see tool calling and the enterprise Spring AI guide. Planning capacity and cost for a streaming feature? Estimate prompt size with the free LLM Token Counter.

Building a production AI feature and want a second pair of eyes on the architecture? Let's talk.