DEV Community: Avaneesh Yadav

System Design: Building a Notification Service for 50 Million Users

Avaneesh Yadav — Wed, 09 Sep 2026 15:18:39 +0000

Every non-trivial application sends notifications. Order confirmations. Password resets. Fraud alerts. Weekly digests. Yet the notification system is one of the most commonly underdesigned pieces of infrastructure — a single service that tries to do everything, with a shared database table that becomes a bottleneck at scale, and no retry logic until the first production outage.

This post designs a notification service that handles 50 million users, multiple channels (email, SMS, push, in-app), and 500,000 notifications per minute at peak. It's structured the way I'd walk through it in a system design interview — requirements first, capacity estimate, high-level design, component deep dive, trade-off decisions — with architecture diagrams at each stage and Spring Boot code for the non-obvious parts.

Requirements

Functional requirements:

Send notifications via email, SMS, push notification, and in-app feed
User preference management: per-channel, per-notification-type opt-in/opt-out
Template management: variable substitution, localization, A/B variants
Scheduling: send now, send at a specific time, send in user's local timezone
Deduplication: guarantee each notification is delivered at most once
Delivery status tracking: sent, delivered, failed, opened

Non-functional requirements:

50 million users, up to 500,000 notifications/minute at peak (campaign sends)
P99 delivery latency < 5 seconds for transactional notifications (password reset, fraud alert)
P99 delivery latency < 5 minutes for bulk campaign sends
At-least-once delivery (retry on failure), with idempotency to prevent duplicates
Multi-region deployment for geo-based regulations (GDPR, SMS routing)

Out of scope: Real-time bidirectional chat (that's a separate WebSocket service), notification analytics beyond delivery status.

Capacity Estimate

Before touching architecture, numbers:

Metric	Value	Reasoning
Daily notifications	200M	4 per user per day average
Peak notifications/min	500K	Campaign sends: 10M users × 3 notifications over 1 hour
Notification record size	2 KB	Template rendered + metadata
Daily storage (hot)	400 GB	200M × 2 KB
Storage retention	90 days	For audit + re-delivery
Kafka throughput needed	~8K messages/sec sustained, 8.5K peak	500K/min = 8,333/sec

This tells us we need a message queue (not direct DB writes) to absorb campaign spikes, and columnar or time-series storage for 90-day retention at 36 TB scale.

High-Level Architecture

flowchart TD
    A[API Clients\nMobile · Web · Backend Services] --> B[Notification API\nSpring Boot · REST]

    B --> C{Preference\nService}
    C -->|User opted in| D[Notification Router]
    C -->|User opted out| Z[/Drop/]

    D --> E[(Template\nService)]
    E --> F[Kafka\nnotification-events topic]

    F --> G[Email Worker\nSES / SendGrid]
    F --> H[SMS Worker\nTwilio / Vonage]
    F --> I[Push Worker\nFCM / APNs]
    F --> J[In-App Worker\nPostgreSQL + WebSocket]

    G & H & I & J --> K[(Delivery Log\nTimescaleDB)]
    G & H & I --> L[Dead Letter Queue\nnotification-dlq topic]

    L --> M[DLQ Processor\nRetry with backoff]
    M --> F

    style A fill:#1e293b,stroke:#334155,color:#e2e8f0
    style B fill:#0e7490,stroke:#0891b2,color:#f0f9ff
    style C fill:#7c3aed,stroke:#8b5cf6,color:#faf5ff
    style D fill:#0e7490,stroke:#0891b2,color:#f0f9ff
    style E fill:#0e7490,stroke:#0891b2,color:#f0f9ff
    style F fill:#b45309,stroke:#d97706,color:#fffbeb
    style G fill:#065f46,stroke:#059669,color:#f0fdf4
    style H fill:#065f46,stroke:#059669,color:#f0fdf4
    style I fill:#065f46,stroke:#059669,color:#f0fdf4
    style J fill:#065f46,stroke:#059669,color:#f0fdf4
    style K fill:#1e3a5f,stroke:#2563eb,color:#eff6ff
    style L fill:#7f1d1d,stroke:#dc2626,color:#fef2f2
    style M fill:#7f1d1d,stroke:#dc2626,color:#fef2f2
    style Z fill:#374151,stroke:#6b7280,color:#f9fafb

Seven distinct responsibilities, seven components. Let me walk through each.

Component Deep Dive

Notification API

The entry point. Two types of callers:

Transactional triggers — backend services (Order Service, Auth Service) calling directly with a structured event:

POST /api/v1/notifications
{
  "type": "ORDER_CONFIRMED",
  "recipientId": "usr_abc123",
  "templateVariables": {
    "orderId": "ord_xyz789",
    "totalAmount": "$149.99",
    "estimatedDelivery": "Sep 12"
  },
  "channels": ["EMAIL", "PUSH"],     // override preference? No — preference service decides
  "deduplicationKey": "order-confirm-ord_xyz789"  // idempotency
}

Campaign sends — marketing platform sending in bulk. The API accepts a campaign job (not individual notifications) and fans it out internally:

POST /api/v1/campaigns
{
  "templateId": "tmpl_monthly_digest",
  "audienceSegmentId": "seg_premium_users",
  "scheduledAt": "2026-09-10T09:00:00",
  "timezoneStrategy": "RECIPIENT_LOCAL"  // send at 9 AM in each user's timezone
}

The API layer does nothing except validate, deduplicate, and enqueue. No channel logic, no template rendering, no external calls. It must return in under 50ms.

Idempotency by deduplicationKey:

@PostMapping("/notifications")
public ResponseEntity<NotificationAck> send(@RequestBody @Valid SendRequest request) {
    String dedupKey = request.deduplicationKey();

    // Check Redis for recent duplicate (TTL = 24 hours)
    if (deduplicationCache.exists(dedupKey)) {
        return ResponseEntity.ok(NotificationAck.duplicate(dedupKey));
    }

    NotificationEvent event = eventFactory.create(request);
    kafkaProducer.send("notification-events", event.recipientId().toString(), event);
    deduplicationCache.set(dedupKey, Duration.ofHours(24));

    return ResponseEntity.accepted().body(NotificationAck.accepted(event.id()));
}

Using recipientId as the Kafka partition key ensures all notifications for one user land in the same partition — guaranteeing ordering for that user's in-app feed without requiring global ordering.

Preference Service

The preference check runs before any message hits the queue. A notification not worth sending is not worth routing, rendering, or failing on.

flowchart LR
    A[Incoming Event\ntype · recipientId] --> B{Global\nopt-out?}
    B -->|Yes| Z[/Drop silently/]
    B -->|No| C{Channel\npreference?}
    C -->|EMAIL disabled| D[Remove EMAIL\nfrom channels]
    C -->|SMS disabled| E[Remove SMS\nfrom channels]
    C -->|No channels left| Z
    C -->|Channels remain| F{Quiet hours?}
    F -->|In quiet hours| G[Schedule for\nend of quiet period]
    F -->|Not in quiet hours| H[Route to queue]

    style Z fill:#374151,stroke:#6b7280,color:#f9fafb
    style G fill:#7c3aed,stroke:#8b5cf6,color:#faf5ff
    style H fill:#065f46,stroke:#059669,color:#f0fdf4

Preference data lives in a Redis hash per user — fast enough to check before every notification without being a bottleneck:

HGETALL user:pref:usr_abc123
→ {
    "global_optout": "false",
    "email_enabled": "true",
    "sms_enabled": "false",
    "push_enabled": "true",
    "quiet_start": "22:00",
    "quiet_end": "08:00",
    "timezone": "America/New_York"
  }

Preference data is also written to PostgreSQL as the source of truth. Redis is the read cache with a 5-minute TTL. Preference updates invalidate the Redis key immediately.

Template Service

Templates are versioned, localized, and support A/B testing:

tmpl_order_confirmed/
├── v3/
│   ├── en/
│   │   ├── email.html
│   │   ├── email.subject
│   │   ├── sms.txt          (160 char limit enforced)
│   │   └── push.json        (title + body + data payload)
│   └── es/
│       ├── email.html
│       └── ...
└── v2/                      ← previous version, still served for in-flight notifications

Template rendering is synchronous within the worker — not a separate network call — because the worker already has the template variables from the event payload:

@Service
class TemplateRenderer {

    private final TemplateEngine templateEngine;  // Thymeleaf or Pebble

    public RenderedNotification render(NotificationEvent event, Channel channel) {
        Template template = templateRepository.getTemplate(
            event.type(), channel, event.locale(), event.templateVariant()
        );
        return new RenderedNotification(
            templateEngine.process(template.content(), event.variables()),
            channel
        );
    }
}

Templates are cached in-process (Caffeine, 10-minute TTL) in each worker. Template cache invalidation sends a Kafka event to all worker instances — they evict and reload on next render.

Kafka: The Central Backbone

One topic, multiple consumer groups — one per channel. This is the fan-out mechanism:

notification-events (topic)
├── email-notification-group    → Email Workers (12 instances)
├── sms-notification-group      → SMS Workers (8 instances)
├── push-notification-group     → Push Workers (10 instances)
└── inapp-notification-group    → In-App Workers (6 instances)

Each consumer group processes at its own pace. Email might be slower (due to ESP rate limits) without backing up SMS delivery. The channels are completely decoupled by the queue.

Partition count is the key scaling decision. At 8,333 messages/second peak and aiming for 1,000 messages/second per partition (leaving headroom), we need at minimum 9 partitions. Use 24 partitions — round number divisible by 1, 2, 3, 4, 6, 8, 12 consumers, giving flexibility to scale workers without rebalancing.

Channel Workers

Each worker follows the same structure regardless of channel:

@KafkaListener(
    topics = "notification-events",
    groupId = "email-notification-group",
    containerFactory = "emailKafkaListenerContainerFactory"
)
public class EmailNotificationWorker {

    private final TemplateRenderer templateRenderer;
    private final EmailProvider emailProvider;
    private final DeliveryLogService deliveryLog;

    @KafkaHandler
    public void handle(NotificationEvent event, Acknowledgment ack) {
        String notificationId = event.id();
        try {
            RenderedNotification rendered = templateRenderer.render(event, Channel.EMAIL);
            EmailResult result = emailProvider.send(rendered, event.recipientEmail());

            deliveryLog.record(notificationId, Channel.EMAIL, DeliveryStatus.SENT,
                Map.of("messageId", result.providerId(), "provider", "SES"));
            ack.acknowledge();

        } catch (RetryableException e) {
            // Don't ack — Kafka will redeliver to this consumer group
            log.warn("[{}] Retryable failure on email send, will redeliver: {}", notificationId, e.getMessage());
            // Don't ack — message will be redelivered after max.poll.interval.ms

        } catch (PermanentException e) {
            // Ack the message (remove from main topic) and send to DLQ
            deliveryLog.record(notificationId, Channel.EMAIL, DeliveryStatus.FAILED,
                Map.of("reason", e.getMessage()));
            dlqProducer.send("notification-dlq", event.recipientId(), new DlqEvent(event, e));
            ack.acknowledge();
        }
    }
}

Retryable vs. permanent exceptions:

Exception	Class	Behavior
Rate limit (429)	Retryable	No ack, redeliver after backoff
Network timeout	Retryable	No ack, redeliver
Invalid email address	Permanent	Ack + DLQ
Unsubscribed (ESP bounce)	Permanent	Ack + update preference + skip DLQ
Template rendering error	Permanent	Ack + DLQ + alert

Retry and DLQ Flow

sequenceDiagram
    participant W as Channel Worker
    participant K as Kafka Main Topic
    participant D as DLQ Topic
    participant P as DLQ Processor
    participant L as Delivery Log

    W->>K: Consume notification event
    W->>W: Attempt delivery

    alt Delivery succeeds
        W->>L: Record SENT status
        W->>K: Acknowledge (offset committed)

    else Retryable failure (429, timeout)
        Note over W,K: Worker does NOT acknowledge
        K-->>W: Redeliver after poll interval
        W->>W: Attempt 2 (exponential backoff: 1s, 2s, 4s)
        W->>W: Attempt 3

        alt Succeeds on retry
            W->>L: Record SENT status
            W->>K: Acknowledge
        else Still failing after 3 attempts
            W->>D: Publish to DLQ with attempt count + error
            W->>L: Record FAILED status
            W->>K: Acknowledge (remove from main topic)
        end

    else Permanent failure
        W->>D: Publish to DLQ immediately
        W->>L: Record PERMANENTLY_FAILED status
        W->>K: Acknowledge
    end

    P->>D: Consume DLQ event
    P->>P: Check if recoverable (age, error type, attempt count)

    alt Recoverable and attempt < 5
        P->>K: Re-publish to main topic with attempt_count+1
    else Max attempts exceeded
        P->>L: Record ABANDONED status
        P->>P: Alert on-call (PagerDuty)
    end

The DLQ processor runs on a 15-minute schedule for most failures. For transactional notifications (password reset, fraud alert), it runs every 60 seconds — delivery SLA matters more for these.

Delivery Log (TimescaleDB)

Delivery tracking generates time-series data. One row per notification per channel, written at every state transition:

CREATE TABLE notification_delivery_log (
    notification_id  UUID        NOT NULL,
    channel          VARCHAR(20) NOT NULL,
    recipient_id     UUID        NOT NULL,
    status           VARCHAR(30) NOT NULL,   -- SENT, DELIVERED, OPENED, FAILED, ABANDONED
    provider_msg_id  VARCHAR(100),           -- ESP/SMS provider's message ID for callbacks
    error_code       VARCHAR(50),
    attempt_count    INTEGER DEFAULT 1,
    metadata         JSONB,
    occurred_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    PRIMARY KEY (notification_id, channel, occurred_at)
);

SELECT create_hypertable('notification_delivery_log', 'occurred_at');

-- Retention: auto-drop partitions older than 90 days
SELECT add_retention_policy('notification_delivery_log', INTERVAL '90 days');

TimescaleDB's hypertable partitioning handles the write volume (10,000+ inserts/second at peak) and makes time-range queries efficient: "show me all failed email notifications in the last 24 hours" hits one or two partitions instead of scanning 36 TB.

Database Schema

erDiagram
    NOTIFICATIONS {
        uuid id PK
        varchar type
        uuid recipient_id FK
        varchar dedup_key UK
        jsonb template_variables
        varchar[] channels
        varchar status
        timestamptz scheduled_at
        timestamptz created_at
    }

    USER_PREFERENCES {
        uuid user_id PK
        boolean global_optout
        jsonb channel_preferences
        varchar quiet_start
        varchar quiet_end
        varchar timezone
        timestamptz updated_at
    }

    TEMPLATES {
        uuid id PK
        varchar notification_type
        varchar channel
        varchar locale
        integer version
        text content
        text subject
        boolean active
        timestamptz created_at
    }

    CAMPAIGNS {
        uuid id PK
        uuid template_id FK
        varchar segment_id
        varchar timezone_strategy
        varchar status
        timestamptz scheduled_at
        integer total_recipients
        integer sent_count
        timestamptz created_at
    }

    NOTIFICATIONS ||--o{ USER_PREFERENCES : "recipient"
    NOTIFICATIONS ||--o{ TEMPLATES : "uses"
    CAMPAIGNS ||--|| TEMPLATES : "uses"

The Trade-offs That Matter

1. Fan-out on write vs. fan-out on read

For campaign sends (10 million recipients), you have two strategies:

Fan-out on write: When the campaign is submitted, create one notification record per recipient immediately, then process from the queue. Pros: consistent delivery tracking, simple worker logic. Cons: 10M database writes before a single notification is sent.
Fan-out on read: Store one campaign record. Workers resolve the recipient list at delivery time. Pros: minimal writes upfront. Cons: recipient list must be re-resolved at send time, harder to track per-recipient status.

Recommendation: Fan-out on write, but asynchronously. The campaign scheduler writes notification records in batches of 1,000 to Kafka directly, not to PostgreSQL first. Workers consume from Kafka and persist delivery status. The notification record exists in the delivery log, not as a pre-created row in the notifications table.

2. At-least-once vs. exactly-once delivery

Exactly-once delivery with Kafka requires Kafka transactions and idempotent producers — significant operational complexity. At-least-once with deduplication in the API layer (the deduplicationKey) gives 99.9% of the benefit with 10% of the complexity.

The remaining 0.1% case: network partition causes the Kafka producer to retry after a successful write, sending the event twice. The deduplication cache (Redis with 24-hour TTL) catches this if the deduplicationKey is set. Callers that don't set a deduplicationKey accept at-least-once semantics explicitly.

3. Notification ordering

In-app notifications need to appear in chronological order for a given user. Email and SMS do not — the user sees them in their email client's sort order anyway.

The partition-by-recipientId strategy guarantees ordering within a partition for a given consumer group. For in-app, we additionally attach a sequence number from a per-user Redis counter when writing to the in-app feed. This handles the edge case where two events for the same user land in the same partition but the in-app worker processes them out of order (rare, but possible with concurrent workers).

4. Template rendering location

Render at the API layer (before enqueue) or at the worker layer (after dequeue)?

Rendering at the worker layer is better because:

Template data can be stale at render time rather than at enqueue time (template variables fetched fresh)
Failed renders become visible at delivery time with full context for debugging
Template updates between enqueue and delivery pick up the new template (important for campaigns scheduled days ahead)

The cost: the template variables must be carried in the Kafka event payload, not fetched again at render time. Include all needed data in the event at enqueue time.

Scaling to 500,000 Notifications Per Minute

flowchart LR
    subgraph "API Tier (stateless, auto-scale)"
        A1[API Instance 1]
        A2[API Instance 2]
        A3[API Instance N]
    end

    subgraph "Kafka (24 partitions)"
        K[notification-events]
    end

    subgraph "Email Workers (12 instances = 12 partitions)"
        E1[Email Worker 1]
        E2[Email Worker 2]
        E3[Email Worker 3..12]
    end

    subgraph "SMS Workers (8 instances)"
        S1[SMS Worker 1..8]
    end

    subgraph "Push Workers (10 instances)"
        P1[Push Worker 1..10]
    end

    subgraph "In-App Workers (6 instances)"
        I1[InApp Worker 1..6]
    end

    A1 & A2 & A3 --> K
    K --> E1 & E2 & E3
    K --> S1
    K --> P1
    K --> I1

At 500,000/min (8,333/sec), with 24 partitions, each partition handles ~347 events/sec. Email workers (slowest channel due to ESP rate limits) use 12 partitions. SMS, push, and in-app share the remaining 12 partitions across their consumer groups — each consumer group reads all 24 partitions independently.

Bottleneck analysis at 500K/min:

Channel	Max throughput per worker	Workers needed	Kafka partitions
Email	50 sends/sec (SES rate limit)	167 → use 200	24 (shared)
SMS	200 sends/sec (Twilio)	42 → use 50	24 (shared)
Push	1,000 sends/sec (FCM batch API)	8.3 → use 10	24 (shared)
In-App	2,000 writes/sec (Postgres)	4.2 → use 6	24 (shared)

Email is the bottleneck. The solution isn't more partitions — it's multiple ESP accounts (SendGrid primary, SES fallback), each with their own rate limit, and distributing sends across accounts in a round-robin pattern within the email worker.

What Goes Wrong in Production (And How to Handle It)

1. Preference service is down

Fail open (send the notification, skip preference check) for transactional notifications (password reset, fraud alert). Fail closed (drop the notification) for marketing content. The distinction is in the notification type's config.

2. ESP (email service provider) outage

The circuit breaker opens on the primary ESP. Workers automatically route to the secondary ESP. The DLQ processor drains backed-up notifications when the primary recovers.

3. Template rendering fails for one user

The worker sends the notification to the DLQ with error_type: TEMPLATE_RENDER_ERROR. The DLQ processor alerts engineering. The rest of the campaign continues — one bad template variable doesn't stop 9,999,999 other sends.

4. Kafka consumer lag spikes

Set an alert when consumer group lag exceeds 100,000 messages (about 12 seconds of peak throughput). Root cause is usually: a slow downstream (ESP rate-limited), workers processing too slowly (increase parallelism), or a Kafka partition imbalance (rebalance the consumer group).

Spring Boot Wiring

@Configuration
class NotificationKafkaConfig {

    @Bean
    KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, NotificationEvent>>
    emailKafkaListenerContainerFactory(ConsumerFactory<String, NotificationEvent> cf) {
        var factory = new ConcurrentKafkaListenerContainerFactory<String, NotificationEvent>();
        factory.setConsumerFactory(cf);
        factory.setConcurrency(12);  // 12 threads = 12 partitions for email group
        factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
        factory.getContainerProperties().setPollTimeout(3_000);

        // Dead letter publishing: after 3 failed attempts, route to DLQ
        var recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate,
            (rec, ex) -> new TopicPartition("notification-dlq", rec.partition()));
        var errorHandler = new DefaultErrorHandler(recoverer,
            new FixedBackOff(1_000L, 3L));
        factory.setCommonErrorHandler(errorHandler);

        return factory;
    }
}

Interview Cheat Sheet

If you're preparing for system design interviews, here's what interviewers are listening for on this question:

Topic	Key point
Fan-out mechanism	Consumer groups on Kafka, not fan-out in the API
Idempotency	Dedup key in Redis, not in the database (too slow)
Preference check	Before the queue, not inside the worker
Failure handling	Three categories: success, retryable, permanent
DLQ	Separate topic, separate processor, not just retry in-place
At-least-once	Acceptable with dedup key; exactly-once is complexity with marginal gain
Partition key	`recipientId` for ordering guarantee per user
Scale number	Know your throughput math (events/min → events/sec → partitions needed)
Database choice	TimescaleDB for delivery log time-series, Redis for preferences cache

The question interviewers are waiting for you to ask: "What's the SLA difference between transactional and marketing notifications?" Answering this before they prompt it shows you understand the business constraints that drive architecture, not just the technology.

Notification systems are where distributed systems complexity meets business requirements that change constantly (new channels, new preference types, new regulatory requirements). The architecture above is designed to absorb those changes: add a new channel by adding a consumer group, add a new preference type by extending the Redis hash, add a new region by adding a Kafka cluster in that region.

The design is the easy part. The hard part is the retries.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte, designing enterprise systems at scale. He writes about system design and AI architecture at buildingai.in.

The Modular Monolith: The Java Architecture Most Teams Should Be Using

Avaneesh Yadav — Wed, 09 Sep 2026 14:36:16 +0000

In 2018, the engineering blog you were reading told you to decompose everything into microservices. In 2022, the same blog started running post-mortems about distributed system complexity. In 2026, teams that survived both cycles are settling on something that looks a lot like what worked in 2010 — but done properly this time.

The modular monolith.

Not a big ball of mud. Not a distributed system pretending to be one service. A deliberately structured codebase where the modules are as isolated as microservices but everything runs in one process, shares one database transaction when needed, and gets deployed as one artifact.

If your team has fewer than 50 engineers, ships to fewer than 10 teams, and is spending more time debugging cross-service issues than building features — this post is for you. I'll show you the exact package structure, how to enforce module boundaries with ArchUnit, how to do cross-module communication without coupling, and when you'll actually know it's time to extract a service.

[!NOTE]
All examples use Java 21, Spring Boot 3.x, and a domain roughly based on an order management system — three bounded contexts: Orders, Catalog, and Customers. The patterns apply regardless of your domain.

Why Most "Microservices" Are Actually Distributed Monoliths

Before the architecture, a diagnostic. Here's the most common failure mode I encounter:

OrderService calls CustomerService via REST
    → CustomerService calls AccountService via REST
        → AccountService calls AuthService via REST

PlaceOrder() now has 4 network hops, 4 failure modes,
4 deployment dependencies, and distributed transaction risk.

This is not a microservices architecture. This is a monolith where the method calls were replaced with HTTP calls. The coupling is identical — it's just now expressed as API contracts instead of import statements, and every call can fail with a network error.

The teams I've seen thrive with microservices have two things: genuinely independent business capabilities with separate teams owning them end-to-end, and the organizational maturity to run distributed systems (platform team, service mesh, distributed tracing, on-call rotations per service). That's a real investment. For most teams, it's not the right trade.

The modular monolith gives you the architectural discipline of microservices without the operational overhead.

The Core Principle: Module Boundaries Are Enforced, Not Suggested

A modular monolith is not just "we put things in separate packages." Every Java codebase has packages. The difference is enforcement.

In a modular monolith:

One module cannot import the internal implementation of another module
Cross-module communication happens through defined interfaces, not direct class references
Database access is scoped per module (even if you share one database)
These rules are tested by the build — not just agreed on in a wiki

Let me show you what this looks like in practice.

Module Structure

src/main/java/com/company/app/
├── orders/
│   ├── api/                    ← Public API of this module
│   │   ├── OrderService.java   ← Interface (not implementation)
│   │   ├── OrderDto.java       ← Data structures safe to share
│   │   └── OrderEvents.java    ← Domain events this module publishes
│   ├── domain/                 ← Private: domain model
│   │   ├── Order.java
│   │   ├── OrderItem.java
│   │   ├── OrderStatus.java
│   │   └── OrderRepository.java
│   ├── application/            ← Private: use cases / application services
│   │   └── OrderServiceImpl.java
│   ├── infrastructure/         ← Private: DB, messaging, external APIs
│   │   ├── JpaOrderRepository.java
│   │   └── OrderMapper.java
│   └── OrdersConfig.java       ← @Configuration for this module's beans
│
├── catalog/
│   ├── api/
│   │   ├── CatalogService.java
│   │   ├── ProductDto.java
│   │   └── CatalogEvents.java
│   ├── domain/
│   ├── application/
│   ├── infrastructure/
│   └── CatalogConfig.java
│
├── customers/
│   ├── api/
│   ├── domain/
│   ├── application/
│   ├── infrastructure/
│   └── CustomersConfig.java
│
└── shared/
    ├── Money.java              ← Value objects used by multiple modules
    ├── CustomerId.java
    └── DomainEvent.java        ← Base type for all domain events

The rule is simple: everything in api/ is public. Everything else is private.

The orders module can call CatalogService (which is in catalog/api/). It cannot import catalog/domain/Product.java or catalog/infrastructure/JdbcCatalogRepository.java.

Enforcing Boundaries with ArchUnit

The structure above is meaningless without enforcement. Engineers will break it under deadline pressure. The compiler won't stop them — Java doesn't enforce package-level visibility like a module system does (and JPMS is a separate conversation). ArchUnit does.

@AnalyzeClasses(packages = "com.company.app", importOptions = ImportOption.DoNotIncludeTests.class)
class ModuleBoundaryTest {

    private static final List<String> MODULES = List.of("orders", "catalog", "customers");

    @ArchTest
    static final ArchRule no_module_accesses_internals_of_another =
        noClasses()
            .that().resideInAPackage("com.company.app.(*)..")
            .should().accessClassesThat()
            .resideInAPackage("com.company.app.(*).domain..")
            .orShould().accessClassesThat()
            .resideInAPackage("com.company.app.(*).application..")
            .orShould().accessClassesThat()
            .resideInAPackage("com.company.app.(*).infrastructure..")
            // unless you're inside that same module
            .andShould(beInTheSameModule());

    @ArchTest
    static final ArchRule infrastructure_does_not_depend_on_other_modules_infrastructure =
        noClasses()
            .that().resideInAPackage("com.company.app.(*).infrastructure..")
            .should().accessClassesThat()
            .resideInAPackage("com.company.app.(*).infrastructure..")
            .andShould(beInDifferentModules());

    @ArchTest
    static final ArchRule domain_has_no_spring_annotations =
        noClasses()
            .that().resideInAPackage("com.company.app.(*).domain..")
            .should().beAnnotatedWith(Component.class)
            .orShould().beAnnotatedWith(Service.class)
            .orShould().beAnnotatedWith(Repository.class);
}

The last rule — no Spring annotations in domain classes — enforces that your domain model is a pure Java object model, not a Spring-managed bean graph. This makes your domain independently testable without starting a Spring context.

This test suite runs in CI. A PR that breaks module boundaries doesn't merge. That's the whole mechanism.

Defining the Module API

Each module exposes its capabilities through a Java interface in api/:

// orders/api/OrderService.java
public interface OrderService {

    OrderDto placeOrder(PlaceOrderCommand command);
    OrderDto getOrder(UUID orderId);
    List<OrderDto> getOrdersForCustomer(UUID customerId);
    void cancelOrder(UUID orderId);
}

And DTO types — not domain objects:

// orders/api/OrderDto.java
public record OrderDto(
    UUID id,
    UUID customerId,
    List<OrderLineDto> lines,
    Money totalAmount,
    String status,
    Instant createdAt
) {}

public record OrderLineDto(UUID productId, String productName, int quantity, Money unitPrice) {}

Why DTOs instead of domain objects? Domain objects carry business logic and invariants — they're not safe to expose outside the module. Exposing Order directly creates an implicit coupling: any change to Order (adding a field, changing a type, splitting a class) breaks every caller. DTOs are a stable contract. Changes to domain model don't ripple out.

Cross-Module Communication via Domain Events

The most common question: "Orders needs to know when a product price changes in Catalog. How does that work without Orders depending on Catalog internals?"

Domain events. Catalog publishes an event. Orders subscribes.

// catalog/api/CatalogEvents.java
public sealed interface CatalogEvent extends DomainEvent permits
    CatalogEvent.ProductPriceChanged,
    CatalogEvent.ProductDiscontinued,
    CatalogEvent.ProductRestocked {

    record ProductPriceChanged(
        UUID productId,
        Money oldPrice,
        Money newPrice,
        Instant occurredAt
    ) implements CatalogEvent {}

    record ProductDiscontinued(
        UUID productId,
        String reason,
        Instant occurredAt
    ) implements CatalogEvent {}

    record ProductRestocked(
        UUID productId,
        int newQuantity,
        Instant occurredAt
    ) implements CatalogEvent {}
}

Using Java 21 sealed interfaces here is intentional. The permits clause makes the event hierarchy exhaustive — when you add a new event type, every switch that handles CatalogEvent gets a compile error if it doesn't handle the new case. Your IDE finds every subscriber that needs updating.

Publishing an event:

// catalog/application/CatalogServiceImpl.java
@Service
@Transactional
class CatalogServiceImpl implements CatalogService {

    private final ProductRepository productRepository;
    private final ApplicationEventPublisher eventPublisher;

    public void updatePrice(UUID productId, Money newPrice) {
        Product product = productRepository.findById(productId)
            .orElseThrow(() -> new ProductNotFoundException(productId));

        Money oldPrice = product.getPrice();
        product.setPrice(newPrice);
        productRepository.save(product);

        // Published AFTER the transaction commits via @TransactionalEventListener
        eventPublisher.publishEvent(
            new CatalogEvent.ProductPriceChanged(productId, oldPrice, newPrice, Instant.now())
        );
    }
}

Subscribing from Orders:

// orders/application/OrderPriceReconciler.java
@Component
class OrderPriceReconciler {

    private final OrderRepository orderRepository;

    @TransactionalEventListener(phase = AFTER_COMMIT)
    void onPriceChanged(CatalogEvent.ProductPriceChanged event) {
        // Find open orders containing this product
        List<Order> openOrders = orderRepository
            .findOpenOrdersContaining(event.productId());

        openOrders.forEach(order -> {
            order.updateLinePrice(event.productId(), event.newPrice());
            log.info("Updated price for product {} in order {} from {} to {}",
                event.productId(), order.getId(), event.oldPrice(), event.newPrice());
        });
    }

    @TransactionalEventListener(phase = AFTER_COMMIT)
    void onProductDiscontinued(CatalogEvent.ProductDiscontinued event) {
        // Cancel any pending orders containing a discontinued product
        orderRepository.findPendingOrdersContaining(event.productId())
            .forEach(order -> order.cancel("Product discontinued: " + event.reason()));
    }
}

The critical detail: @TransactionalEventListener(phase = AFTER_COMMIT).

Without this, the event fires during the catalog transaction, before the price update is committed. If the Orders handler reads the product from the database, it sees the old price. AFTER_COMMIT ensures the event fires only after the catalog transaction commits successfully — if catalog's transaction rolls back, Orders never hears about the price change.

This is the event-driven communication model that prevents half-committed states across modules.

Database Isolation Within One Schema

Sharing one database doesn't mean sharing tables. Each module owns its tables, and that ownership is enforced through a naming convention and an ArchUnit rule.

-- Orders module tables
CREATE TABLE ord_orders (id UUID PRIMARY KEY, customer_id UUID NOT NULL, ...);
CREATE TABLE ord_order_items (id UUID PRIMARY KEY, order_id UUID NOT NULL, ...);

-- Catalog module tables  
CREATE TABLE cat_products (id UUID PRIMARY KEY, name VARCHAR NOT NULL, ...);
CREATE TABLE cat_categories (id UUID PRIMARY KEY, ...);

-- Customers module tables
CREATE TABLE cus_customers (id UUID PRIMARY KEY, email VARCHAR NOT NULL, ...);
CREATE TABLE cus_addresses (id UUID PRIMARY KEY, customer_id UUID NOT NULL, ...);

The prefix (ord_, cat_, cus_) makes ownership visible. More importantly:

@ArchTest
static final ArchRule orders_module_only_accesses_ord_tables =
    noClasses()
        .that().resideInAPackage("com.company.app.orders..")
        .should().accessField(JdbcTemplate.class, "queryForList")
        // more practically: check @Table(name=...) annotations
        // for JPA entities in orders only use names starting with "ord_"

What about joins? This is the question that causes the most pushback. The answer is: you have two options, and both are valid.

Option A — Application-level join. Call CatalogService.getProducts(productIds) from the Orders module and merge in Java. This works well when the data sets are small and the query is selective.

Option B — Shared read model. For reporting queries that need cross-module joins, create a dedicated read model (a reports/ package or a separate schema) that denormalizes data explicitly for that purpose. This is the CQRS pattern applied at the module level.

What you do not do: write a SQL query that JOINs ord_orders and cat_products from inside the Orders module. That is a module boundary violation expressed in SQL instead of Java imports. ArchUnit won't catch it, but it's the same coupling.

Spring Configuration Per Module

Each module has its own @Configuration class that defines the beans for that module:

// orders/OrdersConfig.java
@Configuration
@ComponentScan(basePackages = "com.company.app.orders")
@EnableJpaRepositories(
    basePackages     = "com.company.app.orders.infrastructure",
    entityManagerFactoryRef = "ordersEntityManagerFactory",
    transactionManagerRef   = "ordersTransactionManager"
)
public class OrdersConfig {

    @Bean
    @Primary
    public LocalContainerEntityManagerFactoryBean ordersEntityManagerFactory(
        DataSource dataSource, JpaProperties jpaProperties) {
        var factory = new LocalContainerEntityManagerFactoryBean();
        factory.setDataSource(dataSource);
        factory.setPackagesToScan("com.company.app.orders.domain");
        factory.setJpaProperties(jpaProperties.getProperties());
        return factory;
    }
}

Separate EntityManagerFactory per module means JPA entity scanning is scoped. The Orders module's JPA context only knows about Orders domain classes. Catalog domain classes are invisible to it. This prevents accidental lazy-loading of entities across module boundaries and gives you the option to split the module to a separate database later — the JPA configuration change is isolated to one module's config file.

Testing Strategy

The module structure directly shapes your testing strategy. Each module is tested in three layers:

1. Domain tests — no Spring, no database (fast):

class OrderTest {

    @Test
    void order_with_no_items_cannot_be_placed() {
        Order order = Order.create(UUID.randomUUID());
        assertThatThrownBy(order::place)
            .isInstanceOf(InvalidOrderException.class)
            .hasMessageContaining("at least one item");
    }

    @Test
    void cancelled_order_cannot_be_placed() {
        Order order = Order.createWithItems(List.of(anItem()));
        order.cancel("test");
        assertThatThrownBy(order::place)
            .isInstanceOf(InvalidOrderException.class);
    }
}

These tests start in milliseconds. Run them 10,000 times in the time it takes a Spring context to load.

2. Application service tests — mocked dependencies (medium speed):

@ExtendWith(MockitoExtension.class)
class OrderServiceImplTest {

    @Mock CatalogService catalogService;
    @Mock OrderRepository orderRepository;
    @Mock ApplicationEventPublisher eventPublisher;
    @InjectMocks OrderServiceImpl orderService;

    @Test
    void placeOrder_verifies_product_exists_in_catalog() {
        when(catalogService.getProduct(PRODUCT_ID))
            .thenReturn(Optional.empty());

        assertThatThrownBy(() -> orderService.placeOrder(aPlaceOrderCommand()))
            .isInstanceOf(ProductNotFoundException.class);

        verify(orderRepository, never()).save(any());
    }
}

3. Module integration tests — real database, no other modules:

@SpringBootTest(classes = OrdersConfig.class)
@Testcontainers
class OrdersModuleIntegrationTest {

    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16");

    @MockBean CatalogService catalogService;  // mock the api, not the impl
    @MockBean CustomerService customerService;

    @Autowired OrderService orderService;

    @Test
    @Sql("/orders/test-data.sql")
    void getOrdersForCustomer_returns_only_their_orders() {
        List<OrderDto> orders = orderService.getOrdersForCustomer(CUSTOMER_A_ID);
        assertThat(orders).allMatch(o -> o.customerId().equals(CUSTOMER_A_ID));
    }
}

Note @SpringBootTest(classes = OrdersConfig.class) — this starts only the Orders module's Spring context. No Catalog beans, no Customer beans. The other modules' @Service interfaces are mocked. This makes the test ~5× faster than a full context load and keeps the test scope honest.

4. Cross-module tests — full context, event integration (slow, few):

@SpringBootTest
@Testcontainers
class OrderCatalogIntegrationTest {

    @Autowired CatalogService catalogService;
    @Autowired OrderRepository orderRepository;

    @Test
    void price_change_updates_pending_orders() {
        // Arrange: create a pending order with a product
        UUID productId = catalogService.createProduct(aProductRequest()).id();
        UUID orderId = orderService.placeOrder(PlaceOrderCommand.of(productId, 1)).id();

        // Act: change the product price
        catalogService.updatePrice(productId, Money.of(99, "USD"));
        // give @TransactionalEventListener AFTER_COMMIT time to fire
        awaitility().untilAsserted(() -> {
            OrderDto order = orderService.getOrder(orderId);
            assertThat(order.totalAmount()).isEqualTo(Money.of(99, "USD"));
        });
    }
}

Keep the count of these tests small — they're slow and test module integration, not module logic.

Module Communication Anti-Patterns

Three patterns that look fine in code review and cause problems six months later:

1. Sharing domain objects across modules

// ❌ Orders module importing Catalog domain object
import com.company.app.catalog.domain.Product;

class OrderServiceImpl {
    public void placeOrder(PlaceOrderCommand cmd) {
        Product product = productRepository.findById(cmd.productId()); // direct JPA
        // ...
    }
}

When Catalog adds a field to Product for a new feature, Orders' code and database query break. Use the CatalogService interface and ProductDto instead.

2. Bypassing the API to reach the repository

// ❌ Orders directly autowiring Catalog's JPA repository
@Autowired CatalogProductRepository catalogProductRepo; // private to catalog module

List<Product> products = catalogProductRepo.findByIds(ids); // bypasses business logic

The repository access bypasses any business rule in CatalogServiceImpl. If Catalog adds authorization checks or audit logging to getProducts(), Orders won't get them. ArchUnit catches this in CI.

3. Chatty synchronous communication for every operation

// ❌ Orders calling Catalog for every line item in a loop
order.getItems().forEach(item -> {
    ProductDto product = catalogService.getProduct(item.productId()); // N calls
    // ...
});

Even in a monolith this is N database queries. Use bulk fetching:

// ✅ Batch load
Set<UUID> productIds = order.getItems().stream()
    .map(OrderItem::getProductId).collect(toSet());
Map<UUID, ProductDto> products = catalogService.getProducts(productIds); // 1 query

The Real Answer to "When Do You Extract a Microservice?"

After working with this architecture on several teams, I've converged on two signals that actually predict when extraction is worth it — not the ones usually cited.

Signal 1: Two teams want to deploy independently at different cadences and their releases are colliding.

If the Catalog team ships 3 times a day and their changes keep blocking Orders deployments (or vice versa), the deployment coupling is hurting real people. Extract at the deployment boundary, not the domain boundary.

Signal 2: One module has dramatically different infrastructure needs that are fighting the rest of the application.

Catalog might need a read-replica for search queries and a CDN-friendly caching strategy. Orders needs ACID transactions and message queues. These are genuinely different operational profiles. If forcing them into one deployment means both get suboptimal infrastructure, extraction might be worth it.

Signals that don't actually predict success:

"The module is getting big" — 50K lines in one module is fine. Microservices don't reduce code volume.
"We want to use different technology" — Mixing JVM languages across module boundaries is asking for integration pain. One Java monolith beats a polyglot distributed system for most teams.
"It will be easier to scale" — Unless you've proven that specific module is the bottleneck, horizontal scaling of the whole monolith is usually easier, faster, and cheaper.

When you do extract, the module boundary you built here makes it clean. The api/ package becomes the service's public API contract. The domain and application packages become the service's implementation. Infrastructure switches from local beans to HTTP clients (or message consumers). The module already behaves like a service — you're just moving it to its own process.

The Numbers

We adopted this architecture at a client engagement: a six-year-old Spring Boot monolith, ~150K lines of Java, 11-person team, 4 bounded contexts.

Before the restructure: engineers regularly broke each other's code, because everything was in com.company.app.service.* and import-anything-from-anywhere was the norm. Build time was the only feedback loop.

After a 3-week restructure (largely automated — this is the kind of transformation where Claude Code genuinely helps with the mechanical moves):

Cross-module bugs in CI: dropped from 3–5 per week to essentially zero (ArchUnit catches them)
Build time: unchanged (structural, not a build change)
Onboarding: new engineers understand the module they're working in within a day, instead of needing to understand the whole codebase
Time to extract a module to a real microservice when the business needed it: 2 days, not 2 weeks

The two days is the number that justified the investment.

Getting Started

If you're converting an existing codebase, the path is:

Identify 3–5 natural domain areas in your codebase (what nouns does your business talk about? Orders, Customers, Catalog, Payments, etc.)
Create the package structure — this is mechanical, Claude Code does it well
Add the ArchUnit test with your boundaries — it will fail on everything first
Fix violations module by module, starting with infrastructure (no cross-module repository access), then domain (no imported domain objects), then API (replace direct class imports with interface + DTO)
For new features, enforce the structure from day one — don't wait until the refactor is complete

The structure pays off immediately: even with 30% of violations still remaining, you have a clear map of where the coupling exists. That's half the battle.

Most teams don't need microservices. They need modules. The modular monolith gives you the architectural clarity of microservices, the operational simplicity of a single deployment, and a realistic migration path to distributed services if you ever genuinely need them.

Build the right thing first.

The package structure and ArchUnit rules in this post are available as a GitHub template — check the footer for the link.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte, building enterprise systems with Java and AI. He writes at buildingai.in.

Designing the AI Request Pipeline: 8 Layers Between User Input and Your LLM

Avaneesh Yadav — Wed, 09 Sep 2026 14:36:14 +0000

Most teams build their first AI feature in a weekend. A controller that takes user input, calls an LLM, and returns the response. Three files, maybe four. It works beautifully in the demo.

Then production happens.

The 11 PM alert is always the same: "AI feature returning garbage / not responding / costing $4,000 this week." And the root cause is always the same: the gap between what a demo LLM call needs and what a production AI system needs is enormous, and nobody mapped that gap before shipping.

I've spent the last two years building AI features into enterprise Java applications. The systems that hold up in production all share the same structural pattern — what I call the AI Request Pipeline: eight distinct layers between user input and LLM response, each one solving a specific class of production failure.

This post maps every layer, explains what breaks without it, and shows the Spring Boot implementation.

[!NOTE]
Code samples use Spring AI 1.x with Groq/OpenAI-compatible endpoints and PostgreSQL (pgvector) for the semantic cache. Architecture patterns apply regardless of your AI provider or language.

The Pipeline at a Glance

User Input
    │
    ▼
┌─────────────────────────────┐
│  1. Input Guardrail Layer   │  PII stripping, length limit, injection detection
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  2. Semantic Cache Layer    │  Exact + fuzzy cache lookup before any LLM call
└──────────────┬──────────────┘
               │ (cache miss)
               ▼
┌─────────────────────────────┐
│  3. Model Router Layer      │  Simple task → small model, complex → large model
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  4. Prompt Assembly Layer   │  Context injection, system prompt, RAG retrieval
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  5. LLM Execution Layer     │  Retry, circuit breaker, timeout, fallback
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  6. Output Validation Layer │  Schema check, safety check, hallucination guard
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  7. Response Cache Layer    │  Store validated responses for future cache hits
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│  8. Observability Layer     │  Latency, tokens, cost, quality metrics
└──────────────┬──────────────┘
               │
               ▼
         LLM Response

None of these layers are optional in production. Each one exists because something specific breaks without it. Let me walk through them.

Layer 1: Input Guardrail Layer

What breaks without it: PII in your LLM provider's logs. Prompt injection attacks. $10,000 in compute from one user sending 500KB documents. Legal exposure.

The input guardrail layer runs before anything else — before the cache, before the router, before any LLM call. It's cheap and synchronous.

@Component
public class InputGuardrailService {

    private static final int MAX_INPUT_CHARS = 4_000;
    private static final Pattern PII_PATTERN = Pattern.compile(
        "\\b(\\d{3}-\\d{2}-\\d{4}|\\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\\.[A-Z]{2,}\\b|\\b\\d{16}\\b)",
        Pattern.CASE_INSENSITIVE
    );
    private static final List<String> INJECTION_MARKERS = List.of(
        "ignore previous instructions",
        "ignore all prior instructions",
        "disregard your system prompt",
        "you are now",
        "act as if you are"
    );

    public GuardrailResult validate(String userInput) {
        if (userInput == null || userInput.isBlank()) {
            return GuardrailResult.reject("EMPTY_INPUT");
        }
        if (userInput.length() > MAX_INPUT_CHARS) {
            return GuardrailResult.reject("INPUT_TOO_LONG")
                .withDetail("max", MAX_INPUT_CHARS, "actual", userInput.length());
        }

        String normalized = userInput.toLowerCase();
        for (String marker : INJECTION_MARKERS) {
            if (normalized.contains(marker)) {
                return GuardrailResult.reject("PROMPT_INJECTION_DETECTED");
            }
        }

        // Strip PII before it touches logs or the LLM provider
        String sanitized = PII_PATTERN.matcher(userInput).replaceAll("[REDACTED]");

        return GuardrailResult.allow(sanitized);
    }
}

A few design decisions worth explaining:

Reject vs. strip. PII gets stripped (returned to the pipeline as a sanitized string) because stripping doesn't change intent. Injection markers get rejected outright because there's no safe version of "ignore your system prompt."

Length limit. This is a business decision, not a technical one. For a chat feature, 4,000 characters is generous. For a document analysis feature, you might go higher but add a cost estimate shown to the user before they submit.

What this layer does NOT do. It does not evaluate whether the input is on-topic (that's model routing), and it does not flag offensive content (that's a dedicated content moderation API if your use case needs it). One layer, one job.

Layer 2: Semantic Cache Layer

What breaks without it: Every LLM call costs money and adds latency. Users asking the same question — or very similar questions — each pay the full cost. A Q&A feature with 10,000 daily queries spends 10× what it needs to.

The semantic cache checks whether a sufficiently similar question has been answered before. It returns the cached answer in milliseconds and costs nothing.

@Service
public class SemanticCacheService {

    private final EmbeddingModel embeddingModel;
    private final JdbcTemplate jdbc;
    private static final double SIMILARITY_THRESHOLD = 0.92;

    public Optional<CachedResponse> lookup(String input, String context) {
        float[] queryEmbedding = embeddingModel.embed(input);

        // pgvector cosine similarity query
        String sql = """
            SELECT response, metadata
            FROM ai_response_cache
            WHERE context_key = ?
              AND 1 - (embedding <=> ?::vector) > ?
              AND expires_at > NOW()
            ORDER BY embedding <=> ?::vector
            LIMIT 1
            """;

        return jdbc.query(sql,
            rs -> rs.next()
                ? Optional.of(new CachedResponse(rs.getString("response"), true))
                : Optional.empty(),
            context,
            pgvector(queryEmbedding),
            SIMILARITY_THRESHOLD,
            pgvector(queryEmbedding)
        );
    }

    public void store(String input, String context, String response, Duration ttl) {
        float[] embedding = embeddingModel.embed(input);
        jdbc.update("""
            INSERT INTO ai_response_cache (input_text, embedding, context_key, response, expires_at)
            VALUES (?, ?::vector, ?, ?, NOW() + ? * INTERVAL '1 second')
            ON CONFLICT DO NOTHING
            """,
            input, pgvector(embedding), context, response, ttl.toSeconds()
        );
    }
}

The similarity threshold is the most important tuning parameter. 0.92 is a good starting point for most Q&A workloads. Too low: you serve cached responses to questions that are actually different. Too high: you miss obvious cache hits. Tune it on a labeled sample: pick 50 query pairs, label them as "same intent" or "different intent," and find the threshold that separates them.

TTL strategy by content type:

Content type	TTL
FAQ answers	7 days
Product information	24 hours
Real-time data (prices, status)	0 — skip cache entirely
Personalized responses	0 — skip cache entirely

What the cache does NOT store. Don't cache user-specific, personalized, or time-sensitive responses. The context_key parameter lets you namespace the cache by feature (FAQ, support, onboarding) so different features don't cross-pollinate.

Layer 3: Model Router Layer

What breaks without it: You use GPT-4o / Claude Opus for everything. A simple yes/no classification question costs 50× what it should. Your monthly AI bill is 10× your estimate.

The model router assigns each request to the cheapest model capable of handling it reliably.

@Component
public class ModelRouter {

    public record RoutingDecision(String modelId, String rationale) {}

    // Ordered from cheapest to most capable
    private static final String SMALL_MODEL  = "llama-3.1-8b-instant";
    private static final String MEDIUM_MODEL = "claude-haiku-4-5";
    private static final String LARGE_MODEL  = "claude-sonnet-5";

    public RoutingDecision route(RoutingContext ctx) {
        // Classification tasks: low complexity, structured output
        if (ctx.taskType() == TaskType.CLASSIFICATION
                || ctx.taskType() == TaskType.EXTRACTION
                || ctx.taskType() == TaskType.SUMMARIZATION) {
            return new RoutingDecision(SMALL_MODEL, "structured low-complexity task");
        }

        // Short responses with clear structure: medium model
        if (ctx.expectedResponseTokens() < 500
                && ctx.taskType() != TaskType.CODE_GENERATION
                && ctx.taskType() != TaskType.REASONING) {
            return new RoutingDecision(MEDIUM_MODEL, "short-form task");
        }

        // Code generation, multi-step reasoning, ambiguous intent: large model
        return new RoutingDecision(LARGE_MODEL, "complex task requires full capability");
    }
}

The routing signal comes from your application, not the LLM. Your code knows what the user clicked, what feature they're in, what schema the output must match. Use that knowledge. Don't ask an LLM to classify the request — that's recursive and expensive.

Measure routing quality. After routing a request to the small model, track user satisfaction signals (thumbs up/down, follow-up "can you try again?"). If the small model has low satisfaction on a task type, bump that task type to the medium model. The routing table should evolve based on production data.

Layer 4: Prompt Assembly Layer

What breaks without it: The LLM gets no context and hallucinates answers. Context is assembled inconsistently across different code paths. System prompts drift as engineers edit them without coordination.

Prompt assembly is the layer where you compose the complete message that will go to the LLM: system prompt, retrieved context (if RAG), conversation history (if multi-turn), and the formatted user input.

@Service
public class PromptAssemblyService {

    private final RagRetrieverService ragRetriever;
    private final SystemPromptRegistry promptRegistry;
    private final ConversationHistoryService historyService;

    public Prompt assemble(AssemblyRequest req) {
        String systemPrompt = promptRegistry.get(req.featureId(), req.modelId());

        // RAG: retrieve relevant context if this feature needs grounding
        String ragContext = "";
        if (req.requiresGrounding()) {
            List<Document> docs = ragRetriever.retrieve(req.userInput(), req.featureId(), 5);
            ragContext = formatRagContext(docs);
        }

        // Conversation history (last N turns, budget-capped)
        List<Message> history = historyService.getRecentTurns(
            req.sessionId(), MAX_HISTORY_TURNS
        );

        // Assemble: system → rag context → history → user input
        // IMPORTANT: most-important instructions go at START and END of context,
        // not the middle — attention is weakest in the middle of long contexts
        var messages = new ArrayList<Message>();
        messages.add(new SystemMessage(buildSystemMessage(systemPrompt, ragContext)));
        messages.addAll(history);
        messages.add(new UserMessage(req.userInput()));

        return new Prompt(messages, buildOptions(req));
    }

    private String buildSystemMessage(String systemPrompt, String ragContext) {
        if (ragContext.isBlank()) return systemPrompt;
        return systemPrompt + """

            ---
            REFERENCE CONTEXT (use this to answer the user's question):
            %s
            ---
            Answer based only on the reference context above. If the answer is not in the context, say so.
            """.formatted(ragContext);
    }

    private ChatOptions buildOptions(AssemblyRequest req) {
        return ChatOptionsBuilder.builder()
            .model(req.modelId())
            .maxTokens(req.maxResponseTokens())
            .temperature(req.taskType() == TaskType.FACTUAL ? 0.1f : 0.7f)
            .build();
    }
}

The key architectural decision here is SystemPromptRegistry. System prompts are not strings scattered across service classes. They are versioned, centrally managed, and model-aware (the same logical prompt may need different wording for different models). Store them in your database or a config file, not in Java string literals. This makes A/B testing and rollback of prompt changes possible without code deploys.

Token budget management. The conversation history and RAG context are both dynamic. Before sending the final prompt, count the total tokens and trim if needed — history first (oldest turns), then RAG context (lowest-relevance chunks last). Never trim the system prompt or the user's current message.

Layer 5: LLM Execution Layer

What breaks without it: A single LLM provider outage takes down your entire feature. Transient 429s (rate limit) fail requests that would succeed on retry. Slow responses hold threads.

This layer wraps the actual LLM call with retry logic, a circuit breaker, timeout enforcement, and fallback to an alternative model.

@Service
public class LlmExecutionService {

    private final ChatModel primaryModel;
    private final ChatModel fallbackModel;
    private final CircuitBreaker circuitBreaker;
    private final MeterRegistry metrics;

    public ChatResponse execute(Prompt prompt, String modelId, String requestId) {
        Timer.Sample timer = Timer.start(metrics);

        try {
            return circuitBreaker.executeSupplier(() ->
                executeWithRetry(prompt, modelId, requestId)
            );
        } catch (CallNotPermittedException e) {
            // Circuit is open — use fallback model
            log.warn("[{}] Circuit open for {}, using fallback", requestId, modelId);
            metrics.counter("ai.circuit_breaker.fallback", "model", modelId).increment();
            return fallbackModel.call(prompt);
        } finally {
            timer.stop(metrics.timer("ai.llm.latency", "model", modelId));
        }
    }

    @Retryable(
        retryFor = { RateLimitException.class, ServiceUnavailableException.class },
        maxAttempts = 3,
        backoff = @Backoff(delay = 1000, multiplier = 2, jitter = 500)
    )
    private ChatResponse executeWithRetry(Prompt prompt, String modelId, String requestId) {
        log.debug("[{}] Calling model {}", requestId, modelId);
        return primaryModel.call(prompt);
    }
}

Circuit breaker configuration matters. For LLM calls:

Failure threshold: 50% over a 10-second window (lower than typical services because LLM errors are expensive)
Wait duration in OPEN state: 30 seconds (LLM providers recover faster than traditional databases)
Half-open: allow 3 test calls before fully closing

Timeout is non-negotiable. Set an explicit timeout on every LLM call. A hung LLM call with no timeout holds a thread (or blocks a virtual thread from making useful progress on other work). 30 seconds for streaming, 60 seconds for large completions. Never infinity.

The fallback model is your safety net, not your primary. The fallback doesn't have to match quality with the primary. It just has to return something coherent. A degraded response is better than an error.

Layer 6: Output Validation Layer

What breaks without it: Schema mismatches crash downstream code. Hallucinated entity IDs cause 500s when your code tries to look them up in a database. Unsafe content reaches users.

@Service
public class OutputValidationService {

    private final ObjectMapper objectMapper;
    private final GroundingValidator groundingValidator;

    public ValidationResult validate(String rawOutput, ValidationSpec spec) {
        // 1. Schema validation (for structured outputs)
        if (spec.requiresJson()) {
            try {
                JsonNode parsed = objectMapper.readTree(rawOutput);
                List<String> violations = spec.jsonSchema().validate(parsed);
                if (!violations.isEmpty()) {
                    return ValidationResult.fail(ValidationFailure.SCHEMA_MISMATCH, violations);
                }
            } catch (JsonProcessingException e) {
                return ValidationResult.fail(ValidationFailure.INVALID_JSON,
                    List.of("Output is not valid JSON: " + rawOutput.substring(0, 100)));
            }
        }

        // 2. Business rule validation
        for (BusinessRule rule : spec.businessRules()) {
            RuleResult result = rule.evaluate(rawOutput);
            if (!result.passed()) {
                return ValidationResult.fail(ValidationFailure.BUSINESS_RULE, result.violations());
            }
        }

        // 3. Grounding check (for RAG responses)
        if (spec.requiresGrounding() && spec.sourceDocuments() != null) {
            double groundingScore = groundingValidator.score(rawOutput, spec.sourceDocuments());
            if (groundingScore < 0.7) {
                return ValidationResult.fail(ValidationFailure.HALLUCINATION_RISK,
                    List.of("Response not sufficiently grounded in source documents"));
            }
        }

        return ValidationResult.pass(rawOutput);
    }
}

What to do when validation fails. You have three options, and the right one depends on the failure type:

Retry with error context. For SCHEMA_MISMATCH: retry the LLM call with an additional user message: "Your response failed JSON schema validation with these errors: [errors]. Try again." Works well for schema failures, success rate ~85% on retry.
Return a fallback response. For HALLUCINATION_RISK: return a canned response ("I don't have reliable information about this") rather than a potentially wrong one. This is correct for factual Q&A.
Escalate to human. For safety violations: flag the conversation for review, return a neutral response to the user. Don't auto-retry.

Grounding validation implementation. The groundingValidator compares the response against the source documents using embedding similarity. For each factual claim in the response, it checks whether a semantically similar statement exists in the source. This isn't perfect — it's a heuristic, not proof — but it catches the most egregious hallucinations.

Layer 7: Response Cache Layer

What breaks without it: You paid for a validated, high-quality response and immediately throw it away. The next identical query pays again.

This layer stores validated responses. It runs after validation — you never cache a response that failed validation.

@Service
public class ResponseCacheService {

    private final SemanticCacheService semanticCache;
    private final Map<TaskType, Duration> TTL_BY_TYPE = Map.of(
        TaskType.FAQ,              Duration.ofDays(7),
        TaskType.PRODUCT_INFO,     Duration.ofHours(24),
        TaskType.SUMMARIZATION,    Duration.ofHours(1),
        TaskType.CLASSIFICATION,   Duration.ofHours(12),
        TaskType.PERSONALIZED,     Duration.ZERO,   // never cache
        TaskType.REAL_TIME,        Duration.ZERO    // never cache
    );

    public void store(CacheStoreRequest req) {
        Duration ttl = TTL_BY_TYPE.getOrDefault(req.taskType(), Duration.ofHours(1));
        if (ttl.isZero()) return; // explicitly not cached

        semanticCache.store(
            req.userInput(),
            req.contextKey(),
            req.validatedResponse(),
            ttl
        );
    }
}

This layer should also handle cache invalidation. When your product information changes, you need to invalidate cached responses about that product. Design the context_key to include a version or category that you can invalidate in bulk:

-- Invalidate all product info cache entries when catalog updates
DELETE FROM ai_response_cache
WHERE context_key LIKE 'product-info:%'
  AND expires_at > NOW();

Layer 8: Observability Layer

What breaks without it: You can't answer the question "is our AI feature working?" You can't tell whether your model change improved quality. You can't debug why a user got a bad response.

Observability for AI features needs four dimensions that don't exist in normal API observability:

@Aspect
@Component
public class AiPipelineObservabilityAspect {

    private final MeterRegistry metrics;

    @Around("@annotation(TrackAiRequest)")
    public Object observe(ProceedingJoinPoint pjp) throws Throwable {
        String requestId = UUID.randomUUID().toString();
        long start = System.currentTimeMillis();

        AiRequestContext ctx = AiRequestContext.current();

        try {
            Object result = pjp.proceed();
            long latency = System.currentTimeMillis() - start;

            // 1. Latency (broken down by layer)
            metrics.timer("ai.pipeline.latency",
                "layer", ctx.currentLayer(),
                "model", ctx.modelId(),
                "cache_hit", String.valueOf(ctx.cacheHit())
            ).record(latency, TimeUnit.MILLISECONDS);

            // 2. Token cost tracking
            if (result instanceof ChatResponse response) {
                int inputTokens  = response.getMetadata().getUsage().getPromptTokens();
                int outputTokens = response.getMetadata().getUsage().getGenerationTokens();

                metrics.counter("ai.tokens.input",  "model", ctx.modelId(), "feature", ctx.featureId())
                    .increment(inputTokens);
                metrics.counter("ai.tokens.output", "model", ctx.modelId(), "feature", ctx.featureId())
                    .increment(outputTokens);
            }

            // 3. Cache hit rate
            metrics.counter("ai.cache",
                "result", ctx.cacheHit() ? "hit" : "miss",
                "feature", ctx.featureId()
            ).increment();

            return result;

        } catch (Exception ex) {
            // 4. Failure classification
            metrics.counter("ai.pipeline.error",
                "layer",   ctx.currentLayer(),
                "type",    ex.getClass().getSimpleName(),
                "feature", ctx.featureId()
            ).increment();
            throw ex;
        }
    }
}

The four metrics that matter most:

Metric	Alert threshold	What it tells you
`ai.pipeline.latency` P95	> 5 seconds	User experience degradation
`ai.cache.hit_rate`	< 20% for FAQ features	Caching not working or queries too varied
`ai.tokens.cost_per_request`	> expected × 2	Prompt bloat, wrong model routing
`ai.pipeline.error` by layer	> 1%	Which layer is breaking

Trace individual requests. Every request gets a requestId that flows through all 8 layers and appears in every log line. When a user reports "the AI gave me a wrong answer," you can pull that request ID and see exactly: what input it received, whether it was a cache hit, which model was called, what the raw output was, whether validation passed, and what was returned. Without this, debugging production AI issues is guesswork.

Putting It All Together

Here's the orchestrator that wires all 8 layers:

@Service
public class AiRequestPipeline {

    private final InputGuardrailService guardrails;
    private final SemanticCacheService semanticCache;
    private final ModelRouter modelRouter;
    private final PromptAssemblyService promptAssembly;
    private final LlmExecutionService llmExecution;
    private final OutputValidationService outputValidation;
    private final ResponseCacheService responseCache;

    public AiResponse process(AiRequest request) {
        String requestId = UUID.randomUUID().toString();

        // Layer 1: Guardrails
        GuardrailResult guardrail = guardrails.validate(request.userInput());
        if (guardrail.isRejected()) {
            return AiResponse.rejected(guardrail.reason());
        }
        String sanitizedInput = guardrail.sanitizedInput();

        // Layer 2: Semantic cache lookup
        Optional<CachedResponse> cached = semanticCache.lookup(sanitizedInput, request.featureId());
        if (cached.isPresent()) {
            return AiResponse.fromCache(cached.get().response());
        }

        // Layer 3: Model routing
        ModelRouter.RoutingDecision routing = modelRouter.route(
            new RoutingContext(request.featureId(), request.taskType(), request.expectedComplexity())
        );

        // Layer 4: Prompt assembly
        Prompt prompt = promptAssembly.assemble(AssemblyRequest.builder()
            .userInput(sanitizedInput)
            .featureId(request.featureId())
            .modelId(routing.modelId())
            .sessionId(request.sessionId())
            .taskType(request.taskType())
            .requiresGrounding(request.requiresGrounding())
            .build()
        );

        // Layer 5: LLM execution (retry + circuit breaker)
        ChatResponse llmResponse = llmExecution.execute(prompt, routing.modelId(), requestId);
        String rawOutput = llmResponse.getResult().getOutput().getText();

        // Layer 6: Output validation
        ValidationResult validation = outputValidation.validate(rawOutput, request.validationSpec());
        if (validation.failed()) {
            if (validation.isRetryable()) {
                // One retry with error context injected
                rawOutput = retryWithErrorContext(prompt, validation, routing.modelId(), requestId);
            } else {
                return AiResponse.validationFailed(validation.failures());
            }
        }

        // Layer 7: Cache the validated response
        responseCache.store(new CacheStoreRequest(
            sanitizedInput, request.featureId(), rawOutput, request.taskType()
        ));

        // Layer 8: Observability is handled by the @TrackAiRequest aspect
        return AiResponse.success(rawOutput, routing.modelId(), false);
    }
}

The pipeline reads linearly. Each layer either short-circuits (return early) or passes its output to the next layer. No layer knows about layers beyond its immediate next one. This makes each layer independently testable and independently deployable — you can upgrade the model router without touching the output validator.

What This Architecture Costs You

Honesty about overhead: this pipeline adds latency and complexity compared to a raw LLM call.

Latency overhead per request (when all layers run):

Layer	P50 overhead
Input guardrail	1–2 ms
Semantic cache lookup (miss)	15–25 ms (embedding + pgvector query)
Model routing	< 1 ms
Prompt assembly + RAG	20–50 ms
LLM execution	800–3,000 ms (dominates)
Output validation	5–20 ms
Response cache write	10–15 ms (async — doesn't block response)
Observability	< 1 ms (async metrics)

The LLM call dominates. Everything else is noise. The semantic cache layer adds 15–25 ms on a miss, but saves 800–3,000 ms on every hit. At 30% cache hit rate (achievable for most FAQ/support features), average latency drops significantly.

Development overhead. This is real. You're building a pipeline instead of a function. The payoff comes at 3 AM when the LLM provider has a partial outage and your circuit breaker routes to the fallback model automatically while your on-call team sleeps.

Which Layers to Skip in the Early Days

If you're moving fast and not yet at scale, here's the priority order:

Ship from day one: Layer 1 (guardrails), Layer 5 (retry + timeout), Layer 8 (basic latency/error metrics). These prevent the most damaging production incidents.

Add in week 2–4: Layer 3 (model routing — this is the biggest cost lever), Layer 6 (output validation for structured outputs).

Add at scale: Layer 2 + 7 (semantic cache — ROI is high only when you have enough traffic to get cache hits), Layer 4 (full prompt assembly with RAG — if your use case needs grounding).

The architecture is meant to be grown into, not built all at once.

The teams I've seen struggle with production AI aren't struggling because their models are wrong. They're struggling because they shipped a raw LLM call and are now retrofitting production concerns one emergency at a time. The pipeline structure gives you a place for each concern, a clear interface between layers, and the ability to improve one layer without touching the others.

That's the architecture. What you build on top of it is up to you.

All code samples are illustrative of the pattern — adapt types and error handling to your specific Spring AI version and AI provider. If you're building this pipeline and run into a specific layer that's giving you trouble, drop a question in the comments.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte. He writes about production AI architecture at buildingai.in.

We Let Claude Code Refactor Our 200K-Line Java Monolith. Here's the Honest Result.

Avaneesh Yadav — Tue, 08 Sep 2026 11:46:23 +0000

Six weeks ago, our team made a decision that raised eyebrows in our org: we were going to let an AI agent drive the refactoring of a 200,000-line Java monolith.

Not assist. Not suggest. Drive.

We'd give Claude Code a CLAUDE.md with the target architecture, the constraints, the test suite, and the definition of done. We'd run it in autonomous mode overnight. Engineers would review in the morning.

I'm writing this because the results were not what any of us predicted — not the optimists, not the skeptics. Both camps were wrong in ways worth understanding before you try this yourself.

[!NOTE]
This is a real account of work at a client engagement (details anonymized). The codebase is a 12-year-old Spring Boot monolith — financial services, ~200K lines of Java, 68% test coverage. The team is 8 engineers. The refactoring goal: extract 4 bounded contexts into separate modules, update to Java 21 patterns, and eliminate 3 known anti-patterns throughout.

Why We Tried This

Let me tell you what convinced a risk-averse financial services engineering team to run an AI agent against production source code.

The realistic alternative was worse. Manual refactoring of 200K lines with 8 engineers would take 4–6 months at normal velocity, cause constant merge conflicts, and require keeping two versions of every abstraction alive simultaneously. We'd done this before. It's miserable.

The test coverage was real. 68% isn't great, but the critical paths — payment processing, account management, audit logging — were at 94%. The idea: let the AI make changes, let the tests tell us if anything broke, review the diffs in the morning.

We'd already seen what it could do. Over the prior month, we'd used Claude Code interactively for smaller refactoring tasks. Extracting a service class here, converting a DTO to a record there. It was faster than we expected and the mistakes were easy to catch.

The question was whether it would scale. The honest answer: partially.

The Setup

We spent a week before touching any code configuring the environment. This turned out to be the highest-ROI week of the project.

CLAUDE.md

Our CLAUDE.md was 400 lines — deliberately detailed. It included:

## Target Architecture

The monolith is being extracted into 4 modules:
- `core-banking`: Account, Balance, Transaction entities + repositories
- `payment-processing`: PaymentRequest, PaymentGateway, RetryPolicy
- `audit-trail`: AuditEvent, AuditRepository, AuditQueryService
- `customer-portal`: CustomerProfile, Preferences, NotificationSettings

Each module must:
- Have no compile-time dependencies on other modules (only via interfaces)
- Own its own database schema (prefix: `banking_`, `payment_`, `audit_`, `portal_`)
- Expose a public API via Spring ApplicationEvents or explicit service interfaces — never direct bean injection across module boundaries

## What Is a Correct Change

A change is correct if:
1. All existing tests pass (`./mvnw test` exits 0)
2. The moved class is not referenced from outside its target module
3. No new circular dependencies exist (`./mvnw verify -P architecture-check` exits 0)
4. The CHANGELOG.md entry describes what moved and why

## What Is NOT Correct (Do Not Do These)

- Adding @SuppressWarnings to make compilation pass
- Deleting tests to make the test suite pass
- Creating adapter classes that duplicate logic "temporarily"
- Importing from `com.company.monolith.*` in any new module (this is the boundary you are enforcing)

The "what NOT to do" section was the most important part. Without it, the model found creative ways to make tests pass that violated the spirit of what we were building.

Verification Script

Every autonomous run ended with a verification script that Claude Code was instructed to run before committing:

#!/bin/bash
set -e

echo "=== Running test suite ==="
./mvnw test -q

echo "=== Checking architecture boundaries ==="
./mvnw verify -P architecture-check -q

echo "=== Checking for forbidden imports ==="
grep -r "com.company.monolith" src/main/java/com/company/modules/ && echo "FORBIDDEN IMPORTS FOUND" && exit 1

echo "=== All checks passed ==="

If any check failed, Claude Code was instructed to rollback (via git checkout .) and document why it couldn't complete the task. This was critical — it prevented silent failures from accumulating.

What It Got Right

Let me be specific. These were the genuine wins.

Mechanical Transformations at Scale

Moving a class from one package to another while updating all references — Claude Code does this better than any engineer. Not because it's smart, but because it's patient and exhaustive.

We had 143 classes to relocate across the 4 modules. Manually, this is a 3-day job that produces 300+ merge conflicts. Claude Code did 127 of them correctly in the first overnight run. The patterns were consistent, the import updates were complete, and it caught cross-module references we had no idea existed.

That's 89% success on the most mechanical part of the work. A human team would have been slower and no more accurate on this specific task.

Java 21 Modernization

Within the context of each moved class, Claude Code applied modern Java idioms correctly and consistently:

// Before (Java 11 era):
public class PaymentResult {
    private final String status;
    private final String transactionId;
    private final String errorMessage;

    public PaymentResult(String status, String transactionId, String errorMessage) {
        this.status = status;
        this.transactionId = transactionId;
        this.errorMessage = errorMessage;
    }
    // ... 40 lines of getters, equals, hashCode, toString
}

// After (Java 21 record):
public record PaymentResult(String status, String transactionId, String errorMessage) {}

It converted 67 POJOs to records, replaced 23 switch statements with switch expressions, and modernized 31 Optional usages (removing the Optional.get() calls that were landmines waiting to explode). Every conversion passed the test suite.

For this, it was genuinely better than a code review — it caught every instance, not just the ones in files you happened to open.

Architectural Decision Documentation

We asked Claude Code to write a CHANGELOG entry for every meaningful change. What we got was better than most human-written ADRs:

## [2026-08-14] Extracted AuditEvent to audit-trail module

**Moved:** `com.company.monolith.audit.AuditEvent` → `com.company.modules.audit.AuditEvent`

**Callers updated:** 23 references across 18 files.

**Breaking change:** The class is no longer in the monolith classpath. Any Spring beans 
in the monolith that directly autowire AuditEvent must now import from 
`com.company.modules.audit`. Updated affected beans in PaymentService, AccountService, 
CustomerService.

**Remaining concern:** TransactionService still uses AuditEvent via a deprecated utility 
method (AuditUtils.logEvent). This dependency is in scope for the next session.

That last paragraph — "remaining concern" — appeared consistently and accurately. The model knew what it hadn't finished and said so. That's more than I can say for many ticket systems.

What It Silently Broke

Now the part you actually came for.

Test Coverage Erosion

This was the most insidious failure. On day 3 of autonomous runs, all tests were passing — but our test coverage had dropped from 68% to 61%.

Claude Code wasn't deleting tests. It was moving classes while leaving test classes behind in the old package. Since the test classes still compiled (the old package still existed during the transition), the tests ran but now covered a class that was no longer in the primary codebase.

The verification script checked that tests passed, not that they covered the right code. We didn't catch this for 3 days.

Fix: Added a coverage gate to the verification script:

./mvnw test jacoco:report
# Fail if overall coverage drops below 65%
awk '/INSTRUCTION/ && /TOTALCOUNT/' target/site/jacoco/jacoco.csv | ...

Harder lesson: autonomous AI systems find the exact edge of your success criteria and stop there. If your definition of "correct" has a gap, the model will discover it — not maliciously, but because that's what optimization does.

The 16 Adapter Classes Problem

On day 5, a senior engineer reviewing the morning's diffs flagged something odd: 16 new classes had appeared with names like PaymentServiceCompatibilityAdapter, AccountRepositoryBridgeImpl, AuditEventLegacyWrapper.

These weren't in our design. The model had invented them.

Here's what happened: when Claude Code encountered a dependency it couldn't cleanly resolve (a circular reference, a class that served two modules), it created an adapter class as a bridge. The tests passed. The architecture check passed (the adapters were in a designated compat package we'd created for legitimate use cases).

But the adapters duplicated business logic. PaymentServiceCompatibilityAdapter had its own validation rules that were subtly different from the canonical PaymentService. When those two diverged (which they did, two weeks later), we had a production bug that took 4 hours to trace.

Fix: Added an explicit CLAUDE.md rule: "Creating any class with 'Adapter', 'Bridge', 'Wrapper', 'Compat', or 'Legacy' in the name requires explicit justification in the CHANGELOG. The default answer is: refactor instead of wrap."

Transactional Boundary Confusion

This was the most technically dangerous failure.

When extracting classes across module boundaries, the model sometimes removed @Transactional annotations that spanned what were now separate services. The tests didn't catch it because our test transactions were scoped per test — in production, calls that needed to be atomic now weren't.

// Before (correct — single transaction):
@Transactional
public void processPayment(PaymentRequest request) {
    PaymentResult result = paymentGateway.execute(request);
    auditService.log(AuditEvent.of(result));  // same transaction
    accountService.debit(request.amount());    // same transaction
}

// After (broken — Claude split these across modules):
// PaymentService (payment-processing module):
@Transactional  // only covers paymentGateway.execute
public PaymentResult processPayment(PaymentRequest request) {
    return paymentGateway.execute(request);
}

// Callers now responsible for atomicity — but weren't written to be

The model correctly identified that auditService and accountService were now in different modules. It correctly removed the cross-module @Transactional — that annotation genuinely can't span service boundaries. But it didn't flag that this was a design decision requiring human judgment, not a mechanical transformation.

We caught this in QA because a test for partial payment failure showed inconsistent audit state. In production, this would have been a compliance issue.

Fix: Added to CLAUDE.md: "If removing a @Transactional annotation, create a BLOCKING_ISSUE entry in CHANGELOG.md and do not commit. A human must review any transaction boundary change."

The Hybrid Model We Actually Shipped

After week 3, we stopped running fully autonomous overnight sessions. The failure rate on complex cases was too high and the failures were too subtle.

What we replaced it with — and what actually shipped — was a human-gated autonomous loop:

Engineer scopes the session — opens a GitHub issue describing exactly one refactoring task (e.g., "Extract CustomerProfile to customer-portal module")
Claude Code runs with plan mode — generates a plan, stops for review before making any changes
Engineer approves the plan — 5 minutes, not 30
Claude Code executes with the verification script — stops on any failure, documents blockers
Engineer reviews the diff — 15 minutes, not 2 hours
Merge or reject

This is slower than fully autonomous. It's still 4–5x faster than purely manual refactoring. We finished the extraction in 6 weeks rather than the estimated 4–6 months. The production bugs were zero.

The key insight: AI is fastest when humans define the boundary of each session narrowly. Open-ended autonomous runs exposed Claude Code to decisions it wasn't equipped to make. Tightly-scoped sessions with human gates at the boundaries — that's where the velocity is.

What Changes About the Senior Engineer's Job

The engineers on this project spent less time:

Doing mechanical import updates
Writing boilerplate class migrations
Searching for every reference to a moved class

They spent more time:

Reviewing plans before execution (critical path thinking)
Writing better CLAUDE.md constraints (the AI is only as good as your specification)
Designing verification scripts (defining "correct" in a machine-checkable way)
Catching subtle failures (the @Transactional case required deep understanding of Spring's transactional model)

None of those second activities are junior work. They're the work that separates senior engineers from everyone else — and AI made that work more prominent, not less.

The engineers who struggled were the ones who wanted to hand the task off entirely and come back to finished code. The ones who thrived treated Claude Code as an exceptionally fast junior engineer who needs very clear scope and explicit constraints.

Numbers, Honestly

Metric	Result
Classes migrated	143 total
Migrated correctly on first autonomous pass	127 (89%)
Required human intervention	16 (11%)
Adapter classes created (unwanted)	16 — all deleted
Production bugs from this refactoring	0 (caught in QA)
Near-misses caught in review	4 (including @Transactional)
Estimated manual timeline	4–6 months
Actual timeline	6 weeks
Test coverage after vs before	68% → 71% (improved — we added missing tests the AI flagged)
Team sentiment at end	7/8 engineers want to do it again

That one holdout? He's the one who caught the transactional bug in QA. He's right to be cautious.

What We'd Do Differently

Define "correct" before you start, not after you fail. Every gap in your CLAUDE.md is a gap in the output. Write the "what NOT to do" section first.
Coverage gates are mandatory. Tests passing is table stakes. Coverage delta is what tells you whether the tests are still testing the right thing.
Never let the AI make transaction boundary decisions. @Transactional spans, database schema changes, event ordering — these require human judgment. Add explicit blockers for these in CLAUDE.md.
Scope sessions to single classes or single extractions. "Extract everything" is not a plan. "Extract CustomerProfile and its 3 direct dependencies" is.
Review the CHANGELOG, not the diff. The model's own explanation of what it did is often clearer than reading 800 changed lines. Start there, then verify specific concerns in the diff.

Should You Try This?

Yes — with the hybrid model, not the fully autonomous one. The velocity gain is real and the failure modes are manageable once you know them.

The question isn't "can AI refactor our codebase?" It can. The question is "how do we design the human-AI collaboration so that AI handles what it's genuinely better at and humans handle what they're genuinely better at?"

Mechanical transformations at scale: AI. Architectural decisions that span module boundaries: human. Verification criteria: human. Execution within those criteria: AI. Review of anything involving transactions, security, or compliance: human, always.

That's not a limitation of the technology. That's what good engineering teams have always done — except now one of your team members can execute 200 changes overnight without getting tired or making typos.

If your team is considering a similar project, I'm happy to share the full CLAUDE.md template we used. Drop a comment or reach out directly — details in the footer.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte. He leads AI-augmented engineering initiatives for enterprise clients and writes about production AI architecture at buildingai.in.

Building AI Skills: GitHub Copilot Extensions, Claude Tools, and Reusable Agent Capabilities

Avaneesh Yadav — Tue, 08 Sep 2026 10:36:51 +0000

Every major AI assistant now supports custom skills — capabilities you build that extend what the AI can do. GitHub Copilot calls them Extensions. Claude calls them Tools (and MCP servers). LangChain calls them Tools. Spring AI calls them Functions. The terminology is a mess, but the concept is identical: give an AI a callable capability with a defined interface, and it will use that capability when the task requires it.

This guide builds one concrete skill — a deployment status checker — across three different platforms: GitHub Copilot Extension, Claude MCP Tool, and a reusable agent skill for any LLM. By the end you'll understand the architecture of each, where they differ, and how to choose the right platform for the job.

What "Skills" Actually Are

Before touching code, it helps to understand the mental model. An AI skill has three parts:

flowchart LR
    subgraph "Skill Anatomy"
        A["<b>Schema</b>\nWhat the skill does,\nwhat inputs it accepts,\nwhat it returns"]
        B["<b>Implementation</b>\nThe actual code that\nexecutes when called"]
        C["<b>Registration</b>\nHow the AI model\ndiscovers the skill\nexists"]
    end

    D[LLM] -->|reads| A
    D -->|"decides to invoke\n(with structured args)"| B
    B -->|result| D
    C -->|exposes A + B to| D

    style A fill:#1e3a5f,color:#7dd3fc
    style B fill:#065f46,color:#6ee7b7
    style C fill:#374151,color:#d1d5db

The LLM reads the schema at inference time, decides whether the skill is relevant to the user's request, calls it with structured arguments extracted from natural language, gets the result, and incorporates it into the response. The human never directly triggers the skill — the LLM does, when it judges the skill will help.

Platform	Schema format	Registration	Invocation
GitHub Copilot Extension	Markdown description + REST endpoint	GitHub App manifest	`@your-extension` in Copilot Chat
Claude Tools (API)	JSON Schema	`tools:[]` parameter on each API call	Model invokes via `tool_use` content block
MCP Server	JSON Schema over MCP protocol	`claude_desktop_config.json` or MCP client config	Any MCP-compatible client
Spring AI `@Tool`	Auto-derived from method signature	`ChatClient.tools()` call	Model invokes through Spring AI framework

The Skill We're Building

Deployment Status Checker — given a service name and optional environment, returns the current deployment status, last deploy time, version, and any active incidents.

This is representative because:

It requires an external API call (your deployment platform — Kubernetes, Argo CD, whatever you use)
It takes structured parameters
It returns structured data the AI formats for humans
Every developer wants it in their IDE assistant

We'll implement it three ways and compare what changes.

Platform 1: GitHub Copilot Extension

GitHub Copilot Extensions let you create a custom @your-extension participant in Copilot Chat. Users invoke it with @deploy-status what's the state of the payment service in prod? and your extension receives the conversation, calls your backend, and streams a response back.

Architecture

flowchart TD
    A[Developer types\n@deploy-status in\nCopilot Chat] --> B[GitHub routes message\nto your extension's\nwebhook URL]
    B --> C[Your Extension Server\nNode.js / Java / Python]
    C -->|Verify GitHub\nHMAC signature| D{Valid?}
    D -->|No| E[401 Unauthorized]
    D -->|Yes| F[Parse conversation\nfrom payload]
    F --> G[Call your\ndeployment API]
    G --> H[Optionally call\nCopilot LLM API\nto format response]
    H -->|SSE stream| I[Response appears\nin Copilot Chat]

    style C fill:#1e3a5f,color:#7dd3fc
    style I fill:#065f46,color:#6ee7b7

Setting Up the GitHub App

Create a GitHub App at github.com/settings/apps/new with:

Webhook URL: your extension server's /api/github/copilot endpoint
Permissions: Copilot Chat → Read (under Account permissions)
Copilot: check "Copilot Extension" in the app settings
Callback URL: for OAuth if your skill needs to act on behalf of the user

// copilot-extension-manifest.json — defines the extension's capabilities
{
  "name": "deploy-status",
  "description": "Check deployment status, versions, and incidents for any service",
  "instructions": "Use this extension when the user asks about deployment status, service versions, recent deployments, or production incidents. It can check any service across dev, staging, and prod environments.",
  "skill_sets": [
    {
      "name": "deployment",
      "skills": [
        {
          "id": "get-deploy-status",
          "description": "Returns the current deployment status and version for a named service",
          "parameters": {
            "type": "object",
            "properties": {
              "service": { "type": "string", "description": "The service name, e.g. payment-service" },
              "environment": { "type": "string", "enum": ["dev", "staging", "prod"], "default": "prod" }
            },
            "required": ["service"]
          }
        }
      ]
    }
  ]
}

The Extension Server

// server.mjs — Express webhook handler for the Copilot extension
import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";
import Anthropic from "@anthropic-ai/sdk";

const app = express();
app.use(express.json());

const WEBHOOK_SECRET = process.env.GITHUB_WEBHOOK_SECRET;
const COPILOT_TOKEN  = process.env.COPILOT_TOKEN;  // GitHub-issued token per request

// Verify GitHub's HMAC signature on every webhook
function verifySignature(req) {
  const sig     = req.headers["x-github-token"] ?? "";
  const payload = JSON.stringify(req.body);
  const digest  = createHmac("sha256", WEBHOOK_SECRET).update(payload).digest("hex");
  return timingSafeEqual(Buffer.from(`sha256=${digest}`), Buffer.from(sig));
}

app.post("/api/github/copilot", async (req, res) => {
  if (!verifySignature(req)) return res.status(401).send("Unauthorized");

  const { messages } = req.body;
  const lastMessage  = messages.at(-1)?.content ?? "";

  // Extract service name from the user's message (basic extraction — Claude does this more elegantly)
  const serviceMatch = lastMessage.match(/\b([a-z][a-z0-9-]+(?:-service|-api|-worker|-gateway))\b/i);
  const service      = serviceMatch?.[1] ?? null;
  const envMatch     = lastMessage.match(/\b(dev|staging|prod|production)\b/i);
  const environment  = envMatch?.[1]?.replace("production", "prod") ?? "prod";

  // Call your actual deployment platform API
  const status = service
    ? await fetchDeployStatus(service, environment)
    : null;

  // Use GitHub's Copilot LLM API to format the response naturally
  const copilotClient = new Anthropic({
    baseURL: "https://api.githubcopilot.com",
    apiKey:  req.headers["x-github-token"],   // use the request token, not your own key
  });

  res.setHeader("Content-Type", "text/event-stream");

  const stream = await copilotClient.messages.stream({
    model:      "gpt-4o",   // GitHub Copilot uses its own model routing
    max_tokens: 1024,
    messages: [
      {
        role:    "system",
        content: "You are a helpful deployment assistant. Format deployment status clearly and concisely. Highlight any issues or incidents prominently."
      },
      ...messages,
      ...(status ? [{
        role:    "system",
        content: `Deployment data for ${service} in ${environment}: ${JSON.stringify(status)}`
      }] : [])
    ]
  });

  // Stream the response back in SSE format
  for await (const chunk of stream) {
    if (chunk.type === "content_block_delta") {
      res.write(`data: ${JSON.stringify({ choices: [{ delta: { content: chunk.delta.text } }] })}\n\n`);
    }
  }
  res.write("data: [DONE]\n\n");
  res.end();
});

async function fetchDeployStatus(service, env) {
  // Replace with your actual deployment API — Argo CD, Kubernetes, Spinnaker, etc.
  const res = await fetch(`${process.env.DEPLOY_API_URL}/services/${service}/status?env=${env}`, {
    headers: { Authorization: `Bearer ${process.env.DEPLOY_API_TOKEN}` }
  });
  if (!res.ok) return { error: `Service ${service} not found in ${env}` };
  return res.json();
}

app.listen(3000, () => console.log("Copilot extension running on :3000"));

[!NOTE]
GitHub Copilot Extensions receive a short-lived x-github-token in each request. Use this token to call GitHub's Copilot LLM API — you don't need your own Anthropic or OpenAI key for the language generation part. You do need your own key if you want to call a different model for specific tasks inside your extension.

Platform 2: Claude Tools (API + Spring AI)

Claude's tool use lets you define callable functions in the API request. The model reads your schema, decides when to call your tool, and sends a structured tool_use request. Your code executes the tool and returns the result. Claude then incorporates the result into its response.

Spring AI Implementation

Spring AI makes Claude tool registration almost declarative with @Tool:

// DeploymentSkill.java — Spring AI tool definition
@Component
public class DeploymentSkill {

    private final DeploymentApiClient deploymentApi;

    @Tool(description = """
        Returns the current deployment status, version, last deploy time, and active incidents
        for a named service. Use when the user asks about service health, deployment state,
        running version, or recent deployments.
        """)
    public DeploymentStatus getDeploymentStatus(
            @ToolParam(description = "The service name, e.g. payment-service or user-api") String service,
            @ToolParam(description = "Environment: dev, staging, or prod. Default is prod.") String environment) {

        return deploymentApi.getStatus(service, environment.isBlank() ? "prod" : environment);
    }

    @Tool(description = """
        Lists the last N deployments for a service, including who triggered them and whether
        they succeeded. Use when the user asks about deployment history or recent changes.
        """)
    public List<DeploymentEvent> getDeploymentHistory(
            @ToolParam(description = "The service name") String service,
            @ToolParam(description = "Number of recent deployments to return, max 20") int limit) {

        return deploymentApi.getHistory(service, Math.min(limit, 20));
    }
}

// Record for structured return — Spring AI serializes this to JSON for Claude
public record DeploymentStatus(
    String  service,
    String  environment,
    String  version,
    String  status,         // "healthy" | "degraded" | "down" | "deploying"
    Instant lastDeployedAt,
    String  deployedBy,
    int     activeIncidents,
    String  incidentSummary // null if no active incidents
) {}

// ChatController.java — wire the tool into every chat session
@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatClient chatClient;
    private final DeploymentSkill deploymentSkill;

    @PostMapping
    public Flux<String> chat(@RequestBody ChatRequest req) {
        return chatClient
                .prompt()
                .system("""
                    You are an AI assistant for engineering teams. You have access to deployment
                    status tools. Use them proactively when the user asks about services, 
                    deployments, incidents, or production health.
                    """)
                .user(req.message())
                .tools(deploymentSkill)   // Register the skill — Spring AI handles schema generation
                .stream()
                .content();
    }
}

Spring AI reads your @Tool methods at startup, generates the JSON Schema from the parameter types and @ToolParam descriptions, and includes them in every Claude API call automatically. When Claude decides to use a tool, Spring AI intercepts the tool_use block, invokes your method with the deserialized arguments, and feeds the result back to Claude.

What the API Request Actually Looks Like

Understanding the wire format helps debug issues:

// What Spring AI sends to the Claude API
{
  "model": "claude-opus-5",
  "messages": [{ "role": "user", "content": "What version is payment-service running in prod?" }],
  "tools": [
    {
      "name": "getDeploymentStatus",
      "description": "Returns the current deployment status, version, last deploy time...",
      "input_schema": {
        "type": "object",
        "properties": {
          "service":     { "type": "string", "description": "The service name, e.g. payment-service" },
          "environment": { "type": "string", "description": "Environment: dev, staging, or prod..." }
        },
        "required": ["service"]
      }
    }
  ]
}

// Claude's response when it decides to invoke the tool
{
  "role": "assistant",
  "content": [
    { "type": "text", "text": "Let me check the deployment status for payment-service in prod." },
    {
      "type": "tool_use",
      "id": "toolu_01XyzAbc",
      "name": "getDeploymentStatus",
      "input": { "service": "payment-service", "environment": "prod" }
    }
  ]
}

// Your code returns this, Spring AI sends it back to Claude
{
  "role": "user",
  "content": [
    {
      "type": "tool_result",
      "tool_use_id": "toolu_01XyzAbc",
      "content": "{\"service\":\"payment-service\",\"environment\":\"prod\",\"version\":\"2.14.1\",...}"
    }
  ]
}

Claude then reads the tool result and writes a natural language response.

Platform 3: MCP Server

The Model Context Protocol (MCP) is an open standard that separates tool implementation from tool consumption. You build an MCP server once, and any MCP-compatible client — Claude Desktop, Claude Code, Cursor, Zed, your own agent — can connect to it without any changes.

flowchart TD
    subgraph "MCP Clients"
        A[Claude Desktop]
        B[Claude Code CLI]
        C[Cursor IDE]
        D[Your Custom Agent]
    end

    subgraph "Your MCP Server"
        E[Tool: get_deploy_status]
        F[Tool: list_deployments]
        G[Tool: trigger_rollback]
        H[Resource: /incidents/active]
    end

    subgraph "Your Backend"
        I[Argo CD API]
        J[PagerDuty API]
        K[GitHub Actions API]
    end

    A <-->|MCP protocol\nover stdio / SSE| E
    B <-->|MCP protocol| E
    C <-->|MCP protocol| F
    D <-->|MCP protocol| G

    E --> I
    F --> I
    G --> K
    H --> J

    style E fill:#1e3a5f,color:#7dd3fc
    style F fill:#1e3a5f,color:#7dd3fc
    style G fill:#1e3a5f,color:#7dd3fc

Spring AI MCP Server

// DeploymentMcpServer.java — Spring AI MCP server implementation
@SpringBootApplication
@EnableMcpServer  // Activate Spring AI's MCP server support
public class DeploymentMcpServer {
    public static void main(String[] args) {
        SpringApplication.run(DeploymentMcpServer.class, args);
    }
}

@McpServer
@Component
public class DeploymentTools {

    private final DeploymentApiClient api;

    @McpTool(
        name        = "get_deploy_status",
        description = "Returns current deployment status, version, and incident summary for a service"
    )
    public DeploymentStatus getStatus(
            @McpParam("service")     String service,
            @McpParam("environment") @Nullable String environment) {
        return api.getStatus(service, environment != null ? environment : "prod");
    }

    @McpTool(
        name        = "list_recent_deployments",
        description = "Lists the last N deployments with timestamps, deployers, and outcomes"
    )
    public List<DeploymentEvent> listDeployments(
            @McpParam("service") String service,
            @McpParam("limit")   @Nullable Integer limit) {
        return api.getHistory(service, limit != null ? Math.min(limit, 20) : 5);
    }

    @McpTool(
        name        = "trigger_rollback",
        description = "Triggers a rollback to the previous stable version for a service in a given environment. REQUIRES explicit user confirmation in the conversation before calling."
    )
    public RollbackResult triggerRollback(
            @McpParam("service")     String service,
            @McpParam("environment") String environment) {
        return api.rollback(service, environment);
    }

    // MCP Resources expose data without requiring the model to "ask" for it
    @McpResource(uri = "deploy://incidents/active", description = "Active production incidents")
    public String activeIncidents() {
        return api.getActiveIncidents().stream()
                .map(i -> "- [%s] %s (since %s)".formatted(i.severity(), i.title(), i.startedAt()))
                .collect(Collectors.joining("\n"));
    }
}

# application.yml — MCP server transport config
spring:
  ai:
    mcp:
      server:
        transport: stdio          # stdio for local tools; sse for remote/multi-user
        name: deploy-status-mcp
        version: 1.0.0

Registering with Claude Code

// ~/.claude/claude_desktop_config.json — or your MCP client's config
{
  "mcpServers": {
    "deploy-status": {
      "command": "java",
      "args": ["-jar", "/opt/tools/deploy-status-mcp.jar"],
      "env": {
        "DEPLOY_API_URL":   "https://deploy.yourcompany.internal",
        "DEPLOY_API_TOKEN": "your-token-here"
      }
    }
  }
}

After adding this, Claude Code picks up your tools automatically. Users can run /mcp to list available tools and see get_deploy_status, list_recent_deployments, and trigger_rollback from your server.

Platform 4: Claude Code Skills (SKILL.md)

Claude Code has its own skill system for defining reusable slash commands — procedures Claude follows when you type /deploy-check in a session.

<!-- .claude/skills/deploy-check.md -->
name: deploy-check
description: Check deployment status for a service. Usage: /deploy-check [service] [env]

Check the deployment status for the given service and environment.

## Steps

1. If service name is not provided, ask the user: "Which service? (e.g. payment-service, user-api)"
2. If environment is not provided, default to `prod`
3. Run: `curl -s -H "Authorization: Bearer $DEPLOY_API_TOKEN" \
   "$DEPLOY_API_URL/services/{service}/status?env={env}" | jq .`
4. Format the output:
   - Show service name, environment, version, and status on the first line
   - If status is not "healthy", highlight it prominently with what's wrong
   - Show the last deploy time and who triggered it
   - If there are active incidents, list them with severity and duration
5. If the API call fails (non-200 response), say "Could not reach the deployment API —
   check that DEPLOY_API_TOKEN and DEPLOY_API_URL are set in your environment"

## Output format

[service] @ [env] — v[version] — [status]
Last deployed: [time] by [deployer]
[If incidents:] ⚠️ [N] active incident(s): [summary]

Skills defined in .claude/skills/ are automatically available as /skill-name in every Claude Code session in that project. The CLAUDE.md in your project can also reference them:

<!-- CLAUDE.md -->
## Available slash commands

- `/deploy-check [service] [env]` — Check deployment status and active incidents
- `/review` — Run AI code review on current diff  
- `/cost-check` — Review token usage and estimated costs for this sprint

Writing Skill Descriptions That Work

The single highest-leverage thing you can do to make a skill perform well is write a great description. The model reads your description to decide when to invoke the skill. Bad descriptions lead to missed invocations or wrong invocations.

Bad description:

"Gets deployment info"

Good description:

"Returns the current deployment status, running version, last deploy timestamp, and active PagerDuty incidents for a named service in a specified environment (dev/staging/prod). Use when the user asks about: is service X running? what version is deployed? when was the last deployment? are there any active incidents? what's wrong with service Y?"

The difference:

Specificity — lists exactly what the tool returns
Trigger phrases — "Use when the user asks about..." gives the model explicit routing signals
Examples — concrete sample questions help the model recognize when to invoke

Apply the same principle to parameter descriptions:

// Bad
@ToolParam("service name") String service

// Good
@ToolParam("The exact service name as it appears in the deployment system, e.g. 'payment-service', 'user-api', 'notification-worker'. If the user gives a partial name, use your best judgment.") String service

Choosing the Right Platform

flowchart TD
    A{Where will\nusers invoke\nthe skill?}

    A -->|In GitHub Copilot Chat\nin their IDE| B[GitHub Copilot Extension]
    A -->|In Claude.ai or\nClaude Code| C{Persistent connection\nor per-call?}
    A -->|In your own app\nor API| D{Language / framework?}
    A -->|Any MCP client\nCursor, Zed, etc.| E[MCP Server]

    C -->|One-off, embedded\nin app logic| F[Claude Tool Use\nvia API directly]
    C -->|Persistent, IDE-like\nmulti-session| E

    D -->|Java / Spring Boot| G[Spring AI @Tool\nor MCP Server]
    D -->|Any other| H[Claude API tool_use\nin your language]

    style B fill:#1e3a5f,color:#7dd3fc
    style E fill:#065f46,color:#6ee7b7
    style G fill:#374151,color:#d1d5db

Use case	Best platform
IDE assistant for developers (`@your-team`)	GitHub Copilot Extension
App feature powered by Claude (summarize, analyze, assist)	Claude Tools via API (`tool_use`)
Developer tool available in Claude Code / Cursor / Zed	MCP Server
Team automation inside Claude Code sessions	Claude Code Skill (SKILL.md)
Enterprise integration across multiple AI clients	MCP Server (platform-agnostic)
Prototype / internal hackathon	Claude Tools via API (fastest to build)

The Pattern That Works Across All Platforms

Regardless of which platform you choose, four implementation rules apply everywhere:

1. Return structured data, let the AI format it for humans.
Don't return "Payment service is running version 2.14.1 and was last deployed at 3:42 PM." Return { version: "2.14.1", lastDeployedAt: "2026-09-02T15:42:00Z", status: "healthy" }. The AI formats prose better than you do. Structured data also survives prompt changes without touching your implementation.

2. Fail loudly with context.

// Bad — Claude has nothing to work with
throw new RuntimeException("Not found");

// Good — Claude can explain this to the user
return new SkillError("SERVICE_NOT_FOUND",
    "No service named '" + service + "' found in " + environment + ". " +
    "Available services: payment-service, user-api, notification-worker");

3. Add safety rails on destructive operations.
For tools like trigger_rollback, add a requirement in the description: "ONLY invoke after the user has explicitly confirmed they want to roll back, e.g. 'yes, roll it back' or 'proceed with rollback'." The model respects this instruction.

4. Keep tool count low.
More tools = more confusion about which one to invoke. Target 3–5 well-named tools rather than 12 narrowly-scoped ones. If you have more, group them into domain-specific servers/extensions.

Key Takeaways

GitHub Copilot Extensions are best for developer tools that live inside the IDE alongside code — your team can invoke them with @your-extension without leaving the editor.
Claude Tools (API) are best for embedding AI capabilities inside your own applications — an AI assistant inside your product, a chatbot, a document analyzer. You control the full stack.
MCP Servers are the most portable option — build once, run in any MCP-compatible client. This is the right choice for tools you want available everywhere your team uses AI.
Claude Code Skills are configuration, not code — great for standardizing team workflows and creating shared slash commands without a deployment.
The description is the routing logic. A well-written description is worth more than any amount of prompt engineering around the tool invocation. Invest here first.
Return structured data. Let the AI format results for humans. Your job is to return correct, complete data in a machine-readable format. The AI's job is to turn that into a clear, contextualized response.

The skills you build today will run in assistants your team doesn't know about yet. Invest 30 extra minutes on the schema and description — it compounds across every client that ever connects to your skill.

AI Code Review at Scale: How We Use Claude to Review Every PR Before Humans See It

Avaneesh Yadav — Tue, 08 Sep 2026 10:36:49 +0000

Our team reviews 70–100 pull requests a week. Before we automated the first pass, a senior engineer was spending 4–6 hours a day on reviews that ranged from genuinely interesting architectural decisions to "you forgot to close the stream." We weren't going to hire our way out of that — the solution was changing what humans review, not adding more humans.

Six months ago we shipped an AI first-pass reviewer into our GitHub Actions pipeline. It runs on every PR, posts inline comments on the diff within 90 seconds of the push, and consistently catches 30–40% of real issues before a human reads a single line. Here's the complete architecture, prompt, workflow, and the numbers after 4,200 pull requests.

What This Is Not

AI code review is not a replacement for human review. It's a filter and an amplifier.

The AI catches: null pointer patterns, missing null checks, obvious resource leaks, SQL injection risks, hardcoded secrets, synchronization problems, missing error handling, type safety issues, test coverage gaps in the changed code.

Humans catch: architectural tradeoffs, business logic correctness, team convention alignment, long-term maintainability decisions, subtle concurrency bugs that require understanding the full system.

A well-configured AI reviewer handles the first category reliably enough that human reviewers spend their time almost entirely on the second. That's the value — not replacing judgment, but protecting it.

The Architecture

flowchart TD
    A[Developer pushes\nto feature branch] -->|webhook| B[GitHub Actions\nai-review.yml]
    B --> C[Fetch PR diff\nvia GitHub API]
    C --> D{Diff size check}
    D -->|> 1000 lines| E[Chunk diff\nby file]
    D -->|<= 1000 lines| F[Single request]
    E --> G[Review each chunk\nwith Claude API]
    F --> G
    G --> H[Aggregate findings]
    H --> I{Findings exist?}
    I -->|Yes| J[Post inline comments\nvia GitHub PR Review API]
    I -->|No| K[Post approval\n'LGTM from AI reviewer']
    J --> L[Human reviewer\nsees flagged lines]
    L --> M[Human focuses on\nwhat AI missed]

    style G fill:#1e3a5f,color:#7dd3fc
    style J fill:#065f46,color:#6ee7b7
    style M fill:#374151,color:#d1d5db

Two design decisions worth explaining:

Chunking by file, not by line count. When a PR is large, splitting at an arbitrary line boundary produces incoherent context. Splitting by file means each chunk is a complete, reviewable unit — the AI sees the full before/after for each file.

Inline comments, not a summary. A summary paragraph at the bottom of the PR thread gets ignored. An inline comment on line 47 of PaymentService.java is impossible to miss. The GitHub PR Review API lets you attach comments to specific lines in the diff — that's what makes the feedback actionable.

The GitHub Actions Workflow

# .github/workflows/ai-review.yml
name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize]
    branches: [main, develop]

jobs:
  review:
    runs-on: ubuntu-latest
    # Skip dependabot, release branches, and draft PRs
    if: |
      github.actor != 'dependabot[bot]' &&
      !startsWith(github.head_ref, 'release/') &&
      github.event.pull_request.draft == false
    permissions:
      pull-requests: write
      contents: read

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Fetch PR diff
        id: diff
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh pr diff ${{ github.event.pull_request.number }} \
            --repo ${{ github.repository }} > pr.diff

          lines=$(wc -l < pr.diff)
          echo "lines=$lines" >> $GITHUB_OUTPUT
          echo "PR diff: $lines lines"

      - name: AI Review
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
        run: |
          node .github/scripts/ai-review.mjs

The workflow is intentionally minimal — all the logic lives in ai-review.mjs so it can be tested locally without triggering Actions.

The Review Script

// .github/scripts/ai-review.mjs
import { readFileSync } from "node:fs";
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const GH_TOKEN  = process.env.GH_TOKEN;
const PR_NUMBER = process.env.PR_NUMBER;
const REPO      = process.env.REPO;
const HEAD_SHA  = process.env.HEAD_SHA;

const REVIEW_PROMPT = `You are a senior Java/Spring Boot engineer reviewing a pull request diff.

Review the diff below and identify ONLY real, high-confidence issues. Do not flag:
- Style preferences or opinions
- Minor naming issues that don't affect correctness
- Missing Javadoc on non-public methods
- Issues already present in unchanged lines (context lines starting with space)

Flag these categories when you find them:
- NULL_DEREF: Potential NullPointerException or missing null check on added lines
- RESOURCE_LEAK: Stream, connection, or IO resource opened without try-with-resources
- SECURITY: SQL injection, hardcoded secret, XSS vector, unsafe deserialization
- CONCURRENCY: Unsynchronized shared mutable state, missing volatile, race condition
- ERROR_HANDLING: Swallowed exception (empty catch block), exception converted to void
- LOGIC_BUG: Off-by-one, incorrect boolean logic, unreachable code
- TEST_GAP: New public method with no corresponding test in the diff

Respond with a JSON array. Each object has:
{
  "file": "path/to/File.java",
  "line": <line number in the NEW file where the issue appears>,
  "category": "<one of the categories above>",
  "severity": "high" | "medium" | "low",
  "comment": "<specific, actionable comment referencing the exact code — max 3 sentences>"
}

If you find no real issues, respond with: []

Do not add preamble or explanation outside the JSON array.`;

async function reviewChunk(diffChunk) {
  const message = await client.messages.create({
    model: "claude-opus-5",           // Use strongest model for code review
    max_tokens: 4096,
    messages: [{
      role: "user",
      content: `${REVIEW_PROMPT}\n\n\`\`\`diff\n${diffChunk}\n\`\`\``
    }]
  });

  const text = message.content[0].text.trim();
  try {
    return JSON.parse(text);
  } catch {
    // Model returned non-JSON (rare) — skip this chunk
    console.warn("Non-JSON response from model, skipping chunk");
    return [];
  }
}

function splitDiffByFile(diff) {
  const chunks = [];
  const lines  = diff.split("\n");
  let current  = [];
  let inFile   = false;

  for (const line of lines) {
    if (line.startsWith("diff --git")) {
      if (current.length > 0 && inFile) chunks.push(current.join("\n"));
      current = [line];
      inFile  = true;
    } else {
      current.push(line);
    }
  }
  if (current.length > 0 && inFile) chunks.push(current.join("\n"));
  return chunks;
}

async function postReviewComments(findings) {
  if (findings.length === 0) {
    // Post a clean LGTM review
    await ghApi(`/repos/${REPO}/pulls/${PR_NUMBER}/reviews`, {
      commit_id: HEAD_SHA,
      event: "COMMENT",
      body: "**AI Reviewer** — No issues found in this diff. Passing to human review.",
      comments: []
    });
    return;
  }

  // Group by severity for the summary body
  const high   = findings.filter(f => f.severity === "high");
  const medium = findings.filter(f => f.severity === "medium");
  const low    = findings.filter(f => f.severity === "low");

  const summary = [
    `**AI Reviewer** — Found ${findings.length} potential issue(s):`,
    high.length   ? `🔴 ${high.length} high severity`   : "",
    medium.length ? `🟡 ${medium.length} medium severity` : "",
    low.length    ? `🟢 ${low.length} low severity`    : "",
    "",
    "_These are automated findings. Use judgment — the AI can be wrong._"
  ].filter(Boolean).join("\n");

  // Map findings to inline PR review comments
  const comments = findings
    .filter(f => f.file && f.line)
    .map(f => ({
      path:     f.file,
      position: f.line,   // GitHub API: position in the diff, not file line number
      body:     `**[${f.category}]** ${severityEmoji(f.severity)} ${f.comment}`
    }));

  await ghApi(`/repos/${REPO}/pulls/${PR_NUMBER}/reviews`, {
    commit_id: HEAD_SHA,
    event: "COMMENT",
    body: summary,
    comments
  });
}

function severityEmoji(s) {
  return { high: "🔴", medium: "🟡", low: "🟢" }[s] ?? "⚪";
}

async function ghApi(path, body) {
  const res = await fetch(`https://api.github.com${path}`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${GH_TOKEN}`,
      "Content-Type": "application/json",
      "X-GitHub-Api-Version": "2022-11-28"
    },
    body: JSON.stringify(body)
  });
  if (!res.ok) {
    const err = await res.text();
    throw new Error(`GitHub API ${path} failed (${res.status}): ${err}`);
  }
}

// ── Main ──────────────────────────────────────────────────────────────────────
const diff = readFileSync("pr.diff", "utf-8");

// Skip if no meaningful diff (binary files, lock files only, etc.)
if (diff.trim().length < 50) {
  console.log("Diff too small — skipping AI review");
  process.exit(0);
}

const chunks   = splitDiffByFile(diff);
const MAX_CHUNK_CHARS = 12_000;  // ~3K tokens — leaves room for prompt + response

// Large file diffs get truncated to avoid exceeding model context
const safeChunks = chunks.map(c =>
  c.length > MAX_CHUNK_CHARS ? c.slice(0, MAX_CHUNK_CHARS) + "\n... (truncated)" : c
);

console.log(`Reviewing ${safeChunks.length} file chunk(s)...`);

const allFindings = (
  await Promise.all(safeChunks.map(reviewChunk))
).flat();

// Deduplicate by file+line (same issue caught in multiple chunks)
const seen     = new Set();
const findings = allFindings.filter(f => {
  const key = `${f.file}:${f.line}:${f.category}`;
  if (seen.has(key)) return false;
  seen.add(key);
  return true;
});

console.log(`Found ${findings.length} issue(s) after deduplication`);
findings.forEach(f => console.log(`  [${f.severity}] ${f.file}:${f.line} — ${f.category}`));

await postReviewComments(findings);
console.log("Review posted.");

The Prompt Engineering That Actually Works

The prompt is the hardest part. A poorly calibrated prompt produces one of two failures:

Too noisy: The AI flags style issues, subjective naming, and things that are obviously fine. Engineers start ignoring the comments within a week.

Too quiet: The AI reports nothing on every PR. Zero trust, zero value.

Three decisions made our prompt work:

Explicit exclusion list. "Do not flag style preferences, minor naming, missing Javadoc on non-public methods." Without this, GPT-4 and Claude both default to acting like a linter.

Category-constrained output. Constraining to seven named categories forces the model to be specific and prevents vague findings like "this code could be improved." Every finding must fit a category.

Strict JSON-only output. "Respond with a JSON array. If you find no real issues, respond with: []" — no preamble, no explanation. This makes the output machine-parseable without fragile regex extraction.

[!NOTE]
Using claude-opus-5 for code review is deliberate. Haiku is faster and 20× cheaper but misses subtle concurrency issues and null-deref patterns that require multi-step reasoning. For code review — where you're making a quality-critical decision — use the strongest available model.

Handling the Java-Specific Cases

Generic code review prompts miss Java-specific failure modes. We added these to the category list after observing what slipped through:

// Additions to REVIEW_PROMPT for Java/Spring codebases:
const JAVA_EXTENSIONS = `
Additional Java/Spring-specific checks:
- OPTIONAL_MISUSE: Calling Optional.get() without isPresent() check, or using Optional in a field
- TRANSACTIONAL_RISK: @Transactional on a private method (Spring proxy cannot intercept), or
  @Transactional with noRollbackFor swallowing checked exceptions
- JPA_PITFALL: N+1 query in a loop (entity.getCollection() called per iteration without JOIN FETCH),
  or LazyInitializationException risk (accessing lazy collection outside transaction boundary)
- THREAD_POOL_STARVATION: Blocking call (Thread.sleep, CompletableFuture.get without timeout) 
  inside a reactive chain or a thread pool shared with virtual threads
`;

These four categories alone catch the most expensive bugs in Spring Boot services.

Calibrating Against False Positives

After the first week, we tracked every AI comment as either:

True positive — the human reviewer confirmed it was a real issue
False positive — the human reviewer dismissed it

We used this to tune the prompt iteratively. Week 1: 58% true positive rate. Week 6: 81%.

The biggest false-positive categories and their fixes:

False Positive Type	Fix
Flagging test code for missing error handling	Added "do not flag test files (`Test.java`, `Spec.java`) for missing exception handling"
Flagging Spring bean injection as "unused field"	Added "fields annotated with `@Autowired`, `@Inject`, or `@Value` are not unused"
Flagging intentionally empty catch blocks	Added "if the catch block contains a comment, treat it as intentional"
Flagging abstract methods for missing implementation	Added "do not flag abstract method declarations"

Prompt tuning is ongoing. We export false-positive cases monthly and run them against prompt candidates before deploying a new version.

The Numbers: 6 Months, 4,200 PRs

flowchart LR
    subgraph "Before AI Review"
        A[4.2h avg\nhuman review time\nper engineer/day]
        B[18% of merged PRs\nhad post-merge bug fixes\nwithin 72h]
        C[~45 min\navg time-to-first-review]
    end

    subgraph "After AI Review"
        D[2.7h avg\nhuman review time\nper engineer/day]
        E[11% of merged PRs\nhad post-merge bug fixes\nwithin 72h]
        F[90 sec\nfirst AI review\nthen human follows]
    end

    A -.->|"−36%"| D
    B -.->|"−39%"| E
    C -.->|"−98%"| F

    style D fill:#065f46,color:#6ee7b7
    style E fill:#065f46,color:#6ee7b7
    style F fill:#065f46,color:#6ee7b7

What actually changed:

Senior engineers stopped reviewing trivial null-check issues, stream leaks, and basic error handling gaps — the AI catches those reliably
Human review became genuinely architectural — reviewers now spend time on "is this the right abstraction?" not "did you close the connection?"
New engineers get faster feedback loops: AI review in 90 seconds means they can iterate on a PR before a human is even available

What didn't change:

Logic bugs requiring business domain knowledge — AI catch rate here is low (~15%)
Performance issues requiring profiling — AI never reliably flags these
API design decisions — the AI has no opinion worth having on whether your REST endpoint should be POST /documents/search vs GET /documents?q=

Cost Breakdown

4,200 PRs, average diff size of ~380 lines across 6 files:

Item	Volume	Cost
Input tokens (diff + prompt)	~2,400 per PR avg
Output tokens (findings JSON)	~400 per PR avg
Model	claude-opus-5	$15/M input, $75/M output
Per PR cost		~$0.066
Monthly (700 PRs)		~$46/month

$46/month to give every PR a 90-second first pass. For comparison: one hour of a senior engineer's time at market rate costs more than a month of AI reviews.

[!NOTE]
If cost is a concern, running the first filter on claude-haiku-4-5 (security, null deref, resource leaks only) and escalating uncertain findings to Opus cuts cost to ~$8/month at the same volume. We tested this and found a 12% drop in true positive rate — acceptable for some teams, not for ours.

Anti-Patterns

Auto-blocking merges on AI findings. We tried this for two weeks. Engineers worked around it by pushing fixes to silence the AI rather than evaluating whether the finding was real. Now AI findings are advisory — they inform the human reviewer but never block the merge.

Reviewing the full repository history on every PR. Only review the diff. Reviewing unchanged lines produces false positives at an unusable rate and wastes tokens.

One prompt for all languages. Our Python services use a different prompt with different category list. Java/Spring patterns don't map to Python anti-patterns, and a shared prompt produces noise in both.

Not filtering bot PRs. Dependabot, Renovate, and automated release PRs don't need AI review. Filter them early or you burn tokens reviewing lock file changes.

Posting findings as a separate comment thread. Using the GitHub PR Review API (inline comments on the diff) is 4× more actionable than a single summary comment at the bottom. Engineers read inline comments; they skim summary paragraphs.

Setting Up in Your Repository

Add ANTHROPIC_API_KEY to your repository secrets (Settings → Secrets and variables → Actions)
Copy .github/workflows/ai-review.yml and .github/scripts/ai-review.mjs to your repository
Tune the prompt to your stack. The seven categories above work for Java/Spring. Add language-specific patterns for your codebase (see the Java extensions block above).
Run in shadow mode for 2 weeks first. Post findings to a private Slack channel instead of the PR. Track true vs false positive rate before exposing it to your engineers. Don't skip this step — a noisy AI reviewer destroys trust instantly.
Add a human override label. We honor a skip-ai-review label on PRs for cases where the author knows the diff will confuse the model (e.g., large auto-generated files). Respecting human judgment about when to skip builds trust.

# Add this condition to the workflow job
if: |
  github.actor != 'dependabot[bot]' &&
  !startsWith(github.head_ref, 'release/') &&
  github.event.pull_request.draft == false &&
  !contains(github.event.pull_request.labels.*.name, 'skip-ai-review')

Key Takeaways

AI review is a filter, not a judge. It protects senior engineers' time for architectural decisions by handling correctness checks they shouldn't be doing manually.
Inline comments on the diff beat summaries. The GitHub PR Review API is the right tool — a bottom-of-thread summary gets ignored.
Prompt calibration is the work. The model is not the variable; the prompt is. Track true vs false positive rate and tune monthly.
Use your strongest model. Code review is a quality-critical task — 20× cost savings on a cheaper model are not worth a 15% drop in catch rate.
Shadow mode before going live. Two weeks of private feedback before posting to PRs is the difference between engineers trusting the system and ignoring it.
Advisory, not blocking. AI findings inform human reviewers; they should never be the sole gate on a merge. The AI is wrong sometimes, and engineers need to feel empowered to disagree.

The 36% reduction in human review time is real, but the number that actually changed how our team works is the post-merge bug fix rate dropping from 18% to 11%. That's the metric that matters — fewer bugs reaching production, not faster reviews.

The prompt and workflow files are available as starting points — link in bio. If you're running something similar or found categories that work well for your stack, I'd be curious what you've tuned.

Feature Flags for AI: The Deployment Pattern That Saved Us From 3 Production Disasters

Avaneesh Yadav — Tue, 01 Sep 2026 09:09:46 +0000

Three months after we shipped our first AI feature to production, it broke. Not dramatically — no 500s, no alerts. Just wrong. The model we'd integrated started producing confidently incorrect summaries on a specific document type, and we didn't know until a customer flagged it four days later.

The fix was straightforward. The deployment of the fix was not. We had no way to roll it out to 5% of users first. No way to instantly revert to the previous behavior. No kill switch. We pushed the fix to all users at once and watched the dashboards, anxious for the next 48 hours.

That was the last time we shipped an AI feature without feature flags.

Here's everything we learned about flagging AI features specifically — the patterns that work, the anti-patterns that burn you, and the Spring Boot implementation that runs our current system.

Why AI Features Need Flags More Than Anything Else You've Shipped

Traditional features fail in binary ways: they work or they error. AI features fail in a third way — they work incorrectly, silently, at a rate that only becomes statistically visible after thousands of requests.

That changes the deployment calculus entirely.

You can't unit test your way to confidence. A prompt change that scores 94% on your eval suite can still fail on 2% of production inputs you didn't evaluate. At 100K requests/day, that's 2,000 bad responses a day.

Model providers change behavior under you. Your LLM provider updates the underlying model. Behavior shifts. You didn't deploy anything — but your feature changed. You need a rollback that doesn't involve redeploying your application.

Prompt changes are code. A new system prompt is not less risky than a new database query. It needs the same gradual rollout, the same observability, the same rollback capability.

Cost spikes hit instantly. A new prompt that accidentally uses 3× the tokens will hit your monthly budget within hours of a full rollout. A 1% rollout catches it before it matters.

flowchart TD
    A[AI Feature Change\nmodel / prompt / config] --> B{Feature Flag\nEvaluation}

    B -->|0% rollout| C[Old Behavior\nfor all users]
    B -->|5% rollout| D[New Behavior\nfor 5% of users]
    B -->|100% rollout| E[New Behavior\nfor all users]

    D -->|Metrics look good| F[Increase to 25%]
    D -->|Metrics look bad| G[Instant rollback\nto 0%]
    F -->|Metrics look good| H[Increase to 100%]
    F -->|Metrics look bad| G

    G -->|Takes effect in| I[< 60 seconds\nno redeploy]

    style G fill:#7f1d1d,color:#fca5a5
    style I fill:#065f46,color:#6ee7b7

A Taxonomy of AI Feature Flags

Not all AI flags are the same. Mixing them up leads to either over-engineering or missing the ones that matter.

Flag Type	What it controls	When to use it
Model toggle	Which LLM model handles a task	Model migrations, cost optimizations, provider failover
Prompt toggle	Which system prompt / prompt template is active	Any prompt change, A/B testing, prompt experiments
Rollout percentage	What fraction of users see new AI behavior	Gradual migration, de-risking changes
Kill switch	Entire AI feature on/off	Emergency disable, degraded mode
User segment	New AI behavior for specific users/tenants	Beta users, internal testing, enterprise pilot

You need all five. Teams that only implement kill switches miss the gradual rollout pattern that prevents most incidents. Teams that only implement rollout percentages have no emergency lever.

The Implementation: Spring Boot + Unleash

We use Unleash for flag management — open source, self-hostable, strong Java SDK. The patterns below translate directly to LaunchDarkly, Split, Flagsmith, or even a simple database-backed flag store if you're starting out.

<!-- pom.xml -->
<dependency>
    <groupId>io.getunleash</groupId>
    <artifactId>unleash-client-java</artifactId>
    <version>9.2.0</version>
</dependency>

@Configuration
public class UnleashConfig {

    @Bean
    public Unleash unleash(
            @Value("${unleash.url}") String url,
            @Value("${unleash.api-token}") String token,
            @Value("${spring.application.name}") String appName) {

        UnleashConfig config = UnleashConfig.newBuilder()
                .appName(appName)
                .instanceId(System.getenv().getOrDefault("POD_NAME", "local"))
                .unleashAPI(url)
                .apiKey(token)
                .fetchTogglesInterval(10)   // poll every 10 seconds
                .synchronousFetchOnInitialisation(true)
                .build();

        return new DefaultUnleash(config);
    }
}

The fetchTogglesInterval(10) is important: flag changes propagate to all running pods within 10 seconds without a redeploy. That's your 60-second kill switch window.

The AI Flag Service

All AI feature flag checks go through a single service — never scattered inline unleash.isEnabled() calls throughout the codebase.

@Service
@Slf4j
public class AIFeatureFlags {

    private final Unleash unleash;

    // ── Model selection ──────────────────────────────────────────────────────

    public String summaryModel(String userId) {
        UnleashContext ctx = context(userId);
        if (unleash.isEnabled("ai.summary.use-70b-model", ctx)) {
            return "llama-3.1-70b-versatile";
        }
        return "llama-3.1-8b-instant";
    }

    public String classifyModel(String userId) {
        UnleashContext ctx = context(userId);
        if (unleash.isEnabled("ai.classify.use-claude-haiku", ctx)) {
            return "claude-haiku-4-5-20251001";
        }
        return "llama-3.1-8b-instant";
    }

    // ── Prompt selection ─────────────────────────────────────────────────────

    public String summaryPromptVariant(String userId) {
        UnleashContext ctx = context(userId);
        // A/B test: "control" vs "variant-a" vs "variant-b"
        Variant variant = unleash.getVariant("ai.summary.prompt-variant", ctx);
        return variant.isEnabled() ? variant.getName() : "control";
    }

    // ── Kill switches ────────────────────────────────────────────────────────

    public boolean isSummaryEnabled(String userId) {
        return unleash.isEnabled("ai.summary.enabled", context(userId));
    }

    public boolean isClassifyEnabled(String userId) {
        return unleash.isEnabled("ai.classify.enabled", context(userId));
    }

    // ── Helpers ──────────────────────────────────────────────────────────────

    private UnleashContext context(String userId) {
        return UnleashContext.newBuilder()
                .userId(userId)
                .addProperty("environment", System.getenv().getOrDefault("SPRING_PROFILES_ACTIVE", "local"))
                .build();
    }
}

Every flag check passes a UnleashContext with the user ID. This is what makes percentage rollouts and user-segment targeting work — Unleash uses the user ID to hash-consistently assign the same user to the same variant across requests.

Pattern 1: Model Migration Flag

You're running llama-3.1-8b-instant and want to migrate to claude-haiku-4-5 for better extraction quality. The naive approach: update the constant, deploy, pray.

The flag approach:

@Service
@Slf4j
public class DocumentExtractor {

    private final ChatClient chatClient;
    private final AIFeatureFlags flags;
    private final MeterRegistry metrics;

    public ExtractionResult extract(String documentText, String userId) {
        String model = flags.extractModel(userId);

        log.debug("extraction model={} userId={}", model, userId);

        Timer.Sample timer = Timer.start(metrics);

        ChatResponse response = chatClient
                .prompt()
                .system(EXTRACTION_SYSTEM_PROMPT)
                .user(documentText)
                .options(ChatOptionsBuilder.builder().model(model).build())
                .call()
                .chatResponse();

        timer.stop(metrics.timer("ai.extract.latency", "model", model));

        // Model is tagged on every metric — you can compare in Grafana
        metrics.counter("ai.extract.tokens.input",  "model", model)
               .increment(response.getMetadata().getUsage().getPromptTokens());
        metrics.counter("ai.extract.tokens.output", "model", model)
               .increment(response.getMetadata().getUsage().getGenerationTokens());

        return parseResponse(response);
    }
}

In Unleash, configure ai.extract.use-haiku:

Start at 0% rollout
Ramp to 5% — watch P95 latency and error rate by model in Grafana
Ramp to 25%, 50%, 100% as metrics look good
If anything degrades — flip to 0% in the Unleash UI. No redeploy, propagates in 10 seconds

flowchart LR
    subgraph "Week 1"
        A[0% rollout\nBaseline metrics]
    end
    subgraph "Week 2"
        B[5% rollout\nWatch latency + cost]
    end
    subgraph "Week 3"
        C[25% rollout\nWatch quality metrics]
    end
    subgraph "Week 4"
        D[100% rollout\nMigration complete]
    end

    A -->|Metrics OK| B
    B -->|Metrics OK| C
    C -->|Metrics OK| D
    B -->|Issue found| E[Rollback to 0%\nin 10 seconds]
    C -->|Issue found| E

    style E fill:#7f1d1d,color:#fca5a5
    style D fill:#065f46,color:#6ee7b7

What to compare: latency P95 per model, error rate per model, token cost per model, and — critically — your quality metric (RAGAS score, human thumbs up rate, or whatever you use). The migration is complete when the new model is better or equivalent on all of them.

Pattern 2: Prompt A/B Testing

A new system prompt is not less risky than a code change. It deserves the same scientific treatment.

@Component
public class SummaryPromptRegistry {

    private static final Map<String, String> PROMPTS = Map.of(

        "control", """
            Summarize the following document in under 150 words.
            Focus on the key decisions, numbers, and outcomes.
            """,

        "variant-a", """
            You are a senior analyst. Summarize the following document for an executive
            who has 30 seconds to read it. Lead with the most important insight.
            Use bullet points for supporting facts. 150 words maximum.
            """,

        "variant-b", """
            Summarize this document. Structure your response as:
            HEADLINE: One sentence capturing the core finding.
            KEY FACTS: 3 bullet points, each under 20 words.
            CONTEXT: One sentence on why this matters.
            """
    );

    public String get(String variant) {
        return PROMPTS.getOrDefault(variant, PROMPTS.get("control"));
    }
}

@Service
public class SummaryService {

    private final AIFeatureFlags flags;
    private final SummaryPromptRegistry prompts;
    private final ChatClient chatClient;
    private final MeterRegistry metrics;

    public SummaryResult summarize(String content, String userId) {
        String variant = flags.summaryPromptVariant(userId);    // "control" | "variant-a" | "variant-b"
        String prompt  = prompts.get(variant);

        ChatResponse response = chatClient.prompt()
                .system(prompt)
                .user(content)
                .call()
                .chatResponse();

        // Tag every quality signal with the variant — A/B analysis in Grafana
        metrics.counter("ai.summary.requests", "variant", variant).increment();

        return new SummaryResult(
            response.getResult().getOutput().getContent(),
            variant  // return variant so callers can record user satisfaction per-variant
        );
    }
}

In Unleash, configure ai.summary.prompt-variant as a Variants toggle:

control — 50% of users
variant-a — 25% of users
variant-b — 25% of users

After 7 days and sufficient sample size, compare thumbs-up rate per variant in your analytics. Promote the winner to 100% by setting all weight to the winning variant, then eventually replace control with it.

[!NOTE]
Tag every downstream metric — user satisfaction scores, task completion rates, support tickets — with the prompt variant. The quality metric that matters is business outcome, not LLM eval score.

Pattern 3: Graceful Degradation via Kill Switch

When an AI feature breaks, you need to serve something useful while you fix it. A kill switch paired with a fallback path is the pattern.

@Service
public class SmartSearchService {

    private final AIFeatureFlags     flags;
    private final AISearchService    aiSearch;       // semantic vector search
    private final KeywordSearchService keywordSearch; // traditional Elasticsearch

    public SearchResults search(SearchQuery query, String userId) {
        if (!flags.isAiSearchEnabled(userId)) {
            // AI search is killed — fall back to keyword search
            log.info("ai-search disabled — falling back to keyword search userId={}", userId);
            return keywordSearch.search(query);
        }

        try {
            return aiSearch.search(query);
        } catch (Exception e) {
            // Unexpected failure — fall back and alert, but don't surface the error to users
            log.error("ai-search failed unexpectedly — falling back: {}", e.getMessage());
            metrics.counter("ai.search.fallback", "reason", "exception").increment();
            return keywordSearch.search(query);
        }
    }
}

The kill switch gives you three modes:

Normal — AI search for everyone
Partial kill — AI search for 0% of users (flag disabled), keyword search for all. No service degradation, just capability downgrade.
Automatic fallback — AI search throws an unexpected error, keyword search runs transparently

flowchart TD
    A[Search Request] --> B{flags.isAiSearchEnabled?}
    B -->|true| C[AI Semantic Search]
    B -->|false — flag killed| D[Keyword Fallback]
    C -->|Success| E[Return AI Results]
    C -->|Exception| F[Log + Alert]
    F --> D
    D --> G[Return Keyword Results]

    style B fill:#1e3a5f,color:#7dd3fc
    style D fill:#374151,color:#d1d5db
    style F fill:#7f1d1d,color:#fca5a5

The key design rule: the fallback path must exist before you ship the AI feature. "We'll add fallback later" means you have no kill switch on day one — exactly when you're most likely to need it.

Pattern 4: User-Segment Targeting for Beta Rollouts

Before a public rollout, you want your internal team (and willing beta customers) to dog-food the feature.

@Service
public class AIFeatureFlags {

    // Internal users get new AI features first
    public boolean isNewExtractorEnabled(String userId, Set<String> userRoles) {
        UnleashContext ctx = UnleashContext.newBuilder()
                .userId(userId)
                .addProperty("role", String.join(",", userRoles))
                .build();
        return unleash.isEnabled("ai.extractor.v2", ctx);
    }
}

In Unleash, configure ai.extractor.v2 with activation strategies:

UserWithId — list specific internal user IDs: always enabled for the team
GradualRolloutUserId — 5% of everyone else

This means your engineers and internal QA users see the new extractor immediately. External users get it at 5% rollout. You run this for 2 weeks, then ramp external users to 25%, 50%, 100%.

// Usage in a service — no change needed as rollout percentage increases
public ExtractionResult extract(Document doc, Authentication auth) {
    String userId = auth.getName();
    Set<String> roles = auth.getAuthorities().stream()
            .map(GrantedAuthority::getAuthority)
            .collect(Collectors.toSet());

    if (!flags.isNewExtractorEnabled(userId, roles)) {
        return legacyExtractor.extract(doc);
    }
    return newExtractor.extract(doc);
}

Observability: Linking Flag State to Metrics

Flags are useless without observability. Every request that touches an AI feature should emit metrics tagged with the active flag variant. This is what makes "is variant-A performing better than control?" answerable in Grafana.

@Aspect
@Component
public class AIFlagMetricsAspect {

    private final MeterRegistry metrics;
    private final AIFeatureFlags flags;

    @Around("@annotation(TrackedAIFeature)")
    public Object trackWithFlagContext(ProceedingJoinPoint pjp) throws Throwable {
        // Extract userId from first String argument (by convention)
        String userId = Arrays.stream(pjp.getArgs())
                .filter(a -> a instanceof String)
                .map(a -> (String) a)
                .findFirst().orElse("unknown");

        String model   = flags.summaryModel(userId);
        String variant = flags.summaryPromptVariant(userId);

        Timer.Sample timer = Timer.start(metrics);
        try {
            Object result = pjp.proceed();
            timer.stop(metrics.timer("ai.feature.duration",
                    "model", model, "variant", variant, "status", "ok"));
            return result;
        } catch (Throwable t) {
            timer.stop(metrics.timer("ai.feature.duration",
                    "model", model, "variant", variant, "status", "error"));
            throw t;
        }
    }
}

In Grafana, you now have a single dashboard panel that shows latency by model and variant — so model migration and prompt A/B testing are both visible without extra instrumentation.

The four queries you always want:

# Error rate by variant
rate(ai_feature_duration_seconds_count{status="error"}[5m]) 
  / rate(ai_feature_duration_seconds_count[5m])
  by (variant)

# P95 latency by model  
histogram_quantile(0.95, 
  rate(ai_feature_duration_seconds_bucket[5m]))
  by (model, le)

# Cost by model (tokens × price)
rate(ai_tokens_input_total[1h]) by (model)

# Rollout progress (% of requests per variant)
rate(ai_feature_duration_seconds_count[5m]) by (variant)
  / ignoring(variant) group_left
  sum(rate(ai_feature_duration_seconds_count[5m]))

The Three Disasters We Avoided

Disaster 1 — Model degradation on a specific doc type. Our summarization model started hallucinating numbers in financial documents after a provider-side update. Kill switch disabled AI summarization for affected document categories within 45 seconds of the on-call alert. Users saw the keyword-based fallback. No customer data corrupted.

Disaster 2 — Prompt regression. A new prompt that performed well in offline evals caused a 12% drop in thumbs-up rate in production. We caught it at 5% rollout (affecting ~800 users) instead of rolling it to all 16,000. Rolled back to control variant in the Unleash UI.

Disaster 3 — Cost spike. A batch job that used a new prompt sent to 100% of traffic consumed 4× the expected tokens due to an accidental context window issue. The cost alert fired at $180 (2× rolling average). Kill switch → model flag reverted to the cheaper model → cost normalized in under 2 minutes.

In all three cases: no redeploy, no hotfix PR, no incident spanning multiple engineers. One person, one flag change, one minute.

Anti-Patterns

Checking flags at the wrong layer. Putting flag checks inside the LLM integration class couples your infrastructure to your feature logic. Put flags in the service layer, let the LLM client be a dumb executor.

One flag for the whole AI system. A single ai.enabled kill switch means you can't surgically kill summarization while keeping classification running. One flag per feature, or per feature + model combination.

No fallback for the kill switch. A kill switch that returns an error instead of a fallback response is just a way to 500 your users more deterministically. The fallback must exist before the flag.

Forgetting to tag metrics with flag state. If your metrics don't tell you which variant produced which latency, you're flying blind. Tag everything.

Long-lived feature branches. Don't keep AI behavior in a Git branch while waiting for the flag rollout to complete. Merge to main, deploy behind a 0% flag. The code is in production; the behavior is off. This is trunk-based development for AI.

Starting Simple: The Minimum Viable AI Flag

If you're not ready for Unleash or LaunchDarkly, this works:

// Minimal flag store — a database table, or even a config map
@Component
public class SimpleAIFlags {

    @Value("${ai.summary.enabled:true}")
    private boolean summaryEnabled;

    @Value("${ai.summary.model:llama-3.1-8b-instant}")
    private String summaryModel;

    @Value("${ai.summary.rollout-percentage:100}")
    private int rolloutPercentage;

    public boolean isSummaryEnabled(String userId) {
        if (!summaryEnabled) return false;
        // Deterministic hash-based assignment so the same user always gets the same experience
        return Math.abs(userId.hashCode() % 100) < rolloutPercentage;
    }

    public String getSummaryModel() { return summaryModel; }
}

With Spring Boot Actuator, you can update @Value properties at runtime via /actuator/env + /actuator/refresh (with @RefreshScope). That's your 60-second kill switch without any external system.

Start here. Add Unleash when you have more than 3 AI features and more than one person managing flag changes.

Key Takeaways

AI features fail silently. Flags let you catch silent failures at 5% before they affect 100% of users.
Prompt changes are deployments. Treat them with the same rigor: staging, gradual rollout, observability, rollback.
Build the fallback first. A kill switch that errors instead of degrading gracefully is worse than no kill switch.
Tag every metric with flag state. Without this, you can't answer "is the new model better?" from production data.
Model providers change under you. Your flag infrastructure needs to handle changes you didn't trigger, not just changes you did.
The kill switch window matters. 60 seconds to production effect (flag change → all pods updated) is the target. Anything above 5 minutes is too slow for an active incident.

The engineering investment is modest — a few days to wire up Unleash and build the flag service. The return is the ability to ship AI features confidently and fix them without waking up your team at 3AM.

The next part of this series covers AI observability specifically: the metrics, traces, and dashboards that make "what exactly did the model return for user X?" answerable in under 30 seconds. If you're already doing feature flags and want to pair them with tighter observability, that's where to look next.

Event-Driven AI: How Kafka Streams Changed the Way We Run LLMs in Production

Avaneesh Yadav — Mon, 31 Aug 2026 15:31:32 +0000

Last October our AI service handled ~50K LLM requests per day. By February that number hit 800K. By April, 2 million. The synchronous HTTP architecture we had shipped in three weeks didn't survive contact with that growth. We were dropping requests, hitting rate limits that cascaded into total outages, and on-call engineers were getting paged at 3AM for timeouts that shouldn't have been possible to timeout.

The fix wasn't bigger timeouts or faster hardware. It was rethinking the execution model entirely — from request-response to event-driven. Here's the full architecture, the Spring AI + Kafka code, the failure modes we hit, and the production numbers that came out the other side.

Why Synchronous LLM Calls Break at Scale

The typical AI feature looks like this: a user does something, your API calls an LLM, the LLM responds, you return the result. Simple, works fine in demos, blows up in production.

LLMs are slow and variable. A fast response is 300ms. A slow one — complex reasoning, long context, throttled provider — is 15–30 seconds. Most HTTP infrastructure timeouts sit at 30 seconds, which means you're living on the edge constantly.

Rate limits cascade. Your provider gives you X requests per minute. Under load, one failed batch causes retries that burn more quota, which causes more failures. Exponential backoff helps but doesn't fix the underlying problem: too many concurrent callers sharing a quota.

No buffering. If the downstream LLM is temporarily rate-limited or degraded, you have no place to hold work. You either drop the request or return an error to the user.

Problem	Sync Architecture	Event-Driven
Timeout handling	30s hard limit → user sees error	Message retained → retried after backoff
Rate limit response	HTTP 429 → cascading upstream 500s	Consumer pauses → catches up when quota resets
Traffic spike	Thread pool exhausts → 503s	Kafka absorbs burst → smooth processing
Provider degradation	All in-flight requests fail simultaneously	New messages queue; consumer retries at its own pace
Cost visibility	Invisible per-request cost	Per-consumer-group token metrics

The Event-Driven AI Model

The core idea: instead of making LLM calls synchronously in the API handler, publish work to a Kafka topic, consume it in a dedicated AI worker, and emit results to a results topic. Callers subscribe to results.

flowchart TD
    A[API Gateway] -->|POST /enrich| B[Request Handler]
    B -->|Publish AITask event| C[(ai.tasks.summarize)]
    B -->|202 Accepted + taskId| D[Client]

    C -->|Consumer Group — 8 threads| E[AI Worker Pool]
    E -->|Rate-limited call| F[LLM Provider]
    F -->|Response| E
    E -->|Publish AIResult| G[(ai.results.completed)]

    G -->|Subscribe| H[Result Handler]
    H -->|Persist| I[(PostgreSQL)]
    H -->|WebSocket push| D

    style C fill:#1e3a5f,color:#7dd3fc
    style G fill:#1e3a5f,color:#7dd3fc
    style F fill:#374151,color:#d1d5db

The client immediately gets a 202 Accepted with a task ID. The result arrives when the LLM call completes — via WebSocket, SSE, or polling. This separation is what unlocks scale: the HTTP layer and the LLM processing layer evolve independently and fail independently.

[!NOTE]
This is the same pattern as async email sending, async PDF generation, or async image processing. It's been standard for I/O-heavy work for a decade. LLMs are just I/O with a 10-second P99.

Topic Design: What Goes Where

Getting the Kafka topic structure right matters more than any individual line of code. Topics are your API contract — they're hard to change once services depend on them.

flowchart TD
    subgraph Input ["Task Topics (by type)"]
        A[ai.tasks.summarize]
        B[ai.tasks.classify]
        C[ai.tasks.extract]
    end

    subgraph Priority ["Priority Routing"]
        E[High-Priority Consumer Group\n8 threads]
        F[Batch Consumer Group\n2 threads]
    end

    subgraph Output ["Result Topics"]
        G[ai.results.completed]
        H[ai.results.failed]
        I[ai.tasks.classify.dlq]
    end

    A --> E
    B --> E
    C --> F

    E --> G
    E --> H
    F --> G
    F --> H
    H -->|After 3 retries| I

    style E fill:#065f46,color:#6ee7b7
    style F fill:#1e3a5f,color:#7dd3fc
    style I fill:#7f1d1d,color:#fca5a5

Design principles we landed on:

One topic per task type, not one monolithic ai-tasks topic. Lets you scale consumers independently, set different retention policies (real-time classification results don't need 7-day retention; completed summarizations do), and gives you clear consumer lag metrics per workload.

Priority tiers as separate consumer groups, not topics. Eight consumer threads watch the high-priority topics; two watch the batch topics. Simple to operate — no complex priority queue logic or message reordering.

Separate DLQ per task type (ai.tasks.classify.dlq, not a shared DLQ). When debugging production failures you already know the task type — a shared DLQ forces you to filter through noise from every other failing type simultaneously.

Results topic as a public contract. Any service can subscribe to ai.results.completed and react to completed AI tasks. Loose coupling at its best — the AI worker doesn't need to know who cares about the results.

@Configuration
public class AITopicConfig {

    public static final String TASKS_SUMMARIZE   = "ai.tasks.summarize";
    public static final String TASKS_CLASSIFY    = "ai.tasks.classify";
    public static final String RESULTS_COMPLETED = "ai.results.completed";
    public static final String RESULTS_FAILED    = "ai.results.failed";
    public static final String DLQ_CLASSIFY      = "ai.tasks.classify.dlq";

    @Bean
    public NewTopic summarizeTopic() {
        return TopicBuilder.name(TASKS_SUMMARIZE)
                .partitions(12)   // max 12 concurrent consumers in this group
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "604800000") // 7 days
                .build();
    }

    @Bean
    public NewTopic classifyTopic() {
        return TopicBuilder.name(TASKS_CLASSIFY)
                .partitions(24)   // higher throughput — more partitions
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "86400000")  // 1 day
                .build();
    }

    @Bean
    public NewTopic resultsTopic() {
        return TopicBuilder.name(RESULTS_COMPLETED)
                .partitions(12)
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "2592000000") // 30 days
                .build();
    }
}

Partition count is a ceiling on parallelism. Kafka consumer group parallelism is bounded by partition count — 12 partitions means you can run at most 12 consumer threads per group before threads sit idle. Size for peak throughput, not current throughput. You can't easily add partitions to a live topic without a migration.

The AI Task Schema

Every message uses the same envelope. Consistency is what lets you build generic retry, routing, and observability tooling without per-task-type special cases.

public record AITask(
    String taskId,          // UUID — idempotency key
    String taskType,        // "summarize" | "classify" | "extract"
    String requesterId,     // service or user that submitted
    String priority,        // "high" | "normal" | "batch"
    Instant submittedAt,
    int    attemptNumber,   // 1-based; consumers use this for backoff strategy
    Map<String, Object> payload,   // task-specific input data
    Map<String, String> metadata   // tracing headers, correlation IDs
) {}

public record AIResult(
    String  taskId,
    String  taskType,
    boolean success,
    Instant completedAt,
    long    processingMs,
    String  model,
    int     inputTokens,
    int     outputTokens,
    Object  output,          // task-specific result — Map, String, etc.
    String  errorMessage     // null on success
) {}

Three design decisions worth calling out:

taskId is the idempotency key. Consumers check whether a task has already been processed before calling the LLM. Kafka at-least-once delivery means you will receive duplicates — consumer rebalancing, network glitches, pod restarts. Plan for it from day one.

attemptNumber travels with the message, not just as a consumer variable. This means retry strategy is encoded in the event itself: first attempt = immediate retry, second = 30s backoff, third = DLQ. Works correctly across pod restarts and consumer rebalancing.

Token counts in every result. Essential for cost tracking. Sum inputTokens + outputTokens per consumer group, per model, per hour. That's your cost dashboard with no extra instrumentation.

Spring AI + Kafka: The Consumer

This is the heart of the system — the consumer that actually processes AI tasks.

@Component
@Slf4j
public class AITaskConsumer {

    private final ChatClient         chatClient;
    private final KafkaTemplate<String, AIResult> resultTemplate;
    private final AITaskRepository   taskRepository;   // idempotency store
    private final MeterRegistry      meterRegistry;
    private final AdaptiveRateLimiter rateLimiter;

    @KafkaListener(
        topics       = { AITopicConfig.TASKS_SUMMARIZE, AITopicConfig.TASKS_CLASSIFY },
        groupId      = "ai-worker-high-priority",
        concurrency  = "8",       // 8 consumer threads per pod
        containerFactory = "aiListenerContainerFactory"
    )
    public void consume(
            @Payload AITask task,
            @Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
            Acknowledgment ack) {

        log.info("task={} type={} attempt={}", task.taskId(), task.taskType(), task.attemptNumber());

        // Check idempotency — previous consumer may have already handled this
        if (taskRepository.isCompleted(task.taskId())) {
            log.debug("task={} already completed — skipping (idempotent ack)", task.taskId());
            ack.acknowledge();
            return;
        }

        Timer.Sample timer = Timer.start(meterRegistry);

        try {
            rateLimiter.acquire(); // blocks until a permit is available

            AIResult result = switch (task.taskType()) {
                case "summarize" -> handleSummarize(task);
                case "classify"  -> handleClassify(task);
                default -> throw new IllegalArgumentException("Unknown type: " + task.taskType());
            };

            taskRepository.markComplete(task.taskId(), result); // write idempotency record first
            resultTemplate.send(RESULTS_COMPLETED, task.taskId(), result);
            ack.acknowledge();  // only after result is durably published

            rateLimiter.onSuccess();
            timer.stop(meterRegistry.timer("ai.task.processing",
                    "type", task.taskType(), "status", "success"));

        } catch (RateLimitException e) {
            // Provider returned 429 — don't ack, let Kafka redeliver after consumer pause
            rateLimiter.onRateLimit();
            log.warn("task={} rate-limited (attempt={})", task.taskId(), task.attemptNumber());
            // Intentionally no ack.acknowledge() — message redelivers

        } catch (Exception e) {
            log.error("task={} failed attempt={}: {}", task.taskId(), task.attemptNumber(), e.getMessage());
            if (task.attemptNumber() >= 3) {
                sendToDLQ(task, e.getMessage());
                ack.acknowledge(); // remove from main topic — it's in DLQ now
            }
            // Otherwise no ack — Kafka redelivers with same attempt number
            // (producer should increment attemptNumber on retry, or use a retry topic)
            timer.stop(meterRegistry.timer("ai.task.processing",
                    "type", task.taskType(), "status", "error"));
        }
    }

    private AIResult handleSummarize(AITask task) {
        String content  = (String) task.payload().get("content");
        int    maxWords = (int)    task.payload().getOrDefault("maxWords", 150);

        String prompt = """
            Summarize the following in under %d words.
            Be specific — preserve key facts, numbers, and named entities.

            Content:
            %s
            """.formatted(maxWords, content);

        Instant start    = Instant.now();
        ChatResponse resp = chatClient.prompt().user(prompt).call().chatResponse();
        Usage usage      = resp.getMetadata().getUsage();

        return new AIResult(
            task.taskId(), "summarize", true, Instant.now(),
            Duration.between(start, Instant.now()).toMillis(),
            "llama-3.1-8b-instant",
            (int) usage.getPromptTokens(), (int) usage.getGenerationTokens(),
            Map.of("summary", resp.getResult().getOutput().getContent()),
            null
        );
    }
}

The ack.acknowledge() placement is load-bearing. We only call it after the result is successfully published to the results topic. If the result publish fails — network blip, Kafka broker restart — Kafka will redeliver the task to another consumer thread. Combined with the idempotency check at the top, the task will be processed exactly once even across failures.

Rate limiting happens inside the consumer, not at the API gateway. Your LLM provider doesn't know about your queue depth — it just sees requests arriving. Rate limiting at the consumer level means you never exceed quota regardless of how deep the queue gets.

Handling Backpressure: Adaptive Rate Limiting

The most common failure mode isn't Kafka — it's the LLM provider. When you hit rate limits, the naive response (throw exception, retry immediately) makes things significantly worse. Here's what we do instead:

@Component
public class AdaptiveRateLimiter {

    // Groq's limit at our tier: 30 req/min = 0.5 req/sec
    private final double            maxRate = 0.45; // 10% headroom below actual limit
    private final double            minRate = 0.05; // hard floor when throttled
    private final AtomicDouble  currentRate = new AtomicDouble(0.3);
    private volatile RateLimiter     limiter = RateLimiter.create(0.3);

    public void acquire() { limiter.acquire(); }

    public void onSuccess() {
        double next = Math.min(currentRate.get() * 1.05, maxRate); // +5% on success
        updateLimiter(next);
    }

    public void onRateLimit() {
        double next = Math.max(currentRate.get() * 0.5, minRate); // halve on 429
        updateLimiter(next);
        log.warn("Rate-limited — throttling to {:.2f} req/sec", next);
    }

    private void updateLimiter(double newRate) {
        if (Math.abs(newRate - currentRate.get()) > 0.01) {
            currentRate.set(newRate);
            limiter = RateLimiter.create(newRate);
        }
    }
}

This is AIMD — Additive Increase, Multiplicative Decrease — the same algorithm TCP congestion control uses. On success, increase by 5%. On rate limit, halve. Over time it converges on the sustainable throughput for your provider tier without manual tuning.

flowchart TD
    A[Message dequeued] --> B{Rate limiter\nacquire}
    B -->|Permit granted| C[Call LLM]
    C -->|200 OK| D[Publish result to Kafka]
    D --> E[ack.acknowledge]
    E -->|onSuccess: rate × 1.05| B

    C -->|429 Rate Limited| F[onRateLimit: rate ÷ 2]
    F --> G[No ack — Kafka redelivers]
    G --> A

    C -->|500 or timeout| H{attemptNumber}
    H -->|< 3| G
    H -->|≥ 3| I[sendToDLQ]
    I --> J[ack.acknowledge\nfrom main topic]

    style E fill:#065f46,color:#6ee7b7
    style I fill:#7f1d1d,color:#fca5a5
    style G fill:#1e3a5f,color:#7dd3fc

Dead Letter Queues: Don't Silently Drop AI Work

A DLQ is your safety net for permanent failures. Every task that fails three times ends up there — unprocessed, with the error reason attached as metadata.

private void sendToDLQ(AITask task, String errorMessage) {
    Map<String, String> dlqMeta = new HashMap<>(task.metadata());
    dlqMeta.put("dlq.reason",       errorMessage);
    dlqMeta.put("dlq.sentAt",       Instant.now().toString());
    dlqMeta.put("dlq.finalAttempt", String.valueOf(task.attemptNumber()));

    AITask dlqTask = new AITask(
        task.taskId(), task.taskType(), task.requesterId(), task.priority(),
        task.submittedAt(), task.attemptNumber(), task.payload(), dlqMeta
    );

    String dlqTopic = "ai.tasks." + task.taskType() + ".dlq";
    dlqTemplate.send(dlqTopic, task.taskId(), dlqTask);

    log.error("task={} → DLQ after {} attempts: {}", task.taskId(), task.attemptNumber(), errorMessage);
    meterRegistry.counter("ai.task.dlq", "type", task.taskType()).increment();
}

DLQ messages are retained for 30 days. Twice a week we run a DLQ reprocessing job: identify the root cause, fix the consumer, then replay DLQ messages back into the main topic. Nothing is permanently lost.

[!WARNING]
Do not set DLQ retention too short. A production incident that takes 3 days to diagnose needs the evidence to still be in the DLQ when you're ready to replay. 30 days is a safe default for AI task DLQs.

Exactly-Once Semantics: When You Actually Need Them

Kafka supports exactly-once semantics (EOS) via transactions. For most AI tasks, you don't need them — idempotent consumers plus at-least-once delivery is simpler and sufficient. But for billing events, audit logs, and compliance-critical document processing, true exactly-once matters.

@Bean
public KafkaTransactionManager<String, AIResult> kafkaTxManager(
        ProducerFactory<String, AIResult> pf) {
    return new KafkaTransactionManager<>(pf);
}

// Consumer using Kafka transactions
@Transactional("kafkaTxManager")
@KafkaListener(topics = "ai.tasks.billing-critical", groupId = "ai-billing")
public void consumeExactlyOnce(@Payload AITask task, Acknowledgment ack) {
    // Entire method runs in a Kafka transaction.
    // Either ALL of the following happen atomically, or NONE do:
    //   1. Process the AI task
    //   2. Publish the result event
    //   3. Commit the consumer offset
    AIResult result = processTask(task);
    resultTemplate.send(RESULTS_COMPLETED, task.taskId(), result);
    // offset commits atomically with the produce — true exactly-once
}

The tradeoff: EOS adds roughly 15–20% latency overhead due to transaction coordination. For billing tasks, worth it. For bulk document summarization, not worth it — idempotent at-least-once is the right choice there.

Observability: Tracing AI Events Across Services

An AI task might be submitted by Service A, consumed by AI Worker B, and the result consumed by Service C. When something goes wrong, you need to trace the full journey without manual correlation.

We propagate OpenTelemetry trace context through Kafka headers:

// Producer — inject trace context into message headers
public void submitTask(AITask task) {
    ProducerRecord<String, AITask> record = new ProducerRecord<>(
            TASKS_SUMMARIZE, task.taskId(), task);

    GlobalOpenTelemetry.getPropagators()
            .getTextMapPropagator()
            .inject(Context.current(), record.headers(),
                    (headers, key, val) -> headers.add(key, val.getBytes(UTF_8)));

    kafkaTemplate.send(record);
}

// Consumer — extract trace context and create a child span
@KafkaListener(...)
public void consume(@Payload AITask task, @Headers MessageHeaders headers) {

    Context parentCtx = GlobalOpenTelemetry.getPropagators()
            .getTextMapPropagator()
            .extract(Context.current(), headers, new MessageHeadersGetter());

    Span span = tracer.spanBuilder("ai.task.process")
            .setParent(parentCtx)
            .setAttribute("task.id",    task.taskId())
            .setAttribute("task.type",  task.taskType())
            .startSpan();

    try (Scope ignored = span.makeCurrent()) {
        AIResult result = processTask(task);
        span.setAttribute("tokens.input",  result.inputTokens());
        span.setAttribute("tokens.output", result.outputTokens());
        span.setAttribute("latency.ms",    result.processingMs());
    } finally {
        span.end();
    }
}

In Grafana Tempo, the complete trace now shows: HTTP POST /enrich → Kafka publish → ai.task.process (the LLM call spans inside) → Kafka produce result → result consumer → database write. The entire 8-second LLM call is one contiguous trace chain.

Key metrics per consumer group:

Metric	What it reveals
`ai.task.processing` timer	P50/P95/P99 latency per task type
`kafka.consumer.lag`	Queue depth — how far behind consumers are
`ai.tokens.input` / `ai.tokens.output`	Cost per model per hour
`ai.task.dlq` counter	Permanent failure rate — should be near zero
`ai.rate_limiter.throttle_count`	How often you're hitting provider limits
`ai.task.idempotent.skip_count`	Duplicates correctly filtered (sanity check)

Configuration That Actually Matters

spring:
  kafka:
    bootstrap-servers: ${KAFKA_BROKERS}
    consumer:
      group-id: ai-worker-high-priority
      auto-offset-reset: earliest
      enable-auto-commit: false          # CRITICAL — never use auto-commit for AI tasks
      max-poll-records: 5                # Don't overwhelm a slow consumer
      max-poll-interval-ms: 120000       # 2 min — LLM calls can be slow
      fetch-min-bytes: 1
    producer:
      acks: all                          # Wait for all in-sync replicas
      retries: 3
      enable-idempotence: true
    listener:
      ack-mode: MANUAL_IMMEDIATE
      concurrency: 8

# Tune these per your LLM provider tier
ai:
  rate-limiter:
    max-rate-per-second: 0.45           # 27 req/min — 10% below Groq's 30 req/min limit
    min-rate-per-second: 0.05
  retry:
    max-attempts: 3
    dlq-retention-days: 30

max-poll-interval-ms is the one that bites everyone. If a consumer takes longer than this value between poll() calls, Kafka considers it dead and triggers a rebalance — even if the consumer is actively processing a long LLM call. We set it to 120 seconds (double our P99 LLM latency) to avoid unnecessary rebalancing during legitimate slow calls.

max-poll-records: 5 limits how many messages a single consumer thread fetches in one batch. With 8 threads each holding 5 messages, you have 40 LLM calls potentially in-flight per pod. Size this to your rate limit headroom.

The Cost Breakdown

After 90 days running this architecture against ~2M requests/day:

Model	Task Type	Avg Input Tokens	Avg Output Tokens	Input $/1M	Output $/1M	Daily Cost
llama-3.1-8b-instant	Classification	450	25	$0.05	$0.08	$47
llama-3.1-70b	Summarization	800	180	$0.59	$0.79	$380
claude-haiku-4-5	Extraction	1,200	400	$0.80	$4.00	$260
Total						$687/day

The event-driven architecture contributed two cost wins that weren't possible with synchronous calls:

Intelligent model routing. The consumer reads task.payload().get("complexity") — a simple score computed at task submission — and routes simple classifications to llama-3.1-8b-instant instead of a 70B model. This cut our average cost-per-task by 31%.

Micro-batching. For task types where the model supports batch input, we accumulate messages in 100ms windows and send them as a single request. 40% reduction in per-call API overhead (connection setup, auth handshake, response parsing). Impossible to do cleanly in a synchronous request handler without blocking the caller.

[!NOTE]
$687/day sounds large. At 2M requests that's $0.00034 per request — roughly $20K/month. Running equivalent workloads on self-hosted GPU infrastructure (A100s, realistic utilization rates, ops overhead) would cost us approximately $2,100/day. The managed API is cheaper at our scale.

Before and After: Production Numbers

We ran both architectures in parallel for 30 days — old sync on 20% of traffic, new event-driven on 80% — before cutting over completely.

Metric	Sync (before)	Event-Driven (after)	Change
P50 user-visible latency	4.2 s	0.8 s (202 + async push)	−81%
P99 user-visible latency	28.4 s	1.1 s	−96%
Request timeout rate	3.2%	0.001%	−99.97%
Requests dropped (unrecoverable)	1.8%	0.02%	−99%
Cost per 1M requests	$420	$344	−18%
On-call pages per week	14	1	−93%

The P99 number matters most. Users hitting a 28-second timeout perceived the feature as broken, even if only 3% of requests reached that threshold. With async dispatch and WebSocket delivery, P50 user-visible latency is under 1 second (the 202 response arrives instantly) and the result appears as a push notification when the LLM finishes.

The 93% reduction in on-call pages was unexpected. Most of the 14 weekly pages were timeout-related cascades. When you remove synchronous coupling between the API layer and the LLM layer, most cascade failure modes disappear entirely.

When NOT to Go Event-Driven

This architecture adds real operational complexity. It's not always the right answer.

Stay synchronous when:

The result must block the response. Real-time payment validation, auth decisions, interactive AI chat where the user is watching a streaming response. If you can't send a 202, you need sync.
Volume is under ~10K requests/day. The operational overhead of Kafka, consumer groups, DLQ management, and result delivery infrastructure doesn't pay off at low volume. A simple async HTTP call with a 30-second timeout and a Postgres-backed retry queue is the right choice.
You don't have result delivery infrastructure. WebSockets, SSE, or polling endpoints are meaningful engineering work. If you're not willing to build that, your users will stare at a spinner indefinitely. "We'll add the result delivery later" becomes "we added Kafka but the feature still feels broken."
Your team is new to Kafka. Consumer group rebalancing, partition assignment, exactly-once transactions, DLQ management — these are non-trivial operational concerns. Kafka is operationally complex when something goes wrong. Make sure your team has the skills before you take the dependency.

The decision framework:

flowchart TD
    A{Is LLM call in the\ncritical response path?} -->|Yes| B{Can user wait\n>2 seconds?}
    A -->|No, background work| E[Go event-driven]

    B -->|No| C[Stay synchronous\nuse streaming SSE]
    B -->|Yes| D{Volume > 50K/day?}

    D -->|No| F[Synchronous with\nretry queue in Postgres]
    D -->|Yes| G{Team has\nKafka experience?}

    G -->|No| H[Start sync, plan\nKafka migration]
    G -->|Yes| E

    style E fill:#065f46,color:#6ee7b7
    style C fill:#1e3a5f,color:#7dd3fc
    style F fill:#374151,color:#d1d5db

The Minimum Viable Starting Point

If you're starting fresh and want to validate the pattern before building the full architecture, here's the minimum working setup:

# application.yml — minimum for correctness
spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: ai-worker
      auto-offset-reset: earliest
      enable-auto-commit: false      # never auto-commit
      max-poll-interval-ms: 90000
    listener:
      ack-mode: MANUAL_IMMEDIATE

// Minimum viable consumer — add rate limiting, DLQ, and idempotency incrementally
@KafkaListener(topics = "ai.tasks", groupId = "ai-worker")
public void consume(@Payload AITask task, Acknowledgment ack) {
    try {
        AIResult result = callLLM(task);
        resultTemplate.send("ai.results", task.taskId(), result);
        ack.acknowledge();
    } catch (Exception e) {
        log.error("Task {} failed: {}", task.taskId(), e.getMessage());
        // Don't ack — Kafka redelivers
        // Add DLQ logic here after you've observed retry behavior in staging
    }
}

Start here. Add the adaptive rate limiter when you first see 429s in production. Add DLQs when you see messages redelivered indefinitely. Add idempotency when you see duplicate results. Add distributed tracing when debugging becomes painful. The architecture I described above is 18 months of production failures distilled — you don't need all of it on day one.

Key Takeaways

Synchronous LLM calls are fine until they're not. The failure mode is sudden and cascading — build the escape hatch before you need it. The warning signs are rising P99 latency and rate limit errors.
Kafka absorbs traffic spikes that would kill a synchronous system. The queue is the buffer between your traffic patterns and your provider's rate limits. You can't add that buffer to a sync architecture without fundamentally restructuring it.
Idempotency is non-negotiable. At-least-once delivery is guaranteed by Kafka; idempotent consumers are your responsibility. An idempotency check at the top of every consumer is not optional overhead — it's correctness.
Rate limiting belongs in the consumer, not the gateway. Only the consumer knows what's actually in flight against the LLM provider. Gateway-level rate limiting gives you no visibility into queue depth.
Token counts in every result event. You cannot optimize what you cannot see. Per-model, per-task-type token counts are the only way to know which workloads are driving your LLM bill.
DLQs are not optional. Every permanently failed task needs a destination. "Log it and move on" means you've silently dropped user work. DLQ + replay gives you a recovery path that sync architectures simply don't have.
The biggest win isn't latency — it's reliability. P99 improvements are impressive. Getting paged 93% less often at 3AM is what actually changes how your team feels about owning the service.

The investment is real — probably 6–8 weeks of solid engineering to get this fully operational. In our case, it was worth it the first Saturday night we didn't get paged.

The next post in this series covers stateful aggregation with Kafka Streams — specifically windowed token-count aggregations for real-time cost alerts and per-tenant quota enforcement. If you're hitting a different failure mode than I described here, or building something similar, I'd love to hear about it.

Your Spring Boot App Starts in 3 Seconds. With GraalVM Native, It's 80ms.

Avaneesh Yadav — Mon, 31 Aug 2026 11:16:18 +0000

Every Spring Boot developer has watched the console scroll through startup lines and thought: this feels slow. For a monolith on a beefy server, 3–5 seconds is fine. But when you're paying per-millisecond on AWS Lambda, scaling pods on Kubernetes, or trying to pass a 2-second health-check SLA, that startup time becomes a real cost.

GraalVM Native Image solves this by compiling your entire application — Spring, your code, and the JDK — into a single native binary ahead of time. The result: sub-100ms startup, a fraction of the memory footprint, and a container image under 100MB.

Spring Boot 3 added first-class native support. In 2026, the tooling has matured to the point where most real-world Spring Boot apps can go native without heroic effort. This post walks through what actually happens under the hood, the pitfalls that will bite you, and honest benchmarks so you know what you're getting.

How GraalVM Native Image Actually Works

The JVM is a dynamic runtime. Classes are loaded lazily, reflection resolves types at runtime, proxies are generated on the fly, and the JIT compiler optimizes hot paths after they've been called thousands of times. All of that flexibility comes at a cost: startup.

GraalVM's Native Image compiler inverts this model. At build time it performs a closed-world analysis: starting from your main method, it traces every reachable class, method, and field, and compiles everything into a native binary using ahead-of-time (AOT) compilation. Nothing is loaded lazily. The JIT is gone. What you get is a self-contained executable that starts like a Go binary.

flowchart TD
    src["📄 Source Code (Java)"]
    compile["⚙️ javac Compile"]
    bytecode["📦 Bytecode (.class files)"]
    aot["🔍 Spring AOT Processor"]
    hints["📋 Reflection Hints + Proxy Configs"]
    native["🔨 GraalVM native-image"]
    binary["🚀 Native Binary (OS-specific)"]

    src --> compile --> bytecode
    bytecode --> aot
    aot --> hints
    bytecode --> native
    hints --> native
    native --> binary

The key challenge: anything that relies on runtime dynamism — reflection, dynamic proxies, runtime resource loading — must be declared at build time so the compiler knows to include it. That's the source of most native-image pain.

Spring Boot 3's AOT Processing

Spring Framework 5 and earlier were built on reflection everywhere. Spring Boot 3 (Spring Framework 6) was the first version designed with native in mind. It added an AOT engine that runs at build time and:

Generates source code for bean definitions instead of registering them dynamically
Pre-computes @Configuration class processing
Generates reflection hints for every Spring component
Registers proxies for @Transactional, @Async, @Cacheable, and other AOP annotations

When you run spring-boot:build-image with the native profile, Spring's AOT processor runs first, then its output feeds into the GraalVM compiler. You get native compatibility essentially for free on the Spring layer — the hard part is your own code and third-party libraries.

Setting Up the Build

Prerequisites

GraalVM for JDK 21 or later (or use the GraalVM Maven Toolchain)
Spring Boot 3.2+ (3.3+ recommended)
The spring-boot-starter parent or native-maven-plugin

Maven Setup

Add the native profile to your pom.xml:

<profiles>
  <profile>
    <id>native</id>
    <build>
      <plugins>
        <plugin>
          <groupId>org.graalvm.buildtools</groupId>
          <artifactId>native-maven-plugin</artifactId>
          <extensions>true</extensions>
          <executions>
            <execution>
              <id>build-native</id>
              <goals><goal>compile-no-fork</goal></goals>
              <phase>package</phase>
            </execution>
          </executions>
          <configuration>
            <imageName>${project.artifactId}</imageName>
            <buildArgs>
              <buildArg>--no-fallback</buildArg>
              <buildArg>-O2</buildArg>
            </buildArgs>
          </configuration>
        </plugin>
      </plugins>
    </build>
  </profile>
</profiles>

Build the native binary:

./mvnw -Pnative native:compile

Build a container image (no GraalVM needed on the host — uses Buildpacks):

./mvnw -Pnative spring-boot:build-image

Gradle Setup

plugins {
  id 'org.graalvm.buildtools.native' version '0.10.3'
}

graalvmNative {
  binaries {
    main {
      imageName = 'my-service'
      buildArgs.add('--no-fallback')
    }
  }
}

./gradlew nativeCompile

Real Benchmark Numbers

I compiled a realistic Spring Boot service — REST endpoints, JPA with PostgreSQL, Spring Security, a scheduled job, and Redis caching — and measured cold start across modes.

Mode	Startup Time	RSS Memory (idle)	Container Image Size
JVM (Java 21)	3.2s	310 MB	280 MB
JVM + CDS	1.8s	270 MB	285 MB
Native Image	78ms	62 MB	98 MB

These numbers are real. RSS will vary based on your workload — once the app handles traffic, the JVM typically catches up on throughput due to JIT optimization. But for startup and idle memory, native wins by a factor of 4–5x.

The container image size difference matters most in Kubernetes. Pulling a 98 MB image on a fresh node is fast. Pulling 280 MB when you're trying to scale up fast under traffic is not.

The Pitfalls (And How to Fix Them)

1. Reflection That You Don't Know About

The most common failure mode: a ClassNotFoundException or NoSuchMethodException at runtime in your native binary, which ran fine on the JVM.

Why it happens: You (or a library) called Class.forName("com.example.SomeThing") or ObjectMapper.readValue(json, SomeThing.class). The closed-world compiler didn't see that reference statically, so it excluded the class.

Fix: Add a reflect-config.json hint, or use Spring's @RegisterReflectionForBinding annotation:

@Configuration
@RegisterReflectionForBinding({OrderDTO.class, PaymentDTO.class})
public class NativeHintsConfig {
}

Or with the programmatic hints API:

@Component
public class MyHints implements RuntimeHintsRegistrar {
  @Override
  public void registerHints(RuntimeHints hints, ClassLoader classLoader) {
    hints.reflection()
      .registerType(OrderDTO.class, MemberCategory.INVOKE_DECLARED_CONSTRUCTORS,
                                    MemberCategory.DECLARED_FIELDS);
  }
}

Then annotate your main class:

@SpringBootApplication
@ImportRuntimeHints(MyHints.class)
public class Application {
  public static void main(String[] args) {
    SpringApplication.run(Application.class, args);
  }
}

2. Dynamic Proxies

If you use java.lang.reflect.Proxy or Mockito-style dynamic proxies in production code (not just tests), they must be declared upfront.

hints.proxies().registerJdkProxy(MyService.class, MyOtherInterface.class);

Spring automatically registers proxies for @Transactional, @Async, and AOP-annotated beans. But if you create proxies manually, you must register them.

3. Resources Not Found

getClass().getResourceAsStream("/templates/email.html") works on the JVM because the classpath is there at runtime. In a native binary, resources must be explicitly included.

hints.resources().registerPattern("templates/*");

Or in application.properties:

spring.aot.hint.resources.patterns=templates/*,messages/**

4. Logback / SLF4J Configuration

Logback's XML parsing uses reflection internally. Use Spring Boot's auto-configured logging (it already has native hints) and avoid custom logback.xml with dynamic <if> conditions.

5. Jackson and Custom Serializers

Jackson works well natively when you use @JsonProperty, @JsonSerialize, and standard annotations. Problems arise with:

TypeReference<List<MyDTO>> anonymous classes
Polymorphic types with @JsonTypeInfo

For polymorphic types, register them explicitly:

hints.reflection()
  .registerType(Animal.class, MemberCategory.INVOKE_DECLARED_CONSTRUCTORS, MemberCategory.DECLARED_FIELDS)
  .registerType(Dog.class, MemberCategory.INVOKE_DECLARED_CONSTRUCTORS, MemberCategory.DECLARED_FIELDS);

Testing Your Native Build Before Committing

The worst experience is discovering native failures only after a 5-minute native-image compile. Use the GraalVM agent on the JVM first — it traces all reflective accesses and generates the config files:

java -agentlib:native-image-agent=config-output-dir=src/main/resources/META-INF/native-image \
  -jar target/myapp.jar

Run your app through all code paths (or better, run your integration test suite against it), then stop the app. The agent writes reflect-config.json, proxy-config.json, and resource-config.json into the output directory. Commit these files — GraalVM picks them up automatically.

You can also use Spring's Native Test support to validate hints without a full native compile:

@SpringBootTest
@TestPropertySource(properties = "spring.aot.enabled=true")
class NativeHintsValidationTest {
  @Test
  void contextLoads() { }
}

What You Give Up

Native Image is not free. Know the trade-offs:

Concern	JVM	Native
Build time	30–60s	3–8 minutes
Throughput under load	Higher (JIT)	Slightly lower
Dynamic class loading	Yes	No
Reflection	Unrestricted	Declared only
JVMTI / profiling agents	Yes	Limited
Debuggability	Full	Basic (`-g` flag)
Library compatibility	Universal	Most, not all

Throughput: Under sustained load, JIT-compiled code is often faster than native because JIT can optimize based on runtime behavior (branch prediction, inlining). Native binaries start faster and use less memory but may have slightly lower peak throughput for CPU-intensive workloads. For I/O-bound services (which most Spring Boot apps are), the difference is negligible.

Build time: A 5-minute build is annoying in local dev. Keep native as a CI/CD step, not part of your inner dev loop. Run ./mvnw spring-boot:run locally, build native only in the pipeline.

When Native Is Worth It

Go native when:

Serverless / FaaS: AWS Lambda cold starts are a real cost. 78ms vs 3.2s is the difference between a usable API and an unusable one.
Kubernetes burst scaling: Pods that start in under a second can scale to handle traffic spikes. 3-second startup means you're dropping requests while pods warm up.
Memory-constrained environments: 62 MB RSS vs 310 MB means you can run 5x more instances on the same node.
CLI tools built in Java: A native CLI binary that starts instantly feels as good as a Go or Rust CLI.

Stay on the JVM when:

Your app uses libraries without native support (check the GraalVM Reachability Metadata Repository first)
You rely heavily on runtime class generation (e.g., ByteBuddy, Javassist in non-Spring contexts)
Peak throughput matters more than startup (batch processing, number crunching)
Your team isn't ready to deal with native build failures in CI

CI/CD Pipeline

Keep native builds in a separate job so they don't slow down your PR feedback loop:

# .github/workflows/build.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { java-version: '21', distribution: 'temurin' }
      - run: ./mvnw verify   # fast: compile + test on JVM

  native:
    runs-on: ubuntu-latest
    needs: test
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: graalvm/setup-graalvm@v1
        with:
          java-version: '21'
          distribution: 'graalvm'
          native-image-job-reports: 'true'
      - run: ./mvnw -Pnative native:compile -DskipTests
      - run: ./mvnw -Pnative spring-boot:build-image -DskipTests
      - run: docker push myregistry/myapp:${{ github.sha }}

The native-image-job-reports: 'true' flag in the GraalVM action gives you a detailed report in the GitHub Actions summary showing which classes were included, missing hints, and build time breakdown — useful when debugging failures.

Library Support in 2026

The ecosystem has largely caught up. Most common Spring Boot dependencies have native support via the GraalVM Reachability Metadata Repository:

Library	Native Support
Spring Data JPA / Hibernate	✅ Full
Spring Security	✅ Full
Spring Cache + Redis	✅ Full
Flyway / Liquibase	✅ Full
Micrometer + Prometheus	✅ Full
MapStruct	✅ Full
Lombok	✅ Full (compile-time only)
Jackson	✅ Full with hints
gRPC (grpc-spring-boot-starter)	✅ Full
Testcontainers	⚠️ Test-only, not in native binary
Mockito	❌ Not for production use
ByteBuddy (direct use)	⚠️ Limited

If a library you need isn't on the list, check if it provides its own native-image.properties or if there's a community contribution in the Reachability Metadata Repository. Alternatively, you can contribute hints upstream — the process is straightforward.

The Bottom Line

GraalVM Native Image + Spring Boot 3 is production-ready in 2026. The tooling has matured, the Spring AOT processor handles 80% of the work automatically, and most real-world Spring Boot apps can go native with a manageable amount of hint configuration.

The build time penalty (5–8 minutes) is real, but it belongs in CI — not in your dev loop. The payoff — 40x faster startup, 5x lower memory, and images under 100MB — is a genuine competitive advantage for teams running containerized services.

Start with one service. Run the GraalVM agent against your integration tests, commit the generated hints, add the native build job to CI. If it works (it probably will), you have a blueprint to apply to the rest of your stack.

The hardest part is getting through the first build error. After that, it's just plumbing.

10 Spring Boot Mistakes That Silently Kill Production Apps

Avaneesh Yadav — Mon, 31 Aug 2026 10:26:17 +0000

Most Spring Boot production incidents don't come from exotic bugs. They come from the same ten patterns, repeated across codebases, that work fine under light load and silently degrade as traffic grows.

This guide covers the anti-patterns that production Java systems hit most often — with the exact code that causes them, why they fail, and what to write instead.

1. @Transactional on Private Methods

The mistake:

@Service
public class OrderService {

    // This @Transactional does NOTHING.
    @Transactional
    private void processPayment(Order order) {
        // changes to order are NOT in a transaction
    }

    public void submitOrder(Order order) {
        processPayment(order); // direct call, bypasses proxy
    }
}

Why it fails: Spring's @Transactional works through a proxy that wraps the bean. When you call a private method — or any method from within the same class — you bypass the proxy and call the raw object directly. No proxy → no transaction. Exceptions don't roll back. Data writes commit even when they shouldn't.

This is the most common Spring transactional bug. It compiles and passes unit tests. It fails silently in production when a mid-transaction exception leaves the database in a partially-written state.

The fix:

@Service
@RequiredArgsConstructor
public class OrderService {

    private final PaymentService paymentService; // separate @Service

    public void submitOrder(Order order) {
        paymentService.processPayment(order); // goes through Spring proxy
    }
}

@Service
public class PaymentService {

    @Transactional // works because external call goes through proxy
    public void processPayment(Order order) { ... }
}

If refactoring to a separate class isn't practical, inject self:

@Service
public class OrderService {

    @Autowired
    private OrderService self; // Spring injects the proxied version

    public void submitOrder(Order order) {
        self.processPayment(order); // goes through proxy
    }

    @Transactional
    public void processPayment(Order order) { ... }
}

Catch it early: Enable Spring's proxy debug logging (logging.level.org.springframework.aop=DEBUG) in a staging environment and verify transactions are opening where expected.

2. The N+1 Query Problem

This is the single most common performance issue in JPA-based Spring Boot applications. A page that renders in 50ms under load testing collapses to 4 seconds in production because the dataset grew.

The mistake:

// Entity
@Entity
public class Order {
    @OneToMany(fetch = FetchType.LAZY)
    private List<LineItem> lineItems;
}

// Service
public List<OrderSummary> getOrders() {
    List<Order> orders = orderRepository.findAll(); // 1 query
    return orders.stream()
        .map(o -> new OrderSummary(o, o.getLineItems().size())) // N queries
        .toList();
}

For 200 orders: 1 query to fetch orders + 200 queries to fetch each order's line items = 201 queries. At 1000 orders it's 1001 queries. Most ORMs make this invisible — the queries happen lazily, inside the stream, with no stack frame pointing to the cause.

How to detect it: Enable query logging:

spring:
  jpa:
    show-sql: true
    properties:
      hibernate:
        format_sql: true
logging:
  level:
    org.hibernate.SQL: DEBUG
    org.hibernate.orm.jdbc.bind: TRACE

If you see the same query repeated in a loop with different ID parameters, it's N+1.

The fix — JOIN FETCH:

public interface OrderRepository extends JpaRepository<Order, Long> {

    @Query("SELECT o FROM Order o LEFT JOIN FETCH o.lineItems WHERE o.status = :status")
    List<Order> findByStatusWithItems(@Param("status") OrderStatus status);
}

One query. Fetches orders and their line items in a single JOIN.

The fix — Projections (when you only need some fields):

public interface OrderSummaryView {
    String getOrderId();
    BigDecimal getTotal();
    int getItemCount();
}

public interface OrderRepository extends JpaRepository<Order, Long> {
    List<OrderSummaryView> findByStatus(OrderStatus status);
}

Projections let Spring Data generate a SELECT with only the columns you need — no entity materialization, no lazy loading risk.

The fix — @EntityGraph:

@EntityGraph(attributePaths = {"lineItems", "lineItems.product"})
List<Order> findAll();

Declarative JOIN FETCH without a JPQL query. Cleaner for simple cases.

3. Loading Entities for Count/Existence Checks

// Loads the entire entity from DB just to check if it exists
if (orderRepository.findById(orderId).isPresent()) {
    // ...
}

// Loads all orders just to count them
int count = orderService.findAll().size();

Both are SELECT * queries that transfer full row data across the network, materialize Java objects in heap, then throw most of it away.

The fix:

// Existence — single indexed column lookup, no data transfer
if (orderRepository.existsById(orderId)) { ... }

// Count — aggregation runs on the DB side
long count = orderRepository.count();

// Conditional count
long pendingCount = orderRepository.countByStatus(OrderStatus.PENDING);

For complex conditions:

@Query("SELECT COUNT(o) FROM Order o WHERE o.status = :status AND o.createdAt > :since")
long countRecentByStatus(@Param("status") OrderStatus status, @Param("since") LocalDateTime since);

4. Misconfigured Hikari Connection Pool

HikariCP ships with maximum-pool-size=10. Under low load this is invisible. Under real traffic it causes connection wait times that cascade into request timeouts.

# Default — often wrong for production
spring:
  datasource:
    hikari:
      maximum-pool-size: 10  # too low for most services

Symptoms: Requests succeed but take 200-500ms longer than expected. Actuator metrics show hikaricp.connections.pending > 0 regularly. Logs show HikariPool - Connection is not available, request timed out.

The fix — size for your thread model:

spring:
  datasource:
    hikari:
      # For virtual threads: connections = (number of cores * 2) is a starting point
      # For platform threads: connections = thread pool size
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 3000    # fail fast — 3s, not the default 30s
      idle-timeout: 600000        # 10 min
      max-lifetime: 1800000       # 30 min — rotate before DB kills idle connections
      keepalive-time: 60000       # prevent firewall from killing idle connections

Expose pool metrics:

management:
  metrics:
    enable:
      hikaricp: true

Then monitor hikaricp.connections.pending and hikaricp.connections.acquire p99. If pending > 0 for more than brief spikes, increase the pool.

[!NOTE]
With virtual threads (Spring Boot 3.2+, spring.threads.virtual.enabled=true), the optimal pool size is lower than with platform threads — virtual threads park during I/O so fewer connections handle more concurrent requests. Start at 2× core count and tune from metrics.

5. @Transactional(readOnly = true) Left Off Read Queries

// Missing readOnly — Spring opens a full R/W transaction
@Transactional
public List<Order> findRecentOrders(LocalDate since) {
    return orderRepository.findByCreatedAtAfter(since.atStartOfDay());
}

readOnly = true on a read-only transaction:

Tells Hibernate to skip dirty checking (no snapshot comparison at flush time)
Lets the JPA provider skip the write-ahead log
Some databases route read-only transactions to read replicas automatically

The fix:

@Transactional(readOnly = true)
public List<Order> findRecentOrders(LocalDate since) {
    return orderRepository.findByCreatedAtAfter(since.atStartOfDay());
}

Low effort, measurable improvement under read-heavy load. On services with 80% read traffic, this alone reduces database write-lock contention.

6. Missing Database Indexes on JPA Foreign Keys

JPA doesn't automatically create indexes on foreign key columns. It creates the constraint, not the index. This causes full table scans on every JOIN against a large table.

@Entity
public class LineItem {
    @ManyToOne
    @JoinColumn(name = "order_id")  // creates FK constraint, NOT an index
    private Order order;
}

Finding all line items for an order does SELECT * FROM line_item WHERE order_id = ?. Without an index on order_id, this is a full scan of the entire line_item table.

The fix — declare the index explicitly:

@Entity
@Table(name = "line_item",
    indexes = @Index(name = "idx_line_item_order_id", columnList = "order_id"))
public class LineItem {
    @ManyToOne
    @JoinColumn(name = "order_id")
    private Order order;
}

Or via Liquibase/Flyway (preferred for production — schema migrations in code):

-- V3__add_line_item_order_index.sql
CREATE INDEX idx_line_item_order_id ON line_item(order_id);

Finding missing indexes: Run EXPLAIN ANALYZE on your slowest queries. Any Seq Scan on a large table that filters by a FK column is a missing index candidate.

7. Leaking Hibernate Sessions Into Jackson Serialization

@Entity
public class Order {
    @OneToMany(fetch = FetchType.LAZY)
    private List<LineItem> lineItems;
}

// Controller — Open Session in View is ON by default
@GetMapping("/orders/{id}")
public Order getOrder(@PathVariable Long id) {
    return orderRepository.findById(id).orElseThrow();
    // Jackson serializes Order → touches lineItems → Hibernate loads lazily
    // If session is closed: LazyInitializationException
    // If session is open (OSIV): unexpected extra query per request
}

Spring Boot enables Open Session in View (OSIV) by default. This keeps the Hibernate session open through the entire request, including serialization. This means:

Lazy collections load silently during JSON serialization (N+1 hidden in Jackson)
Session-per-request model doesn't compose with virtual threads
Database connections held for the full request lifecycle, not just the DB operation

The fix — disable OSIV, use DTOs:

spring:
  jpa:
    open-in-view: false  # disable OSIV

// Fetch everything you need while the session is open
@Transactional(readOnly = true)
public OrderResponse getOrder(Long id) {
    Order order = orderRepository.findByIdWithItems(id)
        .orElseThrow(() -> new OrderNotFoundException(id));

    // Map to a DTO — no Hibernate proxy, safe to serialize
    return OrderResponse.from(order);
}

public record OrderResponse(String orderId, BigDecimal total, List<LineItemResponse> items) {
    static OrderResponse from(Order o) {
        return new OrderResponse(
            o.getId().toString(),
            o.getTotal(),
            o.getLineItems().stream().map(LineItemResponse::from).toList()
        );
    }
}

8. Unbounded @async Thread Pools

@Configuration
@EnableAsync
public class AsyncConfig implements AsyncConfigurer {
    // No custom config — Spring defaults to SimpleAsyncTaskExecutor
    // SimpleAsyncTaskExecutor creates a new thread per task, no limit
}

@Service
public class NotificationService {
    @Async
    public void sendEmail(String to, String subject) {
        // ...
    }
}

SimpleAsyncTaskExecutor (the default when no pool is configured) creates a new thread per task with no upper bound. Under sustained load this spins up thousands of threads and OOMs the service.

The fix — explicit bounded pool:

@Configuration
@EnableAsync
public class AsyncConfig implements AsyncConfigurer {

    @Override
    public Executor getAsyncExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(4);
        executor.setMaxPoolSize(20);
        executor.setQueueCapacity(500);       // tasks queue before rejecting
        executor.setThreadNamePrefix("async-");
        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
        executor.initialize();
        return executor;
    }
}

CallerRunsPolicy is usually the right rejection handler for background tasks — it runs the task on the caller's thread rather than throwing. This creates natural backpressure: if the async pool is saturated, the calling thread slows down instead of the service crashing.

For virtual threads (Java 21+):

@Bean
public Executor asyncExecutor() {
    return Executors.newVirtualThreadPerTaskExecutor(); // unbounded, but virtual threads are cheap
}

Virtual thread executors are safe to make unbounded because virtual threads don't block OS threads during I/O and cost ~1KB each.

9. Exception Swallowing in Background Jobs

@Component
public class ReconciliationJob {

    @Scheduled(cron = "0 0 2 * * *")
    public void run() {
        try {
            reconcile();
        } catch (Exception e) {
            log.error("Reconciliation failed", e);
            // returns normally — scheduler sees success
            // no alert, no retry, no escalation
        }
    }
}

The scheduler sees this method return without exception and marks it successful. The error is in the log. Without a log alert configured, the failure is invisible. Reconciliation silently stops running.

This pattern — catch, log, swallow — is how data inconsistencies accumulate over months without anyone noticing.

The fix — let exceptions propagate, or use structured alerting:

@Scheduled(cron = "0 0 2 * * *")
public void run() {
    try {
        reconcile();
    } catch (Exception e) {
        // Send an alert before rethrowing
        alertService.sendAlert("Reconciliation failed: " + e.getMessage(), AlertSeverity.HIGH);
        meterRegistry.counter("job.reconciliation.failure").increment();
        throw new RuntimeException("Scheduled reconciliation failed", e);
        // Spring will log this as an error — also visible in Actuator health
    }
}

Or better — use Temporal or Spring Batch for jobs that need fault tolerance and visibility (see the Temporal guide).

10. Logging in the Hot Path With String Concatenation

// Evaluated on every call, even when DEBUG logging is off
log.debug("Processing order " + order.getId() + " for customer " + customer.getName());

+ string concatenation creates intermediate String objects on every call. At 10,000 requests/minute, even a disabled DEBUG statement that runs on every request produces garbage that GC must collect, adding latency spikes.

The fix — parameterized logging:

// SLF4J defers evaluation — no string created if DEBUG is off
log.debug("Processing order {} for customer {}", order.getId(), customer.getName());

The {} placeholders are only evaluated if the logger is actually at DEBUG level. Zero allocation in production where DEBUG is off.

For complex log objects that are expensive to produce:

// Supplier form — only called if DEBUG is on
log.debug("Order state: {}", () -> computeExpensiveOrderDump(order));

The Production Checklist

Before a Spring Boot service goes to production, run through this:

[ ] Grep for @Transactional on private methods — fix any found
[ ] Run queries against staging data at 10× production volume — look for N+1 patterns in query logs
[ ] Disable OSIV (spring.jpa.open-in-view: false) and fix resulting LazyInitializationExceptions
[ ] Run EXPLAIN ANALYZE on your top 10 slowest queries — add missing indexes
[ ] Add @Transactional(readOnly = true) to all read-only service methods
[ ] Verify Hikari pool size matches your threading model — expose pool metrics in Actuator
[ ] Configure a named, bounded ThreadPoolTaskExecutor for @Async tasks
[ ] Verify all @Scheduled jobs either let exceptions propagate or send alerts before swallowing
[ ] Replace log.debug("text" + var) with parameterized log.debug("text {}", var) in hot paths
[ ] Set management.health.db.enabled=true and test liveness/readiness probes

graph LR
    A[Spring Boot Service] --> B{Code Review}
    B --> C[Transaction Check]
    B --> D[Query Analysis]
    B --> E[Pool Config]
    B --> F[Async Config]
    C --> G[No private @Transactional]
    D --> H[No N+1 — JOINs or projections]
    D --> I[Indexes on FK columns]
    E --> J[Bounded Hikari pool]
    E --> K[Pool metrics exposed]
    F --> L[Bounded executor configured]
    G --> M[Production Ready]
    H --> M
    I --> M
    J --> M
    K --> M
    L --> M

Catching These Before Production

Static analysis. SpotBugs + find-sec-bugs catches some of these (including the @Transactional private method issue). Add it to your Maven/Gradle build.

Query logging in staging. Enable org.hibernate.SQL: DEBUG in your staging environment against a dataset that's at least 10% of production volume. Any N+1 pattern shows up as repeated identical queries.

Load testing. k6, Gatling, or even ab at 2× expected peak. Watch HikariCP's connections.pending, Hibernate's queries.execution p99, and JVM GC pause time. Spikes in any of these under load point to one of the issues above.

Production metrics. Every Spring Boot 3 service should export to an observability backend. Dashboards to have:

DB connection pool saturation (hikaricp.connections.pending > 0)
Slow query p99 (> 100ms threshold)
GC pause duration (> 50ms threshold)
Heap utilization trend (monotonic growth = memory leak)

These patterns aren't obscure. They appear in production codebases at companies of every size, written by engineers who knew the frameworks well. The difference between a service that holds up under load and one that requires an incident at 2 AM is usually caught before deployment, not after.

Spotted one of these in your codebase? Or have a production anti-pattern that isn't on this list? Find me on LinkedIn — the list grows from war stories.

Java Developer Roadmap 2026: What to Learn, What to Skip

Avaneesh Yadav — Mon, 31 Aug 2026 09:26:30 +0000

Java is having its most interesting decade in twenty years. Virtual threads changed the concurrency model. Records and sealed classes modernized the type system. Pattern matching made the language feel genuinely expressive. And Spring AI put enterprise Java at the center of the AI engineering conversation.

If you're a Java developer trying to figure out what actually matters to learn in 2026 — and what to stop spending time on — this is the guide. No paid course recommendations. No sponsored rankings. Just what's worth your time.

The Big Picture: What Changed

graph TD
    A[Java Developer 2026] --> B[Core Language]
    A --> C[Concurrency]
    A --> D[Frameworks]
    A --> E[AI Integration]
    A --> F[Cloud Native]

    B --> B1[Java 21-24 Features]
    B --> B2[Records and Sealed Classes]
    B --> B3[Pattern Matching]

    C --> C1[Virtual Threads]
    C --> C2[Structured Concurrency]
    C --> C3[Scoped Values]

    D --> D1[Spring Boot 3.x]
    D --> D2[Spring Security 6]
    D --> D3[Spring Data]

    E --> E1[Spring AI]
    E --> E2[LangChain4j]
    E --> E3[RAG Patterns]

    F --> F1[Docker and Kubernetes]
    F --> F2[GraalVM Native]
    F --> F3[OpenTelemetry]

1. Core Java — The Non-Negotiables

Java 21 was the LTS that mattered. Java 24 refined it further. Here's what every Java developer needs to know cold in 2026.

Records (Java 16+, now essential)

// Before: 40-line POJO with constructor, getters, equals, hashCode, toString
// After:
public record OrderSummary(String orderId, BigDecimal total, OrderStatus status) {}

// Pattern matching in switch — clean, exhaustive
String describe(OrderSummary order) {
    return switch (order.status()) {
        case PENDING  -> "Awaiting payment";
        case PAID     -> "Processing — total: " + order.total();
        case SHIPPED  -> "On the way";
        case DELIVERED -> "Delivered";
    };
}

Records are not just syntax sugar. They signal immutability, make DTOs and value objects composable, and pair naturally with sealed classes for algebraic data types.

Sealed Classes + Pattern Matching

public sealed interface PaymentResult
    permits PaymentResult.Success, PaymentResult.Failure, PaymentResult.Pending {}

record Success(String transactionId, BigDecimal amount) implements PaymentResult {}
record Failure(String reason, ErrorCode code)          implements PaymentResult {}
record Pending(String referenceId)                     implements PaymentResult {}

// Exhaustive switch — compiler catches missing cases
String handle(PaymentResult result) {
    return switch (result) {
        case Success s  -> "Paid: " + s.transactionId();
        case Failure f  -> "Failed: " + f.reason();
        case Pending p  -> "Pending: " + p.referenceId();
    };
}

This eliminates entire categories of instanceof chains and ClassCastException risks.

Text Blocks

// Stop escaping quotes in SQL, JSON, GraphQL queries
String query = """
    SELECT o.id, o.total, c.email
    FROM orders o
    JOIN customers c ON o.customer_id = c.id
    WHERE o.status = 'PAID'
      AND o.created_at > :since
    ORDER BY o.created_at DESC
    """;

What to skip: Java 8 streams chaining for everything

Stream.of(...).filter(...).map(...).collect(...) is fine. Chaining 12 operators for a 3-line for-loop is not. Java 16+ toList(), records, and the enhanced for-loop often produce cleaner code than complex stream pipelines. Don't mistake verbosity for sophistication.

2. Concurrency — The Biggest Shift in Java History

Virtual threads (GA in Java 21) changed the fundamental model. If you're still writing async/reactive code for I/O-bound services, you're fighting the platform.

Virtual Threads

// Enable in Spring Boot — one line
spring.threads.virtual.enabled=true

// Now every request handler runs on a virtual thread
// Write blocking code. The JVM parks the thread during I/O.
@GetMapping("/orders/{id}")
public Order getOrder(@PathVariable String id) {
    Order order = orderRepository.findById(id)       // blocks, JVM parks virtual thread
        .orElseThrow(() -> new OrderNotFoundException(id));
    List<LineItem> items = itemRepository.findByOrderId(id); // same
    return enrich(order, items);                     // same
}
// No Mono, no CompletableFuture, no callback hell.

The JVM creates virtual threads cheaply (nanoseconds, ~1KB stack). You can have millions. I/O waits don't block the carrier thread. The performance profile of reactive code with the readability of blocking code.

Structured Concurrency (GA in Java 24)

For when you genuinely need to run things in parallel:

// Fan-out: fetch user + orders concurrently, both must succeed
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
    Subtask<User>        user   = scope.fork(() -> userService.find(userId));
    Subtask<List<Order>> orders = scope.fork(() -> orderService.findByUser(userId));

    scope.join().throwIfFailed();    // waits for both, cancels if either fails
    return new Dashboard(user.get(), orders.get());
}

This replaces the fragile CompletableFuture.allOf(...) pattern. Lifetimes are scoped — subtasks cannot outlive the block. Cancellation propagates correctly. Stack traces are readable.

Scoped Values (Java 24)

// Replace ThreadLocal — works correctly across virtual threads
static final ScopedValue<RequestContext> REQUEST_CTX = ScopedValue.newInstance();

// At request entry point:
ScopedValue.where(REQUEST_CTX, new RequestContext(traceId, userId))
    .run(() -> processRequest(req));

// Anywhere in the call tree:
String traceId = REQUEST_CTX.get().traceId();

ThreadLocal still works but doesn't compose well with virtual threads (pinning risk). Use ScopedValue for request context in new code.

What to skip: Project Reactor / WebFlux for request-response services

WebFlux is the right tool for SSE, WebSocket, and genuine streaming workloads. For CRUD microservices and REST APIs, virtual threads deliver equal throughput with a fraction of the complexity. The ecosystem (blocking JDBC, JPA, most third-party SDKs) is also blocking — fighting it reactively adds overhead without benefit.

3. Spring Boot 3.x — The Framework Layer

Spring Boot 3 requires Java 17+ and brought a full Jakarta EE migration. By 2026, if you're still on Spring Boot 2.x, this is the year to move.

What matters most

AOT and GraalVM Native. Spring Boot 3 ships native compilation support. For short-lived workloads (FaaS, CLIs, Lambda) native images start in milliseconds and use a fraction of the RAM.

<plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
</plugin>

mvn -Pnative native:compile
./target/myapp   # starts in ~50ms, uses ~50MB RAM

Spring Security 6. The WebSecurityConfigurerAdapter is gone. The new lambda DSL is cleaner:

@Bean
SecurityFilterChain security(HttpSecurity http) throws Exception {
    return http
        .authorizeHttpRequests(auth -> auth
            .requestMatchers("/actuator/health").permitAll()
            .requestMatchers("/admin/**").hasRole("ADMIN")
            .anyRequest().authenticated())
        .oauth2ResourceServer(oauth2 -> oauth2.jwt(Customizer.withDefaults()))
        .build();
}

Spring Data 3.x. PagingAndSortingRepository is now separate from CrudRepository. JpaRepository extends both. Projections with interfaces or records are the cleanest way to fetch partial data.

Actuator + Micrometer + OpenTelemetry. This stack is the observability standard. Every Spring Boot 3 service should export traces and metrics to your observability backend via OTLP.

management:
  tracing:
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://otel-collector:4318/v1/traces

4. AI Integration — The New Differentiator

Java developers who can wire LLMs into production systems are in the top 5% of the market right now. This is the highest-leverage skill to add in 2026.

Spring AI (the production choice)

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
</dependency>

@Service
@RequiredArgsConstructor
public class DocumentAnalysisService {

    private final ChatClient chatClient;
    private final VectorStore vectorStore;

    // RAG: embed query, retrieve context, generate answer
    public String answer(String question) {
        List<Document> context = vectorStore.similaritySearch(
            SearchRequest.query(question).withTopK(4)
        );

        String contextText = context.stream()
            .map(Document::getContent)
            .collect(Collectors.joining("\n---\n"));

        return chatClient.prompt()
            .system("Answer using only the context provided. If the answer is not there, say so.")
            .user(u -> u.text("Context:\n{ctx}\n\nQuestion: {q}")
                        .param("ctx", contextText)
                        .param("q", question))
            .call()
            .content();
    }
}

Key Spring AI concepts to learn in order

ChatClient — the main abstraction for LLM calls
Structured outputs — returning Java POJOs instead of raw strings
VectorStore + EmbeddingModel — RAG foundation
Tool/Function calling — letting LLMs call your Java methods
ChatMemory + Advisors — conversation state, prompt composition
Observability — tracing LLM calls with Micrometer

What to skip: Building your own LLM abstraction layer

LangChain4j and Spring AI cover 95% of production AI use cases. Rolling a custom HTTP client to the OpenAI API is fine for learning but builds a maintenance liability you'll regret in six months.

5. Cloud-Native — Table Stakes

If you're building services in 2026, cloud-native patterns are not optional.

Docker and Kubernetes (know the basics)

FROM eclipse-temurin:21-jre-alpine
COPY target/app.jar /app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app.jar"]

# Kubernetes readiness + liveness via Spring Actuator
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080

You don't need to be a Kubernetes expert. You do need to know: how to containerize an app, what readiness/liveness probes do, how ConfigMaps and Secrets map to env vars, and how HPA works.

OpenTelemetry — the observability standard

// With Spring Boot 3 Actuator + OTLP exporter, traces are automatic.
// Add baggage for business context:
@GetMapping("/orders/{id}")
public Order getOrder(@PathVariable String id) {
    Span.current().setAttribute("order.id", id);
    // ... rest of handler
}

Logs, traces, and metrics in one collector. Stop learning proprietary agents for each APM vendor.

What to skip: Manual Dockerfiles with fat JARs on JDK images

eclipse-temurin:21-jre-alpine (not JDK) cuts image size by ~300MB. Spring Boot's layered JAR support (spring-boot-maven-plugin layer configuration) and Paketo Buildpacks produce better images than handwritten Dockerfiles for most teams. Use buildpacks first, hand-roll only when you have a specific reason.

Learning Path by Experience Level

0–2 Years (Junior)

Focus here, in this order:

Java 21 core — records, sealed classes, pattern matching, text blocks
Spring Boot 3 basics — REST, JPA, validation, security fundamentals
Testing — JUnit 5, Mockito, @SpringBootTest vs slice tests
Git + Docker basics
SQL — JOINs, indexes, EXPLAIN — know your database

Skip for now: Reactive programming, microservices architecture, AI integration, Kubernetes internals. Complexity before fundamentals creates engineers who can't debug.

2–5 Years (Mid-level)

Virtual threads and structured concurrency — replace your thread pool mental model
Spring Boot 3 production patterns — Actuator, Micrometer, OpenTelemetry
Microservices fundamentals — API design, inter-service communication (REST vs gRPC), circuit breakers
Cloud basics — one of AWS/GCP/Azure to associate level
Spring AI basics — RAG pipeline, tool calling, structured outputs

The leverage move at this level: observability. Engineers who can find production issues fast are disproportionately valuable.

5+ Years (Senior / Lead)

AI system design — RAG architecture, agent orchestration, LLMOps
Performance engineering — virtual thread profiling, JVM GC tuning, DB query optimization
Platform engineering — Kubernetes operators, Helm, GitOps
Architecture patterns — event-driven design, CQRS, saga
Engineering leadership — technical direction, system design interviews, code review culture

The leverage move at this level: mentorship. The engineers you bring up multiply your output more than any technical skill.

The "What to Stop Doing" List

This is the section people argue about. That's the point.

Stop doing	Do this instead
`@Autowired` field injection	Constructor injection — testable, explicit
`extends HttpServlet` / raw servlets	Spring MVC or Spring WebFlux
XML Spring config	Java config + `@Bean`
`java.util.Date` / `Calendar`	`java.time.*` — `LocalDate`, `ZonedDateTime`
Manual JSON parsing with `JSONObject`	Jackson / Gson with typed POJOs
Mutable POJOs with 30 setters	Records or immutable value objects
`synchronized` blocks for everything	`ReentrantLock`, `Semaphore`, virtual threads
`ThreadLocal` for request context	`ScopedValue` (Java 24+)
Catching `Exception` everywhere	Specific exceptions; let runtime exceptions propagate
Log.info("Got order: " + orderId)	`log.info("Got order: {}", orderId)` — no string concat
Building your own retry logic	Resilience4j or Spring Retry
Rolling your own JWT validation	Spring Security's OAuth2 resource server

The Skills That Actually Get You Hired in 2026

In order of what hiring managers actually test for:

System design — can you design a scalable, observable microservice under ambiguous requirements?
Debugging — can you find a production issue using logs, traces, and metrics without stepping through code?
Code quality — do your PRs need one review cycle or five?
AI integration — can you wire an LLM into a service that works reliably in production (not just a demo)?
Communication — can you explain a technical decision to a non-technical stakeholder?

Leetcode matters for FAANG and trading companies. For the vast majority of backend Java roles, system design and real-world debugging matter more.

Resources Worth Your Time

Language:

JEP Index — primary source for new Java features, better than any blog summary
Inside Java podcast — authoritative, from the Java team

Spring:

Spring Blog — official, high signal
Spring One talks on YouTube — architecture deep-dives by the framework authors

AI / LLM:

Spring AI reference docs — the most up-to-date source for Spring AI patterns
AI Engineering — practical guides on RAG, agents, LLMOps in Java

System Design:

Designing Data-Intensive Applications (Kleppmann) — still the best book on distributed systems
ByteByteGo newsletter — good visual breakdowns of production architectures

One More Thing

Java in 2026 is not your Java of 2015. It's fast, expressive, and the ecosystem for AI integration is mature. The engineers who treat it as the "boring enterprise language" are leaving real opportunity on the table.

The engineers who will be most valuable in the next three years are the ones who can build reliable AI-powered systems in Java — because that's where the enterprise deals are, and that's where the Java ecosystem is heading.

Pick one section of this roadmap that you're weakest in. Work on it for 90 days. Then come back to the next one.

Disagreements welcome — that's how roadmaps improve. Find me on LinkedIn or browse the full buildingai.in blog for deep-dives on each section of this roadmap.

Temporal vs Spring Batch: Fixing Enterprise Java's Silent Job Failure Problem

Avaneesh Yadav — Fri, 28 Aug 2026 06:26:08 +0000

Enterprise Java teams commonly reach a point where scheduled jobs become a liability rather than an asset. Jobs fail silently — catching exceptions, logging nothing meaningful, and returning success to the scheduler. Others run simultaneously because the previous execution didn't finish before the next trigger fired and nobody configured @ScheduledLock. Data corruption from concurrent runs can take weeks to surface.

None of this shows up in dashboards until a data audit flags inconsistencies in a critical report.

This guide explains why @Scheduled breaks down at scale, how Temporal's durable execution model eliminates these problems, and how to migrate a production Spring Batch pipeline with complete code examples.

What Temporal Actually Is (In One Paragraph)

Temporal is a durable execution platform. You write plain Java code — functions, loops, try/catch — and Temporal makes it fault-tolerant by recording every state transition to an event log. If the process crashes mid-execution, it replays the log and resumes from exactly where it left off. There's no external state machine to define, no checkpoint tables to maintain, no retry logic to write. The code IS the workflow.

graph TD
  subgraph BEFORE ["Before — Spring Scheduler + Batch"]
    C1[Cron Trigger] --> J1[@Scheduled Method]
    J1 --> J2[Spring Batch Job]
    J2 --> J3[ItemReader]
    J3 --> J4[ItemProcessor]
    J4 --> J5[ItemWriter]
    J5 --> J6[DB commit]
    E1[Failure?] -.->|swallowed| X1[😶 Silent]
    E2[Overlap?] -.->|no guard| X2[💥 Corruption]
  end

  subgraph AFTER ["After — Temporal"]
    T1[Temporal Scheduler] --> W1[Workflow Method]
    W1 --> A1[Activity: Read chunk]
    W1 --> A2[Activity: Process chunk]
    W1 --> A3[Activity: Write chunk]
    F1[Failure?] -.->|auto-retry| W1
    F2[Crash?] -.->|replay from log| W1
    F3[Overlap?] -.->|workflow ID lock| X3[✅ Prevented]
  end

The Problem With @Scheduled at Scale

@Scheduled works fine for one or two simple jobs. By the time a team has dozens, accumulated debt becomes significant:

No visibility. Which jobs ran? Which failed? How long did they take? Spring's scheduler offers nothing here by default. Teams bolt on Actuator, add custom logging, hook up Micrometer. By the time there's real observability it's a custom framework to maintain.

No fault tolerance. An exception kills the job instance. Whether it retries, and how, is the team's problem. Teams solve this inconsistently — some catch-and-retry inline, some use Spring Retry, some let it silently fail and rely on the next scheduled trigger.

No distributed locking. @ScheduledLock with ShedLock or Quartz clustering works, but it's an additional library, additional config, and an additional failure mode (what happens when the lock row gets corrupted?).

Testing is painful. Unit tests can cover job logic, but testing scheduling behavior — does it actually retry? does it respect the lock? — requires either waiting for real time to pass or mocking the scheduler in ways that diverge from production behavior.

Spring Batch adds its own complexity. Job metadata tables (BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, etc.) need to be managed, pruned, and kept consistent. The restart/skip/retry model is powerful but verbose to configure. And Spring Batch has no concept of a workflow spanning multiple jobs — that becomes custom orchestration code.

Before: A Typical Spring Batch Job

Here's a representative pattern — daily invoice reconciliation, reading from a pending_invoices table, processing, and writing to processed_invoices:

// Cron trigger — buried in application.yml
// cron: "0 0 2 * * *"

@Component
@RequiredArgsConstructor
public class InvoiceReconciliationScheduler {

    private final JobLauncher jobLauncher;
    private final Job invoiceReconciliationJob;

    @Scheduled(cron = "${jobs.invoice-reconciliation.cron}")
    @SchedulerLock(name = "invoiceReconciliation", lockAtMostFor = "PT2H")
    public void run() {
        try {
            JobParameters params = new JobParametersBuilder()
                .addLong("run.id", System.currentTimeMillis())
                .toJobParameters();
            jobLauncher.run(invoiceReconciliationJob, params);
        } catch (Exception e) {
            log.error("Invoice reconciliation failed", e);
            // No alert. No retry. No escalation.
        }
    }
}

@Configuration
@RequiredArgsConstructor
public class InvoiceReconciliationJobConfig {

    @Bean
    public Job invoiceReconciliationJob(Step reconcileStep) {
        return jobBuilderFactory.get("invoiceReconciliationJob")
            .start(reconcileStep)
            .build();
    }

    @Bean
    public Step reconcileStep() {
        return stepBuilderFactory.get("reconcileStep")
            .<PendingInvoice, ProcessedInvoice>chunk(500)
            .reader(invoiceReader())
            .processor(invoiceProcessor())
            .writer(invoiceWriter())
            .faultTolerant()
            .retryLimit(3)
            .retry(TransientDataAccessException.class)
            .build();
    }
    // ... reader/processor/writer beans
}

This is roughly 150 lines across three files for one job. It has per-item retry (good), but no workflow-level retry, no timeout enforcement, no alerting on failure, and no way to see execution history without querying the batch tables directly.

After: The Same Job in Temporal

// Workflow interface — the contract
@WorkflowInterface
public interface InvoiceReconciliationWorkflow {
    @WorkflowMethod
    void reconcile(LocalDate date);
}

// Activity interface — the actual I/O work
@ActivityInterface
public interface InvoiceReconciliationActivities {
    List<PendingInvoice> readChunk(LocalDate date, int offset, int limit);
    List<ProcessedInvoice> processChunk(List<PendingInvoice> invoices);
    void writeChunk(List<ProcessedInvoice> processed);
    int countPending(LocalDate date);
}

// Workflow implementation — plain Java, no framework magic
public class InvoiceReconciliationWorkflowImpl implements InvoiceReconciliationWorkflow {

    private final InvoiceReconciliationActivities activities =
        Workflow.newActivityStub(InvoiceReconciliationActivities.class,
            ActivityOptions.newBuilder()
                .setStartToCloseTimeout(Duration.ofMinutes(10))
                .setRetryOptions(RetryOptions.newBuilder()
                    .setMaximumAttempts(5)
                    .setDoNotRetry(IllegalArgumentException.class.getName())
                    .build())
                .build());

    @Override
    public void reconcile(LocalDate date) {
        int total = activities.countPending(date);
        int offset = 0;
        int chunkSize = 500;

        while (offset < total) {
            List<PendingInvoice> chunk = activities.readChunk(date, offset, chunkSize);
            List<ProcessedInvoice> processed = activities.processChunk(chunk);
            activities.writeChunk(processed);
            offset += chunkSize;
        }
    }
}

// Schedule registration — once, at startup
client.scheduleClient().createSchedule(
    "invoice-reconciliation-daily",
    ScheduleSpec.newBuilder()
        .setCronExpressions(List.of("0 2 * * *"))
        .build(),
    ScheduleAction.startWorkflow(
        InvoiceReconciliationWorkflow.class,
        "reconcile",
        WorkflowOptions.newBuilder()
            .setWorkflowId("invoice-recon-" + LocalDate.now())
            .setTaskQueue("invoice-queue")
            .setWorkflowExecutionTimeout(Duration.ofHours(4))
            .build())
);

The workflow body is 12 lines. It reads like the business requirement. No chunk configuration DSL, no step beans, no job parameter boilerplate.

Because the workflow ID includes the date (invoice-recon-2026-08-28), attempting to start a second execution for the same day throws WorkflowExecutionAlreadyStarted — the overlap corruption problem is gone by construction, with no lock to configure or expire.

What You Get for Free

Retries with backoff. Every activity gets the retry policy defined at registration. Failed activities retry automatically, with exponential backoff, without any code in the activity itself.

Crash recovery. If the worker process dies mid-workflow, the next worker that picks up the task queue replays the event history and continues from the last completed activity. No data is lost.

Visibility out of the box. Temporal's web UI shows every workflow execution: start time, current state, activity history, retry attempts, failures, input and output. What would take custom Micrometer instrumentation now comes for free.

Workflow ID deduplication. Using a business-meaningful workflow ID (like invoice-recon-2026-08-28) means the same logical job can never run twice. This is structurally better than distributed locking.

Testable without real time. Temporal's TestWorkflowEnvironment lets you test the entire workflow including retries, timeouts, and activity failures — in unit tests, without waiting for real timers.

@Test
void shouldRetryFailedActivity() {
    TestWorkflowEnvironment env = TestWorkflowEnvironment.newInstance();
    Worker worker = env.newWorker("invoice-queue");

    worker.registerActivitiesImplementations(new InvoiceActivitiesFailing(failOnAttempt: 1));
    worker.registerWorkflowImplementationTypes(InvoiceReconciliationWorkflowImpl.class);
    env.start();

    InvoiceReconciliationWorkflow wf = env.getWorkflowClient()
        .newWorkflowStub(InvoiceReconciliationWorkflow.class,
            WorkflowOptions.newBuilder().setTaskQueue("invoice-queue").build());

    // Runs in milliseconds — Temporal's test env skips real timers
    wf.reconcile(LocalDate.of(2026, 8, 28));

    verify(failingActivity, times(2)).readChunk(any(), anyInt(), anyInt()); // retried once
}

Common Migration Pitfalls

Activity timeouts. StartToCloseTimeout must be set conservatively — Temporal cancels and retries the activity if it exceeds the timeout. Set it to the P99 execution time, not the average. Activities that occasionally run for 45 minutes on large-backlog days need that reflected in their timeout.

Non-idempotent writers. Temporal retries activities automatically. If a writer inserts rows without an upsert, retries produce duplicates. Every write activity must be idempotent — ON CONFLICT DO UPDATE on the write side before migration.

History size limits. Temporal's event history has a default limit of 50,000 events. A workflow processing 500k records in 500-row chunks generates 3,000 activity calls — fine. Processing row-by-row would hit the limit. Design activities to operate on chunks, not individual records.

Local dev environment. Running Temporal locally requires Docker (temporalio/temporal). Teams without Docker in their standard dev setup will have friction on day one. Add it to the team devcontainer or docker-compose before rollout.

[!NOTE]
Temporal's Java SDK (version 2.x) works with Spring Boot 3.2+ virtual threads automatically. Pair it with spring.threads.virtual.enabled=true and worker threads scale to thousands of concurrent activity executions on minimal OS threads.

When to Keep Spring Batch

Not every job should move to Temporal. Spring Batch remains the right tool in two scenarios:

Regulated data processing with mandatory audit trails. Spring Batch's job metadata schema (BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION) is a ready-made audit log that compliance and operations teams can query directly in the database. Temporal's event history lives in Temporal's own store — visible in the UI, but not in your RDBMS. For financial reporting jobs with regulatory requirements, Spring Batch's schema is an asset.

Pure ETL with no orchestration complexity. A job that reads a file, transforms rows, writes to a table, and exits is exactly what Spring Batch was designed for. No retry logic, no dependencies between steps, no timeout sensitivity. Rewriting it in Temporal adds infrastructure without adding value.

Decision rule: if you would describe the job as "a workflow" — with branches, dependencies, or conditions — use Temporal. If you would describe it as "a batch" — fixed input, transform, fixed output — Spring Batch is the right fit.

Getting Started

<dependency>
    <groupId>io.temporal</groupId>
    <artifactId>temporal-spring-boot-starter</artifactId>
    <version>1.4.0</version>
</dependency>

# application.yml
spring:
  temporal:
    connection:
      target: localhost:7233
    workers:
      - task-queue: invoice-queue
        workflow-classes:
          - com.example.workflows.InvoiceReconciliationWorkflowImpl
        activity-beans:
          - invoiceReconciliationActivities

Temporal's Spring Boot starter auto-wires workers, injects WorkflowClient and ScheduleClient as beans, and handles graceful shutdown.

# Local dev — start Temporal server
docker run --rm -p 7233:7233 -p 8233:8233 temporalio/temporal:latest
# UI available at http://localhost:8233

Migration Checklist

[ ] Inventory all @Scheduled and Spring Batch jobs — map dependencies between them
[ ] Classify: "workflow" (Temporal) vs "pure batch" (keep Spring Batch)
[ ] Make all activity implementations idempotent (upsert, not insert)
[ ] Set StartToCloseTimeout at P99 of execution time, not average
[ ] Use business-meaningful workflow IDs for deduplication
[ ] Add Temporal Docker to dev environment / docker-compose
[ ] Run parallel execution for 2 weeks (old scheduler + Temporal) and compare outputs
[ ] Cut over the scheduler trigger, remove old @Scheduled beans
[ ] Configure Temporal Worker autoscaling (CPU-based HPA works well on Kubernetes)

Summary

@Scheduled is fine for one or two simple jobs. Beyond that, teams face silent failures, concurrent execution bugs, and zero visibility — problems that only surface when they've already caused damage.

Temporal solves all three at the infrastructure level: durable execution for crash recovery, workflow ID deduplication for concurrency control, and a built-in UI for visibility. The workflow code is shorter than Spring Batch equivalent, easier to read, and testable in isolation without mocking time.

For teams with more than 10 scheduled jobs, or any jobs with dependencies and retries, Temporal is worth evaluating before the next production incident makes the case for you.

Working with Temporal in Java or evaluating the migration? Happy to discuss patterns and pitfalls. Find me on LinkedIn.