DEV Community: sandeep chagalakonda

The Distributed Transaction Nightmare: Why @Transactional Silently Failed Across Microservices (And the Outbox Pattern Fix)

sandeep chagalakonda — Fri, 16 Oct 2026 10:22:15 +0000

Everything looks flawless in development. You decorate your Spring Boot service method with @Transactional, run an integration test against a local container, simulate a downstream failure, and verify that the database rolls back cleanly.

Then you deploy to production.

At scale, when network sockets hiccup, third-party payment gateways timeout, and message brokers drop TCP connections mid-flight, your database state begins drifting silently. Money gets deducted without inventory decrements. Orders get created in your database, but downstream notification services never hear about them.

Here is why Spring's @Transactional cannot protect distributed microservice architectures, the anatomy of the dreaded Dual-Write Problem, and the exact Transactional Outbox Pattern implementation in Spring Boot 3 and Java 21 that eliminates data drift forever.

The Localhost Illusion: What @Transactional Actually Does Spring's @Transactional is built around a single foundational primitive: a local relational database transaction managed via a single JDBC Connection.

Plaintext

[HTTP Request]
│
▼
┌──────────────────────────────────────────────┐
│ Spring @Transactional Boundary │
│ │
│ 1. conn.setAutoCommit(false) │
│ 2. INSERT INTO orders ... (Local DB) │
│ 3. restTemplate.postForObject(...) ──► [HTTP Downstream Service]
│ 4. conn.commit() │
└──────────────────────────────────────────────┘
When you wrap a method with @Transactional, Spring intercepts the call, grabs a physical connection from your connection pool (HikariCP), sets autoCommit = false, executes your JPA statements, and calls conn.commit() at the boundary exit.

If an unhandled RuntimeException triggers before line 4, Spring issues conn.rollback().

The Fatal Flaw Across Network Boundaries
Consider this typical order-placement method:

Java

@Service
public class OrderProcessingService {

private final OrderRepository orderRepository;
private final RestClient inventoryClient;
private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate;

@Transactional
public Order placeOrder(CreateOrderRequest request) {
    // Step 1: Save local record
    Order order = orderRepository.save(new Order(request));

    // Step 2: Call downstream service over HTTP
    inventoryClient.post()
        .uri("/api/inventory/reserve")
        .body(new ReserveInventoryRequest(order.getId(), request.items()))
        .retrieve()
        .toBstatus();

    // Step 3: Broadcast event to message broker
    kafkaTemplate.send("order.events", new OrderCreatedEvent(order));

    return order;
}

}
This method contains three distinct failure boundaries masquerading as one atomic transaction:

Downstream Timeout After Processing: What happens if inventoryClient successfully reserves stock in the inventory database, but a network blip severs the connection before the HTTP 200 OK reaches your server? The HTTP client throws a ResourceAccessException. Your @Transactional method catches the exception and rolls back the local Order entity. Result: Inventory is permanently locked downstream, but no order exists upstream.

Broker Network Partition: What happens if the database commit succeeds, but your Kafka broker encounters a transient leader re-election during kafkaTemplate.send()? If the broker call throws an exception, your database changes roll back—but if the message was already acknowledged, downstream consumers start processing an order that does not exist in your primary database.

The Dual-Write Race Condition: Even without errors, notice the order of operations: line 3 emits an event before the database transaction commits at line 4. A high-speed downstream consumer reading from Kafka can receive the message and query GET /orders/{id} before your local database transaction has finished committing, resulting in an immediate 404 Not Found.

Why Two-Phase Commit (2PC / XA) is a Trap Historically, distributed systems used Two-Phase Commit (2PC) with XA transactions to maintain atomicity across relational databases and message queues.

In modern cloud environments, 2PC is an anti-pattern:

Blocking Coordinator Protocol: If the transaction coordinator crashes during the PREPARE phase, locks on all participating database rows remain held indefinitely.

Latency Escalation: Every participant must flush state and wait on round-trip network calls before committing, multiplying API latency by 10x–50x.

Cloud Incompatibility: Most modern distributed datastores and message brokers (Kafka, AWS SQS, DynamoDB) do not even support the XA protocol.

To achieve scale and zero data drift, we must separate local persistence from event dispatching.

The Architecture: Transactional Outbox Pattern The Transactional Outbox Pattern ensures guaranteed at-least-once event delivery without distributed locks:

Plaintext

[Client] ──► [Order Service API]
│
▼ (Single Local ACID Transaction)
┌──────────────────────────────────────────────┐
│ 1. INSERT INTO orders (Business State) │
│ 2. INSERT INTO outbox_events (Event Payload) │
└──────────────────────────────────────────────┘
│
▼ (Commit)
[PostgreSQL Database]
│
┌───────────┴────────────────┐
│ │
▼ ▼
[Outbox Poller Service] OR [CDC Engine (Debezium)]
│ │
└──────────────┬─────────────┘
▼
[Apache Kafka / RabbitMQ]
│
▼
[Downstream Microservices]
Instead of sending network calls directly inside the business logic:

You insert the domain entity (Order) into the primary table.

In the exact same database transaction, you insert an event payload into a dedicated outbox_events table.

Because both inserts share a single JDBC connection, they succeed or fail atomically. Zero network calls occur inside the transaction.

An asynchronous worker (either a background polling scheduler or a Change Data Capture tool like Debezium) reads the outbox table and broadcasts the events to the broker.

Production Spring Boot 3 Implementation Step 1: The Outbox Database Schema SQL

CREATE TABLE outbox_events (
id UUID PRIMARY KEY,
aggregate_type VARCHAR(64) NOT NULL,
aggregate_id VARCHAR(64) NOT NULL,
event_type VARCHAR(128) NOT NULL,
payload JSONB NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'PENDING',
retry_count INT NOT NULL DEFAULT 0,
created_at TIMESTAMP WITH TIME ZONE NOT NULL,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL
);

CREATE INDEX idx_outbox_pending ON outbox_events (status, created_at)
WHERE status = 'PENDING';
Step 2: The Outbox Entity and JPA Repository
Java

package com.springforge.outbox.entity;

import jakarta.persistence.*;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;

import java.time.Instant;
import java.util.UUID;

@entity
@Table(name = "outbox_events")
@Getter
@setter
@NoArgsConstructor
public class OutboxEvent {

@Id
private UUID id;

@Column(nullable = false, length = 64)
private String aggregateType;

@Column(nullable = false, length = 64)
private String aggregateId;

@Column(nullable = false, length = 128)
private String eventType;

@JdbcTypeCode(SqlTypes.JSON)
@Column(columnDefinition = "jsonb", nullable = false)
private String payload;

@Enumerated(EnumType.STRING)
@Column(nullable = false, length = 32)
private OutboxStatus status;

@Column(nullable = false)
private int retryCount;

@Column(nullable = false)
private Instant createdAt;

@Column(nullable = false)
private Instant updatedAt;

public enum OutboxStatus {
    PENDING, PROCESSING, PUBLISHED, FAILED
}

public static OutboxEvent from(String aggregateType, String aggregateId, String eventType, String jsonPayload) {
    OutboxEvent event = new OutboxEvent();
    event.setId(UUID.randomUUID());
    event.setAggregateType(aggregateType);
    event.setAggregateId(aggregateId);
    event.setEventType(eventType);
    event.setPayload(jsonPayload);
    event.setStatus(OutboxStatus.PENDING);
    event.setRetryCount(0);
    event.setCreatedAt(Instant.now());
    event.setUpdatedAt(Instant.now());
    return event;
}

}
Java

package com.springforge.outbox.repository;

import com.springforge.outbox.entity.OutboxEvent;
import org.springframework.data.domain.Pageable;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.stereotype.Repository;

import java.util.List;
import java.util.UUID;

@Repository
public interface OutboxRepository extends JpaRepository {

@Query("SELECT o FROM OutboxEvent o WHERE o.status = 'PENDING' ORDER BY o.createdAt ASC")
List<OutboxEvent> findPendingEvents(Pageable pageable);

}
Step 3: Atomic Mutation in Business Service
The business service no longer touches Kafka or external HTTP endpoints. It writes both records in a clean 2ms database transaction:

Java

package com.springforge.outbox.service;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.springforge.outbox.entity.Order;
import com.springforge.outbox.entity.OutboxEvent;
import com.springforge.outbox.repository.OrderRepository;
import com.springforge.outbox.repository.OutboxRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
@RequiredArgsConstructor
@Slf4j
public class OrderService {

private final OrderRepository orderRepository;
private final OutboxRepository outboxRepository;
private final ObjectMapper objectMapper;

@Transactional
public Order createOrder(OrderRequest request) {
    // 1. Persist business state
    Order order = new Order(request.customerId(), request.totalAmount());
    Order savedOrder = orderRepository.save(order);

    // 2. Prepare event payload
    OrderCreatedEvent event = new OrderCreatedEvent(
        savedOrder.getId(),
        savedOrder.getCustomerId(),
        savedOrder.getTotalAmount(),
        savedOrder.getCreatedAt()
    );

    // 3. Persist outbox event in same ACID boundary
    try {
        String payload = objectMapper.writeValueAsString(event);
        OutboxEvent outboxEvent = OutboxEvent.from(
            "ORDER",
            savedOrder.getId().toString(),
            "ORDER_CREATED",
            payload
        );
        outboxRepository.save(outboxEvent);
    } catch (JsonProcessingException e) {
        log.error("Failed to serialize order event for order id: {}", savedOrder.getId(), e);
        throw new IllegalStateException("Event serialization failed", e);
    }

    // Both tables commit simultaneously
    return savedOrder;
}

}
Step 4: The Resilient Outbox Poller Worker
A dedicated scheduler periodically sweeps pending events, streams them to Kafka, and updates the state. If Kafka is down, the records stay safely stored in outbox_events until the broker recovers.

Java

package com.springforge.outbox.worker;

import com.springforge.outbox.entity.OutboxEvent;
import com.springforge.outbox.repository.OutboxRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.data.domain.PageRequest;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;

import java.time.Instant;
import java.util.List;

@Component
@RequiredArgsConstructor
@Slf4j
public class OutboxPublisherWorker {

private final OutboxRepository outboxRepository;
private final KafkaTemplate<String, String> kafkaTemplate;

@Scheduled(fixedDelayString = "${app.outbox.poll-interval-ms:1000}")
public void processOutbox() {
    List<OutboxEvent> pendingEvents = outboxRepository.findPendingEvents(PageRequest.of(0, 100));

    for (OutboxEvent event : pendingEvents) {
        publishSingleEvent(event);
    }
}

@Transactional(propagation = Propagation.REQUIRES_NEW)
public void publishSingleEvent(OutboxEvent event) {
    try {
        // Partition by aggregateId to preserve strict entity ordering in Kafka
        kafkaTemplate.send("order-events", event.getAggregateId(), event.getPayload())
            .whenComplete((result, ex) -> {
                if (ex == null) {
                    event.setStatus(OutboxEvent.OutboxStatus.PUBLISHED);
                    event.setUpdatedAt(Instant.now());
                    outboxRepository.save(event);
                } else {
                    handlePublishFailure(event, ex);
                }
            });
    } catch (Exception ex) {
        handlePublishFailure(event, ex);
    }
}

private void handlePublishFailure(OutboxEvent event, Throwable ex) {
    log.error("Failed to publish outbox event: {}", event.getId(), ex);
    event.setRetryCount(event.getRetryCount() + 1);
    event.setUpdatedAt(Instant.now());

    if (event.getRetryCount() >= 5) {
        event.setStatus(OutboxEvent.OutboxStatus.FAILED);
    }
    outboxRepository.save(event);
}

}

The Golden Rule of At-Least-Once Delivery: Idempotent Consumers The Outbox Pattern guarantees at-least-once delivery. If the publisher crashes right after emitting a message to Kafka but before executing outboxRepository.save(event), the message will be published a second time when the worker reboots.

Downstream consumers must be strictly idempotent.

How to Ensure Idempotence in Consumer Services:
Unique Message Deduplication Table: Store the processed event_id in a local consumer table within a unique index constraint.

Conditional Updates: Use SQL version checks (UPDATE accounts SET balance = balance - 100 WHERE id = ? AND version = ?).

Database Constraints: Rely on natural primary keys (e.g., unique constraints on payment_transaction_id).

Java

@Component
@Slf4j
@RequiredArgsConstructor
public class InventoryOrderConsumer {

private final ProcessedEventRepository processedEventRepository;
private final InventoryService inventoryService;

@KafkaListener(topics = "order-events", groupId = "inventory-service")
@Transactional
public void handleOrderCreated(ConsumerRecord<String, String> record) {
    String eventId = record.key(); // or extract UUID from payload

    // 1. Deduplication check
    if (processedEventRepository.existsByEventId(eventId)) {
        log.info("Duplicate event detected. Skipping event id: {}", eventId);
        return;
    }

    // 2. Execute domain logic
    inventoryService.reserveStock(record.value());

    // 3. Mark processed in same transaction
    processedEventRepository.save(new ProcessedEventRecord(eventId, Instant.now()));
}

}
Conclusion: Architectural Takeaways
Failure Scenario Traditional @Transactional Transactional Outbox Pattern
Downstream HTTP Timeout Inconsistent state / Data drift No external HTTP calls; 100% atomic
Kafka Broker Outage Lost messages or rolled back business transactions Events persist safely in DB; published upon recovery
Throughput / Latency Bottlenecked by slowest downstream API call Sub-5ms database commits; asynchronous dispatch
Delivery Guarantee Best-effort (Zero guarantees) Guaranteed At-Least-Once
Stop relying on single-node transaction abstractions to coordinate distributed systems. Isolate your writes to local relational bounds, make event dispatching asynchronous, and build your consumer microservices to be strictly idempotent.
for cross-posting visit:https://sandeeptechieeblogs.blogspot.com/

The 3-Tier Spring Boot Optimization Playbook: From 800ms Latency to Sub-5ms Reliability

sandeep chagalakonda — Thu, 01 Oct 2026 06:35:15 +0000

In high-throughput systems, backend latency rarely stems from JVM CPU bottlenecks. In over 80% of production incidents, slow API endpoints trace back to three silent architectural traps: unmonitored database query multipliers (Hibernate N+1), misconfigured in-memory caching that leaks stale state, and connection pool starvation under concurrent load.

Here is the step-by-step 3-tier playbook to diagnose, isolate, and systematically reduce an unoptimized Spring Boot REST endpoint from an 800ms bottleneck down to sub-5ms execution.

Tier 1: Eliminate Database Query Multipliers (Hibernate N+1)
Before touching Redis or scaling container replicas, inspect the raw SQL queries your persistence layer emits. Placing an in-memory cache over an inefficient query path merely masks an architectural flaw until cache misses crash your database under peak traffic.

Consider a parent-child relationship where an Order entity contains a collection of OrderItem records:

@entity
@Table(name = "orders")
public class Order {
@id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;

private String customerReference;

@OneToMany(mappedBy = "order", fetch = FetchType.LAZY)
private List<OrderItem> items = new ArrayList<>();

// getters and setters

}
While FetchType.LAZY is appropriate default behavior, calling an unoptimized repository method triggers immediate query multiplication:

// ❌ Emits 1 query for orders + N queries for line items
List orders = orderRepository.findByStatus("COMPLETED");
When downstream serialization or business logic accesses order.getItems() across 50 orders, Hibernate executes 51 separate network round trips to the database:

-- 1 initial query
SELECT id, customer_reference FROM orders WHERE status = 'COMPLETED';

-- Followed by 50 individual queries
SELECT id, order_id, product_name, price FROM order_items WHERE order_id = 1;
SELECT id, order_id, product_name, price FROM order_items WHERE order_id = 2;
-- ... repeats for every single row
The Fix: Explicit JOIN FETCH
Instruct the persistence provider to retrieve the entire entity graph in a single query via an inner or left outer join:

public interface OrderRepository extends JpaRepository {

@Query("SELECT DISTINCT o FROM Order o LEFT JOIN FETCH o.items WHERE o.status = :status")
List<Order> findCompletedOrdersWithItems(@Param("status") String status);

}
Result: 51 queries collapse into exactly 1 single SQL round-trip. Baseline p95 query latency drops from ~800ms down to ~35ms immediately.

Tier 2: Production-Grade Redis Caching with TTL & Eviction
Once relational database queries are consolidated, introduce an in-memory Redis caching layer. The critical failure mode in Spring Boot caching is using @Cacheable without defining explicit Time-To-Live (TTL) or custom serialization, resulting in stale reads or memory leaks.

Configure deterministic cache defaults with JSON serialization in your configuration layer:

@Configuration
@EnableCaching
public class RedisCacheConfig {

@Bean
public RedisCacheManager cacheManager(RedisConnectionFactory connectionFactory) {
    RedisCacheConfiguration cacheConfig = RedisCacheConfiguration.defaultCacheConfig()
        .entryTtl(Duration.ofMinutes(15)) // Strict TTL to prevent stale state persistence
        .disableCachingNullValues()
        .serializeValuesWith(
            RedisSerializationContext.SerializationPair.fromSerializer(
                new GenericJackson2JsonRedisSerializer()
            )
        );

    return RedisCacheManager.builder(connectionFactory)
        .cacheDefaults(cacheConfig)
        .build();
}

}
Ensure write operations proactively clear or synchronize cache entries via @CacheEvict:

@Service
public class OrderService {

@Cacheable(value = "orders", key = "#id")
@Transactional(readOnly = true)
public OrderResponse getOrderById(Long id) {
    return orderRepository.findByIdWithItems(id)
        .map(OrderResponse::from)
        .orElseThrow(() -> new OrderNotFoundException(id));
}

@CacheEvict(value = "orders", key = "#id")
@Transactional
public void updateOrderStatus(Long id, String status) {
    orderRepository.updateStatus(id, status);
}

}
Result: Repeated read traffic hits in-memory RAM rather than disk storage, dropping p95 latency to sub-5ms.

Tier 3: HikariCP Connection Pool & Transaction Tuning
Even with clean queries and fast caching, services will throttle under load if worker threads experience connection pool starvation. When long-running transactions lock database connections unnecessarily, incoming requests back up rapidly.

Standardize connection pool boundaries inside your application.yml:

spring:
datasource:
hikari:
maximum-pool-size: 20
minimum-idle: 10
idle-timeout: 300000
connection-timeout: 20000 # Fast-fail after 20s instead of blocking indefinitely
leak-detection-threshold: 2000 # Emit WARN log if a query holds a connection > 2 seconds
jpa:
open-in-view: false # Prevent lazy-loading leaks outside service transactions
properties:
hibernate:
default_batch_fetch_size: 30 # Secondary safeguard against un-fetched collections
Always apply @Transactional(readOnly = true) on read workflows. This signals Hibernate to skip dirty-checking snapshot creation, cutting memory allocation per request.

Performance Benchmark Telemetry
The table below illustrates benchmark metrics captured using Spring Boot Actuator and Postman under simulated concurrency (100 concurrent worker threads, 50 entities per payload):

Optimization Stage SQL Queries p95 Latency DB CPU Load
Baseline (Default Lazy Loading) 51 820 ms 84%
Tier 1: Explicit JOIN FETCH 1 36 ms 14%
Tier 2 + 3: Redis Cache Hit + Tuned HikariCP 0 4 ms < 2%
Production Takeaways
Audit before caching: Never slap Redis on an unprofiled database endpoint. Fix the underlying SQL round-trips first.
Keep relationships LAZY, but fetch deliberately: Avoid switching entity fields to FetchType.EAGER, which causes unavoidable performance degradation across all usage patterns. Use JOIN FETCH or @EntityGraph only where child data is explicitly required.
Always configure leak detection: A 2-second leak-detection-threshold in HikariCP identifies connection hogging before it causes an outright service outage.
for cross-posting:https://sandeeptechieeblogs.blogspot.com/

I Deployed My Spring Boot App 6 Different Ways: Here’s the One That Stopped the 3 AM Incidents

sandeep chagalakonda — Mon, 28 Sep 2026 10:04:19 +0000

Disclosure: This article contains an affiliate/referral link. If you choose to sign up through it, I may earn a small credit at no additional cost to you. I only recommend infrastructure configurations I run in production.

I have deployed the exact same Spring Boot order management service six different ways over the past two years: Heroku, AWS EC2 with a hand-rolled Nginx reverse proxy, AWS Elastic Beanstalk, an unmanaged bare-metal VPS, Railway, and finally, a containerized setup on DigitalOcean Droplets backed by Managed PostgreSQL.

Each deployment taught me a distinct lesson. Most of those lessons arrived in the form of elevated error rates, silent SSL dropouts, or surprise billing alerts. If you have followed my previous posts on fixing N+1 queries, configuring Redis caching layers, or hardening CI/CD pipelines, you know that code defects are only half the battle. How and where you host that code determines whether your application stays resilient under real traffic.

Below is an honest, production-tested comparison of all six platforms—breaking down real monthly costs, operational burden, JVM memory constraints, and the exact configuration I rely on today.

The 6 Spring Boot Hosting Options Analyzed

Heroku: Frictionless Start, Inflexible Scaling Setup Time: 5 minutes Realistic Monthly Cost: $7 to $25+ per month (Postgres add-on included) Operational Burden: Low The Reality: Git-push deployments make Heroku convenient for proofs of concept. However, since the retirement of the free tier, entry-level dynos suffer from aggressive sleeping mechanisms that cause 15–30 second cold starts—an unacceptable latency spike for REST APIs. Add-ons like managed data stores scale steeply in price, making it inefficient for production JVM workloads.
AWS EC2 (Manual Setup): Total Control, Heavy Maintenance Setup Time: 2 to 3 hours Realistic Monthly Cost: $8 to $20 per month (t3.small / t4g.small) Operational Burden: High The Reality: You provision the instance, configure systemd daemon services, map ports via an Nginx reverse proxy, establish security groups, and manage Let's Encrypt certificate renewals via cron. It grants maximum operational freedom, but transforms you into an on-call systems administrator. If a background automated renewal breaks or an OS-level package conflicts, troubleshooting falls entirely on you.
AWS Elastic Beanstalk: Managed Abstraction, Complex Diagnostics Setup Time: 45 minutes Realistic Monthly Cost: $20 to $50 per month (EC2 + Application Load Balancer + RDS) Operational Burden: Medium-High The Reality: Elastic Beanstalk abstracts core provisioning while leveraging AWS infrastructure. However, debugging deployment failures requires navigating CloudWatch log streams, IAM role permission matrices, and environment health policies. The abstraction layer frequently obscures the underlying root cause when deployments stall.
Railway: Excellent DX, Unpredictable Memory Invoicing Setup Time: 10 minutes Realistic Monthly Cost: $10 to $40+ per month (Usage-based) Operational Burden: Low The Reality: The developer experience is first-class, with seamless GitHub webhook builds and clear dashboards. The challenge stems from Spring Boot's baseline memory behavior. Because the Java Virtual Machine reserves memory upfront (heap plus metaspace and native threads), dynamic resource-metered pricing models can produce unexpected billing spikes even during low-throughput periods.
Bare Unmanaged VPS: Lowest Price, Highest Fragility Setup Time: Ongoing maintenance Realistic Monthly Cost: $4 to $6 per month Operational Burden: Critical The Reality: Inexpensive on paper, but you own every operational failure. You must manually patch Linux kernel vulnerabilities, configure firewall tables (UFW/iptables), mitigate Docker layer disk bloat, and handle host-level failures without automated snapshot recovery.
DigitalOcean (Dockerized Droplet + Managed DB): The Sweet Spot
Setup Time: 15 to 20 minutes
Realistic Monthly Cost: $12 to $27 per month
Operational Burden: Low to Medium

The Reality: A 2GB RAM / 1 vCPU Droplet ($12/month) paired with DigitalOcean's Managed Database tier provides fixed, predictable monthly pricing alongside automated off-site backups and native metrics alerts. It delivers the isolation of a standard Linux VPS without the overhead of enterprise cloud complexity.

Platform Comparison Matrix
Hosting Platform Initial Setup Estimated Monthly Cost Ops Overhead Best Fit
Heroku 5 mins $7 – $25+ Low Prototypes & Demo APIs
AWS EC2 (Manual) 2 – 3 hrs $8 – $20 High Infrastructure Mastery
AWS Elastic Beanstalk 45 mins $20 – $50+ Medium-High Existing AWS Ecosystems
Railway 10 mins Usage-based ($10 – $40+) Low Rapid App Delivery
Bare-Metal VPS Ongoing $4 – $6 Very High Tight Budgets, High Ops Skill

DigitalOcean (Droplet) 15 – 20 mins $12 – $27 Low-Medium Production Backend Services

Production Configuration: Docker Compose + DigitalOcean
To avoid manual environment divergence, I deploy using immutable, commit-SHA tagged container images pulled directly from DigitalOcean Container Registry (DOCR) into a basic docker-compose.yml definition.
The Docker Compose Specification
version: '3.8'

services:
order-service:
image: registry.digitalocean.com/springforge-registry/order-service:${IMAGE_TAG}
container_name: order-service-prod
restart: unless-stopped
ports:
- "8080:8080"
environment:
- SPRING_PROFILES_ACTIVE=prod
- SPRING_DATASOURCE_URL=${DB_JDBC_URL}
- SPRING_DATASOURCE_USERNAME=${DB_USER}
- SPRING_DATASOURCE_PASSWORD=${DB_PASSWORD}
- JAVA_TOOL_OPTIONS=-Xms512m -Xmx1024m -XX:+UseG1GC
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:8080/actuator/health || exit 1"]
interval: 30s
timeout: 5s
retries: 3
start_period: 40s
logging:
driver: "json-file"
options:
max-size: "20m"
max-file: "5"

The Automated Deployment Script This script integrates with your CI runner or local build workflow to ensure zero untracked state on the host:

!/usr/bin/env bash

set -euo pipefail

COMMIT_TAG=$(git rev-parse --short HEAD)
REGISTRY="registry.digitalocean.com/springforge-registry/order-service"

echo "=== Building and Tagging Image: ${COMMIT_TAG} ==="
docker build -t "${REGISTRY}:${COMMIT_TAG}" .

echo "=== Authenticating and Pushing to DOCR ==="
doctl registry login
docker push "${REGISTRY}:${COMMIT_TAG}"

echo "=== Executing Droplet Deployment ==="
ssh deployer@your-droplet-ip \
"export IMAGE_TAG=${COMMIT_TAG} && \
cd /srv/apps/order-service && \
docker-compose pull order-service && \
docker-compose up -d --remove-orphans"

echo "=== Deployment Successful ==="
This approach enforces three critical operational standards:

Explicit JVM Heap Boundaries: Setting -Xms512m -Xmx1024m prevents Java from consuming the Droplet's remaining memory, leaving ample operating space for the OS kernel and metrics agents.
Actuator Health Checks: The container engine automatically marks instances unhealthy if the Spring context drops or database connectivity fails.
Log Rotation Caps: Restricting logs to five 20MB files prevents uncontrolled container logs from filling your storage volume.
Try This Architecture for Free: If you are looking to set up a production-ready environment without immediate overhead, you can use the link below to receive $200 in free cloud credits over 60 days for new DigitalOcean accounts. That provides more than enough bandwidth to run a 2GB Droplet and a Managed PostgreSQL database at zero cost while testing your setup:

DigitalOcean Referral Badge

Frequently Asked Questions
Is a 1GB Droplet sufficient for running Spring Boot?
No. While an empty Spring Boot service might idle at 300–400MB, production workloads involving Jackson serialization, active database connections (HikariCP), and thread stacks can quickly trigger Out-Of-Memory (OOM) kills on a 1GB host. A 2GB RAM instance is the practical minimum baseline for reliable production execution.

How does DigitalOcean differ from AWS for a solo engineer or small team?
AWS provides an expansive ecosystem of managed services, but comes with steep IAM policy overhead, complex VPC networking, and variable data transfer costs. DigitalOcean delivers predictable pricing, direct configuration interfaces, and core infrastructure building blocks without dedicated DevOps overhead.

What is the recommended database strategy for small production setups?
Run your application layer inside a Docker container on the Droplet, but point its datasource to a Managed Database. Managing database replication, failover, and point-in-time recovery on an ephemeral application server introduces significant recovery risk.

Recommendation links:https://sandeeptechieeblogs.the-day-our-green-ci-pipeline-deployed

https://sandeeptechieebhow-my-microservices-architecture
for cross-posting visit:https://sandeeptechieeblogs.blogspot.com

The Day Our Green CI Pipeline Deployed an 11-Day-Old Bug to Production

sandeep chagalakonda — Sat, 19 Sep 2026 14:30:53 +0000

The Day Our Deployment Pipeline Deployed the Wrong Version to Production
We had 47 green checkmarks. Every test passed. CI was happy. The deploy button was practically begging to be pressed.

I pressed it.

Twenty minutes later, a customer support ticket came in: "The export feature is completely broken. It's showing data from three weeks ago."

Then another. Then five more.

We hadn't broken the export feature. We had deployed a version of the code from eleven days earlier — quietly overwriting a critical bug fix and reintroducing a data staleness issue we'd already resolved.

The scariest part? Our pipeline said everything was successful. No errors. No warnings. Green across the board. We had just shipped the wrong version of our own application with total confidence.

Here's how that happened, and the changes we made so it can't happen again.

The Setup: A Pipeline That Looked Solid
Our deployment flow looked reasonable on paper:

Push to main → Run tests → Build Docker image → Push to registry → Deploy to production
Jenkins pipeline, roughly:

groovy
pipeline {
agent any

stages {
    stage('Checkout') {
        steps {
            git branch: 'main', url: 'https://github.com/company/order-service.git'
        }
    }

    stage('Test') {
        steps {
            sh 'mvn test'
        }
    }

    stage('Build') {
        steps {
            sh 'docker build -t order-service:latest .'
        }
    }

    stage('Push') {
        steps {
            sh 'docker push registry.company.com/order-service:latest'
        }
    }

    stage('Deploy') {
        steps {
            sh 'kubectl set image deployment/order-service order-service=registry.company.com/order-service:latest'
        }
    }
}

}
Notice anything? I didn't, for months. Everything is tagged :latest.

That single word — latest — was about to cause a very bad Tuesday.

The Failure: How "Latest" Betrayed Us
Here's the actual sequence of events, once we reconstructed it from logs:

Step 1: A teammate had an old feature branch, checked out locally from 11 days earlier, that he was using to test something unrelated on his machine.

Step 2: He accidentally ran a local script that pushed a Docker image tagged order-service:latest directly to our registry — bypassing CI entirely — while testing a local Docker build command he'd copy-pasted from an old note.

Step 3: Fifteen minutes later, an unrelated, completely correct PR was merged to main. CI ran, all tests passed, and it triggered a deploy.

Step 4: Here's the killer: our Kubernetes deployment was already configured to pull :latest, and due to imagePullPolicy: IfNotPresent on some nodes, several pods didn't even re-pull the image — they just kept running whatever :latest happened to resolve to on that node, which was now his 11-day-old accidental push.

Step 5: New pods scheduled onto different nodes pulled the "real" latest deploy. Old pods on other nodes kept running the stale one.

We ended up with a production cluster running two different versions of the same service simultaneously — split roughly across nodes — for 20+ minutes, with zero visibility into it.

That's not a deployment bug. That's an entire deployment strategy bug.

Why This Happens (The Concept)
This incident had three separate root causes stacked on top of each other:

Mutable Tags (:latest is a Trap)
:latest isn't a version — it's a moving pointer. Anyone, anywhere, with push access can silently redefine what "latest" means. There's no audit trail tying a specific deploy to a specific, immutable artifact.
No Deployment Provenance
We had no way to answer the question: "What commit SHA is actually running in production right now?" Not without SSH-ing into a pod and checking manually.
Inconsistent Image Pull Behavior
imagePullPolicy: IfNotPresent means Kubernetes won't re-pull an image if it already has something cached under that tag name — even if the registry's :latest has moved on. Different nodes, different cache states, different code running.

Put together: a bypass of CI + a mutable tag + inconsistent pull behavior = two versions of production running side-by-side with a green pipeline the whole time.

The Fix: Building a Pipeline You Can Actually Trust
Fix #1: Never Deploy :latest. Tag by Commit SHA.
This is the single highest-leverage change we made.

groovy
pipeline {
agent any

environment {
    IMAGE_TAG = "${env.GIT_COMMIT.take(8)}"
}

stages {
    stage('Checkout') {
        steps {
            checkout scm
            script {
                env.IMAGE_TAG = sh(script: "git rev-parse --short=8 HEAD", returnStdout: true).trim()
            }
        }
    }

    stage('Test') {
        steps {
            sh 'mvn test'
        }
    }

    stage('Build') {
        steps {
            sh "docker build -t order-service:${IMAGE_TAG} ."
        }
    }

    stage('Push') {
        steps {
            sh "docker push registry.company.com/order-service:${IMAGE_TAG}"
        }
    }

    stage('Deploy') {
        steps {
            sh "kubectl set image deployment/order-service order-service=registry.company.com/order-service:${IMAGE_TAG} --record"
        }
    }
}

}
Now every image is immutable and traceable to an exact commit. order-service:a3f92c1e will always be exactly that code, forever. No ambiguity, no accidental overwrites, no "which version is this actually running."

Rule I now enforce on every project: if your deployment artifact's tag can mean something different tomorrow than it does today, you don't have reproducible deployments — you have a guessing game.

Fix #2: Block Manual Pushes to the Registry
The teammate's accidental push should never have been possible in the first place.

yaml

Registry access policy (example: AWS ECR repository policy)

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "OnlyCIPipelineCanPush",
"Effect": "Deny",
"Principal": "*",
"Action": [
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload"
],
"Condition": {
"StringNotEquals": {
"aws:PrincipalArn": "arn:aws:iam::123456789:role/jenkins-ci-role"
}
}
}
]
}
Only the CI service role can push images now. Individual developer credentials — however well-intentioned — cannot write directly to the production registry. If it doesn't go through the pipeline, it doesn't exist as a deployable artifact.

Fix #3: Explicit imagePullPolicy: Always
yaml

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: order-service
spec:
replicas: 4
template:
spec:
containers:
- name: order-service
image: registry.company.com/order-service:a3f92c1e
imagePullPolicy: Always # No ambiguity, no stale caches
Combined with commit-SHA tagging, this is almost redundant now (since each deploy uses a unique tag), but it's cheap insurance against any node serving a cached image under a tag it shouldn't.

Fix #4: A Deployment Manifest That Answers "What's Running Right Now?"
We built a tiny endpoint that every service exposes, populated at build time:

java
@RestController
public class VersionController {

@Value("${app.build.commit:unknown}")
private String commitSha;

@Value("${app.build.timestamp:unknown}")
private String buildTimestamp;

@GetMapping("/actuator/version")
public Map<String, String> version() {
    Map<String, String> info = new HashMap<>();
    info.put("commit", commitSha);
    info.put("builtAt", buildTimestamp);
    return info;
}

}
Injected at build time:

groovy
stage('Build') {
steps {
sh """
docker build \
--build-arg GIT_COMMIT=${IMAGE_TAG} \
--build-arg BUILD_TIME=\$(date -u +%Y-%m-%dT%H:%M:%SZ) \
-t order-service:${IMAGE_TAG} .
"""
}
}
Now, any time, from any environment: curl production-url/actuator/version tells you exactly what's running — no SSH, no guessing, no "let me check with the team who deployed last."

During an incident, this alone can save 20 minutes of "wait, are we even sure what version is live?"

Fix #5: A Rollback That Takes 30 Seconds, Not 30 Minutes
Because every deploy is tagged with an immutable commit SHA, rollback became trivial:

bash

Before: no idea what "the previous version" even means with :latest

After: exact, unambiguous rollback

kubectl set image deployment/order-service \
order-service=registry.company.com/order-service:PREVIOUS_KNOWN_GOOD_SHA \
--record

kubectl rollout status deployment/order-service
We also added a one-command rollback script that pulls the last known-good SHA from our deployment history automatically:

bash

!/bin/bash

rollback.sh

LAST_GOOD_SHA=$(kubectl rollout history deployment/order-service | tail -2 | head -1 | awk '{print $1}')
kubectl rollout undo deployment/order-service --to-revision=$LAST_GOOD_SHA
echo "Rolled back to revision: $LAST_GOOD_SHA"
What used to be a frantic 30-minute scramble (figure out what broke → find the last good version → manually redeploy it → hope it works) became a single command with a predictable outcome.

The Results
Metric Before After
Time to identify "what's currently deployed" 15-20 mins (manual investigation) <10 seconds (/actuator/version)
Time to roll back a bad deploy 25-30 mins Under 2 mins
Unauthorized/manual registry pushes possible Yes No (IAM-blocked)
Ambiguous "latest" deployments Constant risk Eliminated
Incidents caused by tag confusion 1 major (this one) 0 since
The bigger shift wasn't any single fix — it was realizing that a deployment pipeline's job isn't just "make the code run in production." It's "make it provable which code is running in production, and make undoing a mistake boring."

The Lessons

:latest is not a deployment strategy. It's a liability wearing a deployment strategy's clothes.
If your tags aren't immutable, your deployments aren't reproducible — and if they're not reproducible, you can't debug them with confidence.
CI passing green doesn't mean production is running what you think it's running.
Those are two separate claims. Bridge that gap explicitly (version endpoints, deployment records) instead of assuming they're the same thing.
Anything that CAN bypass your pipeline, eventually WILL bypass your pipeline.
Not out of malice — out of a well-meaning developer testing something locally at the wrong moment. Lock down write access to production artifacts at the infrastructure level, not just as a team policy.
Rollback speed matters more than deploy speed.
Everyone optimizes for fast deploys. Fewer teams optimize for "how fast can we undo a mistake at 2 AM." That second number is the one that determines how bad your worst incident gets.
"It passed CI" and "it's safe" are different sentences.
Tests validate code behavior. They say nothing about whether the artifact that reaches production is the artifact you think it is. You need both.

What I'd Tell a Team Setting Up CI/CD Today
Don't just ask "how do we deploy fast?" Ask "if this deploy goes wrong at 2 AM, how do we prove what's running, and how fast can we undo it?"

If you can't answer that second question in under a minute, you don't have a deployment pipeline — you have a deployment hope.

Tag by commit SHA. Lock down your registry. Expose a version endpoint. Build a rollback script before you need it, not while you need it.

Because the day you need it will not send you a warning.

Questions? Comments? Drop them below — I read and reply to every one.

Related Reading
How My Microservices Architecture Collapsed (And How I Fixed It)
Why Your Spring Boot API is Slow: The N+1 Query Problem
Building a Secure JWT Authentication Filter in Spring Boot 3
crossposting:https://sandeeptechieeblogs.blogspot.com/2026/09/the-day-our-green-ci-pipeline-deployed.html

How My Microservices Architecture Collapsed (And How I Fixed It)

sandeep chagalakonda — Fri, 11 Sep 2026 14:58:53 +0000

It was 11 PM on a Friday when my phone started buzzing non-stop.

Not the good kind of buzzing. The kind that means production is on fire.

I opened Slack. 47 unread messages. The order service was throwing timeouts. The payment service couldn't reach the inventory service. Customers were seeing spinning loaders that never stopped spinning.

Our "scalable microservices architecture" — the one I was so proud of three months earlier — had turned into a house of cards. And someone had just sneezed.

Here's what happened, why it happened, and how I fixed it so it never happened again.

The Setup: When Microservices Felt Like a Good Idea
Six months earlier, we split our monolith into five services:

Order Service → Payment Service → Inventory Service → Notification Service → Shipping Service
On paper, this looked great:

Each team owns their service
Independent deployments
Scale services individually
Clean separation of concerns
In practice, we had created a distributed monolith. Every service called every other service synchronously, and none of us had thought about what happens when one link in that chain gets slow — or breaks entirely.

That Friday night, the inventory service started responding slowly because of a bad database index (unrelated issue, we'll get to that another time). But that slowness didn't stay contained. It spread like a virus through every service that depended on it.

The Cascade: How One Slow Service Took Down Everything
Here's the actual chain of failure, step by step:

Step 1: Inventory service's database query started taking 8 seconds instead of 80ms (missing index after a schema migration).

Step 2: Order service calls inventory service synchronously with a default timeout of... nothing. No timeout was configured. It just waited.

// This is what we had - no timeout, no fallback
@Service
public class OrderService {

@Autowired
private RestTemplate restTemplate;

public InventoryResponse checkInventory(String productId) {
    // If inventory service hangs, this hangs FOREVER
    return restTemplate.getForObject(
        "http://inventory-service/api/v1/check/" + productId,
        InventoryResponse.class
    );
}

}
Step 3: Every incoming order request now held a thread hostage for 8+ seconds waiting on inventory.

Step 4: Our order service had a fixed thread pool (Tomcat default: 200 threads). Within minutes, all 200 threads were stuck waiting on slow inventory calls.

Step 5: New order requests couldn't get a thread at all. They started timing out immediately.

Step 6: The payment service, which calls order service to confirm order status before processing payment, started timing out too.

Step 7: Customers saw failed payments, stuck loaders, and some got charged without their order confirming.

One slow database query in ONE service brought down the ENTIRE checkout flow.

That's the nature of distributed systems: failures don't stay isolated unless you design them to.

Why This Happens (The Concept Behind the Chaos)
This is called a cascading failure, and it happens because of three missing safeguards:

No Timeouts
Without a timeout, a slow dependency becomes an infinitely slow dependency from your service's perspective. Your thread just waits.
No Circuit Breakers
A circuit breaker "trips" when a downstream service is failing too often, and stops sending requests to it temporarily — giving it room to recover instead of hammering it with more traffic while it's already struggling.
No Bulkheads
A bulkhead isolates resources (like thread pools) per dependency, so a problem in one integration can't consume ALL your available threads.

Without these three things, every synchronous call in your system is a potential single point of failure for your entire system — even if that call is to a "less important" service.

The Fix: Building Resilience Into the Architecture
Fix #1: Add Timeouts Everywhere (The Bare Minimum)
This should have existed from day one. It didn't.

@Configuration
public class RestTemplateConfig {

@Bean
public RestTemplate restTemplate() {
    HttpComponentsClientHttpRequestFactory factory = 
        new HttpComponentsClientHttpRequestFactory();

    factory.setConnectTimeout(2000);  // 2 seconds to connect
    factory.setConnectionRequestTimeout(2000);
    factory.setReadTimeout(3000);     // 3 seconds to get a response

    return new RestTemplate(factory);
}

}
Rule of thumb: No external call should ever be allowed to hang indefinitely. Pick a timeout that's generous enough for normal traffic but short enough that a slow dependency fails fast instead of holding your service hostage.

Fix #2: Circuit Breaker with Resilience4j
We added Resilience4j to stop hammering a struggling service and to fail fast once we know it's unhealthy.

Add dependency:

io.github.resilience4j
resilience4j-spring-boot3
2.1.0

Configure the circuit breaker:

application.yml

resilience4j:
circuitbreaker:
instances:
inventoryService:
sliding-window-size: 10
failure-rate-threshold: 50
wait-duration-in-open-state: 10s
permitted-number-of-calls-in-half-open-state: 3
automatic-transition-from-open-to-half-open-enabled: true
timelimiter:
instances:
inventoryService:
timeout-duration: 3s
Apply it to the call:

@Service
public class OrderService {

@Autowired
private RestTemplate restTemplate;

@CircuitBreaker(name = "inventoryService", fallbackMethod = "inventoryFallback")
@TimeLimiter(name = "inventoryService")
public CompletableFuture<InventoryResponse> checkInventory(String productId) {
    return CompletableFuture.supplyAsync(() -> 
        restTemplate.getForObject(
            "http://inventory-service/api/v1/check/" + productId,
            InventoryResponse.class
        )
    );
}

// Called automatically when circuit is open or call fails
public CompletableFuture<InventoryResponse> inventoryFallback(String productId, Throwable t) {
    log.warn("Inventory service unavailable, using cached fallback for product: {}", productId);

    // Return cached last-known inventory state, or a safe default
    InventoryResponse fallback = inventoryCacheService.getLastKnownState(productId);
    return CompletableFuture.completedFuture(fallback);
}

}
What this gives us:

After 50% of recent calls fail, the circuit "opens" — we stop calling inventory service entirely for 10 seconds
During that time, we use the fallback (cached inventory data) instead of hammering a struggling service
After 10 seconds, it tries a few test calls ("half-open") to see if inventory service has recovered
If it has, the circuit closes and normal traffic resumes
This alone would have prevented 90% of that Friday night's damage.

Fix #3: Bulkhead Pattern (Isolate Thread Pools)
Even with timeouts, one slow dependency shouldn't be able to eat every thread in your application.

resilience4j:
bulkhead:
instances:
inventoryService:
max-concurrent-calls: 20
max-wait-duration: 500ms
paymentService:
max-concurrent-calls: 30
max-wait-duration: 500ms
@Bulkhead(name = "inventoryService", type = Bulkhead.Type.THREADPOOL)
public CompletableFuture checkInventory(String productId) {
// Only 20 concurrent calls to inventory service allowed,
// regardless of what's happening elsewhere in the app
...
}
Now, even if inventory service goes completely down, only 20 threads get stuck waiting on it — not all 200. Orders that don't depend on inventory keep flowing normally.

Fix #4: Move Non-Critical Calls to Async (Message Queue)
The biggest architectural change: not everything needs to be synchronous.

Before (synchronous, blocking):

Order placed → Wait for Payment → Wait for Inventory update →
Wait for Notification sent → Wait for Shipping label created →
Return response to customer
Every one of those "waits" is a place where things can go wrong and block the customer.

After (event-driven with RabbitMQ):

@Service
public class OrderService {

@Autowired
private RabbitTemplate rabbitTemplate;

public OrderResponse placeOrder(OrderRequest request) {
    // Only the CRITICAL path is synchronous
    Order order = createOrder(request);
    PaymentResult payment = paymentService.charge(order); // Must be sync

    if (payment.isSuccessful()) {
        order.setStatus(OrderStatus.CONFIRMED);
        orderRepository.save(order);

        // Everything else happens asynchronously
        rabbitTemplate.convertAndSend("order.confirmed", new OrderEvent(order.getId()));

        return new OrderResponse(order.getId(), "CONFIRMED");
    }

    order.setStatus(OrderStatus.FAILED);
    orderRepository.save(order);
    return new OrderResponse(order.getId(), "FAILED");
}

}

// Separate consumers handle the non-critical work independently
@Component
public class OrderEventConsumer {

@RabbitListener(queues = "inventory.update.queue")
public void updateInventory(OrderEvent event) {
    // If this fails, it retries independently.
    // It does NOT block the customer's checkout response.
    inventoryService.reserveStock(event.getOrderId());
}

@RabbitListener(queues = "notification.queue")
public void sendNotification(OrderEvent event) {
    notificationService.sendOrderConfirmation(event.getOrderId());
}

@RabbitListener(queues = "shipping.queue")
public void createShippingLabel(OrderEvent event) {
    shippingService.generateLabel(event.getOrderId());
}

}
Why this matters: Inventory service being slow no longer affects whether a customer's payment goes through. It just means the inventory update happens a few seconds later than usual — invisible to the customer, and retryable if it fails.

The rule I now follow: if a step doesn't need to block the customer's response, it doesn't belong in the synchronous path.

Fix #5: Health Checks + Proper Monitoring
We had no visibility into which service was struggling until customers started complaining. That's backwards.

@Component
public class InventoryServiceHealthIndicator implements HealthIndicator {

@Autowired
private RestTemplate restTemplate;

@Override
public Health health() {
    try {
        ResponseEntity<String> response = restTemplate.getForEntity(
            "http://inventory-service/actuator/health", String.class
        );

        if (response.getStatusCode().is2xxSuccessful()) {
            return Health.up().build();
        }
        return Health.down().withDetail("status", response.getStatusCode()).build();

    } catch (Exception e) {
        return Health.down().withException(e).build();
    }
}

}
Combined with Spring Boot Actuator + Prometheus + Grafana, we now get alerted the moment a service's response time creeps up — long before it becomes a full outage.

Alert rule (Prometheus)

alert: SlowInventoryResponses expr: histogram_quantile(0.95, http_request_duration_seconds{service="inventory"}) > 1 for: 2m annotations: summary: "Inventory service p95 latency above 1s for 2 minutes" That Friday night, this alert would have fired 20 minutes before the cascading failure even started.

The Results
Metric Before After
Cascading failures (per quarter) 3 0
Mean time to detect an issue 25+ mins (customer reports) 2 mins (automated alert)
Thread pool exhaustion incidents 2 0
Checkout success rate during partial outages ~15% ~92%
On-call pages at 11 PM Too many Way fewer
The most important number isn't in that table: customer trust. When checkout keeps working even while one internal service is having a bad night, customers never know anything went wrong at all. That's the actual goal of resilience engineering — not eliminating failures (impossible), but containing them so they don't cascade.

The Lessons

Microservices don't remove complexity — they redistribute it.
A monolith fails as one unit. A microservices architecture fails as a graph, and failures can travel along edges you didn't think were critical.
Every synchronous call is a liability until proven otherwise.
Ask of every inter-service call: "What happens to MY service if this one hangs for 30 seconds? For 5 minutes? Forever?" If you don't know the answer, you have a timeout gap.
Timeouts, circuit breakers, and bulkheads are not optional extras.
They're as fundamental to a distributed system as exception handling is to a single application. Skipping them isn't "moving fast" — it's deferring an outage to a worse time (like 11 PM on a Friday).
Not every operation needs to be synchronous.
If the customer doesn't need to wait for it, it belongs in a queue, not in the request path.
You can't fix what you can't see.
Health checks and latency dashboards aren't nice-to-haves. They're what turns a 2 AM outage into a 2 PM "huh, that's interesting, let's fix it" ticket.

What I'd Tell Someone Starting Microservices Today
Don't split a monolith into microservices because it's trendy. Split it because you have a real scaling or team-ownership problem that a monolith can't solve.

And if you do split it, treat resilience patterns — timeouts, circuit breakers, bulkheads, async messaging, and observability — as part of the minimum viable architecture, not as a "phase 2" improvement you'll get to later.

Because "later" showed up for us at 11 PM on a Friday. It probably will for you too.

Questions? Comments? Drop them below. I read and reply to every comment.

Related Reading
Why Your Spring Boot API is Slow: The N+1 Query Problem
How Redis Reduced My Spring Boot API Response Time from 800ms to 5ms
Building a Secure JWT Authentication Filter in Spring Boot 3
Cross-posted from my blog:

Follow for more production engineering insights on Java, Spring Boot, and distributed systems.

"Why Your Spring Boot API is Slow: The N+1 Query Problem (And How I Fixed It in Production)"

sandeep chagalakonda — Sun, 06 Sep 2026 12:50:10 +0000

I was debugging a Spring Boot API at 2 AM on a Tuesday when I realized something that should have been obvious: my database query was being executed 500 times per request.

Not 5 times. Not 50 times.

Five. Hundred. Times.

One single user request → 500 SQL queries → API response time: 3.2 seconds.

That's when I learned about the N+1 query problem. And how it nearly destroyed production.

The Setup: Everything Seemed Fine
Our food delivery microservice was working great. Orders were being processed, customers were happy, and performance looked good in development.

Then we hit production traffic.

Day 1: Latency: 200ms ✅ Day 5: Latency: 500ms ⚠️ Day 10: Latency: 1.2 seconds 🔴 Day 15: Latency: 3+ seconds 💥

Our API was getting slower every single day. And we had no idea why.

I grabbed a profiler and started investigating. That's when I found it: the N+1 query catastrophe.

What Is the N+1 Query Problem?
Imagine you want to fetch a list of orders with their customers.

The naive approach:

@entity
public class Order {
@id
private Long id;
private String orderNumber;

@ManyToOne
private Customer customer; // This relationship is the problem

}

@Service
public class OrderService {
@Autowired
private OrderRepository orderRepository;

public List<Order> getAllOrders() {
    return orderRepository.findAll(); // Query 1: Get all orders
    // For each order, fetch customer // Query 2, 3, 4, 5...
}

}
Here's what happens in your database:

-- Query 1: Get all orders (1 query)
SELECT * FROM orders;

-- Query 2: Get customer for order 1 (N queries)
SELECT * FROM customers WHERE id = 101;

-- Query 3: Get customer for order 2
SELECT * FROM customers WHERE id = 102;

-- Query 4: Get customer for order 3
SELECT * FROM customers WHERE id = 103;

-- ... repeat for every single order ...
If you fetch 500 orders:

1 query to get orders
500 queries to get each customer
Total: 501 queries
That's the N+1 problem. You execute 1 query, then N more queries (one per result).

At scale, this destroys performance.

How I Discovered It (The Hard Way)
I was looking at our order endpoint logs:

GET /api/v1/orders
Database queries: 487
Query time: 2.8 seconds
487 queries for a single API request.

I added Spring Boot's query logging to see what was happening:

application.properties

spring.jpa.properties.hibernate.format_sql=true
spring.jpa.properties.hibernate.use_sql_comments=true
logging.level.org.hibernate.SQL=DEBUG
logging.level.org.hibernate.type.descriptor.sql.BasicBinder=TRACE
Then I made a single API call. Here's what the logs showed:

Hibernate: select order0_.id, order0_.customer_id, order0_.order_number, order0_.total_amount from orders order0_ limit 100
Hibernate: select customer0_.id, customer0_.name, customer0_.email from customers customer0_ where customer0_.id=?
Hibernate: select customer0_.id, customer0_.name, customer0_.email from customers customer0_ where customer0_.id=?
Hibernate: select customer0_.id, customer0_.name, customer0_.email from customers customer0_ where customer0_.id=?
Hibernate: select customer0_.id, customer0_.name, customer0_.email from customers customer0_ where customer0_.id=?
... (repeat 95 more times)
My jaw dropped.

Hibernate was fetching the customer for every single order, one at a time. It was supposed to be doing one fetch. Instead, it was doing 100+ queries.

This is classic N+1 query problem.

The Root Cause: Lazy Loading
By default, JPA uses lazy loading for relationships:

@ManyToOne(fetch = FetchType.LAZY) // Default behavior
private Customer customer;
This means:

When you load an Order, the Customer is NOT loaded
When you access order.getCustomer(), JPA fetches it then
If you have 500 orders and access customer on each one → 500 queries
This was my code:

@GetMapping("/orders")
public ResponseEntity> getAllOrders() {
List orders = orderService.getAllOrders();

// This loop triggers the N+1 problem
List<OrderDTO> dtos = orders.stream()
    .map(order -> new OrderDTO(
        order.getId(),
        order.getOrderNumber(),
        order.getCustomer().getName() // Query executed here! 
    ))
    .collect(Collectors.toList());

return ResponseEntity.ok(dtos);

}
Every time the code accessed order.getCustomer().getName(), Hibernate fired a separate SQL query.

Result: 1 query to get orders + 500 queries to get customers = 501 total queries.

The Fix #1: Eager Loading (The Quick Fix)
The simplest solution: tell JPA to fetch the customer when loading the order.

@entity
public class Order {
@id
private Long id;

@ManyToOne(fetch = FetchType.EAGER) // Change to EAGER
private Customer customer;

}
Now Hibernate does:

SELECT order0_.id, order0_.customer_id, order0_.order_number, customer1_.id, customer1_.name, customer1_.email
FROM orders order0_
LEFT JOIN customers customer1_ ON order0_.customer_id = customer1_.id
Single query. 500 results.

Result:

Before: 501 queries, 2.8 seconds
After: 1 query, 120ms
That's 23x faster.

Why This Isn't Always The Answer
Problem: Eager loading loads customers even if you don't need them.

Example: If you have another endpoint that just needs order numbers:

@GetMapping("/orders/numbers")
public List getOrderNumbers() {
List orders = orderRepository.findAll(); // Loads 500 customers unnecessarily
return orders.stream()
.map(Order::getOrderNumber)
.collect(Collectors.toList());
}
Now you're loading data you don't use. Wastes memory and database resources.

Better approach: Use eager loading only where you need it.

The Fix #2: Fetch Join (The Proper Fix)
Instead of changing the entity, use JPQL fetch join in your query:

@Repository
public interface OrderRepository extends JpaRepository {

@Query("SELECT DISTINCT o FROM Order o " +
       "LEFT JOIN FETCH o.customer c " +
       "WHERE o.id IN :orderIds")
List<Order> findOrdersWithCustomers(@Param("orderIds") List<Long> orderIds);

// Or get all with customers
@Query("SELECT DISTINCT o FROM Order o " +
       "LEFT JOIN FETCH o.customer c")
List<Order> findAllWithCustomers();

}
Now update your service:

@Service
public class OrderService {
@Autowired
private OrderRepository orderRepository;

public List<Order> getAllOrdersWithCustomers() {
    return orderRepository.findAllWithCustomers(); // Uses fetch join
}

}
Hibernate generates:

SELECT DISTINCT order0_.id, order0_.customer_id, order0_.order_number,
customer1_.id, customer1_.name, customer1_.email
FROM orders order0_
LEFT JOIN customers customer1_ ON order0_.customer_id = customer1_.id
Single query. All data loaded.

Why this is better:

✅ Only loads customer when you need it
✅ Still uses one query (no N+1 problem)
✅ You control when eager loading happens
✅ Different queries can load different relationships
The Fix #3: Projection (The Advanced Fix)
Sometimes you don't need the full Order object. You just need specific fields:

public interface OrderDTO {
Long getId();
String getOrderNumber();
String getCustomerName(); // Comes from customer table
BigDecimal getTotalAmount();
}

@Repository
public interface OrderRepository extends JpaRepository {

@Query("SELECT new com.example.dto.OrderDTO(" +
       "o.id, o.orderNumber, c.name, o.totalAmount) " +
       "FROM Order o " +
       "LEFT JOIN o.customer c")
List<OrderDTO> findAllOrderDTOs();

}
Hibernate generates:

SELECT order0_.id, order0_.order_number, customer1_.name, order0_.total_amount
FROM orders order0_
LEFT JOIN customers customer1_ ON order0_.customer_id = customer1_.id
Single query. Only the fields you need.

Why this is best for APIs:

✅ Single query (no N+1)
✅ Returns DTO directly (no mapping overhead)
✅ Database returns only needed columns
✅ Fastest option for REST responses
The Real-World Fix (What I Did)
In production, I did all three:

Identified N+1 queries:

Enable query logging to spot N+1 problems

spring.jpa.properties.hibernate.generate_statistics=true
logging.level.org.hibernate.stat=debug

Fixed the critical endpoints with fetch join:

@GetMapping("/api/v1/orders")
public ResponseEntity> getAllOrders(
@RequestParam(defaultValue = "0") int page,
@RequestParam(defaultValue = "50") int size) {

// Uses fetch join - single query with pagination
Page<OrderDTO> orders = orderRepository.findAllOrderDTOs(
    PageRequest.of(page, size)
);

return ResponseEntity.ok(orders.getContent());

}

Added Spring Data Specification for complex queries:

@Repository
public interface OrderRepository extends
JpaRepository,
JpaSpecificationExecutor {

// Specifications handle complex queries efficiently

}

@Service
public class OrderService {
public List searchOrders(OrderSearchCriteria criteria) {
return orderRepository.findAll((root, query, cb) -> {
Join customerJoin = root.join("customer", JoinType.LEFT);

        // Complex query with joins - still single query
        Predicate predicate = cb.and(
            cb.like(customerJoin.get("name"), criteria.getCustomerName() + "%"),
            cb.greaterThan(root.get("totalAmount"), criteria.getMinAmount())
        );

        return predicate;
    });
}

}
The Results (Before & After)
Metric Before After Improvement
Queries per request 487 1 487x ↓
Response time 2.8s 120ms 23x ↓
Database load 95% CPU 15% CPU 80% ↓
Customer complaints 47 0 100% ↓
After the fix:

Orders endpoint latency: 120ms (was 2.8s)
Database CPU dropped from 95% to 15%
Concurrent users increased from 100 to 500 without slowdown
Zero customer complaints about slow orders
How To Prevent This In The Future

Enable query counting in development:

spring.jpa.properties.hibernate.generate_statistics=true
logging.level.org.hibernate.stat=debug

Monitor queries in your tests:

@test
@Transactional
public void testOrderFetch() {
// Hibernate counts queries during test
List orders = orderService.getAllOrders();

// If this shows > 1 query, N+1 problem detected
// The test fails before production sees it

}

Use database profiling tools:

MySQL: Enable query logs and analyze slow queries
PostgreSQL: Enable auto_explain
Spring Boot Actuator: Monitor database metrics

Code review checklist:

When you see this in a PR:

List orders = orderRepository.findAll();
orders.forEach(order -> {
String customerName = order.getCustomer().getName(); // RED FLAG
});
Ask: "Is there a way to fetch this with a single query?"

The Lesson
The N+1 query problem is invisible until it hits production.

It doesn't show up in:

❌ Unit tests (usually tiny datasets)
❌ Local development (cache hides the problem)
❌ Early production (low traffic)
It explodes when:

✅ Real data volume arrives
✅ Multiple concurrent users
✅ Database connection pool saturated
✅ API timeout starts happening
Prevention is easier than debugging:

Use fetch joins for relationships
Use projections for DTOs
Monitor queries in development
Test with realistic data volumes
Next Steps
If you're experiencing slow Spring Boot APIs:

Enable Hibernate query logging
Make a single API request
Count the SQL queries
If > 2-3 queries for simple endpoint → N+1 problem
Use fetch join or projection to fix
It usually takes 30 minutes to fix and saves your users hours of waiting.

That Tuesday at 2 AM was frustrating. But it taught me something valuable: always think about how many database queries your code executes.

Your users will thank you.

Questions? Comments? Drop them below! I read every single comment and reply within 24 hours.

Related Reading
Spring Boot with Hibernate: Why Your Queries Are Slow
JPA Fetch Strategies: Lazy vs Eager Loading Explained
How to Debug Spring Boot Database Performance

Next Article: "The Connection Pool Mistake That Cost Us $5,000 in RDS Bills"

Cross-posted from my blog: Spring Boot N+1 Query Problem

Follow for more production engineering insights on Java, Spring Boot, and distributed systems.

DEV Community: sandeep chagalakonda

The Distributed Transaction Nightmare: Why @Transactional Silently Failed Across Microservices (And the Outbox Pattern Fix)

The 3-Tier Spring Boot Optimization Playbook: From 800ms Latency to Sub-5ms Reliability

I Deployed My Spring Boot App 6 Different Ways: Here’s the One That Stopped the 3 AM Incidents

DigitalOcean (Droplet) 15 – 20 mins $12 – $27 Low-Medium Production Backend Services

!/usr/bin/env bash

The Day Our Green CI Pipeline Deployed an 11-Day-Old Bug to Production

Registry access policy (example: AWS ECR repository policy)

deployment.yaml

Before: no idea what "the previous version" even means with :latest

After: exact, unambiguous rollback

!/bin/bash

rollback.sh

How My Microservices Architecture Collapsed (And How I Fixed It)

application.yml

Alert rule (Prometheus)

"Why Your Spring Boot API is Slow: The N+1 Query Problem (And How I Fixed It in Production)"

application.properties

Enable query logging to spot N+1 problems

Next Article: "The Connection Pool Mistake That Cost Us $5,000 in RDS Bills"