DEV Community: sandeep chagalakonda

The Day Our Green CI Pipeline Deployed an 11-Day-Old Bug to Production

sandeep chagalakonda — Sat, 19 Sep 2026 14:30:53 +0000

The Day Our Deployment Pipeline Deployed the Wrong Version to Production
We had 47 green checkmarks. Every test passed. CI was happy. The deploy button was practically begging to be pressed.

I pressed it.

Twenty minutes later, a customer support ticket came in: "The export feature is completely broken. It's showing data from three weeks ago."

Then another. Then five more.

We hadn't broken the export feature. We had deployed a version of the code from eleven days earlier — quietly overwriting a critical bug fix and reintroducing a data staleness issue we'd already resolved.

The scariest part? Our pipeline said everything was successful. No errors. No warnings. Green across the board. We had just shipped the wrong version of our own application with total confidence.

Here's how that happened, and the changes we made so it can't happen again.

The Setup: A Pipeline That Looked Solid
Our deployment flow looked reasonable on paper:

Push to main → Run tests → Build Docker image → Push to registry → Deploy to production
Jenkins pipeline, roughly:

groovy
pipeline {
agent any

stages {
    stage('Checkout') {
        steps {
            git branch: 'main', url: 'https://github.com/company/order-service.git'
        }
    }

    stage('Test') {
        steps {
            sh 'mvn test'
        }
    }

    stage('Build') {
        steps {
            sh 'docker build -t order-service:latest .'
        }
    }

    stage('Push') {
        steps {
            sh 'docker push registry.company.com/order-service:latest'
        }
    }

    stage('Deploy') {
        steps {
            sh 'kubectl set image deployment/order-service order-service=registry.company.com/order-service:latest'
        }
    }
}

}
Notice anything? I didn't, for months. Everything is tagged :latest.

That single word — latest — was about to cause a very bad Tuesday.

The Failure: How "Latest" Betrayed Us
Here's the actual sequence of events, once we reconstructed it from logs:

Step 1: A teammate had an old feature branch, checked out locally from 11 days earlier, that he was using to test something unrelated on his machine.

Step 2: He accidentally ran a local script that pushed a Docker image tagged order-service:latest directly to our registry — bypassing CI entirely — while testing a local Docker build command he'd copy-pasted from an old note.

Step 3: Fifteen minutes later, an unrelated, completely correct PR was merged to main. CI ran, all tests passed, and it triggered a deploy.

Step 4: Here's the killer: our Kubernetes deployment was already configured to pull :latest, and due to imagePullPolicy: IfNotPresent on some nodes, several pods didn't even re-pull the image — they just kept running whatever :latest happened to resolve to on that node, which was now his 11-day-old accidental push.

Step 5: New pods scheduled onto different nodes pulled the "real" latest deploy. Old pods on other nodes kept running the stale one.

We ended up with a production cluster running two different versions of the same service simultaneously — split roughly across nodes — for 20+ minutes, with zero visibility into it.

That's not a deployment bug. That's an entire deployment strategy bug.

Why This Happens (The Concept)
This incident had three separate root causes stacked on top of each other:

Mutable Tags (:latest is a Trap)
:latest isn't a version — it's a moving pointer. Anyone, anywhere, with push access can silently redefine what "latest" means. There's no audit trail tying a specific deploy to a specific, immutable artifact.
No Deployment Provenance
We had no way to answer the question: "What commit SHA is actually running in production right now?" Not without SSH-ing into a pod and checking manually.
Inconsistent Image Pull Behavior
imagePullPolicy: IfNotPresent means Kubernetes won't re-pull an image if it already has something cached under that tag name — even if the registry's :latest has moved on. Different nodes, different cache states, different code running.

Put together: a bypass of CI + a mutable tag + inconsistent pull behavior = two versions of production running side-by-side with a green pipeline the whole time.

The Fix: Building a Pipeline You Can Actually Trust
Fix #1: Never Deploy :latest. Tag by Commit SHA.
This is the single highest-leverage change we made.

groovy
pipeline {
agent any

environment {
    IMAGE_TAG = "${env.GIT_COMMIT.take(8)}"
}

stages {
    stage('Checkout') {
        steps {
            checkout scm
            script {
                env.IMAGE_TAG = sh(script: "git rev-parse --short=8 HEAD", returnStdout: true).trim()
            }
        }
    }

    stage('Test') {
        steps {
            sh 'mvn test'
        }
    }

    stage('Build') {
        steps {
            sh "docker build -t order-service:${IMAGE_TAG} ."
        }
    }

    stage('Push') {
        steps {
            sh "docker push registry.company.com/order-service:${IMAGE_TAG}"
        }
    }

    stage('Deploy') {
        steps {
            sh "kubectl set image deployment/order-service order-service=registry.company.com/order-service:${IMAGE_TAG} --record"
        }
    }
}

}
Now every image is immutable and traceable to an exact commit. order-service:a3f92c1e will always be exactly that code, forever. No ambiguity, no accidental overwrites, no "which version is this actually running."

Rule I now enforce on every project: if your deployment artifact's tag can mean something different tomorrow than it does today, you don't have reproducible deployments — you have a guessing game.

Fix #2: Block Manual Pushes to the Registry
The teammate's accidental push should never have been possible in the first place.

yaml

Registry access policy (example: AWS ECR repository policy)

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "OnlyCIPipelineCanPush",
"Effect": "Deny",
"Principal": "*",
"Action": [
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload"
],
"Condition": {
"StringNotEquals": {
"aws:PrincipalArn": "arn:aws:iam::123456789:role/jenkins-ci-role"
}
}
}
]
}
Only the CI service role can push images now. Individual developer credentials — however well-intentioned — cannot write directly to the production registry. If it doesn't go through the pipeline, it doesn't exist as a deployable artifact.

Fix #3: Explicit imagePullPolicy: Always
yaml

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: order-service
spec:
replicas: 4
template:
spec:
containers:
- name: order-service
image: registry.company.com/order-service:a3f92c1e
imagePullPolicy: Always # No ambiguity, no stale caches
Combined with commit-SHA tagging, this is almost redundant now (since each deploy uses a unique tag), but it's cheap insurance against any node serving a cached image under a tag it shouldn't.

Fix #4: A Deployment Manifest That Answers "What's Running Right Now?"
We built a tiny endpoint that every service exposes, populated at build time:

java
@RestController
public class VersionController {

@Value("${app.build.commit:unknown}")
private String commitSha;

@Value("${app.build.timestamp:unknown}")
private String buildTimestamp;

@GetMapping("/actuator/version")
public Map<String, String> version() {
    Map<String, String> info = new HashMap<>();
    info.put("commit", commitSha);
    info.put("builtAt", buildTimestamp);
    return info;
}

}
Injected at build time:

groovy
stage('Build') {
steps {
sh """
docker build \
--build-arg GIT_COMMIT=${IMAGE_TAG} \
--build-arg BUILD_TIME=\$(date -u +%Y-%m-%dT%H:%M:%SZ) \
-t order-service:${IMAGE_TAG} .
"""
}
}
Now, any time, from any environment: curl production-url/actuator/version tells you exactly what's running — no SSH, no guessing, no "let me check with the team who deployed last."

During an incident, this alone can save 20 minutes of "wait, are we even sure what version is live?"

Fix #5: A Rollback That Takes 30 Seconds, Not 30 Minutes
Because every deploy is tagged with an immutable commit SHA, rollback became trivial:

bash

Before: no idea what "the previous version" even means with :latest

After: exact, unambiguous rollback

kubectl set image deployment/order-service \
order-service=registry.company.com/order-service:PREVIOUS_KNOWN_GOOD_SHA \
--record

kubectl rollout status deployment/order-service
We also added a one-command rollback script that pulls the last known-good SHA from our deployment history automatically:

bash

!/bin/bash

rollback.sh

LAST_GOOD_SHA=$(kubectl rollout history deployment/order-service | tail -2 | head -1 | awk '{print $1}')
kubectl rollout undo deployment/order-service --to-revision=$LAST_GOOD_SHA
echo "Rolled back to revision: $LAST_GOOD_SHA"
What used to be a frantic 30-minute scramble (figure out what broke → find the last good version → manually redeploy it → hope it works) became a single command with a predictable outcome.

The Results
Metric Before After
Time to identify "what's currently deployed" 15-20 mins (manual investigation) <10 seconds (/actuator/version)
Time to roll back a bad deploy 25-30 mins Under 2 mins
Unauthorized/manual registry pushes possible Yes No (IAM-blocked)
Ambiguous "latest" deployments Constant risk Eliminated
Incidents caused by tag confusion 1 major (this one) 0 since
The bigger shift wasn't any single fix — it was realizing that a deployment pipeline's job isn't just "make the code run in production." It's "make it provable which code is running in production, and make undoing a mistake boring."

The Lessons

:latest is not a deployment strategy. It's a liability wearing a deployment strategy's clothes.
If your tags aren't immutable, your deployments aren't reproducible — and if they're not reproducible, you can't debug them with confidence.
CI passing green doesn't mean production is running what you think it's running.
Those are two separate claims. Bridge that gap explicitly (version endpoints, deployment records) instead of assuming they're the same thing.
Anything that CAN bypass your pipeline, eventually WILL bypass your pipeline.
Not out of malice — out of a well-meaning developer testing something locally at the wrong moment. Lock down write access to production artifacts at the infrastructure level, not just as a team policy.
Rollback speed matters more than deploy speed.
Everyone optimizes for fast deploys. Fewer teams optimize for "how fast can we undo a mistake at 2 AM." That second number is the one that determines how bad your worst incident gets.
"It passed CI" and "it's safe" are different sentences.
Tests validate code behavior. They say nothing about whether the artifact that reaches production is the artifact you think it is. You need both.

What I'd Tell a Team Setting Up CI/CD Today
Don't just ask "how do we deploy fast?" Ask "if this deploy goes wrong at 2 AM, how do we prove what's running, and how fast can we undo it?"

If you can't answer that second question in under a minute, you don't have a deployment pipeline — you have a deployment hope.

Tag by commit SHA. Lock down your registry. Expose a version endpoint. Build a rollback script before you need it, not while you need it.

Because the day you need it will not send you a warning.

Questions? Comments? Drop them below — I read and reply to every one.

Related Reading
How My Microservices Architecture Collapsed (And How I Fixed It)
Why Your Spring Boot API is Slow: The N+1 Query Problem
Building a Secure JWT Authentication Filter in Spring Boot 3
crossposting:https://sandeeptechieeblogs.blogspot.com/2026/09/the-day-our-green-ci-pipeline-deployed.html

How My Microservices Architecture Collapsed (And How I Fixed It)

sandeep chagalakonda — Fri, 11 Sep 2026 14:58:53 +0000

It was 11 PM on a Friday when my phone started buzzing non-stop.

Not the good kind of buzzing. The kind that means production is on fire.

I opened Slack. 47 unread messages. The order service was throwing timeouts. The payment service couldn't reach the inventory service. Customers were seeing spinning loaders that never stopped spinning.

Our "scalable microservices architecture" — the one I was so proud of three months earlier — had turned into a house of cards. And someone had just sneezed.

Here's what happened, why it happened, and how I fixed it so it never happened again.

The Setup: When Microservices Felt Like a Good Idea
Six months earlier, we split our monolith into five services:

Order Service → Payment Service → Inventory Service → Notification Service → Shipping Service
On paper, this looked great:

Each team owns their service
Independent deployments
Scale services individually
Clean separation of concerns
In practice, we had created a distributed monolith. Every service called every other service synchronously, and none of us had thought about what happens when one link in that chain gets slow — or breaks entirely.

That Friday night, the inventory service started responding slowly because of a bad database index (unrelated issue, we'll get to that another time). But that slowness didn't stay contained. It spread like a virus through every service that depended on it.

The Cascade: How One Slow Service Took Down Everything
Here's the actual chain of failure, step by step:

Step 1: Inventory service's database query started taking 8 seconds instead of 80ms (missing index after a schema migration).

Step 2: Order service calls inventory service synchronously with a default timeout of... nothing. No timeout was configured. It just waited.

// This is what we had - no timeout, no fallback
@Service
public class OrderService {

@Autowired
private RestTemplate restTemplate;

public InventoryResponse checkInventory(String productId) {
    // If inventory service hangs, this hangs FOREVER
    return restTemplate.getForObject(
        "http://inventory-service/api/v1/check/" + productId,
        InventoryResponse.class
    );
}

}
Step 3: Every incoming order request now held a thread hostage for 8+ seconds waiting on inventory.

Step 4: Our order service had a fixed thread pool (Tomcat default: 200 threads). Within minutes, all 200 threads were stuck waiting on slow inventory calls.

Step 5: New order requests couldn't get a thread at all. They started timing out immediately.

Step 6: The payment service, which calls order service to confirm order status before processing payment, started timing out too.

Step 7: Customers saw failed payments, stuck loaders, and some got charged without their order confirming.

One slow database query in ONE service brought down the ENTIRE checkout flow.

That's the nature of distributed systems: failures don't stay isolated unless you design them to.

Why This Happens (The Concept Behind the Chaos)
This is called a cascading failure, and it happens because of three missing safeguards:

No Timeouts
Without a timeout, a slow dependency becomes an infinitely slow dependency from your service's perspective. Your thread just waits.
No Circuit Breakers
A circuit breaker "trips" when a downstream service is failing too often, and stops sending requests to it temporarily — giving it room to recover instead of hammering it with more traffic while it's already struggling.
No Bulkheads
A bulkhead isolates resources (like thread pools) per dependency, so a problem in one integration can't consume ALL your available threads.

Without these three things, every synchronous call in your system is a potential single point of failure for your entire system — even if that call is to a "less important" service.

The Fix: Building Resilience Into the Architecture
Fix #1: Add Timeouts Everywhere (The Bare Minimum)
This should have existed from day one. It didn't.

@Configuration
public class RestTemplateConfig {

@Bean
public RestTemplate restTemplate() {
    HttpComponentsClientHttpRequestFactory factory = 
        new HttpComponentsClientHttpRequestFactory();

    factory.setConnectTimeout(2000);  // 2 seconds to connect
    factory.setConnectionRequestTimeout(2000);
    factory.setReadTimeout(3000);     // 3 seconds to get a response

    return new RestTemplate(factory);
}

}
Rule of thumb: No external call should ever be allowed to hang indefinitely. Pick a timeout that's generous enough for normal traffic but short enough that a slow dependency fails fast instead of holding your service hostage.

Fix #2: Circuit Breaker with Resilience4j
We added Resilience4j to stop hammering a struggling service and to fail fast once we know it's unhealthy.

Add dependency:

io.github.resilience4j
resilience4j-spring-boot3
2.1.0

Configure the circuit breaker:

application.yml

resilience4j:
circuitbreaker:
instances:
inventoryService:
sliding-window-size: 10
failure-rate-threshold: 50
wait-duration-in-open-state: 10s
permitted-number-of-calls-in-half-open-state: 3
automatic-transition-from-open-to-half-open-enabled: true
timelimiter:
instances:
inventoryService:
timeout-duration: 3s
Apply it to the call:

@Service
public class OrderService {

@Autowired
private RestTemplate restTemplate;

@CircuitBreaker(name = "inventoryService", fallbackMethod = "inventoryFallback")
@TimeLimiter(name = "inventoryService")
public CompletableFuture<InventoryResponse> checkInventory(String productId) {
    return CompletableFuture.supplyAsync(() -> 
        restTemplate.getForObject(
            "http://inventory-service/api/v1/check/" + productId,
            InventoryResponse.class
        )
    );
}

// Called automatically when circuit is open or call fails
public CompletableFuture<InventoryResponse> inventoryFallback(String productId, Throwable t) {
    log.warn("Inventory service unavailable, using cached fallback for product: {}", productId);

    // Return cached last-known inventory state, or a safe default
    InventoryResponse fallback = inventoryCacheService.getLastKnownState(productId);
    return CompletableFuture.completedFuture(fallback);
}

}
What this gives us:

After 50% of recent calls fail, the circuit "opens" — we stop calling inventory service entirely for 10 seconds
During that time, we use the fallback (cached inventory data) instead of hammering a struggling service
After 10 seconds, it tries a few test calls ("half-open") to see if inventory service has recovered
If it has, the circuit closes and normal traffic resumes
This alone would have prevented 90% of that Friday night's damage.

Fix #3: Bulkhead Pattern (Isolate Thread Pools)
Even with timeouts, one slow dependency shouldn't be able to eat every thread in your application.

resilience4j:
bulkhead:
instances:
inventoryService:
max-concurrent-calls: 20
max-wait-duration: 500ms
paymentService:
max-concurrent-calls: 30
max-wait-duration: 500ms
@Bulkhead(name = "inventoryService", type = Bulkhead.Type.THREADPOOL)
public CompletableFuture checkInventory(String productId) {
// Only 20 concurrent calls to inventory service allowed,
// regardless of what's happening elsewhere in the app
...
}
Now, even if inventory service goes completely down, only 20 threads get stuck waiting on it — not all 200. Orders that don't depend on inventory keep flowing normally.

Fix #4: Move Non-Critical Calls to Async (Message Queue)
The biggest architectural change: not everything needs to be synchronous.

Before (synchronous, blocking):

Order placed → Wait for Payment → Wait for Inventory update →
Wait for Notification sent → Wait for Shipping label created →
Return response to customer
Every one of those "waits" is a place where things can go wrong and block the customer.

After (event-driven with RabbitMQ):

@Service
public class OrderService {

@Autowired
private RabbitTemplate rabbitTemplate;

public OrderResponse placeOrder(OrderRequest request) {
    // Only the CRITICAL path is synchronous
    Order order = createOrder(request);
    PaymentResult payment = paymentService.charge(order); // Must be sync

    if (payment.isSuccessful()) {
        order.setStatus(OrderStatus.CONFIRMED);
        orderRepository.save(order);

        // Everything else happens asynchronously
        rabbitTemplate.convertAndSend("order.confirmed", new OrderEvent(order.getId()));

        return new OrderResponse(order.getId(), "CONFIRMED");
    }

    order.setStatus(OrderStatus.FAILED);
    orderRepository.save(order);
    return new OrderResponse(order.getId(), "FAILED");
}

}

// Separate consumers handle the non-critical work independently
@Component
public class OrderEventConsumer {

@RabbitListener(queues = "inventory.update.queue")
public void updateInventory(OrderEvent event) {
    // If this fails, it retries independently.
    // It does NOT block the customer's checkout response.
    inventoryService.reserveStock(event.getOrderId());
}

@RabbitListener(queues = "notification.queue")
public void sendNotification(OrderEvent event) {
    notificationService.sendOrderConfirmation(event.getOrderId());
}

@RabbitListener(queues = "shipping.queue")
public void createShippingLabel(OrderEvent event) {
    shippingService.generateLabel(event.getOrderId());
}

}
Why this matters: Inventory service being slow no longer affects whether a customer's payment goes through. It just means the inventory update happens a few seconds later than usual — invisible to the customer, and retryable if it fails.

The rule I now follow: if a step doesn't need to block the customer's response, it doesn't belong in the synchronous path.

Fix #5: Health Checks + Proper Monitoring
We had no visibility into which service was struggling until customers started complaining. That's backwards.

@Component
public class InventoryServiceHealthIndicator implements HealthIndicator {

@Autowired
private RestTemplate restTemplate;

@Override
public Health health() {
    try {
        ResponseEntity<String> response = restTemplate.getForEntity(
            "http://inventory-service/actuator/health", String.class
        );

        if (response.getStatusCode().is2xxSuccessful()) {
            return Health.up().build();
        }
        return Health.down().withDetail("status", response.getStatusCode()).build();

    } catch (Exception e) {
        return Health.down().withException(e).build();
    }
}

}
Combined with Spring Boot Actuator + Prometheus + Grafana, we now get alerted the moment a service's response time creeps up — long before it becomes a full outage.

Alert rule (Prometheus)

alert: SlowInventoryResponses expr: histogram_quantile(0.95, http_request_duration_seconds{service="inventory"}) > 1 for: 2m annotations: summary: "Inventory service p95 latency above 1s for 2 minutes" That Friday night, this alert would have fired 20 minutes before the cascading failure even started.

The Results
Metric Before After
Cascading failures (per quarter) 3 0
Mean time to detect an issue 25+ mins (customer reports) 2 mins (automated alert)
Thread pool exhaustion incidents 2 0
Checkout success rate during partial outages ~15% ~92%
On-call pages at 11 PM Too many Way fewer
The most important number isn't in that table: customer trust. When checkout keeps working even while one internal service is having a bad night, customers never know anything went wrong at all. That's the actual goal of resilience engineering — not eliminating failures (impossible), but containing them so they don't cascade.

The Lessons

Microservices don't remove complexity — they redistribute it.
A monolith fails as one unit. A microservices architecture fails as a graph, and failures can travel along edges you didn't think were critical.
Every synchronous call is a liability until proven otherwise.
Ask of every inter-service call: "What happens to MY service if this one hangs for 30 seconds? For 5 minutes? Forever?" If you don't know the answer, you have a timeout gap.
Timeouts, circuit breakers, and bulkheads are not optional extras.
They're as fundamental to a distributed system as exception handling is to a single application. Skipping them isn't "moving fast" — it's deferring an outage to a worse time (like 11 PM on a Friday).
Not every operation needs to be synchronous.
If the customer doesn't need to wait for it, it belongs in a queue, not in the request path.
You can't fix what you can't see.
Health checks and latency dashboards aren't nice-to-haves. They're what turns a 2 AM outage into a 2 PM "huh, that's interesting, let's fix it" ticket.

What I'd Tell Someone Starting Microservices Today
Don't split a monolith into microservices because it's trendy. Split it because you have a real scaling or team-ownership problem that a monolith can't solve.

And if you do split it, treat resilience patterns — timeouts, circuit breakers, bulkheads, async messaging, and observability — as part of the minimum viable architecture, not as a "phase 2" improvement you'll get to later.

Because "later" showed up for us at 11 PM on a Friday. It probably will for you too.

Questions? Comments? Drop them below. I read and reply to every comment.

Related Reading
Why Your Spring Boot API is Slow: The N+1 Query Problem
How Redis Reduced My Spring Boot API Response Time from 800ms to 5ms
Building a Secure JWT Authentication Filter in Spring Boot 3
Cross-posted from my blog:

Follow for more production engineering insights on Java, Spring Boot, and distributed systems.

"Why Your Spring Boot API is Slow: The N+1 Query Problem (And How I Fixed It in Production)"

sandeep chagalakonda — Sun, 06 Sep 2026 12:50:10 +0000

I was debugging a Spring Boot API at 2 AM on a Tuesday when I realized something that should have been obvious: my database query was being executed 500 times per request.

Not 5 times. Not 50 times.

Five. Hundred. Times.

One single user request → 500 SQL queries → API response time: 3.2 seconds.

That's when I learned about the N+1 query problem. And how it nearly destroyed production.

The Setup: Everything Seemed Fine
Our food delivery microservice was working great. Orders were being processed, customers were happy, and performance looked good in development.

Then we hit production traffic.

Day 1: Latency: 200ms ✅ Day 5: Latency: 500ms ⚠️ Day 10: Latency: 1.2 seconds 🔴 Day 15: Latency: 3+ seconds 💥

Our API was getting slower every single day. And we had no idea why.

I grabbed a profiler and started investigating. That's when I found it: the N+1 query catastrophe.

What Is the N+1 Query Problem?
Imagine you want to fetch a list of orders with their customers.

The naive approach:

@entity
public class Order {
@id
private Long id;
private String orderNumber;

@ManyToOne
private Customer customer; // This relationship is the problem

}

@Service
public class OrderService {
@Autowired
private OrderRepository orderRepository;

public List<Order> getAllOrders() {
    return orderRepository.findAll(); // Query 1: Get all orders
    // For each order, fetch customer // Query 2, 3, 4, 5...
}

}
Here's what happens in your database:

-- Query 1: Get all orders (1 query)
SELECT * FROM orders;

-- Query 2: Get customer for order 1 (N queries)
SELECT * FROM customers WHERE id = 101;

-- Query 3: Get customer for order 2
SELECT * FROM customers WHERE id = 102;

-- Query 4: Get customer for order 3
SELECT * FROM customers WHERE id = 103;

-- ... repeat for every single order ...
If you fetch 500 orders:

1 query to get orders
500 queries to get each customer
Total: 501 queries
That's the N+1 problem. You execute 1 query, then N more queries (one per result).

At scale, this destroys performance.

How I Discovered It (The Hard Way)
I was looking at our order endpoint logs:

GET /api/v1/orders
Database queries: 487
Query time: 2.8 seconds
487 queries for a single API request.

I added Spring Boot's query logging to see what was happening:

application.properties

spring.jpa.properties.hibernate.format_sql=true
spring.jpa.properties.hibernate.use_sql_comments=true
logging.level.org.hibernate.SQL=DEBUG
logging.level.org.hibernate.type.descriptor.sql.BasicBinder=TRACE
Then I made a single API call. Here's what the logs showed:

Hibernate: select order0_.id, order0_.customer_id, order0_.order_number, order0_.total_amount from orders order0_ limit 100
Hibernate: select customer0_.id, customer0_.name, customer0_.email from customers customer0_ where customer0_.id=?
Hibernate: select customer0_.id, customer0_.name, customer0_.email from customers customer0_ where customer0_.id=?
Hibernate: select customer0_.id, customer0_.name, customer0_.email from customers customer0_ where customer0_.id=?
Hibernate: select customer0_.id, customer0_.name, customer0_.email from customers customer0_ where customer0_.id=?
... (repeat 95 more times)
My jaw dropped.

Hibernate was fetching the customer for every single order, one at a time. It was supposed to be doing one fetch. Instead, it was doing 100+ queries.

This is classic N+1 query problem.

The Root Cause: Lazy Loading
By default, JPA uses lazy loading for relationships:

@ManyToOne(fetch = FetchType.LAZY) // Default behavior
private Customer customer;
This means:

When you load an Order, the Customer is NOT loaded
When you access order.getCustomer(), JPA fetches it then
If you have 500 orders and access customer on each one → 500 queries
This was my code:

@GetMapping("/orders")
public ResponseEntity> getAllOrders() {
List orders = orderService.getAllOrders();

// This loop triggers the N+1 problem
List<OrderDTO> dtos = orders.stream()
    .map(order -> new OrderDTO(
        order.getId(),
        order.getOrderNumber(),
        order.getCustomer().getName() // Query executed here! 
    ))
    .collect(Collectors.toList());

return ResponseEntity.ok(dtos);

}
Every time the code accessed order.getCustomer().getName(), Hibernate fired a separate SQL query.

Result: 1 query to get orders + 500 queries to get customers = 501 total queries.

The Fix #1: Eager Loading (The Quick Fix)
The simplest solution: tell JPA to fetch the customer when loading the order.

@entity
public class Order {
@id
private Long id;

@ManyToOne(fetch = FetchType.EAGER) // Change to EAGER
private Customer customer;

}
Now Hibernate does:

SELECT order0_.id, order0_.customer_id, order0_.order_number, customer1_.id, customer1_.name, customer1_.email
FROM orders order0_
LEFT JOIN customers customer1_ ON order0_.customer_id = customer1_.id
Single query. 500 results.

Result:

Before: 501 queries, 2.8 seconds
After: 1 query, 120ms
That's 23x faster.

Why This Isn't Always The Answer
Problem: Eager loading loads customers even if you don't need them.

Example: If you have another endpoint that just needs order numbers:

@GetMapping("/orders/numbers")
public List getOrderNumbers() {
List orders = orderRepository.findAll(); // Loads 500 customers unnecessarily
return orders.stream()
.map(Order::getOrderNumber)
.collect(Collectors.toList());
}
Now you're loading data you don't use. Wastes memory and database resources.

Better approach: Use eager loading only where you need it.

The Fix #2: Fetch Join (The Proper Fix)
Instead of changing the entity, use JPQL fetch join in your query:

@Repository
public interface OrderRepository extends JpaRepository {

@Query("SELECT DISTINCT o FROM Order o " +
       "LEFT JOIN FETCH o.customer c " +
       "WHERE o.id IN :orderIds")
List<Order> findOrdersWithCustomers(@Param("orderIds") List<Long> orderIds);

// Or get all with customers
@Query("SELECT DISTINCT o FROM Order o " +
       "LEFT JOIN FETCH o.customer c")
List<Order> findAllWithCustomers();

}
Now update your service:

@Service
public class OrderService {
@Autowired
private OrderRepository orderRepository;

public List<Order> getAllOrdersWithCustomers() {
    return orderRepository.findAllWithCustomers(); // Uses fetch join
}

}
Hibernate generates:

SELECT DISTINCT order0_.id, order0_.customer_id, order0_.order_number,
customer1_.id, customer1_.name, customer1_.email
FROM orders order0_
LEFT JOIN customers customer1_ ON order0_.customer_id = customer1_.id
Single query. All data loaded.

Why this is better:

✅ Only loads customer when you need it
✅ Still uses one query (no N+1 problem)
✅ You control when eager loading happens
✅ Different queries can load different relationships
The Fix #3: Projection (The Advanced Fix)
Sometimes you don't need the full Order object. You just need specific fields:

public interface OrderDTO {
Long getId();
String getOrderNumber();
String getCustomerName(); // Comes from customer table
BigDecimal getTotalAmount();
}

@Repository
public interface OrderRepository extends JpaRepository {

@Query("SELECT new com.example.dto.OrderDTO(" +
       "o.id, o.orderNumber, c.name, o.totalAmount) " +
       "FROM Order o " +
       "LEFT JOIN o.customer c")
List<OrderDTO> findAllOrderDTOs();

}
Hibernate generates:

SELECT order0_.id, order0_.order_number, customer1_.name, order0_.total_amount
FROM orders order0_
LEFT JOIN customers customer1_ ON order0_.customer_id = customer1_.id
Single query. Only the fields you need.

Why this is best for APIs:

✅ Single query (no N+1)
✅ Returns DTO directly (no mapping overhead)
✅ Database returns only needed columns
✅ Fastest option for REST responses
The Real-World Fix (What I Did)
In production, I did all three:

Identified N+1 queries:

Enable query logging to spot N+1 problems

spring.jpa.properties.hibernate.generate_statistics=true
logging.level.org.hibernate.stat=debug

Fixed the critical endpoints with fetch join:

@GetMapping("/api/v1/orders")
public ResponseEntity> getAllOrders(
@RequestParam(defaultValue = "0") int page,
@RequestParam(defaultValue = "50") int size) {

// Uses fetch join - single query with pagination
Page<OrderDTO> orders = orderRepository.findAllOrderDTOs(
    PageRequest.of(page, size)
);

return ResponseEntity.ok(orders.getContent());

}

Added Spring Data Specification for complex queries:

@Repository
public interface OrderRepository extends
JpaRepository,
JpaSpecificationExecutor {

// Specifications handle complex queries efficiently

}

@Service
public class OrderService {
public List searchOrders(OrderSearchCriteria criteria) {
return orderRepository.findAll((root, query, cb) -> {
Join customerJoin = root.join("customer", JoinType.LEFT);

        // Complex query with joins - still single query
        Predicate predicate = cb.and(
            cb.like(customerJoin.get("name"), criteria.getCustomerName() + "%"),
            cb.greaterThan(root.get("totalAmount"), criteria.getMinAmount())
        );

        return predicate;
    });
}

}
The Results (Before & After)
Metric Before After Improvement
Queries per request 487 1 487x ↓
Response time 2.8s 120ms 23x ↓
Database load 95% CPU 15% CPU 80% ↓
Customer complaints 47 0 100% ↓
After the fix:

Orders endpoint latency: 120ms (was 2.8s)
Database CPU dropped from 95% to 15%
Concurrent users increased from 100 to 500 without slowdown
Zero customer complaints about slow orders
How To Prevent This In The Future

Enable query counting in development:

spring.jpa.properties.hibernate.generate_statistics=true
logging.level.org.hibernate.stat=debug

Monitor queries in your tests:

@test
@Transactional
public void testOrderFetch() {
// Hibernate counts queries during test
List orders = orderService.getAllOrders();

// If this shows > 1 query, N+1 problem detected
// The test fails before production sees it

}

Use database profiling tools:

MySQL: Enable query logs and analyze slow queries
PostgreSQL: Enable auto_explain
Spring Boot Actuator: Monitor database metrics

Code review checklist:

When you see this in a PR:

List orders = orderRepository.findAll();
orders.forEach(order -> {
String customerName = order.getCustomer().getName(); // RED FLAG
});
Ask: "Is there a way to fetch this with a single query?"

The Lesson
The N+1 query problem is invisible until it hits production.

It doesn't show up in:

❌ Unit tests (usually tiny datasets)
❌ Local development (cache hides the problem)
❌ Early production (low traffic)
It explodes when:

✅ Real data volume arrives
✅ Multiple concurrent users
✅ Database connection pool saturated
✅ API timeout starts happening
Prevention is easier than debugging:

Use fetch joins for relationships
Use projections for DTOs
Monitor queries in development
Test with realistic data volumes
Next Steps
If you're experiencing slow Spring Boot APIs:

Enable Hibernate query logging
Make a single API request
Count the SQL queries
If > 2-3 queries for simple endpoint → N+1 problem
Use fetch join or projection to fix
It usually takes 30 minutes to fix and saves your users hours of waiting.

That Tuesday at 2 AM was frustrating. But it taught me something valuable: always think about how many database queries your code executes.

Your users will thank you.

Questions? Comments? Drop them below! I read every single comment and reply within 24 hours.

Related Reading
Spring Boot with Hibernate: Why Your Queries Are Slow
JPA Fetch Strategies: Lazy vs Eager Loading Explained
How to Debug Spring Boot Database Performance

Next Article: "The Connection Pool Mistake That Cost Us $5,000 in RDS Bills"

Cross-posted from my blog: Spring Boot N+1 Query Problem

Follow for more production engineering insights on Java, Spring Boot, and distributed systems.