🧠 Handling Duplicacy in Microservices During Retry Scenarios

In distributed systems and microservices, failures and retries are inevitable — whether it's a temporary network glitch, database latency, or an API failure. To improve reliability, services often retry failed operations. However, this retry behavior can lead to data duplicacy issues, such as:

• Duplicate records in the database
• Duplicate payment processing
• Duplicate email/SMS notifications
• Duplicate Kafka messages

In this blog, we'll explore how to identify, prevent, and design resilient microservices to handle duplicate processing during retries.

---

🕸️ The Problem: Why Duplicacy Happens

Let’s take an example:

Service A → [POST] → Service B (Create Order)

Service B returns HTTP 500 or times out due to network issue.
Service A retries the POST call.
Service B now processes the same request twice.

🔁 Causes of Duplicate Processing:

• Network timeout (client doesn't get a response)
• Service crash after processing but before acknowledgment
• Message broker retries (e.g., Kafka, RabbitMQ)
• Manual replays
• Poorly implemented idempotent APIs

---

✅ Design Strategies to Handle Duplicacy

Here are proven strategies to make your microservices resilient and idempotent:

---

1. 🧾 Idempotency Key Pattern

An idempotency key is a unique key passed by the client with the request. The server uses it to detect and ignore duplicate requests.

🔨 How it Works:

• Client generates a UUID for each operation (e.g., X-Idempotency-Key: abc123)
• Server checks if the key has been processed before
• If yes, return the stored response
• If not, process and store result against the key

✅ Benefits:

• Safe retries from the client
• No duplicate database writes or external API calls

💻 Code Snippet (Spring Boot):

@PostMapping("/orders")
public ResponseEntity<?> createOrder(
    @RequestHeader("X-Idempotency-Key") String idempotencyKey,
    @RequestBody OrderRequest request) {

    Optional<Order> existing = orderRepository.findByIdempotencyKey(idempotencyKey);
    if (existing.isPresent()) {
        return ResponseEntity.ok(existing.get());
    }

    Order newOrder = processOrder(request);
    newOrder.setIdempotencyKey(idempotencyKey);
    orderRepository.save(newOrder);

    return ResponseEntity.ok(newOrder);
}

---

2. 📦 Deduplication Using Unique Constraints in DB

Use unique constraints in the database to prevent multiple inserts for the same key or identifier.

Example:

CREATE TABLE payments (
  id UUID PRIMARY KEY,
  transaction_id VARCHAR UNIQUE,
  amount DECIMAL
);

🔄 Retry-safe flow:

• Retry inserts the same transaction ID
• If already inserted, DB throws a DuplicateKeyException
• Handle it gracefully in the application

---

3. 🛡️ Idempotent APIs

Design APIs to be idempotent by nature, especially PUT and DELETE operations.

Characteristics:

• Repeating the same request doesn’t change the state beyond the initial application
• Safe for retries

Example:

PUT /users/123
Body: { "name": "Nitesh" }

Can be called multiple times with the same effect.

---

4. 🎯 Use of External Request IDs or Correlation IDs

Maintain and propagate a request ID across services. Each downstream service logs and checks if the request has already been processed.

Flow:

1. Gateway generates X-Request-ID
2. All downstream services use this ID to check for duplicates

---

5. 🧵 Exactly-Once Semantics in Messaging

In asynchronous systems (Kafka, RabbitMQ), exactly-once processing is complex but can be mimicked by:

• Storing processed message IDs
• Using transactional outbox and change data capture (CDC)
• Kafka's idempotent producer + transactional consumer

Pseudo-Code:

if (messageStore.exists(messageId)) {
    return; // duplicate message
}
messageStore.save(messageId);
processMessage();

---

6. 🕵️‍♂️ Distributed Locks or Leases

In critical sections (e.g., payment processing), use Redis locks or database-based leases to ensure only one instance processes the job.

Redis Example:

Boolean lockAcquired = redisTemplate.opsForValue().setIfAbsent(lockKey, value, 10, TimeUnit.SECONDS);
if (!lockAcquired) {
    return; // already being processed
}

---

🧪 Real-World Use Cases

Use Case	Strategy
Payment gateway	Idempotency key + DB unique constraint
Email notification	Request ID + message deduplication
File uploads	File checksum (hash) as key
Kafka consumer	Processed message ID tracking
Order creation	Idempotent API + unique constraint

---

🧰 Best Practices

✅ Always design critical operations (like payment, billing, account creation) to be idempotent.
✅ Use request IDs and correlation IDs for end-to-end tracing.
✅ Use dedicated tables for storing processed operations or keys.
✅ Log enough metadata to identify duplicates in logs.
✅ Keep retry mechanisms smart, i.e., exponential backoff, max attempts.
✅ Add visibility and alerts for duplicate operation detection.

---

🧩 Conclusion

Retries are good — they improve reliability. But without deduplication, they can lead to data corruption, financial loss, or poor user experience. Building idempotent APIs and resilient system design practices is essential for any microservices architecture.

By using idempotency keys, unique constraints, proper message deduplication, and robust tracing, we can ensure safe and repeatable operations, even in the face of inevitable failures.

---

📌 What’s Next?

In upcoming blogs, we’ll cover:

• Implementing exactly-once semantics in Kafka consumers
• Using transactional outbox patterns to prevent double delivery
• Handling network partitions and consistency tradeoffs

Stay tuned!