DEV Community

Ankit Sood
Ankit Sood

Posted on

🚀 How We Preserved Event Order in Kafka While Streaming From Cosmos DB: A Real-World Journey

#cosmosdb #eventdriven #springboot #outbox

Just publish the event after saving to the database. How hard can it be?

Famous last words. 😅

What started as a simple requirement in one of my projects quickly turned into a deep dive on ordering guarantees, idempotence, and the quirks of using Cosmos DB’s Mongo API.

This is the story of how we used the Outbox Pattern, Spring Boot, and Apache Kafka to build a resilient system that preserved strict ordering of events—even in the face of failures.

🎯 Goal

We were building an accounts payable backend on Azure:

📦 Cosmos DB (Mongo API) for storing invoices.
🔥 Apache Kafka as the backbone for event-driven workflows.
🌱 Spring Boot for application logic.

🎯 Requirements

✅ When a user places an order, save it to the database
✅ Immediately publish an OrderConfirmed event to Kafka
✅ Ensure downstream consumers always process events in order

Sounds simple enough… right?

🎯 Approaches

😅 The First Naive Attempt and Here’s how we started:

// 1. Save order
mongoTemplate.insert(order);

// 2. Publish to Kafka
kafkaTemplate.send("order-events", orderId, payload);
Enter fullscreen mode Exit fullscreen mode

This worked for some time But pretty soon, things blew up.

🔥 Problem #1: Lost Events
If Kafka was down after the DB write, the event was lost forever. Downstream services never found out about the order.

🔥 Problem #2: Out-of-Order Delivery
Even with retries, we saw scenarios like:

OrderShipped(order-123) arrives BEFORE OrderConfirmed(order-123)
Enter fullscreen mode Exit fullscreen mode

This broke downstream systems relying on proper sequencing.

🧩 Enter the Outbox Pattern to save the day

Instead of trying to send events directly to Kafka, we:

  • Extended our DB schema with an outbox collection
  • When saving an order, also saved a pending outbox event in the same transaction
  • Built a background publisher to read outbox events and send them to Kafka in order
  • Marked events as "SENT" only after Kafka acknowledged

Here’s the flow:

[App writes Order + Outbox event] → [Outbox Processor] → [Kafka]
Enter fullscreen mode Exit fullscreen mode

📦 Outbox pattern acts as a reliable buffer between your DB and Kafka. This pattern works well with any SQL Database but doesn't work with Cosmos DB mongo API.

⚡ Challenges With Cosmos DB Mongo API

We knew Cosmos DB’s Mongo API isn’t MongoDB and Transactions only work if:
✅ Both writes are in the same collection
✅ Documents share the same partitionKey

So we designed a shared collection:

🗄 Structure of Documents

  • Order Document 
{
  "_id": "order-123",
  "type": "order",
  "userId": "user-456",
  "status": "confirmed",
  "partitionKey": "user-456"
}
Enter fullscreen mode Exit fullscreen mode
  • Outbox Event Document 
{
  "_id": "event-789",
  "type": "event",
  "eventType": "OrderConfirmed",
  "payload": { "orderId": "order-123" },
  "createdAt": "2025-06-27T12:00:00Z",
  "status": "PENDING",
  "partitionKey": "user-456"
}
Enter fullscreen mode Exit fullscreen mode

Both uses the same partitionKey to enable transactions.

💻 The Implementation:

We have used Java and Spring boot to implement this. Below is how we put it all together:

🏗 Atomic Write Service
We ensured order + outbox event writes were atomic:

@Service
public class OrderService {
    @Autowired private MongoTemplate mongoTemplate;

    @Transactional
    public void createOrderWithEvent(Order order, OutboxEvent event) {
        mongoTemplate.insert(order, "sharedCollection");
        mongoTemplate.insert(event, "sharedCollection");
    }
}
Enter fullscreen mode Exit fullscreen mode

🚀 Kafka Outbox Publisher
A scheduled Spring component polls pending outbox events and sends them to Kafka in creation order.

@Component
public class OutboxEventPublisher {
    @Autowired private MongoTemplate mongoTemplate;
    @Autowired private KafkaTemplate<String, String> kafkaTemplate;

    @Scheduled(fixedDelay = 1000)
    public void publishEvents() {
        Query query = Query.query(Criteria.where("type").is("event").and("status").is("PENDING"))
                .with(Sort.by("createdAt")).limit(10);

        List<OutboxEvent> events = mongoTemplate.find(query, OutboxEvent.class, "sharedCollection");

        for (OutboxEvent event : events) {
            try {
                kafkaTemplate.send("order-events", event.getPartitionKey(), event.getPayload()).get();

                // Mark as SENT
                mongoTemplate.updateFirst(
                    Query.query(Criteria.where("_id").is(event.getId())),
                    Update.update("status", "SENT"),
                    "sharedCollection"
                );

                System.out.println("Published event: " + event.getId());

            } catch (Exception e) {
                System.err.println("Kafka send failed for event: " + event.getId());
                break; // Stop to avoid out-of-order sends
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

🔐 Kafka Producer Configuration
We enabled idempotence and sync sends to preserve order:

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);
Enter fullscreen mode Exit fullscreen mode

🏗 Transaction Manager Configuration
Spring’s @Transactional is not same as Transaction for Cosmos or Mongo.

  • It's designed for relational databases (JPA, JDBC, etc.)

  • It Relies on a transaction manager like PlatformTransactionManager

  • For MongoDB, it needs MongoDB 4.x+ with replica sets and Spring Data MongoDB transaction support.

@Bean
    MongoTransactionManager transactionManager(MongoDatabaseFactory dbFactory) {
        return new MongoTransactionManager(dbFactory);
    }
Enter fullscreen mode Exit fullscreen mode

To use MongoDB transactions via Spring Boot, all the below things must be true:
✅ MongoDB server version ≥ 4.0
✅ Connected to a replica set (Cosmos DB Mongo API v4.0+ does support transactions but only under certain conditions)
✅ Use MongoTransactionManager explicitly in Spring

But Cosmos DB Mongo API:
⚠ Only supports transactions within the same partition (documents must share partitionKey)
⚠ Has subtle differences from native MongoDB transactions

In case, a transaction should be started explicitly, we can use

try (ClientSession session = mongoTemplate.getMongoDbFactory().getMongoClient().startSession()) {
    session.startTransaction();

    try {
        mongoTemplate.withSession(() -> session).insert(order, "sharedCollection");
        mongoTemplate.withSession(() -> session).insert(event, "sharedCollection");
        session.commitTransaction();
    } catch (Exception e) {
        session.abortTransaction();
        throw e;
    }
}
Enter fullscreen mode Exit fullscreen mode

🎯 Final Thoughts

The Outbox Pattern helped us bridge the gap between Cosmos DB and Kafka. It’s a robust, production-ready solution for ordered, reliable event streaming—even when your DB doesn’t support native CDC.

CDC was one more approach that we could have used. In the past, we have used cosmos change stream which had its own issues and hence we decided to go ahead with this.

👉 Have you faced ordering challenges in Kafka pipelines? How did you solve them? Share your experiences below!

References

Top comments (0)