Just publish the event after saving to the database. How hard can it be?
Famous last words. 😅
What started as a simple requirement in one of my projects quickly turned into a deep dive on ordering guarantees, idempotence, and the quirks of using Cosmos DB’s Mongo API.
This is the story of how we used the Outbox Pattern, Spring Boot, and Apache Kafka to build a resilient system that preserved strict ordering of events—even in the face of failures.
🎯 Goal
We were building an accounts payable backend on Azure:
📦 Cosmos DB (Mongo API) for storing invoices.
🔥 Apache Kafka as the backbone for event-driven workflows.
🌱 Spring Boot for application logic.
🎯 Requirements
✅ When a user places an order, save it to the database
✅ Immediately publish an OrderConfirmed event to Kafka
✅ Ensure downstream consumers always process events in order
Sounds simple enough… right?
🎯 Approaches
😅 The First Naive Attempt and Here’s how we started:
// 1. Save order
mongoTemplate.insert(order);
// 2. Publish to Kafka
kafkaTemplate.send("order-events", orderId, payload);
This worked for some time But pretty soon, things blew up.
🔥 Problem #1: Lost Events
If Kafka was down after the DB write, the event was lost forever. Downstream services never found out about the order.
🔥 Problem #2: Out-of-Order Delivery
Even with retries, we saw scenarios like:
OrderShipped(order-123) arrives BEFORE OrderConfirmed(order-123)
This broke downstream systems relying on proper sequencing.
🧩 Enter the Outbox Pattern to save the day
Instead of trying to send events directly to Kafka, we:
- Extended our DB schema with an outbox collection
- When saving an order, also saved a pending outbox event in the same transaction
- Built a background publisher to read outbox events and send them to Kafka in order
- Marked events as "SENT" only after Kafka acknowledged
Here’s the flow:
[App writes Order + Outbox event] → [Outbox Processor] → [Kafka]
📦 Outbox pattern acts as a reliable buffer between your DB and Kafka. This pattern works well with any SQL Database but doesn't work with Cosmos DB mongo API.
⚡ Challenges With Cosmos DB Mongo API
We knew Cosmos DB’s Mongo API isn’t MongoDB and Transactions only work if:
✅ Both writes are in the same collection
✅ Documents share the same partitionKey
So we designed a shared collection:
🗄 Structure of Documents
- Order Document
{
"_id": "order-123",
"type": "order",
"userId": "user-456",
"status": "confirmed",
"partitionKey": "user-456"
}
- Outbox Event Document
{
"_id": "event-789",
"type": "event",
"eventType": "OrderConfirmed",
"payload": { "orderId": "order-123" },
"createdAt": "2025-06-27T12:00:00Z",
"status": "PENDING",
"partitionKey": "user-456"
}
Both uses the same partitionKey to enable transactions.
💻 The Implementation:
We have used Java and Spring boot to implement this. Below is how we put it all together:
🏗 Atomic Write Service
We ensured order + outbox event writes were atomic:
@Service
public class OrderService {
@Autowired private MongoTemplate mongoTemplate;
@Transactional
public void createOrderWithEvent(Order order, OutboxEvent event) {
mongoTemplate.insert(order, "sharedCollection");
mongoTemplate.insert(event, "sharedCollection");
}
}
🚀 Kafka Outbox Publisher
A scheduled Spring component polls pending outbox events and sends them to Kafka in creation order.
@Component
public class OutboxEventPublisher {
@Autowired private MongoTemplate mongoTemplate;
@Autowired private KafkaTemplate<String, String> kafkaTemplate;
@Scheduled(fixedDelay = 1000)
public void publishEvents() {
Query query = Query.query(Criteria.where("type").is("event").and("status").is("PENDING"))
.with(Sort.by("createdAt")).limit(10);
List<OutboxEvent> events = mongoTemplate.find(query, OutboxEvent.class, "sharedCollection");
for (OutboxEvent event : events) {
try {
kafkaTemplate.send("order-events", event.getPartitionKey(), event.getPayload()).get();
// Mark as SENT
mongoTemplate.updateFirst(
Query.query(Criteria.where("_id").is(event.getId())),
Update.update("status", "SENT"),
"sharedCollection"
);
System.out.println("Published event: " + event.getId());
} catch (Exception e) {
System.err.println("Kafka send failed for event: " + event.getId());
break; // Stop to avoid out-of-order sends
}
}
}
}
🔐 Kafka Producer Configuration
We enabled idempotence and sync sends to preserve order:
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);
🏗 Transaction Manager Configuration
Spring’s @Transactional is not same as Transaction for Cosmos or Mongo.
It's designed for relational databases (JPA, JDBC, etc.)
It Relies on a transaction manager like PlatformTransactionManager
For MongoDB, it needs MongoDB 4.x+ with replica sets and Spring Data MongoDB transaction support.
@Bean
MongoTransactionManager transactionManager(MongoDatabaseFactory dbFactory) {
return new MongoTransactionManager(dbFactory);
}
To use MongoDB transactions via Spring Boot, all the below things must be true:
✅ MongoDB server version ≥ 4.0
✅ Connected to a replica set (Cosmos DB Mongo API v4.0+ does support transactions but only under certain conditions)
✅ Use MongoTransactionManager explicitly in Spring
But Cosmos DB Mongo API:
⚠ Only supports transactions within the same partition (documents must share partitionKey)
⚠ Has subtle differences from native MongoDB transactions
In case, a transaction should be started explicitly, we can use
try (ClientSession session = mongoTemplate.getMongoDbFactory().getMongoClient().startSession()) {
session.startTransaction();
try {
mongoTemplate.withSession(() -> session).insert(order, "sharedCollection");
mongoTemplate.withSession(() -> session).insert(event, "sharedCollection");
session.commitTransaction();
} catch (Exception e) {
session.abortTransaction();
throw e;
}
}
🎯 Final Thoughts
The Outbox Pattern helped us bridge the gap between Cosmos DB and Kafka. It’s a robust, production-ready solution for ordered, reliable event streaming—even when your DB doesn’t support native CDC.
CDC was one more approach that we could have used. In the past, we have used cosmos change stream which had its own issues and hence we decided to go ahead with this.
👉 Have you faced ordering challenges in Kafka pipelines? How did you solve them? Share your experiences below!
Top comments (0)