In the fintech sector, data integrity and system reliability are paramount. When managing sensitive operations such as credit products, loan disbursements, or transaction processing, message loss or system timeouts are not merely technical incidents; they represent critical operational risks that can lead to significant financial discrepancies, such as duplicate charges or failures in loan approval workflows.
The Challenge: Synchronous Bottlenecks
Consider a credit-limit evaluation scenario. If the Credit microservice executes synchronous, blocking HTTP requests across downstream dependencies (such as the Identity service, Risk engine, and Core banking platform), a single latency spike in any of these components propagates, stalling the entire transaction flow.
This tightly coupled orchestration pattern introduces three critical operational risks:
- Cascading Failures: Service-to-service dependencies create a brittle architecture where the unavailability of a single downstream service inevitably propagates failure throughout the entire call chain.
- Scalability Bottlenecks: Synchronous communication patterns frequently trigger thread pool exhaustion under high concurrency, causing performance degradation and resource contention during peak traffic loads.
- Transactional Inconsistency: A network timeout or transient connectivity disruption mid-transaction can leave the system in a partial state, necessitating complex reconciliation logic to maintain data integrity.
The Solution: Event-Driven Architecture with Kafka
By integrating Apache Kafka as a distributed event-streaming platform, we achieve effective decoupling of our system architecture. Rather than relying on synchronous request-response chains, the primary service emits an event (such as CreditApplicationSubmitted) to a Kafka topic and returns an immediate response to the client. Downstream services (e.g., Risk, Compliance, and Notifications) consume these events asynchronously. This architectural pattern eliminates temporal coupling, ensuring that system availability and data durability remain resilient even during transient downstream service latency or downtime.
The Choice of .NET 10
.NET 10 serves as the current long-term support (LTS) release, providing the runtime stability essential for financial infrastructure, ensuring extended support without the operational overhead of frequent upgrades. It continues to deliver performance enhancements, including optimized memory management, an improved ThreadPool, and Native AOT for performance-critical workloads. In conjunction with the Confluent.Kafka NuGet package, it offers an efficient, high-performance runtime well-suited for processing high-volume financial data streams, serving as the foundation for the implementations discussed in this article.
Publishing Events: Simple Publisher Code
From a producer configuration perspective, optimizing the publishing strategy is fundamental to ensuring financial data integrity. Enabling idempotent production prevents duplicate message generation during retry scenarios, while utilizing Acks.All ensures that message acknowledgment occurs only after all in-sync replicas have persisted the event, effectively mitigating the risk of data loss in the event of a broker failure.
using Confluent.Kafka;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
public interface IEventPublisher
{
Task PublishAsync<T>(string topic, string key, T @event, CancellationToken ct);
Task PublishRawAsync(string topic, string? key, string value, CancellationToken ct);
}
public class KafkaEventPublisher : IEventPublisher, IDisposable
{
private readonly IProducer<string, string> _producer;
private readonly ILogger<KafkaEventPublisher> _logger;
public KafkaEventPublisher(IOptions<KafkaSettings> settings, ILogger<KafkaEventPublisher> logger)
{
_logger = logger;
var config = new ProducerConfig
{
BootstrapServers = settings.Value.BootstrapServers,
EnableIdempotence = true, //no duplicate writes on retry; preserves per-partition order
Acks = Acks.All, //wait for all in-sync replicas — don't lose an event to a crash
MessageSendMaxRetries = 5,
CompressionType = CompressionType.Snappy
};
_producer = new ProducerBuilder<string, string>(config).Build();
}
public Task PublishAsync<T>(string topic, string key, T @event, CancellationToken ct)
=> PublishRawAsync(topic, key, JsonSerializer.Serialize(@event), ct);
public async Task PublishRawAsync(string topic, string? key, string value, CancellationToken ct)
{
var result = await _producer.ProduceAsync(
topic, new Message<string, string> { Key = key, Value = value }, ct);
_logger.LogInformation("Published to {Topic} at {Offset}", topic, result.TopicPartitionOffset);
}
public void Dispose()
{
_producer.Flush(TimeSpan.FromSeconds(5)); // drains in-flight messages before exit
_producer.Dispose();
}
}
Consuming Events: Clean Consumer Implementation
We utilize a hosted BackgroundService to continuously poll and process messages from the Kafka cluster. The following consumer implementation incorporates core enterprise patterns and best practices, including manual offset management, graceful application shutdown, and a dead-letter queue (DLQ) strategy for handling invalid or unprocessable messages:
using Confluent.Kafka;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
public class CreditEventConsumer : BackgroundService
{
private readonly IConsumer<string, string> _consumer;
private readonly IServiceScopeFactory _scopeFactory;
private readonly IEventPublisher _publisher; // routes poison messages to the DLQ
private readonly ILogger<CreditEventConsumer> _logger;
private const string Topic = "credit-applications";
private const string DeadLetterTopic = "credit-applications-dlq";
public CreditEventConsumer(
IOptions<KafkaSettings> settings,
IServiceScopeFactory scopeFactory,
IEventPublisher publisher,
ILogger<CreditEventConsumer> logger)
{
_scopeFactory = scopeFactory;
_publisher = publisher;
_logger = logger;
var config = new ConsumerConfig
{
BootstrapServers = settings.Value.BootstrapServers,
GroupId = "credit-service-group",
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = false // commit manually, only after a message is fully processed
};
_consumer = new ConsumerBuilder<string, string>(config)
.SetErrorHandler((_, e) => _logger.LogError("Kafka error: {Reason}", e.Reason))
.Build();
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// Consume() blocks the thread. Yield first so the host can finish starting up
// (health checks, other hosted services) instead of waiting on the first poll.
await Task.Yield();
_consumer.Subscribe(Topic);
_logger.LogInformation("Subscribed to {Topic}", Topic);
while (!stoppingToken.IsCancellationRequested)
{
ConsumeResult<string, string>? result = null;
try
{
result = _consumer.Consume(stoppingToken);
if (result?.Message is null) continue;
// BackgroundService is a singleton, but our processor depends on scoped
// services (DbContext, repositories). Create a scope per message.
using var scope = _scopeFactory.CreateScope();
var processor = scope.ServiceProvider.GetRequiredService<ICreditEventProcessor>();
await processor.ProcessAsync(result.Message.Value, stoppingToken);
// Commit ONLY after processing has fully succeeded and been persisted.
_consumer.Commit(result);
}
catch (OperationCanceledException)
{
break; // graceful shutdown — not an error
}
catch (ConsumeException e)
{
_logger.LogError(e, "Kafka consume error: {Reason}", e.Error.Reason);
}
catch (Exception e)
{
// A poison message (bad data / unrecoverable failure). Don't block the
// partition — route it to the DLQ, then move the offset past it.
_logger.LogError(e, "Processing failed; routing to DLQ");
try
{
if (result is not null)
{
await _publisher.PublishRawAsync(
DeadLetterTopic, result.Message.Key, result.Message.Value, stoppingToken);
_consumer.Commit(result);
}
}
catch (Exception dlqEx)
{
// If we can't even DLQ, don't commit — let the message be redelivered.
_logger.LogCritical(dlqEx, "Failed to route message to DLQ; will retry");
}
}
}
}
public override void Dispose()
{
_consumer.Close(); // leaves the consumer group cleanly and commits final offsets
_consumer.Dispose();
base.Dispose();
}
}
Processing Events Idempotently
The consumer architecture delegates task execution to a dedicated processor, effectively separating infrastructure management (Kafka mechanics) from core business logic. In financial systems, implementing idempotency is critical. Given Kafka’s at-least-once delivery guarantee - where events may be redelivered following rebalances or network retries, processors must be designed to handle redundant messages without duplicating state changes, such as the inadvertent generation of multiple credit lines.
using System.Text.Json;
using Microsoft.Extensions.Logging;
public interface ICreditEventProcessor
{
Task ProcessAsync(string message, CancellationToken ct);
}
public class CreditEventProcessor : ICreditEventProcessor
{
private readonly ICreditRepository _repository;
private readonly ILogger<CreditEventProcessor> _logger;
public CreditEventProcessor(ICreditRepository repository, ILogger<CreditEventProcessor> logger)
{
_repository = repository;
_logger = logger;
}
public async Task ProcessAsync(string message, CancellationToken ct)
{
var evt = JsonSerializer.Deserialize<CreditApplicationSubmitted>(message)
?? throw new InvalidOperationException("Malformed credit event payload.");
// ApplyOnceAsync does the dedupe check AND the business change in one
// transaction, returning false if we've already handled this EventId.
var applied = await _repository.ApplyOnceAsync(evt, ct);
if (!applied)
_logger.LogInformation("Duplicate event {EventId} ignored", evt.EventId);
else
_logger.LogInformation("Processed credit event {EventId}", evt.EventId);
}
}
The idempotency itself lives in the repository - one transaction, so the dedupe record and the business change commit or roll back together:
public async Task<bool> ApplyOnceAsync(CreditApplicationSubmitted evt, CancellationToken ct)
{
await using var conn = await _db.OpenAsync(ct);
await using var tx = await conn.BeginTransactionAsync(ct);
// INSERT ... ON CONFLICT DO NOTHING returns 0 rows if we've seen this EventId before.
var inserted = await conn.ExecuteAsync(
"INSERT INTO processed_events (event_id) VALUES (@id) ON CONFLICT DO NOTHING",
new { id = evt.EventId }, tx);
if (inserted == 0)
{
await tx.RollbackAsync(ct);
return false; // already processed, skip
}
// ... the real credit decision / balance update, in the SAME transaction ...
await tx.CommitAsync(ct);
return true;
}
Final Important Bit: The Transactional Outbox
However cleanly implemented the publisher is, there is still a subtle trap. Persisting data to a database and publishing events to Kafka constitute two distinct operations; consequently, a system failure occurring between these steps can result in state divergence. If an application approves a credit request but fails to emit the corresponding event, transactional inconsistency is introduced. In critical financial infrastructure, such discrepancies are severe incidents that require immediate operational attention.
The fix is the transactional outbox: write the event into an outbox table in the same transaction as your business data, so they commit together or not at all.
await conn.ExecuteAsync("INSERT INTO credit_applications ...", app, tx);
await conn.ExecuteAsync("INSERT INTO outbox ...", outboxRow, tx);
await tx.CommitAsync(ct); // both commit together, or neither does - atomicity
A background relay service periodically polls the outbox, publishing each entry to Kafka. Since this process may successfully publish an event but fail to update its status before a failure occurs, it inherently guarantees at-least-once delivery. This operational characteristic necessitates consumer-side idempotency, as previously discussed. Ultimately, robust systems require both producer-side and consumer-side idempotency to ensure end-to-end data integrity.
Wiring it up
//Add below lines in your Program.cs file
builder.Services.Configure<KafkaSettings>(builder.Configuration.GetSection("KafkaConfig"));
builder.Services.AddSingleton<IEventPublisher, KafkaEventPublisher>();
builder.Services.AddScoped<ICreditEventProcessor, CreditEventProcessor>();
builder.Services.AddScoped<ICreditRepository, CreditRepository>();
builder.Services.AddHostedService<CreditEventConsumer>();
Conclusion
Moving off the synchronous path changes the failure modes in your favor:
- Isolation: If an external payment gateway crashes, its events sit durably in Kafka until it recovers - user requests aren't blocked on it and nothing is dropped.
- Throughput that scales with you: Consumption is bounded by your consumers and partitions rather than by the slowest synchronous dependency. Need more headroom? Add consumers to the group.
- Steadier databases under load: Work is buffered in the topic and drained at a controlled pace, so a traffic spike no longer translates into a synchronous stampede against your primary database.
None of this is free - you're trading immediate consistency for eventual consistency, and you take on idempotency, dead-letter handling, and offset discipline as the price of admission. But for financial workloads where losing a message is unacceptable, that's a trade well worth making.
Thank you for reading through. Please do well to leave me your comments and/or questions on this post in the comments section.
I'm a Technical Lead specializing in building, scaling, and optimizing backend systems with C# and cloud technologies. If you'd like to talk about distributed systems, event-driven architecture, or mentorship, connect with me on LinkedIn.
Top comments (0)