DEV Community: Zeliot

Kafka Partition Strategy: The Complete Guide to Designing for Scale from Day One

Sachin Kamath — Tue, 21 Jul 2026 07:40:42 +0000

TL;DR

Kafka partition strategy is one of the most important architectural decisions when building scalable event-driven applications. It determines how data is distributed across partitions, sets the upper limit for consumer parallelism, influences message ordering, and directly impacts throughput, broker utilization, and cluster performance. Choosing the right Kafka partition count and partition key from the start helps prevent hotspots, balance workloads, and simplify future scaling. Condense is a Kafka-native unified data streaming platform that simplifies the entire partition lifecycle through Kafka Management, enabling engineering teams to configure partitions during topic creation, scale partition counts as workloads grow, monitor partition health and consumer lag, inspect partition-level data, and manage rebalancing from a unified interface. This guide explains how to design an effective Kafka partition strategy, calculate the right partition count, choose the right partition key, avoid common partitioning mistakes, and scale topics confidently in production

Introduction

Apache Kafka is designed to scale horizontally, but that scalability depends on one fundamental architectural decision: the partition strategy. Every topic in Kafka is divided into partitions, which determine how data is distributed across brokers, how many consumers can process events in parallel, and whether related events maintain their ordering. A well-designed partition strategy enables applications to scale predictably as data volumes grow, while a poor strategy can create bottlenecks that become increasingly difficult to resolve in production.

Partition strategy extends beyond simply deciding how many partitions a topic should have. It involves calculating the right partition count based on throughput requirements, selecting partition keys that balance workloads without compromising ordering guarantees, avoiding hotspots caused by uneven data distribution, and planning for future growth. Since partitioning decisions influence producers, consumers, stream processing applications, and downstream systems, modifying them after deployment often requires careful migration planning.

While Apache Kafka provides the flexibility to configure and scale partitions, managing them across production environments introduces additional operational complexity. Condense simplifies this process through its Kafka Management capabilities, allowing engineering teams to configure partition counts during topic creation, increase partitions as workloads grow, inspect partition-level data, monitor consumer lag and partition assignments, and manage consumer rebalancing from a unified interface. This enables teams to implement and operate scalable partition strategies more efficiently while continuing to leverage the flexibility of Apache Kafka.

In this guide, we'll explore why Kafka partition strategy is one of the most important design decisions in event streaming, how to calculate the right partition count, design effective partition keys, prevent hotspots, and safely scale partitions as application workloads evolve.

Why Is Kafka Partition Strategy the Most Consequential Design Decision?

Every Kafka topic is divided into one or more partitions, making partitions the fundamental unit of scalability, parallel processing, and fault tolerance. As records are written to a topic, Kafka distributes them across these partitions, allowing producers and consumers to process data concurrently. Because of this, the partition strategy directly influences application throughput, consumer parallelism, message ordering, broker utilization, and the overall efficiency of the Kafka cluster.

Unlike configuration settings such as retention policies, compression, or replication factors, partitioning decisions become deeply integrated into application architecture. Producers determine how records are distributed using partition keys, consumers process records based on partition assignments, and stateful stream processing applications rely on consistent partitioning to maintain local state and perform joins or aggregations. Changing the partition strategy after applications are deployed often requires creating new topics, migrating producers and consumers, and validating downstream processing, making it one of the most expensive architectural changes in a Kafka deployment.

A well-designed partition strategy delivers several benefits:

Maximizes producer and consumer throughput
Enables horizontal scaling through partition-level parallelism
Preserves message ordering for related events
Balances workload evenly across brokers
Reduces the risk of hotspots and consumer bottlenecks
Simplifies long-term capacity planning

Conversely, a poorly designed strategy can create challenges that become more pronounced as workloads grow. Topics with too few partitions restrict consumer parallelism, ineffective partition keys create uneven workload distribution, and insufficient planning for future growth can force disruptive repartitioning efforts in production.

Although partition strategy is an architectural decision, implementing and managing that strategy at scale requires operational visibility. Condense simplifies Kafka partition management through its Kafka Management capabilities, enabling engineering teams to configure partitions during topic creation, increase partition counts as workloads grow, monitor partition health and consumer lag, inspect partition-level messages, and manage consumer group rebalancing from a unified interface. This allows teams to focus on designing the right partition strategy while simplifying its implementation and ongoing operation.

Key Takeaways

Kafka Partition Strategy Influences	Why It Matters
Consumer Parallelism	Determines the maximum number of active consumers in a consumer group
Throughput	Controls how much data producers and consumers can process concurrently
Message ordering	Preserves the sequence of related events within a partition
Broker Utilization	Distributes storage and processing workloads across the cluster
Scalability	Enables topics to grow with increasing data volumes
Operational Complexity	Poor partitioning decisions are difficult and costly to change after deployment

How Does Kafka Partition Count Determine Consumer Parallelism?

One of Kafka's biggest strengths is its ability to process large volumes of data in parallel. However, the level of parallelism that a Kafka application can achieve is determined by a single factor: the number of partitions in a topic.

Each partition acts as an independent, ordered log that can be processed by only one consumer within a consumer group at any given time. This guarantees message ordering within the partition while allowing multiple partitions to be processed simultaneously by different consumers. As a result, the number of partitions defines the maximum number of consumers that can actively process records in parallel.

Consumer Parallelism in Action

Consider a Kafka topic with four partitions.

Topic Partitions	Consumers in the Consumer Group	Active Consumers	Idle Consumers
4	2	2	0
4	4	4	0
4	6	4	2

With four partitions:

Two consumers process two partitions, leaving two partitions assigned across the available consumers.
Four consumers can each process one partition simultaneously.
Adding more than four consumers does not increase throughput because there are no additional partitions to assign.

This limitation is commonly referred to as the consumer parallelism ceiling. Regardless of how many application instances are deployed, a consumer group cannot process records in parallel beyond the number of partitions available in the topic.

Why Consumer Parallelism Matters

Imagine a connected mobility platform ingesting telemetry from hundreds of thousands of vehicles. As more vehicles come online, the volume of incoming events increases significantly. If the telemetry topic has only four partitions, only four consumers can actively process those events at any given time. Deploying additional consumer instances does not improve throughput because the partition count limits parallel processing.

By increasing the partition count, Kafka can distribute the workload across more consumers, allowing the application to scale horizontally as data volumes grow.

More Partitions Don't Always Mean Better Performance

Although increasing the number of partitions enables greater consumer parallelism, creating excessive partitions introduces additional operational overhead. Every partition consumes broker resources, increases metadata management, extends consumer group rebalancing, and adds complexity to cluster operations.

The goal is to provision enough partitions to support current and future workloads without creating unnecessary operational overhead.

How Condense Simplifies Partition Scaling

As applications evolve, maintaining the right balance between partition count and consumer parallelism becomes an ongoing operational task. Condense simplifies this through Kafka Management, allowing engineering teams to configure partition counts during topic creation and increase partitions as workloads grow. Combined with partition-level observability, consumer lag monitoring, and consumer group health tracking, teams can validate whether additional partitions are improving throughput and parallelism without manually correlating broker-level metrics.

Best Practice: Size Kafka topics based on the maximum consumer parallelism your application is expected to require over time, not just the number of consumers deployed today. As workloads grow, periodically review partition utilization and consumer lag to determine whether increasing the partition count is necessary.

How Do You Calculate the Right Kafka Partition Count?

Choosing the right Kafka partition count is one of the most important capacity planning decisions when designing an event streaming platform. Too few partitions can limit throughput and consumer parallelism, while too many increase broker overhead, metadata management, and operational complexity. Instead of relying on fixed recommendations, partition count should be determined using expected throughput, consumer processing capacity, and future growth projections.

A commonly used approach is to calculate the partition count based on both producer and consumer throughput requirements and choose the larger value.

Kafka Partition Count Formula

Partition Count = Max ( 
    Target Producer Throughput ÷ Producer Throughput per Partition, 
    Target Consumer Throughput ÷ Consumer Throughput per Partition 
)

Where:

Target Producer Throughput is the expected rate at which data is produced.
Producer Throughput per Partition is the maximum throughput a single partition can sustain.
Target Consumer Throughput is the rate at which data must be processed.
Consumer Throughput per Partition is the maximum processing capacity of a single partition.

Example Calculation

Consider a real-time streaming application with the following requirements:

Metric	Value
Target throughput	600 MB/s
Producer throughput per partition	50 MB/s
Consumer throughput per partition	30 MB/s

Producer calculation:

600 ÷ 50 = 12 partitions

Consumer calculation:

600 ÷ 30 = 20 partitions

Recommended partition count:

Max(12, 20) = 20 partitions

Although producers can achieve the required throughput with 12 partitions, consumers require 20 partitions to process data without creating bottlenecks. In this scenario, 20 partitions is the appropriate starting point.

Factors That Influence Partition Count

There is no universal recommendation for the ideal number of Kafka partitions. The optimal partition count depends on several workload-specific factors, including:

Message size
Producer batching and compression
Replication factor
Acknowledgement (acks) configuration
Consumer processing complexity
Broker CPU, memory, and storage performance
Expected traffic growth
Maximum consumer parallelism

Benchmarking representative workloads is significantly more reliable than relying on generic partition sizing recommendations.

Plan for Growth, Not Just Today's Traffic

Kafka allows partition counts to be increased after a topic is created, but increasing partitions changes how future records are distributed across the topic. While this makes scaling possible, resizing production topics still requires planning, especially for applications that depend on key-based routing or message ordering.

A better approach is to estimate expected workload growth over the next two to three years and provision enough partitions to support that scale from the beginning. This reduces the need for repartitioning as applications evolve.

How Condense Simplifies Partition Planning

Once the required partition count has been determined, Condense simplifies implementation through Kafka Management. Engineering teams can configure partition count, replication factor, retention policies, cleanup policies, and segment settings when creating a topic from a unified interface. As workloads grow, partition counts can be increased without recreating the topic, allowing streaming applications to scale while continuing to operate on the same Kafka deployment. Combined with partition-level observability and consumer lag monitoring, teams can validate whether additional partitions are delivering the expected improvements in throughput and parallelism.

Best Practice: Calculate Kafka partition count using measured throughput, expected consumer parallelism, and future growth projections. Configure topics with sufficient partitions during initial deployment, and periodically review partition utilization as workloads evolve.

How Should You Design a Kafka Partition Key?

Choosing the right Kafka partition count determines how much a topic can scale, but choosing the right partition key determines how efficiently that scale is utilized. The partition key controls how records are distributed across partitions, whether related events remain ordered, and how evenly workloads are balanced across brokers and consumers. An ineffective partition key can create hotspots, uneven resource utilization, and reduced application throughput even when a topic has an adequate number of partitions.

When a producer sends a record with a key, Kafka uses a hashing algorithm to determine the destination partition.

Partition = hash(partition_key) % number_of_partitions

This calculation ensures that every record with the same partition key is consistently routed to the same partition, preserving message ordering for that key.

Why Does the Partition Key Matter?

The partition key influences several critical aspects of a Kafka deployment:

Distribution of records across partitions
Message ordering for related events
Consumer workload distribution
Stateful stream processing
Stream joins and aggregations
Overall cluster efficiency

Because the partition key affects both application correctness and cluster performance, it should be selected based on business requirements rather than convenience.

Semantic Keys vs Random Keys

Partition keys generally fall into two categories.

Semantic Keys	Random Keys
Customer ID	UUID
Vehicle ID	Random String
Device ID	Auto-generated Identifier
Order ID	Null Key (Round-Robin Distribution)

When Should You Use Semantic Keys?

Semantic keys represent a business entity whose events must remain in sequence.

Examples include:

Customer ID
Vehicle ID
Device ID
Order ID
Account ID

For example, if every telemetry event generated by Vehicle_1042 uses the vehicle ID as its partition key, all events for that vehicle are written to the same partition.

> Vehicle_1042 
 
  Engine Started 
        ↓ 
  Location Updated 
        ↓ 
  Speed Updated 
        ↓ 
  Engine Stopped

Because all events remain within a single partition, Kafka preserves their order throughout processing.

Semantic keys are recommended when applications require:

Ordered event processing
Stateful stream processing
Event sourcing
Stream joins and aggregations
Entity state management

When Should You Use Random or Null Keys?

Not every workload requires message ordering.

If records are independent and can be processed in any sequence, producers can use random keys or publish records without specifying a partition key. Kafka distributes these records across partitions, resulting in a more even workload distribution.

Random or null keys are commonly used for:

Log ingestion
Metrics collection
Clickstream analytics
Independent telemetry events
High-throughput event ingestion

The trade-off is that events for the same entity may be written to different partitions, meaning Kafka no longer guarantees their relative order.

Designing an Effective Partition Key

A good partition key should:

Preserve ordering where required.
Have high cardinality to distribute traffic evenly.
Remain stable throughout the application's lifecycle.
Support future scaling without creating hotspots.
Reflect the application's processing requirements rather than temporary business attributes.

Selecting fields such as Country, Region, Status, or Event Type often results in uneven traffic distribution because these values have relatively low cardinality. Business identifiers with many unique values typically provide a much better balance across partitions.

How Condense Simplifies Partition Validation

Choosing a partition key is only the first step. Validating that it distributes records as expected is equally important in production.

Condense simplifies this through Kafka Management, allowing engineering teams to inspect messages at the partition level by selecting a specific partition, browsing records from any offset, viewing messages in JSON, Avro, or Protobuf formats, and publishing test messages directly to a chosen partition. These capabilities make it easier to verify partition key behavior, troubleshoot routing issues, and understand how records are distributed across Kafka topics without relying on external tooling.

Best Practice: Choose partition keys based on application behavior rather than convenience. High-cardinality business identifiers generally provide the best balance between message ordering, workload distribution, and long-term scalability. Validate the chosen strategy early using partition-level inspection before workloads reach production scale.

How Do Poor Partition Keys Create Hotspots in Kafka Clusters?

Even with the right number of partitions, a Kafka topic can still suffer from poor performance if records are distributed unevenly across those partitions. This typically happens when an ineffective partition key causes a disproportionate amount of traffic to be routed to a small number of partitions, creating hotspots.

A hotspot occurs when one or more partitions receive significantly more records than others. Since each partition is processed independently, these overloaded partitions become bottlenecks, increasing consumer lag, producer latency, and broker resource utilization while other partitions remain underutilized.

What Causes Partition Hotspots?

Hotspots are usually caused by partition keys with low cardinality or uneven business data distribution.

Consider an application that uses Country as the partition key.

Partition Key	Percentage of Incoming Events
India	82%
United States	8%
Germany	4%
Other Countries	6%

Although the topic may contain multiple partitions, most events associated with India are consistently routed to the same partition. As traffic increases, that partition becomes overloaded while the remaining partitions process comparatively little data.

This imbalance reduces the effectiveness of Kafka's parallel processing model.

Operational Impact of Hot Partitions

Partition hotspots affect the entire streaming pipeline, not just a single consumer.

Component	Impact
Producers	Increased request latency and retries
Consumers	Higher consumer lag and slower event processing
Brokers	Uneven CPU, memory, disk, and network utilization
Storage	Faster growth of individual partitions
Applications	Reduced throughput and inconsistent performance

Over time, the busiest partition becomes the limiting factor for the application, regardless of the available capacity across the rest of the cluster.

Common Partition Key Mistakes

The following partition keys frequently lead to workload imbalance.

Avoid Using	Why It Creates Hotspots	Better Alternative
Country	Uneven geographic traffic	Customer ID
Region	Low cardinality	Device ID
Status	Limited possible values	Order ID
Event Type	Popular events dominate traffic	Vehicle ID
Boolean Fields	Only two possible values	Session ID

The objective is to select partition keys with high cardinality, allowing Kafka to distribute records more evenly across partitions while preserving ordering where required.

Detecting Hotspots Before They Become Production Issues

Traffic patterns evolve over time. A partition key that distributes records evenly today may become imbalanced as specific customers, devices, or tenants generate significantly more events than others.

Engineering teams should continuously monitor:

Consumer lag by partition
Records processed per partition
Partition assignment across consumers
Broker CPU and memory utilization
Partition storage growth
Consumer group health

Monitoring these metrics helps identify workload imbalance before it impacts application performance.

How Condense Simplifies Hotspot Detection

Condense provides partition-level visibility through its Kafka Management and observability capabilities, making it easier to identify workload imbalance as traffic patterns evolve. Engineering teams can monitor consumer lag for individual partitions, track partition ownership and assignments across consumer groups, view committed offsets and lag, and monitor consumer group health, including stable, rebalancing, and inactive states. These insights help teams detect hotspots early, validate whether workloads are evenly distributed, and determine when partition scaling or partition key changes may be required.

Best Practice: Designing an effective partition key is only part of the solution. Continuously monitor partition-level metrics to ensure workloads remain balanced as applications scale, and adjust partition strategy proactively before hotspots become production bottlenecks.

What Happens If You Have Too Few or Too Many Kafka Partitions?

Choosing the right Kafka partition count is a balancing act. While partitions are the foundation of scalability and parallel processing, both under-partitioning and over-partitioning can introduce performance and operational challenges. The objective is not to maximize the number of partitions, but to provision enough partitions to support current workloads, accommodate future growth, and maintain efficient cluster operations.

What Happens When There Are Too Few Partitions?

Topics with too few partitions limit the amount of work that can be processed concurrently. Since each partition can be consumed by only one consumer within a consumer group, insufficient partitions restrict consumer parallelism regardless of how many consumer instances are deployed.

As workloads grow, this often results in:

Limited consumer parallelism
Higher consumer lag
Lower overall throughput
Underutilized consumer instances
Reduced ability to scale horizontally

For example, if a topic has 8 partitions but an application requires 20 consumers to keep pace with incoming traffic, only 8 consumers can actively process records. The remaining 12 consumers remain idle because there are no partitions available to assign.

What Happens When There Are Too Many Partitions?

Although increasing partition count improves scalability, excessive partitioning introduces additional operational overhead. Every partition requires metadata management, broker resources, storage, and coordination during cluster operations.

Creating significantly more partitions than required can lead to:

Increased broker memory consumption
More metadata to manage
Longer consumer group rebalancing
Higher storage and file handle usage
Slower broker startup and recovery
Greater operational complexity

These effects become increasingly noticeable in large production clusters hosting thousands of partitions across multiple brokers.

Comparing Both Approaches

Design Consideration	Too Few Partitions	Too Many Partitions
Consumer Parallelism	Limits horizontal scaling	Supports more consumers than required
Throughput	Creates processing bottlenecks	Additional partitions provide little benefit once throughput requirements are met
Broker Resources	Efficient metadata usage	Higher memory and metadata overhead
Consumer Rebalancing	Faster	Can take longer as partition count increases
Broker Recovery	Faster	Recovery and leader election involve more partitions
Long-Term Scalability	May require repartitioning sooner	Better prepared for future growth, but with increased operational cost

The ideal partition count balances these trade-offs while leaving sufficient headroom for expected growth.

Design for Growth, Not Just Current Workloads

Many engineering teams size Kafka topics based only on current traffic, only to discover later that additional consumers cannot be utilized because the partition count has become the limiting factor.

A more sustainable approach is to consider:

Expected peak throughput
Maximum consumer parallelism
Business growth projections
Seasonal traffic spikes
Infrastructure capacity

Planning with future growth in mind reduces the likelihood of repartitioning production topics as workloads evolve.

How Condense Simplifies Partition Scaling

As streaming workloads increase, engineering teams often need to increase partition counts to maintain throughput and consumer parallelism. Condense simplifies this process through Kafka Management, allowing partition counts to be increased after topic creation while preserving the existing topic lifecycle. As data volumes grow, the platform helps engineering teams scale Kafka brokers and partitions together, enabling applications to handle higher event volumes without manual infrastructure coordination. This is particularly valuable for workloads with rapidly changing traffic patterns, such as connected vehicle platforms, IoT deployments, or event-driven applications that experience predictable peak events.

Best Practice: Select a partition count that supports both current and projected workloads rather than optimizing solely for today's traffic. As throughput and consumer demand increase, periodically review partition utilization and scale partitions proactively to maintain balanced workloads and consistent application performance.

How Can You Safely Increase Kafka Partition Count Without Downtime?

As applications scale, increasing the number of Kafka partitions is often necessary to support higher throughput and greater consumer parallelism. Apache Kafka allows partition counts to be increased for existing topics without taking producers or consumers offline, making it possible to scale running applications with minimal disruption.

However, increasing the partition count is more than a capacity upgrade. It changes how future records are distributed across partitions and can affect applications that depend on partition keys for ordering or stateful processing. Understanding these implications helps engineering teams scale topics safely while maintaining application correctness.

Can Kafka Increase and Decrease Partition Count?

Kafka supports increasing the number of partitions for an existing topic, but it does not allow partitions to be removed.

Operation	Supported	Notes
Increase partition count	✓ Yes	Enables additional consumer parallelism and throughput
Decrease partition count	✗ No	Requires creating a new topic and migrating producers and consumers

Because partition counts cannot be reduced, capacity planning should consider future growth during the initial topic design.

What Happens When You Increase the Partition Count?

Kafka determines the destination partition for keyed records using the partition key and the total number of partitions.

Partition = hash(partition_key) % number_of_partitions

When the number of partitions changes, the calculated destination partition for future records may also change.

For example:

Before Scaling	After Scaling
8 Partitions	16 Partitions
Customer_123 → Partition 3	Customer_123 → Partition 11

Existing records remain in their original partitions, while new records are written according to the updated partition mapping.

This distinction is important because Kafka does not redistribute existing records when partitions are added.

What Should Engineering Teams Consider Before Scaling?

Before increasing partition counts, evaluate whether your applications depend on:

Message ordering for a business entity
Stateful stream processing
Stream joins and aggregations
Consumer applications maintaining local state
Historical event replay

If ordering across the complete event history is a requirement, increasing partition count should be planned carefully.

Recommended Migration Strategies

The safest approach depends on the application architecture.

Strategy	Recommended For	Downtime
Increase partitions in-place	Stateless event processing	None
Create a new topic with the desired partition count	Stateful applications	None
Dual-write to both topics during migration	Large production systems	None
Gradually migrate consumers	Enterprise deployments	None

Many engineering teams choose to create a new topic with the desired partition count for large production workloads. Producers publish to both topics during the migration period, consumers are validated against the new topic, and the original topic is retired once the migration is complete.

How Condense Simplifies Partition Scaling

Scaling Kafka topics should not require engineering teams to manually coordinate infrastructure changes across multiple tools.

Using Kafka Management, Condense allows teams to increase partition counts as application workloads grow while continuing to manage the same Kafka topic throughout its lifecycle. As additional partitions are introduced, Condense provides visibility into consumer lag, partition assignments, committed offsets, and consumer group health, helping teams verify that workloads are being redistributed effectively.

For consumer groups, Condense supports cooperative incremental rebalancing, allowing Kafka to move only the affected partitions when consumers join or leave the group instead of reassigning every partition. This significantly reduces disruption, minimizes unnecessary data movement, and enables smoother scaling for production workloads.

Combined with broker scaling, partition management, and partition-level observability, engineering teams can expand streaming capacity while maintaining application stability during periods of sustained growth or peak traffic.

Best Practice: Plan partition growth before throughput becomes a bottleneck. Increase partitions based on measured workload trends, validate workload distribution using partition-level metrics, and use cooperative incremental rebalancing to minimize disruption during consumer scaling.

What Are the Most Common Kafka Partition Strategy Mistakes?

Even experienced engineering teams can encounter partitioning issues as streaming workloads evolve. In many cases, performance bottlenecks are not caused by Kafka itself but by partitioning decisions made during the initial design phase. Understanding these common mistakes helps teams build Kafka deployments that remain scalable, balanced, and resilient as data volumes increase.

Choosing Partition Keys with Low Cardinality

Partition keys such as Country, Region, Status, or Event Type have a limited number of unique values. As traffic grows, these values often create hotspots by routing a disproportionate amount of data to a small number of partitions.

Instead, choose high-cardinality identifiers such as:

Customer ID
Device ID
Vehicle ID
Order ID
Session ID

These keys distribute records more evenly while preserving ordering for related events.

Ignoring Future Consumer Parallelism

Many teams size partition counts based only on their current deployment.

For example:

Current deployment: 4 consumers
Topic configured with: 4 partitions

As the application grows, the team deploys 12 consumers, expecting throughput to increase. However, only four consumers actively process records because the partition count has become the limiting factor.

Always size partition counts based on expected peak consumer parallelism rather than today's deployment.

Prioritizing Throughput Over Ordering

Using random or null partition keys can improve distribution across partitions, but it also removes ordering guarantees for related events.

Applications involving:

Financial transactions
Fleet telemetry
Inventory updates
Customer state
Event sourcing

typically require semantic partition keys to preserve event order.

Select the partition key based on application behavior rather than throughput alone.

Creating Excessive Partitions

More partitions do not automatically improve Kafka performance.

Excessive partition counts increase:

Broker metadata
Memory usage
File handles
Consumer rebalancing time
Recovery time after failures

Partition count should be determined through capacity planning and benchmarking instead of arbitrary sizing rules.

Not Monitoring Partition Health

Partition strategies should evolve as workloads change.

Traffic patterns that are evenly distributed today may become skewed as new customers, devices, or business regions generate significantly more events.

Engineering teams should continuously monitor:

Consumer lag
Partition assignments
Broker utilization
Partition growth
Consumer group health

These metrics provide early warning signs that partition counts or partition keys should be reviewed.

With Condense, teams can monitor partition-level consumer lag, partition assignments, committed offsets, and consumer group health from a unified Kafka Management interface, making it easier to identify imbalance before it affects production workloads.

Treating Partition Strategy as a One-Time Activity

Partition strategy should evolve alongside the application.

As throughput, consumer groups, and traffic distribution change, engineering teams should periodically review:

Partition utilization
Consumer parallelism
Workload distribution
Broker capacity
Future scaling requirements

Rather than waiting for consumer lag or hotspots to appear, proactively reviewing these metrics helps maintain consistent performance as streaming workloads grow.

Common Mistakes at a Glance

Mistake	Business Impact	Recommended Approach
Choosing low-cardinality partition keys	Creates hotspots and uneven workloads	Use high-cardinality business identifiers
Sizing partitions only for current traffic	Restricts future consumer scaling	Plan for projected throughput and growth
Ignoring ordering requirements	Breaks event sequencing	Use semantic partition keys where ordering matters
Creating excessive partitions	Increases operational overhead	Size partitions using throughput benchmarks
Not monitoring partition-level metrics	Delays detection of workload imbalance	Continuously monitor partition health and consumer lag
Treating partition strategy as static	Limits long-term scalability	Review and refine partition strategy as workloads evolve

Best Practice: Kafka partition strategy should be treated as an ongoing architectural practice rather than a one-time configuration task. Combining effective partition sizing, well-designed partition keys, continuous monitoring, and proactive capacity planning enables streaming applications to scale predictably while maintaining performance and reliability.

Conclusion

Kafka partition strategy is the foundation of every scalable Kafka deployment. It influences how data is distributed across brokers, determines the maximum level of consumer parallelism, preserves message ordering, and directly impacts throughput, resource utilization, and long-term cluster performance. Decisions made during topic design continue to influence application behavior long after workloads move into production, making partition strategy one of the most important architectural considerations in event streaming.

Designing an effective partition strategy requires balancing multiple factors. The right partition count should support expected throughput and future consumer parallelism without introducing unnecessary operational overhead. Partition keys should distribute workloads evenly while preserving ordering for events that belong together. As traffic patterns evolve, continuous monitoring helps identify hotspots, consumer lag, and partition imbalance before they affect application performance.

While Apache Kafka provides the flexibility to configure and scale partitions, implementing and operating an effective partition strategy across production environments requires the right operational capabilities. Condense simplifies the complete partition lifecycle through Kafka Management, enabling engineering teams to configure partitions during topic creation, scale partition counts as workloads grow, inspect partition-level data, monitor consumer lag and partition assignments, and manage consumer group rebalancing from a unified interface. This allows teams to focus on designing scalable streaming applications while simplifying the deployment, monitoring, and evolution of their Kafka infrastructure.

Whether you're building connected mobility platforms, IoT applications, real-time analytics pipelines, or event-driven microservices, investing time in the right Kafka partition strategy from day one helps build streaming systems that remain scalable, resilient, and easier to operate as data volumes continue to grow.

Frequently Asked Questions (FAQs)

What is a Kafka partition strategy?

A Kafka partition strategy defines how records are distributed across partitions within a topic. It involves selecting the appropriate partition count and partition key to balance workloads, maximize consumer parallelism, preserve message ordering where required, and support long-term scalability. A well-designed partition strategy improves throughput, reduces hotspots, and enables applications to scale efficiently
How do I calculate the right Kafka partition count?

The partition count should be based on the larger of the producer throughput requirement and the consumer processing requirement. Factors such as message size, consumer processing complexity, replication factor, and future workload growth should also be considered. Rather than relying on fixed recommendations, benchmark representative workloads and size partitions for expected peak traffic
Can I increase the number of Kafka partitions after creating a topic? Yes. Kafka allows partition counts to be increased after a topic is created, enabling additional consumer parallelism as workloads grow. However, increasing the partition count changes how future records are distributed because the partitioning algorithm uses the total number of partitions. Existing records remain in their original partitions. Using Condense, engineering teams can increase partition counts through Kafka Management while monitoring consumer lag, partition assignments, and consumer group health to validate that scaling has improved workload distribution
How do I choose the right Kafka partition key?

Choose a partition key based on how the application processes data. If related events must remain in order, use a stable business identifier such as Customer ID, Device ID, Vehicle ID, or Order ID. If ordering is not required, random or null keys can distribute records more evenly across partitions and improve load balancing
What causes hotspots in Kafka partitions?

Hotspots occur when a small number of partitions receive significantly more traffic than others. This is commonly caused by partition keys with low cardinality or uneven business data distribution. Hotspots can increase consumer lag, overload brokers, and reduce overall throughput. Monitoring partition-level metrics helps identify these issues before they affect production workloads
Does adding more partitions always improve Kafka performance?

No. While additional partitions increase consumer parallelism and throughput capacity, they also increase broker metadata, memory usage, file handles, and consumer rebalancing overhead. The optimal partition count balances scalability with operational efficiency rather than simply maximizing the number of partitions
Can Condense help manage Kafka partitions?

Yes. Condense simplifies Kafka partition management throughout the topic lifecycle. Engineering teams can configure partition counts during topic creation, increase partitions as workloads grow, inspect partition-level messages, monitor consumer lag and partition assignments, track consumer group health, and manage partition scaling from a unified Kafka Management interface. This helps teams implement and operate partition strategies more efficiently while continuing to leverage Apache Kafka
What metrics should I monitor to validate my partition strategy?

A healthy partition strategy should be validated using operational metrics rather than assumptions. Engineering teams should regularly monitor:

Consumer lag by partition
Partition throughput
Partition assignments
Broker CPU, memory, and storage utilization
Consumer group health
Partition growth over time

Monitoring these metrics helps identify hotspots, workload imbalance, and scaling requirements before they impact application performance

Why Your Kafka Stack Is Holding You Back (And How to Fix It)

Sachin Kamath — Sat, 18 Jul 2026 13:14:23 +0000

TL;DR

Apache Kafka remains foundational but managing it alone is complex and inadequate for modern real-time needs.

Condense modernizes Kafka by providing a fully managed, Kafka-native streaming platform deployed inside your cloud (BYOC) with:

Prebuilt, domain-specific transforms
Low-code, Git-integrated IDE for stream logic
Cloud-native autoscaling and zero-downtime upgrades
Full observability and lifecycle management
Data sovereignty and cost control via BYOC

Leading enterprises use Condense to build scalable, domain-aware, production-ready real-time applications quickly and reliably, moving beyond raw Kafka infrastructure to real business outcomes.

Apache Kafka has remained the backbone of event-driven architectures for over a decade. Its immutable log abstraction, scalable broker design, and stream-first philosophy have powered countless real-time systems, from fraud detection and e-commerce analytics to telematics ingestion and industrial automation.

But the world around Kafka has evolved. Data volumes have exploded. Cloud economics have shifted. Developer expectations have changed. And most critically, the business demands from real-time systems have moved far beyond what an isolated Kafka cluster can provide.

In 2025, continuing to operate Kafka as it was done a few years ago like manually managed, loosely integrated, and layered with custom scripts is increasingly unsustainable. Here are five deeply technical and operational reasons why modernizing the Kafka stack is no longer optional, but strategic.

Reason 1: Kafka Alone Isn’t a Platform

Running Kafka by itself delivers transport but not outcomes. Most real-time use cases depend on an entire ecosystem of critical components around Kafka, including:

Schema registries for versioned serialization
Stream processors for business logic execution
Connectors for integration with databases, filesystems, APIs, or telemetry streams
Monitoring agents to observe lag, consumer health, and throughput bottlenecks
Security layers for multi-tenant isolation, role-based access, and encryption

When these components are stitched together manually, organizations inherit the burden of lifecycle management: upgrades, patching, configuration drift, dependency mismatches, downtime orchestration, and incident response.

Modernizing the Kafka stack means adopting a cohesive, cloud-native runtime where these components work in unison, ideally under a single operational contract. This creates a predictable, observable, and sustainable foundation for stream-first workloads.

Reason 2: Developer Velocity Demands Better Abstractions

The Kafka ecosystem has traditionally favored infrastructure engineers and backend specialists. Defining stream joins, windowing logic, or repartitioning flows requires deep knowledge of Kafka Streams, KSQL, or Flink plus careful handling of topic schemas, backpressure, and message formats.

As event-driven logic becomes part of core business applications, whether it’s scoring driver behavior, flagging transaction anomalies, or transforming IoT telemetry, developer experience becomes a bottleneck.

Modern stacks must support:

Low-code interfaces for operational workflows
GitOps workflows for versioned stream deployments
AI-assisted IDEs to auto-generate transformation templates
Live testing environments that simulate events before production rollout

Without these capabilities, real-time use cases become slower to deliver and harder to iterate, putting Kafka-centric architectures at odds with agile product cycles.

Reason 3: Cloud-Native Architecture Is Now Table Stakes

Migrating to a cloud-aligned architecture reduces operational complexity, increases utilization efficiency, and enables faster scale-out for peak workloads, without human intervention.

In 2025, most Kafka workloads run on cloud infrastructure, whether in VMs, managed Kubernetes clusters, or fully serverless runtimes. Yet traditional Kafka deployments often ignore cloud-native principles:

Manual node provisioning leads to overprovisioning or underperformance.
No support for autoscaling brokers or connectors based on demand.
Lack of integration with cloud IAM, logging, and billing complicates security and cost attribution.
Self-managed high availability adds operational tax for each region or zone.

Modern platforms treat Kafka as one component in a broader elastic data plane. Brokers auto-scale. Connectors spin up based on load. Stream processors run in serverless containers. Failovers are orchestrated automatically. Monitoring is pushed into existing cloud-native observability stacks.

Migrating to a cloud-aligned architecture reduces operational complexity, increases utilization efficiency, and enables faster scale-out for peak workloads, without human intervention.

Reason 4: Real-Time Use Cases Now Depend on Domain-Aware Processing

Kafka is a generic tool. But most real-time applications are domain-specific. Consider:

In mobility, real-time logic might involve VIN-based trip formation, geofence entry/exit events, and harsh braking classification.
In logistics, it may involve cargo temperature violation alerts, trip ETA updates, and route compliance tracking.
In finance, real-time use cases often involve transaction scoring, KYC triggers, or payment retry orchestration.

These patterns cannot be implemented through raw Kafka APIs or SQL-like interfaces. They demand prebuilt, domain-native transforms that understand context. e.g., how to interpret an OBD-II message, what constitutes a loading zone, or how to calculate SLA breach probability in transit.

Modern Kafka platforms incorporate verticalized logic libraries, deployable out-of-the-box, saving engineering months of effort while improving accuracy and operational trust.

Reason 5: Cost Optimization and BYOC Are Now Strategic Priorities

As enterprise cloud bills grow, organizations are rethinking the economics of managed Kafka. Traditional hosted platforms run Kafka inside the vendor’s cloud account, which leads to:

Double billing (vendor cost + unused cloud credits)
Lack of visibility into runtime costs
Inability to apply reserved instances or volume discounts
No control over data egress patterns or compliance enforcement

Modern Kafka platforms support Bring Your Own Cloud (BYOC) where all infrastructure runs in the enterprise’s cloud account, using its cloud credits and governance tools. This offers:

Full cost control and transparency
Better alignment with existing cloud agreements
Data sovereignty and compliance retention
Direct integration with internal monitoring, alerting, and IAM systems

BYOC is not just about infrastructure flexibility; now it is a financial, legal, and strategic enabler for Kafka adoption at scale.

Kafka Needs a Platform, Not Just Brokers

The technical power of Kafka is undiminished. But its role has changed. Kafka is no longer the end goal. It’s the foundation upon which real-time business logic, domain-aware intelligence, and operational outcomes are built.

Modernizing the Kafka stack means wrapping it with the necessary abstractions, integrations, and delivery systems required to thrive in production. The shift is from Modernizing the Kafka stack means wrapping it with the necessary abstractions, integrations, and delivery systems required to thrive in production. The shift is from running brokers to delivering applications. From managing infrastructure to enabling decisions in motion.

Why Condense?

Condense is built for this new era of real-time streaming. It is a Kafka-native platform, delivered via BYOC, and tailored to industries like mobility, logistics, industrial automation, and connected infrastructure.

With prebuilt transforms, low-code development, AI-assisted IDEs, and full cloud integration, Condense reduces time-to-value while increasing platform trust. It brings together Kafka, stream logic, deployment tooling, and observability, without requiring a dedicated SRE team to keep things running.

In 2025, Kafka alone is no longer enough. The future belongs to streaming platforms that don’t just deliver logs, but understand the domain behind every message. Systems where VINs aren’t just strings, but identifiers for operational context. Where a harsh brake isn't just a sensor value, but a signal that may affect safety, routing, or warranty.

Condense leads that transformation. It extends Kafka with domain semantics, real-time transforms pre-aligned with industry workflows, and infrastructure that runs inside the enterprise’s own cloud environment. Kafka becomes more than transport; it becomes the foundation for intelligent, outcome-driven applications that speak the language of the domain.

That’s why enterprises like Volvo, Eicher, Royal Enfield, Michelin, CEAT and TVS have moved beyond generic Kafka clusters and toward streaming platforms like Condense, where real-time pipelines are not just technically correct, but operationally meaningful.

Each of these industries requires different connectors, semantic models, latency expectations, and deployment constraints. Condense abstracts that complexity through domain-aligned transforms, BYOC infrastructure, and a Kafka-native architecture, so organizations don’t just stream data, but operationalize it.

Frequently Asked Questions (FAQs)

Is Apache Kafka being replaced?

No. Apache Kafka remains a foundational component for event streaming. What's changing is the ecosystem around it. Modern organizations are moving away from raw Kafka clusters and toward integrated platforms that combine Kafka with stream processing, domain logic, observability, security, and deployment automation. The goal is not to replace Kafka, but to make it production-grade and outcome-oriented.
What does it mean to "modernize" a Kafka stack?

Modernization involves evolving from a loosely assembled set of Kafka services to a platform where stream processing is:

Domain-aligned (industry-specific logic and semantics)
Cloud-native (autoscaling, managed failover, integrated monitoring)
Developer-ready (GitOps, low-code, AI-assisted transforms)
Cost-efficient (BYOC, cloud credit utilization)

It's about increasing delivery speed and reducing operational burden, without losing Kafka's core strengths.

3. Why is developer velocity relevant to Kafka architecture?

Kafka was historically managed by infrastructure teams. But today, product and application teams are building on top of Kafka for use cases like real-time pricing, routing intelligence, maintenance prediction, and alerting. If the underlying stack requires custom JVM code or complex DSLs for every transformation, delivery slows down.

Modern platforms provide abstractions that let domain experts and developers collaborate at speed, without needing to be Kafka internals experts.

4. What is the role of domain awareness in Kafka-based systems?

Raw Kafka doesn't know the difference between a vehicle ID and a sensor type. But real-time systems increasingly depend on contextual interpretation: route IDs, fleet zones, compliance flags, shipment IDs, etc. Domain-aware platforms bring this intelligence closer to the data plane, embedding semantic understanding into transforms, alerting, and visualization. This eliminates the need to re-encode business logic downstream in BI tools or service code.

5. What is BYOC, and why does it matter for Kafka?

BYOC (Bring Your Own Cloud) allows the Kafka platform and supporting services to run fully inside the enterprise's own cloud account (AWS, Azure, GCP). The platform is still vendor-operated but leverages the customer's cloud credits, IAM policies, observability stack, and compliance posture.

This ensures data sovereignty, cost efficiency, and deep integration, without requiring the enterprise to self-manage Kafka infrastructure.

6. How does Condense modernize Kafka differently?

Condense builds on Kafka's architecture but adds:

Prebuilt, domain-specific transforms (e.g., for mobility, logistics, energy)
A low-code/IDE interface for defining and deploying stream logic
CI/CD pipelines for stream application lifecycle management
Native BYOC deployment support across AWS, Azure, and GCP
Isolation by default, with full auditability and customer-bounded operations

It enables streaming-native applications to be built and deployed in days, not quarters, without requiring deep Kafka expertise or large ops teams.

7. What kinds of organizations are using Condense?

Condense is trusted by a broad spectrum of enterprises and system integrators operating in data-intensive, real-time environments. These span across:

Automotive OEMs including Volvo, Royal Enfield, and TVS Motor, using Condense for OTA updates, remote diagnostics, vehicle analytics, and feature lifecycle control
Fleet and Mobility Platforms such as Eicher, SML Isuzu, and Taabi Mobility, relying on Condense for trip intelligence, predictive maintenance, panic alerting, and live telematics processing
Logistics and Transportation Networks including Michelin and various freight, mining, and container mobility platforms using Condense for multi-modal tracking, cold chain eventing, and geofenced security
Industrial and Manufacturing Operations streaming real-time production telemetry, detecting bottlenecks, balancing workloads, and ensuring operational continuity using data from PLCs and SCADA systems
Financial Services where Condense enables fraud detection pipelines, transaction anomaly flagging, and secure, compliant integration with downstream rule engines and audit layers
Smart Cities and Public Infrastructure powering streaming use cases in traffic signal networks, emergency response coordination, and public transportation tracking with millisecond latency
Travel and Hospitality Systems: Unifying data from property management systems (PMS), shuttle tracking, booking engines, and mobile apps to enable dynamic rate optimization, real-time availability, and multilingual customer notifications

Condense allows hotel chains, airport service providers, and hospitality tech platforms to detect and react to changes, such as flight delays, booking conflicts, or room state transitions, in real time.

What Is Apache Kafka? Plain-English Guide for Engineers (2026)

Sachin Kamath — Wed, 15 Jul 2026 06:15:59 +0000

TL;DR

Apache Kafka is a high-throughput, fault-tolerant distributed event streaming platform ideal for real-time data processing. It organizes data into replicated topics and partitions, ensuring scalability and reliability. Kafka uses a pull-based consumer model and supports hybrid messaging (pub/sub and queuing). It offers flexible data retention, security features, and APIs for stream processing and integrations.

Condense builds on Kafka by providing fully managed, scalable, and cost-optimized Kafka deployments with industry-specific intelligence and verticalized data pipelines. It removes operational complexity and accelerates real-time insights for enterprises, enabling easier adoption and faster value delivery.

Introduction to Apache Kafka

Apache Kafka is an open-source distributed event streaming platform developed by the Apache Software Foundation. It is designed to handle high-throughput, fault-tolerant, durable, and scalable real-time data feeds. Kafka is widely used in real-time data pipelines, event-driven architectures, and Kafka Streams libraries for stream processing applications.

Originally developed at LinkedIn in 2010 to address growing data processing needs, Kafka was open-sourced in 2011 and has since become an integral part of modern data architectures.

Kafka as a Distributed System

Kafka operates as a distributed system, meaning data is stored and processed across multiple machines to ensure high availability and fault tolerance. This architecture allows Kafka to handle millions of messages per second, making it ideal for large-scale, real-time applications.

Kafka is horizontally scalable, allowing organizations to add more servers (brokers) as demand increases. Unlike traditional messaging systems, Kafka employs log-based storage, where data is written sequentially, reducing disk I/O bottlenecks and improving performance.

Kafka as an Event Streaming Platform

Kafka is more than just a messaging system; it enables applications to capture, process, and react to real-time data changes. This capability is valuable for:

Real-time monitoring (e.g., log analysis, security alerts).
Streaming analytics (e.g., fraud detection, stock trading, IoT analytics).
Decoupling microservices (i.e., enabling efficient service-to-service communication via event streams).

Kafka integrates seamlessly with cloud-native environments, including Kubernetes, containerized applications, and managed cloud services. Managing data schemas across producers and consumers is one of the most critical operational challenges teams face.

Kafka’s Core Concepts

Topics, Partitions, and Offsets

Kafka organizes data into:

Topics: Logical channels for message streams.
Partitions: Subdivisions of topics that distribute data across brokers.
Offsets: Unique identifiers assigned to each message within a partition, ensuring ordered message sequences.

Each partition is replicated across brokers for fault tolerance. If a broker fails, Kafka automatically redirects traffic to another broker with a replica.

Kafka’s High Availability & Fault Tolerance

Kafka achieves reliability through leader-follower replication:

Each partition has a leader handling read/write requests.
Follower replicas synchronize with the leader and take over in case of failure.
This ensures continuous data availability and prevents data loss.

Kafka’s Pull-Based Consumer Model

Unlike traditional push-based messaging systems, Kafka follows a pull-based model, where consumers retrieve messages at their own pace. Benefits include:

Backpressure handling: Prevents overwhelming consumers with excessive data.
Flexible message processing: Consumers can reprocess messages by adjusting offsets.
Efficient batching: Consumers can read multiple messages at once for better performance.

Kafka’s Internal Architecture

Zookeeper’s Role in Kafka

Kafka uses Apache Zookeeper for:

Leader election and failover handling.
Configuration management.
Tracking broker metadata.

Kafka 4.0.0 Update: The removal of ZooKeeper in Kafka 4.0 simplified this architecture significantly.

Producer

Partitioning Strategy: Messages are distributed across partitions based on a key or a round-robin method.
Batching & Compression: Kafka supports gzip, Snappy, and LZ4 to optimize data transmission.
Acknowledgment Levels:
- acks=0 → No acknowledgment (fastest but risky).
- acks=1 → Acknowledged by leader only (some risk).
- acks=all → Acknowledged by leader and all in-sync replicas (safest).

Consumer

Consumer Groups: Consumers are grouped to distribute workload efficiently.
Offset Management: Kafka tracks processed messages using internal consumer_offsets topics.
Dynamic Rebalancing: If a consumer joins or leaves, Kafka dynamically redistributes partitions.

Kafka’s Pub-Sub and Message Queuing Hybrid Model

Kafka blends publish-subscribe (pub-sub) and message queuing models:

Message Queuing: Each consumer reads different messages, ensuring parallel processing.
Publish-Subscribe: Multiple consumers can read from the same topic, allowing multiple applications to process the same data stream in real-time.

Kafka’s Retention, Deletion, and Compaction

Time-based Retention: Messages persist for a defined period (default: 7 days).
Size-based Retention: Kafka deletes older messages if the topic exceeds a configured size.
Log Compaction: Instead of deleting messages, Kafka retains only the latest version of a message per key.

Kafka’s Replication Mechanism

Kafka follows a leader-follower model with In-Sync Replicas (ISR)

ISR contains follower replicas that are synchronized with the leader.
Unclean Leader Election: If all ISR replicas fail, Kafka can elect an out-of-sync replica unless explicitly disabled (unclean.leader.election.enable=false).

Kafka Security Mechanisms

Kafka offers multiple security features:

Authentication: Supports SASL, Kerberos, and SSL-based authentication.
Authorization: Role-based access control (RBAC) using Kafka ACLs.
Data Encryption: SSL/TLS for data in transit, cloud-based encryption for data at rest.

Kafka’s Stream Processing & APIs

Kafka offers several APIs for real-time data processing:

Kafka Streams API: Transforms, aggregates, and enriches data streams (e.g., real-time fraud detection).
KSQL (Kafka SQL): Enables SQL-like querying on Kafka topics (e.g., filtering IoT sensor data in real-time).
Kafka Connect API: Integrates Kafka with external databases and cloud storage (e.g., syncing Kafka with a cloud data warehouse).

Kafka Performance Optimization

Increase Partition Count: More partitions allow parallelism but increase metadata overhead.
Broker Tuning:
- log.segment.bytes: Defines segment size before Kafka rolls to a new log file.
- log.retention.hours: Configures data retention duration.
- num.network.threads: Handles network request concurrency.
Producer & Consumer Tuning:
- batch.size: Controls message batching.
- linger.ms: Introduces delays to improve batching.
- fetch.min.bytes: Determines the minimum amount of data consumers request per fetch.

Kafka in Multi-Datacenter & Cross-Region Setups

Kafka supports cross-region replication using MirrorMaker, ensuring:

Disaster recovery.
Regulatory compliance.
Efficient geographically distributed workloads.

Apache Kafka is a scalable, fault-tolerant event streaming platform that enables real-time data processing, analytics, and microservices communication. With its log-based storage, pub-sub hybrid model, high availability, and security features, Kafka remains a key component of modern cloud-native architectures. With the Kafka 4.0 update, the architecture has shifted significantly. ZooKeeper is gone, replaced by KRaft.

Condense: A Vertical Data Streaming Platform

While Kafka is powerful, managing it requires expertise and operational effort. Security configurations, including mTLS, SASL, and ACLs is one of the most complex operational requirements. Condense builds upon Kafka, offering a fully managed streaming platform with an optimized, industry-specific verticalized ecosystem.

Key Benefits of Condense

Fully Managed BYOC (Bring Your Own Cloud): Ensures data sovereignty by deploying within the customer’s cloud environment, removing infrastructure management burdens.
Fully Managed Kafka with 99.95% Availability: Eliminates downtime risks and ensures uninterrupted data streaming. Monitoring consumer lag, broker health, and pipeline throughput is critical for maintaining that availability.
Autonomous Scalability: Automatically adjusts resources based on demand.
Enterprise Support and Zero-Touch Management: 24/7 support, removing operational complexity.
Verticalized Cloud Cost Optimization: Reduces cloud expenses while maintaining performance. Condense brings in the domain expertise to govern the optimal utilization of the resources.
No Latency Issues, Regardless of Throughput: Guarantees ultra-low latency even under extreme data loads.

Why Choose Condense Over Self-Managed Kafka?

Managing Kafka in-house requires extensive DevOps resources, monitoring, and scaling expertise. Condense eliminates these challenges, allowing businesses to leverage Kafka’s full potential without the complexity.

Kafka has revolutionized real-time data streaming, but Condense takes it further, providing a fully managed, highly available, and cost-optimized platform. With zero-latency issues, automated scaling, and enterprise-grade support, Condense ensures seamless data streaming for modern businesses.

Apache Kafka and Condense together empower organizations with scalable, fault-tolerant event streaming capabilities.

Frequently Asked Questions (FAQs)

What is Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform originally built at LinkedIn in 2010 and open-sourced in 2011. It's designed for high-throughput, fault-tolerant, real-time data feeds and is widely used in data pipelines, event-driven architectures, and stream processing applications.
What is Kafka used for?

Kafka powers real-time monitoring (log analysis, security alerts), streaming analytics (fraud detection, IoT analytics, stock trading), and microservices decoupling, where services communicate via event streams instead of direct calls. It integrates natively with Kubernetes, containers, and managed cloud services.
How does Kafka work?

Kafka organizes data into topics, which are split into partitions distributed across brokers. Each message in a partition gets a unique offset, ensuring ordered sequences. Partitions are replicated across brokers using a leader-follower model, so if a broker fails, a synchronized replica takes over automatically with no data loss.
Is Kafka a message queue or a pub-sub system?

Both. Kafka is a hybrid. Within a consumer group, each consumer reads different messages (queuing, for parallel processing), while multiple consumer groups can independently read the same topic (pub-sub, so several applications can process the same stream in real time).
What is the difference between Kafka and traditional messaging systems?

Two big ones: Kafka uses log-based storage where data is written sequentially, reducing disk I/O bottlenecks, and it uses a pull-based consumer model where consumers fetch messages at their own pace. This enables backpressure handling, message reprocessing by adjusting offsets, and efficient batching.
Does Kafka still need ZooKeeper?

No. As of Kafka 4.0, ZooKeeper has been removed entirely and replaced by KRaft, which simplifies the architecture. Previously, ZooKeeper handled leader election, configuration management, and broker metadata.
How long does Kafka retain data?

By default, 7 days (time-based retention). You can also configure size-based retention, where older messages are deleted once a topic exceeds a set size, or log compaction, which keeps only the latest version of each message per key.
Is Kafka secure?

Kafka supports SASL, Kerberos, and SSL-based authentication, role-based access control via ACLs, and SSL/TLS encryption in transit. That said, configuring mTLS, SASL, and ACLs correctly is one of the most complex operational requirements teams face when self-managing Kafka.
How do you scale Kafka?

Kafka scales horizontally by adding brokers, and parallelism increases with partition count (though more partitions add metadata overhead). Performance tuning also involves broker settings (segment size, retention, network threads) and producer/consumer settings like batch size, linger time, and fetch minimums.
What is Condense and how does it relate to Kafka?

Condense is a fully managed data streaming platform built on Kafka. It removes the operational burden of running Kafka: security configuration, scaling, monitoring, and DevOps expertise, while adding an industry-verticalized ecosystem on top.
What does BYOC mean in Condense?

Bring Your Own Cloud: Condense deploys entirely within your own cloud environment, so your data never leaves your account and you retain full data sovereignty, while Condense manages the infrastructure end-to-end.
Why choose Condense over self-managed Kafka?

Self-managing Kafka demands extensive DevOps resources for monitoring consumer lag, broker health, and pipeline throughput, plus deep scaling and security expertise. Condense delivers 99.95% availability, autonomous scaling, ultra-low latency at any throughput, and 24/7 enterprise support without a dedicated platform team.

Privatise your Data Streams with Bring Your Own Cloud (BYOC)

Sachin Kamath — Tue, 14 Jul 2026 03:25:10 +0000

TL;DR

Traditional SaaS streaming requires exporting sensitive data to a vendor cloud, creating security risks and egress costs. BYOC reverses this model by running the data plane inside the customer’s cloud while the vendor manages the control plane. This keeps data within the enterprise perimeter while still providing a managed platform. Condense builds on this model with AI-driven automation, unified monitoring, and marketplace deployment, enabling private, compliant, and cost-efficient real-time data streaming.

The enterprise data landscape is currently defined by a conflict between real-time AI data streaming utility and the strict requirements of data sovereignty. For years, the standard SaaS model forced a compromise. To access premium analytics, companies had to export sensitive telemetry to a vendor cloud. This created massive cloud egress costs and introduced significant security vulnerabilities.

Bring Your Own Cloud (BYOC) for data streaming platforms has emerged as the professional solution to this dilemma. It allows a business to keep data within its own perimeter while benefiting from a fully managed, high-performance ecosystem.

The BYOC Architecture: Privacy by Design

An experienced analyst views BYOC as a clean separation of concerns. The architecture splits the environment into two distinct layers to ensure raw data never leaves the authorized environment.

SaaS Control Plane: This is the management layer hosted by the provider. It handles the brain of the operation. It manages orchestration, user access, and pipeline configuration without ever seeing the actual data packets.
Private Data Plane: This is the muscle. The managed Kafka clusters, Kubernetes (K8s) nodes, and storage engines like ClickHouse live inside the customer Virtual Private Cloud (VPC).

By keeping the data plane inside the customer perimeter, telemetry collection remains private. This architecture is the most direct path to satisfying internal security audits and global regulatory bodies.

The Economic Advantage of Cloud Marketplace Integration

The move to BYOC is a financial strategy as much as a technical one. When a platform like Condense is deployed through a Cloud Marketplace such as AWS, Azure, or GCP, it unlocks three major economic levers.

Commitment Drawdown: Most large enterprises have pre-existing spending commitments with cloud providers. Purchasing through the marketplace allows these software costs to count toward those totals. It turns a budget into high-value infrastructure.
Ending the Data Tax: One of the primary killers of ROI in data projects is the egress fee. By co-locating the streaming producers and consumers in the same region, the tax for moving data to an external SaaS cloud is eliminated.
Procurement Speed: Consolidating costs into a single cloud provider invoice bypasses months of vendor onboarding and complex billing cycles.

Intelligence and Automation via AI Agents

A privatised data stream is only useful if it is manageable. The Condense platform provides a unified ecosystem that goes beyond simple hosting. It includes specialized AI Agents designed to handle the heavy lifting of distributed systems.

Infrastructure and K8s Agents: These handle the automated management of Kafka brokers and Kubernetes clusters. They ensure high availability without needing a massive internal DevOps team.
Pipeline and Code Assistants: These agents help R&D teams build data pipelines and custom code frameworks much faster than manual coding.
Native Monitoring: Instead of jumping between multiple tools, everything from ingestion to transformation is visible in one managed interface.

Sovereignty and the DPDP Act in India

For businesses operating under the DPDP Act in India, the BYOC model is a primary compliance tool. It provides the technical evidence required to prove that sensitive information is handled correctly.

Data Residency: Physical storage stays within the country. This prevents unauthorized cross-border data movement.
Advanced Key Management: Support for Customer-Managed Keys (CMK) and BYOK ensures that even the cloud provider cannot decrypt the data.
Confidential Computing: For the most sensitive workloads, data remains encrypted even while it is being processed in memory.

Strategic Recommendations

To get the most out of a private data stream, focus on these three pillars.

Respect Data Gravity: Place workloads where the data already lives to reduce latency and architectural complexity.
Build Clean Paths: For real-time AI inference, use private metro connectivity and deterministic routing. This guarantees the low latency needed for agentic workloads.
Security First: Use identity-first security and threat detection to protect the distributed nodes that collect your data.

BYOC represents the maturity of the cloud era. It offers the speed and intelligence of a fully managed Kafka platform with the absolute control of a private data center.

Frequently Asked Questions (FAQs)

What makes Condense different from other BYOC Kafka platforms?

Unlike platforms where BYOC is an afterthought, Condense is 100% BYOC-native. It is engineered to run entirely within your VPC (AWS, Azure, or GCP), ensuring that 100% of your data remains under your ownership while eliminating "SaaS" networking taxes and egress fees.
How does Condense handle scaling without manual intervention? Condense features Autonomous Scaling. It monitors real-time consumer lag and event throughput, automatically provisioning compute for your custom Java, Python, or Go transforms. When surges pass, it scales back down to optimize costs, removing the need for a dedicated DevOps team to size clusters.
Does Condense support real-time data processing?

Yes. Condense provides a Unified Streaming Layer that merges the broker and application environment. By running application logic "local" to the Kafka broker, it achieves Zero-Hop processing, which eliminates the network latency found in external processing tiers like Flink.
How does Condense ensure compliance with the DPDP Act in India?

By using a managed BYOC model, Condense satisfies the DPDP Act’s strict data residency requirements. All sensitive telemetry is stored and processed within India’s borders, and with Customer-Managed Keys (CMK), even the platform provider cannot access your decrypted data.
Can I use Condense to build industry-specific AI pipelines?

Yes. Condense includes a Verticalized Ecosystem with pre-built, domain-aware transforms. For industries like Mobility (VIN parsing, trip decoding) or FinTech, these pre-tuned assets allow R&D teams to bypass months of custom coding and go to market faster.
How do Condense AI Agents assist in operations?

Condense uses specialized AI Agents for Intelligent Observability. These agents monitor event streams and infrastructure (K8s/Kafka) continuously, providing actionable insights and automated failovers, which reduces the Total Cost of Ownership (TCO) by up to 40%.

Real-Time Inventory Management with Kafka: How Retailers Are Eliminating Stockouts

Sachin Kamath — Fri, 10 Jul 2026 06:37:45 +0000

TL;DR

Retailers process thousands of inventory transactions every second across physical stores, eCommerce platforms, warehouses, suppliers, and fulfillment centers. Yet many inventory systems still rely on scheduled synchronization, causing stock levels to become outdated within minutes. The result is overselling, delayed replenishment, inaccurate inventory visibility, and avoidable stockouts.

Apache Kafka enables real-time inventory management by treating every inventory movement as an event that is streamed the moment it occurs. Sales, returns, warehouse transfers, supplier deliveries, and IoT sensor updates are continuously processed to maintain a consistent inventory view across all retail systems.

This event-driven approach helps retailers improve inventory accuracy, automate replenishment, detect stockouts before they occur, and respond to changing demand in near real time.

In this guide, you'll learn how Apache Kafka powers real-time inventory management, explore a production-ready reference architecture, understand how inventory events are processed across retail systems, and discover implementation best practices for building scalable, resilient inventory streaming applications.

Introduction

Retail inventory management has evolved far beyond tracking products on warehouse shelves. Today's retailers operate across physical stores, eCommerce platforms, online marketplaces, distribution centers, and supplier networks, where inventory levels change continuously throughout the day. Every sale, return, warehouse transfer, supplier delivery, and inventory adjustment impacts product availability, making accurate inventory visibility essential for delivering a seamless customer experience.

However, many retailers still rely on scheduled synchronization between Point-of-Sale (POS) systems, Warehouse Management Systems (WMS), Enterprise Resource Planning (ERP) platforms, and online storefronts. While these systems perform different functions, they all depend on accurate inventory data. When updates occur only every few minutes, hourly, or overnight, each application gradually develops its own view of inventory, leading to inconsistent stock levels across the business.

This lack of synchronization creates operational challenges that extend beyond inventory accuracy. Customers may order products that are no longer available, store associates struggle to locate stock, replenishment teams react too late to changing demand, and forecasting models rely on outdated information. As retailers expand across multiple sales channels, traditional inventory synchronization becomes increasingly difficult to scale.

Apache Kafka addresses these challenges by enabling real-time inventory management through event streaming. Instead of periodically exchanging inventory data, every inventory movement is published as an event and shared immediately with the systems that need it. This event-driven approach keeps inventory synchronized across retail applications while enabling capabilities such as real-time stock tracking, proactive stockout detection, automated replenishment, and demand sensing.

In this guide, we'll explore how retailers use Apache Kafka to build scalable, event-driven inventory platforms, examine a production-ready reference architecture, and discuss implementation best practices for reducing stockouts and improving inventory visibility across the supply chain.

Why Traditional Inventory Management Creates Stockouts

Retail inventory changes continuously. Every customer purchase, online order, return, warehouse transfer, supplier delivery, and inventory adjustment affects product availability. While these events happen in real time, many retail systems still exchange inventory data on a fixed schedule, creating a gap between actual inventory and what different applications believe is available.

As retailers expand across physical stores, eCommerce platforms, marketplaces, and fulfillment centers, this delay becomes increasingly difficult to manage. Inventory may be available in one system but unavailable in another, leading to inconsistent stock visibility, delayed replenishment, and dissatisfied customers.

The Problem with Nightly Batch Inventory Synchronization

For years, retailers relied on batch processing to synchronize inventory between operational systems. Scheduled jobs, file transfers, or periodic API calls updated inventory at fixed intervals, often every hour or overnight. While this approach reduced integration complexity, it also meant inventory data remained outdated until the next synchronization cycle.

A typical retail ecosystem includes several independent systems, each maintaining its own inventory records.

Point-of-Sale (POS) systems
Warehouse Management Systems (WMS)
Enterprise Resource Planning (ERP) platforms
eCommerce applications
Supplier management systems
Mobile inventory applications

Without continuous synchronization, these systems gradually develop different views of inventory throughout the day.

Synchronization Method	Update Frequency	Business Impact
Nightly batch jobs	Every 24 hours	Inventory remains outdated for most of the day.
Hourly synchronization	Every 60 minutes	Delayed visibility into inventory changes.
API polling	Every 5–30 minutes	Higher infrastructure overhead with stale data between polls.
Event streaming	Continuous	Inventory is updated as business events occur.

Even a short synchronization delay can have a significant business impact. During promotional campaigns or peak shopping periods, inventory may change hundreds of times before the next scheduled update, increasing the risk of overselling and stockouts.

Organizations moving away from scheduled synchronization often begin by replacing traditional ETL pipelines with streaming architectures.

How Inventory Becomes Inconsistent Across Retail Systems

Consider a retailer that starts the day with 150 units of a product in stock.

During the day, customers purchase products in-store, online orders reserve inventory, and warehouses replenish stock. Each system records these activities independently, but until the next synchronization cycle, every application reports a different inventory value.

Business Event	Actual Inventory	ERP	Online Store	POS
Opening inventory	150	150	150	150
12 products sold in-store	138	150	150	138
8 online orders placed	130	150	142	138
Warehouse replenishment received	170	150	142	138
Scheduled synchronization	170	170	170	170

For several hours, every system maintains a different inventory count. While the POS reflects recent sales, the ERP and online store continue operating with outdated information until synchronization completes.

This inconsistency creates several operational challenges.

Customers purchase products that are no longer available.
Online channels display incorrect stock levels.
Replenishment teams respond too late to inventory changes.
Store associates struggle to locate available inventory.
Safety stock increases to compensate for poor inventory visibility.
Demand forecasting relies on outdated inventory data.

As inventory volumes grow across multiple channels, these issues become increasingly difficult to solve using scheduled synchronization alone.

How Apache Kafka Enables Real-Time Inventory Management

Instead of synchronizing inventory databases at scheduled intervals, Apache Kafka enables retailers to process every inventory movement as an event the moment it occurs. Each business event—whether it's a product sale, customer return, warehouse transfer, or supplier delivery—is published to Kafka and made available to every downstream application in real time.

This event-driven approach ensures that inventory changes are shared continuously rather than periodically. As new events arrive, systems such as ERP, WMS, eCommerce platforms, analytics applications, and mobile inventory tools consume the same stream independently, allowing them to maintain a consistent view of inventory without relying on frequent database synchronization.

Beyond improving inventory visibility, this architecture decouples producers and consumers. A POS system only needs to publish a sales event once, while multiple downstream services can consume and process that event according to their own business requirements. This simplifies system integration, reduces operational complexity, and allows retail applications to scale independently.

Inventory Events That Drive Real-Time Stock Tracking

An effective real-time inventory management system captures every business event that changes inventory availability. Rather than streaming only sales transactions, retailers continuously publish events from across the supply chain to build an accurate and up-to-date inventory state.

Event Source	Inventory Event	Business Outcome
POS System	Product sold	Reduce available inventory immediately
eCommerce Platform	Order placed or cancelled	Synchronize inventory across sales channels
Warehouse Management System	Inventory received, picked, packed, or transferred	Maintain warehouse inventory accuracy
Supplier System	Shipment dispatched or delivered	Track incoming inventory
ERP	Inventory adjustments and purchase orders	Keep enterprise inventory aligned
Returns Management	Product returned	Restore sellable inventory
Smart Shelf Sensors	Item removed or shelf replenished	Monitor physical inventory in real time

Together, these events create a continuous stream of inventory activity that reflects what is happening across the business at any given moment.

From Business Events to a Unified Inventory View

Unlike traditional integrations, where every application communicates directly with multiple systems, Apache Kafka acts as a central event backbone.

When a product is sold, the POS system publishes a sales event to Kafka. The Inventory Service consumes the event and updates the available stock. At the same time, the ERP, warehouse management system, eCommerce platform, analytics tools, and monitoring dashboards can independently consume the same event without requiring additional integrations.

This publish-and-subscribe model enables retailers to build loosely coupled systems that remain synchronized while reducing the complexity of maintaining dozens of point-to-point integrations.

By processing every inventory event as it occurs, retailers gain a near real-time view of stock availability across stores, warehouses, and online channels. This provides the foundation for capabilities such as real-time stock tracking, automated replenishment, demand sensing, and proactive stockout detection.

Real-Time Inventory Management Architecture with Apache Kafka

A production-ready inventory platform combines operational systems, IoT devices, event streaming, and stream processing to maintain a continuously updated view of inventory across the retail ecosystem.

Instead of synchronizing inventory databases, every inventory movement is published as an event and processed in real time. This ensures that all downstream applications receive the same information as soon as it becomes available.

Every system publishes only the events it owns. A POS system publishes completed sales, warehouse systems publish inventory movements, suppliers publish shipment updates, and smart shelf sensors report physical inventory changes. Kafka distributes these events to every downstream consumer, allowing each application to process the same inventory stream independently.

This architecture eliminates point-to-point integrations and provides a single event backbone for inventory operations.

Using MQTT for Real-Time Shelf Monitoring

While transactional systems capture sales and warehouse operations, they often don't detect what happens on the store floor before a transaction occurs.

For example:

A customer removes an item from a shelf.
A store associate replenishes inventory.
Products are moved to a different display.
Items are misplaced or damaged.
Shelf inventory reaches a critical threshold.

These events are invisible to traditional inventory systems until a manual inventory count or a sales transaction updates the database.

Smart shelves equipped with RFID readers, weight sensors, barcode scanners, or computer vision systems can continuously publish these changes using MQTT. An MQTT source connector ingests these events into Kafka, making them immediately available to inventory services, ERP platforms, monitoring dashboards, and analytics applications.

For organizations building IoT-enabled retail solutions, MQTT connectors simplify the ingestion of real-time device data into streaming pipelines.

Simplifying Retail Inventory Pipelines with Condense

Building the architecture above involves more than deploying an Apache Kafka cluster. Engineering teams must integrate data from retail applications, ingest IoT events, process inventory streams, monitor pipeline health, and ensure the platform scales reliably as transaction volumes grow.

Condense simplifies these operational challenges by providing a unified platform for building and managing real-time streaming applications. With fully managed Kafka, enterprise connectors, visual pipeline development, and built-in stream processing, teams can develop production-ready inventory pipelines without managing the underlying streaming infrastructure.

For retail inventory workloads, Condense enables teams to:

Ingest events from POS systems, ERP platforms, warehouses, suppliers, databases, and MQTT-enabled devices.
Design and deploy inventory pipelines through a visual interface or code-based workflows.
Process inventory events using built-in transformations and stream processing capabilities.
Monitor pipeline health, throughput, and consumer performance from a centralized dashboard.
Deploy securely within their own cloud environment using a Bring Your Own Cloud (BYOC) model.

By abstracting much of the operational complexity, Condense allows engineering teams to focus on building inventory applications instead of maintaining the streaming platform that powers them.

Detecting Stockouts and Predicting Demand

Real-time inventory management isn't just about knowing how many products are available. The real value comes from continuously analyzing inventory events to detect potential stockouts, automate replenishment, and respond to changing customer demand before it impacts sales.

With Apache Kafka acting as the event backbone, retailers can correlate events from multiple systems as they occur, creating a live inventory view that reflects the current state of every product across stores, warehouses, and fulfillment centers.

Correlating Inventory Events in Real Time

A stockout rarely occurs because inventory suddenly reaches zero. More often, it happens because different systems have different views of inventory.

Consider a common retail scenario:

A customer purchases the last two units of a product in a store.
The POS system immediately records the sale.
A warehouse has already dispatched replacement inventory.
The shipment is still in transit.

The ERP continues to show inventory as available because the latest warehouse update hasn't been processed.

Without a shared event stream, each system operates independently, making it difficult to determine the true inventory position.

By streaming these events through Apache Kafka, retailers can correlate sales, warehouse movements, supplier deliveries, and inventory updates as they occur. Instead of waiting for scheduled synchronization, every new event contributes to a continuously updated inventory state.

Event Source	Event	Inventory Impact
POS	Product sold	Reduce available inventory
Warehouse	Shipment dispatched	Inventory replenishment in transit
Store Receiving	Shipment received	Increase available inventory
Supplier	Delivery confirmed	Update inbound inventory
Shelf Sensor	Shelf empty	Confirm physical stock depletion

This continuous event processing enables inventory services to maintain an accurate inventory position while ensuring every downstream application works with the same information.

Real-Time Stock Tracking Across Retail Channels

Modern inventory exists in multiple locations simultaneously. The same product may be available in a retail store, regional warehouse, fulfillment center, or online marketplace.

As inventory moves between these locations, every change needs to be reflected across all customer-facing and operational systems.

With an event-driven architecture, each inventory movement is published once and consumed by multiple applications independently. This enables:

Consistent inventory visibility across physical and online channels.
Faster inventory updates for ERP, WMS, and eCommerce platforms.
Improved order allocation and fulfillment decisions.
Reduced risk of overselling.
Better inventory accuracy across the retail network.

Instead of synchronizing databases throughout the day, retailers continuously update inventory as business events occur.

Using Streaming Data for Demand Sensing

Traditional forecasting models primarily rely on historical sales reports. While useful for long-term planning, they often fail to capture sudden changes in customer demand caused by promotions, seasonal events, regional trends, or unexpected buying patterns.

Streaming data provides a more dynamic approach.

As inventory events flow through Kafka, retailers can continuously evaluate metrics such as:

Current sales velocity
Inventory depletion rate
Product return patterns
Warehouse replenishment frequency
Supplier lead times
Regional demand trends

Rather than generating reports at the end of the day, these metrics are updated continuously, giving supply chain teams a real-time view of changing demand.

This allows retailers to identify fast-moving products earlier, prioritize replenishment, and make inventory decisions based on current business activity instead of historical snapshots.

Enabling Automated Restocking

Once inventory events are available as a continuous stream, retailers can automate replenishment based on predefined business rules.

For example, a stream processing application can evaluate conditions such as:

Shelf inventory falls below the minimum threshold.
Sales velocity exceeds the expected rate.
Replacement inventory is available in a nearby warehouse.
Supplier lead times remain within acceptable limits.

When these conditions are met, the application can automatically generate a restocking recommendation or notify store operations before customers encounter empty shelves.

By combining real-time stock tracking with continuous event processing, retailers shift from reacting to stockouts after they occur to preventing them altogether. This improves product availability, reduces lost sales, and enables more responsive inventory operations across the retail network.

Best Practices for Building Real-Time Inventory Management Pipelines

Building a real-time inventory management platform requires more than deploying Apache Kafka. Retailers need to ensure inventory events are accurate, ordered, scalable, and resilient as transaction volumes grow across stores, warehouses, suppliers, and online channels.

The following best practices help engineering teams build reliable inventory streaming applications while maintaining inventory consistency across the business.

Design Events Around Business Activities

Instead of publishing periodic inventory snapshots, publish business events that represent changes to inventory.

Examples include:

Product Sold
Inventory Reserved
Inventory Released
Goods Received
Warehouse Transfer Completed
Customer Return Processed
Inventory Adjusted

An event-driven model preserves the complete history of inventory changes, making it easier to replay events, audit inventory movement, and recover application state when needed.

Organize Topics for Scalability

Create dedicated Kafka topics for different inventory event types instead of combining unrelated events into a single stream.

A common topic strategy includes:

Sales-events
Returns-events
Warehouse-events
Supplier-events
Inventory-adjustments
shelf-events

This separation simplifies event processing, improves scalability, and allows individual consumers to subscribe only to the events they require.

Preserve Event Ordering

Inventory updates for the same product should always be processed in sequence.

Partitioning topics using business identifiers such as SKU ID, Store ID, or Warehouse ID helps maintain event ordering while allowing the platform to scale horizontally.

Proper partitioning reduces the risk of inventory inconsistencies caused by out-of-order processing.

Monitor Streaming Pipelines Continuously

Operational visibility is essential for maintaining a healthy streaming platform.

Engineering teams should continuously monitor:

Consumer lag
Processing latency
Event throughput
Failed messages
Topic storage utilization
Consumer availability

Monitoring these metrics helps identify bottlenecks before they impact inventory accuracy or downstream applications.

Secure and Validate Inventory Events

Because inventory data directly influences purchasing, fulfillment, and replenishment decisions, every event should be validated before it updates inventory state.

Production deployments should also implement encryption, role-based access control, and schema validation to ensure inventory data remains secure and consistent across all connected systems.

Implementation Checklist

Before deploying a real-time inventory management platform, verify the following:

Area	Recommendation
Event Design	Publish business events instead of inventory snapshots.
Topic Strategy	Use dedicated topics for sales, returns, warehouse, supplier, and inventory events.
Partitioning	Partition by SKU, Store ID, or Warehouse ID to preserve event ordering.
Schema Management	Use versioned schemas to support application evolution.
Consumer Scaling	Scale consumer groups independently based on workload.
Monitoring	Track consumer lag, throughput, processing latency, and failed events.
Security	Encrypt data in transit and implement role-based access control.
Resilience	Configure replication, retries, and disaster recovery for high availability.

Following these practices helps retailers build scalable inventory streaming pipelines that maintain accurate inventory visibility while supporting millions of inventory events across distributed retail operations.

Building Real-Time Inventory Applications with Condense

Apache Kafka provides the event streaming foundation for real-time inventory management, but building a production application requires more than moving events between systems. Inventory events need to be transformed, enriched with business context, validated, routed to downstream applications, and monitored throughout their lifecycle.

Condense extends Apache Kafka with a unified application platform that enables engineering teams to build, deploy, and operate these event-driven applications without assembling multiple development and operational tools.

For a retail inventory use case, a typical application built on Condense can:

Enrich POS events with product and store metadata before updating inventory.
Correlate sales, warehouse, supplier, and shelf events to maintain a real-time inventory state.
Trigger restocking workflows when inventory falls below predefined thresholds.
Route inventory updates to ERP, WMS, eCommerce platforms, analytics systems, and operational dashboards from a single processing pipeline.
Monitor application performance, processing latency, and event flow throughout the pipeline.

By combining application development, stream processing, deployment, and operations in a single platform, Condense enables teams to focus on delivering business logic instead of integrating and maintaining multiple components across the streaming stack.

Conclusion

Modern retail depends on accurate and timely inventory information. As businesses expand across physical stores, eCommerce platforms, warehouses, and supplier networks, scheduled inventory synchronization can no longer keep pace with the volume and speed of inventory changes.

By adopting real-time inventory management with Apache Kafka, retailers can process inventory events as they occur, maintaining a consistent view of stock across every operational system. This event-driven approach supports real-time stock tracking, faster replenishment, proactive stockout detection, and demand sensing, enabling retailers to respond quickly to changing customer demand.

Building these applications requires more than a messaging platform. Teams need the ability to develop, deploy, operate, and scale event-driven applications that transform inventory events into business outcomes. Condense simplifies this process by providing a unified platform for building production-ready streaming applications on Apache Kafka, enabling retailers to accelerate development while maintaining the flexibility and scalability of an event-driven architecture.

Frequently Asked Questions

What is real-time inventory management?
Real-time inventory management continuously updates inventory as business events occur instead of relying on scheduled synchronization. Every sale, return, warehouse transfer, supplier delivery, or inventory adjustment immediately updates inventory across connected systems, providing accurate stock visibility throughout the retail network.

Why is Apache Kafka used for inventory management?
Apache Kafka acts as the event backbone for inventory management. It enables retailers to stream inventory events in real time so that POS systems, ERP platforms, Warehouse Management Systems (WMS), eCommerce applications, and analytics services all consume the same inventory data without relying on batch synchronization.

How does event-driven inventory management reduce stockouts?
Event-driven inventory management processes inventory events as they occur. By correlating sales, returns, warehouse transfers, supplier deliveries, and shelf events in real time, retailers can detect low inventory sooner, trigger replenishment workflows automatically, and maintain accurate inventory across every sales channel.

Can Apache Kafka integrate with existing retail systems?
Yes. Apache Kafka integrates with systems such as POS, ERP, Warehouse Management Systems (WMS), supplier applications, eCommerce platforms, databases, and IoT devices. This allows organizations to modernize inventory management without replacing existing business applications.

How do MQTT-enabled shelf sensors improve inventory visibility?
Smart shelves equipped with RFID readers, barcode scanners, weight sensors, or computer vision systems publish inventory events using MQTT. These events are streamed into Kafka, providing near real-time visibility into shelf inventory and enabling faster replenishment decisions.

What role does Condense play in a real-time inventory architecture?
While Apache Kafka provides the event streaming foundation, Condense provides the application platform for building inventory solutions on top of Kafka. Teams can develop applications that correlate inventory events, maintain inventory state, automate replenishment workflows, expose APIs, and deploy production-ready streaming applications without assembling multiple development and operational tools.

Why build inventory applications with Condense instead of using Apache Kafka alone?
Apache Kafka excels at transporting and storing events, but production applications also require business logic, application development, deployment, monitoring, lifecycle management, and operations. Condense brings these capabilities together in a unified platform, allowing engineering teams to focus on solving business problems instead of integrating and maintaining multiple frameworks.

Can Condense work with an existing Kafka deployment?
Yes. Condense is designed to build and run applications on existing Apache Kafka deployments, including self-managed Kafka clusters, managed Kafka services, and Bring Your Own Cloud (BYOC) environments. Organizations can continue using their preferred Kafka infrastructure while accelerating application development and operations.

How does Condense accelerate the development of retail streaming applications?
Condense combines application development, stream processing, deployment, observability, and lifecycle management into a single platform. This enables engineering teams to move from architecture to production faster while reducing the operational effort required to build and maintain event-driven inventory applications.

Is this architecture suitable only for retail inventory management?
No. The same event-driven architecture can be applied to connected manufacturing, logistics, supply chain visibility, fleet management, healthcare, industrial IoT, and other industries where business decisions depend on processing continuous streams of real-time events.

Unlock Kafka Schemas with Karapace: A Hands-On Guide

Sachin Kamath — Thu, 18 Jun 2026 11:29:21 +0000

TL;DR

Kafka handles event streaming: producers write to topics, consumers read. JSON parsing lags at scale, so use Karapace Schema Registry (open-source Confluent alternative) with Protobuf for efficient, schema-enforced serialization. This Java Spring Boot guide shows producer/consumer setup, compile .proto schemas, use TopicNameStrategy, SCRAM/Basic auth tested on Condense's Kafka+Karapace. Ideal for structured telematics; avoid for ad-hoc data.

A Brief Introduction to Kafka

Kafka is a distributed system consisting of servers and clients that communicate using the TCP network protocol. Kafka is used as an event-based system that records a specific event that has happened.

For example "Transaction of 500 USD has occurred in Alice's account."

Events are created and written to Kafka by client applications called Producers and consumed by client applications called Consumers. Kafka provides the feature to create topics, which can be used to group events as per the user's requirements. Single or multiple producers can produce events/messages to a single topic, and single or multiple consumers can consume from one topic.

Common Throughput Issues When Operating at Scale

By default, Kafka producers and consumers exchange event data in the String UTF-8 format. Even though Kafka offers features to compress the data using gzip and other algorithms, this creates additional overhead on the producers and consumers to decompress the data and then validate it. Most events are generally sent in the JSON format, which is helpful for human readability, but this slows down the whole pipeline if high throughput is needed. The effects can only be felt on production clusters where the incoming data transfer rate is high, but the consumers start lagging due to complex JSON deserialization if there are nested JSONs present.

To overcome this issue, we need a different serialization and deserialization format that can reduce the time taken to construct or deconstruct the binary data sequence and ensure that the data follows a certain schema. We can either write our own serializer and deserializer or use open-source libraries from Confluent to achieve it. Confluent offers JSON, Avro, and Protobuf serde (serializer and deserializers) to compact the data and enforce a schema on it.

Karapace Schema Registry & Kafka

Karapace is an open source near drop-in replacement for the Confluent Schema Registry, which aims to be compatible with the Confluent Schema Registry Client libraries, which are offered in Java, C#, etc. In this blog post, we will be looking at the Java libraries for implementing a producer and consumer.

The Schema Registry is a separate component from Kafka, as it is not offered by Apache. It is designed to run as a separate node that is dependent on Kafka, but Kafka does not need Schema Registry as a core functionality. This can lead to confusion as Kafka does not enforce the schema for a particular topic or record, but the producer and consumer can choose to do so by using the Schema Registry. Unlike a relational database schema, where a table can enforce a very strict schema down to the datatypes, size of the data in a column, and other constraints regardless of the users connecting to it, different Kafka producers could still write very different data to a topic while not following a schema or format, causing downstream issues in a pipeline. To avoid this, we can use different strategies to minimize the chances of schema drift or unintended data storage in topics, which we will cover in this post.

DISCLAIMER

Setting up Kafka and Schema Registry in a local environment requires a lot of configuration steps. I will be using a platform called Condense by the company Zeliot, which provides an integrated Kubernetes + Kafka + Schema Registry (Karapace), along with the options to set ACLs for Kafka and Schema Registry from a browser UI instead of a command line. I will cover setting up a local Kafka cluster with Schema Registry in a separate blog post.

How Kafka & Schema Registry Work Together

Since Kafka and Schema Registry are separate components, only the producers and consumers are responsible for setting up a loosely coupled connection between them. A basic flow diagram of how they work together is given below:

Here, this architecture assumes that we are using the Confluent Schema Registry Client in both the producer and consumer. We can use compiled classes without the Schema Registry, but that approach has disadvantages, such as a lack of versioning support without downtime, maintenance of compiled classes, complex logic, and library setup needed to generate and use compiled classes.

With Kafka 4.0 and its KRaft-only architecture, schema registry compatibility is an important consideration for teams upgrading.

The process of generating a message in JSON, Avro, or Protobuf format involves the producer specifying the naming strategy and version of the schema (default is the latest version), which is fetched from the Confluent client, and then we need to use the format-specific libraries as specified in the next sections to convert the message from String format to the required format. Note that the message format must be the same as specified in the schema type, or else the Kafka producer library will throw a serialization exception. After serialization, this message is sent to the Kafka topic, which is specified.

For the consumer, the Confluent client library automatically sees the 1st byte called the magic byte, which determines whether the message is using the Schema Registry or not. 1 means yes and 0 means no. If yes, the next 4 bytes of the message determine the schema ID present in the Registry, and the client automatically fetches the correct schema for the message. Then the message is deserialized into the String UTF-8 format and can be viewed and processed normally.

Subject Naming Strategies

There are 3 different naming strategies offered by the Confluent Schema Registry Client libraries to determine how the serializer libraries fetch the schema to correctly serialize the message.

TopicName Strategy

This is the default strategy for naming the subject and is <topic-name> + ”-value”. If a topic name is car-topic, then the corresponding subject name is car-topic-value. For the key, the subject name is <topic-name> + ”-key”. The serializer and deserializer using this strategy assume that all the messages in this topic correspond to this schema, and the presence of messages in other formats can disrupt the consumer with errors. Best used in cases where the topic name is used to differentiate the type of data present in the topic.

RecordNameStrategy

This is a naming strategy for naming the subject using the record name of the schema (in the above example, the Record Name is Car). This does not tie the producer and consumer to a particular topic, but consuming messages in other formats/schemas can cause the consumer to crash if error-handling mechanisms aren’t implemented correctly. Best used in cases where producers need to immediately push data into any available topic with a high number of partitions, replication factors, etc.

TopicRecordNameStrategy

This strategy uses both the topic name and the record name to determine the subject name. This strategy can be used when we are sure that a topic can have multiple record types.

Setting Up a Producer with Protobuf Schema

First, we need to register a subject within Karapace, which defines the schema of the message containing the fields and their data types. I am defining the following proto schema:

syntax = "proto3";

package com.example.GenericProducer.schema;

option java_package = "com.example.GenericProducer.schema";

option java_outer_classname = "CarProto";

message Car {

string carId = 1;

string carNumber = 2;

double speed = 3;

double latitude = 4;

double longitude = 5;

}

This defines a schema in which the data should contain the fields carId, carNumber of type string, and speed, latitude, and longitude of type double. To get a higher data generation speed, I am using compiled classes for Protobuf.

In your Spring Java project, create a folder src/main/java named schema and create a file named Car.proto. Paste the above schema and modify your Java package and java_package to your requirements.

Next, install the protobuf compiler for your OS by following the instructions here. Next, go to the root directory of your project and enter the command to compile the proto file and generate a compiled class. Modify the path if needed. The first parameter sets the path for the output compiled class, and the second accepts the path for the proto file.

protoc --java_out=src/main/java/ src/main/java/schema/car.proto

We can check the generated file to see if it is created or not. Do not edit the file.

Next, we can add the required dependencies pom.xml for the producer application. Below are the following dependencies. We can use the latest versions of these libraries from the mvn repository and the Confluent repository.

<dependency> 

  <groupId>org.apache.kafka</groupId> 

  <artifactId>kafka-clients</artifactId> 

  <version>3.9.0</version> 

</dependency> 

<dependency> 

  <groupId>io.confluent</groupId> 

  <artifactId>kafka-protobuf-serializer</artifactId> 

  <version>8.0.1</version> 

</dependency> 

<dependency> 

  <groupId>io.confluent</groupId> 

  <artifactId>kafka-protobuf-provider</artifactId> 

  <version>8.0.1</version> 

</dependency> 

<dependency> 

  <groupId>com.google.protobuf</groupId> 

  <artifactId>protobuf-java</artifactId> 

  <version>4.33.4</version> 

  <scope>compile</scope> 

</dependency>

To add the io.confluent packages, we need to add the Confluent repository, as these packages are not present in the Maven repository.

We can add it just before the </project> tag in the pom.xml like this:

...

</plugin> 

</plugins> 

</build> 

<repositories> 

<repository> 

<id>confluent</id> 

<name>Confluent Maven Repository</name> 

<url>https://packages.confluent.io/maven/</url> 

</repository> 

</repositories> 

</project>

Since this is a sample producer, we shall randomly generate data every 5 seconds to Kafka to simulate telematics data. We shall define a class just like our schema.

import lombok.AllArgsConstructor; 

import lombok.Data; 

import lombok.NoArgsConstructor; 


@Data 

@AllArgsConstructor 

@NoArgsConstructor 

public class Car { 

private String carId; 

private String carNumber; 

private double speed; 

private double latitude; 

private double longitude; 

}

Next, we shall define a KafkaProducer as a Spring component that can be invoked once within a service and reused.

@Component 

public class KafkaProducerClient { 


public static final String FALSE = "false"; 

private static final String CLIENT_ID = "car-producer"; 

private static final String SECURITY_PROTOCOL = "SASL_PLAINTEXT"; 

private static final String SASL_MECHANISM = "SCRAM-SHA-512"; 

private static final String SASL_JAAS_CONFIG = "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"%s\" password=\"%s\";"; 

private final String bootstrapServerUrl; 


public KafkaProducerClient(@Value("${kafka.bootstrap.server.url}") String bootstrapServerUrl) { 

this.bootstrapServerUrl = bootstrapServerUrl; 

} 

public <K, V> KafkaProducer<K, V> getDefaultProducerClientWithoutPartitioner(String username, String password, @Value("${schema.registry.url}") String schemaRegistryUrl) { 

Properties props = new Properties(); 

props.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, “org.apache.kafka.common.serialization.StringSerializer”); 


props.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer"); 


props.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServerUrl); 

props.setProperty(ProducerConfig.CLIENT_ID_CONFIG, CLIENT_ID); 

props.setProperty(ProducerConfig.RETRIES_CONFIG, "0"); 

props.setProperty(ProducerConfig.MAX_BLOCK_MS_CONFIG, "3000"); 

props.setProperty(AdminClientConfig.SECURITY_PROTOCOL_CONFIG, SECURITY_PROTOCOL); 

props.setProperty(SaslConfigs.SASL_MECHANISM, SASL_MECHANISM); 

props.setProperty(SaslConfigs.SASL_JAAS_CONFIG, String.format(SASL_JAAS_CONFIG, username, password)); 

props.setProperty(AbstractKafkaSchemaSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl); 

props.setProperty("basic.auth.credentials.source", "USER_INFO"); 

props.setProperty("basic.auth.user.info", username + ":" + password); 

props.setProperty(AbstractKafkaSchemaSerDeConfig.AUTO_REGISTER_SCHEMAS, “true”); 

props.setProperty(AbstractKafkaSchemaSerDeConfig.LATEST_COMPATIBILITY_STRICT, "false"); 

props.setProperty(AbstractKafkaSchemaSerDeConfig.VALUE_SUBJECT_NAME_STRATEGY,  

"io.confluent.kafka.serializers.subject.TopicNameStrategy"); 

return new KafkaProducer<>(props); 

} 

}

Here, we set the properties of the key and value serializers to String and Protobuf, respectively. We then set the authentication mechanism for the Kafka cluster (SCRAM-SHA-512 in my case) with the username and password.

I am using basic authentication in Karapace, so we need to set the "basic.auth.credentials.source" to “USER_INFO” which sets the KafkaProducer property to authenticate with the Schema registry using the Basic Authentication headers. We then need to set "basic.auth.user.info" the user’s credentials in the format shown in the code. If we are not using any authentication for the Schema Registry, we can omit these properties.

I am using the Topic Name Strategy for subjects for simplicity and demo purposes for the AbstractKafkaSchemaSerDeConfig.VALUE_SUBJECT_NAME_STRATEGY property. We can set AUTO_REGISTER_SCHEMAS to true if the schema doesn’t exist in the registry, or it will create a new version of the schema if the underlying compiled class schema has changed. The created subject will follow the naming strategy as defined in the above property. To allow compatibility between various versions and generate messages, we have set the LATEST_COMPATIBILITY_STRICT to false.

Now, we shall set up a random data generator class to generate some sample data.

@Component 

@NoArgsConstructor 

public class RandomCarDataGenerator { 


@Value("${CAR_ID}") 

private String carIDString; 


@Value("${CAR_NUMBER}") 

private String carNumberString; 


private static final Random random = new Random(); 


public Car generateRandomCar() { 

Car car = new Car(); 

String carId = carIDString; 

String carNumber = carNumberString; 



if (carId == null || carId.isEmpty()) { 

carId = "default-car-id"; 

} 

if (carNumber == null || carNumber.isEmpty()) { 

carNumber = "default-car-number"; 

} 

car.setCarId(carId); 

car.setCarNumber(carNumber); 

car.setSpeed(generateRandomSpeed()); 

car.setLatitude(generateRandomLatitude()); 

car.setLongitude(generateRandomLongitude()); 

return car; 

} 


private static double generateRandomLatitude() { 

// Range: -90 to 90 

return -90 + (180 * random.nextDouble()); 

} 


private static double generateRandomLongitude() { 

// Range: -180 to 180 

return -180 + (360 * random.nextDouble()); 

} 


private static double generateRandomSpeed() { 

// Random speed between 0 and 180 km/h 

return random.nextDouble() * 180; 

} 

} 


Next, we shall set up the producer class, which initializes a KafkaProducer and has a lightweight function to convert the generated data into a Protobuf message using the earlier generated class “CarProto.Car”. Ensure the environment variables are configured properly to avoid errors.


@Service 

@RequiredArgsConstructor 

@Slf4j 

public class ProtobufProducer { 



private static final String PROTO_TOPIC = "test-schema-car-protobuf"; 



private final KafkaProducerClient kafkaProducerClient; 

private KafkaProducer<String, CarProto.Car> protoProducer; 



@Value("${schema.registry.url}") 

private String schemaRegistryUrl; 



@Value("${kafka.username}") 

private String username; 



@Value("${kafka.password}") 

private String password; 



@PostConstruct 

private void initProtoProducer(){ 

protoProducer = kafkaProducerClient.getDefaultProducerClientWithoutPartitioner( 

username, password, schemaRegistryUrl); 

} 



public void produceCarProto(Car car) { 

try { 

// Direct conversion from POJO to Protobuf - NO JSON overhead! 

CarProto.Car protoCar = CarProto.Car.newBuilder() 

.setCarId(car.getCarId()) 

.setCarNumber(car.getCarNumber()) 

.setSpeed(car.getSpeed()) 

.setLatitude(car.getLatitude()) 

.setLongitude(car.getLongitude()) 

.build(); 



ProducerRecord<String, CarProto.Car> producerRecord =  

new ProducerRecord<>(PROTO_TOPIC, car.getCarId(), protoCar); 

protoProducer.send(producerRecord, (metadata, exception) -> { 

if (exception == null) { 

log.info("Produced Proto message topic={} partition={} offset={}", 

metadata.topic(), metadata.partition(), metadata.offset()); 

} else { 

log.error("Error producing Proto message", exception); 

} 

}); 

} catch (Exception e) { 

log.error("Error producing Proto message", e); 

} 

} 

}

The above code initializes an KafkaProducer at startup and whenever the producerCarProto is called, the builder associated with the CarProto.Car is built from the car object and a ProducerRecord is initialized with the CarProto.Car type. Then the protobuf message is sent to the Kafka topic test-schema-car-protobuf.

Now, we shall initialize a class to periodically send messages every 5 seconds.

@Service 

@RequiredArgsConstructor 

public class ProducerScheduleService { 



private final RandomCarDataGenerator carDataGenerator; 

private final ProtobufProducer protobufProducer; 

private final AvroProducer avroProducer; 

@Scheduled(fixedRate = 5000) 

public void produceCarToBothFormats() { 

Car car = carDataGenerator.generateRandomCar(); // generate once 

@Cleanup 

ExecutorService executorService = Executors.newVirtualThreadPerTaskExecutor(); 

executorService.submit(()->protobufProducer.produceCarProto(car)); 

} 

}

This will produce a Protobuf message to the Kafka topic every 5 seconds with random data.

Similarly, we shall have a separate consumer that can consume messages from this topic and deserialize these messages faster than the normal String JSON format. Let us create a new Spring Boot service with a Kafka Consumer that is set up with a Protobuf deserializer and specifically requests this schema to deserialize the messages.

@Component 

public class KafkaConsumerClient { 



public static final String FALSE = "false"; 

private static final String GROUP_ID = "car-consumer"; 

private static final String CLIENT_ID = "car-consumer1"; 

private static final String SECURITY_PROTOCOL = "SASL_PLAINTEXT"; 

private static final String SASL_MECHANISM = "SCRAM-SHA-512"; 

private static final String SASL_JAAS_CONFIG = "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"%s\" password=\"%s\";"; 

private final String bootstrapServerUrl; 



public KafkaConsumerClient(@Value("${kafka.bootstrap.server.url}") String bootstrapServerUrl) { 

this.bootstrapServerUrl = bootstrapServerUrl; 

} 



public <K, V> KafkaConsumer<K, V> getDefaultConsumer(String username, String password, @Value("${schema.registry.url}") String schemaRegistryUrl){ 

Properties props = new Properties(); 

props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServerUrl); 

props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"); 

props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer"); 

props.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP_ID); 

props.put(ConsumerConfig.CLIENT_ID_CONFIG, CLIENT_ID); 

props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); 

props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true"); 

props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 100); 

props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 30000); 

props.setProperty(AdminClientConfig.SECURITY_PROTOCOL_CONFIG, SECURITY_PROTOCOL); 

props.setProperty(SaslConfigs.SASL_MECHANISM, SASL_MECHANISM); 

props.setProperty(SaslConfigs.SASL_JAAS_CONFIG, String.format(SASL_JAAS_CONFIG, username, password)); 

props.setProperty(AbstractKafkaSchemaSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl); 

props.setProperty("basic.auth.credentials.source", "USER_INFO"); 

props.setProperty("basic.auth.user.info", username + ":" + password); 

props.put("specific.protobuf.value.type", "com.example.schema.CarProto$Car"); 

return new KafkaConsumer<>(props); 

} 

}

Here, we initialize a KafkaConsumer that has a String key deserializer and a Protobuf value deserializer. We can set the consumer group ID and client ID by using the ConsumerConfig.GROUP_ID_CONFIG and ConsumerConfig.CLIENT_ID_CONFIG.
We can set properties such as the auto reset, auto offset commit, and polling periods based on the requirements and hardware capabilities.
We need to set the Schema Registry URL in the properties and the authentication properties, if needed, just like the producer.
Since we are using the Topic Name Strategy for the subject, we are sure that all messages in this topic will be of the CarProto.Car schema for which we generated the compiled class in the producer. This is an optional property and can be omitted if we are not sure all messages are of the same schema. This helps deserialize messages from the topic if there are multiple topics or the Record Naming Strategy or TopicRecord Naming Strategy is used.

We shall now create a consumer class that keeps running once the application is started.

package com.example.GenericConsumer.services; 

import java.time.Duration; 

import java.util.Collections; 

import java.util.concurrent.ExecutorService; 

import java.util.concurrent.Executors; 

import java.util.concurrent.atomic.AtomicBoolean; 



import org.apache.kafka.clients.consumer.ConsumerRecord; 

import org.apache.kafka.clients.consumer.ConsumerRecords; 

import org.apache.kafka.clients.consumer.KafkaConsumer; 

import org.apache.kafka.common.errors.WakeupException; 

import org.springframework.beans.factory.annotation.Value; 

import org.springframework.context.SmartLifecycle; 

import org.springframework.stereotype.Service; 



import com.example.GenericConsumer.clients.KafkaConsumerClient; 

import com.example.GenericConsumer.enums.KafkaDeserializerTypes; 

import com.example.schema.CarProto.Car; 



import lombok.RequiredArgsConstructor; 

import lombok.extern.slf4j.Slf4j; 



@Service 

@RequiredArgsConstructor 

@Slf4j 

public class CarConsumerService implements SmartLifecycle { 



private final KafkaConsumerClient kafkaConsumerClient; 



@Value("${kafka.username}") 

private String username; 



@Value("${kafka.password}") 

private String password; 



@Value("${schema.registry.url}") 

private String schemaRegistryUrl; 



private static final String PROTO_TOPIC = "test-schema-car-protobuf"; 

private ExecutorService executorService; 

private KafkaConsumer<String, Car> consumer; 

private final AtomicBoolean running = new AtomicBoolean(false); 

private volatile boolean started = false; 



@Override 

public void start() { 

if (started) { 

return; 

} 

log.info("Starting Kafka consumer in virtual thread..."); 

// Create thread executor 

executorService = Executors.newSingleThreadExecutor(r -> { 

Thread thread = new Thread(r); 

thread.setName("kafka-consumer-thread"); 

thread.setDaemon(false); 

return thread; 

}); 

running.set(true); 

started = true; 

// Submit consumer task to executor 

executorService.submit(this::consumeCar); 

log.info("Kafka consumer started successfully"); 

} 



private void consumeCar() { 

try { 

consumer = kafkaConsumerClient.getDefaultConsumer( 

username,  

password,  

schemaRegistryUrl 

); 

consumer.subscribe(Collections.singleton(PROTO_TOPIC)); 

log.info("Subscribed to topic: {}", PROTO_TOPIC); 

while (running.get()) { 

try { 

ConsumerRecords<String, Car> records = consumer.poll(Duration.ofMillis(1000)); 

for (ConsumerRecord<String, Car> carRecord : records) { 

log.info("Consumed Car Record - Key: {}, Value: {}, Partition: {}, Offset: {}",  

carRecord.key(), 

carRecord.value(),  

carRecord.partition(), 

carRecord.offset() 

); 

} 

if (!records.isEmpty()) { 

consumer.commitAsync(); 

} 

} catch (WakeupException e) { 

log.info("Consumer wakeup called, shutting down..."); 

break; 

} 

} 

} catch (Exception e) { 

log.error("Error in consumer loop", e); 

} finally { 

log.info("Closing Kafka consumer..."); 

if (consumer != null) { 

try { 

consumer.close(Duration.ofSeconds(5)); 

} catch (Exception e) { 

log.error("Error closing consumer", e); 

} 

} 

log.info("Kafka consumer closed"); 

} 

} 



@Override 

public void stop() { 

log.info("Shutting down Kafka consumer service..."); 

running.set(false); 

if (consumer != null) { 

consumer.wakeup(); 

} 

if (executorService != null) { 

executorService.shutdown(); 

try { 

if (!executorService.awaitTermination(10, java.util.concurrent.TimeUnit.SECONDS)) { 

log.warn("Executor did not terminate in time, forcing shutdown"); 

executorService.shutdownNow(); 

} 

} catch (InterruptedException e) { 

log.error("Shutdown interrupted", e); 

executorService.shutdownNow(); 

Thread.currentThread().interrupt(); 

} 

} 

started = false; 

log.info("Kafka consumer service shutdown complete"); 

} 



@Override 

public boolean isRunning() { 

return started; 

} 



@Override 

public int getPhase() { 

// Return a phase value to control startup order 

// Higher values start later and stop earlier 

return Integer.MAX_VALUE; 

} 



@Override 

public boolean isAutoStartup() { 

// Return true to start automatically when Spring context is ready 

return true; 

} 



@Override 

public void stop(Runnable callback) { 

stop(); 

callback.run(); 

} 

}

This will initialize a KafkaConsumer, which reads and deserializes messages from the test-schema-car-protobuf topic and displays them in the logs.

When to Use Protobuf in Your Pipeline

Protobuf messages are efficient for high-volume, high-throughput scenarios where messages follow a schema with very few optional attributes and variances in the fields.
If the messages are machine-to-machine, and the components in the pipeline are capable of serialization or deserialization.
If there are schemas enforced on the data and evolve with a clear forward/backward pattern

When Not to Use Protobuf in Your Pipeline

If the data needs to be viewed by humans immediately, with no extra computing overhead, then use a normal String format
If the data doesn’t really follow any schemas and can change on an ad-hoc basis, all advantages of Protobuf are negated, and complexity increases unnecessarily.
Protobuf isn’t suitable for applications that require continuous streaming, like file data, videos, etc., as each message is meant to be processed as a whole record.

GitHub Links

Resources

For production teams, managing Karapace alongside Kafka brokers, connectors, and stream processors adds significant operational overhead, one of the core reasons teams move to fully managed streaming platforms.

Frequently Asked Questions (FAQs)

Why does adding Schema Registry make Kafka harder to operate?
Schema Registry is a separate component that producers and consumers must talk to for serialization and compatibility. You need to set up, secure, monitor, and upgrade it alongside Kafka brokers, which adds operational overhead and complexity.
How does Condense simplify Kafka + Schema Registry?
Condense bundles Kafka-native streaming with integrated Schema Registry (Karapace-compatible), ACLs, and observability in one managed platform. You deploy from cloud marketplaces and get schema governance without running a separate registry node.
What subject naming strategies does Condense support?
Condense supports the standard Confluent naming strategies: TopicNameStrategy, RecordNameStrategy, and TopicRecordNameStrategy. These determine how subjects are named and how schema versions are managed for keys and values.
Why is Protobuf better than JSON for high-throughput Kafka pipelines? Protobuf uses compiled classes and binary serialization, which is faster and smaller than JSON's text-based format. This reduces deserialization overhead and consumer lag, especially with nested structures or high data rates.
How does Condense help teams ship Protobuf-based streaming pipelines faster?
Condense provides prebuilt connectors, a Git-enabled IDE, and no-code/low-code builders for stream logic, so you can focus on Protobuf message design instead of cluster ops. You get Kafka-native performance with reduced setup time and operational burden.

Kafka Streams 101: A Developer’s Guide to Real-Time Application Logic

Sachin Kamath — Tue, 16 Jun 2026 10:06:59 +0000

TL;DR

Kafka Streams enables real-time stream processing inside applications using local state backed by Kafka logs. However, deploying and managing multiple Kafka Streams microservices at scale is complex, requiring custom CI/CD, state recovery, and observability tooling.

Condense simplifies this by providing a fully managed, unified streaming platform inside your cloud (BYOC). It integrates Kafka Streams with built-in IDE, Git versioning, prebuilt domain logic, and native observability, eliminating operational overhead while accelerating development and scaling real-time apps reliably.

Introduction

Apache Kafka has long been a cornerstone of modern data infrastructure, providing a distributed, fault-tolerant backbone for event ingestion at scale. But ingestion is only half the equation. Business value lies in what happens after events are received, how raw data is filtered, joined, aggregated, enriched, and ultimately transformed into decisions.

This is where Kafka Streams comes in. As a native stream processing library built on Kafka itself, Kafka Streams enables developers to write real-time logic using a simple yet powerful programming model. This blog walks through the foundations of Kafka Streams, explores how it powers real-world applications, and examines the architectural implications for engineering teams. At the end, we’ll also see how modern platforms are simplifying this journey further by eliminating unnecessary complexity from the development lifecycle.

Understanding the Kafka Streams Programming Model

Kafka Streams is fundamentally a Java library that allows developers to treat Kafka topics not just as message queues, but as unbounded data tables or continuously updating datasets. Its core abstractions include:

KStream: A continuous stream of records. Think of this as the raw event log.
KTable: A changelog stream that represents the latest value for each key, essentially a materialized view.
GlobalKTable: A read-only table replicated on each instance, often used for joining reference data.

Stream logic is expressed using the Streams DSL or the Processor API. Most applications use the DSL to define transformations like map(), filter(), join(), and aggregate(), while the Processor API gives lower-level control over state and custom operators.

Stateful Processing and Local Stores

One of Kafka Streams’ defining features is its local state management. Stateful operations, like groupByKey().windowedBy().aggregate(), require storing intermediate state. Instead of centralizing this in a database, Kafka Streams maintains RocksDB-based state stores on the local disk of each processing instance.

This state is backed by a changelog topic in Kafka. If a failure occurs, the processor recovers by replaying the changelog. This design allows for scalable, distributed stream processing, but it also introduces critical operational requirements:

Persistent disk access for RocksDB.
Monitoring of state restoration and checkpointing.
Partitioned processing tied to Kafka topic partitioning.

Real-World Application Deployment: Microservices and Beyond

In most enterprises, Kafka Streams applications are deployed as microservices. Each stream processing unit, fraud detection, ETA computation, SLA tracking is packaged as a Spring Boot or Quarkus application, then deployed into Kubernetes or another container orchestrator.

This approach introduces certain responsibilities per service:

Maintain a complete lifecycle (build, deploy, monitor, patch).
Handle schema compatibility between topics and application code.
Implement backpressure handling, logging, and metrics.
Define partitioning logic that matches Kafka topic partitioning.

This model is manageable at small scale, but quickly becomes burdensome as the number of real-time applications grows. Teams often end up building:

Custom CI/CD tooling for streaming services.
State migration routines for schema evolution.
Monitoring layers to track per-operator lag, backpressure, and failures.
Homegrown governance to version and deploy transforms safely.

The reality is that while Kafka Streams simplifies the programming model, it does not eliminate operational complexity. Most Kafka Streams microservices still need to be treated like full-fledged backend services, each with infrastructure, observability, and deployment overhead.

Managing Failures and Stateful Recovery

Stateful stream processing introduces unique challenges not seen in stateless services:

Processor crashes require replaying changelogs to restore state.
Version upgrades must avoid state corruption or key mismatch.
Hot deployments risk double processing or record duplication if not orchestrated carefully.
Event time processing with out-of-order data requires complex watermarking and windowing strategies.

Kafka Streams supports exactly-once semantics (EOS) with idempotent producers and transactional writes, but this adds additional configuration burden and requires careful coordination between input/output topics and processing guarantees.

In practice, engineering teams often need to build custom scaffolding to make these patterns reliable, transform state inspection, window replay, timestamp alignment, and state migration versioning.

Why Observability Remains an Under-Addressed Challenge

While Kafka itself provides metrics on broker health and topic lag, Kafka Streams applications demand pipeline-aware observability:

Is a specific stream join introducing backpressure?
Are certain partitions processing slower due to skewed keys?
Is the state store nearing disk exhaustion?
Which application version is currently deployed and processing which partitions?

These questions often require setting up Prometheus exporters, embedding Micrometer, and integrating with tools like Grafana, Jaeger, or OpenTelemetry. In many cases, visibility across multi-stage pipelines (e.g., “raw event → session builder → score assigner → alert emitter”) is fragmented and hard to debug during incident response.

CI/CD and Versioned Transform Pipelines

Deploying changes to streaming logic requires particular discipline:

Stateful operators must be deployed carefully to avoid dropping or reprocessing records.
Version control is critical, not just for source code, but for schemas and processing topology.
Teams must implement rollback strategies for failed deployments without corrupting stream state.
Developers often struggle to test stream topologies locally, especially when logic is embedded deep inside a containerized microservice.

While Kafka Streams supports topology versioning and testing via TopologyTestDriver, there’s no built-in support for seamless, multi-version CI/CD integration.

What This Means for Real-Time Engineering Teams

By now, the picture is clear: Kafka Streams provides the primitives, but not the platform. To make real-time work in production, teams must shoulder:

Lifecycle management of dozens of services.
CI/CD pipelines that are stream-aware.
Governance across schemas, state, and partitioning.
Ops playbooks for fault tolerance, state recovery, and lag monitoring.
A documentation trail so that new engineers can maintain existing stream logic safely.

This fragmentation can be a major blocker, not because the underlying code is difficult, but because the integration burden scales with every new pipeline.

How Platforms Like Condense Change the Equation

Modern real-time platforms are increasingly collapsing this complexity.

Condense, for example, retains Kafka Streams’ power while eliminating the need for separate microservices per application. Instead of building, deploying, and observing independent logic units:

Developers write Kafka Streams-style logic inside an integrated IDE, with support for no-code and low-code operators (merge, delay, alert, window).
All transforms are version-controlled and Git-integrated, enabling safe rollouts, rollbacks, and collaborative development.
The platform handles orchestration, state recovery, partition scaling, and observability as first-class features.
Prebuilt domain-specific operators (e.g., CAN decoder, trip builder, geofence engine) reduce redundant engineering effort.
All Kafka brokers and processors run inside the customer’s cloud account via BYOC, ensuring data sovereignty without operational burden.

By removing the need to wrap each stream job in its own microservice, Condense makes it feasible to scale from 5 to 50+ real-time workflows without growing operational debt linearly.

Closing Thoughts

Kafka Streams remains a powerful tool in the real-time developer’s toolkit. But making it work at scale involves far more than just calling stream.map().filter().join(), it demands operational rigor, architectural forethought, and careful coordination across the development lifecycle.

For organizations moving from raw events to real-time decisions, the choice is not just about code, it’s about platform strategy.

As real-time becomes core infrastructure, platforms like Condense that provide an integrated, streaming-native runtime, from ingestion to logic to deployment are proving to be not just convenient, but essential.

Challenges in Updating Managed Kafka Platforms to Kafka 4.3.0

Sachin Kamath — Wed, 03 Jun 2026 17:28:05 +0000

TL;DR

Updating managed Kafka platforms to Kafka 4.3.0 is not a simple version upgrade. The removal of ZooKeeper, KRaft migration requirements, infrastructure validation, compatibility testing, recovery optimization, and operational changes introduces significant engineering effort for managed Kafka providers. Condense simplifies this complexity by handling Kafka upgrades, infrastructure management, monitoring, scaling, and operational workflows centrally.

Apache Kafka 4.3.0 introduces major architectural and operational changes across KRaft, storage recovery, consumer coordination, security, and observability. While these improvements strengthen Kafka for production-scale environments, upgrading managed Kafka platforms to Kafka 4.3.0 requires significant engineering effort.

For managed Kafka providers, upgrades are not limited to changing broker versions. Every infrastructure layer, operational workflow, monitoring pipeline, client compatibility model, and recovery mechanism must be validated carefully before production rollout.

The move to KRaft-only architecture in Kafka 4.3.0 increases this complexity further because ZooKeeper support is completely removed.

Managed Kafka providers must ensure:

Cluster stability
Data safety
Upgrade compatibility
Operational continuity
Multi-tenant reliability
Security consistency
Zero or minimal downtime

These requirements make Kafka version upgrades operationally intensive.

KRaft Migration Complexity

One of the biggest changes in Kafka 4.3.0 is the complete removal of ZooKeeper support. Kafka clusters now operate entirely on KRaft mode. For managed Kafka providers, this is not simply a configuration update.

Major Efforts Involved:

Migrating existing ZooKeeper-based clusters
Validating metadata consistency
Updating controller management workflows
Reworking infrastructure automation
Rebuilding deployment pipelines
Updating monitoring systems for KRaft

Providers must validate that KRaft behaves consistently across:

Small clusters
Large multi-tenant environments
High-throughput workloads
Disaster recovery scenarios

Migration errors at the metadata layer can directly impact cluster availability and operational stability.

Infrastructure Validation and Compatibility Testing

Managed Kafka environments support multiple customer workloads with different:

Kafka clients
Consumer patterns
Security configurations
Connector ecosystems
Streaming applications

Upgrading Kafka versions requires extensive compatibility validation.

Major Efforts Involved

Client compatibility testing
Connector validation
Schema registry testing
Security integration validation
Consumer group behavior testing
Kafka Streams compatibility verification

Providers cannot assume every customer application will behave identically after upgrades

Even small protocol-level changes can impact:

Rebalance behavior
Throughput patterns
Latency
Connector operations
Stream processing workflows

This makes pre-production validation extremely important.

Operational Risk During Upgrades

Managed Kafka providers operate production-critical environments where downtime risks must remain minimal.

Kafka upgrades require careful operational planning.

Major Efforts Involved

Rolling upgrade orchestration
Replica synchronization validation
Partition reassignment handling
Traffic balancing
Recovery workflow testing
Rollback strategy preparation

Upgrades become even more sensitive in:

High-throughput environments
Multi-region clusters
Tiered storage deployments
Mission-critical systems

Any instability during upgrades can impact production data pipelines directly.

Tiered Storage Recovery Validation

Kafka 4.3.0 introduces improvements for tiered storage replica recovery.

While these improvements provide operational advantages, managed Kafka providers must validate recovery behavior thoroughly before enabling them at scale.

Major Efforts Involved

Recovery testing across large datasets
Remote storage validation
Replica synchronization benchmarking
Failure scenario simulation
Recovery performance tuning

Tiered storage environments usually operate with massive historical data volumes. Recovery inefficiencies can increase operational overhead significantly if not validated properly.

Consumer Group Coordination Changes

Kafka 4.3.0 improves consumer group assignment handling through assignment batching and configurable assignment intervals.

For managed Kafka providers, consumer group behavior is extremely sensitive because customers operate different scaling models and workload patterns.

Major Efforts Involved

Rebalance behavior validation
Autoscaling compatibility testing
Coordinator load benchmarking
Consumer lag analysis
Throughput stability testing

Even improvements intended to optimize coordination must be validated carefully across different workload patterns before broad rollout.

Monitoring and Observability Updates

Kafka 4.3.0 introduces new operational metrics and observability improvements, including retention headroom metrics.

Managed Kafka platforms usually maintain centralized observability systems for:

Metrics
Alerts
Dashboards
Capacity planning
Operational analytics

Every Kafka release requires updates to these monitoring systems.

Major Efforts Involved

Updating monitoring pipelines
Creating new dashboards
Alert validation
Storage visibility integration
Operational analytics updates

Without proper monitoring updates, new Kafka capabilities cannot be utilized effectively.

Security and IAM Integration Validation

Kafka 4.3.0 introduces OAuth client assertion support for enterprise authentication workflows.

Managed Kafka providers supporting enterprise customers must validate:

IAM integrations
Token-based authentication flows
Access control behavior
Security policy compatibility
Authentication performance

Major Efforts Involved

Identity provider testing
Security workflow validation
Multi-tenant access verification
Compliance testing
Zero-trust architecture validation

Security upgrades require careful validation because authentication inconsistencies directly affect customer workloads.

Upgrade Coordination Across Multi-Tenant Environments

Managed Kafka platforms usually host multiple customer environments on shared infrastructure layers.

This creates additional operational complexity during upgrades.

Major Efforts Involved

Tenant-aware rollout planning
Cluster isolation validation
Workload impact analysis
Upgrade scheduling coordination
SLA management

Providers must ensure upgrades do not create cascading impact across customer environments.

This becomes significantly more complex at scale.

Engineering Effort Behind Kafka Upgrades

From the outside, Kafka upgrades may appear straightforward.

Internally, managed Kafka providers must coordinate across:

Platform engineering teams
Infrastructure teams
SRE teams
Security teams
Support teams
Customer operations teams

Kafka Upgrades Involve:

Infrastructure automation updates
Recovery validation
Observability changes
Operational testing
Security integration updates
Documentation and support readiness

The engineering effort behind production-grade Kafka upgrades is substantial.

How Condense Simplifies Kafka Upgrades

At Condense, Kafka infrastructure management, upgrades, scaling, observability, and operational workflows are centrally managed as part of the platform.

Condense simplifies Kafka version adoption by handling:

Kafka cluster management
Upgrade orchestration
Infrastructure automation
Monitoring and observability
Security integration
Scaling workflows
Recovery operations
Operational maintenance

This allows organizations to adopt newer Kafka versions such as Kafka 4.3.0 without managing the operational complexity internally.

As Kafka evolves with architectural changes like KRaft, tiered storage optimization, and operational improvements, Condense ensures these capabilities are integrated and operationalized efficiently within production environments.

Frequently Asked Questions (FAQs)

Why is KRaft migration a major challenge?
KRaft completely removes ZooKeeper dependency, requiring metadata migration, infrastructure changes, monitoring updates, and operational workflow redesign.
Does Kafka 4.3.0 improve operational efficiency?
Yes. Kafka 4.3.0 improves recovery behavior, consumer coordination, observability, security integration, and infrastructure simplification through KRaft architecture.
How does Condense simplify Kafka 4.3.0 adoption?
Condense is already built on KRaft-based Kafka architecture, enabling organizations to adopt Kafka 4.3.0 capabilities without handling underlying infrastructure transitions, upgrade orchestration, monitoring, scaling, or operational maintenance internally.
Can Condense help organizations adopt Kafka 4.3.0 faster?
Yes. Since Condense already operates on modern Kafka architecture principles including KRaft, organizations can leverage Kafka 4.3.0 operational improvements such as simplified metadata management, recovery optimizations, and enhanced observability without redesigning their Kafka infrastructure stack.
Why is KRaft adoption easier with Condense?
KRaft migration usually requires infrastructure redesign, operational workflow changes, metadata management updates, and monitoring modifications. With Condense, these infrastructure complexities are abstracted through a managed streaming platform already aligned with modern Kafka operational architecture.
What operational improvements does Kafka 4.3.0 bring to real-time streaming platforms?
Kafka 4.3.0 improves operational efficiency through faster tiered storage recovery, broker cordoning, retention headroom metrics, improved consumer group coordination, and better enterprise security integration. These improvements help organizations run large-scale streaming environments more efficiently.
What should organizations consider before upgrading to Kafka 4.3.0?
Organizations should evaluate client compatibility, recovery workflows, monitoring updates, consumer group behavior, connector validation, and operational readiness before upgrading to Kafka 4.3.0. Production rollout should always include staging validation and rollback planning.
Does Kafka 4.3.0 require infrastructure changes?
Yes. Kafka 4.3.0 introduces architectural and operational changes that may require updates to deployment workflows, monitoring systems, maintenance processes, and infrastructure automation depending on the existing Kafka environment.
How does Kafka 4.3.0 improve Kafka maintenance workflows?
Kafka 4.3.0 introduces broker and log directory cordoning, allowing operators to stop new partition assignments during maintenance operations. This simplifies hardware replacement, broker migration, and infrastructure maintenance workflows.
Is Kafka 4.3.0 better for large-scale streaming workloads?
Yes. Kafka 4.3.0 improves recovery efficiency, consumer coordination, observability, and operational stability, making it more suitable for large-scale real-time streaming environments with high throughput and large storage volumes.

Real-Time Data Streaming vs Batch Data ETL: Why Timing Matters

Sachin Kamath — Tue, 02 Jun 2026 02:26:41 +0000

TL;DR

Batch ETL moves and processes data on a schedule, delivering insights with built-in latency, ideal for historical analysis and compliance, but ineffective for urgent, real-time business actions. Real-Time Streaming pipelines process each event instantly, enabling on-the-fly fraud detection, predictive maintenance, and hyper-personalized engagement. Timing isn’t just a throughput metric; it determines whether data delivers competitive value or is just hindsight. Condense makes real-time streaming practical and production-ready, letting enterprises turn events into actions within their own cloud, while traditional batch workflows remain valuable for long-term reporting and analytics.

For decades, batch ETL defined how enterprises integrated and analyzed data. Jobs were scheduled, data was extracted from sources, transformed into a unified schema, and loaded into warehouses or lakes for reporting.

This was enough when businesses primarily asked: what happened yesterday?

But the operational environment has changed. Industries now compete on the ability to respond instantly whether blocking fraud at the moment of authorization, detecting anomalies in connected fleets, or personalizing customer engagement as interactions unfold. In this landscape, Real-Time Data Streaming and modern streaming pipelines are not optimizations. They are requirements.

This blog examines the technical differences between batch ETL and Real-Time streaming, explains why timing is more than a performance metric, and explores how streaming pipelines are reshaping enterprise architectures.

Batch ETL: Strengths and Boundaries

Batch ETL (Extract, Transform, Load) pipelines move data in discrete intervals. They typically operate as follows:

Extract
Pull records from transactional systems, APIs, or files.
Transform
Apply schema normalization, deduplication, or business logic in staging.
Load
Insert processed batches into a target system (warehouse or data lake).

Technical strengths

Throughput: Bulk processing of millions of records is efficient on modern compute clusters.
Determinism: Fixed jobs are easier to validate and audit, making them suitable for compliance.
Maturity: Tooling (Informatica, Talend, dbt, Airflow) is well established and battle-tested.

Limitations inherent to the design

Latency: The time between data generation and availability is at least the batch interval minutes, hours, or days.
Operational blind spots: Events between runs remain invisible. Failures may not be discovered until the next batch completes.
Rigid scheduling: Workflows are brittle under changing workloads. Rescheduling impacts dependencies downstream.
Resource spikes: Large jobs create uneven load, with clusters often over provisioned to handle peak windows.

Batch ETL is indispensable for historical analysis and compliance reporting, but unsuitable when insights must drive immediate operational action.

Real-Time Data Streaming: A Continuous Model

Real-Time Data Streaming inverts this paradigm. Instead of moving data in scheduled intervals, every event is treated as a discrete, time ordered signal that can be processed immediately. Kafka and similar log based systems provide the backbone for this architecture.

Core mechanics of streaming pipelines:

Immutable logs: Events are appended to partitions, guaranteeing order and durability.
Replayability: Consumers can reprocess events from any offset, enabling recovery and backfills.
Stateful stream processing: Operators maintain state across windows, joins, and aggregations (e.g., “total purchases by customer in the last 5 minutes”).
Continuous enrichment: Streams are augmented with contextual data (e.g., geolocation, device metadata) in motion.
Low-latency sinks: Events are delivered to APIs, dashboards, or control systems within milliseconds to seconds.

This model does not merely accelerate batch. It enables workflows that batch cannot support because the business outcome depends on acting while the event is still unfolding.

Why Timing Is Strategic

Timing is not a secondary concern; it directly determines the value of data.

Fraud detection: A fraudulent card transaction must be flagged before the authorization completes. A nightly batch report identifies fraud after the funds are gone.
Predictive maintenance: An abnormal vibration detected mid route can prevent breakdown. Batch ETL will surface it only after the vehicle is already sidelined.
Customer personalization: Recommending a product while a customer is browsing drives conversion. A next day email is often irrelevant.
Logistics visibility: A delayed shipment must trigger re routing in the moment. Reporting it after delivery deadlines have passed is operationally useless.
Cybersecurity: Intrusion attempts must be analyzed in flight to prevent compromise. Batch ETL provides forensic evidence, not active defense.

In each case, the same data is processed. The difference is timing. Batch delivers hindsight. Streaming delivers foresight.

Demand for Streaming Pipelines

Enterprises are increasingly building streaming pipelines because the nature of their industries leaves no tolerance for latency.

Financial services: Real-Time AML checks, fraud detection, and instant payment processing are both competitive and regulatory mandates.
Mobility and automotive: Vehicles generate telemetry that must be analyzed continuously for safety and efficiency.
Telecom and IoT: Billions of device signals require filtering, aggregation, and anomaly detection at scale.
Retail and digital platforms: Context aware personalization drives customer engagement. Delayed data undermines the business model.

The demand side is clear: data is only valuable if it can be acted upon within the time window that matters.

Coexistence: Batch and Streaming Together

This is not a zero sum choice. Batch ETL and streaming coexist in most enterprises:

Batch ETL: Best for historical analytics, compliance archiving, financial reporting, and periodic aggregations.
Real-Time Data Streaming: Best for operational intelligence, anomaly detection, personalization, SLA monitoring, and IoT telemetry.

The shift is not about replacement, but about recognizing that streaming pipelines increasingly occupy the critical front line of enterprise decision making.

Why Real Time Data Streaming Platforms like Condense Matters Here

This is where Condense makes a difference. It is a Kafka Native platform designed to deliver production-ready streaming pipelines inside the enterprise’s own cloud environment (BYOC). With Condense, organizations don’t just get Managed Kafka brokers they get a complete runtime that manages ingestion, stream processing, stateful recovery, observability, and domain-specific transforms.

That means enterprises can move from raw events to actionable insights in minutes, without taking on the operational weight of building pipelines from scratch.

Batch ETL will remain valuable, but the competitive edge lies in Real-Time. Condense enables enterprises to capture that edge by making Real-Time Data Streaming both practical and production-ready.

Frequently Asked Questions (FAQs)

What is the main difference between batch ETL and Real-Time Data Streaming?
Batch ETL processes data in scheduled intervals, while Real-Time Data Streaming processes each event as it happens.
Why are streaming pipelines faster than batch ETL?
Streaming pipelines handle events continuously with low latency, unlike batch jobs that wait for scheduled runs.
When should enterprises use batch ETL instead of streaming?
Batch ETL is best for historical reporting, compliance archives, and workloads where timing is not critical.
Why is timing important in Real-Time Data Streaming?
Timing ensures events drive immediate actions, such as fraud blocking, predictive maintenance, or real-time personalization.
Can batch ETL and streaming pipelines coexist?
Yes, most enterprises use streaming pipelines for live operations and batch ETL for long-term analytics.
What industries benefit most from Real-Time Data Streaming?
Finance, mobility, logistics, IoT, and retail depend on Real-Time Data Streaming for mission-critical decisions.
How does Condense improve the adoption of streaming pipelines?
Condense is a Kafka Native platform that lets enterprises build production-ready streaming pipelines in minutes inside their own cloud.

AIDL deep dive post does not exist! Part - I

Prasanna — Sat, 18 Jul 2020 06:33:30 +0000

Introduction

Hello there! I am going to talk about Android's parliamentary conversations. What do I mean by that? I mean, the IPC(Inter Process Communication) that is happening inside Android system. We are going to use AIDL for the so called "conversation". I will explain better and deeper in this article.

We all know it!

Why is this a brand new post? Because, it's not about the IPC of components inside one Android application. It's about the communication of two different applications. Most people agree to disagree that it's pretty easy to understand AIDLs. But, there is saying(of mine), "If you understand interfaces in OOP, you shall understand AIDL!". Well, Let's see what that saying really means, in detail.

The AIDL unbolted.

Android Interface Definition Language(AIDL), is what you every time you Google AIDL. This doesn't explain what it really is. Let me explain! AIDL is an Android implementation to achieve Inter Process Communication(IPC) in between Android components. Diving deep, there is a program called aidl, which compiles the AIDL source code and generates client(Proxy) and server(Stub) Java interfaces.

Initial building blocks.

Let us start with the basics. To create an AIDL, we need to open an Android Project and click File -> New -> AIDL -> AIDL File. Write your first AIDL file.

interface IMyAidlInterface {
    String getMessage();
}

Then, do not forget to Build -> Rebuild. The IMyAidlInterface class will be generated. Create a class which extends the IMyAidlInterface.Stub. Please remember to override getMessage() method in this class.

public class MyImplementor extends IMyAidlInterface.Stub {
    @Override
    public String getMessage() {
        return "Hello from AIDL Stub!";
    }
}

Once we have extended the Stub class and implemented(overridden) the method, we must create a Service which is going to expose the APIs to the applications which need bind the AIDL functionalities.

public class MyService extends Service {
    @Nullable
    @Override
    public IBinder onBind(Intent intent) {
        return new MyImplementor();
    }
}

Also, add the <service> tag in AndroidManifest.xml inside <application> tag.

<service
    android:name=".MyService"
    android:enabled="true"
    android:exported="true"
    android:process=":remote">
    <intent-filter>
        <action android:name="MyService" />
    </intent-filter>
</service>

Now that we have implemented the Service part, we should go ahead and create another Android project for the client part. Please note that the AIDLs are very case-sensitive. Unfortunately, Android Studio is not mature enough to detect the syntax errors in an AIDL file. Any changes in an AIDL file would need a rebuild of the project.

The Real River to cross!

We are done with the server(Service) part. Now, we should create another application which will be our client part. So, create a new project in Android Studio. After creating the project, we have to do one of the most important things.

In Android Studio, select the Project view in the left top menu. Inside app/src/main/, create a folder called aidl.

After this, go to the AIDL service application, and copy the contents of aidl folder.

Switch back to the client application and paste the contents into aidl folder. Build -> Rebuild. The project should build without any errors.

Go to MainActivity.java and create a ServiceConnection object and IMyAidlInterface instance.

private IMyAidlInterface iMyAidlInterface;
private final ServiceConnection mServiceConnection =
    new ServiceConnection() {
        @Override
        public void onServiceConnected(
                    ComponentName name, IBinder service) {

            iMyAidlInterface =
                    IMyAidlInterface.Stub.asInterface(service);

            Log.d(TAG, "Service Connected.");
            Toast.makeText(MainActivity.this, "Service Connected.", Toast.LENGTH_SHORT).show();
        }

        @Override
        public void onServiceDisconnected(ComponentName name) {
            iMyAidlInterface = null;
            Toast.makeText(MainActivity.this, "Service Disconnected.", Toast.LENGTH_SHORT).show();
        }
    };

In onCreate() of MainActivity.java, bind the service. Note that we need the action and the package name to successfully bind to the service.

protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    Intent intent = new Intent();
    intent.setPackage("com.zeliot.aidldemo");
    intent.setAction("MyService");
    bindService(intent, mServiceConnection, BIND_AUTO_CREATE);
}

Now, create a button and while clicking the button, call iMyAidlInterface.getMessage() to receive the String from the AIDL service.

We are done with client and server parts. As of now, the Service is connected from another application. We can do any type of operations using the same. We can have data trasnfer,aynchronous calls, callbacks. More detailed implementation will be released in the second part of this series.

AIDL source code