The Confluent Data Streaming World Tour 2026 in Mumbai highlighted a significant shift in how we look at and process the data. The core message was simple: if you want AI to work in production, real-time data is no longer just a "nice-to-have".
The event showed a transition from old-school static pipelines to "data in motion" in order to build systems that are truly scalable.
You can consider this post as my neutral commentary on the event. (Any of my own thoughts/reflections or opinions are written in parenthesis and formatted italics, like this one)
(While there were many sessions, the following three stood out for their architectural insights.)
Opening Keynote: Greg Taylor on "Why real-time matters?"
The event started with a keynote by Greg Taylor with a sharp analogy: "Would you cross the street based on snapshots of where the cars were yesterday?" This set the tone for the whole discussion throughout the day around why real-time data streaming and processing is essential.
Taylor spoke about the shift from Business Intelligence (BI) to Artificial Intelligence (AI):
- Business Intelligence: The speaker defined it as a software built for humans. Data usually moves in batches, and actions happen periodically. Taylor pointed out a classic corporate struggle: often, by the time data is cleaned, put onto a dashboard, and reviewed by executives to make a call, the ground reality has already changed. Making big moves based on "stale" information is a huge risk that many companies are still carrying.
- Artificial Intelligence: The speaker defined it as a software being used by other software. In this world, we don't have the luxury of "human-in-the-loop" delays . Data has to move in real-time and actions must be continuous for the system to actually stay relevant and performant.
(Although, I do believe that human should be in the loop as it ensures correctness)
To help businesses make this jump, Confluent highlighted its "governance triad" to bring together the best industry standards:
- Apache Kafka: The go-to standard for operational streaming.
- Apache Flink: The gold standard for stream processing.
- Apache Iceberg and Delta Lake: The leading table formats for unified analytics.
Aerospike: Infrastructure for the Agentic AI Era
Shekhar Suman, who is a Solutions Architect at Aerospike, gave a solid session on why traditional data layers often choke under the weight of modern AI. He explained how a "Real-time database for AI" solves these bottlenecks.
Moving from linear to agentic workflows
A major takeaway was the shift from "Traditional inference" where you have simple and known steps to Agentic systems. In these new setups, one single user interaction can trigger over 100 different dependent operations, creating what he called "unbounded decision chains."
Predictability: The Metric that actually counts
Suman argued that when you are in production, "predictability is the ultimate performance metric." He showed how Aerospike keeps performance flat even when you’re pushing hardware to its absolute limits. This ensures:
- Shifting Patterns: The system handles changes in read/write ratios without needing a team to manually retune things.
- Linear Scaling: When you add more capacity, you get the results you expect. This lets the teams ship early and evolve without the fear of crashing production.
Handling "Production entropy"
The talk touched upon the "Gap between design-time order and production entropy." Systems always look perfect on a whiteboard (Precision geometry), but production brings in system decay and volatility. Aerospike’s architecture is built to handle this "Entropy" by keeping real-time and batch workflows in one place.
Trust, Compliance, and Checkpointing
One very cool technical bit was using LangGraph with Aerospike for checkpointing AI decisions. By saving the state at every single step (Trigger -> Tool Call -> Reasoning -> Decision), Aerospike gives you:
- Full Auditability: Regulators can literally replay the AI's logic step-by-step.
- No Black Box: You get a clear "Chain of Thought" for every decision made.
- Resilience: If any step fails, the system doesn't just crash; it can recover and try again from the last checkpoint.
Practical use: Real-time fraud detection
Suman compared Real-time fraud detection with traditional fraud detection, to which he called a "rear-view mirror" approach that can take days to catch a fraudulent activity, but with the use of Aerospike and by bringing real-time data, feature engineering, and AI agents into one layer, fraud can be spotted and blocked the second it happens.
ClickHouse: A New Way to Look at Analytics
Alexey Milovidov shared some great insights on how ClickHouse and Confluent are powering the "Agentic AI" era. Since autonomous AI needs super-fast data, ClickHouse is becoming the industry standard for high-performance analytical databases.
Growth and Market Position
ClickHouse has seen incredible growth, picking up over 3,000 customers in less than three years. Their clients include everyone from tech giants like Microsoft, Meta, and Netflix to big traditional players like Deutsche Bank. After a $400M Series D earlier in 2026, the company is now valued at $15B.
Performance Benchmarks
The technical side was impressive. ClickHouse consistently beats out rivals like Snowflake, Redshift, and ElasticSearch:
- Ingestion Speed: ClickHouse loads data in 140s, while Redshift takes 1,829s.
- Query Latency: For big analytical queries, ClickHouse finishes in 2.57s, miles ahead of Snowflake at 12.33s.
- Storage Efficiency: Using top-tier compression, it only needs 9.27 GiB for a dataset that takes 99.18 GiB in Postgres.
The Scaling Paradox
A big theme was the "Small to Big" paradox. Most databases either work well on small data but choke on billions of rows (like MySQL), or they are "Big Data" tools that take forever to even start up (like Spark). ClickHouse, written in C++, uses Vectorized Query Execution and SIMD instructions to work just as well on a single laptop as it does on a massive cloud cluster.
Why Hadoop is Fading
The session noted that people are moving away from Hadoop. Unlike the "zoo" of services Hadoop needs (Zookeeper, NameNodes, etc.), ClickHouse is way easier to deploy, gives results in milliseconds, and is much more efficient with hardware.
One of the personal highlights for me was getting a chance to chat with Alexey Milovidov (Co-founder and CTO of ClickHouse) backstage. Which was a fun time.
Porter’s scaling and architectural evolution
Ambuj Singh, Head of Engineering at Porter, gave a very honest account of moving from a monolithic setup to microservices. Porter is a Digital Goods Transport Agency (GTA) focused on solving intra-city logistics for MSMEs, and their scale is massive:
- Global reach: 41 cities worldwide.
- MSME base: Over 30 lakh customers.
- Workforce: 6 lakh active driver-partners every month.
- Model: An "asset-light" approach where sustainability is a core value.
The "Distributed Monolith" Mistake
Ambuj shared a common pitfall they faced: the Distributed monolith. They split their code into separate services (Order system, pricing, allocation, etc.), but kept them "tightly coupled."
In this setup, if the "pricing" service went down, the "order system" crashed because it was waiting for a direct response. They had red arrows (dependencies) crisscrossing everywhere. They realized that just putting code in different folders or servers doesn't mean you have microservices if they still depend on each other for every single task.
The Solution: Event-Driven Architecture (EDA)
To fix this, Porter shifted to an Event-Driven Architecture powered by Kafka.
- The Central Hub: Kafka acts as the "data streaming" pipe in the middle.
- Loose Coupling: Now, services only talk to Kafka. If the "order system" does something, it just sends an event like "order created."
- Independence: The "analytics" or "notifications" services just "listen" and pick up that info when they are ready. If one crashes, the rest of the business keeps running.
Leadership perspectives from the industry
- BFSI Sector: Banking, Financial Services, and Insurance Sector Leaders in banking are doubling down on trust and speed. They are looking at these as the pillars for modern financial systems, especially for data protection.
- GCC Evolution: Global Capability Centres are no longer just "back offices." They are becoming strategic tech hubs, using AI to drive real business value.
- GiniMinds: They focused on why a solid data foundation is a must for AI. Without proper structure and governance, AI just isn't sustainable in production.
- Aerospike: Showcased how high-performance systems directly impact the user experience by keeping things fast and reliable.
- YuVerse: Mathangi Sri Ramachandran pointed out that the real hurdle for GenAI isn't building the model, it's feeding it continuous, reliable data.
- Confluent Keynote: Andrew Sellers made it clear: static data is a bottleneck for AI. Real-time data is what lets it scale in the real world.
- Meesho: Shubham Sharma shared how they build for "India scale," where acting on data instantly is the only way to stay efficient.
My Personal Takeaways?
All in all, the event was a fantastic learning experience. It’s one thing to read about these architectures on GitHub, but quite another to see them working at an "India scale" for companies like Meesho and Porter. It was also a brilliant networking opportunity chatting with other engineers and architects in Mumbai really puts into perspective how everyone is tackling the same scaling headaches.
Honestly, it feels like we’ve finally hit the point where the "big data tax" is dead. We no longer have to choose between a "small and fast" database or a "big and slow" warehouse. With architectures like ClickHouse, Aerospike, and Confluent, systems can stay fast and predictable whether you’re looking at a thousand rows or a trillion.
However, I want to be clear, while this sounds like a massive shift, it isn't necessarily for everyone. The need for this level of real-time infrastructure differs heavily from industry to industry. I don't believe we are at a time where every single business needs to rush to these tools immediately. Instead, we are at a point where adoption is the easiest it has ever been. If your business case actually demands it, the barrier to entry has finally dropped.
The biggest thing I'm taking home is that "real-time" isn't just about speed anymore. it’s about relevance. Moving to an Event-Driven Architecture isn't just a technical migration; it's a completely different way of thinking about how a business breathes and reacts.
It’s an exciting time to be building in this space. The tools are finally catching up to our ambitions.
Please feel free to share any thoughts and questions in the comment section below.
Top comments (0)