We were at Current 2026 in London two weeks ago. Keynotes about agentic AI and streaming agents but conferences conversations were about something else entirely.
1. The IBM-Confluent acquisition?
Six months after IBM acquired Confluent for $11B, I expected hot takes. What I got was mostly 'indifference'?
A data engineering lead at a major US bank called himself a "doomer", Kafka and Cassandra are his two favorite open-source projects, and both now sit under IBM. A staff engineer at a streaming vendor was more optimistic, speculating about Confluent's AI team merging with Watson.
The majority were more like "I try to concentrate on the tech side." "I don't know enough to comment." Moving on.
The acquisition was in the air, but practitioners had more pressing problems to talk about.
2. Kafka costs are the third certainty (after death and taxes)
Every cost conversation we had mapped to the same things:
- Partition overprovisioning: inherited from not-knowing or templates tuned for a different workload
- Retention mismatches: defaults that outlive the use case they were set for
- Cluster proliferation: one-off clusters that never got consolidated
- Orphan and duplicate topics: experiments or dead projectsthat never got cleaned up
- Inefficient client patterns: batching, compression, and serialization left on defaults + JVM vs librdkafka differences
- Static capacity: paying for peak when the load is 10% of that most of the time
Some examples of scale where they had these problems:
- six years of unmanaged topics inherited during a Confluent Cloud migration
- one team had twenty MSK clusters running multi-regions
- MSK, Confluent, and Redpanda all running in parallel, looking for a single control plane as totally separated.
A data engineer at a German energy consultancy put it well:
"Kafka resource costs should not be taken for granted. Maybe I need sub-second latency, or maybe daily is enough. These are considerations you must make up front, before it's too late."
The solution is to change how to use Kafka: guardrails at topic creation, retention defaults that reflect reality, and ownership at the source. We typically see 25-40% of infrastructure costs come back this way.
We wrote up the full analysis in Your Platform Team Can't Fix Kafka Costs Alone.
3. Everyone wants self-service. Nobody has the operating model for it.
Self-service provisioning is a common topic now. Developers want to create topics and request permissions without filing a Jira ticket (many still do). Platform teams want to provide it to not be a bottleneck. The how is the question. Home-made is time consuming considering all variations, and using a vendor, it has to be flexible to adopt.
A senior engineer at a Danish financial firm told us he was too busy managing tickets to build the system that would have made the tickets unnecessary. An architect at a UK telecom had no self-service at all, just Jira. A senior developer at a UK bank said her biggest frustration was waiting on the ticketing system.
This is no joke. Self-service resource provisioning is a capability and Federated ownership is what produces it.
- The platform team sets standards and guardrails.
- Domain teams operate within them autonomously.
The teams that have this working got the operating model right before building the tooling. You need to slow down to accelerate better. Everyone else stopped at "we can create topics, we have gitops", they think this is enough and are missing the whole point. This is like 5% of a real solution.
"The key thing for us is approval gates. Guardrails. If somebody adds a thousand partitions, we need someone to have eyes on that." — Lead streaming data engineer at a major African bank
Self-service without federated ownership becomes either a free-for-all nobody can govern, or a tightly controlled environment nobody uses.
4. Nobody knows what a Kafka proxy actually does
This was the surprise. Kafka proxies came up in more conversations than any other architectural topic, but most people still think a proxy does one thing: route traffic for legacy clients.
An engineer at a European ISP runs one in production. It handles message routing. We asked about encryption, masking, or transformation. "We see a proxy as more of a routing tool."
Here's the range of what serious Kafka proxies like Conduktor Gateway can do:
- Routing: bridging legacy or non-native clients (most common)
- DR and migration: failover and provider switching
- Security: encryption and masking at the message level
- Full policy enforcement: aliasing, schema validation, virtual clusters, field-level encryption, audit, and access control
A VP at a global bank walked us through his checklist for a Kafka gateway: topic aliasing for migration, payload validation, virtual clusters for multi-tenant isolation, field-level encryption, data masking, and audit. It's a bank requirements doc.
Another engineer at a UK building society had built application-layer encryption for PII. He didn't know a proxy could do it at the message level, saving hours of per-application work. Plenty of teams are paying the same tax right now, doing it client-side and it's a pain to manage at scale.
A proxy is where policy and control live when they don't belong in the cluster or the application. Here's what a Kafka proxy actually does.
5. AI on Kafka is an ownership problem, not a streaming problem
The keynotes kept talking about agentic AI as the bottleneck. Kafka developers disagree:
"Spinning up infrastructure and writing code is rarely the bottleneck in a full end-to-end business solution." — Staff engineer at a UK retailer
A senior engineer at a major social platform building on-call AI agents put it directly: "Building the AI agent isn't the difficult thing. The data needs to be in the right structure."
A software engineer at a UK challenger bank has been feeding an AI agent context from ~40 microservices. "You have to give it the entire Java classes of all those microservices. That's usually when it starts hallucinating."
The hallucination is a legibility problem. The bottleneck is data quality, which sits downstream of governance, which sits downstream of your operating model. Every prerequisite practitioners named like ownership, schema discipline, topic visibility, federated governance, is foundational plumbing, not AI tooling.
The teams that will run AI on Kafka in 2027 are the teams getting their ownership and governance right in 2026.
All these discussions are coming from people running Kafka in production at banks, telcos, retailers, and energy companies. Costs are too high and too complex to undersand, self-service needs an operating model, proxies are underused, and AI isn't a shortcut past the governance work you haven't done yet.
If any of these hit home, the Conduktor blog goes deeper on each one.
Top comments (0)