DEV Community: Stéphane Derosiaux

I let an AI agent set up my entire Kafka platform. Here's what actually happened.

Stéphane Derosiaux — Mon, 08 Jun 2026 13:34:35 +0000

Your AI coding assistant can explain consumer groups, rebalancing, and exactly-once semantics. Ask it to actually set up a Kafka platform with governance, though, and it won't be able to do that on its own.

Between hallucinations, misunderstanding, production impact (I really saw Claude messing up a rolling upgrade of Kafka brokers), and the lack of knowledge of the products your Kafka infra is relying on, there's a lot working against it

The models, besides their training, have zero context about your infra. They've never seen your cluster, don't know your policies (technical, governance), and often have no way to check anything against your actual environment.

You can give it the missing context using Conduktor.

The thing that was missing

There is an open-source Conduktor skill you install into your AI assistant. It works with Claude Code, Cursor, VS Code Copilot, Gemini CLI:

npx skills add conduktor/skills

This is teaching the agent the whole platform and how to run process against it: Console, Gateway, and the CLI, so it can be efficient and not hallucinate.

After the install, the agent discovers your environment (Kafka clusters, Schema Registry, policies, etc.), asks questions based on what it finds, generates configs with real values and best practices, and runs everything with dry-run validation before it touches anything.

The CLI are really its "hands" as more deep than just MCP. The skill is the playbook where all the experience and practices from years of usage are written. This does a big difference VS "generate some YAML and cross fingers"

Starting from absolutely nothing

You can start from scratch with just Docker running and nothing else. No Kafka, no Conduktor, no config. When I just ask this (with the Conduktor skill setup):

install Conduktor and set it up so I can login

It checked my environment, asked what I was trying to do, wrote a docker-compose.yml, spun up the containers, hit one error along the way, self-corrected, and handed me a working platform, Kafka & Console perfectly configured.

I could ask the same but on my production Kubernetes. It would follow best practices too, use Helm, discover my environment, etc., and in minutes everything would be wired perfectly, with policies already in place.

This is much more powerful than a "human" quickstart, as the range of applications it covers is just wider and more production-ready already. The agent knows the Kafka domain, and with the skill it knows Conduktor, so the combination of both makes it ask me the right questions.

Governance, without becoming a Kafka lawyer

Running Kafka isn't the hard part anymore. Making it safe for a team to share is the hard part: naming conventions, ownership boundaries, policies. This is what prevent a Kafka cluster from turning into a wasteland of test-topic-final-v2.

The beautiful thing is to be able to ask large prompts like this now:

set up governance for two teams, Payments and Analytics, with topic policies and cross-team permissions

It worked in stages and figured out the dependency ordering itself. When the API rejected something, it read the rejection, restructured the YAML, and retried, with minimal hand-holding from me (just asking what policies I want based on what's possible). It ended up creating the following:

TopicPolicy objects: locking down naming per team, enforcing safe defaults (retention, replication, required labels) across every topic.
Application objects with non-overlapping resource boundaries to define ownership of resources and teams.
Topics with descriptions and labels in the catalog.
Cross-team permission giving Analytics read access to payments.orders.*.

This is federated ownership in practice: the platform team sets the boundaries, developers move freely inside them. Normally that knowledge takes months to accumulate and lives spreadsheet or Jira tickets. Here it lives in a skill file that every agent on the team can read.

Now flip to the developer side

Once those guardrails exist, a developer on the Payments team installs the same skill and never has to know any of it happened. No ApplicationInstance, no TopicPolicy, no YAML. They just talk.

"What topics do we have?"

The agent runs conduktor get Topic and shows the catalog — descriptions, owners, labels, visibility.

"I need a topic for my service."

The agent checks their ApplicationInstance, reads the policy constraints (naming prefix payments.*, retention one-to-seven days, a required data-criticality label), asks what the topic is for, generates compliant YAML, dry-runs it, and applies:

Topic/payments.fulfillment.shipped: Created

The developer just got a topic that's compliant by default. Without the skill, that's a JIRA ticket most likely, and asking platform team what's the right shape and what to put.

"How do I produce to my topic?"

It reads the cluster config, grabs the real bootstrap server, and hands back working code:

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "localhost:19092",
})

producer.produce(
    "payments.fulfillment.shipped",
    key="ord-123",
    value='{"orderId": "ord-123", "status": "shipped"}',
)
producer.flush()

Copy, paste, run.

"I need to read the Analytics team's clickstream."

The agent finds that analytics.clickstream.pageviews belongs to the Analytics team, then writes a read-only permission scoped to exactly that topic, at both the Kafka and Console layers. The developer doesn't know what an ACL is or what patternType: LITERAL means. They asked in English and got access.

What I actually take away from this

This walkthrough only touched governance and onboarding. The skill also covers Gateway (Kafka proxy) encryption, data quality rules, Terraform export, and CI/CD scaffolding.

Try it, it's one command:

npx skills add conduktor/skills

It's open source, so if you hit a workflow it handles badly, open a PR. And if you're new to Conduktor, the Community Edition is free and self-hosted, the skill will do the install for you.

This post was adapted from the original on the Conduktor blog.

5 Things Kafka Practitioners Actually Said at Current 2026

Stéphane Derosiaux — Mon, 01 Jun 2026 13:23:50 +0000

We were at Current 2026 in London two weeks ago. Keynotes about agentic AI and streaming agents but conferences conversations were about something else entirely.

1. The IBM-Confluent acquisition?

Six months after IBM acquired Confluent for $11B, I expected hot takes. What I got was mostly 'indifference'?

A data engineering lead at a major US bank called himself a "doomer", Kafka and Cassandra are his two favorite open-source projects, and both now sit under IBM. A staff engineer at a streaming vendor was more optimistic, speculating about Confluent's AI team merging with Watson.

The majority were more like "I try to concentrate on the tech side." "I don't know enough to comment." Moving on.

The acquisition was in the air, but practitioners had more pressing problems to talk about.

2. Kafka costs are the third certainty (after death and taxes)

Every cost conversation we had mapped to the same things:

Partition overprovisioning: inherited from not-knowing or templates tuned for a different workload
Retention mismatches: defaults that outlive the use case they were set for
Cluster proliferation: one-off clusters that never got consolidated
Orphan and duplicate topics: experiments or dead projectsthat never got cleaned up
Inefficient client patterns: batching, compression, and serialization left on defaults + JVM vs librdkafka differences
Static capacity: paying for peak when the load is 10% of that most of the time

Some examples of scale where they had these problems:

six years of unmanaged topics inherited during a Confluent Cloud migration
one team had twenty MSK clusters running multi-regions
MSK, Confluent, and Redpanda all running in parallel, looking for a single control plane as totally separated.

A data engineer at a German energy consultancy put it well:

"Kafka resource costs should not be taken for granted. Maybe I need sub-second latency, or maybe daily is enough. These are considerations you must make up front, before it's too late."

The solution is to change how to use Kafka: guardrails at topic creation, retention defaults that reflect reality, and ownership at the source. We typically see 25-40% of infrastructure costs come back this way.

We wrote up the full analysis in Your Platform Team Can't Fix Kafka Costs Alone.

3. Everyone wants self-service. Nobody has the operating model for it.

Self-service provisioning is a common topic now. Developers want to create topics and request permissions without filing a Jira ticket (many still do). Platform teams want to provide it to not be a bottleneck. The how is the question. Home-made is time consuming considering all variations, and using a vendor, it has to be flexible to adopt.

A senior engineer at a Danish financial firm told us he was too busy managing tickets to build the system that would have made the tickets unnecessary. An architect at a UK telecom had no self-service at all, just Jira. A senior developer at a UK bank said her biggest frustration was waiting on the ticketing system.

This is no joke. Self-service resource provisioning is a capability and Federated ownership is what produces it.

The platform team sets standards and guardrails.
Domain teams operate within them autonomously.

The teams that have this working got the operating model right before building the tooling. You need to slow down to accelerate better. Everyone else stopped at "we can create topics, we have gitops", they think this is enough and are missing the whole point. This is like 5% of a real solution.

"The key thing for us is approval gates. Guardrails. If somebody adds a thousand partitions, we need someone to have eyes on that." — Lead streaming data engineer at a major African bank

Self-service without federated ownership becomes either a free-for-all nobody can govern, or a tightly controlled environment nobody uses.

4. Nobody knows what a Kafka proxy actually does

This was the surprise. Kafka proxies came up in more conversations than any other architectural topic, but most people still think a proxy does one thing: route traffic for legacy clients.

An engineer at a European ISP runs one in production. It handles message routing. We asked about encryption, masking, or transformation. "We see a proxy as more of a routing tool."

Here's the range of what serious Kafka proxies like Conduktor Gateway can do:

Routing: bridging legacy or non-native clients (most common)
DR and migration: failover and provider switching
Security: encryption and masking at the message level
Full policy enforcement: aliasing, schema validation, virtual clusters, field-level encryption, audit, and access control

A VP at a global bank walked us through his checklist for a Kafka gateway: topic aliasing for migration, payload validation, virtual clusters for multi-tenant isolation, field-level encryption, data masking, and audit. It's a bank requirements doc.

Another engineer at a UK building society had built application-layer encryption for PII. He didn't know a proxy could do it at the message level, saving hours of per-application work. Plenty of teams are paying the same tax right now, doing it client-side and it's a pain to manage at scale.

A proxy is where policy and control live when they don't belong in the cluster or the application. Here's what a Kafka proxy actually does.

5. AI on Kafka is an ownership problem, not a streaming problem

The keynotes kept talking about agentic AI as the bottleneck. Kafka developers disagree:

"Spinning up infrastructure and writing code is rarely the bottleneck in a full end-to-end business solution." — Staff engineer at a UK retailer

A senior engineer at a major social platform building on-call AI agents put it directly: "Building the AI agent isn't the difficult thing. The data needs to be in the right structure."

A software engineer at a UK challenger bank has been feeding an AI agent context from ~40 microservices. "You have to give it the entire Java classes of all those microservices. That's usually when it starts hallucinating."

The hallucination is a legibility problem. The bottleneck is data quality, which sits downstream of governance, which sits downstream of your operating model. Every prerequisite practitioners named like ownership, schema discipline, topic visibility, federated governance, is foundational plumbing, not AI tooling.

The teams that will run AI on Kafka in 2027 are the teams getting their ownership and governance right in 2026.

All these discussions are coming from people running Kafka in production at banks, telcos, retailers, and energy companies. Costs are too high and too complex to undersand, self-service needs an operating model, proxies are underused, and AI isn't a shortcut past the governance work you haven't done yet.

If any of these hit home, the Conduktor blog goes deeper on each one.

How to analyze the cost of Kafka?

Stéphane Derosiaux — Mon, 25 May 2026 15:19:36 +0000

Which side are you on: "This is just what Kafka costs at scale" or "We should switch to a cheaper Kafka provider"?

At Conduktor, our field team works inside Kafka environments that have been running for a long time. We see this: most Kafka teams are overpaying by 25 to 40 percent. Not because anyone did anything wrong, but because of how Kafka got built up over time.

The cost drivers of Kafka are weirdly context-dependent: the infrastructure and the provider are a tiny part of the full picture.

The "how" it's being used is the real question.

Five bad patterns eating budget

Below is what see, the same patterns show up everywhere, and are the first things we work with our customers.

1. Partition overprovisioning

"How many partitions?" is the most common question with Kafka. I heard last week someone telling me an org just defaults to "64". I was shocked. Not only providers may price per partitions, but from a Kafka point of view: this takes metadata and open files etc.

Partitions depend on throughput and concurrency expected (consumer parallelism). If a 64-partitions topic is sitting in a cluster with barely no traffic, you're just losing money on all sides. Multiply by dozens or hundreds of topics at scale.

2. Retention that makes no sense

Long retention on topics that nobody reads past the last few hours. Do you need replay? Default is 7-day retention, but it's often applied uniformly, when some topics only need a couple of hours and others genuinely need weeks.

Tips: when using compacted topics and/or Kafka streams (changelog etc.), data is being stored indefinitely, that can cause some security/regulations issues.

3. Let's spin up another cluster

One-cluster-per-team was a reasonable isolation strategy a long time ago. We saw this multiple times, more than 500 clusters, with tons of mirroring to share data. Throwing money down the drain.

You're paying for underutilized clusters instead of consolidating onto fewer well-managed ones.

4. Zombie topics

Topics created for experiments, migrations, or one-off tests that were never cleaned up. It's a simple thing but cost so much money as no one is looking. Every one of them is replicated and has retention costs. We've seen enterprises with hundreds of zombie topics, who were so surprised when we showed them.

5. Runaway egress

We had a customer where egress was running 30x higher than ingress on a single topic because of a misconfigured consumer. Buggy consumers, unnecessary fan-out, and chatty clients create traffic patterns that are invisible without dedicated infra monitoring. Egress is rarely free.

How to deal with it

Pick your starting point based on where the waste is concentrated.

Stop the bleeding: better defaults

Low-coordination work that pays off over time. It's better to have exceptions rather than wrong defaults you can't rollback.

Set sensible low partition defaults (3) and short retention (1 day). Increase if necessary only.
Enforce client-side compression. (Conduktor Gateway)
Require ownership metadata at topic creation. (Conduktor)

This won't reduce your bill right away, but it will prevent it from getting worse.

Trim the fat: optimize what's running

Tune retention where it's drifted, analyze consumer patterns.
Retire topics with no active producers or consumers.
Right-size partition counts (this is the hard one, since it means recreating topics and coordinating with every producer and consumer). - Consolidate Kafka clusters, introduce multi-tenancy (Conduktor)

This work easily moves the infrastructure bill, we saw reductions of $500k just doing this.

Now, keep it clean, be disciplined

After a cleanup, the same "drift" will start operating again.

To help you keeping the direction, have absolute visibility into what you Kafka ecosystems contains and what it costs (chargeback is powerful for this), clear ownership so every topic and cluster has a team accountable for it, and a regular review cadence to catch drift before it becomes permanent. Not heavyweight governance. Just enough discipline that the cleanup doesn't have to be repeated every year.

Where to start

The diagnostic question is simple: which of these patterns are present in your environment, and what are they costing you?

The original deep-dive goes further into the four layers of Kafka cost (infrastructure, ecosystem tooling, vendor/licensing, and operational) and includes a framework for sequencing the work.

If you want to look at your own estate, Conduktor's field team does a free cost analysis where they walk through your environment with you and give you concrete numbers.

1500 Lines of Markdown vs 15000 Lines of Python.

Stéphane Derosiaux — Wed, 31 Dec 2025 13:25:39 +0000

The best orchestrator might be the one you don't have to build. Claude Code is already an orchestrator. Stop building infrastructure around it.

I spent a weekend studying GPT-Researcher, an open-source project with 24,000+ GitHub stars. It builds an autonomous research agent that generates comprehensive reports with citations. The architecture is elegant: multiple specialized agents coordinate through LangGraph, parallel execution speeds up research, and quality gates ensure reliable output.

It uses LLM calls to decide which agent to run. It uses LLM calls to generate sub-queries. It uses LLM calls to select tools. It uses LLM calls to coordinate parallel work.

We're wrapping LLMs in infrastructure to teach them orchestration... when they can already orchestrate.

The orchestration overhead

Consider a typical agent workflow:

LLM API call to analyze the task
LLM API call to plan the approach
LLM API call to select tools
LLM API call to execute (finally, the actual work)
LLM API call to verify results

Four out of five calls are orchestration overhead. The LLM produces structured outputs (JSON) that code interprets to decide... what to ask the LLM next and passing it the right context.

This pattern is everywhere. LangChain and LangGraph provide graph-based workflows. AutoGen from Microsoft enables multi-agent conversations. CrewAI offers role-based agent coordination, used by Oracle, PwC, and NVIDIA. Each framework solves real problems: managing state, coordinating agents, handling failures.

But they all share the same assumption: the LLM needs infrastructure to orchestrate.

What GPT-Researcher does well

Credit where it's due: GPT-Researcher is excellent. According to its maintainers, it outperforms Perplexity, OpenAI's research tools, and other systems in benchmarks on citation quality, report quality, and information coverage.

The architecture is sophisticated:

Multi-agent roles: Chief Editor orchestrates the process. Researchers investigate subtopics. Editors plan structure. Reviewers validate quality. Revisers incorporate feedback. Writers compile reports.
Parallel execution: Research happens concurrently across subtopics. Multiple retrievers (Tavily, Google, Bing) run in parallel. Web scraping is asynchronous.
Quality gates: Review cycles catch errors. Revision loops improve output.

The architecture tax

Here's what the orchestration layer requires:

# agent_creator.py - LLM decides which agent to use
response = await llm.call(
    "Analyze this query and return JSON with agent type..."
)
agent_type = parse_json(response)  # error handling, retries

# query_processing.py - LLM generates sub-queries
response = await llm.call(
    "Generate search queries for this task..."
)
queries = parse_list(response)  # more parsing, more error handling

# tool_selector.py - LLM selects MCP tools
response = await llm.call(
    "Select relevant tools from this list..."
)
tools = parse_tool_selection(response)  # yet more parsing

Each step requires prompt engineering, output parsing, error handling, and retry logic. The orchestration layer is substantial.

Claude Code's native capabilities

Here's what Claude Code provides out of the box:

Claude Code doesn't need an LLM call to decide what tools to use. It IS the LLM. It reasons about the task and uses tools directly, in the same context, without round-trips.

When you ask Claude Code to research a topic:

It analyzes the query (no separate LLM call)
It generates sub-queries (no separate LLM call)
It executes parallel searches (native Task agents)
It synthesizes results (no separate LLM call)
It writes the report (native Write tool)

What required infrastructure now requires prompts.

The rewrite

I created Claude Researcher to test this. It's not a Python package. It's four commands and one skill file.

Commands:

The /research-team command implements the full multi-agent pattern:

Claude Code (Chief Editor)
      │
      ├── [PARALLEL] Research Agent 1 → findings
      ├── [PARALLEL] Research Agent 2 → findings
      └── [PARALLEL] Research Agent 3 → findings
              ↓
         Draft Report
              ↓
         Reviewer Agent → feedback
              ↓
         Reviser Agent → improved draft
              ↓ (repeat until quality gate passes)
         Final Report

Quality gates are defined in the command file. Review cycles repeat until scores meet thresholds. The multi-agent patterns from GPT-Researcher, expressed as instructions rather than code.

The difference in sub-query generation:

GPT-Researcher:

prompt = f"""Write {max_iterations} google search queries...
You must respond with a list of strings in the following format: [{example}].
The response should contain ONLY the list."""

response = await llm.call(prompt)
queries = json.loads(response)  # parsing, error handling, retries

Claude Researcher:

Generate 5 search queries to research: "{query}"
- Each query should explore a different angle
- Include queries for recent information when relevant

Claude Code executes the queries directly. No parsing layer. No error handling for malformed output. The LLM produces the queries and uses them in the same context.

When orchestration frameworks make sense

This isn't a claim that frameworks are useless. They solve real problems:

Production systems with strict SLAs. When you need guaranteed response formats, retry logic, circuit breakers, and observability, frameworks provide battle-tested infrastructure. Claude Code is conversational, not transactional.
Non-Claude environments. If you're building on GPT-4, Gemini, or open-source models, Claude Code isn't available. Frameworks provide the coordination layer those environments lack.
Complex state machines. Research is relatively linear: gather, synthesize, write. Workflows with branching logic, human-in-the-loop steps, or long-running state benefit from explicit orchestration.
Team standardization. Frameworks enforce patterns. When multiple developers build agents, shared infrastructure ensures consistency. Markdown commands are flexible but less structured.
Audit requirements. Enterprise deployments often need detailed logs of every decision. Frameworks with explicit orchestration make this easier than conversational interfaces.

The question isn't "frameworks vs no frameworks". It's "do you need the framework for THIS task?"

Try it yourself

Clone the repo:

git clone https://github.com/sderosiaux/claude-researcher

Copy to your Claude Code config:

cp commands/_*.md ~/.claude/commands/
cp skills/researcher.md ~/.claude/skills/

Run a research task:

/research "Impact of AI agents on software development" --depth=deep

/research-team "Comparison of vector databases" --quality=high

/lookup "What is Claude Opus 4.5 context window?"

No pip install. No API keys beyond what Claude Code already uses. No configuration files.

Claude Code IS the Orchestrator, a really good one

There's a pattern in software engineering: we build abstractions to solve problems, then build abstractions to manage our abstractions. Each layer adds capability but also complexity, configuration, and cognitive load.

What if the base layer already does what I need?

Claude Code is an LLM with native tool access, parallel execution, and context management. GPT-Researcher is infrastructure that makes LLMs do those things. For research tasks, the native capabilities are sufficient.

The best orchestrator might be the one you don't have to build.