DEV Community: Hamdi Mechelloukh

Real-time streaming pipeline with Apache Flink 2.0, Kafka and Iceberg

Hamdi Mechelloukh — Tue, 31 Mar 2026 11:09:58 +0000

It's 2:03 PM. A flash sale just started.

In the warehouse, an operator is entering incoming orders into the management system. He types a quantity, makes a mistake, corrects it immediately. Two events, one reality. Thirty seconds apart.

The batch job that runs at 2 AM will see both. It won't know which one is right. Depending on how the reconciliation logic is written, if it exists at all, it picks one of the two, often non-deterministically. And if the correction falls into the next batch window, the problem doesn't surface right away: the morning's numbers are wrong, cleanly, with no technical error in sight.

This is a real and recurring source of data quality problems in data teams.

Processing events as they arrive, in order, with their temporal context intact, fundamentally changes how this problem is handled. That's the starting point for this project: an end-to-end streaming pipeline on the Olist e-commerce dataset, built with Apache Flink 2.0, Kafka and Iceberg.

The dataset and the problem

The Olist dataset is a public Brazilian e-commerce dataset: orders, products, sellers, customers, reviews. 100,000 orders over two years.

I had already built a batch lakehouse on this same dataset. The logical next step was to go to the other extreme: stream processing, one-minute calculation windows, anomaly detection at the second level. Three concrete needs:

Revenue by category in real time — know which category is performing at every minute
Anomaly detection — a customer placing multiple orders within a few minutes, or an order at an abnormal price
Global KPIs — average order value, order rate, total revenue in real time

These are the three jobs that make up the pipeline.

Why Apache Flink?

The question is worth asking. There are other options for streaming in Java:

Kafka Streams — easy to operate, no separate cluster, but limited to Kafka-in/Kafka-out topologies
Apache Spark Structured Streaming — micro-batches, minimum latency of a few seconds, but familiar if you already know Spark
Flink — true event-by-event streaming, native event-time processing, built-in CEP

Flink was the natural choice for two reasons.

CEP (Complex Event Processing). Detecting "3 orders from the same customer within 5 minutes" is not an aggregation, it's a temporal correlation between events. Flink CEP handles this natively with a pattern DSL. In Kafka Streams, it requires maintaining manual state and writing the temporal logic by hand.

Flink 2.0. Version 2.0 brought native Java 21 support. Working on the current version rather than an end-of-life one was a deliberate choice.

The architecture

Olist CSV → Simulator → Kafka (orders)
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
    RevenueAggregation  AnomalyDetection  RealtimeKpi
    (tumbling 1 min)    (CEP 5 min)       (windowAll 1 min)
              │               │               │
              ▼               ▼               ▼
      Kafka (revenue)  Kafka (alerts)  Kafka (kpis)
              │               │               │
              └───────────────┴───────────────┘
                              │
                    Apache Iceberg (MinIO)

Three independent jobs, one shared source topic, three output topics, and an optional Iceberg data lake.

The independence of the jobs is a deliberate choice. In production, you want to be able to restart AnomalyDetectionJob without affecting RevenueAggregationJob. Each job has its own checkpoint, its own state, its own topology.

Job 1: RevenueAggregationJob

The simplest of the three. It aggregates revenue by product category over one-minute windows.

orders → filter nulls → map to RevenueByCategory → keyBy(category)
       → TumblingWindow(1 min) → reduce + ProcessWindowFunction
       → Kafka sink + Iceberg sink (optional)

A few details that matter.

Watermark strategy. The pipeline uses event time, meaning the event timestamp in the Kafka message, not the arrival time. The strategy is forBoundedOutOfOrderness(10 seconds) with a 5-second idleness timeout.

Why idleness? If a stream is empty for several minutes (the simulator is stopped, for example), Flink can no longer advance its watermark. Without withIdleness, windows never close. With withIdleness(5s), Flink ignores silent partitions and advances anyway.

Side outputs. Invalid events (null price, missing timestamp) are not silently dropped. They are routed to a side output that logs them. This avoids the scenario where events disappear without a trace.

Two-phase reduction. Before the window is applied, a reduce combines events by category on the fly. The ProcessWindowFunction then only attaches the window start and end timestamps. Less state to store, less work at window closure.

Job 2: AnomalyDetectionJob

This one is more interesting. It detects two types of anomalies through two different mechanisms.

Threshold detection: price anomaly

A filter on price:

ordersStream
    .filter(event -> event.getPrice() != null
                  && event.getPrice().compareTo(priceThreshold) > 0)
    .map(event -> OrderAlert.priceAnomaly(event.getCustomerId(), event.getPrice()))

The threshold (500 BRL by default) is configurable via environment variable. One subtlety: the filter is > 0, not >= 0. An order at exactly 500 BRL is not an anomaly. This behavior is covered by a specific test.

Pattern detection: suspicious frequency

This is where CEP comes in.

Pattern<OrderEvent, ?> pattern = Pattern.<OrderEvent>begin("orders")
    .timesOrMore(suspiciousOrderCount)
    .within(Duration.ofMinutes(5));

PatternStream<OrderEvent> patternStream = CEP.pattern(
    ordersStream.keyBy(OrderEvent::getCustomerId),
    pattern
);

This pattern says: if the same customer places 3 or more orders within a 5-minute window, it's suspicious.

The key is keyBy(customerId). Without it, Flink would compare orders from different customers. With keyBy, each customer has their own independent CEP state.

Both streams, price alerts and frequency alerts, are then merged with union() before being sent to the Kafka output topic.

Job 3: RealtimeKpiJob

Global KPIs: average order value, orders per minute, total revenue. The calculation is straightforward, but the implementation reveals an interesting trade-off.

windowAll: the acknowledged bottleneck

To calculate total revenue across all orders in one minute, you need to aggregate all events together, without splitting by key. In Flink, this is called windowAll.

windowAll forces all events through a single processing instance. It's a bottleneck by design. At this volume (50 events per second), it's more than sufficient. If throughput rose to 50,000 events per second, a pre-aggregation by key followed by a merge would be necessary. We don't do that here because adding complexity for a hypothetical need is not good engineering.

Two-phase aggregation

The KPI calculation uses the AggregateFunction + ProcessAllWindowFunction pattern:

KpiAggregateFunction accumulates the count and sum as events arrive, continuously
KpiWindowFunction computes the average and derived metrics at window closure

This separation maintains minimal state (two numbers) instead of buffering all raw events. The ProcessAllWindowFunction only receives the final accumulator.

The BoundedHistogram

An optional but interesting detail: a custom Flink Histogram implementation.

The Flink Metrics API exposes three standard types: Counter, Gauge, Histogram. For a Histogram, Flink expects an implementation that returns percentiles, mean and standard deviation via a HistogramStatistics interface.

The BoundedHistogram is a fixed-size circular buffer (1000 values). When the buffer is full, new values overwrite the oldest ones.

public synchronized void update(long value) {
    values[writeIndex % values.length] = value;
    writeIndex++;
}

Simple, thread-safe, bounded memory. It allows Grafana to show the distribution of average order values, not just the latest single value.

Iceberg integration: what I didn't anticipate

The Apache Iceberg integration was optional in the initial architecture. In practice, this is where I spent the most time.

The classloader problem

Flink 2.0 loads its filesystem plugins (including flink-s3-fs-hadoop) in an isolated classloader, invisible to user code. When iceberg-flink-runtime tries to instantiate S3AFileSystem at write time, it can't find the class provided by the Flink plugin.

The solution: bundle hadoop-aws and the AWS SDK directly in the fat JAR, with aggressive exclusions to avoid dependency conflicts.

implementation("org.apache.hadoop:hadoop-aws:3.4.1") {
    exclude(group = "com.amazonaws", module = "aws-java-sdk-bundle")
}
implementation("com.amazonaws:aws-java-sdk-s3:1.12.780")
implementation("com.amazonaws:aws-java-sdk-sts:1.12.780")

The fat JAR reaches ~710 MB. Not ideal, but that's the real cost of an Iceberg + Flink + S3 integration outside a managed service.

Credential timing

Second surprise: HadoopCatalog reads its S3 configuration at construction time, not after. The intuitive pattern of creating the catalog and then injecting configuration doesn't work:

// Credentials are injected too late
HadoopCatalog catalog = new HadoopCatalog();
catalog.setConf(hadoopConf);

// Credentials must be in the Configuration before construction
Configuration hadoopConf = new Configuration();
config.toProperties().forEach(hadoopConf::set);
HadoopCatalog catalog = new HadoopCatalog(hadoopConf, config.warehouse());

Same applies to CatalogLoader.hadoop(). This behavior is not prominently documented. It's the kind of error you only discover through end-to-end testing.

Docker Compose and .env resolution

A less expected issue: Docker Compose v2 resolves the .env file from the directory containing docker-compose.yml, not from the current working directory.

# From the project root, this command ignores the .env at the root
docker compose -f docker/docker-compose.yml up -d

# You need to pass the path explicitly
docker compose --env-file .env -f docker/docker-compose.yml up -d

Without this, ICEBERG_ENABLED=true in the .env is ignored and jobs start without an Iceberg sink, with no error message.

Observability

Flink exposes its metrics via Prometheus on port 9249. Each job exposes custom metrics:

Metric	Type	Job
`windowsEmitted`	Counter	RevenueAggregationJob
`kpiWindowsEmitted`	Counter	RealtimeKpiJob
`lastWindowOrderCount`	Gauge	RealtimeKpiJob
`orderValueDistribution`	Histogram	RealtimeKpiJob
`priceAnomalyAlertsEmitted`	Counter	AnomalyDetectionJob
`suspiciousFrequencyAlertsEmitted`	Counter	AnomalyDetectionJob
`deserializationErrors`	Counter	All

These metrics land in Prometheus every 15 seconds and are visualized in Grafana. The deserializationErrors metric is particularly useful: if the simulator sends a malformed message, the counter rises and you see it immediately in the dashboard, without the job crashing.

Testing

The tests use Flink's MiniCluster, an embedded Flink cluster that runs in the test process, with no external infrastructure.

This choice has a cost: tests are slower (a few seconds each). But they test the actual behavior of Flink operators, not a mock. The AnomalyDetectionJobTest specifically validates CEP edge cases:

2 orders in 5 minutes → no alert
3 orders in 5 minutes → alert triggered
Order at exactly 500 BRL → no price alert

18 tests in total, covering all three jobs, the BoundedHistogram and the deserialization schema. The CI (GitHub Actions) compiles and runs all tests on every push, with a JaCoCo report as an artifact.

Batch or streaming: the real debate

Back to the opening scene.

Streaming is often perceived as expensive: the cluster runs continuously, the infrastructure never shuts down. That's true. But this comparison is incomplete.

A batch pipeline that handles events which correct themselves over time accumulates its own debt. Timeline reconciliation logic. Re-processing when an event arrives late. Alerts, manual interventions, data engineers spending time explaining why numbers are inconsistent across two windows. This cost is diffuse: it doesn't appear on any cloud bill, but it accumulates in sprints, in support, in technical debt.

Nobody actually does this calculation in practice — because it's too costly to conduct seriously.

This project doesn't claim to settle the debate. What it shows is that an end-to-end streaming pipeline with Flink 2.0 is accessible today without managed infrastructure, without Databricks, without Confluent Cloud. A docker compose up and the pipeline runs. The complexity is in the integration details, not in the paradigm itself.

The code is on GitHub. The start-e2e.sh script launches the entire pipeline in a single command.

You can also read this and other articles on my portfolio.

Building an open-source vendor-neutral lakehouse

Hamdi Mechelloukh — Fri, 20 Mar 2026 11:05:38 +0000

When you work in data, you always end up asking the same question: what happens if we need to switch platforms tomorrow?

I've seen firsthand that software vendors can be aggressive with pricing, and they won't hesitate to sunset a product that isn't generating enough revenue. When that happens, you need to migrate quickly, or face massive costs in migration, redevelopment, and lost time.

This conviction led me to build an end-to-end open-source, vendor-neutral lakehouse, from messaging to visualization. Here are the architecture choices, the trade-offs, and what I learned.

The stack: Kafka → Spark → Iceberg

Sources → Kafka → Spark (Bronze) → Spark (Silver) → Spark (Gold) → Streamlit
                                                                   ↑
                                                          Great Expectations
                                                          (quality at each layer)

Kafka for ingestion

Choosing event-driven for data transfer isn't trivial. In computing, managing time is one of the hardest problems. On the operational side, we're moving more and more toward event-driven architectures precisely for this reason: an event arrives when it arrives, and the system processes it. No batch window to respect, no "the file should have arrived at 6 AM".

Kafka is the de facto standard for this kind of architecture. Open-source, battle-tested, and crucially: no vendor lock-in. You can deploy it on any cloud or on-premise.

Spark for compute

You might ask: why Spark in an event-driven architecture? My position is pragmatic. Pure streaming via Kafka works well for ingestion into bronze, or even silver, to handle temporality upstream. But once you reach heavy transformations (aggregations, joins, enrichments), Spark remains the most battle-tested and portable tool.

Spark's advantage is that it runs everywhere: on a YARN cluster, on Kubernetes, on Databricks, on EMR, or locally for development. It's one of the few compute tools that doesn't lock you in.

Iceberg for the table format

Iceberg is the open table format that's gaining momentum. My choice was partly technical curiosity: I use Delta Lake daily at work, so I wanted to explore the alternative.

But beyond curiosity, Iceberg has properties that make it particularly suited for a vendor-neutral lakehouse:

Open format — no dependency on a specific vendor
Time travel — query data at any point in time
Schema evolution — add or modify columns without rewriting data
Partition evolution — change partitioning scheme without migration
Compatible with all engines — Spark, Trino, Flink, Dremio, Athena...

The project could just as well run with Delta Lake or Hudi. In fact, it would be interesting to offer format choice to anyone forking the project.

The layered architecture: bronze, silver, gold

The medallion pattern (bronze/silver/gold) structures data in three levels of refinement:

Bronze — raw data as it arrives, no transformation
Silver — cleaned, deduplicated, properly typed data
Gold — aggregated data ready for business consumption

Honestly, these terms are recent. A few years ago, we called them dataraw, dataprep, dataset. The vocabulary changes, the principle stays the same. What matters is to follow this progressive refinement logic without being rigid. Functional reality always takes precedence over technical rules. If data doesn't need three layers, it doesn't need three layers.

MinIO: S3-compatible without the lock-in

One point that might surprise you: why MinIO rather than S3 directly?

Because S3 is an AWS service, and using S3 means locking yourself into AWS. MinIO implements the S3 API identically: every tool that speaks S3 speaks MinIO without modification. You can develop and test locally, deploy on any cloud, and migrate to S3, GCS or Azure Blob Storage without changing a single line of application code.

That's exactly the vendor-neutral principle: use open standards rather than proprietary managed services.

Data quality: Great Expectations and its limits

Great Expectations is the most widely used data validation tool in the Python/Spark ecosystem. I integrated it at each pipeline layer to validate data on input and output.

The tool does its job well for simple quality rules: nullability, uniqueness, value ranges, formats. It's also a tool I've seen used in enterprise settings, which validated the choice.

But it has real limitations:

Complex quality rules (cross-table consistency, conditional business rules) are hard to express
Resource-intensive checks (massive joins for cross-source duplicate detection) don't scale easily
And most importantly: discovering quality issues is not enough

This last point is crucial and comes directly from my production experience at Decathlon. You can set up all the quality alerts in the world. If source teams have no commitment to fix the issues, nothing will change. You need to work on data quality service-level agreements: SLAs on fix turnaround, shared responsibilities, clear escalation paths. Without that, source teams will make little effort to resolve quality problems.

The difficulty of vendor-neutral

The biggest challenge of this project wasn't technical in the traditional sense. It was resisting the temptation of managed services.

At every step, there's a managed option that saves time:

Why manage your own Kafka when there's Amazon MSK or Confluent Cloud?
Why MinIO when S3 is there, configured in 2 clicks?
Why self-hosted Airflow when there's MWAA?

The answer is always the same: because the day the pricing changes or the service is deprecated, you need to be able to leave. This doesn't mean you should never use managed services. It means you should do it knowingly, and make sure the abstraction layer allows switching.

In practice, building vendor-neutral requires more upfront effort:

Terraform for declarative, multi-cloud infrastructure management
Docker for isolation and portability
Standard interfaces everywhere (S3 API, JDBC, etc.)

But once it's in place, the freedom it provides is invaluable.

Orchestration: Airflow

Airflow is the natural choice for orchestration in a vendor-neutral stack. Open-source, extensible, and above all: the community is massive. When you have an Airflow problem, someone has already had it and posted the solution on Stack Overflow.

Alternatives would be Dagster or Prefect, but Airflow remains the most widely deployed in production and the most in-demand on the market. Pragmatism.

IaC: Terraform for multi-cloud

Terraform is the piece that makes vendor-neutral viable at scale. Infrastructure is described in code, versioned in Git, and deployable on AWS, GCP or Azure with provider changes, no complete rewrite needed.

In this project, Terraform modules provision AWS infrastructure, but the same logic could be ported to another cloud without rebuilding the application architecture.

What I took away

Vendor-neutral has a cost, but so does lock-in

Building vendor-neutral requires more upfront work. But lock-in has a hidden cost that explodes the day you need to migrate. And that day always comes sooner than you think.

Open formats are your data's life insurance

Iceberg, Parquet, Avro: as long as your data is in an open format, you can switch compute engines without losing your data. It's the most important decision in a data architecture.

Data quality is an organizational problem, not a technical one

Tools like Great Expectations are necessary but not sufficient. Without service-level agreements with sources, quality alerts are just noise.

Functional reality takes precedence over patterns

Bronze/silver/gold is a good guide, not a religion. If your data only needs two layers, don't make three to respect a pattern. Architecture should serve the business need, not the other way around.

Streaming doesn't replace batch, it complements it

Kafka for real-time ingestion, Spark for heavy transformations. The two coexist, and that's healthy. Trying to do everything in streaming is as dogmatic as doing everything in batch.

Going further

The source code is available on GitHub. The project uses the Olist dataset (Brazilian e-commerce) as a data source, making it testable without heavy infrastructure.

You can also read this and other articles on my portfolio.

Lessons from 2 years as Production Manager at Decathlon Digital

Hamdi Mechelloukh — Fri, 20 Mar 2026 10:59:30 +0000

For two and a half years, I stepped away from code to manage data production for sales at Decathlon Digital. A role I discovered upon arrival: the job title said "Production Expert", and I quickly realized it was going to be a full-time commitment.

Here's what I learned from switching to the other side.

Context: Perfeco and sales data

Perfeco was the data product that served the company's economic performance and sales data. In practice, it meant:

An ingestion pipeline built on Talend and Redshift — data was processed and stored in Redshift, then pushed to S3
2 to 3 million sales per day ingested
Data exposed in the datalake and via an API consumed by multiple business teams
Kafka messages with XML payloads converted to CSV before loading
A scheduler (OpCon) to orchestrate ingestion jobs

My role: make sure all of this runs, every day, without interruption.

What "Production Manager" actually means day-to-day

Coming from development, you'd think production is about monitoring and a few alerts. Reality is very different.

Reducing the operational burden

My main goal wasn't to react to incidents, but to reduce their frequency. That meant:

Proactive alerting — setting up the right dashboards (QuickSight, Tableau) and alerts to detect anomalies before they become incidents. Automatic Jira ticket creation when a threshold is breached.
Data quality at the source — analyzing and detecting quality issues upstream, then escalating them to source teams. This is facilitation work, not code: convincing an upstream team that their data is poorly formatted takes time and diplomacy.
Run documentation — writing and maintaining on-call procedures so that any team member can intervene at 3 AM without relying on one person's memory.
Run KPIs — scripting metrics collection to objectively measure stability: incident count, resolution time, data availability.

Facilitating, not coding

The biggest surprise was the relational dimension of the role. I spent more time managing people than technology:

Facilitating consumer teams — improving incident communication. When an ingestion is delayed, 5 different teams need to know why and when it will be resolved. You need a clear channel, a clear message, and consistency.
Facilitating source teams — working with teams that produce upstream data so they fix quality issues at the root.
On-call planning — organizing rotations for the team, making sure everyone is trained and the load is fairly distributed.
Postmortems — I organized regular meetings with both data sources and consumers. Postmortems were filled collaboratively during these sessions: what happened, why, and what actions to take to prevent recurrence. This collaborative format aligned everyone and avoided the blame game.

The typical incident: when the scheduler crashes

My nemesis during those two years was the OpCon scheduler client crashing on the machine. Silently.

The scenario was always the same:

The OpCon client crashes → no jobs are launched
Sales keep arriving via Kafka (messages with XML payloads)
Messages pile up, hundreds of thousands within hours
When we restart the scheduler, the XML → CSV conversion job faces a massive backlog
The Talend job struggles, processing times explode, Redshift is overwhelmed, data arrives late in S3

The biggest incidents we had were all tied to this problem. What made it frustrating was that the client crash was silent: no alert, no explicit log. We'd only discover it by noticing the absence of data downstream.

The lesson: monitoring the absence of events is as important as monitoring errors. If a job that runs every 15 minutes hasn't executed in 30 minutes, that's a strong signal.

What I learned

Production is engineering

Reducing operational burden isn't just "adding alerts". It's designing an observability system, automating detection, documenting procedures, and measuring improvement. It's engineering work in its own right.

Communication is a technical skill

Writing a clear incident message, running a blameless postmortem, convincing a source team to fix a data format. These are skills as important as writing code. And they can be practiced.

Proactive alerting changes everything

The difference between a PM who reacts and one who manages is proactivity. When you discover an incident from an automatic alert at 8 AM instead of a call from a business team at 10 AM, you've gained 2 hours and a lot of peace of mind.

Monitor the silence

The most dangerous incidents don't generate errors: they generate silence. A pipeline that stops running, a scheduler that has crashed, a message that never arrives. Alerts on the absence of activity saved me more often than alerts on errors.

Documentation is not optional

In dev, you can sometimes get by with readable code and a few comments. In production, if the on-call procedure isn't written down, it doesn't exist. The person on call at 3 AM doesn't have time to guess.

Why I went back to technical work

After two and a half years, I decided to return to a Data Engineer role. The reason is simple: I felt I was regressing technically.

The PM day-to-day is fascinating: the diversity of problems, the human dimension, the direct impact on data reliability. But I was spending my days facilitating, documenting and communicating, and less and less designing and coding.

I was afraid of falling behind, of no longer being up to speed on fast-evolving technologies: Spark, Databricks, lakehouse architectures. The risk of becoming a purely managerial profile without technical expertise didn't sit well with me.

Today, looking back, I don't regret the experience. It gave me an understanding of production that many developers don't have. When I design a pipeline now, I naturally think about observability, error recovery, and operational documentation. These are reflexes that code alone wouldn't have given me.

In summary

If you're a developer and someone offers you a production-oriented role, here's what I'd tell you:

It's a real job, not a support role. It requires engineering, rigor, and a lot of soft skills.
You'll learn things that development will never teach you — crisis communication, priority management under pressure, the end-to-end view of a data product.
Set a time limit. It's enriching, but if your core expertise is technical, don't stay too long or you risk falling behind.
Bring those reflexes back into your code. Observability, documentation, monitoring the silence — these are skills that make better engineers.

You can also read this and other articles on my portfolio.

AgenticDev: a multi-LLM framework for generating tested code

Hamdi Mechelloukh — Fri, 20 Mar 2026 10:58:27 +0000

In late 2025, after spending hours prompting LLMs one by one to generate code, a question kept nagging me: what if multiple LLM agents could collaborate to produce a complete project? Not a single agent doing everything, but a specialized team (an architect, a developer, a tester), each with its own role, tools, and constraints.

That's how AgenticDev was born, a Python framework that orchestrates 4 LLM agents to turn a plain-text request into tested, documented code.

In this article, I share the architecture decisions, the problems I ran into, and the lessons learned.

Starting point: testing the limits of multi-agent collaboration

My initial goal was simple: explore how far LLM agents can collaborate autonomously. Not a throwaway POC, but a real pipeline where each agent has a clear responsibility:

Architect — analyzes the request and produces a technical specification (spec.md)
Designer — generates SVG assets from the spec
Developer — implements the code following the spec and integrating the assets
Tester — writes and runs tests, then sends failures back to the Developer

The idea is the Agent as Tool pattern: each agent is a node in an execution graph, not an LLM calling other LLMs chaotically.

Architecture: why LangGraph over an LLM orchestrator

My first approach was letting an orchestrator agent (Gemini) dynamically decide which sub-agent to call, via function calls. It worked, but I quickly identified a problem: the more generic the system, the more unpredictable it became.

The LLM orchestrator could decide to skip the Designer, call the Tester before the Developer, or loop indefinitely. For a framework that needs to produce reliable code, that's a deal-breaker.

So I chose to delegate orchestration to LangGraph, a deterministic graph framework. The pipeline becomes explicit:

Architect → Designer → Developer → Tester
                                      │
                                      ▼ (tests fail?)
                                   Developer ← fix loop (max 3×)

Each node is an autonomous agent, but execution order and retry logic are deterministic. The LLM controls the what (generated content), but not the when (execution flow).

_builder = StateGraph(PipelineState)
_builder.add_edge(START, "architect")
_builder.add_edge("architect", "designer")
_builder.add_edge("designer", "developer")
_builder.add_edge("developer", "tester")
_builder.add_conditional_edges(
    "tester",
    should_fix_or_end,
    {"fix": "fix_developer", "end": END},
)
_builder.add_edge("fix_developer", "tester")

The should_fix_or_end function is pure Python: it parses the Tester's output and decides whether to rerun the Developer or finish. No LLM in the decision loop.

The prompt caching problem and the switch to full Gemini

During the exploration phase, I very quickly hit API rate limits on Gemini. Every agent call sent the full system prompt, tool definitions, project context, thousands of tokens per request.

The solution: prompt caching. But Gemini and Claude handle it very differently.

Gemini: implicit caching

Gemini automatically caches repeated prefixes. If the system prompt and initial instructions are identical between two calls, Google reuses the cached context. On the code side, there's nothing to do: caching is transparent.

# Savings show up in usage metadata
cached = getattr(meta, "cached_content_token_count", 0)
total = getattr(meta, "prompt_token_count", 0)
logger.info("cache hit: %d/%d tokens (%d%%)", cached, total, cached * 100 // total)

Claude: explicit caching

Claude requires explicit cache_control: ephemeral markers on the blocks you want cached: the system prompt, tool definitions, and the first user message.

system = [{
    "type": "text",
    "text": self.instructions,
    "cache_control": {"type": "ephemeral"}
}]

claude_tools = [self._fn_to_claude_tool(fn) for fn in self.tools]
if claude_tools:
    claude_tools[-1]["cache_control"] = {"type": "ephemeral"}

Why I switched to full Gemini

I started with a multi-LLM architecture: Gemini for the Architect and Tester, Claude for the Developer. The idea was appealing: use each LLM where it excels.

In practice, Claude's API cost quickly made this approach unsustainable. A full pipeline run with Claude as Developer cost significantly more than with Gemini, especially during fix iterations where the context grows with each turn. So I decided to switch to full Gemini as the default pipeline, while keeping the ClaudeAgent in the framework as a configurable option.

This pragmatic choice also let me fully benefit from Gemini's implicit caching across the entire pipeline, without managing two different caching strategies in production.

The contrast between both approaches still pushed me to design the class hierarchy to isolate these differences:

BaseAgent (ABC)
├── GeminiAgent    → implicit caching, google-genai SDK
│   ├── ArchitectAgent
│   ├── DesignerAgent
│   ├── DeveloperAgent
│   └── TesterAgent
└── ClaudeAgent    → explicit caching, anthropic SDK
    └── DeveloperAgent

Each agent inherits its backend's caching strategy without having to worry about it.

Agent hierarchy: ABC and specialization

The core of the framework relies on a simple hierarchy:

BaseAgent (ABC) — defines the contract: run(context) → AgentResult, tool management
GeminiAgent — implements the agentic loop for Gemini (chat + tool calls)
ClaudeAgent — implements the agentic loop for Claude (messages + tool_use blocks)

Specialized agents (Architect, Developer, Tester) inherit from GeminiAgent and only define their instructions and tools:

class ArchitectAgent(GeminiAgent):
    def __init__(self):
        super().__init__(
            name="Architect",
            instructions="You are a software architect...",
            tools=[web_search, write_file],
            model_name="gemini-3.1-pro-preview",
        )

To add a new agent, just create a class, define its instructions, and add it as a node in the LangGraph pipeline. No need to touch the chat logic, tool calling, or caching.

The Designer: a special case

The DesignerAgent is an interesting case. Unlike other agents that use the standard agentic loop (chat → tool call → response → tool call → ...), the Designer makes direct API calls to generate SVG.

Why? Because SVG generation is a well-defined two-step workflow:

Planning — "what assets does this project need?" → returns JSON
Generation — "generate these N SVG sprites" → returns parsable text

No need for an agentic loop with tools here. The Designer still inherits from GeminiAgent (for the API client and key validation), but it overrides run() with its own logic.

The automatic fix loop

One of the most useful aspects of the pipeline is the fix loop. When the Tester detects failures, the Developer is relaunched in FIX MODE:

def should_fix_or_end(state: PipelineState) -> Literal["fix", "end"]:
    if (
        _has_test_failures(state.get("test_results", ""))
        and state.get("fix_iterations", 0) < MAX_FIX_ITERATIONS
    ):
        return "fix"
    return "end"

The Developer then receives the test output in its context, with a clear instruction:

"You are in FIX MODE — read existing files and fix these. Do NOT rewrite all files from scratch."

In practice, 3 iterations are enough in most cases to go from 60-70% passing tests to 100%.

Shared tools

Agents interact with the file system through 4 simple tools:

Tool	Role
`write_file(path, content)`	Write a file (creates parent directories)
`read_file(path)`	Read an existing file
`execute_code(command)`	Execute a shell command
`web_search(query)`	Web search via DuckDuckGo

These tools are plain Python functions, passed to agents through their constructor. The framework handles exposing them to the LLM in the right format (Gemini function declarations or Claude tool definitions).

The limits: a solid foundation, not a finished product

Let's be honest about what the framework can and can't do. AgenticDev excels at generating a functional project base: file structure, initial code, tests, documentation. For simple projects (CLI tools, libraries, small APIs), the output is often usable as-is.

But as complexity grows (intricate business logic, multiple integrations, performance constraints), the generated code will be a starting point, not the final product. There will be technical limitations (overly naive architectures, uncovered edge cases) and functional gaps (the LLM doesn't know your business context) that you'll need to fix manually or by vibe-coding with a tool like Claude Code or Cursor.

This is actually the workflow I recommend: let AgenticDev generate the skeleton, then iterate on it with a coding assistant to refine the details. The framework saves you the first hours of setup, not the last hours of polish.

What I learned

Specialization beats generality

An agent that "does everything" is less reliable than a team of specialized agents. The Architect can't code, the Developer can't test, and that's by design. Each agent has precise instructions and a limited scope.

Deterministic orchestration is non-negotiable

Letting an LLM decide the execution flow means accepting that the pipeline behaves differently on every run. For a code generation tool, that's unacceptable. LangGraph let me keep the LLMs' creativity while enforcing a predictable execution order.

Prompt caching is essential in multi-agent systems

Without caching, a 4-agent pipeline easily consumes 100k+ tokens per run, 80% of which is repeated context. Caching significantly reduces both costs and latency.

Cost dictates architecture

Starting with multi-LLM was intellectually satisfying, but economic reality caught up. Keeping the multi-backend abstraction while using a single provider by default is the right trade-off: you only pay for what you use, without sacrificing flexibility.

Agent instructions are code

Agent prompts aren't vague sentences: they're precise specifications with rules, examples, and edge cases. For instance, the Developer's prompt includes rules on Python vs TypeScript conventions, placeholder handling, and a mandatory completion audit before returning its response.

Going further

The source code is available on GitHub. The framework is designed to be extended: adding a new agent takes about ten lines of code.

You can also read this and other articles on my portfolio.

Next steps I'm considering:

Support for new LLM backends (Mistral, Llama)
Quality metrics on generated code
Interactive mode with human validation between each step