Winning bids in real-time: Building high-throughput AdTech RTB pipelines with Redpanda, Iceberg, and AI

Aykut Bulgu — Mon, 20 Jul 2026 21:19:41 +0000

In programmatic advertising, every millisecond counts. Real-time bidding (RTB) systems have to compete in auctions and return a bid in sub-100ms windows to win impressions, while simultaneously capturing clickstream events like views and device/campaign metadata to power audience segmentation, personalization, and model retraining. I wanted to figure out how to do both at scale, without latency spikes.

To do that, I built an RTB-style pipeline where clickstream events are ingested with Redpanda Connect, written to a data lake as Apache Iceberg tables using Redpanda Iceberg Topics, and then used to train and serve a small AI scoring model that feeds predictions back into a bidder loop.

My goal wasn't to fully implement OpenRTB end-to-end, but to build a high-throughput, low-latency pipeline skeleton that could grow into a production RTB architecture. Here's how I put it together, so you can follow along and build your own.

RTB bidding optimization

Imagine an RTB pipeline with three recurring problems:

Latency spikes during traffic bursts (you miss auctions when you exceed the response window).
High infrastructure cost from overprovisioning just to survive peak load.
Slow model refresh because historical data lives in silos and is painful to query and retrain from.

The goal is to improve the system so it can:

Ingest clickstream events (views/clicks/device/campaign) in real time.
Store the events cost-effectively as Iceberg tables for analytics and model training.
Use AI to predict impression value (CTR proxy/value score).
Stream predictions to the bidder so it can make faster and smarter decisions.

Here's the rough architecture:

To revamp the system, you'll implement the following components:

Redpanda Connect handles HTTP ingestion of bid requests and acts as the "glue", consuming requests and producing predictions.
Redpanda Iceberg Topics streams data into Iceberg tables in object storage. Redpanda writes the data in Iceberg-compatible format, making it queryable with Iceberg clients.
Spark/Jupyter (from the lab) queries Iceberg tables and produces a dataset for training.
A tiny inference service scores bid requests and publishes bid_predictions.
A simple bidder loop consumes predictions and decides how aggressively to bid.

Prerequisites

To keep this tutorial simple and repeatable, you'll use Redpanda’s official Iceberg Docker Compose lab as the base environment. It includes Redpanda, MinIO (S3-compatible storage), and Spark + Jupyter configured to query Iceberg tables.

You'll also need:

Docker and Docker Compose
rpk CLI
Python 3.11 or newer (for a tiny inference service and generators)

Bringing up the Iceberg Playground (Redpanda, MinIO, Spark/Jupyter)

To start, clone the labs repo and enter the project directory:

git clone https://github.com/redpanda-data/redpanda-labs.git && cd redpanda-labs/docker-compose/iceberg

This lab expects Redpanda >= 24.3.1. Define the required Redpanda versions:

export REDPANDA_VERSION=v25.3.2
export REDPANDA_CONSOLE_VERSION=v3.3.2

Start the environment. This provides a complete pipeline from producing data to querying Iceberg tables in Spark/Jupyter:

docker compose build && docker compose up

Next, create an rpk profile for the lab:

rpk profile create docker-compose-iceberg \
  --set=admin_api.addresses=localhost:19644 \
  --set=brokers=localhost:19092 \
  --set=schema_registry.addresses=localhost:18081

Creating the RTB topics

In this section, you'll create three topics:

clickstream (Iceberg-enabled)
bid_requests (messages to score)
bid_predictions (scored output)

For clickstream, start with key_value mode because it’s the easiest path for HTTP ingestion (you can POST JSON and treat it as the "value").

Run the following set of commands to create the topics:

rpk topic create clickstream --topic-config=redpanda.iceberg.mode=key_value ;
rpk topic create bid_requests ;
rpk topic create bid_predictions ;

Iceberg Topics are controlled via redpanda.iceberg.mode. Once you produce to an Iceberg-enabled topic, the data becomes available in object storage for Iceberg clients to consume.

Running clickstream ingestion with Redpanda Connect

In AdTech, clickstream often arrives via HTTP collectors. Redpanda Connect has an http_server input for receiving POSTed events.

Select a working directory on your local computer, create a YAML file called connect-clickstream.yaml, then paste the following content:

input:
  http_server:
    address: 0.0.0.0:4196
    path: /events
    allowed_verbs: [ POST ]

pipeline:
  processors:
    # Keep it simple: accept JSON and pass through.
    - mapping: |
        root = this

output:
  kafka_franz:
    seed_brokers: [ "redpanda:9092" ]
    topic: "clickstream"

kafka_franz is a Kafka output in Redpanda Connect.

Note: In Redpanda Cloud documentation, kafka_franz is marked deprecated in favor of unified Redpanda components. If your target environment is Cloud, consider switching outputs accordingly.

Run Redpanda Connect as a container on the same Docker network as the lab (you may need to adjust the network name depending on your Docker Compose project name):

docker run --rm -it \
  --network redpanda-labs_default \
  -p 4196:4196 \
  -v "$PWD/connect-clickstream.yaml:/connect.yaml:ro" \
  docker.redpanda.com/redpandadata/connect:latest \
  run /connect.yaml

Then, send a couple of sample clickstream events:

curl -X POST http://localhost:4196/events \
  -H 'Content-Type: application/json' \
  -d '{"user_id":101,"ad_id":55,"campaign_id":9,"event_type":"click","clicked":1,"ts":"2025-12-17T09:00:00Z"}'

curl -X POST http://localhost:4196/events \
  -H 'Content-Type: application/json' \
  -d '{"user_id":102,"ad_id":55,"campaign_id":9,"event_type":"view","clicked":0,"ts":"2025-12-17T09:00:01Z"}'

To verify the generated clickstream data, consume the clickstream data with the following command in a new terminal window:

rpk topic consume clickstream

The output should look like this:

{
  "topic": "clickstream",
  "value": "{\"user_id\":101,\"ad_id\":55,\"campaign_id\":9,\"event_type\":\"click\",\"clicked\":1,\"ts\":\"2025-12-17T09:00:00Z\"}",
  "timestamp": ...,
  "partition": 0,
  "offset": 0
}
{
  "topic": "clickstream",
  "value": "{\"user_id\":102,\"ad_id\":55,\"campaign_id\":9,\"event_type\":\"view\",\"clicked\":0,\"ts\":\"2025-12-17T09:00:01Z\"}",
  "timestamp": ...,
  "partition": 0,
  "offset": 0
}
...output omitted...

Verifying the Iceberg Writes

On your browser, open http://localhost:9001/browser and enter the credentials minio / minio123:

Open another tab and navigate to http://localhost:8888, which is the page for Jupyter Labs.

The lab notebook should guide you through querying Iceberg tables created from Redpanda topics.

You can also do a quick sanity check with Spark SQL. Run the following command to start spark-iceberg and spark-sql:

docker exec -it spark-iceberg spark-sql

On the open interface, you can query the Iceberg table.

Note: the exact catalog/table naming is shown in the lab notebook; the lab demonstrates querying lab.redpanda.<topic>.

Building a "bid value" model from Iceberg data

To create a small “bid value” model, you need to first prepare the data by parsing it from JSON and then save it as a parquet file on the disk of the Jupyter Labs container. Then you can train the model using the saved training dataset.

Preparing the data

At this point, you’ve validated the base path: HTTP -> Redpanda -> Iceberg. The next challenge is to train a model on fresh behavioral data and turn it into a low-latency scoring service.

Because you created the Iceberg topic in key_value mode, the Iceberg table stores your payload in a binary value column (plus a redpanda struct that holds record metadata).

That means the training workflow has two phases:

Extract: Cast/decode value and parse your JSON into typed columns
Train: Fit a lightweight model and save an artifact you can load in an API

Go back to Jupyter in your browser and create a new Python notebook. In the first cell, load the Iceberg table. In this lab, tables are queryable as lab.redpanda.<topic> (the lab explicitly shows querying lab.redpanda.value_schema_id_prefix, so your clickstream topic will be lab.redpanda.clickstream).

Enter the following script in the first row and run it:

clickstream_raw = spark.table("lab.redpanda.clickstream")
clickstream_raw.printSchema()

You should see a schema that includes:

A redpanda struct column (metadata)
A value column (binary payload)

To look at the raw payload, copy and paste the following script content in the second row of the notebook:

from pyspark.sql.functions import col

(clickstream_raw
  .select(col("redpanda.timestamp").alias("rp_timestamp"),
          col("value").cast("string").alias("json_value"))
  .show(5, truncate=False)
)

If your events were JSON, you should see JSON strings in json_value. If you see null or an error, it usually means one of two things:

You didn't actually publish JSON (unlikely in your curl case).
The bytes aren't UTF-8, so the cast doesn't decode cleanly.

Now you have to paste the JSON payload into typed columns. To do that, you need to define the schema you expect from your clickstream collector. In your Jupyter notebook, add the following and run it:

from pyspark.sql.functions import from_json
from pyspark.sql.types import StructType, StructField, IntegerType, StringType

payload_schema = StructType([
    StructField("user_id", IntegerType(), True),
    StructField("ad_id", IntegerType(), True),
    StructField("campaign_id", IntegerType(), True),
    StructField("event_type", StringType(), True),
    StructField("clicked", IntegerType(), True),
    StructField("ts", StringType(), True),
])

To parse the value, add the following script and run it on your notebook:

parsed = (clickstream_raw
    .select(from_json(col("value").cast("string"), payload_schema).alias("e"))
    .select("e.*")
    .dropna(subset=["user_id", "ad_id", "campaign_id", "clicked"])
)

parsed.show(5, truncate=False)

You should be looking for clean columns like this:

user_id, ad_id, campaign_id as integers
event_type as a string
clicked as 0/1

If the clicked event is sometimes missing, you can default it (for demo) to 0:

from pyspark.sql.functions import when, lit

parsed = parsed.withColumn("clicked", when(col("clicked").isNull(), lit(0)).otherwise(col("clicked")))

In this tutorial, you won't train directly over millions of rows. You should take a sample so the notebook stays fast and repeatable:

training_df = parsed.limit(100_000)

training_path = "/home/jovyan/work/training_clickstream.parquet"
training_df.write.mode("overwrite").parquet(training_path)

training_path

At this point, you have a stable dataset on disk inside the Jupyter container.

Training the Model

To train the model, you'll use a logistic regression pipeline with one-hot encoding for event_type. This is intentionally small and "demo-grade," but it's enough to prove the pipeline.

In the same Jupyter notebook, copy and paste the following content into the next available row:

# If needed in the notebook environment:
# %pip install -q scikit-learn joblib pandas pyarrow

import pandas as pd

train_pd = pd.read_parquet(training_path)

X = train_pd[["ad_id", "campaign_id", "event_type"]]
y = train_pd["clicked"].astype(int)

from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
import joblib

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

pre = ColumnTransformer(
    transformers=[
        ("cat", OneHotEncoder(handle_unknown="ignore"), ["event_type"]),
        ("num", "passthrough", ["ad_id", "campaign_id"]),
    ]
)

model = Pipeline(steps=[
    ("pre", pre),
    ("clf", LogisticRegression(max_iter=200))
])

model.fit(X_train, y_train)

probs = model.predict_proba(X_test)[:, 1]
print("AUC:", roc_auc_score(y_test, probs))

Then save the artifact:

artifact_path = "/home/jovyan/work/bid_value_model.joblib"
joblib.dump(model, artifact_path)
artifact_path

Now you have a model artifact that's easy to ship into a tiny inference service.

Serving the model as an inference endpoint

Now that you have a trained model artifact (bid_value_model.joblib), you need to make it usable by your online RTB path. To do so, you'll copy the model artifact out of the Jupyter container, then run the pre-created inference service.

Copy the model artifact out of the Jupyter container

You should have saved the artifact inside the Jupyter/Spark container at: /home/jovyan/work/bid_value_model.joblib.

On your host machine, open a terminal in the directory where you want to keep your inference service files and copy the model out:

docker cp spark-iceberg:/home/jovyan/work/bid_value_model.joblib ./bid_value_model.joblib

Do a quick sanity check:

ls -lh ./bid_value_model.joblib

You should see a non-zero file size (usually KBs–MBs depending on the pipeline).

WARNING: If the container name isn't spark-iceberg, run docker ps and replace it with whatever your lab uses.

Run the inference service

Run the following command on your terminal to clone the inference service demo, which uses FastAPI:

git clone https://github.com/SystemCraftsman/redpanda-rtb-iceberg-demo

Navigate into the cloned repository directory and create a Python virtual environment and activate it:

python3 -m venv .venv ;
source .venv/bin/activate

Then, install the demo project requirements by executing the following command:

pip install -r requirements.txt

Since you have all the required dependencies installed now, you can run the service:

uvicorn inference_service:app --host 0.0.0.0 --port 8000

Verifying the inference service

In a new terminal, run the following command:

curl -i http://localhost:8000/health

Your output should look like this:

{"status":"ok"}

Test the scoring with the following command:

curl -s -X POST http://localhost:8000/score \
  -H 'Content-Type: application/json' \
  -d '{"ad_id":55,"campaign_id":9,"event_type":"click"}'

Your output should look like this:

{"p_click":0.37}

If you get a 500 error, it usually means one of these things:

The model file wasn’t found (bid_value_model.joblib in the same directory, or wrong path)
A dependency mismatch (e.g., scikit-learn version issues)
Your request fields don’t match the feature names/types the pipeline expects

Scoring bid requests in-stream

In a terminal on your host, produce a couple of requests:

echo '{"request_id":"req-1","ad_id":55,"campaign_id":9,"event_type":"view"}' \
  | rpk topic produce bid_requests --format '%v\n'

echo '{"request_id":"req-2","ad_id":55,"campaign_id":9,"event_type":"click"}' \
  | rpk topic produce bid_requests --format '%v\n'

You need to consume these requests via a Redpanda Connect connector. Create a YAML file called connect-score-bids.yaml and paste the following data to configure this connector:

input:
  kafka_franz:
    seed_brokers: [ "redpanda:9092" ]
    topics: [ "bid_requests" ]
    consumer_group: "bid-scorer"

pipeline:
  processors:
    # Store the original request so we can merge it back after the HTTP call.
    - mapping: |
        meta original_request = content()

    # Call the inference API.
    # Important: because this Connect pipeline runs in Docker, "localhost" would refer to the container.
    # On macOS/Windows, use host.docker.internal to reach your host machine.
    - http:
        url: "http://host.docker.internal:8000/score"
        verb: POST
        headers:
          Content-Type: application/json

    # Merge original request + score response into one enriched event
    - mapping: |
        let req = meta("original_request").parse_json()
        let resp = this.parse_json()
        root = req
        root.p_click = resp.p_click

output:
  kafka_franz:
    seed_brokers: [ "redpanda:9092" ]
    topic: "bid_predictions"
    key: ${! json("request_id") }

Run Redpanda Connect on the same Docker network as the lab:

docker run --rm -it \
  --network redpanda-labs_default \
  -v "$PWD/connect-score-bids.yaml:/connect.yaml:ro" \
  docker.redpanda.com/redpandadata/connect:latest \
  run /connect.yaml

Leave this terminal open and the container running as it's your scoring pipeline.

To verify the bid predictions are being produced, open a new terminal and consume bid_predictions topic:

rpk topic consume bid_predictions -n 2 -f '%k %v\n'

You should see keys and enriched values similar to this:

req-1 {"request_id":"req-1","ad_id":55,"campaign_id":9,"event_type":"view","p_click":0.12}
req-2 {"request_id":"req-2","ad_id":55,"campaign_id":9,"event_type":"click","p_click":0.37}

Alternatively, you can use a bidder script written in Python to consume the data. Here's an example bidder.py:

import json
from kafka import KafkaConsumer

BROKER = "localhost:19092"
TOPIC = "bid_predictions"

consumer = KafkaConsumer(
    TOPIC,
    bootstrap_servers=[BROKER],
    auto_offset_reset="earliest",
    enable_auto_commit=True,
    value_deserializer=lambda v: json.loads(v.decode("utf-8")),
)

THRESHOLD = 0.30

print("Bidder is listening for predictions...")

for msg in consumer:
    pred = msg.value
    req_id = pred.get("request_id", "unknown")
    p = float(pred.get("p_click", 0.0))

    if p >= THRESHOLD:
        decision = "BID_AGGRESSIVE"
        bid_multiplier = 1.5
    else:
        decision = "BID_CONSERVATIVE"
        bid_multiplier = 0.5

    print(f"[{req_id}] p_click={p:.3f} => {decision} (multiplier={bid_multiplier})")

After you run bidder.py, you have a tiny “decision loop” sitting on top of your streaming pipeline. The script continuously consumes messages from the bid_predictions topic, extracts the model’s p_click score, and applies a simple threshold rule to choose a bid strategy. If p_click is above 0.30, it prints an aggressive decision as higher bid multiplier; otherwise it prints a conservative decision as lower multiplier. In a real RTB bidder, you’d plug in your actual pricing logic (budget constraints, floor prices, pacing, frequency caps), but the core idea is the same: stream in predictions, make a decision immediately, and respond fast enough to win auctions.

Conclusion

In this tutorial, I showed you how to build an RTB-style pipeline that separates high-throughput learning and low-latency decisioning. You've created a streaming backbone with Redpanda that can absorb traffic spikes, a lakehouse layer that keeps storage and historical analysis cost-effective, and an online scoring loop that can evolve independently as your features and models improve. It's not a production RTB architecture, but it's a skeleton you could grow into one.

You can find all the relevant code for the demo on Github.

DEV Community: Aykut Bulgu