Tiered Storage in Apache Kafka — Hot Data on Disk, Old Data in S3 (and the Same Trick That Cuts a Databricks Bill)

Every Kafka cluster I've worked on had the same problem. Not throughput. Disk.

The business wants a year of history kept. The brokers can hold maybe a week. So retention gets cut, data gets deleted, and three months later somebody asks for it.

Tiered storage fixed this in open-source Kafka, and the idea is honestly simple.

Recent data stays on the broker's local disk. That's the hot data, the stuff consumers are actually reading all day. When a segment is done and gets old, Kafka ships it to S3 and removes the local copy.

Nobody notices. Same topic, same offsets. A consumer can still read six-month-old data — the broker just pulls it back from S3 when asked. It's slower, sure. Old data always is. But it works, and we didn't have to build some separate archiving pipeline on the side to make it happen.

On EKS this setup is almost unfair. Tiny fast disks on the brokers, S3 doing the heavy lifting behind them. Retention stops being a math problem you solve with a spreadsheet and a prayer.

On-prem works too. The storage layer is pluggable, so MinIO, Ceph, whatever S3-compatible box is already sitting in the data center — it all works the same way.

And the same thinking saves money outside Kafka. Databricks is where I've seen storage bills quietly explode. Delta keeps old versions of files around for time travel, old partitions sit on full-price storage, and nobody ever goes back to clean it up.

The workaround is the same idea. Leave the recent data in Delta where it's fast. Take the old partitions, write them out once as plain Parquet into an archive bucket, and let an S3 lifecycle rule push them down to Glacier pricing. Drop them from the table. If someone actually needs last year's data — and that happens maybe twice a year — read the Parquet directly. Ten minutes of work beats paying hot-storage prices forever.

One honest warning. Tiered storage is not a backup. Delete the topic and the S3 data goes with it. It solves cost. It doesn't solve accidents.

Hot data where it's read. Cold data where it's cheap. That's the whole trick.

---

The system design

Write path. Producers write to the leaders as always. Nothing about the hot path changes — records land in the active log segment on local disk and are acknowledged per your normal acks and replication settings.

Offload path. When a segment closes and ages past the local retention window, the broker uploads it to remote storage through the Remote Storage Manager (RSM), records where it went in the Remote Log Metadata Manager (RLMM — an internal Kafka topic in the production setup), and only then deletes the local copy. Uploads are chunked (4 MiB here) so a multi-GB segment is many small, resumable S3 writes, not one giant one.

Read path. A fetch for a recent offset is served from local disk, exactly as before. A fetch for an old offset is resolved through the RLMM to the segment's S3 location; the broker streams the needed chunks back (through a local chunk cache) and serves them as a normal fetch response. The consumer cannot tell the difference — only the latency changes.

The two retention knobs are the whole design:

Setting	Meaning
local.retention.ms / local.retention.bytes	How long data stays on broker disk — the hot window
retention.ms / retention.bytes	Total history kept, local + remote together

On EKS the brokers run as a StatefulSet with small gp3 volumes, credentials come from IRSA (no keys in the pod), and the S3 plugin arrives via an init container. On-prem, point the same RSM at MinIO or Ceph — the design doesn't change, only the endpoint.

---

The configs

1. Broker config — server.properties

Apache Kafka ships the tiered-storage framework (KIP-405) but no S3 implementation, so the Remote Storage Manager is Aiven's open-source plugin — the standard choice.

# Turn the tiered-storage system on for this broker
remote.log.storage.system.enable=true

# Remote Log Metadata Manager: topic-based (production choice)
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.impl.prefix=rlmm.config.
rlmm.config.remote.log.metadata.topic.replication.factor=3
rlmm.config.remote.log.metadata.topic.num.partitions=50

# Remote Storage Manager: Aiven plugin, S3 backend
remote.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.storage.manager.impl.prefix=rsm.config.
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3StorageBackend
rsm.config.storage.s3.bucket.name=my-kafka-tiered-storage
rsm.config.storage.s3.region=us-east-1
# Credentials come from the environment (on EKS: the pod's IRSA role).
# Never put keys in this file.

# 4 MiB chunks keep S3 requests sane on big segments
rsm.config.chunk.size=4194304

2. Per-topic opt-in — topic-config.sh

kafka-configs.sh --bootstrap-server kafka-0.kafka-headless:9092,kafka-1.kafka-headless:9092,kafka-2.kafka-headless:9092 \
  --entity-type topics --entity-name events --alter \
  --add-config remote.storage.enable=true,\
local.retention.ms=86400000,\
retention.ms=15552000000
# local.retention.ms = 1 day on disk (hot window)
# retention.ms       = 180 days total history

3. The pod spec — kafka-on-eks.yaml

The number to look at: 50 Gi of disk. Without tiered storage this same topic needed terabytes per broker.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kafka
  namespace: kafka
  annotations:
    # IRSA: this role carries the S3 permissions (see iam-policy.json).
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/kafka-tiered-storage
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-config
  namespace: kafka
data:
  server.properties: |
    process.roles=broker,controller
    node.id=0
    controller.quorum.voters=[e-mail]:9093,[e-mail]:9093,[e-mail]:9093
    listeners=PLAINTEXT://:9092,CONTROLLER://:9093
    advertised.listeners=PLAINTEXT://kafka-0.kafka-headless.kafka.svc:9092
    log.dirs=/var/lib/kafka/data
    num.partitions=12
    remote.log.storage.system.enable=true
    remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
    remote.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
    remote.storage.manager.impl.prefix=rsm.config.
    rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3StorageBackend
    rsm.config.storage.s3.bucket.name=my-kafka-tiered-storage
    rsm.config.storage.s3.region=us-east-1
    rsm.config.chunk.size=4194304
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels: { app: kafka }
  template:
    metadata:
      labels: { app: kafka }
    spec:
      serviceAccountName: kafka
      initContainers:
        # Kafka ships no S3 RSM — fetch Aiven's open-source plugin jars.
        - name: fetch-tiered-storage-plugin
          image: curlimages/curl:8.10.1
          command: ["sh", "-c", "cd /plugin && curl -sSLO https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/v0.10.0/tiered-storage-for-apache-kafka-0.10.0.tgz && tar xzf tiered-storage-for-apache-kafka-0.10.0.tgz --strip-components=1"]
          volumeMounts:
            - { name: plugin, mountPath: /plugin }
      containers:
        - name: kafka
          image: apache/kafka:3.9.0
          ports:
            - { containerPort: 9092, name: client }
            - { containerPort: 9093, name: controller }
          volumeMounts:
            - { name: data, mountPath: /var/lib/kafka/data }
            - { name: config, mountPath: /opt/kafka/config/server.properties, subPath: server.properties }
            - { name: plugin, mountPath: /opt/kafka/libs/tiered-storage }
          resources:
            requests: { cpu: "2", memory: 4Gi }
            limits:   { cpu: "4", memory: 8Gi }
      volumes:
        - name: config
          configMap: { name: kafka-config }
        - name: plugin
          emptyDir: {}
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources:
          requests:
            storage: 50Gi   # plenty, when disk only holds the hot window

4. The only S3 permissions the brokers need — iam-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "ListTieredStorageBucket", "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::my-kafka-tiered-storage" },
    { "Sid": "ReadWriteSegments", "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-kafka-tiered-storage/*" }
  ]
}

---

The Databricks half: storage workaround + EKS compute split

Storage. The archive job below runs weekly: old Delta partitions are written once as plain Parquet into an archive bucket, deleted from the table, and VACUUMed so the space is genuinely released. The bucket's lifecycle rule then does the pricing work.

# databricks_archive_job.py (essentials)
from pyspark.sql import functions as F

HOT_WINDOW_DAYS = 180
TABLE = "prod.events"
ARCHIVE = "s3://my-archive-bucket/archive/events"
DATE_COL = "event_date"

cutoff = F.date_sub(F.current_date(), HOT_WINDOW_DAYS)
old = spark.read.table(TABLE).filter(F.col(DATE_COL) < cutoff)

# Land the copy BEFORE touching the table
old.write.mode("overwrite").partitionBy(DATE_COL).parquet(ARCHIVE)

spark.sql(f"DELETE FROM {TABLE} WHERE {DATE_COL} < date_sub(current_date(), {HOT_WINDOW_DAYS})")
spark.sql(f"VACUUM {TABLE} RETAIN 168 HOURS")   # 7 days of time-travel grace

# Reading it back, the rare time anyone asks:
#   spark.read.parquet("s3://my-archive-bucket/archive/events")

// s3-lifecycle-archive.json
{ "Rules": [ { "ID": "archive-to-glacier", "Status": "Enabled",
  "Filter": { "Prefix": "archive/" },
  "Transitions": [
    { "Days": 30,  "StorageClass": "GLACIER_IR" },
    { "Days": 180, "StorageClass": "DEEP_ARCHIVE" } ] } ] }

Compute. The archive reads, backfills, and compaction don't need Databricks at all. You can't run Databricks on EKS — its compute is Databricks-managed — but you don't need to: Delta Lake is open source, and the same tables and Parquet archives on S3 are readable and writable by plain Apache Spark running as EKS pods, at raw EC2 prices, with zero DBUs. Keep Databricks for the interactive work where its speed earns the markup; let cheap pods do the scheduled heavy lifting against the cold tiers, scaling to zero in between.

---

The evidence

What the storage tiers actually cost (AWS list prices, us-east-1):

Tier	$ / GB-month	20 TB of history
S3 Standard (and broker-disk equivalent thinking)	$0.023	~$460 / month
S3 Glacier Instant Retrieval	$0.004	~$80 / month
S3 Glacier Deep Archive	$0.00099	~$20 / month

Worked example — 20 TB total history, 1-day hot window: ~1 TB stays hot (~$23), 19 TB sits in Glacier Instant Retrieval (~$76). ≈ $99/month instead of ≈ $460 — about 78% off the storage line, before counting the broker disks you no longer buy (50 Gi gp3 per broker above, versus terabytes).

What Databricks actually charges: every workload pays twice — the cloud VM cost plus a per-DBU charge (Flexera's 2026 Databricks pricing guide: Jobs compute $0.15/DBU Premium, serverless jobs $0.35/DBU, all-purpose compute $0.40/DBU). All-in, Databricks typically lands at roughly 1.5–2.5× the cost of the raw compute it runs on (Reintech, Apache Spark vs Databricks 2026 comparison). Moving a scheduled batch job to Spark-on-EKS removes the entire DBU half of that bill for that job.

Why the split is legal, technically: Delta Lake is an open-source Linux Foundation project — "not locked to Databricks and can be used anywhere" (Databricks' own Delta Lake overview). External engines — Apache Spark batch and streaming, Flink, DuckDB — can now create, read, and write even Unity Catalog–managed Delta tables with commits coordinated by Unity Catalog (Databricks Community technical blog, 2026).

The honest limits: open-source Delta lacks Databricks-only optimizations (Photon, predictive I/O); Glacier Deep Archive objects must be restored in S3 before they can be read, which is why the lifecycle stops at Instant Retrieval first; self-managed Spark on EKS trades engineer time for the DBU savings; and tiered storage — Kafka or Delta — is a cost tool, not a backup.

---

Sources: Flexera, "Databricks pricing guide (2026)"; Reintech, "Apache Spark vs Databricks 2026"; Dremio, "Keep Databricks for ETL. Cut Analytics Costs"; Databricks Community, "Performing DML Operations on Unity Catalog Managed Tables from External Engines"; Delta Lake documentation (docs.delta.io); AWS S3 pricing (list, us-east-1). Config files: companion bundle kafka-tiered-storage/.

Comments

[Karan Pulyala]

The local.retention vs retention table is the clearest explanation of the two knobs I've seen. One thing to watch in the ConfigMap as shown: node.id=0 and advertised.listeners pointing at kafka-0 are baked into a single server.properties shared by every pod in the StatefulSet, so all three brokers would claim the same ID and advertise the same host. Those usually get templated per pod from the ordinal in an init container or entrypoint. On the Delta side, a small addition to the archive idea: deleting old partitions from the table doesn't reduce the storage bill until VACUUM runs and the deleted-file retention period (7 days by default) has passed, because the files are still kept for time travel. It's also worth stressing that the lifecycle rule should only target the separate archive bucket, since a Delta table whose files move to Glacier will fail on read.