Databricks Real-Time Mode Is GA. Your Pipeline Still Won't Be Real-Time.

On October 29, real-time mode went generally available in Lakeflow pipelines, on classic and serverless compute. The number in the docs is the one everybody quoted: end to end latency as low as five milliseconds.

Five milliseconds, in Spark. That is a genuine piece of engineering, and it closes a gap people have been complaining about ever since Flink started winning the "but we actually need low latency" arguments.

Then I read the rest of the page. Specifically the part that never makes it into a keynote slide: the list of things real-time mode does not support. And that list tells you far more about your architecture than the latency number does.

Here is the short version, and it is the sentence that should make you sit up:

Delta is not supported as a source or as a sink in real-time mode.

Read it again if you run a medallion architecture.

What actually changed under the hood

Micro-batch has always worked the same way. Spark plans a batch, schedules the tasks for stage one, waits for stage one to finish, writes a shuffle, schedules stage two, and so on. Each stage boundary is a barrier. Your floor latency is the time it takes to walk that whole staircase, plus the trigger interval, plus the checkpoint write.

Real-time mode changes the shape of that. It runs long-lived batches, five minutes by default, and inside a batch:

• all stages of the query are scheduled at once, not one after another
• data moves between stages through a streaming shuffle as soon as a stage produces a record, instead of waiting for the stage to complete
• checkpointing and metrics publishing happen between batches, not between micro-batches

So the trigger interval stops being "how often I process" and becomes "how often I checkpoint". Processing is continuous. That is the whole trick, and it is why the latency drops by three orders of magnitude.

It also means the cost model flips. Because every stage is resident, you need enough task slots for all of them simultaneously, for the entire life of the batch. The docs give a clean example: a two stage stateful pipeline with maxPartitions set to 8 and 20 shuffle partitions needs 28 slots. Not 8. Not 20. Twenty eight, held open, whether traffic is flowing or not.

Turning it on

Three things have to line up: the pipeline has to be continuous, the Spark config has to enable real-time mode, and the individual flow has to ask for the RealTime trigger.

{
  "continuous": true,
  "spark_conf": {
    "spark.databricks.streaming.realTimeMode.enabled": "true"
  }
}

from pyspark import pipelines as dp

dp.create_sink("alerts_out", "kafka", {
    "kafka.bootstrap.servers": broker,
    "topic": "fraud-alerts",
})

@dp.update_flow(
    name="fraud_rtm_flow",
    target="alerts_out",
    spark_conf={
        "pipelines.trigger": "RealTime",
        # how often we checkpoint, NOT how often we process
        "pipelines.trigger.interval": "5 minutes",
    },
)
def fraud_rtm_flow():
    return (
        spark.readStream
            .format("kafka")
            .option("kafka.bootstrap.servers", broker)
            .option("subscribe", "card-authorizations")
            .option("startingOffsets", "latest")
            .load()
            .selectExpr("CAST(key AS STRING) AS card_id",
                        "CAST(value AS STRING) AS payload",
                        "timestamp")
    )

Note what the source and the sink are. Kafka in, Kafka out. Event Hubs works too, through the Kafka connector. Kinesis works as a source but not as a sink. Delta works as neither.

That constraint is not an oversight. It is the design. A streaming shuffle that never materializes a stage boundary cannot also commit an ACID table transaction per record without destroying the thing that makes it fast. So real-time mode is a message bus to message bus transform, and your lakehouse sits outside of it.

The unsupported list is the real specification

Read this as a design document rather than a gap analysis:

• update output mode only, append and complete are out
• no session windows, tumbling and sliding only
• stream to table joins must be broadcast, which means the table has to fit in roughly 10 MB
• stream to stream inner joins need Databricks Runtime 18 LTS or later plus extra Spark config, and outer joins are not supported at all
• forEachBatch is out, forEach is the custom sink escape hatch
• mapPartitions, flatMapGroupsWithState and mapGroupsWithState are out
• event time timers are not supported

Every one of those is a direct consequence of removing the batch boundary. forEachBatch needs a batch. Complete mode needs a finished aggregation. An outer join needs to know that the other side definitively did not arrive, which is a decision you can only make at a boundary.

The broadcast-only join limit is the one that will quietly break the most designs. Enriching a payment stream with a customer dimension is the canonical streaming use case, and a real customer dimension is not 10 MB. You will end up pushing that lookup into state, or into the consumer, or accepting a stale copy.

The per row trap in transformWithState

This is the sharpest edge in the release, and it is easy to miss because the code compiles and runs.

In micro-batch, handleInputRows is called once per key with every row for that key in the batch. In real-time mode it is called once per row. Same method signature, completely different contract.

So code like this, which is perfectly correct in micro-batch:

# micro-batch assumption: inputRows holds ALL rows for this key in the batch
def handleInputRows(self, key, inputRows, timerValues):
    rows = list(inputRows)
    total = sum(r.amount for r in rows)
    if total > self.threshold:          # batch level decision
        yield Row(card_id=key, flagged_total=total, n=len(rows))
    self.running.update((total,))

becomes wrong under real-time mode. It still executes, it just sees one row at a time, so total is one amount, len(rows) is always 1, and the threshold logic silently stops working. You have to move the accumulation into state:

# real-time mode: one row per call, so state IS the aggregate
def handleInputRows(self, key, inputRows, timerValues):
    prev = self.running.get()
    total = (prev[0] if prev else 0.0)
    for r in inputRows:               # exactly one row here
        total += r.amount
    self.running.update((total,))
    if total > self.threshold:
        yield Row(card_id=key, flagged_total=total)

A few more things worth knowing before you port anything: transformWithStateInPandas is not supported, so you need the row based API. Processing time timers fire late when data is sparse, because they are driven by arriving records, so a timer set for 10:00:00 on a quiet stream fires when the next record shows up. And if you are running Python UDFs you will see guidance to set spark.sql.execution.arrow.maxRecordsPerBatch to 1, which minimizes latency by trading away throughput. Pandas UDFs behave badly at batch size 1, so prefer Arrow UDFs.

Checkpoints cut both ways

The default five minute batch is a latency to recovery trade, not a latency to throughput trade. Longer batches checkpoint less often, which means a failure replays more work and your metrics arrive later. Shorter batches checkpoint more often, which can start to interfere with the latency you bought the feature for.

The useful part is that the metrics tell you where the time actually goes. processingLatencyMs, sourceQueuingLatencyMs and e2eLatencyMs are published at p50, p90, p95 and p99. Look at sourceQueuingLatencyMs first. If your records are sitting in Kafka waiting for a slot, no amount of streaming shuffle will help you, and the fix is partitions and task slots, not trigger configuration.

The other half of the problem is on the serving side

Here is where the Microsoft side of the stack matters, and where "five milliseconds" meets reality.

A Kafka to Kafka transform in five milliseconds is only useful if something downstream can act in a comparable window. That consumer is usually a dashboard, an alerting rule, or an operational store. On the Fabric side, that is what Real-Time Intelligence has been building out, and FabCon Barcelona pushed a batch of it to GA: Eventhouse accelerated shortcuts over OneLake Delta tables, inline transformations through update policies, and a self-optimizing engine. Business events in Real-Time hub can now trigger Activator, and Reference Data Join, in preview, lets an Eventstream enrich events with Lakehouse tables.

That update policy pattern is the piece I would pair with a real-time flow, because it keeps the shaping work in the serving engine instead of in the low latency path:

.create table authorizations_raw (payload: dynamic)

.create-or-alter function ShapeAuthorizations() {
    authorizations_raw
    | extend card_id   = tostring(payload.card_id),
             amount    = toreal(payload.amount),
             event_ts  = todatetime(payload.ts)
    | project card_id, amount, event_ts
}

.alter table authorizations_shaped policy update

[{
  "IsEnabled": true,
  "Source": "authorizations_raw",
  "Query": "ShapeAuthorizations()",
  "IsTransactional": true,
  "PropagateIngestionProperties": true
}]

Notice the division of labor. Databricks real-time mode decides, in milliseconds, on a bus. Fabric Real-Time Intelligence serves and reacts. Neither one is pretending to be your medallion.

What I would actually do in production

I would not convert an existing pipeline. I would split the decision path out of it.

Keep the medallion exactly where it is, in micro-batch, writing Delta, with forEachBatch and complete mode and all the comfortable things. Then take the one or two decisions that genuinely need sub-second latency, fraud scoring, inventory holds, a time boxed offer, and give each of them its own continuous pipeline with a single real-time flow. The docs recommend one flow per pipeline on classic compute and on the serverless current channel anyway, because extra flows compete for the same task slots.

Then size for the slots, not for the data volume. Count the stages, count the shuffle partitions, add them up, and provision for the sum. A real-time pipeline that is 30 percent underprovisioned does not degrade gracefully into a slower pipeline. It queues at the source, and your p99 goes to seconds while the dashboard still says real-time mode is enabled.

Five milliseconds was never the hard part. The hard part is that sub-second latency is a property of an entire path, and the moment any hop in that path is a table commit, a broadcast that no longer fits, or a consumer polling every thirty seconds, the number you bought is gone.

So the question I keep coming back to: for the workloads where your business actually asked for real time, what is the slowest hop in the path today? My bet is it is not the trigger.

Comments

[Karan Pulyala]

"Delta is not supported as a source or as a sink" is the line everyone's going to skim past, and it's the whole story. Splitting the one or two decisions that really need sub-second latency into their own pipeline, and leaving the medallion alone, feels like the only sane way to adopt this. The transformWithState change from per-key to per-row calls is scary, though: same signature, it still compiles, and the logic is silently wrong. Have you seen any linting or test pattern that catches that before it ships, or is it just "read the docs very carefully"?