📅 November 5, 2026 • databricks.com
Apache Spark 4.3 has been released and is available now, both as open source and inside Databricks Runtime 19. Per Databricks, the headline change is stateful processing in Real-Time Mode: streaming aggregation, deduplication through dropDuplicates and the Scala and Java transformWithState API now run in the Structured Streaming low-latency mode with sub-150 ms P99 end-to-end latency. The figure comes from three reference benchmarks published by the vendor, so it is a measured result on those workloads rather than a universal guarantee.
Stateful operators were previously the main reason teams kept a separate low-latency engine next to Spark for millisecond-class workloads. Real-Time Mode now carries state across the streaming timeline, so aggregation and deduplication behave consistently from batch-scale analytics down to per-event latency targets. That covers patterns such as fraud detection scoring, deduplication of event feeds and live monitoring of device fleets without moving the stateful part of the pipeline to another system. The operators involved are exactly the ones that hold running state across events, which is what previously forced teams to re-implement deduplication and aggregation logic outside the engine or reach for a second, latency-focused system.
The release also open sources SCD Type 2 Auto CDC, the change-data-capture pattern that preserves full record history through START AT and END AT marker columns instead of overwriting rows. It is callable from Python, SQL and Spark Connect, so analysts can build slowly changing dimensions without leaving the engine, and every recorded state change stays queryable afterwards. The marker-column layout keeps every historical version of a row addressable, which is exactly what slowly changing dimension jobs require.
Performance work arrives under the name Project Feather: shuffle-free single-task execution for eligible small single-partition query fragments, plus Arrow-native caching that keeps hot data in columnar format close to the executor. On the language surface, Spark 4.3 adds VARIANT transformations, the JSON TABLE function, temporal joins and an opt-in preview of nanosecond-precision timestamps.
The release aggregates work from more than 200 contributors across more than 1,100 resolved Jira tickets. Per Databricks, version 4.3 also starts a quarterly release cadence, replacing the previous six-month cycle and putting new versions on a predictable quarterly calendar. For teams planning upgrades, that moves feature adoption from a twice-a-year to a quarterly rhythm.
IMPACT: One open engine now covers batch processing, analytics, AI workloads and millisecond-scale streaming, so supported workloads can drop a separate streaming engine and the double operational burden that comes with it. The natural first candidates are fraud detection, deduplication feeds and live device monitoring, where the data path already runs on Spark.
Sources: databricks.com
Confirmations could not be found at this time.
Sources:
Continue the conversation: https://t.me/AIAutomationArchitecture · Bluesky
Top comments (0)