Microsoft Fabric Runtime 2.0 has moved from preview to general availability. This is not a minor version bump. Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, and a significantly enhanced Native Execution Engine land together in a single runtime upgrade. This article breaks down what changed, what it means for real Fabric workloads, and what to watch out for before you migrate.
Why This Release Is Different
Most runtime upgrades are incremental. A version bump here, a library update there, a deprecation notice you file away to deal with later.
Runtime 2.0 is not that.
This release represents the biggest component jump in Fabric's Spark history. The move from Runtime 1.3 (Spark 3.5, Delta Lake 3.2, Python 3.11, Java 11) to Runtime 2.0 spans multiple major versions across every core component simultaneously. Spark crosses the 4.x threshold. Delta Lake moves to 4.2. Python jumps to 3.13. Java moves from 11 to 21. Scala shifts from 2.12 to 2.13.
Each of those individually would be a meaningful migration. Together, they require careful evaluation before any production workload is touched.
The good news: Microsoft has been deliberate about the rollout. Runtime 2.0 is GA and production-ready, but it is not the default runtime yet. Existing workspaces stay on Runtime 1.3 until you explicitly opt in. The plan is to make Runtime 2.0 the default for new workspaces and environment items in late September 2026, which gives teams a defined window to validate and migrate at their own pace.
What Is in Runtime 2.0
The full component stack:
Apache Spark: 4.1
Delta Lake: 4.2
Python: 3.13
Java: 21
Scala: 2.13
R: 4.5.2
OS: Azure Linux 3.0 (Mariner 3.0)
That is the foundation. On top of it sit several capability upgrades worth understanding individually.
The Native Execution Engine: The Performance Story
The headline performance claim for Runtime 2.0 is significant: up to six times faster than open-source Spark on TPC-DS benchmarks, driven by the Native Execution Engine (NEE).
The NEE is not a new concept in Fabric — it shipped as part of Runtime 1.3 as well — but Runtime 2.0 brings meaningful enhancements. The engine works by offloading supported Spark operators from JVM-based execution to a vectorized C++ execution path via Apache Gluten and Velox. The result is columnar, SIMD-accelerated processing with native support for Parquet and Delta formats.
What makes this practical rather than just impressive in benchmarks:
No code changes required. Existing notebooks and Spark job definitions run through the NEE automatically when it is enabled. Unsupported operators fall back to JVM-based Spark transparently.
No vendor lock-in. The NEE can be disabled to revert to standard open-source Spark behaviour, and it supports both Parquet and Delta formats.
Environment-level control. The NEE is enabled at the environment item level, meaning all notebooks and job definitions that reference that environment inherit the performance gain without individual configuration.
Vectorized CSV parsing is now included in the NEE, accelerating CSV ingestion and query workloads. Vectorized JSON parsing and Spark Structured Streaming support are planned for future updates.
For data engineering workloads that are compute-heavy — large joins, aggregations over wide fact tables, complex transformations — the NEE is the single most impactful change in this release. Enable it on a non-production environment first, validate your critical jobs, then promote.
Apache Spark 4.1: What Engineers Should Know
Spark 4.0 was the inaugural release of the 4.x series — a significant milestone driven by the open-source community. Runtime 2.0 ships on Spark 4.1, which builds on that foundation.
The SQL layer received substantial attention in the 4.x cycle. Key additions relevant to Fabric analytics engineers:
VARIANT data type support. A new semi-structured data type designed for storing and querying JSON-like data natively in Spark SQL without pre-defining schemas. For teams ingesting event streams or API responses with variable structures, this reduces the need for complex schema inference logic.
SQL user-defined functions (UDFs). SQL-native UDFs without requiring PySpark or Scala. Teams that work primarily in SQL can now define reusable logic without leaving the SQL layer.
Session variables. Variables scoped to a Spark session, making parameterised notebook workflows cleaner. No more hacking variables through widget parameters or global configurations.
Pipe syntax. A more readable query composition style that chains transformations, reducing deeply nested subquery structures.
String collation support. Proper locale-aware string comparison and sorting, which matters for any workload handling multilingual data.
On the PySpark side:
Native plotting API. A built-in plotting capability directly in PySpark DataFrames without requiring a separate visualisation library import for exploratory analysis.
Python Data Source API. A new API for defining custom Python-based data sources, extending Spark's connectivity to external systems without writing JVM-based connectors.
Python UDTFs (User-Defined Table Functions). Functions that return tables rather than scalar values, written in Python. For teams that need complex row-generation logic, this is meaningfully more expressive than traditional UDFs.
Structured Streaming improvements. The Arbitrary State API v2 provides more flexible stateful stream processing, and the new State Data Source makes debugging streaming state considerably easier.
One important deprecation: SparkR is deprecated in Spark 4.x and may be removed in a future version. Teams with R-based Spark workloads should evaluate migration paths before committing to Runtime 2.0.
Delta Lake 4.2: The Lakehouse Foundation Upgrade
Delta Lake 4.2 is the most consequential component for Fabric lakehouse architects. It brings interoperability improvements, performance optimisations, and new table management capabilities.
Three features deserve specific attention:
Z-ordering. Organises data within Delta table files by specified columns to improve query performance for filtered queries. If a table is frequently queried with a filter on region or product_category, Z-ordering on those columns reduces the data scanned per query. This directly reduces CU consumption under both the old and new Fabric billing models.
Liquid Clustering. A more flexible approach to data organisation that automatically optimises layout without the manual maintenance that Z-ordering requires. Where Z-ordering demands explicit re-ordering when query patterns change, Liquid Clustering adapts incrementally. For tables with evolving access patterns, this is a lower-maintenance path to query acceleration.
Parallel Delta snapshot loading. The NEE loads Delta table snapshots in parallel, reducing query startup time for large tables. For workloads that frequently cold-start queries against large Delta tables, this reduces the latency before actual computation begins.
Critical caveat on Delta Lake 4.x features: Delta Lake 4.x-specific features are marked experimental and currently only work on Spark experiences — notebooks and Spark job definitions. If the same Delta Lake tables are used across multiple Fabric workloads (Warehouse SQL endpoint, Power BI Direct Lake, Dataflow Gen2), enabling Delta 4.x features on those tables may break compatibility. Review the Delta Lake table format interoperability documentation before enabling any Delta 4.x-specific features on shared tables.
Compute Management: Two New Capabilities
Runtime 2.0 introduces two compute management features worth noting for teams running Fabric at scale.
Resource profiles. Configure predefined resource allocations for Spark sessions — executor memory, cores, and instance counts — matched to specific workload requirements. For environments where different jobs have significantly different resource needs, resource profiles allow right-sizing without creating separate workspaces or environment items for each job type.
Custom live pools (preview). Pre-warmed Spark pools that reduce session startup time. Standard Spark session startup in Fabric carries a cold-start penalty while the cluster provisions. Custom live pools keep a pool ready, reducing that latency. This is in preview for Runtime 2.0 workloads and particularly relevant for interactive notebook workflows where startup latency affects developer productivity.
How to Enable Runtime 2.0
Two paths are available, depending on whether you want a workspace-wide default or targeted adoption.
Workspace-level (all Spark workloads in the workspace):
Go to Workspace settings
Select Data Engineering/Science → Spark settings → Environment tab
Under Runtime version, select 2.0 (Spark 4.1, Delta 4.2)
Save
Environment item level (specific notebooks or Spark job definitions):
Create or open an Environment item
Under the Runtime dropdown, select 2.0 (Spark 4.1, Delta 4.2)
Save and Publish
Attach the Environment to specific notebooks or job definitions
The environment item approach is the right migration path for most teams — it allows side-by-side validation of Runtime 2.0 against 1.3 without disrupting existing production workloads.
What to Watch Before You Migrate
Three things require action before migrating production workloads.
The Python library breaking change. The Runtime 2.0 rollout includes a Python environment upgrade that introduces a breaking change for environment items with Python and wheel libraries. Affected workloads will surface one of two errors:
warning: 1 deprecation (since 2.13.0); for details, enable :setting -deprecation
LibraryManagementError: An upgrade to the base Spark Python environment has been detected.
Please republish the environment.
The fix is straightforward but manual: remove all libraries from the environment, publish, re-add all libraries, publish again. This recreates the environment against the updated Python runtime. The important thing is knowing this is required before the first notebook failure surfaces in production.
SparkR deprecation. Any R-based Spark workload needs evaluation. SparkR is deprecated in Spark 4.x and may be removed in a future version. Migration options include rewriting in PySpark (which now has a native plotting API and improved UDF support) or using R in a non-Spark context where appropriate.
Delta Lake 4.x feature compatibility. As noted above, Delta 4.x-specific features are experimental and limited to Spark experiences. Enabling them on tables shared across Fabric workloads risks breaking those workloads. Audit which tables are accessed from multiple Fabric surfaces before enabling any Delta 4.x features.
The Migration Timeline
The current state:
Runtime 2.0 is GA and production-ready
Runtime 1.3 remains the default for existing workspaces
Runtime 2.0 becomes the default for new workspaces and environment items in late September 2026
Runtime 1.3 enters Long Term Support (LTS) on October 1, 2026, with support extending through March 2027
The practical implication: teams that have not migrated by late September will find new workspaces defaulting to Runtime 2.0. Any new notebooks created in those workspaces will run on Spark 4.1 and Delta 4.2 unless explicitly overridden. The LTS window on Runtime 1.3 gives existing workloads a six-month runway to validate and migrate before Runtime 1.3 reaches end of support.
The recommended approach for most Fabric teams:
Create a Runtime 2.0 environment item in a development workspace now
Run your critical Spark job definitions and notebooks against it — pay attention to Python library warnings and any SparkR usage
Validate Delta Lake table compatibility if Delta 4.x features are a consideration
Enable the NEE on the environment and measure performance on representative workloads
Promote to production ahead of the September default switch
What This Means for Fabric Workloads
Runtime 2.0 is a production platform that is genuinely better than what it replaces. Spark 4.1 is meaningfully more capable than Spark 3.5. Delta Lake 4.2 brings table management features that reduce manual maintenance. The Native Execution Engine's performance gains are real and applicable to most analytics workloads without code changes.
The migration is not without friction — the Python library issue requires manual action, SparkR users need a path forward, and Delta 4.x experimental features require careful handling on shared tables. None of these are blockers, but they are worth addressing deliberately rather than discovering them reactively after the September default switch.
The window between now and late September is exactly the right time to validate, fix, and migrate — with Runtime 1.3 LTS available as a safety net through March 2027 if any production workload needs more time.
Gilbert Kiptoo Lelon is a Microsoft Fabric analytics engineer and freelance consultant based in Nairobi, Kenya, operating under BluePeak Analytics. He holds the DP-600 (Fabric Analytics Engineer) and PL-300 (Power BI Data Analyst) certifications and works with US and European clients on Fabric lakehouse design, data engineering, and analytics architecture. Connect on LinkedIn: linkedin.com/in/gilbertkiptoo
Sources: Microsoft Fabric Runtime 2.0 official documentation (Microsoft Learn, updated August 12, 2026); Apache Spark 4.0 and 4.1 release notes; Delta Lake 4.2 release notes; Fabric July 2026 Feature Summary.
If you're working through a Runtime 2.0 migration — or have already moved production workloads to it — what did you find? Drop a comment below.
Top comments (0)