DEV Community: Gilbert Kiptoo Lelon

Microsoft Fabric Runtime 2.0 Is Now Generally Available; Here Is What It Actually Changes By Gilbert Kiptoo Lelon | DP-600 | DP-700

Gilbert Kiptoo Lelon — Fri, 14 Aug 2026 20:11:51 +0000

Microsoft Fabric Runtime 2.0 has moved from preview to general availability. This is not a minor version bump. Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, and a significantly enhanced Native Execution Engine land together in a single runtime upgrade. This article breaks down what changed, what it means for real Fabric workloads, and what to watch out for before you migrate.

Why This Release Is Different

Most runtime upgrades are incremental. A version bump here, a library update there, a deprecation notice you file away to deal with later.

Runtime 2.0 is not that.

This release represents the biggest component jump in Fabric's Spark history. The move from Runtime 1.3 (Spark 3.5, Delta Lake 3.2, Python 3.11, Java 11) to Runtime 2.0 spans multiple major versions across every core component simultaneously. Spark crosses the 4.x threshold. Delta Lake moves to 4.2. Python jumps to 3.13. Java moves from 11 to 21. Scala shifts from 2.12 to 2.13.

Each of those individually would be a meaningful migration. Together, they require careful evaluation before any production workload is touched.

The good news: Microsoft has been deliberate about the rollout. Runtime 2.0 is GA and production-ready, but it is not the default runtime yet. Existing workspaces stay on Runtime 1.3 until you explicitly opt in. The plan is to make Runtime 2.0 the default for new workspaces and environment items in late September 2026, which gives teams a defined window to validate and migrate at their own pace.

What Is in Runtime 2.0

The full component stack:

Apache Spark: 4.1
Delta Lake: 4.2
Python: 3.13
Java: 21
Scala: 2.13
R: 4.5.2
OS: Azure Linux 3.0 (Mariner 3.0)

That is the foundation. On top of it sit several capability upgrades worth understanding individually.

The Native Execution Engine: The Performance Story

The headline performance claim for Runtime 2.0 is significant: up to six times faster than open-source Spark on TPC-DS benchmarks, driven by the Native Execution Engine (NEE).

The NEE is not a new concept in Fabric — it shipped as part of Runtime 1.3 as well — but Runtime 2.0 brings meaningful enhancements. The engine works by offloading supported Spark operators from JVM-based execution to a vectorized C++ execution path via Apache Gluten and Velox. The result is columnar, SIMD-accelerated processing with native support for Parquet and Delta formats.

What makes this practical rather than just impressive in benchmarks:

No code changes required. Existing notebooks and Spark job definitions run through the NEE automatically when it is enabled. Unsupported operators fall back to JVM-based Spark transparently.
No vendor lock-in. The NEE can be disabled to revert to standard open-source Spark behaviour, and it supports both Parquet and Delta formats.
Environment-level control. The NEE is enabled at the environment item level, meaning all notebooks and job definitions that reference that environment inherit the performance gain without individual configuration.
Vectorized CSV parsing is now included in the NEE, accelerating CSV ingestion and query workloads. Vectorized JSON parsing and Spark Structured Streaming support are planned for future updates.

For data engineering workloads that are compute-heavy — large joins, aggregations over wide fact tables, complex transformations — the NEE is the single most impactful change in this release. Enable it on a non-production environment first, validate your critical jobs, then promote.

Apache Spark 4.1: What Engineers Should Know

Spark 4.0 was the inaugural release of the 4.x series — a significant milestone driven by the open-source community. Runtime 2.0 ships on Spark 4.1, which builds on that foundation.

The SQL layer received substantial attention in the 4.x cycle. Key additions relevant to Fabric analytics engineers:

VARIANT data type support. A new semi-structured data type designed for storing and querying JSON-like data natively in Spark SQL without pre-defining schemas. For teams ingesting event streams or API responses with variable structures, this reduces the need for complex schema inference logic.

SQL user-defined functions (UDFs). SQL-native UDFs without requiring PySpark or Scala. Teams that work primarily in SQL can now define reusable logic without leaving the SQL layer.

Session variables. Variables scoped to a Spark session, making parameterised notebook workflows cleaner. No more hacking variables through widget parameters or global configurations.

Pipe syntax. A more readable query composition style that chains transformations, reducing deeply nested subquery structures.

String collation support. Proper locale-aware string comparison and sorting, which matters for any workload handling multilingual data.

On the PySpark side:

Native plotting API. A built-in plotting capability directly in PySpark DataFrames without requiring a separate visualisation library import for exploratory analysis.

Python Data Source API. A new API for defining custom Python-based data sources, extending Spark's connectivity to external systems without writing JVM-based connectors.

Python UDTFs (User-Defined Table Functions). Functions that return tables rather than scalar values, written in Python. For teams that need complex row-generation logic, this is meaningfully more expressive than traditional UDFs.

Structured Streaming improvements. The Arbitrary State API v2 provides more flexible stateful stream processing, and the new State Data Source makes debugging streaming state considerably easier.

One important deprecation: SparkR is deprecated in Spark 4.x and may be removed in a future version. Teams with R-based Spark workloads should evaluate migration paths before committing to Runtime 2.0.

Delta Lake 4.2: The Lakehouse Foundation Upgrade

Delta Lake 4.2 is the most consequential component for Fabric lakehouse architects. It brings interoperability improvements, performance optimisations, and new table management capabilities.

Three features deserve specific attention:

Z-ordering. Organises data within Delta table files by specified columns to improve query performance for filtered queries. If a table is frequently queried with a filter on region or product_category, Z-ordering on those columns reduces the data scanned per query. This directly reduces CU consumption under both the old and new Fabric billing models.

Liquid Clustering. A more flexible approach to data organisation that automatically optimises layout without the manual maintenance that Z-ordering requires. Where Z-ordering demands explicit re-ordering when query patterns change, Liquid Clustering adapts incrementally. For tables with evolving access patterns, this is a lower-maintenance path to query acceleration.

Parallel Delta snapshot loading. The NEE loads Delta table snapshots in parallel, reducing query startup time for large tables. For workloads that frequently cold-start queries against large Delta tables, this reduces the latency before actual computation begins.

Critical caveat on Delta Lake 4.x features: Delta Lake 4.x-specific features are marked experimental and currently only work on Spark experiences — notebooks and Spark job definitions. If the same Delta Lake tables are used across multiple Fabric workloads (Warehouse SQL endpoint, Power BI Direct Lake, Dataflow Gen2), enabling Delta 4.x features on those tables may break compatibility. Review the Delta Lake table format interoperability documentation before enabling any Delta 4.x-specific features on shared tables.

Compute Management: Two New Capabilities

Runtime 2.0 introduces two compute management features worth noting for teams running Fabric at scale.

Resource profiles. Configure predefined resource allocations for Spark sessions — executor memory, cores, and instance counts — matched to specific workload requirements. For environments where different jobs have significantly different resource needs, resource profiles allow right-sizing without creating separate workspaces or environment items for each job type.

Custom live pools (preview). Pre-warmed Spark pools that reduce session startup time. Standard Spark session startup in Fabric carries a cold-start penalty while the cluster provisions. Custom live pools keep a pool ready, reducing that latency. This is in preview for Runtime 2.0 workloads and particularly relevant for interactive notebook workflows where startup latency affects developer productivity.

How to Enable Runtime 2.0

Two paths are available, depending on whether you want a workspace-wide default or targeted adoption.

Workspace-level (all Spark workloads in the workspace):

Go to Workspace settings
Select Data Engineering/Science → Spark settings → Environment tab
Under Runtime version, select 2.0 (Spark 4.1, Delta 4.2)
Save

Environment item level (specific notebooks or Spark job definitions):

Create or open an Environment item
Under the Runtime dropdown, select 2.0 (Spark 4.1, Delta 4.2)
Save and Publish
Attach the Environment to specific notebooks or job definitions

The environment item approach is the right migration path for most teams — it allows side-by-side validation of Runtime 2.0 against 1.3 without disrupting existing production workloads.

What to Watch Before You Migrate

Three things require action before migrating production workloads.

The Python library breaking change. The Runtime 2.0 rollout includes a Python environment upgrade that introduces a breaking change for environment items with Python and wheel libraries. Affected workloads will surface one of two errors:

warning: 1 deprecation (since 2.13.0); for details, enable :setting -deprecation

LibraryManagementError: An upgrade to the base Spark Python environment has been detected.
Please republish the environment.

The fix is straightforward but manual: remove all libraries from the environment, publish, re-add all libraries, publish again. This recreates the environment against the updated Python runtime. The important thing is knowing this is required before the first notebook failure surfaces in production.

SparkR deprecation. Any R-based Spark workload needs evaluation. SparkR is deprecated in Spark 4.x and may be removed in a future version. Migration options include rewriting in PySpark (which now has a native plotting API and improved UDF support) or using R in a non-Spark context where appropriate.

Delta Lake 4.x feature compatibility. As noted above, Delta 4.x-specific features are experimental and limited to Spark experiences. Enabling them on tables shared across Fabric workloads risks breaking those workloads. Audit which tables are accessed from multiple Fabric surfaces before enabling any Delta 4.x features.

The Migration Timeline

The current state:

Runtime 2.0 is GA and production-ready
Runtime 1.3 remains the default for existing workspaces
Runtime 2.0 becomes the default for new workspaces and environment items in late September 2026
Runtime 1.3 enters Long Term Support (LTS) on October 1, 2026, with support extending through March 2027

The practical implication: teams that have not migrated by late September will find new workspaces defaulting to Runtime 2.0. Any new notebooks created in those workspaces will run on Spark 4.1 and Delta 4.2 unless explicitly overridden. The LTS window on Runtime 1.3 gives existing workloads a six-month runway to validate and migrate before Runtime 1.3 reaches end of support.

The recommended approach for most Fabric teams:

Create a Runtime 2.0 environment item in a development workspace now
Run your critical Spark job definitions and notebooks against it — pay attention to Python library warnings and any SparkR usage
Validate Delta Lake table compatibility if Delta 4.x features are a consideration
Enable the NEE on the environment and measure performance on representative workloads
Promote to production ahead of the September default switch
What This Means for Fabric Workloads

Runtime 2.0 is a production platform that is genuinely better than what it replaces. Spark 4.1 is meaningfully more capable than Spark 3.5. Delta Lake 4.2 brings table management features that reduce manual maintenance. The Native Execution Engine's performance gains are real and applicable to most analytics workloads without code changes.

The migration is not without friction — the Python library issue requires manual action, SparkR users need a path forward, and Delta 4.x experimental features require careful handling on shared tables. None of these are blockers, but they are worth addressing deliberately rather than discovering them reactively after the September default switch.

The window between now and late September is exactly the right time to validate, fix, and migrate — with Runtime 1.3 LTS available as a safety net through March 2027 if any production workload needs more time.

Gilbert Kiptoo Lelon is a Microsoft Fabric analytics engineer and freelance consultant based in Nairobi, Kenya, operating under BluePeak Analytics. He holds the DP-600 (Fabric Analytics Engineer) and PL-300 (Power BI Data Analyst) certifications and works with US and European clients on Fabric lakehouse design, data engineering, and analytics architecture. Connect on LinkedIn: linkedin.com/in/gilbertkiptoo

Sources: Microsoft Fabric Runtime 2.0 official documentation (Microsoft Learn, updated August 12, 2026); Apache Spark 4.0 and 4.1 release notes; Delta Lake 4.2 release notes; Fabric July 2026 Feature Summary.

If you're working through a Runtime 2.0 migration — or have already moved production workloads to it — what did you find? Drop a comment below.

The One-Minute Trap: What Microsoft Fabric's New Warehouse Billing Model Means for Your Workloads By Gilbert Kiptoo Lelon | DP-700|DP-600

Gilbert Kiptoo Lelon — Fri, 14 Aug 2026 19:51:37 +0000

Microsoft is changing how Fabric Data Warehouse charges for compute starting August 2026. Most teams will discover what that means on their next capacity bill. This article is a technical breakdown of the change, what the math actually looks like, and how to rethink workload design before the surprise arrives.

Why This Change Matters More Than It Sounds

When a cloud platform adjusts its billing model, the announcement usually reads like a changelog — a couple of dry sentences buried in a notification email. Most teams file it away to review later. Later sometimes never arrives.

This one is worth reading now.

Starting in August 2026, Microsoft Fabric Data Warehouse (and SQL analytics endpoints of Lakehouse) is moving away from per-query CPU-time metering and toward per-workspace virtual-node time metering. The distinction sounds like an accounting technicality. It is not. It changes the cost shape of nearly every Fabric Warehouse workload, and the direction of that change depends entirely on how your queries are structured.

Dense, heavy ETL windows may get cheaper. Sparse, chatty workloads — think dashboards, monitoring probes, and single-query wakeups — may get significantly more expensive. Not because the queries got slower or heavier, but because the billing floor changed.

I want to walk through the mechanics, the math, and the practical rethink this requires — drawing on the excellent technical breakdown by Nikola Ilic (Data Mozart), official Microsoft documentation, and my own experience designing Fabric Warehouse workloads for clients across the US and European markets.

What Is Actually Changing
The Old Model: Per-Query CPU Time

Under the current model, CU consumption for Fabric Warehouse is driven by the CPU time a query actually burns. If a query completes in 20 seconds and uses modest compute, the CU charge reflects that modest usage. The metering is closely tied to what the query did.

The New Model: Per-Workspace Virtual-Node Time

Starting August 2026, the metering unit shifts to the virtual node — a 4-vCore unit of Warehouse compute that Fabric allocates automatically based on workload demand.

The rate also changes:

Old rate: 2 CU per vCore
New rate: 0.53 CU per vCore

So the new per-virtual-node math is:

1 virtual node = 4 vCores
1 vCore = 0.53 CUs
1 virtual node = 2.12 CUs while active

And here is the clause that changes everything:

There is a one-minute minimum per workspace. If Warehouse activity lasts less than one minute, the virtual-node uptime is rounded up to 60 seconds. After that threshold, billing is per second.

That floor is where most of the cost surprise lives.

The Math in Four Scenarios

The calculations below use the unit definitions from Microsoft's update. They are not dollar-cost estimates — actual costs depend on your F-SKU pricing and how many virtual nodes Fabric allocates. The point is to illustrate the metering shape.

Scenario 1: The Lonely Short Query

A workspace wakes up one virtual node for a 20-second query. The one-minute floor applies:

1 virtual node × 60 seconds × 2.12 CUs = 127.2 CU-seconds

Under the old model, a 20-second query with modest CPU usage would have generated far fewer CU-seconds. The new model charges for the full allocation window, not the useful work inside it. For a query that barely did anything, 127.2 CU-seconds is steep.

This is the dashboard-with-one-lonely-query problem. The query is cheap to run. The compute wake-up is not.

Scenario 2: Same Window, More Work

Now the same workspace runs five queries inside that first minute. The virtual node is already allocated. More useful work packs into the same billing window.

1 virtual node × 60 seconds × 2.12 CUs = 127.2 CU-seconds

Same total charge. But now five queries share it instead of one. Workload density directly reduces the effective cost per query.

This is why the mental model has to shift. The question is no longer only "is this query efficient?" It is "how much useful work happened during the compute window this query opened?"

Scenario 3: A Proper ETL Window

Ten virtual nodes running for 20 minutes during a scheduled transformation:

10 virtual nodes × 1,200 seconds × 2.12 CUs = 25,440 CU-seconds

No one-minute weirdness here — the workload runs long enough that the floor is irrelevant. If those nodes are genuinely busy during that window (scans, joins, aggregations, statistics), the lower CU-per-vCore rate can actually reduce consumption compared to the old CPU-time model.

Dense ETL workloads may come out ahead. The key condition is that compute stays occupied while it is allocated.

Scenario 4: Workspace Fragmentation — the Expensive Hidden Pattern

This is the scenario I flag most urgently for Fabric architects.

Ten different workspaces each run a tiny 20-second query. Each workspace hits the one-minute minimum independently:

10 workspaces × 1 virtual node × 60 seconds × 2.12 CUs = 1,272 CU-seconds

Now suppose those same queries run close together in a single workspace:

1 workspace × 1 virtual node × 60 seconds × 2.12 CUs = 127.2 CU-seconds

Same queries. Same data. Same analytical intent. Ten times the metered cost under the fragmented model.

This does not mean collapsing every workspace into one — governance, security, lifecycle, and team ownership all matter enormously. But it does mean workspace design is now part of cost design in a way it was not before.

How This Compares to Snowflake and Databricks

The direction of this change is not surprising if you have worked on Snowflake or Databricks. Fabric is converging toward an industry-standard model — and understanding the nuances of that convergence matters.

Snowflake

Snowflake charges for virtual warehouses based on size, number of clusters, and compute runtime. There is a 60-second minimum when compute provisions, and per-second billing after that.

The economics are structurally similar to the new Fabric model: allocated compute over time, with a floor. Dense usage is rewarded. Constant start-stop patterns are penalised.

The key difference is control. In Snowflake, the engineer explicitly chooses warehouse size, auto-suspend settings, and multi-cluster behaviour. The warehouse is the named cost-and-performance boundary that the team configures and monitors.

In Fabric, Microsoft handles allocation and scaling automatically. The workspace becomes the effective metering boundary, but there is no manual sizing lever equivalent to Snowflake's warehouse size. Fabric is becoming Snowflake-like in economics but not in explicit compute controls.

Databricks

Databricks SQL Warehouses, particularly serverless SQL Warehouses with Intelligent Workload Management, follow the same pattern: managed elastic compute, dynamic scaling, per-second granularity after provisioning.

Databricks still exposes SQL warehouses as named resources with configurable cluster size, scaling, and auto-stop settings. Idle SQL warehouses continue to accumulate DBU charges until stopped.

Fabric hides more of that infrastructure layer. The upside is less compute plumbing to manage. The risk is that noisy Warehouse behaviour can affect the same shared capacity pool that Power BI, Data Factory notebooks, Eventhouse, and every other Fabric workload depends on — without the same visibility that named Databricks warehouses provide.

The new Fabric model is arguably more elegant from a management perspective, and more dangerous from a cost-visibility perspective. Both things are true simultaneously.

The Workload Patterns to Review

Based on the mechanics above, here are the specific patterns worth auditing before August:

Workspaces with frequent short Warehouse activity. Any workspace where the pattern is "wake compute, ask one small question, sleep" is now a one-minute minimum factory. Aggregate those queries. Batch them. Stagger them thoughtfully.

Dashboard queries that cold-start compute. A Power BI report backed by DirectQuery against a Fabric Warehouse, loading at 8 AM when compute is cold, now pays for a full virtual-node minute even if the queries complete in seconds. Consider caching strategies or aggregation layers for these patterns.

Monitoring and health-check probes. If operational scripts or pipeline sensors query the Warehouse every few minutes to check status, each probe may trigger its own billing window. Route these checks elsewhere — Lakehouse tables, metadata APIs, or KQL Eventhouse if latency allows.

System-generated activity. Microsoft's documentation notes that Warehouse Query metering includes both user-generated and system-generated T-SQL statements. Background optimisation and statistics operations count. Understanding what Fabric generates on your behalf is now a billing-relevant question.

ETL windows where compute stays busy. These are worth confirming under the new model, because they may genuinely benefit from the lower CU-per-vCore rate. Review your Capacity Metrics app for ETL job windows and calculate whether sustained compute time at 0.53 CU/vCore comes out better than what you were paying before.

Workspace fragmentation across similar workloads. If the organisation has proliferated workspaces for reasons of convenience rather than governance necessity, the cost of that fragmentation is now directly measurable. It may be worth a consolidation conversation.

The Mental Model Shift

The old question for Fabric Warehouse cost management was: "Which query burned the most CPU?"

The query insights views, execution history, and Capacity Metrics app were all pointed at that question. Identify the expensive queries. Optimise them. Reduce CPU burn. That was the playbook.

The new question is: "When Fabric allocated Warehouse compute for this workspace, did we actually use it?"

That is a fundamentally different lens. It shifts attention from individual query efficiency to workspace-level compute utilisation during active windows. A highly optimised query that runs alone in a cold workspace at 15-second intervals is now more expensive per unit of work than a moderately inefficient query that runs alongside several others in a busy window.

The operational checklist changes accordingly:

Stop measuring cost purely by query duration or rows scanned
Start measuring cost by compute window utilisation — how much useful work happened per virtual-node minute
Build schedules that create dense ETL windows rather than spreading queries across many small wakeups
Review workspace architecture with cost implications as a first-class concern, not an afterthought
Monitor system-generated activity, not just user queries
What This Means for Fabric Architecture Decisions

For analytics engineers and data architects working on Fabric today, this change is an argument for a few structural principles that were already good practice and are now financially reinforced.

Consolidate analytical workloads into coherent refresh windows. If ten semantic models refresh independently and each one pings the Warehouse, that is ten billing windows. A coordinated refresh orchestration that sequences them inside a single active compute window is better architecture and lower cost.

Think carefully before routing every operational query through the Warehouse. Fabric offers multiple SQL surfaces. The KQL Eventhouse is purpose-built for real-time monitoring and operational queries at low latency. Lakehouse SQL endpoints are appropriate for exploratory analytics. Not every query needs to live in the Warehouse, and routing chatty operational patterns away from the Warehouse avoids waking compute unnecessarily.

Use the Capacity Metrics app differently. The useful signal is no longer just "which operation consumed the most CUs." It is "which workspaces have a pattern of frequent short Warehouse activity." Build a view of your workspace-level compute windows and their utilisation before August.

Document workspace purpose and query patterns. This is the kind of governance artifact that often gets skipped in fast-moving Fabric projects. After August, the cost of undocumented ad-hoc workspace proliferation becomes explicit. Better to audit and rationalise now than explain the bill later.

A Note on the Broader Trend

The Fabric team is making a deliberate architectural bet with this change. By moving to virtual-node time metering, Microsoft aligns Fabric Warehouse economics with the dominant model in enterprise data warehousing. Snowflake built its business on this model. Databricks's push toward serverless SQL is the same thesis in different packaging.

The bet is that compute density — keeping allocated resources genuinely busy — is a better optimisation incentive than per-query CPU minimisation. There is real merit to that argument. Dense, well-orchestrated workloads tend to produce better throughput, better caching behaviour, and more predictable performance.

The risk is that many Fabric environments were designed under the old model, with workload patterns that made sense when billing was per-query. August 2026 is an inflection point. Teams that audit their patterns before the change have time to adjust. Teams that discover it on the bill will be in a more reactive position.

Summary

The Fabric Data Warehouse billing change rolling out in August 2026 replaces per-query CPU-time metering with per-workspace virtual-node time metering. The key facts:

One virtual node is four vCores at 0.53 CU per vCore, totalling approximately 2.12 CUs while active
A one-minute minimum applies per workspace — 20-second queries pay for 60 seconds
Dense workloads that keep compute busy during active windows may get cheaper under the lower rate
Sparse, chatty patterns — single queries waking compute, fragmented workspaces, monitoring probes — may get meaningfully more expensive
The design question shifts from query-level CPU optimisation to workspace-level compute utilisation

The new mental model is not complex. But acting on it requires looking at workload patterns differently, and doing that before August is considerably easier than explaining an unexpected capacity bill afterward.

Gilbert Kiptoo Lelon is a Microsoft Fabric analytics engineer and freelance consultant based in Nairobi, Kenya, operating under BluePeak Analytics. He holds the DP-600 (Fabric Analytics Engineer) and PL-300 (Power BI Data Analyst) certifications and works with US and European clients on Fabric lakehouse design, Warehouse optimisation, and analytics engineering. Connect on LinkedIn: linkedin.com/in/gilbertkiptoo

Credit to Nikola Ilic (Data Mozart) for the original technical breakdown of this change and the side-by-side comparisons with Snowflake and Databricks. Sources: Microsoft Fabric Data Warehouse billing documentation (Microsoft Learn); Snowflake warehouse documentation; Databricks SQL Warehouse documentation.