Microsoft Fabric has extended customer-managed key (CMK) encryption to cover temporary data generated during Spark job execution. This is not a minor security patch. It closes a significant compliance gap that has prevented regulated industries from adopting cloud analytics platforms.
This article explains what was protected before, what is protected now, and why the distinction matters for organizations with strict data governance requirements.
What actually changed
Fabric's CMK encryption previously covered data at rest,tables, files, and persisted data in OneLake. What it did not cover was the temporary working data Spark generates during job execution.
The gap was specific:
Data stored on cluster disks during Spark job execution
Shuffle data retained on cluster nodes during processing
Temporary spill files created during large transformations
This data existed in a protected gray zone: not quite "at rest" in the traditional sense, but not ephemeral enough to ignore from a compliance perspective.
The GA announcement changes that. CMK now extends across the entire Spark data lifecycle — from storage through active processing. The encryption keys you control now protect temporary cluster data the same way they protect your OneLake tables.
Why this matters: the compliance reality
For organizations in regulated industries, data protection is not a technical preference. It is a contractual, regulatory, and audit requirement. Security assessments typically ask: "Does this cover everything?"
The previous honest answer was: "Everything that is persisted. But Spark generates temporary data during processing that is encrypted with platform-managed keys."
That answer triggered risk assessments, legal reviews, and project delays. The temporary data gap was not theoretical — it was a real compliance barrier.
The new answer is: "Yes. Customer-managed keys protect data at rest, data in transit, and temporary data during Spark processing. The entire data lifecycle is covered."
That single change moves Fabric from "requires exception" to "standard approved platform" in many enterprise security frameworks.
The Spark data flow — what is now protected
When a Spark job runs in Fabric, data moves through five stages:
- Input data read from OneLake ← CMK-protected (before)
- Shuffle operations between executors ← NOW CMK-protected
- Spill files written to local disk ← NOW CMK-protected
- Temporary data cached on nodes ← NOW CMK-protected
- Output written back to OneLake ← CMK-protected (before)
Shuffle data: When Spark performs joins or aggregations, data is exchanged between cluster nodes. This shuffle data was previously written to local disks with platform-managed encryption. Now it uses customer-managed keys.
Spill files: When Spark operations exceed available memory, data is spilled to disk. These spill files persisted for the duration of the job and were previously outside CMK scope. Now they are encrypted with customer keys.
Cluster disk storage: Temporary data cached on cluster nodes during job execution is now CMK-protected throughout the job lifecycle.
Implementation — what you need to do
The critical detail: no changes are required to existing Spark jobs or code.
For new workspaces: Enable CMK at workspace creation. Spark jobs automatically use CMK for all data types.
For existing workspaces: If CMK is already enabled, the extension applies automatically. No job code changes, no pipeline modifications, no performance impact.
This is a platform-level change. Encryption happens at the infrastructure layer, transparent to the Spark jobs running above it.
The key hierarchy
Fabric uses a hierarchical key model:
Customer-managed key (CMK) ← You control this in Azure Key Vault
↓
Platform-managed keys ← Intermediate keys derived from CMK
↓
Data encryption keys ← Encrypt actual data blocks
Previously, platform-managed keys protected temporary data — breaking the chain from your CMK. Now CMK flows through the entire hierarchy, including temporary data paths.
The audit trail in Key Vault now shows complete key usage across the entire data lifecycle, simplifying compliance reporting.
Who this unblocks
Financial services: Banks with strict requirements that all customer financial data be encrypted with customer-controlled keys. The temporary data gap meant Spark workloads handling transaction data required security exceptions. Spark-based fraud detection, risk modeling, and customer analytics can now proceed with standard security approval.
Healthcare: HIPAA and HITRUST frameworks require encryption of protected health information throughout processing. Temporary data containing PHI encrypted with platform keys created compliance exposure. Healthcare analytics workloads on Spark can now meet encryption requirements without architectural workarounds.
Government and public sector: Government contracts often specify CMK requirements for all data handling. The previous gap prevented Fabric from meeting these specifications for Spark workloads. Government data processing pipelines can now use Spark with full CMK coverage.
Multi-tenant SaaS providers: SaaS companies building on Fabric often promise customers CMK encryption. The temporary data gap meant they could not fully deliver that promise for Spark-based features. SaaS providers can now offer complete CMK coverage across all data processing paths.
Before and after: the security review path
Before:
Document the temporary data gap
Obtain risk acceptance from security leadership
Implement compensating controls
Accept audit findings related to encryption coverage
After:
Enable CMK at workspace creation
Proceed with standard security review
No exceptions required
Clean audit posture
For architects designing Fabric solutions for regulated industries, this removes a significant friction point from the security conversation.
Practical steps if you are implementing this now
New workspaces:
Enable CMK during workspace provisioning — not after
Configure the Key Vault and key before workspace creation
Establish key rotation procedures upfront
Set up Key Vault access alerts
Existing workspaces:
Verify CMK is currently enabled (the extension applies automatically if it is)
Review Spark workloads and update compliance documentation
Inform security teams that the gap is now closed
Migration planning:
Identify Spark jobs that were previously blocked by CMK gaps
Remove compensating controls and workarounds that addressed the temporary data issue
Update runbooks and security assessments to reflect complete coverage
Summary
The technical change is straightforward: temporary data on cluster disks, shuffle data, and spill files are now encrypted with customer-managed keys, with no code changes required.
The impact is substantial. Regulated industries can now adopt Spark-based analytics with standard security approval. SaaS providers can offer complete encryption coverage. Security teams can approve Fabric without exceptions.
For organizations with strict data governance requirements, this announcement moves Fabric from "requires security exception" to "standard approved platform" for Spark workloads.
Is your organization in a regulated industry that was previously blocked by the CMK gap? Drop a comment — would be interested to hear how teams handled the workarounds before this change.
Gilbert Kiptoo Lelon is a Microsoft Fabric analytics engineer and freelance consultant based in Nairobi, Kenya, operating under BluePeak Analytics. He holds the DP-600 (Fabric Analytics Engineer) and PL-300 (Power BI Data Analyst) certifications and works with US and European clients on Fabric security architecture, compliance, and analytics engineering. Connect on LinkedIn: linkedin.com/in/gilbertkiptoo
Top comments (0)