DEV Community

dinakar
dinakar

Posted on

🔐 Storage Credentials in Databricks Unity Catalog (AWS) — The Security Backbone You Can't Ignore

#databricks #aws #dataengineering #deltlake

As organisations move to the Databricks Lakehouse Platform on AWS, two priorities consistently surface at the top of every architecture conversation: data governance and secure storage access.

Unity Catalog addresses both — but one foundational component powers nearly every secure storage action underneath the hood:

Storage Credentials — the security backbone that connects Unity Catalog governance with AWS S3 permissions.

## 🚀 Why Did Storage Credentials Need to Exist?****

Before Unity Catalog, Databricks relied on cluster-level instance profiles to access S3. The problems with this approach were significant:

  • Every user on the cluster got identical S3 permissions — no isolation
  • No per-table, per-schema, or per-user access control
  • No centralised governance layer
  • Difficult to audit who accessed what
  • Dangerous for data exfiltration scenarios
  • Cross-account S3 architectures were painful to manage

Unity Catalog replaced this pattern entirely with object-based governance — and Storage Credentials sit at the centre of that improvement.

## 🧩 What Exactly Is a Storage Credential?

A Storage Credential is an AWS IAM Role (roles are strongly recommended over static keys) that Databricks can assume via AWS STS to access S3 securely.

You register it in Unity Catalog with a single SQL statement:

CREATE STORAGE CREDENTIAL finance_sc
WITH ROLE 'arn:aws:iam::123456789012:role/databricks-finance-access';
Enter fullscreen mode Exit fullscreen mode

Once registered, it becomes the governed identity that Databricks uses when accessing specific S3 locations — completely abstracted away from individual users or cluster configurations.

🔍 What Problems Do Storage Credentials Solve?

1. Fine-Grained, UC-Governed Access to S3

Storage Credentials enforce exactly:

  • Who can read or write files
  • Who can register external locations
  • Which S3 buckets a team or table can access
  • Which catalog or schema maps to which S3 path

This makes S3 permissions fully consistent with table-level governance in Unity Catalog — no more out-of-band IAM management.

2. They Replace Instance Profiles Completely

SQL Warehouses — especially Serverless SQL — do not support instance profiles at all.

If you are running Serverless compute (which Databricks is pushing as the default going forward), all S3 access must go through Storage Credentials.

Without them → your SQL Warehouse simply cannot query your data.

3. They Are Essential for External Locations

You cannot create an External Location without first having a Storage Credential:

CREATE EXTERNAL LOCATION bronze_loc
URL 's3://company-lakehouse/bronze/'
WITH STORAGE CREDENTIAL finance_sc;
Enter fullscreen mode Exit fullscreen mode

This allows Unity Catalog to enforce access at directory-level granularity within S3 — not just at the bucket level.

4. Secure Cross-Account S3 Access

Most enterprise AWS environments use multi-account architectures — a data account, a workload account, a security account. Managing S3 access across accounts with instance profiles is messy and risky.

Storage Credentials simplify this significantly:

Databricks Account
       ↓ assumes
IAM Role in Data S3 Account (via STS)
       ↓ scoped to
Specific S3 bucket prefixes via bucket policy
Enter fullscreen mode Exit fullscreen mode

No cluster-level over-permissions. No credential sprawl. Clean, auditable cross-account access.

5. Strong, Built-In Auditability

All S3 access routed through Storage Credentials is:

  • Fully auditable via system.access.audit
  • Tied to the UC object hierarchy (catalog → schema → table)
  • Tied to the user identity via Unity Catalog's identity passthrough

This gives compliance and data governance teams complete visibility into who accessed which data, when, and through which pipeline.

-- Audit all storage credential usage
SELECT
    event_time,
    user_identity.email       AS user,
    action_name,
    request_params
FROM system.access.audit
WHERE action_name IN (
    'createStorageCredential',
    'updateStorageCredential',
    'getStorageCredential'
)
ORDER BY event_time DESC;
Enter fullscreen mode Exit fullscreen mode

🏗️ How It All Fits Together

The user never holds AWS credentials directly. The IAM role is never exposed in notebooks, job configs, or Spark settings. Unity Catalog is the single control plane for everything.

⚡ Quick Setup Checklist 

✅ Create IAM Role in AWS with S3 permissions
✅ Set up trust policy for Databricks Account IAM Role
✅ Register Storage Credential in Unity Catalog (UC Admin)
✅ Create External Location pointing to S3 path
✅ GRANT CREATE EXTERNAL LOCATION to appropriate groups
✅ Verify with VALIDATE STORAGE CREDENTIAL
Enter fullscreen mode Exit fullscreen mode 
-- Verify storage credential works
VALIDATE STORAGE CREDENTIAL finance_sc;

-- Check who has access to it
SHOW GRANTS ON STORAGE CREDENTIAL finance_sc;
Enter fullscreen mode Exit fullscreen mode

💡 Key Takeaway

Storage Credentials are not optional in a modern Databricks architecture — they are the foundational security primitive that makes Unity Catalog's governance model work in practice on AWS.

If you are still using cluster-level instance profiles, or if you are planning to adopt Serverless SQL, migrating to Storage Credentials is not just recommended — it is required.

🔗 Resources

Top comments (0)