How Apache Polaris Vends Credentials: Securing Data Access Without Sharing Keys

Prithvi S — Thu, 09 Apr 2026 09:10:24 +0000

The modern data warehouse demands a fundamental shift in how we think about access control. When you build multi-tenant systems at scale, the traditional approach - distributing long-lived API keys or database credentials - becomes a security nightmare. Apache Polaris solves this elegantly: vend temporary, scoped credentials on demand, revoke instantly, audit everything.

The Problem: Why Long-Lived Credentials Don't Scale

At Netflix, Cloudera, or any major data platform, you're managing access across hundreds of users, services, and applications. If you hand out permanent API keys:

Revocation is impossible - a compromised key stays valid until you manually rotate it
Audit trails are fuzzy - you don't know which key accessed which data when
Compliance is painful - SOC2, HIPAA, PCI-DSS demand temporal, traceable access
Key rotation is a nightmare - updating thousands of clients, coordinating across teams
Scope is too broad - a key that works today still works tomorrow, even if access should have expired

This is why cloud providers moved away from permanent credentials. AWS uses temporary STS tokens. GCP uses short-lived access tokens. Azure has managed identities. The pattern is clear: trust should be ephemeral, scoped, and revocable.

Polaris applies this principle to data catalogs and table access.

How Polaris Vends Credentials

Polaris is an open-source, REST-first Iceberg catalog that implements the Iceberg REST API. Unlike traditional catalogs (which require direct database access or assume long-lived credentials), Polaris mints temporary credentials on every access.

Here's the flow:

1. Authorization Check (Who Are You? What Can You Do?)

When a client requests data access, Polaris first checks:

Is this principal (user/service) authenticated?
Do they have a role with permission to access this table?
Is the access read-only or read-write?

This uses Polaris's two-tier RBAC model:

Principal Roles - assigned to service principals (identities)
Catalog Roles - define actual permissions (TABLE_READ_DATA, TABLE_WRITE_DATA, etc.)

Example: A data analyst gets role analyst_prod, which is granted TABLE_READ_DATA on catalog.sales.transactions. A service account gets role etl_writer, which gets TABLE_WRITE_DATA on catalog.etl.staging.

2. Storage Configuration Lookup

Polaris queries its configuration:

Which cloud provider hosts this table? (AWS S3, GCS, Azure Blob)
What credentials should be used for minting? (Polaris's service role)
Are there any table-specific overrides?

3. Credential Minting

Here's where the magic happens. Polaris calls the cloud provider's API to mint temporary, scoped credentials:

For AWS (S3):

assume-role(
  role_arn=polaris-service-role,
  session_name=client-session-xyz,
  session_duration=15m,
  policy=restrict-to-s3://bucket/table-path/
)

Returns: temporary AWS credentials (access key + secret key) valid for 15 minutes, scoped to the specific table path.

For GCS:

create-service-account-key(
  service_account=polaris-sa@project.iam.gserviceaccount.com,
  lifetime=15m,
  custom-claims={ "resource": "gs://bucket/table-path" }
)

Returns: short-lived JWT valid for 15 minutes, scoped to the table path.

For Azure:

get-managed-identity-token(
  resource=https://storage.azure.com,
  lifetime=15m,
  scope=/subscriptions/xxx/resourceGroups/yyy/providers/Microsoft.Storage/storageAccounts/zzz/blobServices/default/containers/table-path
)

Returns: short-lived bearer token (OAuth2) valid for 15 minutes, scoped to the container path.

4. Scope Restriction

The credentials are scoped to:

Path - exact table location (e.g., s3://data/catalog/table/)
Operations - read-only (GET) vs read-write (GET, PUT, DELETE)
Duration - typically 15 minutes (configurable)

A client can't use these credentials to:

Access other tables
Write data to a read-only table
Perform actions after expiration

5. Return to Client

Polaris returns the temporary credentials to the client. The client's query engine (Spark, Trino, Presto, DuckDB, etc.) receives these credentials and uses them to read/write data directly to object storage.

No long-lived secrets are distributed. The client never sees Polaris's service credentials.

Why This Matters

Security Benefits

Instant Revocation - Delete a principal's role, all future requests are denied instantly
Fine-Grained Access - per-table, per-operation, per-principal permissions
Auditability - every credential mint event is logged (who, when, which table, read/write)
Compliance-Ready - temporal credentials, immutable audit trails, no shared secrets
Blast Radius - if a credential leaks, it's only valid for 15 minutes and only for one table

Operational Benefits

No Credential Rotation - credentials are automatically rotated every request
No Key Distribution - no need to distribute, store, or rotate permanent keys
Multi-Cloud Ready - same API works with S3, GCS, Azure, MinIO
Client Simplicity - clients just receive credentials and query - they don't manage them

Business Benefits

Compliance Aligned - meets SOC2, HIPAA, PCI-DSS, FedRAMP requirements
Cost Control - audit who accessed what, charge accordingly
Governance - enforce data mesh principles (teams own their data, Polaris mediates access)

Real-World Example: Data Mesh at Scale

Imagine you're running a data mesh with 50 teams, each owning their own datasets. Without Polaris:

Each team issues permanent API keys to consumers
Keys spread across configuration files, CI/CD pipelines, notebooks
A leaked key compromises an entire dataset
Revocation requires manual updates across dozens of systems
Audit trails are incomplete (keys used by multiple systems)

With Polaris:

Each consumer requests access via Polaris API (authenticated via OIDC, OAuth2, mTLS)
Polaris checks if consumer's identity has permission
Polaris mints a 15-minute credential scoped to the specific table and operation
Consumer queries the data with the temporary credential
On next request, the credential is already expired - a new one is minted
Revoke a consumer's role, and all future requests fail instantly
Audit logs show exactly which identity accessed which table at what time

Version 1.3.0 Features (January 2026)

Recent Polaris releases added:

Federated Credential Vending - Polaris can mint credentials for external catalogs (Snowflake, AWS Glue) instead of clients using their own credentials
OPA Integration - externalize authorization logic to Open Policy Agent for complex policies
Generic Tables - support Delta Lake and Hudi alongside Iceberg
Metrics Reporting - pluggable framework to report table metrics (row/byte counts, commits)

Getting Started

Polaris is on Apache Foundation:

GitHub: https://github.com/apache/polaris
Docs: https://polaris.apache.org/
REST API: Implements Iceberg REST Catalog spec

If you're building a data platform, data mesh, or multi-tenant system, Polaris's credential vending model is worth studying. It's a pattern that applies beyond Iceberg - any system managing shared resource access can benefit from temporal, scoped credentials.

About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. Follow my work on GitHub.

DEV Community: Prithvi S