Is Your Databricks Job Still Using Your Personal Access Token?

Why this matters: Hardcoded personal access tokens are a primary vector for privilege escalation and data exfiltration in managed data environments. Moving to Service Principals decouples authentication from individual identities, ensuring that pipeline access persists beyond the lifecycle of any single user account.

Consider a hypothetical pipeline that syncs data from a third-party API into a Delta Lake table. The job is configured to run on a cluster using a Python script that requires an API key. For months, the job runs perfectly. Then, the engineer who originally set up the job leaves the organization. Three days later, the job fails with a generic 401 Unauthorized or 403 Forbidden error, despite the code remaining unchanged.

This is the "Orphaned Token" anti-pattern. When a user account is deactivated, any Personal Access Token (PAT) associated with that account is immediately revoked, silently breaking every downstream automation tied to that identity.

Symptoms

An engineer investigating this failure often sees the following:

• The "It Works on My Machine" trap: The pipeline runs successfully when executed manually by a senior engineer with broad permissions, leading to the false conclusion that the issue is transient or related to "network blips."
• Confusing Logs: Spark logs might show java.io.IOException or com.databricks.backend.common.rpc.DatabricksSqlException with cryptic authentication messages.
• Silent Failures: If the code doesn't explicitly log the full stack trace of the authentication error, the job might just hang while waiting for an API response that will never arrive.

Photo by Lu on Unsplash

1. Finding the Hardcoded Secret

The most common cause is a string literal hidden in a configuration file or a dbutils call. To confirm, check the job configuration in the Databricks workspace.

How to confirm:
Search the codebase for dbutils.secrets.get calls or, worse, raw strings like api_key = "secret123". If you are using Databricks Asset Bundles (DABs), check your databricks.yml for environment variables that might be pulling from local files rather than the Databricks Secret Manager.

How to fix:
Stop referencing secrets directly in your code. Instead, use a Secret Scope. If you are on Databricks, use the Databricks Secret CLI to create a scope and add the secret:

# Create a scope backed by Databricks
databricks secrets create-scope my-app-scope

# Add the secret (do this interactively to avoid shell history logging)
databricks secrets put-secret my-app-scope my-api-key

2. The Identity Mismatch (Personal vs. Service Principal)

Even if you use secret scopes, the job might still be running under the execution context of a user. If the cluster is configured to use "Single User" mode, it inherits the identity of the user who started the cluster.

How to confirm:
Check the Job Run details. Does it list an individual user as the "Run As" identity? In a secure architecture, this should always be a Service Principal.

How to fix:
Create a Service Principal in your cloud provider (e.g., Entra ID, AWS IAM) and grant it the necessary permissions in your Unity Catalog or Hive Metastore. Then, update your Databricks Job definition to use this Service Principal for the run.

If using Databricks Asset Bundles, configure the run_as block in your bundle.yml:

resources:
  jobs:
    my_sync_job:
      name: sync-api-to-delta
      run_as:
        service_principal_name: <application-id-of-service-principal>
      tasks:
        - task_key: main
          spark_python_task:
            python_file: ./src/sync.py

3. Scope ACLs and Permissions

Sometimes the job fails because the Service Principal lacks the "READ" permission on the secret scope itself.

How to confirm:
Run a test script using the Service Principal’s token (or via a job run as the SP) to access the secret:

# Test script to verify secret access
try:
    secret_value = dbutils.secrets.get(scope="my-app-scope", key="my-api-key")
    print("Access successful")
except Exception as e:
    print(f"Access denied: {e}")

How to fix:
Grant the Service Principal access to the scope using the Databricks CLI:

databricks secrets put-acl my-app-scope <service-principal-application-id> READ

4. Environment Variable Injection

If you are using Docker containers or custom Spark configurations, you might be injecting secrets as environment variables in the cluster config. This is dangerous because these values are often visible in the Spark UI "Environment" tab to anyone with "Can View" access to the cluster.

How to fix:
Instead of setting MY_API_KEY in the Spark Config, keep it in the Secret Scope. Update your Python code to fetch it at runtime:

# Use the Databricks utility to fetch the secret at runtime
# This keeps the secret out of the Spark UI environment variables
api_key = dbutils.secrets.get(scope="my-app-scope", key="my-api-key")

Preventing it

To prevent these issues from recurring, implement these systemic safeguards:

1. Enforce "Run As" Policies: Configure your CI/CD pipelines to reject any Databricks Job definition that does not specify a run_as block pointing to a Service Principal.
2. Linting: Use custom scripts in your CI pipeline to scan your Python code for patterns like os.environ.get("SECRET_KEY") or hardcoded strings that look like API keys.
3. Unity Catalog: Move away from legacy secret scopes where possible and leverage Unity Catalog’s external locations and credentials. By using Unity Catalog, you can grant the Service Principal direct access to cloud storage (S3/ADLS/GCS) without ever needing to manage a secret key for the storage layer itself.
4. Audit Logs: Regularly review Databricks Audit Logs for jobs running as individual users, and use that list to target your migration efforts.

By removing the reliance on personal tokens, you move your platform from a state of "fragile convenience" to "hardened automation." The goal is for your pipelines to be identity-agnostic, running with the same permission set regardless of who triggered the deployment.

---

Tags: #databricks #security #dataengineering #cicd

Cover photo by Stephen Hickman on Unsplash.

---

This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.