Building a Configuration-Driven Databricks Platform on AWS (Part 2)

In the first part, we explored the architecture behind a Databricks platform on AWS and how GitHub, Terraform, and YAML configuration fit together. We also looked at why infrastructure, workspace access, compute, and data permissions have separate responsibilities. In this second part, we’ll put that design into practice.

Before running the deployment workflow, we need to prepare a few things that cloning the repository does not provide: an AWS identity for GitHub Actions, a place to store Terraform state, and credentials to access the Databricks account. These bootstrap dependencies are easy to overlook because the provisioning code expects them to already exist.

This walkthrough follows the working dev/demo path in the Databricks Platform repository, from initial setup and configuration through validation, deployment, and verification.

By the end, we’ll have provisioned the AWS infrastructure, a Databricks workspace, persona entitlements, and approved compute configuration. Unity Catalog data objects and dataset permission requests follow separate workflows, which sit outside the deployment covered here.

1. Prepare the accounts and tools

Start with:

• An AWS account and permission to bootstrap IAM and state storage.
• A Databricks account and an administrator who can establish the deployment identity.
• A GitHub repository where you can configure Actions and Environments.
• Git, Python, and Terraform installed locally.

The Python tooling requires Python 3.10 or later. The current workflows use Python 3.12 and Terraform 1.10.5. Match the workflow version when reproducing its behavior locally.

Clone the repository, or clone your fork:

git clone https://github.com/mysticrenji/databricks-platform.git
cd databricks-platform/AWS/databricks-foundation

python3 -m venv .venv
source .venv/bin/activate
pip install -r tools/environment/requirements.txt

Keep local credentials, generated plans, state, and secret-bearing variable files out of Git. This example uses eu-west-1. Check the current Databricks supported regions before choosing another region, including availability of the features you intend to use.

2. Bootstrap Terraform state storage

Create the S3 bucket for Terraform state before running the workspace workflow. Use a globally unique bucket name, enable versioning and encryption, block public access, and restrict access to the deployment identities and recovery administrators. This bucket is separate from the workspace storage bucket declared later in YAML.

The workflow needs permission to list the appropriate state prefix and read and write state objects. S3 lockfiles also need read, write, and delete permissions on the lock objects. If the state bucket uses a customer-managed KMS key, include the corresponding key permissions. The S3 backend documentation describes these requirements.

The state bucket region may differ from the workspace region; record both accurately. Keep state storage under independent ownership. Removing a development workspace should leave its Terraform history available for investigation and recovery.

3. Establish the GitHub-to-AWS identity

The repository includes an OIDC bootstrap root:

tools/aws-oidc-github-runner/

Despite its name, this root does not create a runner machine. It creates the GitHub OIDC provider and an IAM role that workflows can assume.

From the foundation directory:

cd tools/aws-oidc-github-runner
cp terraform.tfvars.example terraform.tfvars

Edit the file with your region, allowed GitHub subjects, and an existing deployment policy ARN. The deployment policy must cover the enabled AWS modules, state access, and the IAM operations required by the Databricks integration. This bootstrap root attaches the policies you supply; it does not generate a complete deployment policy or attach administrator access by default.

For dev/demo, authorize the subjects for:

dev-demo-plan
dev-demo-apply

Use the exact OIDC subject format configured for your repository, including immutable owner and repository IDs where applicable. The example file contains the source repository’s values, which must be replaced for a fork.

An environment subject is different from a main branch subject. Configure the environment subjects required by the jobs, then restrict the branches allowed to use those GitHub Environments. GitHub explains the subject formats in its AWS OIDC guide.

Using your existing administrative AWS session, create and inspect a bootstrap plan:

terraform init
terraform plan -out=bootstrap.tfplan
terraform show bootstrap.tfplan
terraform apply bootstrap.tfplan

These commands change AWS resources. Run the apply after reviewing the plan.

⚠️ OIDC Collision Warning: If your account already has the GitHub OIDC provider, resolve its ownership before using a root that declares another one. Keep the bootstrap state protected and outside the workspace teardown lifecycle.

Record the resulting role ARN, then return to the foundation directory:

cd ../..

4. Create the Databricks deployment identity

Create an account-level Databricks service principal for the automation. Have an account administrator grant the privileges required for workspace provisioning and establish the identity’s access for subsequent workspace operations. Keep an independent recovery administrator.

Generate an OAuth client secret and record the client ID. Follow the Databricks OAuth M2M setup guide for the credential setup.

The workflow uses:

DATABRICKS_ACCOUNT_ID
DATABRICKS_CLIENT_ID
DATABRICKS_CLIENT_SECRET

Store the client secret in GitHub secrets. The account ID and client ID are configuration values. Successful authentication only proves the identity can sign in. Account operations and workspace operations still require the appropriate authorization.

5. Define the dev/demo deployment

Edit:

platform/environments/dev/demo.yaml

Use your own identifiers and a unique workspace bucket name. This example contains placeholders:

version: 2

workspace:
  name: demo

environment:
  name: dev
  aws_account_id: "123456789012"
  aws_region: eu-west-1
  deployment_role_arn: arn:aws:iam::123456789012:role/github-actions-oidc

databricks:
  account_id: "00000000-0000-0000-0000-000000000000"
  pricing_tier: PREMIUM

network:
  nat_gateway_mode: single
  vpc_cidr: 10.40.0.0/16

  workspace_subnets:
    a:
      availability_zone: eu-west-1a
      cidr_block: 10.40.0.0/20
    b:
      availability_zone: eu-west-1b
      cidr_block: 10.40.16.0/20

  nat_subnets:
    a:
      availability_zone: eu-west-1a
      cidr_block: 10.40.32.0/24

  workspace_subnet_nat_gateway_keys:
    a: a
    b: a

  endpoint_subnets:
    a:
      availability_zone: eu-west-1a
      cidr_block: 10.40.34.0/24
    b:
      availability_zone: eu-west-1b
      cidr_block: 10.40.35.0/24

storage:
  workspace_bucket_name: replace-with-a-unique-dev-demo-bucket

features:
  customer_managed_kms: false
  observability: true

Replace the account ID, role ARN, Databricks account ID, and bucket name before validation.

Network Sizing Strategy: Notice the CIDR breakdown. The overall VPC is given a /16 (65,536 IPs). The workspace subnets are given /20s (4,096 IPs each) to allow for heavy cluster autoscaling, while the NAT and endpoint subnets require much less space and are constrained to /24s (256 IPs each).

This example uses one NAT gateway (nat_gateway_mode: single) and maps both workspace subnets to it. Review that choice against your availability and network requirements. Check the address ranges against networks you may connect later, rather than checking only for overlap inside this document.

Keep dev/demo for this walkthrough. The additional persona entitlement and compute jobs are currently enabled specifically for that target.

6. Review the group and compute inputs

Open:

platform/terraform-inputs/dev/demo.02-account.json

The supplied configuration declares these workspace roles:

• dev-demo-admins: ADMIN
• dev-demo-engineers: USER
• dev-demo-analysts: USER
• dev-demo-ml: USER

The membership maps are empty. Terraform creates the configured groups and assignments without creating membership resources. Review the matching persona and principal configuration under platform/access/. Their names and roles must agree with the account inputs.

Then inspect:

platform/compute/dev.yaml

This defines the approved compute choices. The current persona policies include bounded worker counts and automatic termination. The analytics warehouse is non-serverless and has a bounded cluster count and automatic stop.

Before deploying, check the approved runtimes and node types. The live compute preflight verifies availability and rejects unavailable choices.

💸 Cost Warning: Creating cluster policies starts no clusters, but applying the SQL warehouse configuration can start billable compute. Include it in your deployment cost review.

7. Validate before configuring live execution

Run the environment and access tests:

python3 -m unittest discover -s tools/environment/tests -v
python3 -m unittest discover -s tools/databricks-access/tests -v

Validate the completed environment configuration:

python3 tools/environment/environment.py validate platform/environments/dev/demo.yaml

These checks do not deploy resources. The validation workflow also tests the generated configuration through mocked Terraform providers. Let those checks pass on the pull request before merging deployment configuration.

A clean result means the configuration satisfies the tested rules. Live IAM permissions, service quotas, bucket availability, and runtime availability remain separate checks.

8. Configure the GitHub Environment pair

In repository settings, create:

dev-demo-plan
dev-demo-apply

Set matching values in both environments:

• ENVIRONMENT_CONFIG_PATH: AWS/databricks-foundation/platform/environments/dev/demo.yaml
• AWS_DEPLOYMENT_ROLE_ARN: bootstrap role ARN matching YAML
• DATABRICKS_ACCOUNT_ID: account ID matching YAML
• DATABRICKS_CLIENT_ID: deployment service principal client ID
• TF_STATE_BUCKET: existing state bucket
• TF_STATE_REGION: state bucket region
• TF_INFRA_STATE_KEY: databricks/dev/demo/01-infra.tfstate
• TF_ACCOUNT_STATE_KEY: databricks/dev/demo/02-account.tfstate
• TF_DATA_ACCESS_STATE_KEY: databricks/dev/demo/03-data-access.tfstate
• TF_WORKSPACE_STATE_KEY: databricks/dev/demo/04-workspace.tfstate

Add DATABRICKS_CLIENT_SECRET as a secret in both environments.

The feature and permissions runners enforce their own state paths. Adding variables for the data and workspace states does not make the provisioning workflow execute every root.

The repository also provides a GitHub Environment setup helper. It can preview and create these settings, but it does not configure AWS trust or environment approval rules.

Configure branch restrictions and the approval policy appropriate to the target. Dev/demo has a documented single-administrator exception. Set ENVIRONMENT_YAML_APPLY_ENABLED=true in dev-demo-apply when ready to enable that path. Other targets retain their protected-environment review requirements, and dataset permission apply has its own separate requirements.

9. Run a plan and understand the first-deployment limit

Merge the reviewed configuration and workflows to main. Open Actions, select Provision environment YAML, and run it with:

Branch: main
target_environment: dev
target_workspace: demo
deploy: false

The infrastructure job produces a plan without applying it.

🚧 Expected Failure: On a completely new deployment, the later account job cannot finish planning until infrastructure state contains the required outputs. Feature and compute plans similarly require their upstream resources. This dependency failure is expected on the first plan-only run.

Inspect the infrastructure plan and resolve any unexpected resources, names, network settings, or replacements before enabling deployment. The next deployment run creates fresh plans. It does not reuse a plan from this earlier run.

10. Deploy through the manual workflow

For dev/demo, a repository administrator starts the workflow from main with:

target_environment: dev
target_workspace: demo
deploy: true
confirm_environment: dev/demo

The workflow checks administrator permission, the target confirmation, and the apply opt-in before creating infrastructure. It checks authorization again at the relevant apply stages.

Each resource stage plans and then applies its exact saved plan when the checks pass. Downstream stages stop if a dependency fails.

The dev/demo exception allows this progression without a separate reviewer pause for each plan. If your deployment process requires independent approval of every plan, establish that protection before using the workflow for a broader target. Saved plan manifests bind the deployment to its source, inputs, backend, and relevant state. Changed inputs or expired artifacts require a fresh plan.

Provisioning Pipeline

De-provisioning Pipeline

11. Verify the platform as a user

After deployment, inspect the workspace URL and ID produced by the account layer. Have the identity administrator add test users to the appropriate account groups through the established membership process. Use one test identity per persona before onboarding a wider team.

Check that:

• Engineers can use their approved engineering policy.
• ML users can use their approved ML policy.
• Analysts can use the analytics SQL warehouse.
• Analysts receive no cluster-policy grant from this configuration.
• Unrestricted cluster creation remains disabled for the intended personas.

Inspect other groups and direct grants too. A user may have broader effective access than the persona alone describes.

Do not expect a successful query against business data yet. This provisioning path does not create the Unity Catalog objects or apply the separate data grants. For data onboarding, establish the objects through their owning root, then use the permissions workflow for approved access to those existing objects. Preserve one owner for each grant set and resource.

12. Repeat the process and plan retirement

For another development workspace, add its complete environment YAML, matching supplemental inputs, state namespace, GitHub Environment pair, and AWS OIDC subjects. Infrastructure and account provisioning support additional development and staging targets. The automatic persona feature and compute stages remain dev/demo-specific until deliberately extended.

Before retiring the workspace, inspect all downstream states. Remove separately managed data grants and objects before removing the workspace they depend on.

The deprovisioning workflow has an independent destroy opt-in and target confirmation. Storage is retained by default. Permanent S3 deletion requires additional confirmation and checks, while state history remains preserved.

The first successful deployment is a useful milestone. The stronger proof is being able to make a small configuration change, review its effect, deploy through the same controls, and retire the workspace without losing track of its resources or data.

Conclusion

We started with a repository and worked through the steps needed to turn its configuration into a Databricks platform on AWS. That included preparing authentication and Terraform state, validating the YAML, and running the deployment workflow to create the infrastructure, workspace, persona entitlements, and approved compute configuration.

The value of this approach goes beyond the first deployment. Changes can be reviewed in GitHub, checked before they reach the cloud, and applied through saved Terraform plans. Separate states give each layer clear ownership and make the platform easier to maintain as it grows.

There is still a distinction between having a workspace ready and having data ready to use. Unity Catalog objects and dataset permissions follow their own workflows. With the foundation in place, those become the next steps toward giving teams access to the data they need.