Building a Configuration-Driven Databricks Platform on AWS (Part 1)

If your platform team is bottlenecked by manual AWS console clicks and fragmented Terraform scripts every time someone needs a new Databricks workspace, you have a scaling problem.

A request for another Databricks workspace involves more than creating the workspace itself. Someone must choose the AWS network, configure storage and IAM, assign access, decide which compute options teams can use, and make sure the next workspace follows the same rules.

Terraform can describe those resources. The harder problem is deciding how people request changes, how those requests become Terraform inputs, and how the changes reach AWS and Databricks.

A configuration-driven approach gives those decisions a consistent, repeatable path:

This article explains that design using the Databricks Platform Engineering repository. Part two walks through setting up its development deployment from zero.

The repository has working development automation, with additional infrastructure and account provisioning for staging. Its production files are configuration scaffolds. That distinction matters when deciding which parts to reuse.

Make the workspace request explicit

Suppose a platform team needs a development workspace for data engineers, analysts, and machine learning users. The request includes several decisions:

• Which AWS account and region will host it?
• Which address ranges belong to its network?
• Which S3 bucket will hold workspace storage?
• Which groups receive workspace access?
• Which compute policies and SQL warehouses are available?

These decisions can live in reviewed configuration files. In this implementation, each deployment has two identifiers:

version: 2

environment:
  name: dev

workspace:
  name: demo

This is an excerpt; the complete document also contains AWS, Databricks, network, storage, and feature settings. The pair dev/demo identifies the deployment, determining resource naming, Terraform state paths, and GitHub Environment names. The cloud workspace name becomes dev-demo.

That gives the platform a repeatable way to distinguish dev/demo, dev/sandbox, and staging/shared. A configuration filename alone does not establish that identity. Adding another workspace means defining another deployment within supported boundaries, then running the same provisioning machinery.

Give configuration a clear owner

Configuration becomes difficult to govern when every file can change every setting. A strong developer experience requires separating platform decisions from team requests:

platform/
  environments/
  compute/
  access/
  terraform-inputs/

workloads/

Platform administrators own the environment definitions, approved compute settings, personas, and access boundaries. Workload authors request a persona and access to existing data through workloads/. They do not choose arbitrary infrastructure settings or create their own compute policies through those requests.

For example, the platform can define an analyst persona with SQL access. A team can request that persona without deciding the SQL warehouse size or receiving unrestricted cluster creation.

Some settings still use supplemental JSON under platform/terraform-inputs/. Those inputs are allowlisted and cannot override settings owned by environment YAML. This is a practical way to extend the configuration model gradually. A setting should move into YAML when its meaning and validation rules are clear.

Use Python to validate the request

YAML is easy to read, but readability does not make it safe to deploy.

Consider a configuration that contains valid YAML syntax but assigns overlapping subnet ranges. Another might name one AWS account while pointing to a deployment role in a different account. Both should fail before cloud authentication.

The Python adapter checks the document against a schema and applies additional semantic checks. The repository rejects unknown fields, duplicate keys, YAML aliases, invalid types, overlapping subnets, and role/account mismatches. Once validation succeeds, Python renders the inputs consumed by Terraform.

The responsibilities stay clear:

• YAML: declares the requested configuration
• Python: validates and translates that configuration
• Terraform: manages the resources
• Actions: controls execution

The runners also use isolated copies of the Terraform source. Local variable files, backend files, cached provider directories, and Terraform environment overrides do not silently become deployment inputs. That makes the committed configuration a more dependable description of what the workflow will use.

⚠️ Validation Limitations: Offline validation still has limits. It cannot prove that an S3 bucket name is available, that the deployment identity has every required permission, or that an AWS quota allows another resource. Those checks need live planning or service calls.

Split Terraform state by responsibility

One Terraform state for the whole platform is easy to begin with, but as the platform grows, that state accumulates resources with different owners and different dependencies.

This implementation uses separate execution roots:

• 01-infra owns AWS networking, IAM, storage, optional encryption, and monitoring.
• 02-account owns Databricks workspace creation, account groups, workspace assignments, and budgets.
• 02-workspace-access owns dev/demo workspace feature entitlements.
• 03-data-access owns Unity Catalog integration, storage credentials, locations, catalogs, schemas, and volumes.
• 04-workspace owns workspace configuration, including policies, warehouses, and jobs.
• 05-permissions owns approved grants on existing data objects.

Each root owns a separate state file. For example:

databricks/dev/demo/01-infra.tfstate
databricks/dev/demo/02-account.tfstate
databricks/dev/demo/02-workspace-access.tfstate
databricks/dev/demo/04-workspace.tfstate

The state namespace contains both deployment identifiers. The runner rejects keys outside the selected namespace. The account root reads the infrastructure outputs and verifies their environment and workspace identity. It does not simply trust that whichever state file it receives belongs to the requested deployment.

Separate state also preserves a useful boundary between Databricks account APIs and workspace APIs. Workspace creation uses the account provider. Workspace configuration uses the workspace provider after the workspace exists.

The tradeoff is orchestration: these roots have dependencies, so they must run in the right order.

Follow the actual dependency chain

The current dev/demo provisioning path is:

graph LR
    A[01-infra] --> B[02-account]
    B --> C[02-workspace-access]
    C --> D[04-workspace compute]

The infrastructure root creates the AWS foundation. The account root uses its outputs to create the workspace and configured groups. The entitlement root then configures which workspace features those groups can use. Finally, the compute runner creates the approved policies, SQL warehouse configuration, and associated permissions.

This path does not require datasets or a metastore to exist. The dev/demo compute runner deliberately avoids a dependency on 03-data-access when it has no Unity Catalog volume references.

Data setup follows its own dependency path. If compute policies reference a Unity Catalog volume for logs or an init script, that storage must exist first. The important rule is to express real dependencies. Putting every task into one long workflow can create unnecessary prerequisites, while running everything independently can leave later stages without the IDs and resources they need.

Treat access as several decisions

“Give the engineering team access” is too vague for automation. There are several separate questions:

1. Does the group exist at account level?
2. Is it assigned to this workspace?
3. Which workspace features can it use?
4. Which compute resources can it use?
5. Which data can it read or change?

The repository gives these decisions different owners.

For dev/demo, 02-account creates the admin, engineer, analyst, and ML groups and assigns their workspace roles. Group membership remains external. The entitlement root controls workspace, SQL, and unrestricted compute-creation capabilities.

The compute root grants access to approved policies or warehouses. Engineers receive access to engineering-standard; ML users receive access to ml-standard. Analysts receive access to the analytics SQL warehouse without a cluster-policy grant.

Unity Catalog grants remain separate. A warehouse permission does not grant permission to read a table, and a workspace role does not replace catalog and schema privileges. Effective access also depends on other memberships, direct grants, workspace defaults, and administrator privileges. Testing only the declared persona configuration can miss permissions granted through another route.

Keep pull request checks credential-free

A configuration pull request should receive useful feedback before anyone starts a deployment.

The validation workflow runs Python tests, validates development and staging templates, renders generated inputs, checks Terraform formatting, and runs mocked Terraform tests. It does not need AWS or Databricks credentials.

That allows the repository to test invalid configuration and ownership rules without giving pull request code access to the platform. Live planning is a different operation. It can read state and query services, so it runs through manual workflows with the appropriate identities and target settings.

A plan-only run also depends on existing upstream resources. For a new deployment, the account plan cannot use infrastructure outputs that have not been created yet. Feature planning cannot resolve groups that do not exist.

💡 The Reality of Planning: “Plan only” means no changes are applied. It does not mean Terraform can plan every dependent layer before the first layer exists.

Authenticate to AWS and Databricks separately

The workflow talks to two control planes, each with its own identity.

For AWS, GitHub Actions uses OpenID Connect to assume an IAM role and obtain temporary credentials. The role trust policy restricts which repository and GitHub Environment subjects can assume it. GitHub documents this flow in its AWS OIDC guide.

The trust policy and the role permissions answer different questions:

• The trust policy decides who may assume the role.
• The permission policies decide what the role may do.

For Databricks, the supplied workflow uses a service principal with OAuth machine-to-machine credentials. Its client secret lives in GitHub Environment secrets.

Databricks also supports workload identity federation, but that is not the authentication path configured by these workflows. Changing to federation requires changes to the integration, rather than simply deleting the existing secret. See the Databricks federation documentation. The current setup therefore avoids stored AWS access keys while still requiring management of a Databricks OAuth secret.

Apply the plan that was produced

The deployment workflow saves the Terraform plan and applies that saved file.

Its manifest binds the plan to the source, commit, configuration, generated inputs, backend, and plan hash. The account layer also binds upstream state metadata. The compute runner adds checks against the relevant state snapshots and its expected ownership boundaries. If those inputs change, the workflow requires a new plan.

Approval policy remains a separate concern. Most supported targets use protected GitHub Environments. Dev/demo has a limited single-administrator exception: a repository administrator must manually start the workflow from main, enable deployment, provide the exact target confirmation, and satisfy the apply opt-in. That exception does not provide a human pause to approve every individual plan. It authorizes the checked workflow to plan and apply through the selected run.

Protect state and plan artifacts

State isolation reduces accidental overlap, but state storage still needs protection.

The S3 backend uses locking to coordinate state mutations. S3 lockfiles are enabled with use_lockfile, and bucket versioning supports recovery from accidental changes. HashiCorp documents both in its S3 backend guide.

GitHub Actions also serializes provisioning and deprovisioning for the same environment/workspace pair. Operators running Terraform elsewhere must still coordinate with that workflow. Plans deserve restricted access too. Marking a Terraform value as sensitive does not remove it from state or saved plans. See HashiCorp’s sensitive data guidance.

The repository retains deployment artifacts for one day. Short retention reduces exposure, but it does not replace access controls.

Include cost and retirement in the design

A cluster policy defines the settings a cluster may use. Creating that policy does not start a cluster. A SQL warehouse apply can start billable compute. AWS networking, storage, and monitoring can also incur charges without anyone running a notebook.

The platform configuration therefore includes bounded compute choices and automatic shutdown settings. Its Databricks budget alerts are notifications, not hard spending caps or AWS budgets.

Retirement needs equally explicit rules. Dependent compute and permissions must be removed before the groups, workspace, and infrastructure they rely on. The provisioning workflow does not automatically own every data resource or grant, so teardown checks for remaining downstream state.

Storage is retained by default. Permanent storage deletion requires separate confirmation and checks, while Terraform backend history remains preserved.

A useful starting point is one workspace with a complete lifecycle: validated configuration, clear state ownership, a controlled deployment, persona verification, and a documented retirement path. Once that works, another workspace can reuse the same process with its own identity and reviewed settings.

How is your team currently handling Databricks workspace sprawl? Do you use a similar config-driven approach? Let me know in the comments, or check out the full implementation in the GitHub repo.