DEV Community: Renjith Ravindranathan

Building a Configuration-Driven Databricks Platform on AWS (Part 1)

Renjith Ravindranathan — Thu, 22 Oct 2026 16:46:36 +0000

If your platform team is bottlenecked by manual AWS console clicks and fragmented Terraform scripts every time someone needs a new Databricks workspace, you have a scaling problem.

A request for another Databricks workspace involves more than creating the workspace itself. Someone must choose the AWS network, configure storage and IAM, assign access, decide which compute options teams can use, and make sure the next workspace follows the same rules.

Terraform can describe those resources. The harder problem is deciding how people request changes, how those requests become Terraform inputs, and how the changes reach AWS and Databricks.

A configuration-driven approach gives those decisions a consistent, repeatable path:

This article explains that design using the Databricks Platform Engineering repository. Part two walks through setting up its development deployment from zero.

The repository has working development automation, with additional infrastructure and account provisioning for staging. Its production files are configuration scaffolds. That distinction matters when deciding which parts to reuse.

Make the workspace request explicit

Suppose a platform team needs a development workspace for data engineers, analysts, and machine learning users. The request includes several decisions:

Which AWS account and region will host it?
Which address ranges belong to its network?
Which S3 bucket will hold workspace storage?
Which groups receive workspace access?
Which compute policies and SQL warehouses are available?

These decisions can live in reviewed configuration files. In this implementation, each deployment has two identifiers:

version: 2

environment:
  name: dev

workspace:
  name: demo

This is an excerpt; the complete document also contains AWS, Databricks, network, storage, and feature settings. The pair dev/demo identifies the deployment, determining resource naming, Terraform state paths, and GitHub Environment names. The cloud workspace name becomes dev-demo.

That gives the platform a repeatable way to distinguish dev/demo, dev/sandbox, and staging/shared. A configuration filename alone does not establish that identity. Adding another workspace means defining another deployment within supported boundaries, then running the same provisioning machinery.

Give configuration a clear owner

Configuration becomes difficult to govern when every file can change every setting. A strong developer experience requires separating platform decisions from team requests:

platform/
  environments/
  compute/
  access/
  terraform-inputs/

workloads/

Platform administrators own the environment definitions, approved compute settings, personas, and access boundaries. Workload authors request a persona and access to existing data through workloads/. They do not choose arbitrary infrastructure settings or create their own compute policies through those requests.

For example, the platform can define an analyst persona with SQL access. A team can request that persona without deciding the SQL warehouse size or receiving unrestricted cluster creation.

Some settings still use supplemental JSON under platform/terraform-inputs/. Those inputs are allowlisted and cannot override settings owned by environment YAML. This is a practical way to extend the configuration model gradually. A setting should move into YAML when its meaning and validation rules are clear.

Use Python to validate the request

YAML is easy to read, but readability does not make it safe to deploy.

Consider a configuration that contains valid YAML syntax but assigns overlapping subnet ranges. Another might name one AWS account while pointing to a deployment role in a different account. Both should fail before cloud authentication.

The Python adapter checks the document against a schema and applies additional semantic checks. The repository rejects unknown fields, duplicate keys, YAML aliases, invalid types, overlapping subnets, and role/account mismatches. Once validation succeeds, Python renders the inputs consumed by Terraform.

The responsibilities stay clear:

YAML: declares the requested configuration
Python: validates and translates that configuration
Terraform: manages the resources
Actions: controls execution

The runners also use isolated copies of the Terraform source. Local variable files, backend files, cached provider directories, and Terraform environment overrides do not silently become deployment inputs. That makes the committed configuration a more dependable description of what the workflow will use.

⚠️ Validation Limitations: Offline validation still has limits. It cannot prove that an S3 bucket name is available, that the deployment identity has every required permission, or that an AWS quota allows another resource. Those checks need live planning or service calls.

Split Terraform state by responsibility

One Terraform state for the whole platform is easy to begin with, but as the platform grows, that state accumulates resources with different owners and different dependencies.

This implementation uses separate execution roots:

01-infra owns AWS networking, IAM, storage, optional encryption, and monitoring.
02-account owns Databricks workspace creation, account groups, workspace assignments, and budgets.
02-workspace-access owns dev/demo workspace feature entitlements.
03-data-access owns Unity Catalog integration, storage credentials, locations, catalogs, schemas, and volumes.
04-workspace owns workspace configuration, including policies, warehouses, and jobs.
05-permissions owns approved grants on existing data objects.

Each root owns a separate state file. For example:

databricks/dev/demo/01-infra.tfstate
databricks/dev/demo/02-account.tfstate
databricks/dev/demo/02-workspace-access.tfstate
databricks/dev/demo/04-workspace.tfstate

The state namespace contains both deployment identifiers. The runner rejects keys outside the selected namespace. The account root reads the infrastructure outputs and verifies their environment and workspace identity. It does not simply trust that whichever state file it receives belongs to the requested deployment.

Separate state also preserves a useful boundary between Databricks account APIs and workspace APIs. Workspace creation uses the account provider. Workspace configuration uses the workspace provider after the workspace exists.

The tradeoff is orchestration: these roots have dependencies, so they must run in the right order.

Follow the actual dependency chain

The current dev/demo provisioning path is:

graph LR
    A[01-infra] --> B[02-account]
    B --> C[02-workspace-access]
    C --> D[04-workspace compute]

The infrastructure root creates the AWS foundation. The account root uses its outputs to create the workspace and configured groups. The entitlement root then configures which workspace features those groups can use. Finally, the compute runner creates the approved policies, SQL warehouse configuration, and associated permissions.

This path does not require datasets or a metastore to exist. The dev/demo compute runner deliberately avoids a dependency on 03-data-access when it has no Unity Catalog volume references.

Data setup follows its own dependency path. If compute policies reference a Unity Catalog volume for logs or an init script, that storage must exist first. The important rule is to express real dependencies. Putting every task into one long workflow can create unnecessary prerequisites, while running everything independently can leave later stages without the IDs and resources they need.

Treat access as several decisions

“Give the engineering team access” is too vague for automation. There are several separate questions:

Does the group exist at account level?
Is it assigned to this workspace?
Which workspace features can it use?
Which compute resources can it use?
Which data can it read or change?

The repository gives these decisions different owners.

For dev/demo, 02-account creates the admin, engineer, analyst, and ML groups and assigns their workspace roles. Group membership remains external. The entitlement root controls workspace, SQL, and unrestricted compute-creation capabilities.

The compute root grants access to approved policies or warehouses. Engineers receive access to engineering-standard; ML users receive access to ml-standard. Analysts receive access to the analytics SQL warehouse without a cluster-policy grant.

Unity Catalog grants remain separate. A warehouse permission does not grant permission to read a table, and a workspace role does not replace catalog and schema privileges. Effective access also depends on other memberships, direct grants, workspace defaults, and administrator privileges. Testing only the declared persona configuration can miss permissions granted through another route.

Keep pull request checks credential-free

A configuration pull request should receive useful feedback before anyone starts a deployment.

The validation workflow runs Python tests, validates development and staging templates, renders generated inputs, checks Terraform formatting, and runs mocked Terraform tests. It does not need AWS or Databricks credentials.

That allows the repository to test invalid configuration and ownership rules without giving pull request code access to the platform. Live planning is a different operation. It can read state and query services, so it runs through manual workflows with the appropriate identities and target settings.

A plan-only run also depends on existing upstream resources. For a new deployment, the account plan cannot use infrastructure outputs that have not been created yet. Feature planning cannot resolve groups that do not exist.

💡 The Reality of Planning: “Plan only” means no changes are applied. It does not mean Terraform can plan every dependent layer before the first layer exists.

Authenticate to AWS and Databricks separately

The workflow talks to two control planes, each with its own identity.

For AWS, GitHub Actions uses OpenID Connect to assume an IAM role and obtain temporary credentials. The role trust policy restricts which repository and GitHub Environment subjects can assume it. GitHub documents this flow in its AWS OIDC guide.

The trust policy and the role permissions answer different questions:

The trust policy decides who may assume the role.
The permission policies decide what the role may do.

For Databricks, the supplied workflow uses a service principal with OAuth machine-to-machine credentials. Its client secret lives in GitHub Environment secrets.

Databricks also supports workload identity federation, but that is not the authentication path configured by these workflows. Changing to federation requires changes to the integration, rather than simply deleting the existing secret. See the Databricks federation documentation. The current setup therefore avoids stored AWS access keys while still requiring management of a Databricks OAuth secret.

Apply the plan that was produced

The deployment workflow saves the Terraform plan and applies that saved file.

Its manifest binds the plan to the source, commit, configuration, generated inputs, backend, and plan hash. The account layer also binds upstream state metadata. The compute runner adds checks against the relevant state snapshots and its expected ownership boundaries. If those inputs change, the workflow requires a new plan.

Approval policy remains a separate concern. Most supported targets use protected GitHub Environments. Dev/demo has a limited single-administrator exception: a repository administrator must manually start the workflow from main, enable deployment, provide the exact target confirmation, and satisfy the apply opt-in. That exception does not provide a human pause to approve every individual plan. It authorizes the checked workflow to plan and apply through the selected run.

Protect state and plan artifacts

State isolation reduces accidental overlap, but state storage still needs protection.

The S3 backend uses locking to coordinate state mutations. S3 lockfiles are enabled with use_lockfile, and bucket versioning supports recovery from accidental changes. HashiCorp documents both in its S3 backend guide.

GitHub Actions also serializes provisioning and deprovisioning for the same environment/workspace pair. Operators running Terraform elsewhere must still coordinate with that workflow. Plans deserve restricted access too. Marking a Terraform value as sensitive does not remove it from state or saved plans. See HashiCorp’s sensitive data guidance.

The repository retains deployment artifacts for one day. Short retention reduces exposure, but it does not replace access controls.

Include cost and retirement in the design

A cluster policy defines the settings a cluster may use. Creating that policy does not start a cluster. A SQL warehouse apply can start billable compute. AWS networking, storage, and monitoring can also incur charges without anyone running a notebook.

The platform configuration therefore includes bounded compute choices and automatic shutdown settings. Its Databricks budget alerts are notifications, not hard spending caps or AWS budgets.

Retirement needs equally explicit rules. Dependent compute and permissions must be removed before the groups, workspace, and infrastructure they rely on. The provisioning workflow does not automatically own every data resource or grant, so teardown checks for remaining downstream state.

Storage is retained by default. Permanent storage deletion requires separate confirmation and checks, while Terraform backend history remains preserved.

A useful starting point is one workspace with a complete lifecycle: validated configuration, clear state ownership, a controlled deployment, persona verification, and a documented retirement path. Once that works, another workspace can reuse the same process with its own identity and reviewed settings.

How is your team currently handling Databricks workspace sprawl? Do you use a similar config-driven approach? Let me know in the comments, or check out the full implementation in the GitHub repo.

Static Code Analysis using Semgrep App

Renjith Ravindranathan — Thu, 20 Oct 2022 10:14:00 +0000

In this short article, we will explore the static code analysis capability of Semgrep. Semgrep is a fast, open-source, static analysis tool that supports most modern languages. It works on a set of rules and rules are customizable as well according to your requirements. The tool is available in the CLI (OSS) version as well as in the SaaS version(Semgrep App). Also, it is very flexible to integrate with your CI pipelines. Let's look at integrating Semgrep with GitLab.

Pre-requisites
Gitlab account
Semgrep App Account

Let's build
Semgrep works in such a way that the whole code analysis is done in the agents aka build machines and no sensitive data is being sent to the cloud. The only requirement is to generate an API TOKEN from Semgrep App and pass it to your GitLab pipelines, so they can talk with each other.

You can create an API token from this link

Once you have the token generated, the next step is to add the API TOKEN to the GitLab Variables — SEMGREP_APP_TOKEN in your project

Let's look at the GitLab pipelines. The pipeline has been just to do the static code analysis for illustration purposes.

stages:
  - build

# Semgrep static code analysis
semgrep:
  stage: build
  # A Docker image with Semgrep installed.
  image: returntocorp/semgrep
  # Run the "semgrep ci" command on the command line of the docker image.
  script: semgrep ci

  rules:
  # Scan changed files in MRs, (diff-aware scanning):
  - if: $CI_MERGE_REQUEST_IID

  # Scan mainline (default) branches and report all findings.
  - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

  variables:
    # Connect to Semgrep App through your SEMGREP_APP_TOKEN.
    # Generate a token from Semgrep App > Settings
    # and add it as a variable in your GitLab CI/CD project settings.
    SEMGREP_APP_TOKEN: $SEMGREP_APP_TOKEN

Result
For scanning, I am utilizing an old repository hosted here that consists of Python scripts, Dockerfiles, and Kubernetes Manifests. The outcome is quite nice, as Semgrep was able to catch a few common mistakes in the development. The dashboard shows an overall summary of the findings.

Let's zoom in on one of the findings run-as-non-root . Typically this message is raised when you have allowed your container to be run as root or in order words you have not specified the user by which the container is executed.

There is a rule editor which shows the pre-defined rules, against which your code is compared. You may go around it to see, how it evaluates the condition.

Conclusion
Semgrep is a good tool to integrate with and it can be integrated with major CI/CD engines like GitLab, GitHub Actions, Azure DevOps, Jenkins, CircleCI, Bitbucket Pipelines, etc. It is worth checking out if it suits your organization's requirements. That’s all for now. Hope you find the article useful and feedbacks are always welcome. Cheers.

References
GitLab Repo
Semgrep Docs

In case of any queries, please feel to connect me via the below links

LinkedIn
Twitter
Medium

Essential plugins for Kubectl CLI

Renjith Ravindranathan — Thu, 20 Oct 2022 06:57:17 +0000

This is a short article about handy plugins that can be used along with kubectl CLI , which helps you to can reduce a few of your operational tasks from an admin perspective. Basically, we can extend the capabilities kubectl with the power of plugins, like checking the deprecations of components in the cluster or SSHing to the nodes.

We can install plugins using a plugin manager called krew and there are quite a few available on the website listed here as well. Let's look at how we can get started with plugins :)

Setup
The plugins are added using krew and the primary requirement is to install krew them in the system. You can use the below code snippet in Mac/Linux to install it

(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

Once you have krew it in the system, now it is time to install the additional plugins.

Plugins
There are many plugins available in the OSS GitHub repos, but below are some of my personal favorites.

1. Kubepug
KubePug/Deprecations is intended to be a kubectl plugin, which:

Downloads a swagger.json from a specific Kubernetes version
Parses this Json finding deprecation notices
Verifies the current kubernetes cluster or input files checking whether exists objects in this deprecated API Versions, allowing the user to check before migrating

kubectl krew install deprecations
#Usage
kubepug --k8s-version=v1.18.6 # Will verify the current context against v1.18.6 swagger.json

2. net-forward

Simple plugin that lest you create a local port listener on your personal machine that redirects to an arbitrary TCP service that the Cluster can see. This is similar to kubectl port-forward without the restriction that forwarding can only use Pods or Services as destinations.

kubectl krew install net-forward

#Usage
kubectl net-forward -i 169.254.169.254 -p 3389 -l 3389
I personally use this a lot to debug issues related to environment specific inside the Windows/Linux nodes.

Please feel to try out the plugins on the krew website also, as there are other plugins that could be beneficial to your work.
Hope this info was helpful and thanks for reading the article.!

References
Kubepug
net-forward
Krew

In case of any queries, please feel to connect me via the below links

LinkedIn
Twitter
Medium