Deeper into Dataform 2: Other API features

Ben Watson — Sun, 24 May 2026 15:03:00 +0000

In part 1 we looked at the Dataform CompilationResult and WorkflowInvocation objects. Let's follow up with a quick look at some other API endpoints that I find useful.

Workspace management - `delete_workspace` and `create_workspace`

delete_workspace is essential for Dataform instances used by large teams as it allows workspaces to be automatically deleted post-merge, saving the bi-annual "can everyone please let me know which workspaces they're finished with?" message. This can be integrated into CI/CD tools such as GitHub Actions.

from google.cloud import dataform_v1

client = dataform_v1.DataformClient()

workspace_path = client.workspace_path(
    PROJECT_ID,
    REGION,
    REPOSITORY_ID,
    WORKSPACE_ID,
)

try:
    operation = client.delete_workspace(name=workspace_path)

There's also create_workspace - very useful for automatically creating a workspace if a code change is made on a new branch outside of the Dataform UI.

Git operations - `push_git_changes` and `pull_git_changes`

Another set of useful endpoints that allow code changes to be automatically applied outside of the Dataform UI:

from google.cloud import dataform_v1

client = dataform_v1.DataformClient()

workspace_path = client.workspace_path(
    project_id,
    region,
    repository_id,
    workspace_id,
)

# ------------------------------------------------------------
# 1. Pull latest changes from Git into the workspace
# ------------------------------------------------------------
print("Pulling latest Git changes into workspace...")

pull_request = dataform_v1.PullGitChangesRequest(
    name=workspace_path
)

pull_operation = client.pull_git_changes(request=pull_request)
print("Pull initiated:", pull_operation.operation.name)

# Optional: wait for completion (simplified polling)
pull_result = pull_operation.result()
print("Pull completed successfully")

# ------------------------------------------------------------
# (Optional) 2. Make programmatic changes here
# ------------------------------------------------------------
# e.g. update config blocks, inject variables, generate models, format SQL, etc.

print("Applying automated workspace changes...")

# ------------------------------------------------------------
# 3. Push workspace changes back to Git
# ------------------------------------------------------------
print("Pushing workspace changes to Git...")

push_request = dataform_v1.PushGitChangesRequest(
    name=workspace_path
)

push_operation = client.push_git_changes(request=push_request)
push_result = push_operation.result()
print("Push completed successfully")
print("Workspace successfully synced with Git")

Deeper into Dataform 1: Exploring the API

Ben Watson — Sun, 24 May 2026 15:02:20 +0000

Series overview

This series of blog posts is aimed at Dataform users who are looking to explore beyond its core functionality. Among other things we'll dig into Dataform's powerful API, create automated CI/CD processes in GitHub Actions, build pipelines to monitor costs, and modify the config {} block through code.

Dataform's API is the natural starting point for diving into Dataform's more powerful and less-documented capabilities. Understanding of the API allows users to utilise Dataform as the all-powerful hub of an automated modern data platform.

Dataform API key concepts

The API has two objects that serve as essential components of any workflow:

CompilationResult - the compiled state of a Dataform workspace at a specific point in time (the result of building the DAG).
WorkflowInvocation - a single execution of a CompilationResult (the result of running the DAG).

`CompilationResult`

from google.cloud import dataform_v1

client = dataform_v1.DataformClient()

project_id = "my-project"
region = "europe-west2"
repository_id = "analytics"
workspace_id = "dev"

workspace = client.workspace_path(
    project_id,
    region,
    repository_id,
    workspace_id,
)

repository = client.repository_path(
    project_id,
    region,
    repository_id,
)

compilation_result = dataform_v1.CompilationResult(
    workspace=workspace
)

response = client.create_compilation_result(
    parent=repository,
    compilation_result=compilation_result,
)

print(response.name)

The response takes the form projects/<project_id>/locations/<region>/repositories/<repository_id>/compilationResults/<compilation_result_id> (where <compilation_result_id> is a UUID v4).

A CompilationResult object is generated in the Dataform UI each time a code change is detected, although the UI conceals its ID. The ID can be seen in the Executions page in the UI following a run.

The CompilationResult includes information about each job inside the Dataform workspace, allowing users to traverse the DAG through code and extract information such as the config {} block of each file. Each job can then be iterated over via:

request = dataform_v1.QueryCompilationResultActionsRequest(
    name=COMPILATION_RESULT
)
response = client.query_compilation_result_actions(
    request=request
)
for action in response.compilation_result_actions:
    ...

`WorkflowInvocation`

from google.cloud import dataform_v1

client = dataform_v1.DataformClient()

project_id = "my-project"
region = "europe-west2"
repository_id = "analytics"

repository = client.repository_path(
    project_id , region, repository_id 
)

# compilation_result comes from the previous snippet
invocation = dataform_v1.WorkflowInvocation(
    compilation_result=compilation_result.name
)

response = client.create_workflow_invocation(
    parent=repository,
    workflow_invocation=invocation,
)

print(f'{response.name}: {response.state}')

A WorkflowInvocation is generated in the Dataform UI each time an Execution is run:

The WorkflowInvocation includes information about each job that has been executed, allowing users to tie jobs to their BigQuery job IDs, and see the objects created (such as tables or views). Each executed job can be iterated over via:

request = dataform_v1.QueryWorkflowInvocationActionsRequest
    name=WORKFLOW_INVOCATION
)
response = client.query_workflow_invocation_actions(
    request=request
)
for action in response.workflow_invocation_actions:
    ...

Tying them together

The real power of Dataform’s API emerges when these two objects are combined. A typical workflow looks like this:

A workspace is updated (manually or via Git),
A CompilationResult is generated from the workspace,
A WorkflowInvocation is created from that compilation,
BigQuery executes the resulting DAG.

With just these two objects it's possible to automate building and running Dataform DAGs outside of the Dataform UI. This enables easy CI/CD pipelines, which we'll cover in a later post.

DEV Community: Ben Watson

Deeper into Dataform 2: Other API features

Workspace management - delete_workspace and create_workspace

Git operations - push_git_changes and pull_git_changes