DEV Community: Sami Chibani

CI/CD in the Era of AI and Platform Engineering: A Deep Dive into Dagger CI (Part 4)

Sami Chibani — Fri, 27 Mar 2026 00:26:11 +0000

Part 4: The AI-Native CI/CD Stack: Agents, Modules, and Spec-Driven Development

Fixed pipelines for speed and reliability. AI agents to write them and fix them when they break.

In Part 1 we built pipelines as real code. In Part 2 we decoupled them from infrastructure. In Part 3 we built AcmeCorp's private module library (acme-backend, acme-frontend, and acme-deploy) that wraps public daggerverse modules with organization-specific compliance, naming, and security.

Now let's talk about where AI actually belongs in CI/CD, and where it doesn't.

The thesis is simple: AI doesn't replace the pipeline. It writes the pipeline and fixes it when it breaks. The pipeline itself stays fixed, deterministic, and fast.

What Is a Dagger Agent?

Just as container primitives allow us to build CI pipelines, Dagger introduces an LLM() primitive that lets you create agents the same way you'd call any other pipeline function. Under the hood, dag.llm() connects to any supported model (Claude, GPT, Gemini) and gives you a composable builder to layer on system prompts, environment bindings, and tool access.

What makes this powerful is the tool story. Any Dagger module, including the same private modules we built in Part 3, can be exposed as MCP tools that the agent calls at runtime. Your acme-deploy module becomes a cloud_run tool. Your acme-backend module becomes build and test tools. You can also attach any local MCP server (a language linter, a CLI wrapper, a documentation server) alongside those module tools, giving the agent both your custom CI abstractions and third-party capabilities in a single environment.

The result is a tight synergy between modules and agents: modules are the typed, testable building blocks; agents are the orchestration layer that composes them through natural language. You don't choose between writing pipelines and using AI. You write modules once, and agents compose them for you.

LLM provider required. The dag.llm() primitive needs access to a language model. Dagger detects your provider from environment variables — set one of:

Provider API Key Variable Model Variable

Anthropic (Claude) ANTHROPIC_API_KEY ANTHROPIC_MODEL

OpenAI (GPT) OPENAI_API_KEY OPENAI_MODEL

Google (Gemini) GEMINI_API_KEY GEMINI_MODEL

Local (Ollama) OLLAMA_HOST (defaults to http://localhost:11434) OLLAMA_MODEL

In CI, add the key as a repository secret and pass it via env:. Locally, export the variable in your shell before running dagger call. If no provider is detected, agent functions will fail at runtime with a clear error message.

Provider	API Key Variable	Model Variable
Anthropic (Claude)	`ANTHROPIC_API_KEY`	`ANTHROPIC_MODEL`
OpenAI (GPT)	`OPENAI_API_KEY`	`OPENAI_MODEL`
Google (Gemini)	`GEMINI_API_KEY`	`GEMINI_MODEL`
Local (Ollama)	`OLLAMA_HOST` (defaults to `http://localhost:11434`)	`OLLAMA_MODEL`

The Problem With Setting Up CI

We solved the YAML problem in Part 1. Pipelines are real code now. And in Part 3, we went further: toolchains let you install AcmeCorp's private modules as zero-code CI, with dagger check running all your checks from a single dagger.json. No SDK, no .dagger/ directory, no pipeline code.

But there's still a bottleneck: configuring that setup requires knowing the module library. AcmeCorp's platform team maintains a growing set of private modules: acme-backend, acme-frontend, acme-deploy. Each module has its own @check functions, its own parameters, its own DefaultPath conventions. Knowing which modules to install as toolchains, which customizations to add for a monorepo layout, and how to wire the deployment step in GitHub Actions still requires familiarity with the internal module library.

What if you could point an AI agent at your private modules and your source code, and have it generate the complete toolchain setup and CI workflow for you?

Generating the Setup With Daggie

Daggie is a Dagger CI specialist agent. It reads module source code, understands their APIs, and generates the right toolchain configuration for your project. You give it your source directory and the Git URL of your module repository. Daggie discovers all available modules inside it and picks the ones relevant to the assignment.

Let's pick up from where we left off in Part 3. We're in the dagger-ci-demo monorepo (FastAPI backend + Angular frontend), and AcmeCorp's private modules live at github.com/telchak/acme-dagger-modules. AcmeCorp's coding agents (Monty, Angie, Daggie) live at github.com/telchak/daggerverse.

If you still have local changes from Part 3, you can stash them with git stash -u or simply delete the repo and clone it fresh — we want a clean starting point with no existing Dagger configuration.

First, initialize Dagger in the project and write the assignment file:

dagger init

cat > daggie-assignment.md << 'EOF'
Set up this monorepo with toolchains from the module library.
The backend (backend/) is FastAPI/Python, the frontend (frontend/) is Angular.
Install acme-backend, acme-frontend, and acme-deploy as toolchains
with the right source path customizations for this monorepo layout.
Also generate a .github/workflows/ci.yml with dagger check for PRs,
and a deploy step for Cloud Run (backend) and Firebase (frontend) on main.
On check failure, call Monty on the backend and Angie on the frontend
to post inline fix suggestions on the PR.
EOF

Then point Daggie at both repositories — the module library and the daggerverse (so it can discover Monty and Angie's real URLs and versions):

dagger call -m github.com/telchak/daggerverse/daggie@v0.3.0 \
  --module-urls="https://github.com/telchak/acme-dagger-modules.git" \
  --module-urls="https://github.com/telchak/daggerverse.git" \
  assist \
    --assignment-file=./daggie-assignment.md \
    --source=. \
  export --path=.

Daggie clones both repositories and auto-discovers all Dagger modules within them by finding dagger.json files. It reads each module's source code and @check-decorated functions (acme-backend (test, lint), acme-frontend (test, lint, audit), acme-deploy (scan)), detects the monorepo layout, and finds the coding agents (Monty, Angie) with their version tags. It also fetches the latest dagger/dagger-for-github action version automatically. The export --path=. writes the generated dagger.json and .github/workflows/ci.yml to your project root, ready to review, test with dagger check, and commit.

Meet the Agents

Before we look at what Daggie generates, let's introduce the three agents that work together in this setup. They're all Dagger modules, and you call them the same way you call any other module:

Daggie: the CI specialist. It reads your source code and available modules, then generates the toolchain configuration and CI workflow. You've just seen it in action. Daggie writes the setup; it doesn't run in the pipeline.
Monty: the Python coding agent. When a check fails on Python code (a test failure, a lint error, a broken import), Monty reads the error output and the source code, analyzes the root cause, and posts an inline code fix suggestion directly on the pull request.
Angie: the Angular/TypeScript coding agent. Same role as Monty, but for the frontend stack. When an Angular build or test fails, Angie diagnoses the issue and suggests the fix.

The key design: Daggie generates the toolchain setup once. Monty and Angie are called from the CI workflow only when something fails. The happy path (dagger check: lint, test, audit, scan) is pure deterministic module execution with no LLM involved. AI only enters the picture when a human needs help.

The Generated Setup

Here's what Daggie generates. No .dagger/ directory, no SDK, no Python pipeline code. Just a dagger.json with toolchains and a CI workflow. The code blocks below are what Daggie consistently produced as output after 10+ runs with gemini-2.5-pro:

{
  "name": "acme-monorepo",
  "engineVersion": "v0.20.3",
  "toolchains": [
    {
      "name": "backend",
      "source": "github.com/telchak/acme-dagger-modules/acme-backend@v1.0.0",
      "customizations": [
        {
          "function": ["test"],
          "argument": "source",
          "defaultPath": "/backend"
        },
        {
          "function": ["lint"],
          "argument": "source",
          "defaultPath": "/backend"
        }
      ]
    },
    {
      "name": "frontend",
      "source": "github.com/telchak/acme-dagger-modules/acme-frontend@v1.0.0",
      "customizations": [
        {
          "function": ["test"],
          "argument": "source",
          "defaultPath": "/frontend"
        },
        {
          "function": ["lint"],
          "argument": "source",
          "defaultPath": "/frontend"
        },
        {
          "function": ["audit"],
          "argument": "source",
          "defaultPath": "/frontend"
        }
      ]
    },
    {
      "name": "deploy",
      "source": "github.com/telchak/acme-dagger-modules/acme-deploy@v1.0.0",
      "customizations": [
        {
          "function": ["scan"],
          "argument": "source",
          "defaultPath": "/backend"
        }
      ]
    }
  ]
}

Notice what Daggie understood from the project structure and the module library:

Monorepo layout detected. Daggie saw backend/ and frontend/ and added customizations to route each check's source argument to the right subdirectory.
All @check functions discovered. It read the module source, found every function decorated with @check (test, lint on backend; test, lint, audit on frontend; scan on deploy), and installed the modules as toolchains so dagger check picks them all up.
No pipeline code. No dag.container(), no base image selection, no pip install. The private modules encapsulate all of that. The project gets AcmeCorp-compliant CI from a single JSON file.
Deployment stays explicit. cloud_run and firebase are regular functions, not checks. They don't run via dagger check — they're called explicitly from the CI workflow's deploy step, because deployments should be intentional.

Running the Checks

No GCP credentials needed for the checks — they run entirely in containers:

dagger check

Output:

✔ acme-backend:lint    (12.9s)  OK
✔ acme-backend:test    (15.2s)  OK
✔ acme-deploy:scan     (18.4s)  OK
✔ acme-frontend:lint   (58.0s)  OK
✔ acme-frontend:test   (62.0s)  OK
✔ acme-frontend:audit  (25.6s)  OK

Six checks, three toolchains, zero lines of code. All six run in parallel. No tokens consumed. The private modules handled base images, cache volumes, coverage thresholds, and vulnerability scanning — all invisible to the project.

When a Check Fails

Let's say a developer pushes a PR and the backend tests fail:

$ dagger check
✔ acme-backend:lint    (3.1s)   OK
✘ acme-backend:test    (6.4s)   ERROR
┇ .test(
  ┆ source: context ./backend
  ) ›
✘ withExec pytest -v --tb=short --cov=src ...  (4.2s)  ERROR
FAILED tests/test_auth.py::test_validate_token
AssertionError: Expected 401, got 200
auth.py:47 — missing token expiry check

✔ acme-deploy:scan     (18.4s)  OK
✔ acme-frontend:lint   (2.9s)   OK
✔ acme-frontend:test   (9.1s)   OK
✔ acme-frontend:audit  (25.6s)  OK

The CI workflow's failure step kicks in. Monty reads the error output and the source code, analyzes the root cause, and posts an inline code suggestion directly on the PR:

🐍 Monty suggested a fix for backend/auth.py:
def validate_token(token: str) -> bool:
    payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
    if payload["exp"] < time.time():
        raise HTTPException(status_code=401, detail="Token expired")
    return True
The test expects a 401 when the token is expired, but validate_token doesn't check the exp claim. This adds the expiry check before returning.

The developer gets actionable fix suggestions, with code they can accept in one click, instead of a wall of logs to interpret.

Integrating Into CI/CD

Daggie also generates the GitHub Actions workflow. Here's what it produces: dagger check for PRs, deployment on main, and a failure handler that calls Monty or Angie directly:

# .github/workflows/ci.yml (generated by Daggie)
name: CI

on:
  push:
    branches:
      - main
  pull_request:

# Grant permissions for OIDC and for agents to post comments/suggestions
permissions:
  contents: read
  pull-requests: write
  id-token: write

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      - name: Run all Dagger checks
        id: checks
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3" # Must match engineVersion in dagger.json
          verb: check
        # Allow the job to continue on failure so the fix suggestion steps can run
        continue-on-error: true

      - name: Call Monty to suggest fixes for the backend
        if: failure() && steps.checks.outcome == 'failure' && github.event_name == 'pull_request'
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: >-
            -m github.com/telchak/daggerverse/monty@v0.2.0
            suggest-github-fix
            --source=./backend
            --github-token=env:GITHUB_TOKEN
            --pr-number=${{ github.event.pull_request.number }}
            --repo=${{ github.repository }}
            --commit-sha=${{ github.event.pull_request.head.sha }}
            --error-output="${{ steps.checks.outputs.stderr }}"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY
          # depending on the agent's LLM provider
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - name: Call Angie to suggest fixes for the frontend
        if: failure() && steps.checks.outcome == 'failure' && github.event_name == 'pull_request'
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: >-
            -m github.com/telchak/daggerverse/angie@v0.2.0
            suggest-github-fix
            --source=./frontend
            --github-token=env:GITHUB_TOKEN
            --pr-number=${{ github.event.pull_request.number }}
            --repo=${{ github.repository }}
            --commit-sha=${{ github.event.pull_request.head.sha }}
            --error-output="${{ steps.checks.outputs.stderr }}"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - name: Fail the job if checks failed
        if: steps.checks.outcome == 'failure'
        run: exit 1

  deploy:
    runs-on: ubuntu-latest
    needs: check
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    steps:
      - uses: actions/checkout@v6

      - name: Deploy Backend to Cloud Run
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: >-
            deploy cloud-run
            --source=./backend
            --service-name=acme-backend-api
            --team=api-team
            --project-id=${{ vars.GCP_PROJECT_ID }}
            --region=${{ vars.GCP_REGION }}
            --environment=production
            --oidc-request-token=env:ACTIONS_ID_TOKEN_REQUEST_TOKEN
            --oidc-request-url=env:ACTIONS_ID_TOKEN_REQUEST_URL
        env:
          ACTIONS_ID_TOKEN_REQUEST_TOKEN: ${{ secrets.ACTIONS_ID_TOKEN_REQUEST_TOKEN }}
          ACTIONS_ID_TOKEN_REQUEST_URL: ${{ secrets.ACTIONS_ID_TOKEN_REQUEST_URL }}

      - name: Deploy Frontend to Firebase
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: >-
            deploy firebase
            --source=./frontend
            --project-id=${{ vars.GCP_PROJECT_ID }}
            --oidc-request-token=env:ACTIONS_ID_TOKEN_REQUEST_TOKEN
            --oidc-request-url=env:ACTIONS_ID_TOKEN_REQUEST_URL
        env:
          ACTIONS_ID_TOKEN_REQUEST_TOKEN: ${{ secrets.ACTIONS_ID_TOKEN_REQUEST_TOKEN }}
          ACTIONS_ID_TOKEN_REQUEST_URL: ${{ secrets.ACTIONS_ID_TOKEN_REQUEST_URL }}

The check job uses zero LLM tokens. It's pure dagger check — six deterministic checks from three toolchains. The suggest-fix steps only run on failure, calling Monty and Angie directly as Dagger modules (not pipeline functions). The deploy job calls acme-deploy's functions via dagger call on the installed toolchain. You get deterministic, fast CI with intelligent failure handling.

The Developer Experience Shift

Before: Pipeline Specialists

Developer → writes app code
DevOps    → writes pipeline code per stack
DevOps    → maintains deployment scripts per framework
DevOps    → debugs pipeline failures
Everyone  → waits for DevOps

After: Agents Configure and Fix CI

Platform team → builds and maintains private modules (acme-backend, acme-frontend, acme-deploy)
Platform team → builds and maintains agents (Daggie, Monty, Angie)
Developer     → asks Daggie to set up toolchains from the private module library
dagger check  → runs deterministically — no LLM, no tokens
On failure    → coding agents analyze errors and suggest fixes on the PR

The platform team builds the modules and agents. Daggie configures toolchains and generates the CI workflow. dagger check runs fast and deterministic. When things break, coding agents step in with targeted fixes.

From Developer Platform to Agent Factory

So far we've seen how Dagger improves CI performance, maintainability, and developer experience. But there's a larger shift happening.

As coding agents become more capable, the developer's core role is evolving, from pure coder to agent orchestrator. You still need to understand the code, review the output, and make architectural decisions. But more and more of the mechanical work (implementing a well-specified feature, writing tests for existing code, fixing a lint error) can be delegated to agents that understand your codebase.

Follow this evolution to its conclusion, and an Internal Developer Platform starts looking like an Internal Agent Factory: a system that manages not just infrastructure and deployments, but how coding agents are built, composed, and deployed: which agents run on which tasks, with what models, under what constraints, producing what artifacts.

The building blocks are already here. We have:

Typed, testable modules that encapsulate domain expertise (Part 3)
Coding agents that read source code and produce changes (Monty, Angie)
A CI specialist that generates pipelines from module libraries (Daggie)

What's missing is the orchestration layer, something that takes a feature request, breaks it into agent-assignable tasks, and dispatches them through CI. That's Speck.

Spec-Driven Development With Speck

Speck is a Dagger agent that implements spec-driven development, inspired by GitHub's spec-kit methodology. The idea is simple: specifications first, code second.

Given a feature request (either a prompt or a GitHub issue), Speck runs a three-step pipeline:

Specify: generate a structured specification with user stories, acceptance criteria, and requirements
Plan: produce a technical implementation plan grounded in the actual codebase
Decompose: break the plan into ordered, dependency-aware tasks with agent and model assignments

The output is a structured JSON object designed for GitHub Actions fromJson() + matrix strategy consumption. Each task includes a suggested_agent (which Dagger agent should execute it), a suggested_model (which LLM complexity tier it needs), and an order field that defines the execution sequence.

The Pipeline Pattern

When --include-tests and --include-review are enabled, Speck organizes tasks into phases that follow an implement → test → review pipeline:

Phase 1: T001 (implement, haiku) → T002 (test, sonnet) → T003 (review, sonnet)
Phase 2: T004 (implement, sonnet) → T005 (implement, sonnet) → T006 (test, opus) → T007 (review, sonnet)

Phases run in parallel (each on its own CI runner). Tasks within a phase use the prompt chaining pattern, a workflow where the output of one agent becomes the input of the next, forming a sequential pipeline. Concretely, each agent receives a source Directory, modifies it, and exports the result back to the workspace. The next agent in the chain picks up that modified workspace as its input. This is different from running agents independently: the test agent sees the code the implementation agent wrote, and the review agent sees both the implementation and the tests. One PR is created per phase from the accumulated changes.

The model assignment is automatic: Speck maps task complexity to concrete model IDs based on the chosen provider family. Simple config changes get Haiku. Standard feature implementations get Sonnet. Cross-cutting architectural changes get Opus. Test tasks get one tier above their implementation task's complexity, since understanding the implementation requires more context.

Setting Up the Workflow

Let's see this in action. We'll fork a real-world application, the FastAPI RealWorld Example App (a production-like REST API with authentication, articles, comments, and favorites), and turn GitHub Actions into a spec-driven development platform.

Step 1: Fork the repository

gh repo fork nsidnev/fastapi-realworld-example-app --clone
cd fastapi-realworld-example-app

Step 2: Add the Speck workflow

Create .github/workflows/speck.yml:

# Spec-Driven Development with Speck
# Triggers on "speck" label → decompose → execute phases → create PRs

name: Spec-Driven Development

on:
  issues:
    types: [labeled]

permissions:
  contents: write
  pull-requests: write
  issues: write

jobs:
  decompose:
    if: github.event.label.name == 'speck'
    runs-on: ubuntu-latest
    outputs:
      result: ${{ steps.run.outputs.result }}
    steps:
      - uses: actions/checkout@v6
      - uses: dagger/dagger-for-github@v8.4.1
        with:
          verb: version

      - name: Decompose issue into phases
        id: run
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          ANTHROPIC_MODEL: claude-opus-4-6
        run: |
          dagger call -m github.com/telchak/daggerverse/speck@feat/speck \
            --allow-llm=all \
            --source=. \
            decompose \
            --issue-id=${{ github.event.issue.number }} \
            --repository="https://github.com/${{ github.repository }}" \
            --github-token=env:GITHUB_TOKEN \
            --create-pr \
            --include-tests \
            --include-review \
            --agents='[{"name":"monty","source":"github.com/telchak/daggerverse/monty@feat/speck","specialization":"Python backend development","capabilities":["assist","review","write_tests","build","upgrade"]}]' \
            --tech-stack="Python, FastAPI, PostgreSQL" \
            > /tmp/speck.json

          echo "result=$(jq -c '.' /tmp/speck.json)" >> $GITHUB_OUTPUT

      - name: Post summary to issue
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          RESULT: ${{ steps.run.outputs.result }}
        run: |
          TASKS=$(echo "$RESULT" | jq '.total_tasks')
          PHASES=$(echo "$RESULT" | jq '.execution_plan.total_phases')
          PRETTY=$(echo "$RESULT" | jq '.')

          gh issue comment "${{ github.event.issue.number }}" \
            --body "## Speck Decomposition Complete
          **Tasks**: ${TASKS} | **Phases**: ${PHASES}
          <details><summary>Full JSON</summary>

          \`\`\`json
          ${PRETTY}
          \`\`\`
          </details>"

  execute-phase:
    needs: decompose
    runs-on: ubuntu-latest
    strategy:
      matrix:
        phase: ${{ fromJson(needs.decompose.outputs.result).execution_plan.phases }}
      max-parallel: 3
      fail-fast: false
    steps:
      - uses: actions/checkout@v6
      - uses: dagger/dagger-for-github@v8.4.1
        with:
          verb: version

      - name: Execute phase ${{ matrix.phase.phase }} and create PR
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          SPECK_JSON: ${{ needs.decompose.outputs.result }}
          PHASE_NUM: ${{ matrix.phase.phase }}
        run: |
          TASKS=$(echo "$SPECK_JSON" | jq -c \
            "[.tasks[] | select(.phase == ($PHASE_NUM | tonumber))] | sort_by(.order)")

          for i in $(seq 0 $(( $(echo "$TASKS" | jq 'length') - 1 ))); do
            TASK=$(echo "$TASKS" | jq -c ".[$i]")
            AGENT=$(echo "$TASK" | jq -r '.suggested_agent.source // empty')
            ENTRY=$(echo "$TASK" | jq -r '.suggested_agent.entrypoint // "assist"' | tr '_' '-')
            DESC=$(echo "$TASK" | jq -r '.description')
            MODEL=$(echo "$TASK" | jq -r '.suggested_model')

            echo "--- [$(echo "$TASK" | jq -r '.id')] $(echo "$TASK" | jq -r '.title') (model=$MODEL) ---"
            [ -z "$AGENT" ] && echo "Skipping: no agent" && continue

            if [ "$ENTRY" = "review" ]; then
              ANTHROPIC_MODEL="$MODEL" dagger call -m "$AGENT" --allow-llm=all \
                --source=. "$ENTRY" --assignment="$DESC"
            else
              ANTHROPIC_MODEL="$MODEL" dagger call -m "$AGENT" --allow-llm=all \
                --source=. "$ENTRY" --assignment="$DESC" export --path=.
            fi
          done

          # Create PR from accumulated changes
          dagger call -m github.com/kpenfound/dag/github-issue \
            --token=env:GITHUB_TOKEN \
            create-pull-request \
            --repo="https://github.com/${{ github.repository }}" \
            --source=. \
            --branch="${{ matrix.phase.pr_branch }}" \
            --base=master \
            --title="Phase ${{ matrix.phase.phase }}: ${{ matrix.phase.name }}" \
            --body="Implements **${{ matrix.phase.name }}**. Closes #${{ github.event.issue.number }}. Generated by Speck." \
            url

A few things to note in this workflow:

Two jobs, one workflow. The decompose job uses Opus for planning, since it needs the most reasoning power to analyze the codebase and produce a good decomposition. The execute-phase job uses the suggested_model from each task: Haiku for simple changes, Sonnet for standard work, Opus for complex logic.
--allow-llm=all is required in CI. In interactive mode, Dagger prompts for LLM API access approval. In GitHub Actions there's no TTY, so we bypass the prompt.
PR creation uses a Dagger module. Instead of raw git commands, we use the github-issue module's create-pull-request function, which takes a --source Directory and handles branch creation, commit, push, and PR creation internally.
Review tasks skip export. The review entrypoint returns a string (the review text), not a Directory. Other entrypoints return a Directory that gets exported back to the workspace for the next task in the chain.

Step 3: Configure secrets

The workflow needs an LLM API key. Add it as a repository secret:

gh secret set ANTHROPIC_API_KEY --repo your-org/fastapi-realworld-example-app

The GITHUB_TOKEN is provided automatically by GitHub Actions with the permissions declared in the workflow.

Step 4: Commit, push, and create a test issue

git add .github/workflows/speck.yml
git commit -m "Add Speck spec-driven development workflow"
git push origin master

Now create a GitHub issue with a feature request (see issue #1):

Title: Add article bookmarking/favorites list endpoint

Body:

Add the ability for authenticated users to retrieve their list of favorited articles with pagination and optional filtering.

Requirements:

New GET /api/articles/feed endpoint that returns articles the current user has favorited

Support pagination via limit and offset query parameters (defaults: limit=20, offset=0)

Support optional tag filter to narrow favorites by tag

Response format must match the existing GET /api/articles response shape (articles array + articlesCount)

Only accessible to authenticated users (return 401 if unauthenticated)

Running the Workflow

Add the speck label to the issue. This triggers the workflow.

Step 1, Decomposition (Opus):

Speck reads the issue, explores the FastAPI codebase (models, routes, repositories, existing test patterns), and produces a structured decomposition. It posts the result as a comment on the issue:

In this case, Speck decomposed the feature into 3 phases with 9 tasks:

Phase 1, Repository and Query Layer (T001-T004): add repository method to fetch favorited articles (assist, sonnet), add dependency function and filter schema (assist, haiku), write tests for repository and schema (write-tests, sonnet), review (review, sonnet)
Phase 2, API Endpoint (T005-T007): implement GET /api/articles/favorites endpoint (assist, sonnet), write integration tests (write-tests, opus), review endpoint and tests (review, sonnet)
Phase 3, Error Handling and Strings (T008-T009): add error handling strings and edge case coverage (assist, haiku), final integration review (review, sonnet)

Each task has a suggested_model based on complexity: simple schema additions get claude-haiku-4-5, standard implementations get claude-sonnet-4-6, and comprehensive test writing gets claude-opus-4-6 (since tests need to understand the full implementation context).

Step 2, Execution (parallel phases, sequential tasks):

GitHub Actions matrices by phase. Each phase runs on its own runner. Within each phase, tasks are chained sequentially. Monty implements the feature, then writes tests on top of the implementation, then reviews the accumulated changes:

Phase 1 runner (Repository and Query Layer):
  T001 (assist, sonnet)        → Monty adds repository method           → exports to .
  T002 (assist, haiku)         → Monty adds filter schema + dependency  → exports to .
  T003 (write-tests, sonnet)   → Monty writes tests                     → exports to .
  T004 (review, sonnet)        → Monty reviews all changes              → logs review
  → github-issue creates PR #3 from accumulated changes

Phase 2 runner (API Endpoint):
  T005 (assist, sonnet)        → Monty implements the endpoint           → exports to .
  T006 (write-tests, opus)     → Monty writes integration tests          → exports to .
  T007 (review, sonnet)        → Monty reviews endpoint + tests          → logs review
  → github-issue creates PR #4 from accumulated changes

Phase 3 runner (Error Handling and Strings):
  T008 (assist, haiku)         → Monty adds error handling + edge cases  → exports to .
  T009 (review, sonnet)        → Monty does final integration review     → logs review
  → github-issue creates PR #5 from accumulated changes

Step 3, Pull Requests:

Each phase produced one PR with all accumulated changes, linked to the original issue:

PR #3: Phase 1: Repository and Query Layer — 361 additions across 5 files
PR #4: Phase 2: API Endpoint — 317 additions across 5 files
PR #5: Phase 3: Error Handling and Strings — 74 additions across 3 files

Each PR includes implementation, tests, and a review pass, all generated by Monty working sequentially on the same codebase within the phase.

What Just Happened

A developer wrote a feature request with acceptance criteria. The system:

Opus analyzed the codebase and decomposed the request into 9 tasks across 3 phases
Haiku, Sonnet, and Opus implemented the feature, wrote tests, and reviewed the code, each task using the right model for its complexity
Three PRs were created automatically, linked to the issue, ready for human review — 752 lines of code across 13 files

No pipeline code was written. No agent was invoked manually. The developer's job is now to review the PRs: read the code, check the tests, verify the approach. The mechanical work of translating a spec into code, tests, and PRs happened automatically.

This is the shift from Internal Developer Platform to Internal Agent Factory: the platform doesn't just run your CI. It runs your agents, manages their model costs, chains their outputs, and produces reviewable artifacts from natural language specifications.

Image generated with Google's Gemini "Nano Banana Pro"

Agents That Learn: Self-Improvement Across Runs

There's one more capability worth covering. Every agent (Monty, Angie, Daggie, and Goose, a GCP deployment orchestrator) reads per-repo context files to understand project conventions. But until now, the context was static. The developer wrote it once and maintained it by hand.

With --self-improve, the agents can update those files themselves:

dagger call -m github.com/telchak/daggerverse/monty@v0.2.0 \
  --self-improve=write \
  assist \
    --source=. \
    --assignment="Add input validation to all API endpoints"

As Monty works through the codebase (reading models, tracing routes, checking existing patterns), it discovers things: "This project uses Pydantic v2 field validators, not v1-style @validator." "Tests use httpx.AsyncClient, not the sync test client." "Custom exceptions live in app/errors.py."

Instead of those discoveries dying with the session, Monty records them in two files:

MONTY.md, Python-specific knowledge:

## Learned Context

- Pydantic v2 with field validators (`field_validator`), not v1 `@validator`
- All route handlers are async; tests use `httpx.AsyncClient` with `pytest-asyncio`
- Input validation pattern: Pydantic model as request body, raises `ValidationError` → 422

AGENTS.md, general project knowledge shared across all agents:

## Learned Context

- Custom exception hierarchy in `app/errors.py`, handlers in `app/middleware.py`
- Project uses src layout with `app/` as the main package
- CI runs pytest with coverage; minimum threshold is 80%

The next time any agent runs on this repo, whether it's Monty, Angie, or a different developer, it reads both the agent-specific file and the shared AGENTS.md, starting with better knowledge. Python patterns stay in MONTY.md where only Monty reads them; project-wide conventions go in AGENTS.md where every agent benefits. No one had to write documentation. The agents documented the project by working on it.

Three modes

Mode	What happens
`off` (default)	Current behavior: context files are read-only
`write`	Agent updates context files in the returned workspace directory
`commit`	Agent updates context files and creates a git commit

The commit mode is useful for automation. When combined with develop-github-issue, the context file updates get included in the PR:

dagger call -m github.com/telchak/daggerverse/monty@v0.2.0 \
  --self-improve=commit \
  develop-github-issue \
    --github-token=env:GITHUB_TOKEN \
    --issue-id=42 \
    --repository="https://github.com/owner/my-python-api" \
    --source=.

The PR includes both the code changes and a commit like:

chore(monty): update context files with learned discoveries

Over time, the context files become living documents, a compressed summary of the project's architecture, conventions, and gotchas, maintained by the agents that work on it.

Would I Recommend Dagger CI in Production Right Now?

I've been following the Dagger project for several years now. And I can say with confidence: it has never been closer to production-ready than it is today.

The core primitives (typed functions, composable modules, containerized execution, deterministic caching) are solid. The dagger call experience is genuinely portable across local development and CI. The module ecosystem is growing. And as we've seen throughout this series, the LLM integration through the dag.llm() primitive opens up a category of workflows that simply didn't exist before.

That said, there are areas where the platform still needs to mature. Here's what I'd like to see, and what's already on the roadmap.

What the Agent Layer Needs

The current LLM primitive is functional but minimal. To build truly capable agents in Dagger, a few key features would make a significant difference:

External memory: Right now, agents forget everything between runs. Connecting the LLM to persistent memory stores (a vector database, a knowledge graph, or even a simple key-value store) would let agents accumulate project knowledge beyond what --self-improve and context files can provide.
RAG integration: Being able to connect the LLM to external retrieval-augmented generation engines would allow agents to reason over large documentation sets, internal wikis, or historical CI logs without stuffing everything into the context window.
Remote MCP servers: Dagger currently supports stdio-based MCP servers. Adding support for remote HTTP MCP servers would unlock integration with hosted tool services (SonarQube, Jira, Slack, observability platforms) without needing to bundle everything as a local process.
Broader BYOK compatibility: Dagger natively recognizes only four provider configurations: ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENAI_API_KEY, and Ollama. To use any other provider (Mistral, Qwen, DeepSeek), you have to route through the OPENAI_BASE_URL compatibility layer, which works but feels like a workaround. Native support for more provider environment variables (MISTRAL_API_KEY, DEEPSEEK_API_KEY, etc.) would make BYOK a first-class experience rather than an OpenAI-compat hack.
Native multi-agent patterns: Right now, orchestrating multiple agents (like Speck decomposing work across Monty and Angie) requires external coordination (a GitHub Actions matrix, a shell loop, or a custom orchestrator). A native Graph type in Dagger's core, similar to what LangGraph provides, would let you define agent workflows as typed, cacheable DAGs. Imagine declaring "run these three agents in parallel, merge their outputs, then run a review agent" as a first-class Dagger construct.

What's Coming, and Why It Matters

Some of the most exciting changes are already in active development:

Cloud Engines: Fully managed Dagger execution environments with auto-scaling and distributed caching built in. Run dagger --cloud and your pipeline executes on managed infrastructure, with secrets and local context securely streamed to the cloud. No more managing Kubernetes daemonsets or custom cache layers.

Cloud Checks: This is the big one. Cloud Checks connects directly to your Git provider and triggers dagger check on every change, running on Cloud Engines. No YAML. No vendor syntax. No orchestration layer. Just your Dagger modules.

Those two previous features are welcome because the more complex our Dagger workflows get, the more trying to fit them into GitHub Actions or GitLab CI feels like forcing circles into squares. Our Speck-driven development workflow is a perfect example: a decompose job that outputs dynamic JSON, a matrix strategy that fans out phases, shell scripts converting snake_case to kebab-case, environment variables carrying JSON between steps, conditional export commands based on return types... All of this ceremony exists because GitHub Actions was designed for static, declarative workflows, not for the kind of dynamic, graph-shaped execution that Dagger naturally produces. Cloud Checks would eliminate that entire translation layer. Your Dagger module is the CI platform. Add to that a native Graph core type, and you could have a full native multi-agent workflow completely independent from GitHub Actions or any other CI engine. Dagger CI would go from a "CI development toolkit" to a fully operational CI/CD platform.

Modules V2: A fundamental redesign of how modules interact with projects. Today, modules can't see your project structure unless you thread it through manually with --source flags, custom boilerplate, and static path patterns. Modules V2 introduces a typed Workspace API that lets modules parse configuration files, traverse directory trees, and adapt to any project layout, all through executable code rather than rigid pragmas. A new .dagger/config.toml file declares which modules a project uses in a human-editable format, and a lockfile ensures reproducible resolution across teams. This shifts complexity from users to module authors, which is exactly where it belongs.

These three features together (managed compute, native CI triggering, and smarter module integration) would close the gap between "Dagger as a portable pipeline SDK" and "Dagger as a complete CI platform." And from everything I've seen in the project's trajectory, that gap is closing fast.

Conclusion

This is where the whole series comes together.

Part 1: Pipelines as real code — typed, testable, portable. No more YAML guesswork.
Part 2: Decoupled from infrastructure. Same pipeline on any runner, any cloud.
Part 3: A module library. Public daggerverse modules for generic operations, private modules for organizational compliance. Domain expertise encoded as deterministic, versioned functions.
Part 4: AI agents at the edges. Daggie generates the toolchain setup from the module library. dagger check runs fast and deterministic — no LLM, no tokens. When checks fail, Monty and Angie post fix suggestions on the PR. Speck decomposes feature requests into phased, agent-executable task plans. And with --self-improve, every agent interaction leaves the project better documented.

The key insight: CI checks need to be fast, reliable, and deterministic. AI belongs at the edges — generating the configuration, diagnosing failures, decomposing specs into tasks, and learning from every run. Never in the hot path.

The example apps and Dagger module are at github.com/telchak/dagger-ci-demo. The AcmeCorp private modules from Part 3 are at github.com/telchak/acme-dagger-modules.

Useful links:

github.com/dagger/dagger: the main Dagger repository
dagger.io/changelog: follow what features are being actively developed (Cloud Engines, Cloud Checks, Modules V2, and more)
daggerverse.dev: the public module registry, where you can discover and reuse community modules
github.com/telchak/daggerverse: all the modules I've been personally creating throughout this series (Daggie, Monty, Angie, Goose, Speck, and the GCP infrastructure modules), and that I'll continue to support to bring my little piece to this open-source project

Key Takeaways

Deterministic checks, not LLM-routed ones: CI needs speed and reliability; keep AI out of the hot path
Daggie configures toolchains: point it at your module library and your project, and it generates the right dagger.json and CI workflow
Agents fix failures: Monty and Angie analyze errors and post code suggestions on PRs
Agents learn: --self-improve lets agents update context files (agent-specific + shared AGENTS.md) with project discoveries, getting smarter across runs
Spec-driven development: Speck decomposes high-level specs into structured task lists, dispatching work across specialized agents with model-appropriate assignments
Modules are the foundation: public modules for generic operations, private modules for org compliance, both composable by humans and agents
AI at the edges: configure the setup (before), fix the failures (after), learn from the work. Never in the hot path

This concludes the 4-part series. Thanks for reading.

Full Series:

Tags: #cicd #dagger #ai-agents #platform-engineering #mcp #cloudrun #firebase #spec-driven-development

CI/CD in the Era of AI and Platform Engineering: A Deep Dive into Dagger CI (Part 3)

Sami Chibani — Fri, 27 Mar 2026 00:20:27 +0000

Part 3: From Scripts to a Platform: Your CI/CD Module Library

Let's be clear: Dagger doesn't eliminate the complexity of modern CI/CD systems. It tames that complexity into a testable, maintainable, and reusable system suitable for modern Platform Engineering practices.

In Part 1 we wrote pipelines as real code. In Part 2 we decoupled them from infrastructure. Our dagger-ci-demo module works. It builds, tests, and runs identically on a laptop and in CI. But it's a single module in a single repository. What happens when the organization grows?

The Growth Problem

Let's say you're at AcmeCorp. Your platform team adopted Dagger six months ago. The first project went well: a single .dagger/ module with build, test, and deploy functions. Then things accelerated.

New repositories appeared. The mobile team needs CI. The data team wants to containerize their pipelines. The infrastructure team is building internal tools. Each team creates their own .dagger/ module, and within weeks, you see the pattern:

Copy-paste proliferation. Every team has its own GCP authentication function. Some use service account keys. Some use OIDC. Some hardcode the project ID. None of them handle credential rotation.
Security drift. The data team's deploy function doesn't validate image signatures. The mobile team allows unauthenticated Cloud Run services in production. Nobody enforces the naming convention.
Maintenance burden. When GCP deprecates the old gcloud auth activate-service-account flow, someone has to find and update every copy. Nobody knows how many there are.

This is the exact same problem that Terraform modules and Helm charts solved for infrastructure. The solution is the same too: a module library.

The difference is that Dagger already has this built in.

The Daggerverse: Community Modules

Before building anything custom, check what already exists. Dagger is a community-driven project, and the community has published hundreds of modules for common use cases, all centralized in what's called the Daggerverse.

The Daggerverse is Dagger's public module registry. Any module published to a Git repository and indexed by Dagger Cloud becomes discoverable there. You can browse by category, search by keyword, and inspect the typed API of any module before installing it.

For example, if AcmeCorp runs on Google Cloud, a quick search reveals modules that already handle the GCP fundamentals:

Module	What it does
`gcp-auth`	GCP authentication via OIDC or service account key
`gcp-artifact-registry`	Publish container images to Artifact Registry
`gcp-cloud-run`	Deploy services to Cloud Run
`gcp-firebase`	Deploy to Firebase Hosting
`python-build`	Build Python applications with pip/uv
`angular`	Build and test Angular applications
`trivy`	Container vulnerability scanning with Trivy

These are real, production-tested modules. Each one has a typed API, documentation auto-generated from the code, and a version you can pin.

Full disclosure: I'm the author of most of the GCP modules listed above (gcp-auth, gcp-artifact-registry, gcp-cloud-run, gcp-firebase, python-build, angular). I'm recommending them not because they're mine, but because I built them to solve real problems I kept running into over several years of running production workloads on Google Cloud. They encode patterns and guardrails that my team and I needed — OIDC authentication that actually works across CI providers, Artifact Registry publishing with proper tagging, Cloud Run deployments with sane defaults. If you're on GCP, I genuinely think they'll save you time. If they don't fit your needs, fork them or build your own — that's the beauty of the module system.

You install them with a single command:

dagger install github.com/telchak/daggerverse/gcp-auth@gcp-auth/v0.2.1

This adds the module as a dependency in your dagger.json. You can then call its functions from your pipeline code via dag.gcp_auth(), or directly from the CLI:

# Test GCP authentication from your terminal using a service account key
dagger call -m github.com/telchak/daggerverse/gcp-auth@gcp-auth/v0.2.1 \
  gcloud-container \
  --credentials=file:./my-service-account-key.json \
  --project-id=my-project \
  with-exec --args="gcloud","auth","list" \
  stdout

The module system handles dependency resolution, version pinning, and cross-module type compatibility. You compose modules the same way you compose functions, by passing outputs as inputs:

# Authenticate
gcloud = dag.gcp_auth().from_service_account_key(key=sa_key, project=project)

# Publish (takes the authenticated container from gcp-auth)
image_uri = await dag.gcp_artifact_registry().publish(
    container=my_app, gcloud=gcloud, project_id=project,
    region="us-central1", repository="my-repo", image_name="api", tag="v1.0.0",
)

# Deploy (takes the image URI from artifact-registry)
url = await dag.gcp_cloud_run().service().deploy(
    gcloud=gcloud, service_name="api", image=image_uri, region="us-central1",
)

This is the first layer of the module system: public modules that handle generic, well-understood operations. They're open-source, community-maintained, and available to anyone.

But for most organizations, public modules alone aren't enough.

Two-Layer Module Architecture

A typical production setup has two layers of modules:

┌──────────────────────┐     ┌──────────────────────────┐     ┌──────────────────────┐
│  Daggerverse         │────▶│  Private Modules Repo     │────▶│  Your CI Pipelines    │
│  (public modules)    │     │  (org-specific layer)      │     │  (.dagger/ per repo)  │
└──────────────────────┘     └──────────────────────────┘     └──────────────────────┘

 Generic operations          Security, compliance,           Project-specific
 (GCP auth, Docker,          naming, defaults,               build/test/deploy
  language builds)           org business logic              logic

Layer 1: Public modules handle generic operations: authenticate to GCP, push a container image, deploy to Cloud Run. They have no opinion about your organization's naming conventions, security policies, or environment structure.

Layer 2: Private modules wrap public modules with organization-specific logic. They encode your security requirements, enforce naming conventions, inject audit trails, and provide a curated interface that teams consume without needing to understand the underlying complexity.

This separation matters because:

Public modules evolve independently. When gcp-auth adds a new authentication method, you get it for free on the next version bump.
Private modules enforce your rules. Every deploy goes through your security checks, regardless of which team triggered it.
Teams consume the private layer only. They don't need to know which public modules are underneath, or how authentication works internally. They call dag.acme_deploy() and get a compliant deployment.

Let's see what this looks like in practice for AcmeCorp. But first, a few principles that apply to both layers.

Module Development Best Practices

Whether you're building public daggerverse modules or private organizational ones, the same practices apply. Dagger modules are consumed across SDKs (Python, Go, TypeScript, Java) and from the CLI, so the way you write them directly impacts the experience for every consumer.

Documentation is your API

This is the single most important practice. Every docstring, every Doc() annotation, every class description you write is automatically used to generate documentation across all Dagger SDKs and the CLI. When someone runs dagger call your-function --help, they see your docstrings. When someone browses your module on the Daggerverse, they see your Doc() annotations. When someone uses your module from Go or TypeScript, their IDE shows your descriptions as inline documentation.

@object_type
class GcpAuth:
    """Authenticate to Google Cloud Platform.           ← Shows in module description

    Supports two authentication methods:
    - Service account key (local dev, testing)
    - OIDC / Workload Identity Federation (CI/CD)
    """

    @function
    async def from_oidc(
        self,
        token: Annotated[                              # ← Each parameter gets its own
            dagger.Secret,                             #   help text in CLI and IDE
            Doc("OIDC token from the CI provider "
                "(e.g. GitHub Actions ID token)"),
        ],
        provider: Annotated[
            str,
            Doc("Full Workload Identity Provider resource name "
                "(projects/{id}/locations/global/workloadIdentityPools/{pool}/providers/{provider})"),
        ],
    ) -> dagger.Container:
        """Authenticate using OIDC / Workload Identity. ← Shows in `dagger call --help`

        Best for CI/CD pipelines — no long-lived
        credentials to rotate or leak.
        """

Write docstrings as if they're the only documentation someone will ever read, because for most consumers, they are.

Single responsibility per module

Each module should do one thing well. gcp-auth handles authentication. gcp-cloud-run handles Cloud Run deployments. Don't create a gcp-everything module that does auth, registry, Cloud Run, and Firebase. Smaller modules are easier to test, version, and compose.

A good rule of thumb: if your module needs two unrelated sets of credentials, it's probably two modules.

Use `field()` for constructor dependencies

When a module needs an authenticated context or shared state, accept it as a constructor field rather than repeating it on every function:

@object_type
class ArtifactRegistry:
    """Manage container images in Google Artifact Registry."""

    gcloud: Annotated[
        dagger.Container,
        Doc("Authenticated gcloud container from gcp-auth"),
    ] = field()

This makes composition natural. You wire up authentication once and pass it in:

gcloud = dag.gcp_auth().from_service_account_key(key=key, project=project)
registry = dag.artifact_registry(gcloud=gcloud)  # Authenticated for all calls

Return Dagger types, not strings

When possible, return Container, Directory, or File rather than strings. This enables chaining: the output of your module becomes the input of the next one. Return str only for terminal values (URLs, status messages) or when the consumer explicitly needs text output.

# Good: returns Container — composable with other modules
@function
async def build(self, source: ...) -> dagger.Container: ...

# Good: returns str — terminal value (a URL)
@function
async def deploy(self, image: ...) -> str: ...

Use `dagger.Secret` for sensitive data

Never accept credentials, tokens, or keys as plain str parameters. Dagger's Secret type ensures sensitive data is never logged, cached, or written to disk. The CLI handles secret injection transparently:

# From environment variable
dagger call deploy --token=env:MY_TOKEN

# From file
dagger call deploy --key=file:./service-account.json

Private functions stay private

Not every function in your module needs to be part of the public API. Dagger follows each SDK's language conventions for visibility. In Python, any method prefixed with an underscore (_) is treated as private and won't be exported by Dagger. It won't show up in dagger call --help, won't be callable from the CLI, and won't appear in the Daggerverse documentation:

@object_type
class AcmeDeploy:

    def _validate_and_resolve(self, team, service_name, environment, region):
        """Internal helper — not exported by Dagger."""
        ...

    def _authenticate(self, oidc_token, project):
        """Internal helper — not exported by Dagger."""
        ...

    @function
    async def cloud_run(self, ...) -> str:
        """Public function — exported and callable."""
        ctx = self._validate_and_resolve(...)
        gcloud = self._authenticate(...)
        ...

Use this to keep your public API clean while organizing internal logic into reusable helpers. In Go, unexported (lowercase) functions serve the same role. In TypeScript, omit the @func() decorator.

Pin your dependencies

Always pin module dependencies to a specific version tag, not @main or @latest. This ensures reproducible builds and prevents upstream changes from breaking your pipelines unexpectedly:

{"name": "gcp-auth", "source": "github.com/telchak/daggerverse/gcp-auth@gcp-auth/v0.2.1"}

Building the Private Module Layer

AcmeCorp's platform team creates a private repository, github.com/telchak/acme-dagger-modules, that contains organization-specific modules. These modules wrap the public daggerverse modules and add AcmeCorp's business logic.

Repository Structure

telchak/acme-dagger-modules/
├── acme-backend/                   # Python/FastAPI build, test, lint, SBOM
│   ├── dagger.json
│   ├── pyproject.toml
│   └── src/acme_backend/
│       └── main.py
├── acme-frontend/                  # Angular build, test, lint, audit
│   ├── dagger.json
│   ├── pyproject.toml
│   └── src/acme_frontend/
│       └── main.py
├── acme-deploy/                    # Compliant deployment (Cloud Run + Firebase)
│   ├── dagger.json                 # + Trivy scanning, production branch gating
│   ├── pyproject.toml
│   └── src/acme_deploy/
│       └── main.py
└── tests/                          # Integration tests for all modules
    ├── dagger.json
    ├── pyproject.toml
    └── src/tests/
        └── main.py

Each module is a self-contained Dagger module with its own dagger.json, dependencies, and source code. They're versioned together via Git tags (e.g. acme-deploy/v1.2.0) or as a monorepo with a single version.

Let's walk through the three core modules that cover AcmeCorp's full-stack pipeline: build, test, and deploy.

`acme-backend`: Python Build & Test

This module wraps the public python-build module with AcmeCorp's standards: base image policy, cache volume conventions, health check endpoint, and standard entrypoint configuration.

"""AcmeCorp Python backend builder — standardized, cached, production-ready."""

from typing import Annotated

import dagger
from dagger import DefaultPath, Doc, check, dag, function, object_type


# Approved base images — only these are allowed in production containers.
APPROVED_BASE_IMAGE = "python:3.13-slim"

# Minimum test coverage threshold enforced across all backend services.
MIN_COVERAGE_PERCENT = 80


@object_type
class AcmeBackend:
    """Build and test Python backends the AcmeCorp way.

    Enforces:
    - Organization-approved base images (python:3.13-slim)
    - Standardized cache volumes for pip
    - Health check endpoint convention (/health)
    - Cloud Run-compatible entrypoint and port
    - Minimum 80% test coverage
    - CycloneDX SBOM generation for vulnerability tracking
    """

    def _base_container(self, source: dagger.Directory) -> dagger.Container:
        """Standard Python container with source and cached deps."""
        return (
            dag.container()
            .from_(APPROVED_BASE_IMAGE)
            .with_workdir("/app")
            .with_directory("/app", source)
            .with_mounted_cache("/root/.cache/pip", dag.cache_volume("acme-pip"))
            .with_exec(["pip", "install", "-r", "requirements.txt"])
        )

    @function
    def build(
        self,
        source: Annotated[dagger.Directory, Doc("Python backend source directory")],
        port: Annotated[int, Doc("Application port")] = 8080,
    ) -> dagger.Container:
        """Build a production-ready FastAPI container.

        Uses the organization-approved base image and standardized
        entrypoint. Returns a Container that can be passed directly
        to acme-deploy.
        """
        return (
            self._base_container(source)
            .with_env_variable("PORT", str(port))
            .with_exposed_port(port)
            .with_label("org.opencontainers.image.vendor", "AcmeCorp")
            .with_entrypoint([
                "uvicorn", "src.main:app",
                "--host", "0.0.0.0", "--port", str(port),
            ])
        )

    @function
    @check
    async def test(
        self,
        source: Annotated[
            dagger.Directory,
            Doc("Python backend source directory"),
            DefaultPath("."),
        ],
        coverage: Annotated[bool, Doc("Enforce minimum coverage threshold")] = True,
    ) -> str:
        """Run the test suite with AcmeCorp conventions.

        Uses pytest with verbose output, short tracebacks, and coverage
        reporting. Fails if coverage drops below the org-wide threshold.
        """
        pytest_args = ["pytest", "-v", "--tb=short"]
        if coverage:
            pytest_args.extend([
                "--cov=src", "--cov-report=term-missing",
                f"--cov-fail-under={MIN_COVERAGE_PERCENT}",
            ])
        pytest_args.append("tests/")

        return await (
            self._base_container(source)
            .with_exec(pytest_args)
            .stdout()
        )

    @function
    @check
    async def lint(
        self,
        source: Annotated[
            dagger.Directory,
            Doc("Python backend source directory"),
            DefaultPath("."),
        ],
    ) -> str:
        """Run linting (ruff) on the source code."""
        return await (
            dag.container()
            .from_(APPROVED_BASE_IMAGE)
            .with_workdir("/app")
            .with_directory("/app", source)
            .with_exec(["pip", "install", "ruff"])
            .with_exec(["ruff", "check", "src/"])
            .stdout()
        )

    @function
    async def sbom(
        self,
        source: Annotated[dagger.Directory, Doc("Python backend source directory")],
    ) -> dagger.File:
        """Generate a CycloneDX SBOM for vulnerability tracking.

        Produces a Software Bill of Materials in CycloneDX JSON format,
        suitable for upload to Dependency-Track or similar platforms.
        """
        return (
            self._base_container(source)
            .with_exec(["pip", "install", "cyclonedx-bom"])
            .with_exec([
                "cyclonedx-py", "requirements",
                "--input", "requirements.txt",
                "--output", "/app/sbom.json",
                "--format", "json",
            ])
            .file("/app/sbom.json")
        )

Its dagger.json:

{
  "name": "acme-backend",
  "engineVersion": "v0.20.3",
  "sdk": { "source": "python" },
  "dependencies": [
    { "name": "python-build", "source": "github.com/telchak/daggerverse/python-build@python-build/v0.2.0" }
  ]
}

`acme-frontend`: Angular Build & Test

Same pattern for the frontend: wraps the public angular module with AcmeCorp's Node version policy, cache conventions, and build configuration.

"""AcmeCorp Angular frontend builder — standardized builds and testing."""

from typing import Annotated

import dagger
from dagger import DefaultPath, Doc, check, dag, function, object_type


# Approved Node.js version — pinned to LTS for security compliance.
APPROVED_NODE_VERSION = "20"


@object_type
class AcmeFrontend:
    """Build and test Angular frontends the AcmeCorp way.

    Enforces:
    - Organization-approved Node.js version (20 LTS)
    - Standardized npm cache volumes
    - Production build configuration with source map exclusion
    - Vitest-based testing via Angular's built-in test runner
    - Audit check for known vulnerabilities in dependencies
    """

    def _base_container(self, source: dagger.Directory) -> dagger.Container:
        """Standard Node container with source and cached deps."""
        return (
            dag.container()
            .from_(f"node:{APPROVED_NODE_VERSION}-slim")
            .with_workdir("/app")
            .with_directory("/app", source)
            .with_mounted_cache("/root/.npm", dag.cache_volume("acme-npm"))
            .with_exec(["npm", "ci"])
        )

    @function
    def build(
        self,
        source: Annotated[dagger.Directory, Doc("Angular project source directory")],
    ) -> dagger.Directory:
        """Build the Angular app for production.

        Returns a Directory containing the dist/ output,
        ready to be passed to acme-deploy for Firebase Hosting.
        """
        return (
            dag.angular()
            .build(
                source=source,
                configuration="production",
                node_version=APPROVED_NODE_VERSION,
                npm_cache=dag.cache_volume("acme-npm"),
            )
        )

    @function
    @check
    async def test(
        self,
        source: Annotated[
            dagger.Directory,
            Doc("Angular project source directory"),
            DefaultPath("."),
        ],
    ) -> str:
        """Run the Angular test suite."""
        return await (
            self._base_container(source)
            .with_exec(["npx", "ng", "test"])
            .stdout()
        )

    @function
    @check
    async def lint(
        self,
        source: Annotated[
            dagger.Directory,
            Doc("Angular project source directory"),
            DefaultPath("."),
        ],
    ) -> str:
        """Run Angular linting."""
        return await (
            self._base_container(source)
            .with_exec(["npx", "ng", "lint"])
            .stdout()
        )

    @function
    @check
    async def audit(
        self,
        source: Annotated[
            dagger.Directory,
            Doc("Angular project source directory"),
            DefaultPath("."),
        ],
    ) -> str:
        """Check npm dependencies for known vulnerabilities.

        Runs npm audit at the 'moderate' severity level. Fails if any
        vulnerabilities at moderate or higher are found.
        """
        return await (
            self._base_container(source)
            .with_exec(["npm", "audit", "--audit-level=moderate"])
            .stdout()
        )

Its dagger.json:

{
  "name": "acme-frontend",
  "engineVersion": "v0.20.3",
  "sdk": { "source": "python" },
  "dependencies": [
    { "name": "angular", "source": "github.com/telchak/daggerverse/angular@angular/v0.2.0" }
  ]
}

`acme-deploy`: Compliant Deployment

Required Google APIs — The deployment module needs these APIs enabled on each target GCP project (staging and production):
gcloud services enable \
  run.googleapis.com \
  artifactregistry.googleapis.com \
  firebasehosting.googleapis.com \
  iam.googleapis.com
These cover Cloud Run deployment, container image storage in Artifact Registry, Firebase Hosting for the frontend, and IAM for service account authentication.

This module handles the full deployment lifecycle: vulnerability scanning, authentication, registry, Cloud Run for backends, and Firebase Hosting for frontends. It wraps four public daggerverse modules plus Trivy with organization-specific defaults, security checks, and production safeguards:

"""AcmeCorp deployment module — compliant, opinionated, simple."""

from typing import Annotated

import dagger
from dagger import DefaultPath, Doc, check, dag, function, object_type


ALLOWED_REGIONS = ["europe-west1", "us-central1"]
ALLOWED_ENVIRONMENTS = ["staging", "production"]

# Pin Trivy to a known safe version.
# Versions 0.69.4–0.69.6 were compromised in a supply chain attack (CVE-2026-33634).
TRIVY_VERSION = "0.69.3"

# Resource defaults enforced across all Cloud Run services.
CLOUD_RUN_DEFAULTS = {
    "min_instances": {"staging": 0, "production": 1},
    "max_instances": {"staging": 5, "production": 50},
    "cpu": "1",
    "memory": "512Mi",
    "concurrency": 80,
    "timeout": "300s",
}


@object_type
class AcmeDeploy:
    """Deploy services the AcmeCorp way.

    Wraps public daggerverse modules with org-specific defaults:
    - Enforces naming conventions (acme-{team}-{service}-{env})
    - Supports OIDC authentication (CI) and local ADC (developer laptops)
    - Deploys to the org's standard regions
    - Injects required labels and metadata for cost tracking and audit
    - Production services always authenticated (no public endpoints)
    - Git metadata (branch, commit SHA) attached to every deployment
    """

    def _validate_and_resolve(self, team: str, service_name: str, environment: str, region: str):
        """Shared validation and naming logic. Returns the full service name."""
        if region not in ALLOWED_REGIONS:
            msg = f"Region {region} not allowed. Must be one of: {ALLOWED_REGIONS}"
            raise ValueError(msg)
        if environment not in ALLOWED_ENVIRONMENTS:
            msg = f"Environment {environment} not allowed. Must be one of: {ALLOWED_ENVIRONMENTS}"
            raise ValueError(msg)

        return f"acme-{team}-{service_name}-{environment}"

    def _validate_production_branch(self, environment: str, git_branch: str) -> None:
        """Refuse production deployments from non-main branches."""
        if environment == "production" and git_branch != "main":
            msg = (
                f"Production deployment forbidden from branch '{git_branch}'. "
                f"Only the 'main' branch can deploy to production."
            )
            raise ValueError(msg)

    def _authenticate(
        self,
        project_id: str,
        oidc_request_token: dagger.Secret | None = None,
        oidc_request_url: dagger.Secret | None = None,
        gcloud_config: dagger.Directory | None = None,
    ) -> dagger.Container:
        """Authenticate to GCP — CI (OIDC) or local (ADC from host).

        In CI: pass oidc_request_token and oidc_request_url (from GitHub Actions).
        Locally: pass gcloud_config (your ~/.config/gcloud directory).
        """
        if gcloud_config:
            return dag.gcp_auth().gcloud_container_from_host(
                project_id=project_id,
                gcloud_config=gcloud_config,
            )

        if oidc_request_token and oidc_request_url:
            return dag.gcp_auth().gcloud_container_from_github_actions(
                workload_identity_provider="projects/123456/locations/global/workloadIdentityPools/github/providers/github-actions",
                project_id=project_id,
                oidc_request_token=oidc_request_token,
                oidc_request_url=oidc_request_url,
                service_account_email=f"ci-deployer@{project_id}.iam.gserviceaccount.com",
            )

        msg = "Provide either gcloud-config (local) or oidc-request-token + oidc-request-url (CI)"
        raise ValueError(msg)

    @function
    @check
    async def scan(
        self,
        source: Annotated[
            dagger.Directory,
            Doc("Backend source directory"),
            DefaultPath("."),
        ],
        port: Annotated[int, Doc("Application port")] = 8080,
    ) -> str:
        """Scan the built container for HIGH and CRITICAL CVEs.

        Builds the container from source using acme-backend, then runs
        Trivy against it. Fails if any vulnerabilities at HIGH severity
        or above are found. Pinned to a safe Trivy version.
        """
        container = dag.acme_backend().build(source=source, port=port)
        return await dag.trivy(version=TRIVY_VERSION).container(container).output(format="table")

    def _build_labels(
        self, team: str, environment: str,
        git_branch: str = "", git_sha: str = "",
    ) -> dict[str, str]:
        """Standard labels for cost tracking, audit, and ownership."""
        labels = {
            "team": team,
            "environment": environment,
            "managed-by": "dagger",
        }
        if git_branch:
            labels["git-branch"] = git_branch
        if git_sha:
            labels["git-sha"] = git_sha[:8]
        return labels

    @function
    async def cloud_run(
        self,
        source: Annotated[dagger.Directory, Doc("Backend source directory")],
        service_name: Annotated[str, Doc("Service name (without prefix)")],
        team: Annotated[str, Doc("Team name for naming and labels")],
        project_id: Annotated[str, Doc("GCP project ID to deploy to")],
        oidc_request_token: Annotated[dagger.Secret | None, Doc("ACTIONS_ID_TOKEN_REQUEST_TOKEN (CI)")] = None,
        oidc_request_url: Annotated[dagger.Secret | None, Doc("ACTIONS_ID_TOKEN_REQUEST_URL (CI)")] = None,
        gcloud_config: Annotated[dagger.Directory | None, Doc("Host gcloud config dir for local auth (~/.config/gcloud)")] = None,
        environment: Annotated[str, Doc("Target environment")] = "staging",
        region: Annotated[str, Doc("GCP region")] = "europe-west1",
        port: Annotated[int, Doc("Application port")] = 8080,
        repository: Annotated[str, Doc("Artifact Registry repository name (defaults to acme-{team})")] = "",
        git_branch: Annotated[str, Doc("Git branch (for audit labels)")] = "",
        git_sha: Annotated[str, Doc("Git commit SHA (for audit labels)")] = "",
        # NOTE: This flag is exposed here for demo/testing purposes only.
        # In a real production setup, IAM invoker checks should remain enabled
        # and access should be controlled via proper IAM bindings or a load balancer.
        disable_invoker_iam_check: Annotated[bool, Doc("Disable Cloud Run IAM invoker check (for demo/testing only)")] = False,
    ) -> str:
        """Build and deploy a backend service to Cloud Run with AcmeCorp compliance.

        Builds the container from source using acme-backend, scans for
        vulnerabilities, pushes to Artifact Registry, and deploys to
        Cloud Run — all in a single call. Enforces naming conventions,
        region whitelist, production branch gate, and access controls.

        Authentication: pass gcloud-config for local development, or
        oidc-request-token + oidc-request-url for CI (GitHub Actions).

        Production services are never publicly accessible — they require
        IAM authentication. Staging services allow unauthenticated access
        for testing convenience.
        """
        self._validate_production_branch(environment, git_branch)
        full_name = self._validate_and_resolve(team, service_name, environment, region)
        gcloud = self._authenticate(
            project_id=project_id,
            oidc_request_token=oidc_request_token,
            oidc_request_url=oidc_request_url,
            gcloud_config=gcloud_config,
        )
        labels = self._build_labels(team, environment, git_branch, git_sha)

        # Build the container from source
        container = dag.acme_backend().build(source=source, port=port)

        # Scan for vulnerabilities before shipping
        await dag.trivy(version=TRIVY_VERSION).container(container).output(format="table")

        # Publish to Artifact Registry
        image_uri = await dag.gcp_artifact_registry().publish(
            container=container,
            gcloud=gcloud,
            project_id=project_id,
            region=region,
            repository=repository or f"acme-{team}",
            image_name=service_name,
            tag=f"{environment}-latest",
        )

        # Deploy to Cloud Run with org-standard configuration
        url = await dag.gcp_cloud_run().service().deploy(
            gcloud=gcloud,
            service_name=full_name,
            image=image_uri,
            region=region,
            allow_unauthenticated=(environment != "production"),
            min_instances=CLOUD_RUN_DEFAULTS["min_instances"][environment],
            max_instances=CLOUD_RUN_DEFAULTS["max_instances"][environment],
            cpu=CLOUD_RUN_DEFAULTS["cpu"],
            memory=CLOUD_RUN_DEFAULTS["memory"],
            concurrency=CLOUD_RUN_DEFAULTS["concurrency"],
            timeout=CLOUD_RUN_DEFAULTS["timeout"],
            disable_invoker_iam_check=disable_invoker_iam_check,
        )

        return url

    @function
    async def firebase(
        self,
        source: Annotated[dagger.Directory, Doc("Frontend source directory")],
        service_name: Annotated[str, Doc("Service name for the hosting site")],
        team: Annotated[str, Doc("Team name")],
        project_id: Annotated[str, Doc("GCP project ID to deploy to")],
        oidc_request_token: Annotated[dagger.Secret | None, Doc("ACTIONS_ID_TOKEN_REQUEST_TOKEN (CI)")] = None,
        oidc_request_url: Annotated[dagger.Secret | None, Doc("ACTIONS_ID_TOKEN_REQUEST_URL (CI)")] = None,
        gcloud_config: Annotated[dagger.Directory | None, Doc("Host gcloud config dir for local auth (~/.config/gcloud)")] = None,
        environment: Annotated[str, Doc("Target environment")] = "staging",
        git_branch: Annotated[str, Doc("Git branch (for audit trail)")] = "",
    ) -> str:
        """Build and deploy a frontend to Firebase Hosting with AcmeCorp compliance.

        Builds the frontend from source using acme-frontend, then deploys
        to production channel ('live') or a preview channel matching the
        environment name. Production deploys are gated to the main branch only.

        Authentication: pass gcloud-config for local development, or
        oidc-request-token + oidc-request-url for CI (GitHub Actions).
        """
        self._validate_production_branch(environment, git_branch)
        self._validate_and_resolve(team, service_name, environment, region="europe-west1")

        # Authenticate to Firebase: use ADC credentials (local) or OIDC (CI).
        # gcp-firebase has its own auth — it doesn't use gcloud containers.
        firebase_auth: dict = {}
        if gcloud_config:
            credentials = dag.set_secret(
                "firebase_credentials",
                await gcloud_config.file("application_default_credentials.json").contents(),
            )
            firebase_auth["credentials"] = credentials
        elif oidc_request_token and oidc_request_url:
            gcloud = self._authenticate(
                project_id=project_id,
                oidc_request_token=oidc_request_token,
                oidc_request_url=oidc_request_url,
            )
            token_output = await gcloud.with_exec(["gcloud", "auth", "print-access-token"]).stdout()
            firebase_auth["access_token"] = dag.set_secret("firebase_access_token", token_output.strip())

        channel = "live" if environment == "production" else environment
        build_command = f"npm run build -- --configuration={environment}"

        if channel == "live":
            return await dag.gcp_firebase().deploy(
                project_id=project_id,
                source=source,
                build_command=build_command,
                deploy_functions=False,
                **firebase_auth,
            )

        return await dag.gcp_firebase().deploy_preview(
            project_id=project_id,
            channel_id=channel,
            source=source,
            build_command=build_command,
            **firebase_auth,
        )

Its dagger.json:

{
  "name": "acme-deploy",
  "engineVersion": "v0.20.3",
  "sdk": { "source": "python" },
  "dependencies": [
    { "name": "gcp-auth", "source": "github.com/telchak/daggerverse/gcp-auth@gcp-auth/v0.2.1" },
    { "name": "gcp-artifact-registry", "source": "github.com/telchak/daggerverse/gcp-artifact-registry@gcp-artifact-registry/v0.2.0" },
    { "name": "gcp-cloud-run", "source": "github.com/telchak/daggerverse/gcp-cloud-run@gcp-cloud-run/v0.3.0" },
    { "name": "gcp-firebase", "source": "github.com/telchak/daggerverse/gcp-firebase@gcp-firebase/v0.2.0" },
    { "name": "trivy", "source": "github.com/sagikazarmark/daggerverse/trivy@v0.6.0" },
    { "name": "acme-backend", "source": "../acme-backend" },
    { "name": "acme-frontend", "source": "../acme-frontend" }
  ]
}

What the private layer gives you

Compare what a developer writes without the private layer, a full-stack deploy across two GCP services:

# Without private modules — every team reinvents this (per service!)
gcloud = dag.gcp_auth().from_oidc(
    token=oidc_token,
    provider="projects/123456/locations/global/workloadIdentityPools/github/providers/github-actions",
    service_account="ci-deployer@acmecorp-staging.iam.gserviceaccount.com",
    project="acmecorp-staging",
)

# Backend: build, scan, push, deploy
backend = (
    dag.container().from_("python:3.13-slim")
    .with_workdir("/app")
    .with_file("/app/requirements.txt", backend_source.file("requirements.txt"))
    .with_exec(["pip", "install", "-r", "requirements.txt"])
    .with_directory("/app/src", backend_source.directory("src"))
    .with_env_variable("PORT", "8080").with_exposed_port(8080)
    .with_entrypoint(["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080"])
)
# Vulnerability scan — easy to forget, and every team configures it differently
await dag.trivy().container(backend).output("table")
image_uri = await dag.gcp_artifact_registry().publish(
    container=backend, gcloud=gcloud, project_id="acmecorp-staging",
    region="europe-west1", repository="acme-backend", image_name="api",
    tag="staging-latest",
)
backend_url = await dag.gcp_cloud_run().service().deploy(
    gcloud=gcloud, service_name="acme-backend-api-staging", image=image_uri,
    region="europe-west1", allow_unauthenticated=True,
)

# Frontend: build, deploy
dist = (
    dag.container().from_("node:20-slim").with_workdir("/app")
    .with_directory("/app", frontend_source)
    .with_exec(["npm", "ci"])
    .with_exec(["npx", "ng", "build", "--configuration=production"])
    .directory("/app/dist/angular-frontend/browser")
)
token_output = await gcloud.with_exec(["gcloud", "auth", "print-access-token"]).stdout()
access_token = dag.set_secret("firebase_access_token", token_output.strip())
frontend_url = await dag.gcp_firebase().deploy_preview(
    project_id="acmecorp-staging", channel_id="staging", source=dist,
    access_token=access_token, skip_build=True,
)

With the private layer, the same full-stack deploy becomes:

# With private modules — two calls, compliance baked in everywhere
backend_url = await dag.acme_deploy().cloud_run(
    source=backend_source, service_name="api", team="backend",
    project_id="acmecorp-staging",
)
frontend_url = await dag.acme_deploy().firebase(
    source=frontend_source, service_name="web", team="frontend",
    project_id="acmecorp-staging",
)

Two calls instead of thirty lines. No authentication boilerplate, no naming convention to remember. The naming convention, region whitelist, authentication flow, label requirements, vulnerability scanning, production branch gating, Cloud Run resource defaults, SBOM generation — all of it is encoded once in the private modules and enforced everywhere. A new team member doesn't need to know which base image is approved, that containers are Trivy-scanned before shipping, or that production services must not be publicly accessible. The modules handle it.

Consuming Modules in Your Pipelines

With both layers in place, here's what a typical project's .dagger/ looks like at AcmeCorp:

Project Structure

dagger-ci-demo/
├── backend/                  # FastAPI application code
│   ├── src/
│   ├── tests/
│   └── requirements.txt
├── frontend/                 # Angular application code
│   ├── src/
│   └── package.json
├── .dagger/                  # Dagger module (CI/CD pipeline)
│   ├── dagger.json
│   ├── pyproject.toml
│   └── src/dagger_ci_demo/
│       └── main.py
└── .github/
    └── workflows/
        └── ci.yml            # GitHub Actions (just calls dagger)

`dagger.json`: Dependencies

The project only depends on the private layer. Public modules are transitive dependencies; the project never references them directly:

{
  "name": "dagger-ci-demo",
  "engineVersion": "v0.20.3",
  "sdk": { "source": "python" },
  "dependencies": [
    { "name": "acme-backend", "source": "github.com/telchak/acme-dagger-modules/acme-backend@v1.0.0" },
    { "name": "acme-frontend", "source": "github.com/telchak/acme-dagger-modules/acme-frontend@v1.0.0" },
    { "name": "acme-deploy", "source": "github.com/telchak/acme-dagger-modules/acme-deploy@v1.0.0" }
  ]
}

To install these modules, run dagger install for each dependency:

dagger install github.com/telchak/acme-dagger-modules/acme-backend@v1.0.0
dagger install github.com/telchak/acme-dagger-modules/acme-frontend@v1.0.0
dagger install github.com/telchak/acme-dagger-modules/acme-deploy@v1.0.0

Pipeline Code

If you want to follow along, you can come back to the same dagger-ci-demo repository from Part 1. If you're starting fresh, run dagger init --sdk=python to scaffold a new module. Once you've installed the AcmeCorp modules above, replace the content of .dagger/src/dagger_ci_demo/main.py with the following:

This is where everything comes together. The project pipeline doesn't contain a single line of build logic, authentication boilerplate, or deployment configuration. It orchestrates the private modules:

"""CI/CD pipeline for dagger-ci-demo — powered by AcmeCorp modules."""

from typing import Annotated

import dagger
from dagger import Doc, dag, function, object_type


@object_type
class DaggerCiDemo:
    """Full-stack pipeline for the Angular + FastAPI product."""

    @function
    async def test(
        self,
        backend_source: Annotated[dagger.Directory, Doc("Backend source directory")],
        frontend_source: Annotated[dagger.Directory, Doc("Frontend source directory")],
    ) -> str:
        """Run all tests (backend + frontend)."""
        backend_result = await dag.acme_backend().test(source=backend_source)
        frontend_result = await dag.acme_frontend().test(source=frontend_source)
        return f"Backend:\n{backend_result}\n\nFrontend:\n{frontend_result}"

    @function
    async def lint(
        self,
        backend_source: Annotated[dagger.Directory, Doc("Backend source directory")],
        frontend_source: Annotated[dagger.Directory, Doc("Frontend source directory")],
    ) -> str:
        """Run linting on all source code."""
        backend_result = await dag.acme_backend().lint(source=backend_source)
        frontend_result = await dag.acme_frontend().lint(source=frontend_source)
        return f"Backend:\n{backend_result}\n\nFrontend:\n{frontend_result}"

    @function
    async def deploy(
        self,
        backend_source: Annotated[dagger.Directory, Doc("Backend source directory")],
        frontend_source: Annotated[dagger.Directory, Doc("Frontend source directory")],
        project_id: Annotated[str, Doc("GCP project ID to deploy to")],
        oidc_request_token: Annotated[dagger.Secret | None, Doc("ACTIONS_ID_TOKEN_REQUEST_TOKEN (CI)")] = None,
        oidc_request_url: Annotated[dagger.Secret | None, Doc("ACTIONS_ID_TOKEN_REQUEST_URL (CI)")] = None,
        gcloud_config: Annotated[dagger.Directory | None, Doc("Host gcloud config dir for local auth (~/.config/gcloud)")] = None,
        environment: Annotated[str, Doc("Target environment")] = "staging",
    ) -> str:
        """Test and deploy the full stack.

        1. Tests both apps (fails fast if anything breaks)
        2. Deploys backend to Cloud Run, frontend to Firebase Hosting
           (build, scan, push are handled internally by acme-deploy)

        Authentication: pass gcloud-config for local development, or
        oidc-request-token + oidc-request-url for CI (GitHub Actions).
        """
        # 1. Test first — fail fast
        await self.test(
            backend_source=backend_source,
            frontend_source=frontend_source,
        )

        # 2. Deploy both apps — build, scan, push, and deploy are all internal
        backend_url = await dag.acme_deploy().cloud_run(
            source=backend_source,
            service_name="api",
            team="product",
            project_id=project_id,
            oidc_request_token=oidc_request_token,
            oidc_request_url=oidc_request_url,
            gcloud_config=gcloud_config,
            environment=environment,
        )

        frontend_url = await dag.acme_deploy().firebase(
            source=frontend_source,
            service_name="web",
            team="product",
            project_id=project_id,
            oidc_request_token=oidc_request_token,
            oidc_request_url=oidc_request_url,
            gcloud_config=gcloud_config,
            environment=environment,
        )

        return f"Backend:  {backend_url}\nFrontend: {frontend_url}"

Read this pipeline again. There are zero dag.container() calls. No base image choices. No pip install. No npm ci. No Artifact Registry URIs. No Cloud Run flags. No Firebase channel logic. No Trivy configuration. No branch gating logic. The only project-specific value is the project_id, which the caller provides. Every other operational detail is encapsulated in the three private modules, and every project at AcmeCorp gets the same standards, the same security, the same compliance, without any effort from the project team.

Try It Locally

You can test the pipeline right away on your machine. The test and lint functions work without any cloud credentials:

# Run the test suite on both apps
dagger call test \
  --backend-source=./backend \
  --frontend-source=./frontend

# Run linting on both apps
dagger call lint \
  --backend-source=./backend \
  --frontend-source=./frontend

The deploy function requires GCP credentials. In CI, it uses OIDC tokens from GitHub Actions. For local development, you can pass your host's gcloud config directory instead:

# Deploy from your laptop using local ADC credentials
dagger call deploy \
  --backend-source=./backend \
  --frontend-source=./frontend \
  --project-id=acmecorp-staging \
  --gcloud-config=$HOME/.config/gcloud \
  --environment=staging

# Check that the module loads and all dependencies resolve
dagger functions

GitHub Actions Workflow

The workflow file stays minimal. It just calls dagger:

name: CI/CD

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read
  id-token: write

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      - name: Lint
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: lint --backend-source=./backend --frontend-source=./frontend

      - name: Test
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: test --backend-source=./backend --frontend-source=./frontend

  deploy:
    if: github.ref == 'refs/heads/main'
    needs: check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      - name: Deploy to staging
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: >
            deploy
            --backend-source=./backend
            --frontend-source=./frontend
            --oidc-token=env:ACTIONS_ID_TOKEN_REQUEST_TOKEN
            --environment=staging

The entire CI configuration calls three Dagger functions: lint, test, and deploy. All the logic (base images, dependency installation, build steps, authentication, compliance, registry push, Cloud Run flags, Firebase channels) lives in typed, testable, version-pinned modules.

The Local Developer Experience

This is where Dagger's CLI really shines. A developer clones dagger-ci-demo, and without reading any documentation, they can discover every available pipeline function, run them individually, and even inspect intermediate build artifacts, all from their terminal.

Discovering what's available:

$ dagger functions
Name    Description
deploy  Build, test, and deploy the full stack.
lint    Run linting on all source code.
test    Run all tests (backend + frontend).

Every function name and description comes directly from the Python docstrings we wrote. The Doc() annotations on parameters become --help text:

$ dagger call deploy --help
Deploy the full stack.

  1. Tests both apps (fails fast if anything breaks)
  2. Builds both apps via acme-backend and acme-frontend
  3. Deploys backend to Cloud Run, frontend to Firebase Hosting

  Authentication: pass gcloud-config for local development, or
  oidc-request-token + oidc-request-url for CI (GitHub Actions).

USAGE
  dagger call deploy [arguments]

ARGUMENTS
      --backend-source Directory    Backend source directory (required)
      --frontend-source Directory   Frontend source directory (required)
      --project-id string           GCP project ID to deploy to (required)
      --oidc-request-token Secret   ACTIONS_ID_TOKEN_REQUEST_TOKEN (CI)
      --oidc-request-url Secret     ACTIONS_ID_TOKEN_REQUEST_URL (CI)
      --gcloud-config Directory     Host gcloud config dir for local auth (~/.config/gcloud)
      --environment string          Target environment (default "staging")

No wiki page. No Confluence document. No Slack thread asking "how do I deploy again?" The CLI is the documentation, and it's always in sync with the code because it's generated from it.

Running individual functions:

A developer working on the backend doesn't need to run the full pipeline. Since acme-backend, acme-frontend, and acme-deploy are already installed as dependencies in the project's dagger.json, they can call any dependency's functions by name:

# Run backend tests only (calling the project's own pipeline function)
dagger call test --backend-source=./backend --frontend-source=./frontend

# Lint just the backend (calling the installed dependency directly)
dagger call -m acme-backend lint --source=./backend

# Generate an SBOM for the backend
dagger call -m acme-backend sbom --source=./backend -o ./sbom.json

# Run the frontend dependency audit
dagger call -m acme-frontend audit --source=./frontend

The -m flag selects which module to call. When you use it with a dependency name (acme-backend), Dagger resolves it from your dagger.json, so there's no need for the full Git path. You can also use -m with a full path to call modules that aren't installed, which is useful for trying out a new daggerverse module before adding it as a dependency.

Inspecting build artifacts with terminal:

This is one of Dagger's most powerful features for local development. Any function that returns a Container can be chained with terminal to drop into an interactive shell inside the built container:

$ dagger call -m acme-backend build --source=./backend terminal
● Attaching terminal:
container: Container!
.from(address: "docker.io/library/python:3.13-slim@sha256:739e7213..."): Container!
withWorkdir /app
.withDirectory(
    ┆ ┆ path: "/app"
    ┆ ┆ source: Host.directory(path: "./backend"): Directory!
    ┆ ): Container!
.withMountedCache(
    ┆ ┆ path: "/root/.cache/pip"
    ┆ ┆ cache: cacheVolume(key: "acme-pip", ...): CacheVolume!
    ┆ ): Container!
withExec pip install -r requirements.txt
withEnvVariable PORT=8080
.withExposedPort(port: 8080): Container!
.withLabel(name: "org.opencontainers.image.vendor", value: "AcmeCorp"): Container!
.withEntrypoint(args: ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080"]): Container!

dagger /app $

This means a developer can build the exact same container that will ship to Cloud Run, then poke around inside it: check installed packages, verify file layout, test imports, inspect environment variables. The container is identical whether you're on your laptop or in CI, because Dagger builds are hermetic.

The same trick works with any Container return type in the chain, even transitive dependencies you haven't installed directly. For example, to inspect the authenticated gcloud container that acme-deploy uses under the hood:

# Call a transitive dependency by its full Git path
$ dagger call -m github.com/telchak/daggerverse/gcp-auth@gcp-auth/v0.2.1 \
    gcloud-container \
    --credentials=file:./my-service-account-key.json \
    --project-id=acmecorp-staging \
    terminal

root@def456:/# gcloud auth list
         Credentialed Accounts
ACTIVE   ACCOUNT
*        ci-deployer@acmecorp-staging.iam.gserviceaccount.com
root@def456:/# gcloud config get-value project
acmecorp-staging

Chaining functions from the CLI:

Any Container or Directory return type can be chained further. This means you can compose operations from the CLI the same way you compose them in code:

# Build the backend, then run a custom command inside the result
dagger call -m acme-backend \
  build --source=./backend \
  with-exec --args="python","-c","import sys; print(sys.version)" \
  stdout

# Build the frontend dist, then list its contents
dagger call -m acme-frontend \
  build --source=./frontend \
  entries

Why this matters:

Every function the platform team writes (build, test, lint, sbom, audit, deploy) is immediately available as a CLI command with full documentation, tab completion, and composability. A developer who has never seen the codebase can run dagger functions, understand what's available, run dagger call test --help, understand what parameters are needed, and execute the pipeline, all without leaving the terminal. The same commands work identically in CI. There is no "it works on my machine" gap.

Testing Modules

Modules are code, and code should be tested. Each module repository should include integration tests that validate the public API:

"""Integration tests for AcmeCorp modules."""

import dagger
from dagger import dag, function, object_type


@object_type
class Tests:
    """Test suite for AcmeCorp private modules.

    Validates naming conventions, region whitelists, production branch
    gating, and module composition without hitting real GCP services.
    """

    @function
    async def test_region_whitelist_rejects_invalid(self) -> str:
        """Verify that regions outside the whitelist are rejected."""
        try:
            await dag.acme_deploy().cloud_run(
                source=dag.directory().with_new_file("requirements.txt", "fastapi\n"),
                service_name="test",
                team="platform",
                project_id="acmecorp-staging",
                region="asia-east1",  # Not in whitelist
            )
            return "[FAIL] Should have rejected region"
        except dagger.ExecError:
            return "[OK] Invalid region rejected"

    @function
    async def test_invalid_environment_rejected(self) -> str:
        """Verify that unknown environments are rejected."""
        try:
            await dag.acme_deploy().cloud_run(
                source=dag.directory().with_new_file("requirements.txt", "fastapi\n"),
                service_name="test",
                team="platform",
                project_id="acmecorp-staging",
                environment="development",  # Not staging/production
            )
            return "[FAIL] Should have rejected environment"
        except dagger.ExecError:
            return "[OK] Invalid environment rejected"

    @function
    async def test_production_deploy_requires_main_branch(self) -> str:
        """Verify that production deploys are blocked from non-main branches."""
        try:
            await dag.acme_deploy().cloud_run(
                source=dag.directory().with_new_file("requirements.txt", "fastapi\n"),
                service_name="test",
                team="platform",
                project_id="acmecorp-prod",
                environment="production",
                git_branch="feature/my-branch",
            )
            return "[FAIL] Should have blocked non-main production deploy"
        except dagger.ExecError:
            return "[OK] Production deploy blocked from non-main branch"

    @function
    async def test_backend_build_returns_container(self) -> str:
        """Verify that acme-backend build produces a container with the expected port."""
        source = (
            dag.directory()
            .with_new_file("requirements.txt", "fastapi\nuvicorn\n")
            .with_new_file("src/__init__.py", "")
            .with_new_file("src/main.py", (
                "from fastapi import FastAPI\n"
                "app = FastAPI()\n"
                "@app.get('/health')\n"
                "def health(): return {'status': 'ok'}\n"
            ))
        )
        container = dag.acme_backend().build(source=source)
        ports = await container.exposed_ports()
        port_numbers = [await p.port() for p in ports]
        if 8080 not in port_numbers:
            return f"[FAIL] Expected port 8080, got {port_numbers}"
        return "[OK] Backend build produces container with port 8080"

# Run tests locally
dagger call -m tests/ test-region-whitelist-rejects-invalid
dagger call -m tests/ test-production-deploy-requires-main-branch
dagger call -m tests/ test-backend-build-returns-container

For example, running the region whitelist test shows exactly how the module enforces compliance at the function level:

$ dagger call -m tests/ test-region-whitelist-rejects-invalid

✔ tests: Tests! 3.4s
✘ .testRegionWhitelistRejectsInvalid: String! 27.3s ERROR
✘ AcmeDeploy.cloudRun(
  ┆ container: Container.from(address: "alpine"): Container!
  ┆ serviceName: "test"
  ┆ team: "platform"
  ┆ oidcToken: setSecret(name: "test-token"): Secret!
  ┆ region: "asia-east1"
  ): String! 11.7s ERROR

ValueError: Region asia-east1 not allowed. Must be one of: ['europe-west1', 'us-central1']

The test deliberately passes a forbidden region (asia-east1) and verifies the module rejects it with a clear error. The branch gating test works the same way:

$ dagger call -m tests/ test-production-deploy-requires-main-branch

✔ tests: Tests! 0.0s
✘ .testProductionDeployRequiresMainBranch: String! 23.2s ERROR
✘ AcmeDeploy.cloudRun(
  ┆ container: Container.from(address: "alpine"): Container!
  ┆ serviceName: "test"
  ┆ team: "platform"
  ┆ oidcToken: setSecret(name: "test-token"): Secret!
  ┆ environment: "production"
  ┆ gitBranch: "feature/my-branch"
  ): String! 11.9s ERROR

ValueError: Production deployment forbidden from branch 'feature/my-branch'.
  Only the 'main' branch can deploy to production.

This is policy-as-code: region restrictions, naming conventions, and branch gating are all enforced by the module itself, not by documentation or code review.

Versioning Strategy

For a monorepo with multiple modules, use per-module Git tags:

telchak/acme-dagger-modules/
├── acme-backend/           → git tag: acme-backend/v1.0.0
├── acme-frontend/          → git tag: acme-frontend/v1.0.0
├── acme-deploy/            → git tag: acme-deploy/v1.2.0
└── tests/

Automate with conventional commits:

# "feat(acme-deploy): add Firebase Hosting support" → acme-deploy/v1.3.0 (minor)
# "fix(acme-deploy): handle empty labels"           → acme-deploy/v1.2.1 (patch)

Consuming teams pin to a version and upgrade on their own schedule:

{ "name": "acme-deploy", "source": "github.com/telchak/acme-dagger-modules/acme-deploy@v1.0.0" }

When the platform team releases v1.3.0, teams can upgrade by bumping the version in dagger.json. The typed API ensures that breaking changes are caught at development time, not in production.

Toolchains: Zero-Code Consumption

So far, every project at AcmeCorp writes a .dagger/ module, a few dozen lines of Python that orchestrates the private modules. That's already much simpler than raw CI scripts, but Dagger recently introduced a feature that eliminates even that: toolchains.

A toolchain is a Dagger module installed directly into your project's dagger.json with no SDK code required. You install it, and its functions become available via dagger call and dagger check immediately.

Official toolchains

Dagger already maintains a growing set of official toolchains for popular tools and frameworks, ready to install with zero configuration:

Toolchain	What it does	Install
`pytest`	Python test framework	`dagger toolchain install github.com/dagger/pytest`
`jest`	JavaScript testing (Jest)	`dagger toolchain install github.com/dagger/jest`
`vitest`	Test framework for Vite.js	`dagger toolchain install github.com/dagger/vitest`
`eslint`	JavaScript static analysis	`dagger toolchain install github.com/dagger/eslint`
`prettier`	Multi-language code formatter	`dagger toolchain install github.com/dagger/prettier`
`biomejs`	Web application formatter (Biome)	`dagger toolchain install github.com/dagger/biomejs`
`mochajs`	JavaScript testing (Mocha)	`dagger toolchain install github.com/dagger/mochajs`
`bun`	Bun JavaScript runtime	`dagger toolchain install github.com/dagger/bun`

These official toolchains follow the single-tool-per-module philosophy: each integrates deeply with one specific tool, provides sensible defaults, and exposes checks that run via dagger check. You can mix and match them freely: install pytest for your backend and eslint + prettier for your frontend, and dagger check runs all of them.

But the real power of toolchains is that you can build your own, which is exactly what AcmeCorp's private modules become when you add @check and DefaultPath.

Making modules toolchain-ready

The key ingredients are the @check decorator and DefaultPath("."). Adding @check to an existing function makes it discoverable via dagger check. Adding DefaultPath(".") to the source parameter makes it optional. Dagger automatically passes the project's source directory when no --source flag is provided. Here's what it looks like in acme-backend:

from dagger import DefaultPath, Doc, check, dag, function, object_type

@object_type
class AcmeBackend:
    # ... build(), sbom() as before ...

    @function
    @check
    async def lint(
        self,
        source: Annotated[
            dagger.Directory,
            Doc("Python backend source directory"),
            DefaultPath("."),
        ],
    ) -> str:
        """Run linting (ruff) on the source code."""
        ...

    @function
    @check
    async def test(
        self,
        source: Annotated[
            dagger.Directory,
            Doc("Python backend source directory"),
            DefaultPath("."),
        ],
        coverage: Annotated[bool, Doc("Enforce minimum coverage threshold")] = True,
    ) -> str:
        """Run the test suite with AcmeCorp conventions."""
        ...

No separate wrapper functions needed. @check goes directly on the existing lint and test functions. They keep their str return type and remain callable via dagger call as before, but they're now also registered as checks.

acme-frontend follows the same pattern: lint, test, and audit all get @check + DefaultPath(".").

The zero-code project setup

Our dagger-ci-demo pipeline is powerful, but it still required writing Python code. What if a team just wants standard CI, with no custom pipeline logic at all?

Let's try it. In our dagger-ci-demo repository, we can strip everything back: remove the .dagger/ directory, remove the SDK, remove the dependencies, and instead use dagger toolchain install to add AcmeCorp modules directly as toolchains:

# Remove the existing module code
rm -rf .dagger/

# Reset dagger.json to a minimal config (no SDK, no dependencies)
echo '{"name": "dagger-ci-demo"}' > dagger.json

# Install AcmeCorp modules as toolchains
dagger toolchain install github.com/telchak/acme-dagger-modules/acme-backend@acme-backend/v1.0.0
dagger toolchain install github.com/telchak/acme-dagger-modules/acme-frontend@acme-frontend/v1.0.0
dagger toolchain install github.com/telchak/acme-dagger-modules/acme-deploy@acme-deploy/v1.0.0

This produces a dagger.json with no SDK, no Python, no .dagger/ directory:

{
  "name": "dagger-ci-demo",
  "engineVersion": "v0.20.3",
  "toolchains": [
    {
      "name": "acme-backend",
      "source": "github.com/telchak/acme-dagger-modules/acme-backend@acme-backend/v1.0.0",
      "pin": "2fc60326472c2141e7d6cc6cdfb3382f2d38b303"
    },
    {
      "name": "acme-frontend",
      "source": "github.com/telchak/acme-dagger-modules/acme-frontend@acme-frontend/v1.0.0",
      "pin": "2fc60326472c2141e7d6cc6cdfb3382f2d38b303"
    },
    {
      "name": "acme-deploy",
      "source": "github.com/telchak/acme-dagger-modules/acme-deploy@acme-deploy/v1.0.0",
      "pin": "f6d4de715efc78610f43900334eabd667d918b84"
    }
  ]
}

That's it. One file, no code. Now they can run:

# Call any function directly
$ dagger call acme-backend lint --source=./backend
$ dagger call acme-backend test --source=./backend
$ dagger call acme-backend build --source=./backend terminal

No Python. No .dagger/ directory. No pipeline code to maintain. The team gets AcmeCorp-compliant linting, testing, and building from a single JSON file.

Customizing toolchain defaults

Toolchains work out of the box, but you can override any argument's default value directly in dagger.json, without touching code. The customizations array lets you scope overrides to specific functions using the function field.

For example, say one team's backend doesn't follow the default coverage threshold and they want to target a specific function's argument:

{
  "name": "simple-api",
  "toolchains": [
    {
      "name": "acme-backend",
      "source": "github.com/telchak/acme-dagger-modules/acme-backend@v1.0.0",
      "customizations": [
        {
          "function": ["test"],
          "argument": "coverage",
          "default": "false"
        }
      ]
    }
  ]
}

The function field scopes the override. Here, only the test function's coverage argument is changed. Without a function field, the override targets the module's constructor arguments instead.

This also works for directory paths. When a module uses DefaultPath(".") to pick up your source automatically, you can redirect it to a subdirectory. Use the function field to scope the override to specific functions:

{
  "name": "monorepo-api",
  "toolchains": [
    {
      "name": "acme-backend",
      "source": "github.com/telchak/acme-dagger-modules/acme-backend@v1.0.0",
      "customizations": [
        {
          "function": ["test"],
          "argument": "source",
          "defaultPath": "/services/api"
        },
        {
          "function": ["lint"],
          "argument": "source",
          "defaultPath": "/services/api"
        }
      ]
    }
  ]
}

Each customization targets a specific function via the function field. Without it, the override applies to the module's constructor — not to individual functions. Here, both test and lint checks will automatically operate on ./services/api instead of the project root. No --source flag needed.

You can also filter out checks you're not ready for using ignoreChecks with glob patterns:

{
  "name": "simple-api",
  "toolchains": [
    {
      "name": "acme-backend",
      "source": "github.com/telchak/acme-dagger-modules/acme-backend@v1.0.0",
      "ignoreChecks": [
        "test"
      ]
    }
  ]
}

This is useful for gradual adoption. A team can start with just linting and add test enforcement later. Run dagger check -l to see which checks are active after filtering.

Full-stack toolchains

Let's go back to our dagger-ci-demo repository and set it up as a full-stack toolchain project. Since our backend lives in ./backend and our frontend in ./frontend, we need to tell each toolchain's check functions where to find their source:

{
  "name": "dagger-ci-demo",
  "engineVersion": "v0.20.3",
  "toolchains": [
    {
      "name": "acme-backend",
      "source": "github.com/telchak/acme-dagger-modules/acme-backend@acme-backend/v1.0.0",
      "pin": "2fc60326472c2141e7d6cc6cdfb3382f2d38b303",
      "customizations": [
        {
          "function": ["test"],
          "argument": "source",
          "defaultPath": "/backend"
        },
        {
          "function": ["lint"],
          "argument": "source",
          "defaultPath": "/backend"
        }
      ]
    },
    {
      "name": "acme-frontend",
      "source": "github.com/telchak/acme-dagger-modules/acme-frontend@acme-frontend/v1.0.0",
      "pin": "2fc60326472c2141e7d6cc6cdfb3382f2d38b303",
      "customizations": [
        {
          "function": ["test"],
          "argument": "source",
          "defaultPath": "/frontend"
        },
        {
          "function": ["lint"],
          "argument": "source",
          "defaultPath": "/frontend"
        },
        {
          "function": ["audit"],
          "argument": "source",
          "defaultPath": "/frontend"
        }
      ]
    },
    {
      "name": "acme-deploy",
      "source": "github.com/telchak/acme-dagger-modules/acme-deploy@acme-deploy/v1.0.0",
      "pin": "f6d4de715efc78610f43900334eabd667d918b84",
      "customizations": [
        {
          "function": ["scan"],
          "argument": "source",
          "defaultPath": "/backend"
        }
      ]
    }
  ]
}

Each customizations entry uses the function field to target a specific check function. This tells acme-backend:test and acme-backend:lint to use ./backend, acme-frontend:test, acme-frontend:lint, and acme-frontend:audit to use ./frontend, and acme-deploy:scan to build and scan the backend container from ./backend. Now dagger check runs all checks from all three toolchains with the right source directories:

$ dagger check
✔ acme-backend:lint    (12.9s)  OK
✔ acme-backend:test    (15.2s)  OK
✔ acme-deploy:scan     (18.4s)  OK
✔ acme-frontend:lint   (58.0s)  OK
✔ acme-frontend:test   (62.0s)  OK
✔ acme-frontend:audit  (25.6s)  OK

When a check fails, Dagger surfaces the full error inline — no need to dig through logs. Here's what it looks like when acme-frontend:audit catches a real vulnerability in a transitive dependency:

$ dagger check
✔ acme-backend:lint    (0.1s)   OK
✔ acme-backend:test    (0.1s)   OK
✘ acme-frontend:audit  (26.2s)  ERROR
┇ .audit(
  ┆ source: context /tmp/dagger-ci-demo/frontend (exclude: [])
  ) ›
✘ withExec npm audit --audit-level=moderate  (2.3s)  ERROR
# npm audit report

undici  7.0.0 - 7.23.0
Severity: high
Undici has an HTTP Request/Response Smuggling issue
  - https://github.com/advisories/GHSA-2mjp-6q6p-2qxm
fix available via `npm audit fix --force`
Will install @angular/build@20.3.21, which is a breaking change
node_modules/undici
  @angular/build  21.0.0-next.0 - 21.2.3
  Depends on vulnerable versions of undici

2 high severity vulnerabilities
! exit code: 1

✔ acme-frontend:lint   (4.4s)   OK
✔ acme-frontend:test   (4.4s)   OK

Notice how the failing check doesn't block the others — all six run in parallel, and Dagger reports the full audit output so the team knows exactly what to fix. This is the value of @check: the module author defined the security audit once, and every project that installs the toolchain gets it automatically.

Six checks, three toolchains, zero lines of code. The CI workflow becomes even simpler:

# .github/workflows/ci.yml
name: CI
on: [push, pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: check

One step. One command. Every AcmeCorp standard enforced.

When to use toolchains vs. pipeline code

Both consumption models are valid. The right choice depends on how much customization the team needs:

	Toolchains	Pipeline code (`.dagger/`)
Setup	`dagger.json` only, no SDK code	`.dagger/` module with Python/Go/TS
Best for	Standard projects that follow org patterns	Custom orchestration, conditional logic
Checks	Automatic via `dagger check`	Define your own `@check` functions
Customization	`customizations` in JSON	Full SDK, compose modules in code
Deployment	Install `acme-deploy` as toolchain	Call `dag.acme_deploy()` in pipeline

Most teams start with toolchains. When they need custom orchestration ("run tests in parallel, then deploy backend before frontend"), they graduate to a .dagger/ module that imports the private modules. The same modules power both paths.

Deploying with `acme-deploy` as a toolchain

Vulnerability scanning shouldn't wait until deploy time. In acme-deploy, we've promoted the Trivy scan from a private helper to a public @check function called scan. It builds the container from source using acme-backend, then runs Trivy against it:

@function
@check
async def scan(
    self,
    source: Annotated[
        dagger.Directory,
        Doc("Backend source directory"),
        DefaultPath("."),
    ],
    port: Annotated[int, Doc("Application port")] = 8080,
) -> str:
    """Scan the built container for HIGH and CRITICAL CVEs."""
    container = dag.acme_backend().build(source=source, port=port)
    return await dag.trivy(version=TRIVY_VERSION).container(container).output(format="table")

Install it as a third toolchain alongside acme-backend and acme-frontend:

dagger toolchain install github.com/telchak/acme-dagger-modules/acme-deploy@acme-deploy/v1.0.0

With the scan function's source customized to point at /backend:

{
  "name": "acme-deploy",
  "source": "github.com/telchak/acme-dagger-modules/acme-deploy@acme-deploy/v1.0.0",
  "customizations": [
    {
      "function": ["scan"],
      "argument": "source",
      "defaultPath": "/backend"
    }
  ]
}

Now dagger check catches CVEs alongside linting and tests:

$ dagger check
✔ acme-backend:lint    OK
✔ acme-backend:test    OK
✔ acme-frontend:lint   OK
✔ acme-frontend:test   OK
✔ acme-frontend:audit  OK
✔ acme-deploy:scan     OK    # ← vulnerability scanning before any deploy

The deployment functions (cloud_run, firebase) stay as regular functions — they're not checks, because they mutate external state. Deployments happen intentionally, with full orchestration, not as part of dagger check.

Implementing the deployment step in CI

To actually deploy, you need a CI workflow that calls acme-deploy's cloud_run and firebase functions after checks pass. This requires some GCP setup first.

Step 1: Set up Workload Identity Federation on GCP

Workload Identity Federation lets GitHub Actions authenticate to GCP without storing service account keys. Create a Workload Identity Pool and Provider tied to your GitHub repository:

# Create the Workload Identity Pool
gcloud iam workload-identity-pools create "github" \
  --location="global" \
  --display-name="GitHub Actions"

# Create the OIDC Provider for GitHub
gcloud iam workload-identity-pools providers create-oidc "github-actions" \
  --location="global" \
  --workload-identity-pool="github" \
  --issuer-uri="https://token.actions.githubusercontent.com" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository" \
  --attribute-condition="assertion.repository_owner == 'your-github-org'"

# Create the CI service account
gcloud iam service-accounts create "ci-deployer" \
  --display-name="CI Deployer" \
  --project="acmecorp-staging"

# Grant Cloud Run permissions (deploy services, push to Artifact Registry)
gcloud projects add-iam-policy-binding "acmecorp-staging" \
  --member="serviceAccount:ci-deployer@acmecorp-staging.iam.gserviceaccount.com" \
  --role="roles/run.admin"

gcloud projects add-iam-policy-binding "acmecorp-staging" \
  --member="serviceAccount:ci-deployer@acmecorp-staging.iam.gserviceaccount.com" \
  --role="roles/artifactregistry.writer"

# Grant Firebase Hosting permissions
gcloud projects add-iam-policy-binding "acmecorp-staging" \
  --member="serviceAccount:ci-deployer@acmecorp-staging.iam.gserviceaccount.com" \
  --role="roles/firebase.admin"

# Cloud Run needs to pull images — grant the service agent access
gcloud projects add-iam-policy-binding "acmecorp-staging" \
  --member="serviceAccount:ci-deployer@acmecorp-staging.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

# Link the service account to the Workload Identity Pool
# This allows GitHub Actions from your repo to impersonate ci-deployer
gcloud iam service-accounts add-iam-policy-binding \
  "ci-deployer@acmecorp-staging.iam.gserviceaccount.com" \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/123456/locations/global/workloadIdentityPools/github/attribute.repository/your-github-org/dagger-ci-demo"

The last command is the critical link: it tells GCP that GitHub Actions workflows running in the your-github-org/dagger-ci-demo repository are allowed to impersonate the ci-deployer service account. Without it, the OIDC token exchange will fail with a permission denied error. The attribute.repository condition ensures that only workflows from your specific repository can authenticate — other repositories in the same GitHub organization cannot.

Step 2: No secrets required

No GCP credentials or identifiers to store in GitHub. The WIF provider, service account, and project mappings are all encoded in the acme-deploy module itself — that's the whole point of centralizing deployment logic. The only values the CI workflow passes are the GitHub Actions OIDC environment variables (ACTIONS_ID_TOKEN_REQUEST_TOKEN and ACTIONS_ID_TOKEN_REQUEST_URL), which are automatically available when the workflow has id-token: write permission.

Step 3: CI workflow with checks + deployment

# .github/workflows/ci.yml
name: CI
on: [push, pull_request]

permissions:
  contents: read
  id-token: write  # Required for WIF OIDC token

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: check

  deploy:
    needs: check
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: >-
            acme-deploy cloud-run
            --source=./backend
            --service-name=api
            --team=backend
            --project-id=acmecorp-staging
            --oidc-request-token=env:ACTIONS_ID_TOKEN_REQUEST_TOKEN
            --oidc-request-url=env:ACTIONS_ID_TOKEN_REQUEST_URL
            --environment=staging
            --region=europe-west1
            --git-branch=${{ github.ref_name }}
            --git-sha=${{ github.sha }}

The check job runs all toolchain checks (lint, test, audit, scan). The deploy job only runs on pushes to main, after checks pass. No google-github-actions/auth step needed — the acme-deploy module handles the OIDC token exchange internally via the gcp-auth Dagger module. GitHub Actions automatically exposes ACTIONS_ID_TOKEN_REQUEST_TOKEN and ACTIONS_ID_TOKEN_REQUEST_URL when the workflow has id-token: write permission. The gcp-auth module uses the oidc-token module to exchange these for a GCP-compatible OIDC token, then authenticates via Workload Identity Federation. Authentication stays inside the pipeline, not in the CI glue.

Trying it locally

For local deployment testing, pass your host's gcloud config directory instead of the OIDC tokens. The acme-deploy module detects which authentication method to use based on the arguments you provide:

Artifact Registry setup: The cloud-run function pushes container images to Google Artifact Registry in the europe-west1 region by default. The target repository is named acme-{team} (e.g. acme-backend for --team=backend), but you can override it with the --repository flag. If the repository doesn't exist yet, create it first:
gcloud artifacts repositories create acme-backend \
  --repository-format=docker \
  --location=europe-west1 \
  --project=acmecorp-production \
  --description="Docker images for the backend team"
This creates a Docker-format repository in your GCP project. You only need to do this once per team/repository.

Cloud Run access for Firebase Hosting: In this setup, the frontend on Firebase Hosting communicates with the backend on Cloud Run through a Firebase Hosting rewrite. The rewrite proxies /api/** requests to your Cloud Run service, so the frontend uses relative URLs and never exposes the backend's address. However, Cloud Run services are private by default — they require IAM authentication for every request, including those from Firebase Hosting. Rather than granting public access (which may be blocked by your organization's IAM policies), you can disable the IAM invoker check using the --disable-invoker-iam-check flag on the cloud-run function. In a production setup, this would typically be handled by your infrastructure-as-code (Terraform, Pulumi, etc.) alongside network-level controls like VPC ingress rules.

# Deploy backend to Cloud Run
dagger call acme-deploy cloud-run \
  --source=./backend \
  --service-name=api \
  --team=backend \
  --project-id=acmecorp-production \
  --gcloud-config=$HOME/.config/gcloud \
  --environment=production \
  --git-branch=main \
  --disable-invoker-iam-check

# Deploy frontend to Firebase Hosting
dagger call acme-deploy firebase \
  --source=./frontend \
  --service-name=web \
  --team=backend \
  --project-id=acmecorp-production \
  --gcloud-config=$HOME/.config/gcloud \
  --environment=production \
  --git-branch=main

Under the hood, acme-deploy calls gcp-auth's gcloud-container-from-host function, which mounts your local Application Default Credentials into the pipeline container. No service account keys needed — just run gcloud auth application-default login once on your machine. The checks (dagger check) don't require any GCP authentication at all.

Module Visibility with Dagger Cloud

Dagger Cloud provides a dashboard that automatically tracks every module used across your organization. When you enable module scanning on your GitHub repositories, Dagger Cloud indexes them and shows:

Which modules exist, their API documentation, and versions
Which pipelines depend on which modules
Pipeline traces linked to module function calls

This gives the platform team visibility into module adoption without any manual tracking, which is useful for knowing when it's safe to deprecate an old version or to identify teams that haven't upgraded yet.

Summary: The Module Architecture

Image generated with Google's Gemini "Nano Banana Pro"

                     ┌────────────────────────────────────┐
                     │          Daggerverse               │
                     │   gcp-auth, gcp-cloud-run,         │
                     │   gcp-firebase, python-build,      │
                     │   angular, trivy, ...              │
                     └──────────────┬─────────────────────┘
                                    │
                                    ▼
                     ┌────────────────────────────────────┐
                     │     Private Modules Repo           │
                     │   acme-backend, acme-frontend,     │
                     │   acme-deploy                      │
                     │                                    │
                     │   Encodes: security, compliance,   │
                     │   naming, defaults, audit,         │
                     │   vulnerability scanning           │
                     └──────────────┬─────────────────────┘
                                    │
                  ┌─────────────────┼─────────────────┐
                  ▼                 ▼                  ▼
          ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
          │  Service A   │  │  Service B   │  │  Service C   │
          │              │  │              │  │              │
          │  .dagger/    │  │  dagger.json │  │  dagger.json │
          │  main.py     │  │  only        │  │  only        │
          │              │  │              │  │              │
          │  Custom      │  │  Toolchains  │  │  Toolchains  │
          │  pipeline    │  │  + checks    │  │  + checks    │
          │  code        │  │              │  │              │
          │  (SDK)       │  │  Zero code   │  │  Zero code   │
          └──────────────┘  └──────────────┘  └──────────────┘

           Full control:        Standard projects:
           compose modules      install as toolchains,
           in Python/Go/TS,     customize via JSON,
           custom orchestration  dagger check runs all

Two consumption paths, same modules. Teams that need custom orchestration (conditional deploys, parallel builds, multi-step workflows) write a .dagger/ pipeline module that imports the private modules via SDK code. Teams that follow the standard pattern install them as toolchains in dagger.json and get lint, test, and audit checks with zero code. Both paths enforce the same security, compliance, and naming rules.

The pattern scales: public modules handle the generic complexity (GCP auth, Trivy scanning, language builds), private modules enforce your rules (naming, branch gating, vulnerability gates, resource defaults), and project pipelines stay simple, whether they're written in Python or declared in JSON. When something changes (a new compliance requirement, a GCP API deprecation, a CVE severity policy update), you update one module and every pipeline benefits on their next version bump.

What's Next

In Part 4, we'll add AI to the mix, by exploring Dagger's agentic capabilities. The modules we built today become the building blocks of a deterministic pipeline that an AI agent (Daggie) generates for you. And when that pipeline fails, coding agents (Monty for Python, Angie for Angular) analyze the error and post fix suggestions directly on the PR.

The pipeline stays fixed, deterministic, and fast. The AI writes it, reviews it, and fixes it. We'll also approach how we could build a Spec-driven development agentic platform on top of those foundations.

The full source code for AcmeCorp's private modules is available at github.com/telchak/acme-dagger-modules. Clone it as a starting point for your own private module library.

Up Next: Part 4: The AI-Native CI/CD Stack: Agents, Modules, and Spec-Driven Development

This is Part 3 of a 4-part series. Follow for updates.

Tags: #cicd #dagger #gcp #platform-engineering #modules

CI/CD in the Era of AI and Platform Engineering: A Deep Dive into Dagger CI (Part 2)

Sami Chibani — Fri, 27 Mar 2026 00:17:57 +0000

Part 2: Decoupling Pipelines from Infrastructure

In a Platform Engineering world, developers shouldn't care where CI runs. The pipeline code stays the same; only the runner configuration changes.

In Part 1, we built our first Dagger pipeline: typed, testable code that runs identically on a developer's laptop and in GitHub Actions. We left off with a minimal, working CI workflow: a few lines of YAML that run dagger call on GitHub's shared runners, with zero infrastructure to manage.

That setup works, but it has limits. Every job starts with a cold Dagger cache, builds share GitHub's 2-vCPU runners with the rest of the world, and there's no persistent state between runs. For a small project, that's fine. For a team shipping dozens of PRs a day, it's a bottleneck.

This part is more SRE/Platform Engineer oriented. We'll dig into caching internals, runner infrastructure, and the operational tooling that makes Dagger viable at scale. We'll explore three approaches, each with different tradeoffs between simplicity, performance, and control.

The Runner Spectrum

Approach	Setup Time	Cost	Control	Best For
GitHub Actions + Dagger Action	5 minutes	$0 (free tier)	Low	Getting started, small teams
Depot.dev Managed Runners	15 minutes	~$0.04/min	Medium	Fast builds, medium teams
Kubernetes ARC + Shared Cache	2-4 hours	Your infra	Full	Enterprise, high volume

Option 1: GitHub Actions with dagger-for-github

The simplest approach. Zero infrastructure to manage.

Create .github/workflows/ci.yml:

name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      - name: Build and Test
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: all --source=.

The dagger/dagger-for-github action installs Dagger, starts the engine, and runs your command.

Understanding Dagger's Cache Architecture

Before we talk about CI runners, we need to understand how Dagger caches work, because caching is probably the single most impactful optimization you'll make. OpenMeter cut their CI pipeline from 25 minutes to 5 minutes primarily through better caching, and the same principles apply here.

Dagger maintains three distinct cache categories:

Cache Type	What It Stores	Scope
Layers	Build instructions and results of API calls	Content-addressed by input hash
Volumes	Persistent filesystem directories (package caches, build artifacts)	Named, persisted across engine sessions
Function calls	Return values from module function invocations	Keyed by function signature + arguments

The engine runs a background garbage collector that keeps cache storage below 75% of total disk capacity while preserving at least 20% free space. You can inspect and manage the cache manually:

# View cache usage summary
dagger core engine local-cache entry-set

# View detailed cache entries
dagger core engine local-cache entry-set entries

# Prune all unused cache entries
dagger core engine local-cache prune

# Prune using the default GC policy
dagger core engine local-cache prune --use-default-policy

Now let's see how these three cache types work in practice.

Layer Cache: Content-Addressed Operation Caching

Every operation in Dagger's execution graph is cached based on the hash of its inputs. When you chain .with_exec(["pip", "install", ..."]) after .with_file("requirements.txt", ...), Dagger hashes the requirements file content and the base container state. If neither changed, the entire install step is skipped, not re-executed.

This is similar to Docker layer caching, but with an important difference: Dagger's cache is content-addressed, not order-dependent. Docker invalidates all layers after the first changed layer. Dagger evaluates each operation independently based on its actual inputs.

The ordering trick: Just like Docker, you should place instructions that change infrequently before those that change often. Install dependencies before copying source code:

# Good: dependencies cached separately from source code
(
    dag.container()
    .from_("python:3.13-slim")
    .with_file("/app/requirements.txt", source.file("requirements.txt"))
    .with_exec(["pip", "install", "-r", "requirements.txt"])  # Cached until requirements.txt changes
    .with_directory("/app/src", source.directory("src"))        # Only this invalidates on code changes
)

# Bad: source code change invalidates dependency install
(
    dag.container()
    .from_("python:3.13-slim")
    .with_directory("/app", source)                             # Any file change invalidates everything below
    .with_exec(["pip", "install", "-r", "requirements.txt"])   # Re-runs even if requirements.txt didn't change
)

This single reordering can save minutes per build. The layer cache lives in the Dagger engine's local storage, by default at /var/lib/dagger.

Cache Volumes: Persistent Package Manager Storage

Beyond layer caching, Dagger provides cache volumes: named persistent directories you mount into containers. These are the equivalent of Docker's --mount=type=cache but integrated into the Dagger API.

The typical use case: package manager caches. Without a cache volume, pip install re-downloads every package on every run even if the layer cache misses. With a cache volume, pip finds the packages already in its local cache and skips the download.

@function
async def build_backend(
    self,
    source: Annotated[dagger.Directory, Doc("Backend source directory")],
) -> dagger.Container:
    """Build the FastAPI backend container."""
    return (
        dag.container()
        .from_("python:3.13-slim")
        .with_workdir("/app")
        .with_file("/app/requirements.txt", source.file("requirements.txt"))
        # Mount a persistent cache volume for pip's download cache.
        # dag.cache_volume("pip") creates a named volume that persists
        # across runs. pip won't re-download packages it already has.
        .with_mounted_cache("/root/.cache/pip", dag.cache_volume("python-pip"))
        .with_exec(["pip", "install", "-r", "requirements.txt"])
        .with_directory("/app/src", source.directory("src"))
        .with_env_variable("PORT", "8080")
        .with_exposed_port(8080)
        .with_entrypoint(["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080"])
    )

Here's the pattern for common package managers:

# Python (pip)
.with_mounted_cache("/root/.cache/pip", dag.cache_volume("python-pip"))
.with_exec(["pip", "install", "-r", "requirements.txt"])

# Python (uv) — dramatically faster with cache
.with_mounted_cache("/root/.cache/uv", dag.cache_volume("python-uv"))
.with_exec(["uv", "pip", "install", "--system", "-r", "requirements.txt"])

# Node.js (npm)
.with_mounted_cache("/root/.npm", dag.cache_volume("node-npm"))
.with_exec(["npm", "ci"])

# Go
.with_mounted_cache("/go/pkg/mod", dag.cache_volume("go-mod"))
.with_mounted_cache("/root/.cache/go-build", dag.cache_volume("go-build"))
.with_exec(["go", "build", "./..."])

Cache volumes are scoped to the Dagger engine instance. On your laptop, they persist indefinitely. On an ephemeral CI runner, they need to be persisted externally (we'll cover how below).

How the Three Caches Stack

Here's what happens on a typical CI run where you changed one Python source file:

Function call cache: checks if build_backend() was called with the same source directory hash. If yes, returns the cached result immediately (entire function skipped). If no, proceeds to step 2.
Layer cache: the from_("python:3.13-slim") step is cached (image didn't change). The with_file("requirements.txt", ...) step is cached (file didn't change). The pip install step is cached (requirements + base image didn't change). Only the with_directory("/app/src", ...) step and everything after it re-runs.
Cache volume: if the pip install step does need to re-run (e.g., you added a new dependency), pip finds most packages already downloaded in the cache volume. Only the new package is fetched.

The result: a build that takes 3 minutes cold can drop to 30 seconds when only source code changed, and even a dependency change is fast because the volume cache eliminates most network I/O.

Persisting Cache on Ephemeral Runners

GitHub Actions runners are ephemeral. Every job starts from a clean VM. Without persistence, every run starts with a cold cache.

Why `actions/cache` Doesn't Work Here

Your first instinct might be to cache /var/lib/dagger with actions/cache@v4. Unfortunately, this doesn't work. Dagger stores its state inside a Docker volume named dagger-engine, not directly on the host filesystem. The path /var/lib/dagger only exists inside the engine container, not on the host where actions/cache can reach it.

The Solution: Docker Volume Caching

Since the Dagger engine already persists its state in a named Docker volume (dagger-engine), we just need a way to back up and restore that volume across ephemeral CI runs. BYK/docker-volume-cache-action does exactly that:

name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      # 1. Restore the Dagger engine's Docker volume from cache.
      #    Use a per-job cache key so parallel matrix jobs don't
      #    overwrite each other. restore-keys falls back to any
      #    existing cache for this OS if the exact key doesn't match.
      - name: Restore Dagger engine cache
        uses: BYK/docker-volume-cache-action/restore@v1
        with:
          key: dagger-engine-${{ runner.os }}-backend
          restore-keys: |
            dagger-engine-${{ runner.os }}-
          volumes: dagger-engine

      # 2. Run Dagger — the _EXPERIMENTAL_DAGGER_RUNNER_HOST env var
      #    tells the CLI to use a Docker volume named "dagger-engine"
      #    for engine state. This is what makes the restore/save work.
      - name: Build and Test
        uses: dagger/dagger-for-github@v8.4.1
        env:
          _EXPERIMENTAL_DAGGER_RUNNER_HOST: "docker-image://registry.dagger.io/engine:v0.20.3?volume=dagger-engine"
        with:
          version: "0.20.3"
          verb: call
          args: test-backend --source=./backend

      # 3. Stop the engine so the volume is consistent before saving
      - name: Stop Dagger engine
        if: always()
        run: docker stop $(docker ps -q -f 'name=dagger-engine') 2>/dev/null || true

      # 4. Save the volume back to cache — only on main branch.
      #    PRs restore from main's cache but don't save, avoiding
      #    ~1m30 of overhead per job on every pull request.
      - name: Save Dagger engine cache
        if: always() && github.ref == 'refs/heads/main'
        uses: BYK/docker-volume-cache-action/save@v1
        with:
          key: dagger-engine-${{ runner.os }}-backend
          volumes: dagger-engine

A few things to note about this setup:

_EXPERIMENTAL_DAGGER_RUNNER_HOST tells the Dagger CLI to store engine state in a Docker volume named dagger-engine. Without this, the CLI creates an internal volume with an auto-generated name that the cache action can't find.
Per-job cache keys (e.g. -backend, -frontend) are important when you run parallel matrix jobs. With a shared key, the last job to save would overwrite the cache with only its own data. Per-job keys keep each job's cache independent.
Save only on main avoids wasting ~1m30 per job on pull requests. PRs still benefit from restoring the cache built by main.
restore-keys fallback allows a cache miss on the exact key to fall back to any cache for the same OS, which is useful when a job runs for the first time but other jobs have already populated a cache.

On the first run, the cache is cold and everything executes. On the second run, the Docker volume is restored with all Dagger layers, cache volumes, and function call results intact, and Dagger picks up right where it left off.

Performance note: This approach introduces a "cold start" overhead on every job — the restore step decompresses the volume at the start, and the save step compresses it at the end. For a ~1 GB engine cache, this adds roughly 30-60 seconds per job. This overhead partially offsets Dagger's built-in caching gains, making it the least performant of the three runner options. It's a good starting point, but if build speed is critical, Depot runners or self-hosted ARC runners with persistent volumes (Options 2 and 3 below) eliminate this overhead entirely.

Note: GitHub Actions caches are scoped to a branch (with fallback to the default branch) and expire after 7 days of inactivity, with a 10 GB limit per repository. This is generous enough for most projects — but if you hit the limit, Dagger Cloud's distributed caching (covered below) is the next step.

The important point: none of this caching logic is CI-specific. The .with_mounted_cache() calls and operation ordering live in your pipeline code. They work identically on your laptop, on GitHub Actions, on Depot, or on a self-hosted runner. The only CI-specific piece is the volume cache restore/save steps that persist the engine state, and even that becomes unnecessary with persistent runners (see Options 2 and 3 below) or Dagger Cloud.

Dagger Cloud: Distributed Caching and Pipeline Observability

Dagger Cloud is Dagger's centralized control plane. It provides three capabilities that become increasingly valuable as your pipeline usage grows: pipeline visualization, distributed caching, and module management.

Why Dagger Cloud?

The Docker volume caching approach above works well for small projects, but has limitations:

10 GB limit per repository: large monorepos or Docker-heavy builds can blow through this
Branch-scoped: cache from main is available to PRs, but PRs don't share cache between each other
Single runner: the cache only benefits the specific runner that uploaded it
Save/restore overhead: compressing and decompressing a multi-GB Docker volume adds 30-60 seconds per job

Dagger Cloud solves all of these with a distributed cache that syncs layers and volumes across every Dagger Engine connected to your organization. Cache volumes are downloaded at the start of each run and uploaded back at completion. Layers are fetched on demand as the engine needs them. This means a build on one runner benefits from the cache populated by a completely different runner, even on a different branch.

OpenMeter reported cutting their CI from 25 minutes to 10 minutes with Dagger Cloud caching alone (a 2.5x improvement) before adding faster runners for the full 5x speedup.

Setting Up a Dagger Cloud Account

Pricing:

Plan	Cost	Users	Features
Individual	Free	1	Pipeline visualization, traces
Team	$50/month	Up to 10	Visualization + distributed caching + module sharing
Enterprise	Custom	Unlimited	Dedicated support, SLA, advanced features

All plans include a 14-day free trial of Team features.

Step 1: Create your account.

Head to dagger.io/cloud and create an account directly with your GitHub or Google account. Follow the guided setup: create your organization (alphanumeric + dashes, must be unique), select a plan, and optionally invite teammates.

You can then authenticate from the CLI:

dagger login

This opens your browser with a verification link. Confirm the unique key matches what the CLI displays, and you're connected.

Tip: Use your company name or team name for the organization — you can't change it later.

Step 2: Get your token.

Once logged in, navigate to your organization settings:

https://dagger.cloud/{your-org-name}/settings?tab=Tokens

Click the eye icon to reveal your token. Copy it. You'll need it for CI configuration.

Step 3: Add the token to GitHub Actions.

Go to your repository → Settings → Secrets and variables → Actions → New repository secret. Name it DAGGER_CLOUD_TOKEN and paste your token.

Step 4: Update your workflow.

Add the cloud-token parameter to the Dagger action:

- name: Build and Test
  uses: dagger/dagger-for-github@v8.4.1
  with:
    version: "0.20.3"
    verb: call
    args: all --source=.
    cloud-token: ${{ secrets.DAGGER_CLOUD_TOKEN }}

That's it. You can now remove the volume cache restore/save steps. Dagger Cloud handles cache persistence automatically. Alternatively, keep both for redundancy during the transition.

Dagger Cloud also supports GitLab CI, CircleCI, Jenkins, and Argo Workflows. The setup is the same: store DAGGER_CLOUD_TOKEN as a secret/variable in your CI system, and Dagger picks it up automatically via the environment variable.

Pipeline Visualization and Traces

Every Dagger run connected to Dagger Cloud generates a trace, a visual breakdown of your pipeline execution showing:

Which operations executed and their duration
Which operations were cache hits (skipped) vs cache misses (re-executed)
Dependency relationships between operations
Error locations and output for failed steps

This is invaluable for optimization. Instead of guessing which steps are slow, you can see exactly where time is spent and whether your caching strategy is working. If a step that should be cached keeps re-executing, the trace will show you.

Public traces: Dagger Cloud automatically detects if traces come from public repositories and makes them publicly accessible by default. You can toggle this in organization settings under Visibility.

Exemple of Dagger pipeline trace with Gantt view

Install the GitHub App (Optional)

For even tighter integration, install the Dagger Cloud GitHub App. This adds Dagger pipeline status checks directly to your pull requests, with links to the trace view for each run.

Module Management

If you publish Dagger modules (like we do in Part 3), Dagger Cloud can scan your GitHub repositories and display module information: API documentation, activity history, dependencies, and linked traces. Enable this through Settings → Git Sources → Install the GitHub Application → select repositories → Enable module scanning.

Modules page view from Dagger Cloud

Pros: Zero infrastructure setup, free tier for individuals, works with any GitHub repository (personal or organization), automatic updates.
Cons: Slowest of the three options due to volume save/restore overhead on every job, shared infrastructure, 10 GB cache limit per repository (Dagger Cloud removes this limit).

Option 2: Depot.dev Managed Runners

Depot provides managed GitHub runners optimized for container builds. The standout features: Dagger integration with persistent caching and native arm64 support.

Warning: Depot runners require your repository to be part of a GitHub organization. They do not work with personal GitHub repositories. If you're working on a personal repo, you'll need to create an organization (free tier is sufficient) and transfer or fork the repository there before you can use Depot runners.

name: CI (Depot)

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build:
    runs-on: depot-ubuntu-latest,dagger=0.20.3
    steps:
      - uses: actions/checkout@v6

      - name: Build and Test
        run: dagger call all --source=.

That's it. No depot/setup-action, no dagger-for-github action, no environment variables. By appending dagger=0.20.3 to the runs-on label, Depot pre-configures the runner with the Dagger engine, persistent cache, and all necessary plumbing. You just call dagger directly.

Depot runners have 16 CPU / 32GB RAM standard (vs GitHub's 2 CPU / 7GB), persistent NVMe caching, and native arm64 with no emulation. A build that takes 8 minutes on GitHub's free runners often completes in under 2 minutes on Depot.

Pros: 4-10x faster, persistent cache, native multi-arch, simple migration.
Cons: Paid service (~$0.04/minute), another vendor dependency.

Option 3: Self-Hosted Kubernetes with ARC

For maximum control, run your own runners on Kubernetes using GitHub's Actions Runner Controller (ARC).

Architecture

┌──────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster                          │
│                                                               │
│  ┌───────────────────────────────────────────────────────┐   │
│  │     ARC Controller (Namespace: arc-systems)            │   │
│  └───────────────────────────────────────────────────────┘   │
│                            │                                  │
│                            ▼                                  │
│  ┌───────────────────────────────────────────────────────┐   │
│  │   arc-runners namespace                                │   │
│  │                                                        │   │
│  │   ┌──────────┐  ┌──────────┐  ┌──────────┐           │   │
│  │   │ Runner 1 │  │ Runner 2 │  │ Runner 3 │           │   │
│  │   │ (pod)    │  │ (pod)    │  │ (pod)    │           │   │
│  │   └─────┬────┘  └─────┬────┘  └─────┬────┘           │   │
│  │         │              │              │                │   │
│  │         │    kube-pod:// or Unix socket                │   │
│  │         │              │              │                │   │
│  │   ┌─────▼──────────────▼──────────────▼─────┐         │   │
│  │   │      Dagger Engine (Helm chart)         │         │   │
│  │   │      DaemonSet or StatefulSet           │         │   │
│  │   │                                         │         │   │
│  │   │      /var/lib/dagger (cache)            │         │   │
│  │   │      hostPath (DS) or PVC (STS)         │         │   │
│  │   └─────────────────────┬───────────────────┘         │   │
│  └─────────────────────────┼─────────────────────────────┘   │
│                             │                                 │
│                             ▼                                 │
│                  ┌─────────────────────┐                     │
│                  │    Dagger Cloud     │  (optional)          │
│                  │    Magicache        │                     │
│                  └─────────────────────┘                     │
└──────────────────────────────────────────────────────────────┘

The key architectural difference from Options 1 and 2: the Dagger engine runs as a shared, long-lived process (DaemonSet or StatefulSet) rather than being started fresh inside each runner. Runner pods are lightweight: they just run the GitHub Actions runner process and connect to the engine. This separation means the engine's cache persists across jobs, and runner pods can scale independently from the engine.

Prerequisites: Setting Up the GitHub App

ARC authenticates to GitHub via a GitHub App. You need to create one before deploying the infrastructure.

Step 1: Create the app.

Go to your GitHub App creation page:

Organization: https://github.com/organizations/<YOUR_ORG>/settings/apps/new
Personal account: https://github.com/settings/apps/new

Fill in the form:

GitHub App name: something descriptive like ARC Dagger Runners
Homepage URL: your organization's URL
Webhook: uncheck Active (ARC doesn't use webhooks)

Set the required Repository permissions:

Permission	Access
Actions	Read-only
Administration	Read & write
Checks	Read-only
Metadata	Read-only

If your runners will be shared across an organization (recommended), also set this Organization permission:

Permission	Access
Self-hosted runners	Read & write

Under "Where can this GitHub App be installed?", choose Only on this account. Click Create GitHub App.

On the app's settings page, note the App ID (displayed near the top). Then scroll to Private keys and click Generate a private key. Save the .pem file securely.

Step 2: Install the app.

For an organization (all repos can use the runners):

Go to https://github.com/organizations/<YOUR_ORG>/settings/apps
Click Configure next to your app
Under Repository access, choose All repositories (or select specific ones)
Click Save
Note the Installation ID from the URL: .../installations/<INSTALLATION_ID>

For a single repository:

Go to https://github.com/<OWNER>/<REPO>/settings/installations
Click Configure next to your app
Note the Installation ID from the URL

The gh_config_url in the Terraform module should match the scope:

Organization: https://github.com/your-org
Single repo: https://github.com/your-org/your-repo

Tip: Start with organization-level installation even if you only have one repo today. It's easier to add repos later without changing the Terraform configuration.

Terraform Setup

Writing the raw Helm releases and Kubernetes resources by hand gives you full understanding of the moving parts, but for production use, you want a reusable module that handles the boilerplate. We've open-sourced one that builds on the same GKE and ARC foundations as the official terraform-google-github-actions-runners module and adds Dagger-specific features.

Full source: github.com/telchak/terraform-dagger-runners

The module is organized in three layers:

Layer	Module path	What it does
Engine (root)	`/`	Deploys Dagger engines via Helm. CI-agnostic, cloud-agnostic.
CI platform	`modules/github/`	Adds ARC controller + runner scale sets. Calls root for engine.
Cloud	`modules/github/gke/`	Provisions a GKE cluster. Calls CI platform module.

You pick the layer that matches your situation. Need a full GKE cluster? Use modules/github/gke/. Already have a Kubernetes cluster (GKE, EKS, AKS, on-premise)? Use modules/github/ directly.

Required Google APIs — If you're provisioning a GKE cluster for self-hosted runners, enable these APIs on your GCP project:
gcloud services enable \
  container.googleapis.com \
  compute.googleapis.com \
  iam.googleapis.com \
  cloudresourcemanager.googleapis.com
Kubernetes Engine API (container.googleapis.com) — create and manage GKE clusters

Compute Engine API (compute.googleapis.com) — provision VMs, networks, and disks

IAM API (iam.googleapis.com) — service accounts for nodes and workload identity

Cloud Resource Manager API (cloudresourcemanager.googleapis.com) — project-level operations

Scenario A: Create a new GKE cluster with runners

This is the all-in-one option. The module provisions the GKE cluster, networking, ARC controller, runner scale sets, and Dagger engines:

module "dagger-runners" {
  source = "github.com/telchak/terraform-dagger-runners//modules/github/gke?ref=v0.1.0"

  # GCP
  project_id     = var.project_id
  region         = "us-central1"
  create_network = true
  machine_type   = "n2-standard-4"   # 4 vCPU, 16 GB RAM — fits ~15 runners
  disk_size_gb   = 100
  min_node_count = 1
  max_node_count = 4
  spot           = true              # Safe with StatefulSet — cache is on PVC

  # GitHub App credentials from Step 1 & 2 above.
  # Store these in terraform.tfvars (git-ignored) or use TF_VAR_ env vars.
  gh_app_id              = var.gh_app_id
  gh_app_installation_id = var.gh_app_installation_id
  gh_app_private_key     = var.gh_app_private_key
  gh_config_url          = "https://github.com/your-org"  # org or repo URL

  # Each version gets its own runner scale set with a unique label.
  # Use runs-on: dagger-v0.20 in your workflows.
  dagger_versions = ["0.20.3"]

  min_runners = 0   # Scale to zero when idle
  max_runners = 10

  # StatefulSet mode: persistent PVC cache per engine version.
  # Cache survives pod restarts, node preemptions, and scale-downs.
  # Runners connect to the engine via kube-pod:// protocol.
  engine_mode                         = "statefulset"
  persistent_cache_size               = "100Gi"
  persistent_cache_storage_class_name = "premium-rwo"  # GKE SSD-backed PD

  # Runner pod resources — runners are lightweight (they just connect
  # to the shared Dagger engine), so requests are minimal.
  runner_size_templates = {
    "" = {
      runner_requests = { cpu = "50m", memory = "256Mi" }
      runner_limits   = {}
    }
  }

  # Optional: Dagger Cloud for distributed caching + traces
  dagger_cloud_token = var.dagger_cloud_token
}

Scenario B: Deploy onto an existing Kubernetes cluster

If you already have a Kubernetes cluster (GKE, EKS, AKS, on-premise, k3s), use modules/github/ directly. You configure the kubernetes and helm providers yourself; the module deploys only the ARC controller, runner scale sets, and Dagger engines:

module "dagger-runners" {
  source = "github.com/telchak/terraform-dagger-runners//modules/github?ref=v0.1.0"

  # GitHub App credentials from Step 1 & 2 above.
  gh_app_id              = var.gh_app_id
  gh_app_installation_id = var.gh_app_installation_id
  gh_app_private_key     = var.gh_app_private_key
  gh_config_url          = "https://github.com/your-org"

  dagger_versions = ["0.20.3"]
  min_runners     = 0
  max_runners     = 10

  # DaemonSet mode (default): simplest setup, no PVC needed.
  # One engine per node, ephemeral cache at /var/lib/dagger on the host.
  # Works on any cluster without a StorageClass.
  engine_mode = "daemonset"

  # Or use StatefulSet mode for persistent cache:
  # engine_mode       = "statefulset"
  # persistent_cache_size = "100Gi"
  # Leave persistent_cache_storage_class_name empty to use the cluster's
  # default StorageClass, or set it explicitly:
  #   GKE:         "premium-rwo"
  #   EKS:         "gp3"
  #   AKS:         "managed-premium"
  #   On-premise:  "longhorn", "rook-ceph-block", etc.

  # Optional: Dagger Cloud for distributed caching + traces
  # dagger_cloud_token = var.dagger_cloud_token
}

This is the option to use when you manage your own infrastructure, or when running on a cloud provider the module doesn't have a dedicated cloud layer for yet (e.g. AWS, Azure). The interface is identical; only the cluster provisioning is your responsibility.

Step 3: Deploy

# Store credentials safely — never commit the private key.
export TF_VAR_gh_app_private_key="$(cat /path/to/your-app.pem)"

terraform init
terraform plan \
  -var="project_id=my-project" \
  -var="gh_app_id=123456" \
  -var="gh_app_installation_id=12345678"
terraform apply

Tip: If you manage your GitHub App secret through an external system like HashiCorp Vault or External Secrets Operator, you can pass gh_app_existing_secret_name instead of the credentials — the module will reference the pre-existing Kubernetes secret and skip creating one.

The module handles: ARC controller, runner scale sets, Dagger engine deployment (via the official Dagger Helm chart), StatefulSet RBAC, VPA, Dagger Cloud secret injection, and failed pod cleanup. The key Dagger-specific features:

dagger_versions: list of Dagger engine versions to deploy. Each version creates exact labels (dagger-v0.20.3), minor aliases (dagger-v0.20 pointing to the latest patch), and a dagger-latest label pointing to the newest version overall.
Engine modes: daemonset (one engine per node, ephemeral hostPath cache) or statefulset (one engine per version, persistent PVC cache). StatefulSet is recommended for Spot instances and when cache warmth matters.
_EXPERIMENTAL_DAGGER_RUNNER_HOST: automatically set on each runner to point at the correct Dagger engine for that scale set (Unix socket for DaemonSet, kube-pod:// for StatefulSet).
Dagger Cloud token: injected as a Kubernetes secret and set as DAGGER_CLOUD_TOKEN env var on runner pods when provided. Also enables Magicache on the engine.

Multi-Version Runners

One of the most powerful features is running multiple Dagger versions side-by-side. This is invaluable during version upgrades, as teams can migrate repositories at their own pace:

# Deploy two versions simultaneously
dagger_versions = ["0.19.5", "0.20.3"]

This creates five runner scale sets: exact versions, minor aliases, and a latest pointer:

runs-on: dagger-v0.19.5: exact, pinned to 0.19.5
runs-on: dagger-v0.19: minor alias, points to 0.19.5
runs-on: dagger-v0.20.3: exact, pinned to 0.20.3
runs-on: dagger-v0.20: minor alias, points to 0.20.3
runs-on: dagger-latest: always the newest version (0.20.3)

Teams that want reproducibility pin to an exact version. Teams that want automatic patch updates use the minor alias. Repositories that just need "whatever is current" use dagger-latest.

Runner Size Templates

By default, runner pods have no resource constraints, so they get BestEffort QoS and are the first to be evicted under memory pressure. For production workloads, the module supports size templates that define resource requests and limits per runner. Each label gets a size suffix:

runner_size_templates = {
  # Default size (no suffix): runs-on: dagger-v0.20
  "" = {
    runner_requests = { cpu = "50m", memory = "256Mi" }
    runner_limits   = {}
  }
  # Large size (suffix): runs-on: dagger-v0.20-large
  large = {
    runner_requests = { cpu = "250m", memory = "512Mi" }
    runner_limits   = { cpu = "1", memory = "1Gi" }
  }
}

Runner pods are lightweight: they only run the GitHub Actions runner process. The Dagger engine runs separately as a DaemonSet or StatefulSet and is shared across all runners. This separation means you don't need to give runner pods large CPU/memory allocations; the engine handles the heavy lifting independently.

With the above templates and dagger_versions = ["0.20.3"], you get six labels: dagger-v0.20.3, dagger-v0.20, dagger-latest (default size), and dagger-v0.20.3-large, dagger-v0.20-large, dagger-latest-large. Teams pick the size that matches their workload:

jobs:
  lint:
    runs-on: dagger-v0.20           # default size — fast checks
  build:
    runs-on: dagger-v0.20-large     # heavy builds with work outside Dagger

Node Sizing

The minimum viable node is 2 vCPU / 8 GB RAM, but larger nodes are more cost-efficient because they amortize system pod overhead (kube-dns, ARC controller, Dagger engine) across more runner pods:

Machine type	vCPU	RAM	Runners per node	Best for
`e2-standard-2`	2	8 GB	~5	Dev/test, low concurrency
`n2-standard-4`	4	16 GB	~15	Small teams
`n2-standard-8`	8	32 GB	~30	Medium teams, multi-version
`n2-standard-16`	16	64 GB	~50+	Large teams, high concurrency

Runner counts assume 50m CPU / 256Mi memory requests per runner and ~1.5 vCPU / 2 GB overhead for system pods + engine. Actual capacity depends on your workload.

Caching: Engine Modes and Dagger Cloud

With self-hosted runners, you choose how the Dagger engine stores its cache, and the Terraform module supports all options:

DaemonSet mode (simplest): engine_mode = "daemonset". One engine per node, cache on the host disk via hostPath. No PVC needed; works on any cluster. Cache is ephemeral: lost when the node is recycled. Best for stable node pools or when Dagger Cloud handles cache persistence.

StatefulSet mode (persistent): engine_mode = "statefulset". One engine per Dagger version, cache on a PVC managed by the Helm chart. Cache survives pod restarts, node preemptions, and scale-downs. Runners connect via kube-pod:// protocol (RBAC created automatically). Best for Spot instances and when cache warmth is critical.

Dagger Cloud (complementary): Set dagger_cloud_token in the module. Magicache syncs cache layers across all engines connected to your organization, even across clusters and regions. You also get the trace UI and module management described earlier. Works with both DaemonSet and StatefulSet modes.

You can combine them: StatefulSet for fast local cache on PVC + Dagger Cloud for cross-cluster sharing and observability. This is the recommended production setup.

Vertical Pod Autoscaler (VPA)

Getting resource requests right for the Dagger engine is tricky. Too low and the scheduler under-provisions nodes; too high and you waste capacity. The module includes optional Vertical Pod Autoscaler support to solve this:

# Enable VPA on the cluster (GKE-specific, creates the VPA objects automatically)
enable_vertical_pod_autoscaling = true

# Start in recommendation-only mode — no automatic resizing
vpa_update_mode = "Off"

With vpa_update_mode = "Off", the VPA controller observes actual engine resource usage and produces recommendations without touching the pods. You can inspect them with:

kubectl get vpa -n arc-runners -o yaml

Look for the recommendation section, which shows target, lowerBound, and upperBound for CPU and memory. After a few days of representative workload, use these values to tune dagger_engine_requests:

# Apply the VPA recommendations to your engine requests
dagger_engine_requests = {
  cpu    = "500m"    # from VPA target recommendation
  memory = "16Gi"   # from VPA target recommendation
}

In-Place Pod Resizing (Kubernetes 1.33+)

Starting with Kubernetes 1.33, the InPlacePodVerticalScaling feature gate enables resizing pod resources without restarting them. This is particularly valuable for the Dagger engine, since a restart means losing in-memory state and warming the cache from disk. The module supports this via:

vpa_update_mode = "InPlaceOrRecreate"

With this mode, the VPA will resize the engine pod's CPU and memory in place when possible, falling back to a recreate only if the node can't accommodate the new size. This feature graduates to GA in Kubernetes 1.35, at which point it works without any feature gate configuration.

Tip: Start with "Off" to collect recommendations, then move to "InPlaceOrRecreate" once you're confident in the VPA's suggestions and your cluster supports it. This gives you automatic right-sizing without engine downtime.

Scale to Zero

The min_runners = 0 setting means no idle runners consuming resources. Combined with min_node_count = 0 on the GKE node pool and cluster autoscaling, this creates fully elastic CI infrastructure that costs nearly nothing when idle.

Pros: Full control, scale to zero, private networks, infrastructure as code, multi-version Dagger support.
Cons: Initial setup complexity, Kubernetes expertise required, you manage updates.

Image generated with Google's Gemini Imagen "Nano Banana Pro"

Choosing Your Path

If you...	Choose...
Are just starting with Dagger	GitHub Actions + Dagger Action
Need fast builds without infra work	Depot.dev
Have Kubernetes expertise and volume	Self-hosted ARC
Need air-gapped or private network	Self-hosted ARC
Want the best of both worlds	Depot for dev, ARC for production

Dagger Cloud is orthogonal to the runner choice and works with all three options. Start with the free Individual plan for pipeline visualization and traces. Upgrade to Team ($50/month) when you need distributed caching across runners or when the actions/cache 10 GB limit becomes a bottleneck.

Your pipeline code doesn't change. Only the runner configuration does. That's the decoupling that matters.

What's Next

In Part 3, we'll build a library of reusable Dagger modules: GCP Auth, Artifact Registry, Cloud Run, and Firebase Hosting. These modules compose into a single pipeline that deploys both our Angular frontend and FastAPI backend.

The modules will work on all three runner setups we just configured.

Up Next: Part 3: From Scripts to a Platform: Your CI/CD Module Library

This is Part 2 of a 4-part series. Follow for updates.

Tags: #cicd #dagger #kubernetes #github-actions #platform-engineering

CI/CD in the Era of AI and Platform Engineering: A Deep Dive into Dagger CI (Part 1)

Sami Chibani — Fri, 27 Mar 2026 00:16:43 +0000

Part 1: The CI/CD Bottleneck Nobody Talks About

Platform Engineering matured how we provision infrastructure. It hasn't touched how we build and deploy code.

CI/CD was, a decade ago, at the heart of the DevOps revolution. It deeply transformed the way we ship software by providing continuous automation workflows from code to production. It hasn't changed much since then. And like every tool that has been continuously used for so long, the more years pass, the more the flaws and frustrations seem obvious. The "traditional" way of building CI/CD nowadays often feels cumbersome and rusty.

Among the almost always recurrent flaws that any DevOps engineer has witnessed:

Lack of development ergonomics. For most technologies, the only way to really test pipelines is by pushing the code to the server, waiting, and iterating. It's inefficient and time-consuming. The feedback loop for CI/CD code is worse than for any other code in your organization.
YAML sprawl. As soon as pipelines require more logic and complexity, you rapidly end up with thousands of YAML files to maintain across repositories, each slightly different, each owned by whoever touched it last.
Opaque failure investigation. When things go wrong in a pipeline, there is no intelligence. Developers sift through thousands of log lines to manually investigate, with no programmatic way to diagnose or recover.

Meanwhile, Infrastructure-as-Code solved provisioning. GitOps solved configuration drift. Internal Developer Platforms gave teams golden paths for spinning up services, databases, and environments. But open any repository in your organization and look at the CI/CD: it's still bespoke YAML. In a world where platform teams build abstractions for everything else, CI/CD remains the one piece of developer infrastructure that every team reinvents from scratch.

This series explores what a CI/CD pipeline could look like in 2026, one that addresses all three of these flaws, at the intersection of Platform Engineering and AI agents. To do that, we'll use one of the most exciting tools I've seen in the CI/CD landscape in a long time: Dagger. Dagger reimagines pipelines as typed functions written in real programming languages, running inside containers with deterministic caching, portable across any CI engine. But what makes it particularly relevant for this series is that it doesn't stop at build automation: it has native LLM integration, a composable module system, and an emerging agent ecosystem that lets you go from reusable pipeline modules to AI-powered CI/CD workflows. It's the right tool to explore what happens when Platform Engineering meets AI agents in the CI/CD space.

We'll build pipelines as real code, decouple them from the infrastructure they run on, package them as reusable modules, and eventually compose AI agents that handle review, build, and deployment from a single natural language command.

The Series at a Glance

This is a 4-part series. Each part builds on the previous one, progressively transforming how we think about CI/CD:

Part 1 (this article): Pipelines as real code. We replace YAML with typed, testable, locally-runnable pipeline functions using Dagger, and build a complete CI/CD pipeline for an Angular + FastAPI stack deployed to Google Cloud.
Part 2: Decoupling Pipelines from Infrastructure: Same pipeline code, different runners. We explore three options for scaling: GitHub Actions (free tier), managed runners with Depot.dev, and self-hosted Kubernetes runners with ARC, including a Terraform module for production deployments.
Part 3: From Scripts to a Platform: Your CI/CD Module Library: Modules as the unit of CI/CD reuse. We build a two-layer architecture: public daggerverse modules for generic operations (GCP Auth, Cloud Run, Firebase) wrapped by private organizational modules that enforce compliance, security, and conventions.
Part 4: The AI-Native CI/CD Stack: Agents, Modules, and Spec-Driven Development: We leverage the agentic capabilities of Dagger to enhance the developer experience, automate CI failure remediation, and explore what an Internal Developer Factory looks like when built on Spec-Driven Development principles. The pipeline itself stays fast and deterministic, with no LLM in the hot path.

But first: let's look at the root cause.

The YAML Illusion

YAML-based CI/CD looks simple. A few lines, a run: step, and you have a pipeline. But that simplicity is an illusion. The moment a real-world project needs conditional logic, secret management, reusable workflows, matrix builds, or cross-repository dependencies, every CI platform reaches for its own layer of abstractions on top of YAML.

GitHub Actions has reusable workflows, composite actions, and a marketplace of third-party actions you import by reference. GitLab CI has include:, extends:, and multi-project pipelines. CircleCI has orbs. Azure DevOps has templates and task groups. Each platform reinvented inheritance, composition, and parameterization (poorly) on top of a configuration format that was never designed for it.

The result is a system that is declarative in appearance but imperative in practice. A developer looking at a .github/workflows/deploy.yml that imports three reusable workflows, each with their own if: conditions and secret references, is not reading a simple configuration file. They're reading a distributed program written in a language with no type system, no IDE support, no debugger, and no way to run it locally.

This is the fundamental problem: CI/CD, as it exists today, is not part of the software factory. It sits outside the development workflow. It can't be run on a developer's machine. It can't be unit tested. It can't be refactored with confidence. It can't be debugged step by step. It doesn't benefit from the tooling, the practices, or the discipline that every other piece of software in the organization does.

And yet it's the most critical automation in your delivery chain, the one that decides whether code reaches production.

CI/CD as Software

The fix isn't a better YAML syntax. It's a paradigm shift: treat CI/CD as an integral part of the software factory.

That means CI/CD primitives (building a container, running tests, publishing an artifact, deploying a service) should be programmatic abstractions. Written in real programming languages. With types, functions, imports, error handling. Runnable on a developer's laptop. Testable. Debuggable. Versioned alongside the application code. Reviewable in the same pull request.

This is what Dagger does. It decomposes CI/CD operations into typed functions, available through SDKs in Python, Go, and TypeScript. A pipeline is not a configuration file interpreted by a remote server. It's code that runs inside containers, with content-addressed caching at every step.

A note on language choice — Throughout this series, we'll use Python for all our examples. But everything we write here works identically in Go, TypeScript, or Java. Dagger's SDKs expose the same API surface and the same types across all supported languages — the pipeline behavior is determined by the engine, not the SDK. Pick the language your team is most comfortable with.

from typing import Annotated

import dagger
from dagger import Doc, dag, function, object_type


@object_type
class MyPipeline:
    @function
    async def build(
        self,
        source: Annotated[dagger.Directory, Doc("Application source code")],
    ) -> dagger.Container:
        """Build the application container."""
        return (
            dag.container()
            .from_("python:3.12-slim")
            .with_directory("/app", source)
            .with_workdir("/app")
            .with_exec(["pip", "install", "-r", "requirements.txt"])
            .with_entrypoint(["python", "main.py"])
        )

This is a typed, composable function that:

Runs on your laptop with dagger call build --source=.
Runs in GitHub Actions with the exact same command
Runs in GitLab CI with the exact same command
Can be debugged, unit tested, and refactored like any other code

No YAML translation. No platform-specific quirks. The same code, the same command, everywhere.

The shift is subtle but significant. CI/CD is no longer a separate discipline with its own language, its own debugging workflow, and its own specialists. It's software. Developers can read it, understand it, modify it, and run it, the same way they do with application code. That's the self-service promise of Platform Engineering applied to the one area it hadn't reached yet.

Three Properties That Make It Work

When CI/CD becomes software, three properties emerge that YAML pipelines fundamentally cannot provide.

1. Container-Based Execution

Every operation happens inside a container. When you write:

dag.container().from_("python:3.12-slim").with_exec(["pip", "install", "pytest"])

You're building a container execution graph. The engine takes this graph and executes it efficiently. Your pipeline is reproducible (same inputs, same outputs), isolated (no "works on my machine"), and portable (containers run anywhere).

This is what enables local execution. The same container graph that runs on a CI runner runs on your laptop. No emulation, no mocking, just the real thing. A developer can iterate on a pipeline the same way they iterate on application code: change, run, observe, repeat. The feedback loop goes from "push and wait 10 minutes" to "run and see in seconds."

2. Content-Addressed Caching

Every operation is cached based on the hash of its inputs:

@function
async def test(
    self,
    source: Annotated[dagger.Directory, Doc("Application source code")],
) -> str:
    """Run tests with intelligent caching."""
    return await (
        dag.container()
        .from_("python:3.12-slim")
        .with_directory("/app", source)       # Cached if source unchanged
        .with_exec(["pip", "install", "-r", "requirements.txt"])  # Cached if requirements unchanged
        .with_exec(["pytest"])                 # Only re-runs if dependencies changed
        .stdout()
    )

Change a test file? Only the test execution re-runs. Change requirements.txt? Dependencies reinstall, but the base image stays cached. This is content-addressed: Dagger hashes the actual content, not timestamps.

Because it's deterministic, caching works identically on your laptop and in CI. A developer running dagger call test --source=. after changing one test file gets the same instant feedback a CI runner would, with no cold start and no full rebuild.

3. A Module System

Modules are reusable pipeline components that can be published and shared:

# Use a published module
dagger call -m github.com/purpleclay/daggerverse/golang@v0.5.0 build --source=.

# Or reference your own
dagger call -m ./.dagger build --source=.

Modules have typed interfaces, documentation, and versioning. They compose:

@function
async def deploy(
    self,
    source: Annotated[dagger.Directory, Doc("Application source")],
) -> str:
    # Authenticate to GCP
    gcp = dag.gcp_auth().from_service_account_key(key=sa_key, project=project)

    # Publish to Artifact Registry
    image_uri = await dag.gcp_artifact_registry(gcp).publish(
        container=await self.build(source),
        repository="my-repo", image="my-app", tag="latest",
    )

    # Deploy to Cloud Run
    return await dag.gcp_cloud_run(gcp).service().deploy(
        name="my-service", image=image_uri, region="us-central1",
    )

This is the Platform Engineering pattern applied to CI/CD. A platform team builds and maintains modules (gcp-auth, artifact-registry, cloud-run) the same way they'd maintain Terraform modules or Helm charts. Developers consume them as typed dependencies. They don't need to know how GCP authentication works internally; they call dag.gcp_auth() and get an authenticated container. Self-service, with guardrails.

Putting It Into Practice

Enough theory. For the rest of this series, we're going to simulate, as closely as we can, a production deployment. Not a toy example with a single endpoint and a hello world container, but a realistic two-component product with a frontend, a backend, authentication between them, and deployment to a real cloud provider.

The goal is a deep dive: from writing the first pipeline function on your laptop, all the way to a multi-agent CI/CD system that reviews, builds, and deploys both apps from a single natural language command.

Prerequisites

To follow along, you'll need three things set up:

Dagger CLI: Install the Dagger engine and CLI on your machine. This is what runs your pipelines locally and in CI.
→ Install Dagger
A GitHub account: We'll use GitHub as our code server and CI runner (via GitHub Actions). The companion repository with all example code is public.
→ Create a GitHub account if you don't have one
A Google Cloud account with a project: GCP is our deployment target throughout the series. You'll need a project with billing enabled (the free tier is sufficient for everything we do here).
→ Create a GCP account
→ Create a GCP project

Required Google APIs — Enable these APIs on your GCP project before deploying. You can do it from the API Library or via gcloud:
gcloud services enable \
  run.googleapis.com \
  artifactregistry.googleapis.com \
  firebasehosting.googleapis.com \
  firebase.googleapis.com \
  iam.googleapis.com \
  cloudresourcemanager.googleapis.com
Cloud Run API (run.googleapis.com) — deploy the FastAPI backend

Artifact Registry API (artifactregistry.googleapis.com) — store container images

Firebase Hosting API (firebasehosting.googleapis.com) — host the Angular frontend

Firebase API (firebase.googleapis.com) — Firebase Anonymous Auth for the frontend

IAM API (iam.googleapis.com) — manage service accounts and OIDC federation

Cloud Resource Manager API (cloudresourcemanager.googleapis.com) — project-level operations

You'll also want Docker installed locally (Dagger uses it under the hood), and Python 3.12+ and Node.js 20+ for the example apps themselves.

The Product: Angular Frontend + FastAPI Backend

Our product is two apps that talk to each other:

Angular 21 SPA on Firebase Hosting: displays items from the API, authenticates via Firebase Anonymous Auth
FastAPI REST API on Cloud Run: serves items, validates Firebase ID tokens from the frontend

The frontend authenticates anonymously with Firebase, gets an ID token, and sends it as a Bearer header to the backend. The backend validates the token using google-auth. This is a realistic pattern for any Firebase + Cloud Run stack.

Initializing the Module

Clone the companion repository:

git clone https://github.com/telchak/dagger-ci-demo.git
cd dagger-ci-demo

We have a monorepo with backend/ and frontend/. Rather than creating a separate Dagger module for each app, we'll initialize a single module at the repository root. This gives us one place to define pipeline functions for the entire product:

dagger init --sdk=python --name=dagger-ci-demo

This creates a .dagger/ directory (Dagger's convention, like .github/ for GitHub Actions) containing a dagger.json manifest, a pyproject.toml, and a source scaffold where we'll write our pipeline functions.

Key Concepts: Types, Functions, and Chaining

Before looking at the code, let's clarify three core Dagger concepts (docs):

Types. The Dagger API provides specialized types that represent CI/CD primitives. The ones we'll use most:

Container: an OCI container image with its filesystem and configuration
Directory: a filesystem tree (your source code, build output, etc.)
File: a single file artifact
Secret: sensitive data (API keys, credentials) that Dagger never logs or caches
Service: a running network service (for integration testing, local dev, etc.)

These types are the building blocks. They replace the implicit, untyped operations of YAML pipelines with explicit, typed abstractions.

Image generated with Google's Gemini Imagen "Nano Banana Pro"

Functions. A Dagger module exposes functions, decorated with @function, that accept these types as parameters and return them as outputs. Functions are the unit of work. Each one is a self-contained operation: build a container, run tests, publish an image.

Chaining. Each Dagger type comes with its own methods, and these methods return the same (or related) types. This enables chaining: you pass the output of one operation directly into the next, building up a pipeline step by step. When you write dag.container().from_("python:3.12-slim").with_exec(["pip", "install", "..."]), you're chaining three operations on a Container type. Dagger evaluates this chain lazily: nothing executes until a terminal operation (like .stdout() or .publish()) triggers the full graph.

The Pipeline: Backend and Frontend

Here's what we'll write at .dagger/src/dagger_ci_demo/main.py, a single class with functions for both apps:

"""CI/CD pipeline for the dagger-ci-demo product."""

from typing import Annotated

import dagger
from dagger import Doc, dag, function, object_type


# @object_type marks this class as a Dagger module.
# Its public methods become callable pipeline functions.
@object_type
class DaggerCiDemo:
    """Pipeline for the Angular + FastAPI product."""

    # --- Backend (FastAPI) ---

    @function
    async def build_backend(
        self,
        source: Annotated[dagger.Directory, Doc("Backend source directory")],
    ) -> dagger.Container:
        """Build the FastAPI backend container.

        Returns a Container — the core Dagger type representing an OCI image.
        The caller can publish it, export it, or chain further operations on it.
        """
        return (
            dag.container()
            # Start from a base image — returns a Container
            .from_("python:3.12-slim")
            # Set the working directory inside the container
            .with_workdir("/app")
            # Copy only requirements.txt first — this layer is cached
            # as long as requirements.txt doesn't change, even if
            # source code changes. This is the Docker cache optimization
            # pattern, expressed as a function chain.
            .with_file("/app/requirements.txt", source.file("requirements.txt"))
            # Install dependencies — cached if requirements.txt unchanged
            .with_exec(["pip", "install", "--no-cache-dir", "-r", "requirements.txt"])
            # Copy application source — only this layer and below re-run
            # when source code changes
            .with_directory("/app/src", source.directory("src"))
            # Configure the container for Cloud Run
            .with_env_variable("PORT", "8080")
            .with_exposed_port(8080)
            .with_entrypoint([
                "uvicorn", "src.main:app",
                "--host", "0.0.0.0", "--port", "8080", "--workers", "2",
            ])
        )

    @function
    async def test_backend(
        self,
        source: Annotated[dagger.Directory, Doc("Backend source directory")],
    ) -> str:
        """Run the backend test suite.

        Returns a string (stdout) — a terminal type that triggers execution.
        Everything before .stdout() is lazily evaluated; Dagger only runs
        the container when it needs the output.
        """
        return await (
            dag.container()
            .from_("python:3.12-slim")
            .with_workdir("/app")
            # Mount the full source directory (tests need everything)
            .with_directory("/app", source)
            .with_exec(["pip", "install", "--no-cache-dir", "-r", "requirements.txt"])
            # Run pytest — if this command fails, the function raises an error
            .with_exec(["pytest", "-v", "tests/"])
            # .stdout() is the terminal operation: it triggers execution
            # of the entire chain and returns the command's output as a string
            .stdout()
        )

    # --- Frontend (Angular) ---

    @function
    async def build_frontend(
        self,
        source: Annotated[dagger.Directory, Doc("Frontend source directory")],
    ) -> dagger.Directory:
        """Build the Angular frontend for production.

        Returns a Directory — the build output (dist/) that can be
        deployed to Firebase Hosting, exported to the host filesystem,
        or passed to another function.
        """
        return (
            dag.container()
            .from_("node:20-slim")
            .with_workdir("/app")
            # Copy package files first for dependency caching
            .with_file("/app/package.json", source.file("package.json"))
            .with_file("/app/package-lock.json", source.file("package-lock.json"))
            # npm ci installs exact versions from lockfile — cached if
            # package files unchanged
            .with_exec(["npm", "ci"])
            # Copy the rest of the source
            .with_directory("/app/src", source.directory("src"))
            .with_file("/app/angular.json", source.file("angular.json"))
            .with_file("/app/tsconfig.json", source.file("tsconfig.json"))
            .with_file("/app/tsconfig.app.json", source.file("tsconfig.app.json"))
            # Build for production — Angular CLI outputs to dist/
            .with_exec(["npx", "ng", "build", "--configuration=production"])
            # Extract just the build output directory.
            # .directory() returns a Directory type — stripping away the
            # container and keeping only the filesystem artifact we need.
            .directory("/app/dist/angular-frontend/browser")
        )

    @function
    async def test_frontend(
        self,
        source: Annotated[dagger.Directory, Doc("Frontend source directory")],
    ) -> str:
        """Run the Angular test suite."""
        return await (
            dag.container()
            .from_("node:20-slim")
            .with_workdir("/app")
            .with_directory("/app", source)
            .with_exec(["npm", "ci"])
            .with_exec(["npx", "ng", "test"])
            .stdout()
        )

Let's unpack what's going on here.

The class DaggerCiDemo is a module, a collection of functions packaged together. The @object_type decorator registers it with the Dagger engine, and every @function method becomes a callable operation, both from the CLI and from other modules.

Each function takes a Directory as input, Dagger's type for a filesystem tree. When you pass --source=./backend from the CLI, Dagger packages that directory and hands it to the function as a typed Directory object. No string paths, no glob patterns, just a real, content-addressed filesystem reference.

The function bodies are chains of operations on Dagger types. dag.container().from_("python:3.12-slim") creates a Container from a base image. Each .with_*() call returns a new Container with the modification applied. Nothing executes yet. Dagger builds a directed acyclic graph (DAG) of operations. Execution only happens when a terminal operation like .stdout() or .publish() forces the engine to resolve the graph.

This is why caching works so well. Dagger hashes each node in the graph based on its inputs. If requirements.txt hasn't changed, the pip install step is a cache hit, instantly. If only a source file changed, only the layers after .with_directory("/app/src", ...) re-execute. Same content-addressed logic, on your laptop and in CI.

Notice how build_backend returns a Container while build_frontend returns a Directory. The return type determines what the caller can do next. A Container can be published to a registry, started as a service, or chained further. A Directory can be deployed to static hosting, exported, or mounted into another container. Types guide composition. They make it clear what flows where.

Run It Locally

With Dagger installed (see prerequisites above), let's explore the module from the repository root.

Discovering Functions

Before running anything, you can inspect what the module exposes:

dagger functions

Name             Description
build-backend    Build the FastAPI backend container.
build-frontend   Build the Angular frontend for production.
test-backend     Run the backend test suite.
test-frontend    Run the Angular test suite.

This is auto-generated. Dagger reads the @function decorators, their docstrings, and the Doc() annotations on parameters, and produces CLI documentation from them, in any SDK language. The same metadata that describes a function in Python generates the same CLI interface as it would in Go or TypeScript.

You can drill into any function with --help:

dagger call build-backend --help

Build the FastAPI backend container.

Returns a Container — the core Dagger type representing an OCI image.
The caller can publish it, export it, or chain further operations on it.

USAGE
  dagger call build-backend [arguments]

ARGUMENTS
  --source Directory   Backend source directory [required]

Everything here (the description, the argument name, the type, the [required] marker) comes directly from your code. The Doc("Backend source directory") annotation you wrote in the Annotated[dagger.Directory, Doc("...")] parameter becomes the argument's help text. The function's docstring becomes the command's description. There's no separate documentation to write or maintain. The code is the documentation, and Dagger keeps it in sync for both the CLI and the SDK.

This is what makes self-service work. A developer who has never seen this module before can run dagger functions, pick the operation they need, check its arguments with --help, and run it, without reading source code or asking the platform team.

Running Functions

# Test the backend
dagger call test-backend --source=./backend

# Build the backend container
dagger call build-backend --source=./backend

# Test the frontend
dagger call test-frontend --source=./frontend

# Build the frontend (returns the dist/ directory)
dagger call build-frontend --source=./frontend

A few things to notice about the CLI:

dagger call is the universal entry point for running any function. It works the same way whether you're calling a local module or a remote one (dagger call -m github.com/...).
Function names are converted from Python's snake_case to CLI kebab-case automatically (build_backend → build-backend).
--source is not a built-in Dagger flag. It's the parameter we defined with Annotated[dagger.Directory, Doc("...")]. Dagger turns every function parameter into a CLI flag, typed accordingly. A dagger.Directory parameter accepts a local path; a dagger.Secret would accept env:MY_SECRET or file:./key.json.
Return types determine what the CLI shows. test_backend returns a str, so Dagger prints the stdout. build_backend returns a Container, so Dagger shows the image digest. build_frontend returns a Directory, which you can export with dagger call build-frontend --source=./frontend export --path=./dist.

Same functions, same commands, for both apps. A developer debugging a build failure doesn't push a commit and wait. They run the same command on their laptop, inspect the output, and fix the issue with the same tools they use for application code. CI/CD is part of the software factory.

From Laptop to CI: The Same Commands in GitHub Actions

We've been running everything locally. Now let's prove the claim: the same code runs in CI without any changes. Create .github/workflows/ci.yml:

name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      - name: Test Backend
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: test-backend --source=./backend

      - name: Test Frontend
        uses: dagger/dagger-for-github@v8.4.1
        with:
          version: "0.20.3"
          verb: call
          args: test-frontend --source=./frontend

That's it. No Dockerfile to maintain. No shell scripts gluing steps together. No CI-specific logic. The dagger/dagger-for-github action installs the Dagger CLI and engine, and runs the exact same dagger call commands we just ran locally. The pipeline code doesn't know, or care, where it's running.

This is the minimal, working CI setup. Push it, open a pull request, and watch the same tests you just ran on your laptop execute in GitHub Actions. Same functions, same caching, same results.

Why Now

Three trends make this approach timely:

Platform Engineering is maturing. The core promise of Platform Engineering is developer self-service: give teams golden paths so they can ship without waiting for specialists. CI/CD modules are the missing piece in that story: reusable, versioned, self-documenting abstractions that reduce cognitive load the same way a Terraform module does. When CI/CD is software, it fits naturally into the Internal Developer Platform.

AI agents need programmatic interfaces. When your pipeline is real code with typed functions, AI agents can interact with it through tool use. YAML pipelines are opaque to agents. Typed Dagger functions are not. (We'll explore this in Part 4.)

Multi-cloud is the norm. A pipeline that deploys to GCP today can deploy to AWS tomorrow with a module swap. No vendor lock-in at the CI layer. The platform team maintains the modules; developers maintain their code.

What's Next

We now have a working CI pipeline, real code running both locally and in GitHub Actions with the same commands. But this minimal setup has limitations: every job starts with a cold cache, builds run on GitHub's shared 2-vCPU runners, and there's no persistent state between runs.

In Part 2, we'll tackle the infrastructure side: caching strategies that cut build times by 5x, managed runners with persistent NVMe storage, and self-hosted Kubernetes runners with shared Dagger engines. Three approaches, each with different tradeoffs between simplicity and control.

The pipeline code won't change. Only where it runs will.

Up Next: Part 2: Decoupling Pipelines from Infrastructure

This is Part 1 of a 4-part series. Follow for updates.

Tags: #cicd #dagger #platform-engineering #python #angular

DEV Community: Sami Chibani

CI/CD in the Era of AI and Platform Engineering: A Deep Dive into Dagger CI (Part 4)

Part 4: The AI-Native CI/CD Stack: Agents, Modules, and Spec-Driven Development

What Is a Dagger Agent?

The Problem With Setting Up CI

Generating the Setup With Daggie

Meet the Agents

The Generated Setup

Running the Checks

When a Check Fails

Integrating Into CI/CD

The Developer Experience Shift

Before: Pipeline Specialists

After: Agents Configure and Fix CI

From Developer Platform to Agent Factory

Spec-Driven Development With Speck

The Pipeline Pattern

Setting Up the Workflow

Running the Workflow

What Just Happened

Agents That Learn: Self-Improvement Across Runs

Three modes

Would I Recommend Dagger CI in Production Right Now?

What the Agent Layer Needs

What's Coming, and Why It Matters

Conclusion

Key Takeaways

CI/CD in the Era of AI and Platform Engineering: A Deep Dive into Dagger CI (Part 3)

Part 3: From Scripts to a Platform: Your CI/CD Module Library

The Growth Problem

The Daggerverse: Community Modules

Two-Layer Module Architecture

Module Development Best Practices

Documentation is your API

Single responsibility per module

Use field() for constructor dependencies

Return Dagger types, not strings

Use dagger.Secret for sensitive data

Private functions stay private

Pin your dependencies

Building the Private Module Layer

Repository Structure

acme-backend: Python Build & Test

acme-frontend: Angular Build & Test

acme-deploy: Compliant Deployment

What the private layer gives you

Consuming Modules in Your Pipelines

Project Structure

dagger.json: Dependencies

Pipeline Code

Try It Locally

GitHub Actions Workflow

The Local Developer Experience

Testing Modules

Versioning Strategy

Toolchains: Zero-Code Consumption

Official toolchains

Making modules toolchain-ready

The zero-code project setup

Customizing toolchain defaults

Full-stack toolchains

When to use toolchains vs. pipeline code

Deploying with acme-deploy as a toolchain

Implementing the deployment step in CI

Module Visibility with Dagger Cloud

Summary: The Module Architecture

What's Next

CI/CD in the Era of AI and Platform Engineering: A Deep Dive into Dagger CI (Part 2)

Part 2: Decoupling Pipelines from Infrastructure

The Runner Spectrum

Option 1: GitHub Actions with dagger-for-github

Understanding Dagger's Cache Architecture

Layer Cache: Content-Addressed Operation Caching

Cache Volumes: Persistent Package Manager Storage

How the Three Caches Stack

Persisting Cache on Ephemeral Runners

Why actions/cache Doesn't Work Here

The Solution: Docker Volume Caching

Dagger Cloud: Distributed Caching and Pipeline Observability

Why Dagger Cloud?

Use `field()` for constructor dependencies

Use `dagger.Secret` for sensitive data

`acme-backend`: Python Build & Test

`acme-frontend`: Angular Build & Test

`acme-deploy`: Compliant Deployment

`dagger.json`: Dependencies

Deploying with `acme-deploy` as a toolchain

Why `actions/cache` Doesn't Work Here