Abhiuday

Posted on Jun 14

Why I Built pipeline-compose: Ordered GitHub Actions Without YAML Hell

#githubactions #devops #typescript #architecture

You push v1.2.3 and expect a predictable sequence: tests pass, version resolves, GitHub Release publishes. In practice, most teams pick one of two painful options:

One giant workflow — every stage in a single YAML file. It works until you need reuse, different triggers per stage, or a workflow_call boundary.
workflow_run chains — workflow A triggers workflow B. Passing outputs between runs is awkward, and renaming a workflow breaks the chain silently.

I kept hitting both while building release automation for my own repos. The debugging pattern was always the same: click through six job logs to find which dependency edge failed, then grep generated YAML to see if it drifted from the source pipeline file.

That is why I built pipeline-compose — a way to declare stage order in one pipeline file, keep small focused workflow files, and orchestrate them with a single pipeline-compose-run step. v0.3.0 shipped this week; this post is the "why" behind it.

The problem: orchestration is not the same as workflow definition

GitHub Actions is good at running jobs inside a workflow. It is less ergonomic when your unit of reuse is an entire workflow file.

Consider a tag release flow:

ci → version sync → release publish

Each stage is legitimately its own workflow:

CI also runs on branch pushes
Version sync only makes sense on tags
Release publish needs contents: write and different inputs

Native needs: works when all jobs live in one file. The moment you split into separate workflows, you are choosing between:

Approach	What breaks
Mega-workflow	Stages cannot have independent triggers; file grows without bound
`workflow_run` chains	Output passing is indirect; renames fail quietly
Compile-to-YAML tools	Generated workflow becomes a second codebase to review

I tried the mega-workflow first. It shipped. Then every release was a small archaeology exercise — which job failed, which upstream output was empty, whether the if: on job 7 was wrong or job 3 never ran.

The insight: order and wiring are a separate concern from stage implementation.

Design: run vs compile

pipeline-compose splits the problem into two actions:

pipeline-compose-run (start here)

One step on your entry workflow reads .github/pipelines/pipeline.yml, dispatches each stage workflow via workflow_dispatch, waits for completion, collects outputs, and evaluates optional when: expressions.

No generated workflow to commit.

pipeline-compose-compile (optional)

If your org requires a static workflow file with native needs: for review or compliance, compile the pipeline into a generated workflow. Most repos do not need this — but some CI policies do.

Mode	Committed artifact	Best for
Run	Pipeline YAML only	Tag releases, multi-stage deploys, runtime `when:`
Compile	Pipeline YAML + generated workflow	Teams that require native `needs:` in review
Hand-written mega-workflow	One large YAML	Simple repos with 2–3 jobs total

I default to run. Compile stays an escape hatch.

Concrete example: tag release in three stages

The repo ships a copy-paste example: examples/run-tag-release.

On git push origin v*:

release.yml          ← one job, one action step
  └─ pipeline.yml    ← declares order + wiring
       ├─ ci.yml
       ├─ stage-version-sync.yml     → exports version
       └─ stage-release-publish.yml  ← receives version

Entry workflow

.github/workflows/release.yml:

name: Release
on:
  push:
    tags: ["v*"]

permissions:
  contents: write
  actions: write

jobs:
  run-pipeline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: aeswibon/pipeline-compose-run@v0.3.0
        with:
          pipeline_file: .github/pipelines/pipeline.yml
          github_token: ${{ github.token }}

actions: write is required — the run action dispatches stage workflows programmatically.

Pipeline file (order only)

.github/pipelines/pipeline.yml:

name: pipeline
version: 1
stages:
  - id: ci
    workflow: .github/workflows/ci.yml

  - id: version-sync
    workflow: .github/workflows/stage-version-sync.yml
    needs:
      - ci
    outputs:
      - version

  - id: release-publish
    workflow: .github/workflows/stage-release-publish.yml
    needs:
      - version-sync
    inputs:
      version: ${{ context.version-sync.version }}

This file is the source of truth for order. Stage implementations stay in normal workflow files you can run manually or reuse via workflow_call.

The ${{ context.version-sync.version }} syntax resolves at runtime from the completed version-sync stage.

The output-passing detail that actually matters

GitHub's API does not return job outputs for workflow_dispatch runs the way it does for jobs in a single workflow. pipeline-compose collects stage outputs from an artifact instead:

Artifact name: pipeline-compose-<stage-id>
File: outputs.json with your output keys

For version sync:

- name: Export outputs for pipeline-compose
  if: success()
  env:
    VERSION: ${{ steps.version.outputs.value }}
  run: |
    mkdir -p pipeline-compose
    jq -n --arg version "$VERSION" '{version: $version}' > pipeline-compose/outputs.json

- uses: actions/upload-artifact@v4
  if: success()
  with:
    name: pipeline-compose-version-sync
    path: pipeline-compose/outputs.json
    retention-days: 1

The artifact name matches the stage id: pipeline-compose-version-sync. Downstream stages receive values as workflow_dispatch inputs — wired in the pipeline file, not copy-pasted between YAML files.

This is the mechanism that makes multi-workfile orchestration tractable. Without it, you are back to stringly-typed env vars in workflow_run payloads.

Why a TypeScript monorepo for CI tooling

pipeline-compose v0.3.0 is a pnpm workspace: shared @aeswibon/pipeline-compose-core, four action packages (run, compile, eval, context-merge), Vitest tests, and a single pnpm run publish:actions path to ship all four Marketplace actions from one repo.

That choice is pragmatic, not ideological:

GitHub Actions composite and JavaScript actions already run on Node
One core library, four thin action wrappers — edit orchestration logic once
Vitest + schema validation (pipeline-v1.schema.json) catch pipeline mistakes before they hit CI
Bundling produces the single-file dist each action repo publishes

My other backend work is Go and Java. For Actions orchestration tooling, shipping a typed TypeScript core with a schema was faster than fighting JVM startup or cross-compiling Go inside action bundles. Different layer, different tradeoff.

What v0.3.0 changed

The recent release refactored the development surface:

Monorepo with pnpm workspaces instead of scattered packages
Schema moved to packages/core/schema/pipeline-v1.schema.json
CI publishes all four action repos from a single publish-actions stage
Rebuilt run/compile/eval bundles from shared core v0.3.0

If you used an earlier version, the pipeline file format is stable (version: 1); the action tags are what you pin in release.yml.

Full changelog: v0.3.0 release notes

When pipeline-compose is the wrong tool

Be honest about limits:

Simple repos with 2–3 jobs in one workflow — native Actions is fine; do not add orchestration
Teams that forbid programmatic workflow dispatch — run mode will not pass security review
Heavy cross-run state — you still need artifacts or a external store; this is not a database

If you have six stages across four workflow files and tag-triggered releases, that is the sweet spot.