DEV Community: Abhiuday

Why I Built pipeline-compose: Ordered GitHub Actions Without YAML Hell

Abhiuday — Sun, 14 Jun 2026 10:25:25 +0000

You push v1.2.3 and expect a predictable sequence: tests pass, version resolves, GitHub Release publishes. In practice, most teams pick one of two painful options:

One giant workflow — every stage in a single YAML file. It works until you need reuse, different triggers per stage, or a workflow_call boundary.
workflow_run chains — workflow A triggers workflow B. Passing outputs between runs is awkward, and renaming a workflow breaks the chain silently.

I kept hitting both while building release automation for my own repos. The debugging pattern was always the same: click through six job logs to find which dependency edge failed, then grep generated YAML to see if it drifted from the source pipeline file.

That is why I built pipeline-compose — a way to declare stage order in one pipeline file, keep small focused workflow files, and orchestrate them with a single pipeline-compose-run step. v0.3.0 shipped this week; this post is the "why" behind it.

The problem: orchestration is not the same as workflow definition

GitHub Actions is good at running jobs inside a workflow. It is less ergonomic when your unit of reuse is an entire workflow file.

Consider a tag release flow:

ci → version sync → release publish

Each stage is legitimately its own workflow:

CI also runs on branch pushes
Version sync only makes sense on tags
Release publish needs contents: write and different inputs

Native needs: works when all jobs live in one file. The moment you split into separate workflows, you are choosing between:

Approach	What breaks
Mega-workflow	Stages cannot have independent triggers; file grows without bound
`workflow_run` chains	Output passing is indirect; renames fail quietly
Compile-to-YAML tools	Generated workflow becomes a second codebase to review

I tried the mega-workflow first. It shipped. Then every release was a small archaeology exercise — which job failed, which upstream output was empty, whether the if: on job 7 was wrong or job 3 never ran.

The insight: order and wiring are a separate concern from stage implementation.

Design: run vs compile

pipeline-compose splits the problem into two actions:

pipeline-compose-run (start here)

One step on your entry workflow reads .github/pipelines/pipeline.yml, dispatches each stage workflow via workflow_dispatch, waits for completion, collects outputs, and evaluates optional when: expressions.

No generated workflow to commit.

pipeline-compose-compile (optional)

If your org requires a static workflow file with native needs: for review or compliance, compile the pipeline into a generated workflow. Most repos do not need this — but some CI policies do.

Mode	Committed artifact	Best for
Run	Pipeline YAML only	Tag releases, multi-stage deploys, runtime `when:`
Compile	Pipeline YAML + generated workflow	Teams that require native `needs:` in review
Hand-written mega-workflow	One large YAML	Simple repos with 2–3 jobs total

I default to run. Compile stays an escape hatch.

Concrete example: tag release in three stages

The repo ships a copy-paste example: examples/run-tag-release.

On git push origin v*:

release.yml          ← one job, one action step
  └─ pipeline.yml    ← declares order + wiring
       ├─ ci.yml
       ├─ stage-version-sync.yml     → exports version
       └─ stage-release-publish.yml  ← receives version

Entry workflow

.github/workflows/release.yml:

name: Release
on:
  push:
    tags: ["v*"]

permissions:
  contents: write
  actions: write

jobs:
  run-pipeline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: aeswibon/pipeline-compose-run@v0.3.0
        with:
          pipeline_file: .github/pipelines/pipeline.yml
          github_token: ${{ github.token }}

actions: write is required — the run action dispatches stage workflows programmatically.

Pipeline file (order only)

.github/pipelines/pipeline.yml:

name: pipeline
version: 1
stages:
  - id: ci
    workflow: .github/workflows/ci.yml

  - id: version-sync
    workflow: .github/workflows/stage-version-sync.yml
    needs:
      - ci
    outputs:
      - version

  - id: release-publish
    workflow: .github/workflows/stage-release-publish.yml
    needs:
      - version-sync
    inputs:
      version: ${{ context.version-sync.version }}

This file is the source of truth for order. Stage implementations stay in normal workflow files you can run manually or reuse via workflow_call.

The ${{ context.version-sync.version }} syntax resolves at runtime from the completed version-sync stage.

The output-passing detail that actually matters

GitHub's API does not return job outputs for workflow_dispatch runs the way it does for jobs in a single workflow. pipeline-compose collects stage outputs from an artifact instead:

Artifact name: pipeline-compose-<stage-id>
File: outputs.json with your output keys

For version sync:

- name: Export outputs for pipeline-compose
  if: success()
  env:
    VERSION: ${{ steps.version.outputs.value }}
  run: |
    mkdir -p pipeline-compose
    jq -n --arg version "$VERSION" '{version: $version}' > pipeline-compose/outputs.json

- uses: actions/upload-artifact@v4
  if: success()
  with:
    name: pipeline-compose-version-sync
    path: pipeline-compose/outputs.json
    retention-days: 1

The artifact name matches the stage id: pipeline-compose-version-sync. Downstream stages receive values as workflow_dispatch inputs — wired in the pipeline file, not copy-pasted between YAML files.

This is the mechanism that makes multi-workfile orchestration tractable. Without it, you are back to stringly-typed env vars in workflow_run payloads.

Why a TypeScript monorepo for CI tooling

pipeline-compose v0.3.0 is a pnpm workspace: shared @aeswibon/pipeline-compose-core, four action packages (run, compile, eval, context-merge), Vitest tests, and a single pnpm run publish:actions path to ship all four Marketplace actions from one repo.

That choice is pragmatic, not ideological:

GitHub Actions composite and JavaScript actions already run on Node
One core library, four thin action wrappers — edit orchestration logic once
Vitest + schema validation (pipeline-v1.schema.json) catch pipeline mistakes before they hit CI
Bundling produces the single-file dist each action repo publishes

My other backend work is Go and Java. For Actions orchestration tooling, shipping a typed TypeScript core with a schema was faster than fighting JVM startup or cross-compiling Go inside action bundles. Different layer, different tradeoff.

What v0.3.0 changed

The recent release refactored the development surface:

Monorepo with pnpm workspaces instead of scattered packages
Schema moved to packages/core/schema/pipeline-v1.schema.json
CI publishes all four action repos from a single publish-actions stage
Rebuilt run/compile/eval bundles from shared core v0.3.0

If you used an earlier version, the pipeline file format is stable (version: 1); the action tags are what you pin in release.yml.

Full changelog: v0.3.0 release notes

When pipeline-compose is the wrong tool

Be honest about limits:

Simple repos with 2–3 jobs in one workflow — native Actions is fine; do not add orchestration
Teams that forbid programmatic workflow dispatch — run mode will not pass security review
Heavy cross-run state — you still need artifacts or a external store; this is not a database

If you have six stages across four workflow files and tag-triggered releases, that is the sweet spot.

Try it

Copy examples/run-tag-release/.github into your repo
Replace ci.yml test steps with your commands
Push a tag: git tag v0.0.1 && git push origin v0.0.1
Watch Actions → Release — stages run in pipeline order

Marketplace action: pipeline-compose-run

Longer tutorial in the repo: docs/tutorials/tag-release-pipeline.md

Feedback welcome

I opened an RFC on the repo for stage output passing and failure policy: https://github.com/aeswibon/pipeline-compose/discussions/1

Tag release pipelines without a 400-line GitHub Actions workflow

Abhiuday — Sun, 14 Jun 2026 06:20:35 +0000

You push v1.2.3 and expect a predictable sequence: tests pass → version is resolved → GitHub Release is created. In practice, teams usually pick one of two painful options:

One giant workflow — every stage in a single YAML file. It works until you need reuse, workflow_call, or different triggers per stage.
workflow_run chains — workflow A triggers workflow B. Passing outputs between runs is awkward, and renaming a workflow breaks the chain silently.

There is a middle path: keep small, focused stage workflows (the ones you already have), declare order and wiring in one pipeline file, and use a single orchestrator step on tag push.

This tutorial uses pipeline-compose-run — available on the GitHub Marketplace — and a copy-paste example you can drop into any repo.

Full example (copy .github/): examples/run-tag-release

What we are building

On git push origin v*:

release.yml          ← one job, one action step
  └─ pipeline.yml    ← declares order + wiring
       ├─ ci.yml
       ├─ stage-version-sync.yml     → exports version
       └─ stage-release-publish.yml  ← receives version

No generated workflow to commit. No manual workflow_run graph.

Step 1 — Entry workflow

Create .github/workflows/release.yml:

name: Release
on:
  push:
    tags: ["v*"]

permissions:
  contents: write
  actions: write

jobs:
  run-pipeline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: aeswibon/pipeline-compose-run@v0.3.0
        with:
          pipeline_file: .github/pipelines/pipeline.yml
          github_token: ${{ github.token }}

The actions: write permission is required because the action dispatches your stage workflows via workflow_dispatch.

Step 2 — Pipeline file (order only)

Create .github/pipelines/pipeline.yml:

name: pipeline
version: 1
stages:
  - id: ci
    workflow: .github/workflows/ci.yml

  - id: version-sync
    workflow: .github/workflows/stage-version-sync.yml
    needs:
      - ci
    outputs:
      - version

  - id: release-publish
    workflow: .github/workflows/stage-release-publish.yml
    needs:
      - version-sync
    inputs:
      version: ${{ context.version-sync.version }}

This file is the source of truth for order. Stage implementations stay in normal workflow files you can also run manually or reuse via workflow_call.

The ${{ context.version-sync.version }} syntax resolves at runtime from the completed version-sync stage.

Step 3 — Stage workflows

Each stage must include workflow_dispatch. Downstream stages that receive values need matching workflow_dispatch inputs.

CI (`.github/workflows/ci.yml`)

name: CI
on:
  workflow_dispatch:
  push:
    branches: [master]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - run: echo "Replace with your test/lint commands"

Version sync — export outputs

GitHub’s API does not return job outputs for workflow_dispatch runs. pipeline-compose collects stage outputs from an artifact:

Artifact name: pipeline-compose-<stage-id>
File: outputs.json with your output keys

name: Version sync
on:
  workflow_dispatch:

jobs:
  version-sync:
    runs-on: ubuntu-latest
    steps:
      - name: Resolve semver from ref
        id: version
        run: |
          ref="${GITHUB_REF}"
          if [[ "$ref" =~ ^refs/tags/v(.+)$ ]]; then
            echo "value=${BASH_REMATCH[1]}" >> "$GITHUB_OUTPUT"
          else
            echo "Expected a version tag ref, got: $ref" >&2
            exit 1
          fi

      - name: Export outputs for pipeline-compose
        if: success()
        env:
          VERSION: ${{ steps.version.outputs.value }}
        run: |
          mkdir -p pipeline-compose
          jq -n --arg version "$VERSION" '{version: $version}' > pipeline-compose/outputs.json

      - uses: actions/upload-artifact@v4
        if: success()
        with:
          name: pipeline-compose-version-sync
          path: pipeline-compose/outputs.json
          retention-days: 1

Note the artifact name matches the stage id: pipeline-compose-version-sync.

Release publish — consume version input

name: Release publish
on:
  workflow_dispatch:
    inputs:
      version:
        description: Semver without v prefix
        type: string
        required: true

permissions:
  contents: write

jobs:
  create-release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 0

      - name: Create release
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          VERSION: ${{ inputs.version }}
        run: |
          tag="v${VERSION}"
          if gh release view "$tag" >/dev/null 2>&1; then
            echo "Release $tag already exists"
            exit 0
          fi
          gh release create "$tag" --title "$tag" --generate-notes

Step 4 — Ship it

git tag v1.0.0 && git push origin v1.0.0

Open Actions → Release in your repo. Stages run in pipeline order; publish receives the version from sync.

Why not other approaches?

Approach	Pain point
Monolithic workflow	Hard to reuse stages; noisy diffs
`workflow_run` chains	Fragile; outputs don’t flow cleanly
Generated workflow (compile)	Works, but you commit generated YAML
pipeline-compose-run	Ordered dispatch + context; one pipeline file

If you prefer committing a static graph with native GitHub needs:, see pipeline-compose-compile instead.

Troubleshooting

Symptom	Fix
`403` on dispatch	Add `actions: write` on the entry job
Publish stage missing version	Artifact must be `pipeline-compose-version-sync` with `outputs.json`
Stage never runs	Add `workflow_dispatch` to that workflow
Wrong ref in stage	Pass `ref:` to the run action if needed

Try it now

Marketplace: pipeline-compose-run
Copy-paste example: examples/run-tag-release
More examples: pipeline-compose/examples

Part of pipeline-compose. MIT License.

Observation Haki for Manga: Why I Built a Change Data Capture (CDC) Pipeline Just to Read Manga

Abhiuday — Thu, 11 Jun 2026 16:52:40 +0000

Observation Haki for Manga: Why I Built a Change Data Capture (CDC) Pipeline Just to Read Manga

If you are a manga reader, you know the pain.

New chapters are scattered across a dozen scanlation sites (MangaDex, MangaFire, MangaPlus, Asura Scans). Each site has a different layout, different APIs (or none at all), and zero unified way to track updates. Manually checking six websites daily is tedious and slow.

In One Piece, Observation Haki (Kenbunshoku Haki) allows a warrior to sense the presence, strength, and movements of others before they can act.

I wanted that power for my manga reading list. So, I did what any software engineer would do: I over-engineered a real-time Change Data Capture (CDC) pipeline using Go, PostgreSQL, Redpanda, and Spring Boot.

Here is how I built it—and why this hybrid stack is the ultimate pattern for building robust, self-hosted web trackers.

The Grand Line: System Architecture

Rather than writing a monolithic scraper that directly makes API requests and spams webhooks, I designed a decoupled, event-driven architecture:

Let's break down the main components and how they map to our engineering (and anime) concepts.

1. The Scraper: Shadow Clone Jutsu (Go + Colly)

To scrape multiple sites quickly, the scraper must be fast, lightweight, and concurrent. Go is the perfect fit.

I built a concurrent runner that spins up separate workers (Goroutines) for each scanlation platform. Think of them as Naruto’s Shadow Clones (Kage Bunshin no Jutsu). They spread out across the web, gather pages using the Colly scraping framework, extract metadata, and send it back to the main thread.

Here’s a snippet of how the scraper fetches chapters using HTML selectors:

// From scraper/internal/adapter/asurascans.go
c.OnHTML("a[href*=\"/chapter/\"]", func(e *colly.HTMLElement) {
    href := e.Attr("href")
    if href == "" {
        return
    }

    parts := strings.Split(href, "/chapter/")
    if len(parts) != 2 {
        return
    }

    chapterNum, err := strconv.ParseFloat(strings.TrimRight(parts[1], "/"), 64)
    if err != nil {
        return
    }

    chapters = append(chapters, model.Chapter{
        Number: chapterNum,
        URL:    asurascansBase + href,
        IsNew:  true,
    })
})

2. The Change Engine: Observation Haki (PostgreSQL Diff)

Once the raw data is fetched, the system needs to determine what has actually changed. We don't want to receive duplicate alerts. This is where Observation Haki (our Diff Engine) comes in.

The Go scraper runs a Postgres transactional upsert:

It upserts the manga_series table (updating headers, cover image, status).
It attempts to insert the chapters to the chapters table.
The database schema has a unique constraint: UNIQUE(series_id, chapter_num).

Using ON CONFLICT DO NOTHING, if a chapter already exists in our database, Postgres ignores it. If it is new, Postgres saves it and returns a new UUID. The Go code detects this database-generated ID and flags the chapter as a fresh release:

// From scraper/internal/db/postgres.go
func (d *DB) InsertChapter(ctx context.Context, seriesID string, ch model.Chapter) (string, error) {
    var id string
    err := d.pool.QueryRow(ctx, `
        INSERT INTO chapters (series_id, chapter_num, title, url, release_date, is_new)
        VALUES ($1, $2, $3, $4, $5, true)
        ON CONFLICT (series_id, chapter_num) DO NOTHING
        RETURNING id
    `, seriesID, ch.Number, ch.Title, ch.URL, ch.ReleaseDate).Scan(&id)
    // ...
}

3. The Eventing Layer: Domain Expansion (Redpanda / Kafka)

Directly invoking notifications inside the scraper is a classic anti-pattern. If your Discord webhook rate-limits you, or your service goes offline, you lose the event.

To solve this, I activated a Domain Expansion: a pocket message broker (using Redpanda locally and Aiven Kafka in production) to act as a buffer.

Instead of deploying a full, heavy Debezium Connector, the Go scraper serializes database changes into a Debezium-compatible JSON payload directly:

{
  "op": "c",
  "after": {
    "id": "7ca648b2-5f65-4d2c-8067-27083042a3cf",
    "series_id": "bfd0d829-114c-47bc-ad6c-d2c67f407784",
    "chapter_num": "1117.00",
    "url": "https://mangaplus.shueisha.co.jp/viewer/1021287",
    "is_new": true
  }
}

The scraper pushes this event to the message topic, ensuring at-least-once delivery.

4. The Notifier: Hell Butterflies (Spring Boot)

In Bleach, Shinigami use Hell Butterflies (Jigokuchō) to safely guide messages between worlds.

In our pipeline, the Spring Boot application consumes events from the Redpanda stream and dispatches them as Discord, Slack, or Telegram webhook payloads.

Spring Boot is the perfect choice for the consumer layer due to its rich ecosystem of integration libraries and battle-tested thread pool listeners.

// From notification-service/.../service/ChapterEventService.java
public void processChapterEvent(String message) {
    JsonNode root = mapper.readTree(message);
    String op = root.path("op").asText();
    if (!"c".equals(op)) return; // Only notify on 'create' updates

    JsonNode after = root.path("after");
    String chapterId = after.path("id").asText();
    String chapterNum = after.path("chapter_num").asText();
    String url = after.path("url").asText();

    // Route to active webhook endpoints
    Map<String, Boolean> results = notifierRegistry.sendAll(seriesTitle, chapterNum, title, url);

    // Mark chapter as notified so it won't trigger alerts again
    chapterRepo.markNotified(chapterId);
}

Here is how the notification alerts look in action:

Developer Experience (DX) First

A major problem with multi-service stacks is setup complexity. To fix this, I created an interactive setup CLI wizard written in Go. Running go run ./configure automatically guides you through the process:

With a single docker compose up -d, you can have a running Kafka, Prometheus monitoring dashboard, PostgreSQL cluster, and the scraper active in seconds.

Conclusion & Code

Building manga-cdc proved that enterprise concepts like CDC, decoupled message brokers, and hybrid-language microservices aren't just for scaling huge web companies. They are powerful tools for building side-projects that are resilient, modular, and extremely fun to build in public.

You can inspect the complete source code, set up the wizard, and run it yourself on GitHub:

👉 GitHub Repository: aeswibon/manga-cdc

Let me know in the comments: What is the most over-engineered side project you have built to solve a daily minor annoyance?

DEV Community: Abhiuday

Why I Built pipeline-compose: Ordered GitHub Actions Without YAML Hell

The problem: orchestration is not the same as workflow definition

Design: run vs compile

pipeline-compose-run (start here)

pipeline-compose-compile (optional)

Concrete example: tag release in three stages

Entry workflow

Pipeline file (order only)

The output-passing detail that actually matters

Why a TypeScript monorepo for CI tooling

What v0.3.0 changed

When pipeline-compose is the wrong tool

Try it

Feedback welcome

Tag release pipelines without a 400-line GitHub Actions workflow

What we are building

Step 1 — Entry workflow

Step 2 — Pipeline file (order only)

Step 3 — Stage workflows

CI (.github/workflows/ci.yml)

Version sync — export outputs

Release publish — consume version input

Step 4 — Ship it

Why not other approaches?

Troubleshooting

Try it now

Observation Haki for Manga: Why I Built a Change Data Capture (CDC) Pipeline Just to Read Manga

Observation Haki for Manga: Why I Built a Change Data Capture (CDC) Pipeline Just to Read Manga

The Grand Line: System Architecture

1. The Scraper: Shadow Clone Jutsu (Go + Colly)

2. The Change Engine: Observation Haki (PostgreSQL Diff)

3. The Eventing Layer: Domain Expansion (Redpanda / Kafka)

4. The Notifier: Hell Butterflies (Spring Boot)

Developer Experience (DX) First

Conclusion & Code

CI (`.github/workflows/ci.yml`)