DEV Community

Fernando Paladini
Fernando Paladini

Posted on

Measure Your AI Coding Harness Maturity in CI with GitHub Actions

#github #ai #automation #devops

Two teams can use the same model and the same prompt — and get completely different results.

The difference is often not the model. It is the harness: the guides, rules, skills, hooks, tests, and CI feedback wrapped around the agent.

Most repositories build that harness slowly. Almost none of them measure it. Worse: a pull request can quietly delete hooks.json or drop a CI gate, and nobody notices until an agent does something expensive.

This tutorial shows how to put a deterministic maturity score into your GitHub Actions pipeline with Harness Score — so AI-assisted repos get the same kind of ratchet you already expect from tests and lint.

What “AI harness maturity” means

Harness Score scans filesystem evidence across tools like Cursor, Claude Code, Windsurf, Cline, Continue, Codex, and Copilot. It returns:

  • a maturity level from L0 (Unharnessed) to L4 (Self-correcting)
  • a score across six dimensions (up to 108 points)
  • a ranked list of what to fix next
Level Name Rough meaning
L0 Unharnessed Agents rediscover the project every session
L1 Documented Substantive context file exists
L2 Guided Scoped rules / skills + basic hygiene
L3 Sensing Tests, lint, types, and CI feedback exist
L4 Self-correcting Gate + feedback hooks close the loop

Important constraints (by design):

  • No LLM calls during the scan
  • No network during the scan
  • Same commit ⇒ same score

That is what makes the number safe to gate CI on.

Try it locally first:

npx harness-score@1.5.1
Enter fullscreen mode Exit fullscreen mode

Why put this in GitHub Actions

Local scans are useful. CI is the contract.

With the Action in your pipeline you can:

  1. Fail PRs that drop below a minimum maturity level
  2. Show a job summary with the dimension breakdown
  3. Post a sticky PR comment with score deltas vs the base branch
  4. Emit a README badge that updates when maturity changes

Official Action: Harness Score on the GitHub Marketplace

Tutorial: add Harness Score to CI

1) Create the workflow

Add .github/workflows/harness.yml:

name: Harness maturity

on:
  pull_request:
  push:
    branches: [main]

jobs:
  harness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: paladini/harness-score@v1
        with:
          min-level: '0'              # report only for now
          badge: 'harness-badge.svg'
          report: 'harness-report.md'
Enter fullscreen mode Exit fullscreen mode

Commit, push, and open the Actions tab. You should see a job summary like:

Harness Score: L2 · Guided (65% maturity)
Enter fullscreen mode Exit fullscreen mode

plus a per-dimension table.

Tip: for stronger supply-chain hygiene, pin the Action to a full commit SHA instead of @v1.

2) Start gating when you know your baseline

Once you know where the repo sits, raise the floor.

Example: require at least L3 · Sensing on main and PRs:

- uses: paladini/harness-score@v1
  with:
    min-level: '3'
    badge: 'harness-badge.svg'
Enter fullscreen mode Exit fullscreen mode

If maturity falls below L3, the job fails and prints the gaps needed to recover (for example: missing sensors or CI coverage).

A practical rollout:

  1. Week 1: min-level: '0' (observe)
  2. Week 2: set min-level to your current level (prevent regressions)
  3. Later: bump one level at a time as you add guides, sensors, and hooks

3) Add sticky PR comments (optional, highly recommended)

On pull_request events, the Action can compare the PR head against the base branch and update a single sticky comment:

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  harness:
    runs-on: ubuntu-latest
    concurrency:
      group: harness-score-${{ github.event.pull_request.number }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v4

      - uses: paladini/harness-score@v1
        with:
          min-level: '3'
          comment: 'true'
          badge: 'harness-badge.svg'
Enter fullscreen mode Exit fullscreen mode

You need pull-requests: write — the Action cannot grant that for you.

The comment shows level movement (for example L2 → L3), score deltas, and newly passing / newly failing checks. That makes harness regressions visible in review, not only in a red X on the Checks tab.

4) Publish the badge in your README

The Action can write harness-badge.svg on every run. A common pattern is committing it to a badges branch (or uploading it wherever you host static assets), then embedding:

<img alt="Harness Score" src="https://raw.githubusercontent.com/<you>/<repo>/badges/harness-badge.svg" height="20">
Enter fullscreen mode Exit fullscreen mode

You can also pin a static level badge from the docs site if you prefer not to auto-update.

More badge recipes: Show your maturity

Useful Action inputs

Input Default What it does
min-level 0 Fail when maturity is below this level (0–4)
badge harness-badge.svg SVG pill path (empty to skip)
report (empty) Write a Markdown report
comment false Sticky PR comment with score delta
working-directory . Subdirectory to scan (monorepos)
version pinned release harness-score npm version
gate maturity Gate on maturity (repo-only) or effective
include-user-harness false Include user-level harness paths in effective score
config .harness-score.json Custom config path

Outputs you can consume in later steps: level, level-name, percent.

Optional: repository config

If you need team policy (scopes, presets, per-check rules), add .harness-score.json at the repo root.

Example: keep CI gated on repository maturity, while still allowing local diagnosis with user-scope overlays:

{
  "scopes": {
    "user": true,
    "system": false
  },
  "gate": "maturity"
}
Enter fullscreen mode Exit fullscreen mode

Customization is transparent: excluded checks are removed from both earned and available points (no free credit), and security checks for exposed credentials cannot be disabled.

What this does not claim

A high score means the infrastructure for reliable agent work exists:

  • context and rules
  • skills/commands
  • hooks/guardrails
  • sensors (tests/lint/types)
  • CI feedback
  • hygiene/safety

It does not mean your tests are good, your rules are true, or every runtime path honors the committed harness. That honesty is intentional — deterministic scanners should not pretend to judge vibes.

Full starter workflow

Copy-paste version with gate + PR comment + badge + report:

name: Harness maturity

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  pull-requests: write

jobs:
  harness:
    runs-on: ubuntu-latest
    concurrency:
      group: harness-score-${{ github.event.pull_request.number || github.ref }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v4

      - name: Scan AI harness maturity
        id: harness
        uses: paladini/harness-score@v1
        with:
          min-level: '3'
          comment: 'true'
          badge: 'harness-badge.svg'
          report: 'harness-report.md'

      - name: Upload report artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: harness-score
          path: |
            harness-badge.svg
            harness-report.md
Enter fullscreen mode Exit fullscreen mode

Try it, then ratchet

  1. Run npx harness-score@1.5.1 locally
  2. Add the workflow with min-level: '0'
  3. Set min-level to your current level to prevent silent regressions
  4. Climb toward L3/L4 with the ranked fix list from the report

Links:

If you add this to a repo, comment with your starting level (L0–L4) and the first check that surprised you — those reports are the best way to harden the maturity model.

Top comments (0)