I built siftwise: run only the tests your change needs (open source, self-hosted)

Pramod Basavaraj Menasi — Wed, 21 Oct 2026 07:25:08 +0000

Your coding agent just changed one file. CI is now running all 18,000 tests. Again.

Coding agents now write code faster than CI can test it. Every pull request, whether a person or an agent wrote it, still runs the entire test suite. Anthropic reported a 25x increase in CI jobs over six months as agents sped up writing code (Agentic coding is straining CI), and Linear wrote that AI coding has made CI a bottleneck.

That got me thinking: what if CI only ran the tests a change could actually break?

So over the last six weeks I built siftwise, an open-source, self-hosted tool that does exactly that. v0.1.0 is out on PyPI and GHCR.

pip install siftwise

This post is the whole guide: what it does, how to run it on your laptop, how to connect your own repo, your coding agent, and your CI. By the end you'll have everything you need.

The idea in one sentence

siftwise looks at your diff, works out which tests could be affected, runs only those, and falls back to the full suite whenever it isn't sure, because missing a real failure is worse than running a few extra tests.

Test impact analysis isn't a new idea; there are good commercial tools for it. What I wanted was something open source, self-hosted, and built for agent-driven CI.

Why you might care

⚡ Faster feedback: a one-file change can run just the tests that touch that file, not the whole suite.
🛡️ Safe by default: unsure → full suite. Server down → full suite. It never blocks CI (unless you opt into fail-on).
🔍 Every test has a reason: no black box. You see why each test was picked.
🧪 "Is this my fault?": every failure gets a verdict: new regression, already broken on main, flaky, or unknown.
🤖 Agent-ready: an MCP server, so a coding agent can pick tests and triage failures itself.
🏠 Your data stays yours: runs on your own machine or server, on your own Postgres. No SaaS, no account, no test instrumentation.
📏 Proves itself: shadow mode runs the full suite on a sample of PRs and tells you what selection would have missed.

How it works

select: diffs your branch, builds a Python or Go import graph locally, and asks the server which tests the change can affect.
run: you run only those tests (or the full suite if siftwise isn't confident).
record: JUnit results are uploaded to the server.
rollup: per-test history is updated (failures, durations, flakiness, broken-on-main).
triage: every failure gets a verdict.
shadow compare (optional): on sampled runs, the full suite runs anyway and siftwise checks what it would have missed.
In CI, a sticky PR comment shows it all.

Selection signals

Path mapping (test_cart.py ↔ cart.py)
Import graphs for Python and Go (go list -deps -test)
Failure history: tests that failed when these files changed before
Declared rules in .siftwise.toml, for dependencies no import graph can see

Every selected test comes with a reason, like imports changed module rdflib/term.py.

Supported languages

Language	How tests are selected	Command siftwise gives you
Python (pytest)	`foo.py` → `test_foo.py`; test files importing a changed module (incl. parent packages, conftests); changed test files	`pytest` node IDs or test files
Go	the changed package and every package whose tests import it	`go test ./pkg`, or `go test -run` with the selected test names
JS/TS (Jest)	`foo.ts` → `foo.test.`, `foo.spec.`, `__tests__/foo.*` (needs file paths in the JUnit)	`jest <files>`
Java (Maven/Gradle)	recorded with full history (for triage); selection always runs the full suite (for now)	none: you run your usual `mvn test` or `./gradlew test`

Safe fallbacks

Unknown diff, a changed build/CI/dependency file, no history yet, or a changed file that no signal covers: full suite. If the server is down, CI just runs everything.

Failure triage

new_regression  high   tests.test_cart::test_total
    tests/test_cart.py tests changed src/cart.py
pre_existing    high   tests.test_db::test_query
    failing on main on ubuntu-py3.12 since 4f2a9c1 (3 main runs)
flaky           medium tests.test_net::test_fetch
    flaky 1 time in 90 days (1 unrelated fail-then-pass)
siftwise: 3 failing test(s): 1 new regression, 1 pre existing, 1 flaky, 0 unknown

Nobody wastes an hour debugging a test that was already broken on main.

Before you start

You need:

Docker (with Docker Compose) for the server
git
uv (or pip / pipx) for the CLI; Python 3.11+
Go users: go on your PATH, plus gotestsum for JUnit output (go install gotest.tools/gotestsum@latest)

All commands are bash. On Windows, use Git Bash or WSL.

Step 1: Try it in 5 minutes (quickstart)

Start the server (Postgres included):

git clone https://github.com/PramodBasavarajMenasi/siftwise.git
cd siftwise
[ -f .env ] || echo "SIFTWISE_API_TOKEN=$(openssl rand -hex 32)" > .env
docker compose up -d --wait             # pulls the image, waits until healthy
curl -fsS http://localhost:8000/healthz # {"status":"ok"}

The token in .env is a shared secret you choose; the server and every client use it.

Install the CLI and give it the token:

uv tool install siftwise                # or: pip install siftwise
export SIFTWISE_API_TOKEN=$(sed -n 's/^SIFTWISE_API_TOKEN=//p' .env)
siftwise --version                      # 0.1.0

siftwise: command not found? Run uv tool update-shell once and open a new terminal.

Record a run, change a file, and select:

demo="$(mktemp -d)/shop" && cp -r examples/quickstart "$demo" && cd "$demo"
git init -q && git add -A && git commit -qm "A shop"
uv run pytest -q --junitxml=junit.xml
siftwise record junit.xml --repo demo/shop --branch main --main

echo "# reviewed the bulk discount" >> shop/cart.py && git commit -qam "Touch the cart"
siftwise select --repo demo/shop --base HEAD~1 > selected.sh
cat selected.sh                         # pytest tests/test_cart.py
uv run sh selected.sh

siftwise: python imports: 1 test file(s) import changed modules
siftwise: selective (2 of 4 known tests): 2 of 4 known tests selected

You changed cart.py, and siftwise ran only the tests that import it. 🎉

Step 2: Use it on your own project (locally)

Keep the server running. Now point siftwise at your repo.

2.1 Connect the CLI to the server. Two environment variables are the whole connection. First print your token (in the siftwise folder):

sed -n 's/^SIFTWISE_API_TOKEN=//p' .env

Then, in your project:

cd ~/code/my-app
export SIFTWISE_SERVER=http://localhost:8000     # the default; change it for a remote server
export SIFTWISE_API_TOKEN=<paste the token>

Put both lines in your ~/.bashrc (or ~/.zshrc) so you don't have to repeat them. Never commit the token.

2.2 Record a first full run on main. This gives siftwise history to work with. Pick a repo name (owner/name) and use the same one in every command.

git checkout main

# Python
pytest --junitxml=junit.xml

# Go
gotestsum --junitfile junit.xml ./...

# Jest: use a JUnit reporter that writes file paths (e.g. jest-junit with addFileAttribute="true")

siftwise record junit.xml --repo me/my-app --branch main --main

Want history from the start? If your GitHub Actions runs already upload JUnit XML as artifacts, scripts/backfill.py in the siftwise repo can import past runs (see the README's "Seed history from past CI runs").

2.3 Make a change and select.

git checkout -b fix-discount
# ...edit src/my_app/pricing.py...
git commit -am "Fix discount rounding"

siftwise select --repo me/my-app --base main > selected.sh
cat selected.sh            # only the tests your change can affect
sh selected.sh             # in a uv project: uv run sh selected.sh

--json prints every selected test with the reason it was picked.
An empty selected.sh means no tests are affected.
If you touched something siftwise can't reason about (say pyproject.toml or a CI file), selected.sh runs the full suite. That's by design.
Exit code 3 means "full suite needed, but I don't know your command": just run your normal test command.

2.4 A test failed? Ask whether it's your fault.

Run the failing tests with a JUnit report, then triage it:

pytest tests/test_pricing.py --junitxml=junit.xml    # the tests from selected.sh
siftwise triage junit.xml --repo me/my-app --base main

Add --json for the full evidence.

2.5 Keep feeding it. Record your branch runs too (with --base, so triage can link later failures to this change):

siftwise record junit.xml --repo me/my-app --branch fix-discount --base main

The more runs it has, the better its history: flakiness, "already broken on main", and co-failures all come from recorded runs. In practice, CI does this for you (Step 4).

2.6 Add rules for what imports can't see. Drop a .siftwise.toml in your repo root:

always_run = ["tests/smoke/**", "tests.test_health::*"]  # test file paths or test IDs
generated = ["build/reports/**"]          # files your test runs write (never "changed code")
known_flaky = ["tests/test_net.py"]       # triage: unlinked failures are flaky

[[depends]]                               # dependencies no import graph can see
tests = "test/cli/**"                     # test file paths, or <Go package dir>/<TestName>
on = ["**/*.go", "!**/*_test.go"]         # ! excludes

Globs: ** spans directories; * and ? stay within one folder. Go test IDs contain the import path, so match them with **::TestName.

2.7 Stop the server when you're done: docker compose down in the siftwise folder (down -v also deletes its data).

Step 3: Connect your coding agent (MCP)

siftwise mcp is a standard stdio MCP server with three tools:

Tool	The agent calls it	Returns
`select_tests`	before running tests after a change	the mode, the command to run, a reason per test
`triage_failure`	when tests fail, before fixing them	a verdict per failure, with evidence
`explain`	when a selection or verdict surprises it	one test's recent history and why it would (not) be selected

It includes uncommitted work (the diff runs to your working tree, untracked files too), and it fails open: if the server is down, it tells the agent to run the full suite.

3.1 Install:

uv tool install 'siftwise[mcp]'
uv tool update-shell        # once, if ~/.local/bin isn't on your PATH; then restart the agent
uv tool dir --bin           # prints the folder, if you need the full path below

3.2 Add it to your agent. Open the agent inside your repository: siftwise reads the repo name from your origin remote and the base branch from origin/HEAD (else main). Override with SIFTWISE_REPO / SIFTWISE_BASE_REF if they don't match the name you recorded with.

Claude Code (tested ✅):

claude mcp add siftwise --env SIFTWISE_SERVER=http://localhost:8000 \
    --env SIFTWISE_API_TOKEN=your-token -- siftwise mcp

or a project .mcp.json, which reads the token from your environment so it never lands in git:

{
  "mcpServers": {
    "siftwise": {
      "command": "siftwise",
      "args": ["mcp"],
      "env": {
        "SIFTWISE_SERVER": "http://localhost:8000",
        "SIFTWISE_API_TOKEN": "${SIFTWISE_API_TOKEN}"
      }
    }
  }
}

Codex (~/.codex/config.toml):

[mcp_servers.siftwise]
command = "siftwise"
args = ["mcp"]
env = { SIFTWISE_SERVER = "http://localhost:8000", SIFTWISE_API_TOKEN = "your-token" }

Cursor (~/.cursor/mcp.json for every project, or .cursor/mcp.json in one): the same JSON as above, with the token value written in.

Honest note: siftwise is a standard stdio MCP server, so any MCP client should be able to run it. So far I've tested it with Claude Code. The Codex and Cursor configs follow each tool's MCP format but haven't been tested yet. If you try them, please tell me (or open an issue)!

Agent can't find siftwise? (common on Windows: the agent doesn't see your shell's PATH) Use the full path from uv tool dir --bin as the command. In JSON, double the backslashes: "C:\\Users\\you\\.local\\bin\\siftwise.exe". In TOML, use a single-quoted string.

3.3 Tell the agent when to use it, in CLAUDE.md or AGENTS.md:

Before running tests, call the siftwise `select_tests` tool and run the command it returns.
When tests fail, call `triage_failure` before fixing anything: fix `new_regression` failures,
re-run `flaky` ones, and leave `pre_existing` ones (already failing on main).

What that looks like in practice (an example flow):

You: "Fix the discount rounding bug."
The agent edits pricing.py, calls select_tests → gets pytest tests/test_pricing.py tests/test_cart.py instead of the whole suite.
A test fails → it calls triage_failure → new_regression, linked to the file it just changed → it fixes its own bug.
A different failure is pre_existing (already broken on main) → the agent leaves it alone instead of "fixing" something unrelated.

Less waiting, fewer wrong fixes, and a reason for every decision.

The MCP server only reads. History comes from runs you (or CI) record, so it gets smarter once Step 4 is in place.

Step 4: Connect it to your CI (GitHub Actions)

4.1 Host the server where CI can reach it. GitHub's runners can't see your localhost. The quickstart compose file deliberately binds the API to 127.0.0.1, so on a server you put a TLS reverse proxy in front (the token travels in a request header).

On any Linux VM with Docker:

git clone https://github.com/PramodBasavarajMenasi/siftwise.git && cd siftwise
echo "SIFTWISE_API_TOKEN=$(openssl rand -hex 32)" > .env
docker compose up -d --wait

Point a DNS record (e.g. siftwise.your-domain.com) at the VM, open ports 80 and 443, install Caddy, and use this Caddyfile (Caddy gets the HTTPS certificate for you):

siftwise.your-domain.com {
    reverse_proxy localhost:8000
}

sudo systemctl reload caddy
curl -fsS https://siftwise.your-domain.com/healthz   # {"status":"ok"}

Already have Postgres? Run just the image: docker run -d -p 8000:8000 -e SIFTWISE_API_TOKEN=... -e SIFTWISE_DATABASE_URL=postgresql+psycopg://user:password@db-host:5432/siftwise ghcr.io/pramodbasavarajmenasi/siftwise:0.1.0. It applies database migrations on start.

4.2 Add two repo secrets (Settings → Secrets and variables → Actions):

Secret	Value
`SIFTWISE_URL`	`https://siftwise.your-domain.com`
`SIFTWISE_TOKEN`	the server's `SIFTWISE_API_TOKEN` (from its `.env`)

4.3 Add the workflow. The push: main trigger matters: main-branch runs are the history siftwise learns from.

name: CI
on:
  push:
    branches: [main]      # records main history
  pull_request:           # NEVER pull_request_target: it exposes secrets to fork code
permissions:
  contents: read
  actions: read           # the run's start time
  pull-requests: write    # the PR comment
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0  # full history, so the diff works
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt   # your install step

      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@v0.1.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

      - name: Tests
        env:
          MODE: ${{ steps.siftwise.outputs.mode }}
          COMMAND: ${{ steps.siftwise.outputs.command }}
          JUNIT_DIR: ${{ steps.siftwise.outputs.junit-dir }}
        run: |
          if [ "$MODE" = selective ]; then
            # each pytest command writes its own report to JUNIT_DIR
            if [ -n "$COMMAND" ]; then eval "$COMMAND"; else echo "No tests affected."; fi
          else
            pytest --junitxml="$JUNIT_DIR/full.xml"   # your normal full suite
          fi

      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@v0.1.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

A few rules that save you debugging time:

Pass the command through env: as above, never as ${{ }} inside run:.
record uploads `/junit*.xml** plus everything in junit-dir. Name your report files junit*.xml or set its junit` input.
Go: write JUnit with gotestsum --junitfile. Jest: use a JUnit reporter with file paths.
Java: siftwise gives no command, so run mvn test or ./gradlew test in the full-suite branch. Their reports are named TEST-*.xml, which the default glob misses, so set junit: "**/target/surefire-reports/TEST-*.xml" (Maven) or junit: "**/build/test-results/test/TEST-*.xml" (Gradle) on record.
Pin @v0.1.0 (or a commit SHA) for reproducible CI.
Using a test matrix? Give each leg's record a variant, let only one leg comment (comment: ${{ strategy.job-index == 0 }}), and add one actions/rollup job after all legs. Full example in the README.

4.4 Open a PR. You'll get a sticky comment: what ran, what was skipped and why, and a verdict for every failure.

4.5 Optional extras:

Gate on real regressions: set fail-on: new_regression,unknown on record (single job) or rollup (matrix), give the test step continue-on-error: true, and make that job a required check. With fail-on set, siftwise fails closed: if verdicts are unavailable, the step fails too.
Re-run unknown failures to catch flakes (pytest): rerun-unknown: 2 and rerun-prefix: uv run on record.
Shadow mode (recommended at first): add shadow: 10% to select. On 10% of runs the full suite runs anyway, and the PR comment says what selection would have caught or missed. Check your recall any time:

siftwise report --repo me/my-app --days 30 --by week

My suggested rollout

Run the server and record main for a few days (just the push: main trigger).
Turn on PRs with shadow: always or shadow: 10%, and watch the recall.
Happy with the numbers? Let selection run, and add fail-on if you want gating.

Security and limits (read this before you host it)

One token, full access. SIFTWISE_API_TOKEN can read and write every repo on that server. No per-repo or read-only tokens yet. Use a separate server for repos that shouldn't see each other's history.
Fork PRs get no secrets, so they run the full suite with no comment. That's expected. Keep another required check for them.
Protect .github/workflows/ with CODEOWNERS or a ruleset: a PR that edits workflows can change any gate.
What's stored: test IDs, outcomes, durations, failure messages (truncated to 4 KB), changed paths, branch names, SHAs. Treat the database like your CI logs. PR comments never show raw failure messages.
Always full suite: merge-queue, scheduled and manual runs (no diff base). The Action is tested on Linux runners only.

Troubleshooting

You see	Fix
`siftwise: command not found`	`uv tool update-shell`, then open a new terminal
Every endpoint returns 503	the server has no `SIFTWISE_API_TOKEN`; check its `.env`
`select` fails in CI about a shallow clone	add `fetch-depth: 0` to `actions/checkout`
No PR comment	check `pull-requests: write` permission and the two secrets; fork PRs never get one
Always the full suite	no history yet (record `main` first), or the diff touched a build/CI file; run `select --json` to see the reason
Server restarts with `Can't locate revision ...`	you went back to an older image; use the newer one (with `--build` from a checkout: rebuild after pulling)
Agent can't find `siftwise`	use the full path from `uv tool dir --bin` in its MCP config

Does it actually work?

I didn't want to just claim it works, so I replayed real CI history from open-source projects, with no peeking at the future:

Repo	Failures caught	Test time skipped
rdflib (Python, ~18,000 tests)	2,049 of 2,050; 140 of 141 in selective runs (the one miss: a flaky test)	10.4% on the 10 most recent PRs (from lint-only PRs); 0.6% over 19 main commits
substrate (Go, ~9,000 tests)	2,043 of 2,044; 38 of 39 in selective runs, with two declared rules (26 of 38 without)	5.1% on PRs, 13.9% over 40 main commits
Cirq (Python monorepo, 9 packages)	not measurable (main never failed)	37.9% over 132 merged PRs
kubo (Go, ~4,100 tests, older week-2 code)	19 of 19 (those runs fell back to the full suite)	30.7% over 20 main commits; 0% on recent PRs (all dependency/CI bumps)

Triage: on substrate, new_regression verdicts held up on 319 of 319 labelled failures, and only 5 of 406 labelled regressions were wrongly called flaky (4 likely label noise). On rdflib, 0 regressions were called flaky.

Speed: /select takes about 0.4 s on 18,000 tests.

Where it falls short: savings follow package boundaries. In codebases where everything imports everything (like rdflib's core), safe selection still runs ~99% of the suite. siftwise just runs it, rather than guess. These are replays of history, not live production numbers; shadow mode is how you measure it on your own repo.

All the numbers and methodology: full write-up · eval results.

What's next

v0.2: prove it and widen its reach

Live dogfooding: run siftwise on its own CI and publish live shadow-mode recall numbers.
More runners for rerun-unknown: Go, Jest and Java (currently pytest only).
Other MCP clients: test and document Cursor and Codex.
GitLab CI support, beyond GitHub Actions.

Try it, break it, tell me

This is v0.1.0 (alpha), built by one person in six weeks, and fully open source (Apache-2.0). The fastest way to make it better is you running it on a real repo.

⭐ Star it if the idea clicks: it genuinely helps a solo project get seen.
🐛 Open an issue if it picks the wrong tests. Misses are the most valuable bug reports.
🛠️ Contribute: anything on the v0.2 list (re-run runners, MCP client testing, GitLab CI) is a great place to start. See CONTRIBUTING.md.

Point it at your repo and drop your numbers in the comments. I want to know how much CI time it saves you. 👇

DEV Community: Pramod Basavaraj Menasi