DEV Community: Pramod Basavaraj Menasi

siftwise v0.3: riskiest tests first, time budgets, Jest and Java selection, and live dogfooding numbers

Pramod Basavaraj Menasi — Wed, 11 Nov 2026 04:00:30 +0000

siftwise is an open-source, self-hosted tool that runs only the tests your change can affect, and tells you whether a failing test is your fault, already broken on main, or just flaky.

New here? Post 1 explains the idea and sets up the server and the GitHub Action. Post 2 adds GitLab CI and five coding agents over MCP.

Today v0.3.0 is out. The theme: smarter selection.

pip install --upgrade siftwise   # 0.3.0

What's new in v0.3

🎯 Risk order: the tests most likely to fail run first, so a failure shows up early.
⏱️ Time budget: "the riskiest tests that fit in 5 minutes", for fast feedback.
🟨 Jest selection: a JS/TS import graph, with no Node.js needed.
☕ Java selection for Maven and Gradle, including multi-module builds.
🐶 Dogfooding: siftwise now runs on its own CI on every PR and push, with live numbers.
🐛 Follow-ups and fixes from testing v0.2 on real repos, three of which made siftwise silently miss tests.
🙌 First outside contributors: two of them, the same morning I opened my first good first issue.

Like the earlier posts, this one is the whole guide, with the real numbers, including the ones that didn't go my way.

Planned vs shipped

At the end of post 2 I promised five things for v0.3. Here's what happened:

Promised in v0.2	Shipped in v0.3
Run the tests most likely to fail first	✅ Risk order (#42), plus import depth on top (#48)
Time-budget mode	✅ `--budget 5m` everywhere: CLI, Action, GitLab, MCP (#43)
A real JS/TS import graph for Jest	✅ (#44), plus computed imports like `import.meta.glob` (#51)
Test selection for Java	✅ Maven and Gradle (#45), plus module build-file scoping, one command per build and a first Gradle replay (#54)
Live recall numbers from dogfooding	⚠️ Live time-saved numbers, yes. Live recall, not yet: none of siftwise's own runs has failed so far, so there's nothing to catch or miss. More in Part 5.

Not planned, but shipped too:

the v0.2 follow-ups (#46): Jest without addFileAttribute, JUnit 5 @DisplayName re-runs, Go workspaces and nested modules, and the GitLab Tests tab;
go list no longer edits go.mod (#52);
six bug fixes, three of which made siftwise silently miss tests (Part 6);
new evals: time to first failure, recall within a budget, and a Gradle replay.

Upgrading from 0.2

v0.3 has two database migrations (0011 and 0012; both only add columns). The server runs them on start, so with Docker it's the usual:

cd siftwise && git pull                       # the compose file now names the 0.3.0 image
docker compose up -d --wait                   # pulls 0.3.0, migrates, restarts
uv tool upgrade siftwise                      # the CLI; or: pip install --upgrade siftwise

Then move your Action pins to @v0.3.0, and the GitLab template to v0.3.0 with its new integrity hash.

Update the server and the clients together. A new CLI talking to a 0.2 server still works: it sends the request again without the new fields and gets 0.2's selection. The exception is a time budget, which can't be dropped silently: the Action, GitLab and MCP fall back to the full suite with a warning, and siftwise select --budget exits with an error. An old CLI works with a new server unchanged.

Part 1: risk order, riskiest tests first

In v0.2, siftwise picked the right tests but ran them in path order. If the one that fails is last, you still wait for everything before it.

Now every selected test gets a score, a rank and a reason, and the command runs the riskiest first. The score is a fixed, documented formula, with no machine learning:

score = link + 3 × recency + 2 × failure rate   (recency and rate discounted if the test is flaky)
        + 2 if the test is new

link     = 6 if the change reaches the test directly, 3 through other files, 0 by history only
recency  = halves every 7 days since the test last failed

So a test whose file imports the code you changed, and that failed last week, goes to the front. Ties go to the faster test.

Nothing to configure: it's on by default. siftwise select --json shows each test's score, rank and rank_reason; the PR comment and GitLab note name the three riskiest tests. To keep the old order: --order default, the Action's order: default, SIFTWISE_ORDER: default on GitLab, or order: "default" over MCP.

The order never changes which tests run, only when. How much it's respected depends on the runner:

Runner	What the order guarantees
pytest	arguments run in the order given (unless a plugin like pytest-xdist or pytest-randomly reorders them)
Go	packages run one after another, riskiest first
Jest	a hint: Jest runs files in parallel and may reorder them
Maven	modules run in Maven's reactor (dependency) order

Does it help? The numbers

I replayed real CI history, evaluating each run using only the history from before it. The number is how much of a selective run passes before the first failure shows, as a median (lower is better):

Repo	Default order	Risk order
RDFLib/rdflib (Python), 17 runs	37.3% of the time	1.4%
agent-substrate/substrate (Go), 19 runs	42.0%	25.6%

On rdflib, the failure typically shows in the first 1.4% of the run instead of after a third of it. Recall didn't change: the same tests ran, in a different order.

I also tried a "cost-aware" order (risk per second of estimated test time). It wasn't clearly better on both repos and was worse by test count, so it didn't ship.

The feature that made the numbers worse

v0.3 also adds import depth: a test file that imports the changed code itself counts as a direct link (6), above one that reaches it through other files (3).

My first version did this for every language, and substrate got worse: the median went from 23.3% to 34.4% (by test count), with the first failure coming later in 6 runs and earlier in only 1.

Going failure by failure showed why. Go's graph is per package, and in Go most dependents import the changed package directly: 83% of the import-linked packages were at depth 1. So "direct" lifted almost every package above the e2e suites that a [[depends]] rule selects, and those were the ones failing.

The fix: Go packages aren't promoted by depth; Python, Jest and Java test files are. substrate is back to exactly its earlier numbers. I chose that rule because a test file's imports say much more precisely what a test uses than a package's do, not because it scored best on two repos. Honestly, import depth didn't move rdflib's medians either (only the mean improved a little), so I'm not claiming a speed-up from it yet.

Part 2: time budget

Sometimes even the selection is too big for the feedback you want right now. Set a budget, and siftwise runs the riskiest tests whose estimated time fits:

siftwise select --repo acme/shop --base origin/main --budget 5m   # 90s, 5m, 1h30m

Where	Setting
GitHub Action	`budget: 5m` (outputs `budget-limited`, `budget-left-out`)
GitLab	`SIFTWISE_BUDGET: 5m`
MCP `select_tests`	`budget` argument
API	`budget_seconds`

It's off by default. Two things matter before you use it:

A budget is an estimate, not a cap. siftwise uses each test's mean duration from history, so a run can take longer. Keep a real CI timeout if you need a hard limit.
A budget-limited run is not a complete check, and siftwise says so plainly: the PR comment shows "budget-limited (estimated 4m 50s of a 5m target): N tests left out". Run the full selection before merging.

In shadow mode, a failure the budget left out is counted as a miss with the cause budget, so you can measure what a budget costs you on your own repo.

How much a budget catches, on the same replays: runs whose failure is still caught at a budget of 10%, 25% and 50% of each run's time:

Repo	10%	25%	50%	No budget
rdflib	22 of 31	24 of 31	28 of 31	30 of 31
substrate	34 of 72	43 of 72	54 of 72	71 of 72

On rdflib, a 10% budget still catches most failures. On substrate, it catches under half. So a budget is great for a quick first signal, and not a replacement for the full selection.

Part 3: Jest selection

Until now, siftwise matched Jest tests only by file name (foo.ts → foo.test.ts), so a change most tests reach through imports ran the whole suite. Now siftwise reads your JS/TS imports and also runs every test file that imports the changed file, at any depth.

What it understands, without running Node:

relative imports, index files and TypeScript's .js → .ts;
tsconfig/jsconfig paths and baseUrl (with extends);
npm, yarn, pnpm and lerna workspace packages;
a JSON Jest moduleNameMapper;
JSON and other files a test imports;
computed imports with a fixed folder: import(`./locales/${lang}.json`), require('./plugins/' + name), Vite's import.meta.glob and webpack's require.context count as imports of every file they could load (#51).

When it can't see, it plays safe: a test that reaches an import it can't bound (like require(somePath)) runs for every JS change, and changes to tsconfig, Jest, Babel or SWC configs, root manifests or lockfiles run the full suite.

New test files run first, by path, once siftwise can tell where Jest expects them. On jestjs/jest (2,272 JS/TS files), the graph builds in about 2 seconds.

Step 3.1: let siftwise know each test's file

siftwise learns which file each test lives in from the JUnit report. Add jest-junit, ideally with addFileAttribute:

npm install --save-dev jest-junit

{
  "jest": {
    "reporters": ["default", ["jest-junit", { "addFileAttribute": "true" }]]
  }
}

New in v0.3: without addFileAttribute, record searches the checkout for the one test file whose string literals match the suite's test names, and adds it to the report. That works for most files, but titles built from variables can't be matched, so setting the attribute is still the reliable option.

Step 3.2: the workflow

      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci

      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@v0.3.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

      - name: Tests
        env:
          MODE: ${{ steps.siftwise.outputs.mode }}
          COMMAND: ${{ steps.siftwise.outputs.command }}
          JEST_JUNIT_OUTPUT_DIR: ${{ steps.siftwise.outputs.junit-dir }}
        run: |
          export PATH="$PWD/node_modules/.bin:$PATH"   # the command starts with `jest`
          if [ "$MODE" = selective ]; then
            if [ -n "$COMMAND" ]; then eval "$COMMAND"; else echo "No tests affected."; fi
          else
            jest   # your normal full suite
          fi

      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@v0.3.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

JEST_JUNIT_OUTPUT_DIR puts the report where record looks for it. As always, pass the command through env:, never as ${{ }} inside run:.

An honest gap: there's no recall number for Jest yet. Replays need old CI runs that uploaded JUnit with file paths, and no public Jest repo I found does that. I validated the graph by hand on jestjs/jest. If you run siftwise in shadow mode on a JS repo, I'd love your numbers.

Part 4: Java selection, Maven and Gradle

In v0.2, Java tests were recorded and re-run but always ran in full. Now siftwise reads your build without running Maven, Gradle or a JVM:

the modules from pom.xml or settings.gradle[.kts];
a static graph of which test classes use which classes (imports, same-package names, fully-qualified names, Class.forName strings), across modules;
name matches: changing Foo.java selects FooTest, FooTests, TestFoo and FooIT;
a module's src/test/resources runs that module's tests.

One command per build where that's safe:

./mvnw -B -pl core,api -am test -Dtest=CartTest,MoneyTest            # Maven
./gradlew :core:test --tests CartTest :api:test --tests MoneyTest   # Gradle

Build files are scoped too (#54): a change to a module's own pom.xml or build.gradle[.kts] runs the tests of that module and of every module that depends on it. The root build file, settings.gradle, version catalogs, wrappers, parent and BOM POMs still run the full suite, because they can affect everything.

Where it can't see, the full suite runs: Kotlin, Groovy or Scala sources, custom source folders, code-generating annotation processors, or a build it can't read. Reflection (Spring scanning, ServiceLoader) needs a [[depends]] rule.

Step 4.1: the workflow

The only difference from Python is where the reports live, so point record at them:

      - uses: actions/setup-java@v6
        with:
          distribution: temurin
          java-version: 21

      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@v0.3.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

      - name: Tests
        env:
          MODE: ${{ steps.siftwise.outputs.mode }}
          COMMAND: ${{ steps.siftwise.outputs.command }}
        run: |
          if [ "$MODE" = selective ]; then
            if [ -n "$COMMAND" ]; then eval "$COMMAND"; else echo "No tests affected."; fi
          else
            ./mvnw -B test   # or: ./gradlew test
          fi

      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@v0.3.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}
          junit: "**/target/surefire-reports/TEST-*.xml **/build/test-results/test/*.xml"

The numbers

dropwizard/dropwizard (Maven, 42 modules), 148 PR runs: 74 of 74 failures caught. In the 22 selective runs, siftwise skipped a median 90.6% of the estimated test time (26% to 93%).
The catch: most of dropwizard's PRs are dependency bumps to its BOM, which correctly run everything. So over all 148 runs, the time saved is much smaller: 11.9%. Module build-file scoping moved 5 more runs from full to selective. A repo with more code changes and fewer bumps would save more.
naver/fixture-monkey (Gradle): 3 of 3 failures caught. That's a small history, so treat it as a first check, not proof.

The dropwizard replay also found a real bug: Surefire writes a class-level result with an empty test name for a JUnit 5 @Nested class, and siftwise rejected the whole upload for it. Every Windows leg of dropwizard's CI history (781 uploads) was rejected before the fix.

Part 5: dogfooding, siftwise on its own CI

This was the last promise from post 2, and the one I care most about. Replays of other projects' history are useful, but they're replays. Dogfooding is siftwise running live on real changes, as they happen.

How it works

Every siftwise PR and push to main runs a Dogfood workflow in shadow mode:

A siftwise server starts inside the CI job, on the runner's own Postgres, with a random per-job token. There's no hosted server and nothing to pay for.
History comes from the GitHub Actions cache. The job restores the database from main's latest cache entry, and saves a new one at the end.
siftwise stores what it would have selected, then the full suite runs anyway.
It compares: for every failure, would the selection have caught it? How much time would it have saved?
Each run uploads a small summary file. scripts/dogfood_report.py adds them all up.

It gates nothing: no fail-on, no PR comment, and every siftwise step fails open. So it can never block my own CI.

The safety part: a PR's cache entry is scoped to that PR, so a PR (including one from a fork) can read main's history but can never write into it. Only main's own runs build the history. For the numbers, I also leave out the fork PRs' runs, because code from the fork wrote their summaries; each of those changes still counts once, as its push to main.

The numbers (26 Oct to 9 Nov 2026)


Workflow runs	36 (17 PR runs, 19 pushes to main)
Shadow runs counted	32 (the first 2 had no history yet; the 2 fork runs are left out)
Would have been selective	9 of 32
Tests selected in those 9	a median 39% of the known tests
Time saved	14.7% of 23.0 minutes over the 32 runs
Failures	none: every run's suite passed, so recall can't be measured yet
Weekly time saved	21%, 7% and 34%

Read these as early numbers. A merged PR usually counts twice (once as its PR run, once as its push to main), so the 9 selective runs are really about 6 distinct changes.

What dogfooding taught me

Why 23 of 32 ran in full: 20 changed a build or config file (the Dockerfile and docker-compose.yml in 6 runs each, workflows, and my test fixtures' own build.gradle.kts, tsconfig.json and jest.config.ts), and 3 changed a file no test reaches (like glama.json).
The fixture finding is the most useful one. siftwise's tests include tiny sample Gradle and TypeScript projects. Their build files are test data, but siftwise treats them like real build files and runs everything. Scoping them safely is the first item for v0.4. I'd never have found this from replays.
Each run now records the risk rank of every failure, so once a dogfood run does fail, the report shows not just "caught or missed" but how early risk order would have surfaced it.

You can do the same on your repo: the Action's shadow input (always, or a sample like 10%) runs the full suite and reports what siftwise would have caught or missed, before you trust it with anything. Details: Shadow mode in the README.

Part 6: v0.2 follow-ups and fixes

Things I found while testing v0.2 on real projects (#46):

Jest without addFileAttribute: selection and re-runs now work without it (Part 3).
JUnit 5 @DisplayName and parameterized test names map back to their methods, so Gradle and phrased-Maven re-runs work.
Go workspaces (go.work) and nested modules work for selection, commands and re-runs.
The GitLab template now declares its JUnit reports, which the pipeline's Tests tab reads.

Fixes, three of which (🔴) made siftwise silently miss tests:

🔴 Nested Go modules: a nested module's dependents were missed, and its tests ran from the root, where Go can't build them. Now they run as go test -C <module> ./pkg.
🔴 Missed JS edges: import('./a/' + x), import.meta.glob and require.context created no edges, so changing a file they load didn't select the tests that load it.
go list could edit go.mod when GOFLAGS had -mod=mod. It now runs read-only (#52). On Go 1.15 or older, set GOFLAGS=-mod=readonly yourself; making siftwise do it is an open good first issue (#67).
Java uploads rejected for a @Nested class's result (the dropwizard bug above).
🔴 Same-named Java test classes in two modules: a change to one module's test resources ran only the other module's copy.
A deleted Jest test file was still named in the jest command, which then failed with "No tests found".

What building v0.3 taught me

Measure the features you're sure about. Import depth was the "obviously better" change, and it made one repo worse. Without the replay, I'd have shipped it.
Your dependencies change under you. On release day, a new Typer version started requiring pydantic, which broke the check that keeps the light CLI install small. siftwise's code didn't change; the light install grew from 9 MB to 18 MB, and the check now verifies that siftwise itself never imports server code.
good first issue works. I labelled a one-line bug, and by the same morning two people had sent fixes. One fixed it; the other added the test that stops it from coming back. Both got merged.

Community

Thanks to @bfs2021 (#62) and @mursalhayat1 (#63), siftwise's first outside contributors.
siftwise's MCP server is listed in the HOL Guard MCP server catalog (opt-in, after a review of its code and disclosures) and on Glama.

Troubleshooting

You see	Fix
Jest tests always run in full	add `addFileAttribute: "true"` to jest-junit (Step 3.1)
`jest: command not found`	put `node_modules/.bin` on `PATH`, as in Step 3.2
Java tests never selected	`record` didn't find the reports: set its `junit` input (Step 4.1)
Every Java PR runs the full suite	it changed a root or parent build file, which runs everything by design
`--budget` fails with a server error	the server is still on 0.2: upgrade it
The order looks wrong in Jest	Jest reorders files across its workers; the order is a hint there

Known limits

Jest: no recall number yet; jest.config.js can't be read without Node (tsconfig and relative paths still resolve); a custom testMatch isn't read.
Java: Kotlin, Groovy and Scala tests fall back to the full suite; Maven runs modules in reactor order, not risk order.
Budget: an estimate, not a cap.
Dogfooding: 32 runs and no failures yet, so no live recall.
MCP is tested on Windows only; GitLab on gitlab.com Free only.
Everything else: the CHANGELOG.

What's next: v0.4

Safe scoping for build files inside test fixtures (the dogfooding finding).
-mod=readonly for older Go versions.
More recall numbers: a Jest replay, more Gradle history, and the first live recall from dogfooding.
More good first issues. If you'd like to contribute, start here.

Try it, break it, tell me

If you have a Jest or Java repo, try siftwise in shadow mode and tell me what it would have caught or missed: that's exactly the data v0.3 is missing. Misses make the best bug reports. And a ⭐ helps a small project get seen. 👇

siftwise v0.2: run it on GitLab CI and with 5 coding agents (full setup guide)

Pramod Basavaraj Menasi — Wed, 28 Oct 2026 07:27:40 +0000

Update: v0.3 is out: riskiest tests first, time budgets, Jest and Java (Maven, Gradle) selection, and live dogfooding numbers. Read the v0.3 guide →

A week ago I launched siftwise: an open-source, self-hosted tool that runs only the tests your change can affect, and tells you whether a failing test is your fault, already broken on main, or just flaky.

If you missed it, the first post explains the idea and how to set up the server and the GitHub Action.

Today v0.2.0 is out. The theme: use it in more places.

pip install --upgrade siftwise   # 0.2.0

What's new in v0.2

🦊 GitLab CI: the same test selection, failure verdicts and sticky comment as on GitHub, now on merge requests.
🤖 5 coding agents tested: Claude Code, GitHub Copilot, Cursor, Codex CLI and Gemini CLI can ask siftwise which tests to run and why a test failed.
🔁 Flaky-test re-runs for Go, Jest, Maven and Gradle, not just pytest.
🔒 Safer PR comments: a file name in a PR can no longer sneak a link or an @mention into siftwise's comment.
🐶 Dogfooding: siftwise now runs on its own CI, collecting live numbers for v0.3.

Like the first post, this one is the whole guide: by the end you can use every new feature.

Before you start

You need a siftwise server. If you don't have one yet, Step 1 of the first post gets one running locally in 5 minutes with Docker:

git clone https://github.com/PramodBasavarajMenasi/siftwise.git
cd siftwise
[ -f .env ] || echo "SIFTWISE_API_TOKEN=$(openssl rand -hex 32)" > .env
docker compose up -d --wait
curl -fsS http://localhost:8000/healthz   # {"status":"ok"}

Already on 0.1.0? Upgrading is a drop-in, with no database migration:

cd siftwise && git pull                       # the compose file now names the 0.2.0 image
docker compose up -d --wait                   # pulls 0.2.0 and restarts the server
uv tool upgrade siftwise                      # the CLI; or: pip install --upgrade siftwise

All commands are bash; on Windows, use Git Bash or WSL.

Part 1: connect your coding agent (MCP)

siftwise ships an MCP server with three tools your agent can call:

Tool	The agent calls it	It gets back
`select_tests`	before running tests	the test command, with a reason for each test
`triage_failure`	when tests fail, before fixing	a verdict per failure: new regression, pre-existing, flaky or unknown
`explain`	when something looks odd	one test's history and why it was (not) selected

What it looks like in practice:

You: "Fix the bulk discount bug."
The agent edits cart.py, calls select_tests, and runs pytest tests/test_cart.py instead of the whole suite.
A test fails. It calls triage_failure and gets new_regression, linked to the file it just changed, so it fixes its own bug.
A failure that's pre_existing (already broken on main) gets left alone instead of "fixed".

In v0.1 I had only tested this with Claude Code. For v0.2 I tested it by hand on Windows in five clients, and all three tools worked in each:

Client	Config file	How siftwise finds your repo
Claude Code	`claude mcp add` (or `.mcp.json`)	the project folder Claude Code tells it
GitHub Copilot (VS Code, agent mode)	`.vscode/mcp.json`	the open workspace folder
Cursor	`~/.cursor/mcp.json` or `.cursor/mcp.json`	asks Cursor which folder is open, nothing to configure
Codex CLI	`~/.codex/config.toml`	the `cwd` you set
Gemini CLI	`~/.gemini/settings.json`	the `cwd` you set

Step 1.1: install the MCP extra

uv tool install 'siftwise[mcp]'   # or: pip install 'siftwise[mcp]'
uv tool update-shell              # once, if `siftwise` isn't found; then restart the agent
uv tool dir --bin                 # prints the folder, if you need the full path below

You need two values for every client: your server address (http://localhost:8000 locally) and the API token from the server's .env.

Windows tip: agents don't always see your shell's PATH. If a client can't start siftwise, use the full path from uv tool dir --bin, e.g. "C:\\Users\\you\\.local\\bin\\siftwise.exe" in JSON, or 'C:\Users\you\.local\bin\siftwise.exe' (single quotes) in TOML.

Step 1.2: add it to your client

Claude Code (one command, run inside your repo):

claude mcp add --env SIFTWISE_SERVER=http://localhost:8000 \
    --env SIFTWISE_API_TOKEN=your-token --transport stdio siftwise -- siftwise mcp

Check with /mcp in a session.

GitHub Copilot in VS Code: create .vscode/mcp.json in your repo. VS Code asks for the token on first start and stores it securely, so this file is safe to commit:

{
  "inputs": [
    { "type": "promptString", "id": "siftwise-token", "description": "siftwise API token", "password": true }
  ],
  "servers": {
    "siftwise": {
      "type": "stdio",
      "command": "siftwise",
      "args": ["mcp"],
      "env": {
        "SIFTWISE_SERVER": "http://localhost:8000",
        "SIFTWISE_API_TOKEN": "${input:siftwise-token}"
      }
    }
  }
}

Click Start above "siftwise" in the file, then use Copilot Chat in Agent mode.

Cursor: ~/.cursor/mcp.json (all projects). No repo path needed: siftwise asks Cursor which folder you have open.

{
  "mcpServers": {
    "siftwise": {
      "type": "stdio",
      "command": "siftwise",
      "args": ["mcp"],
      "env": {
        "SIFTWISE_SERVER": "http://localhost:8000",
        "SIFTWISE_API_TOKEN": "your-token"
      }
    }
  }
}

Check in Cursor's settings (search "MCP"): siftwise should show a green dot and 3 tools.

Codex CLI: ~/.codex/config.toml. Set cwd to your repo:

[mcp_servers.siftwise]
command = "siftwise"
args = ["mcp"]
cwd = "/path/to/your/repo"   # Windows: 'C:\Users\you\code\repo'

[mcp_servers.siftwise.env]
SIFTWISE_SERVER = "http://localhost:8000"
SIFTWISE_API_TOKEN = "your-token"

Check with /mcp. If Codex says the model "isn't supported with a ChatGPT account", pick another one with /model; that's Codex's account check, not siftwise.

Gemini CLI: ~/.gemini/settings.json. The token must be in env, because Gemini CLI strips inherited variables with TOKEN in their name:

{
  "mcpServers": {
    "siftwise": {
      "command": "siftwise",
      "args": ["mcp"],
      "cwd": "/path/to/your/repo",
      "env": {
        "SIFTWISE_SERVER": "http://localhost:8000",
        "SIFTWISE_API_TOKEN": "your-token"
      }
    }
  }
}

Check with /mcp.

Keep tokens out of git. The Cursor, Codex and Gemini files above live in your home folder, outside any repo. Don't put a real token into a config file inside your project.

Windsurf and Cline configs are in docs/mcp-clients.md, but I haven't tested those yet. I've also only tested on Windows so far; reports from macOS and Linux are very welcome.

Step 1.3: tell your agent when to use it

Add this to your CLAUDE.md, AGENTS.md or the client's rules file:

Before running tests, call the siftwise `select_tests` tool and run the command it returns.
When tests fail, call `triage_failure` before fixing anything: fix `new_regression` failures,
re-run `flaky` ones, and leave `pre_existing` ones (already failing on main).

Step 1.4: try it

Make a change in your repo and ask the agent:

Ask	Expect
"Which tests should I run for my change? Use siftwise."	a test command with a reason per test
"The tests failed. Ask siftwise before fixing anything."	a verdict per failing test
"Why does siftwise select `<a test>`?"	the test's history and the link to your change

Your repo needs some recorded history first (a recorded run on main; see Step 2 of the first post), otherwise siftwise safely says "run the full suite".

How siftwise finds your repo

Clients start MCP servers in different folders, so siftwise uses the first of these that works:

a repo_path the agent passes;
SIFTWISE_REPO_PATH in the config;
the project folder Claude Code sets;
the folders the client has open (MCP roots);
the folder the server started in.

If none is a git repo, every tool says so and how to fix it, and select_tests falls back to "run the full suite".

Part 2: GitLab CI

GitLab merge requests now get what GitHub PRs get. Here's the note from a real test MR on gitlab.com, where I broke a discount calculation on purpose:

Selective: 2 of 4 tests ran.
The verdict: the failing test is a new regression, high confidence, linked to the changed file.
One sticky note, updated on every push, not a new comment each time.
The gate: fail-on failed the pipeline, as configured.

Step 2.1: make your server reachable from GitLab

GitLab's runners can't see your localhost. Run the server somewhere they can reach, behind HTTPS. The first post's Step 4.1 shows a VM with Docker and Caddy. The token travels in a request header, so use https://; siftwise warns if you don't.

Step 2.2: create a GitLab token for the comment

siftwise posts its note through GitLab's API, so it needs a token. Give it the least that works:

GitLab.com Free: a fine-grained personal access token, limited to this one project, with Work Item create, read and update (MR comments are part of it), Merge Request read, User read (so siftwise only edits its own note), and an expiry date. Ideally it belongs to a dedicated bot user with the Reporter role.
Premium, Ultimate or Self-Managed: a project access token with the Reporter role and the api scope.

GitLab's built-in CI_JOB_TOKEN can't write MR comments, which is why a separate token is needed.

Step 2.3: add three CI/CD variables

In your project: Settings → CI/CD → Variables → Add variable:

Key	Value	Visibility
`SIFTWISE_SERVER`	`https://siftwise.your-domain.com`	visible is fine
`SIFTWISE_TOKEN`	your server's `SIFTWISE_API_TOKEN`	Masked and hidden
`SIFTWISE_GITLAB_TOKEN`	the token from Step 2.2	Masked and hidden

Protected or not? Protected variables only reach protected branches, so most MR pipelines wouldn't get the tokens (siftwise then runs the full suite and posts nothing). Unprotected (masked and hidden) works for every MR, but anyone who can push a branch could read them in their pipeline. If everyone with Developer access may use siftwise, go unprotected, and run a separate siftwise server per team or trust boundary. The README's protected or unprotected section has the details.

Step 2.4: add the template to `.gitlab-ci.yml`

Include the template by its release tag and a hash, so the code that runs with your tokens can't change under you. Get the hash once:

curl -sL https://raw.githubusercontent.com/PramodBasavarajMenasi/siftwise/v0.2.0/gitlab/siftwise.gitlab-ci.yml \
  | openssl dgst -sha256 -binary | openssl base64 -A

Then:

include:
  - remote: https://raw.githubusercontent.com/PramodBasavarajMenasi/siftwise/v0.2.0/gitlab/siftwise.gitlab-ci.yml
    integrity: sha256-PASTE-THE-HASH

workflow:  # merge request pipelines for open MRs, branch pipelines otherwise
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

siftwise-select:
  extends: .siftwise-select

test:
  extends: .siftwise-record
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - |
      if [ "$SIFTWISE_MODE" = selective ]; then
        if [ -s "$SIFTWISE_COMMAND_FILE" ]; then sh "$SIFTWISE_COMMAND_FILE"; else echo "No tests affected."; fi
      else
        pytest --junitxml=.siftwise-ci/junit/full.xml   # your full suite
      fi
  artifacts:
    reports:
      junit: .siftwise-ci/junit/*.xml

siftwise-rollup:
  extends: .siftwise-rollup
  variables:
    SIFTWISE_FAIL_ON: new_regression,unknown   # optional: fail the pipeline on real regressions

What each job does:

siftwise-select picks the tests and passes the result to your test job.
Your test job runs them, and its after_script uploads the results.
siftwise-rollup gives each failure a verdict, updates the MR note and applies fail-on.

GIT_DEPTH: "0" (full history) is already set by the template, since GitLab clones only 20 commits by default.

Step 2.5: open a merge request

Push .gitlab-ci.yml to your default branch first. That pipeline records main's history.
Make a change on a branch and open an MR.
The MR shows the siftwise note, and .siftwise-ci/ in the job artifacts has the full reports.

Using parallel: matrix:? Give each leg a SIFTWISE_VARIANT; the README's GitLab section shows it.

Just want to try it first? examples/gitlab is a tiny project that runs its own siftwise server inside the pipeline, so you need no server at all. Push it to a new private gitlab.com project, add SIFTWISE_GITLAB_TOKEN, and open an MR that breaks shop/cart.py. That's exactly the test behind the screenshot above.

What keeps it safe

Tokens only come from environment variables, never command arguments. They're redacted from every log line, file and note, and tests check that.
siftwise only edits a note its own token's user wrote, so nobody can hijack it with a fake one.
Test names come from the MR's code, so they're shown as code: they can't add links, @mentions or GitLab quick actions.
Fork MRs get no tokens: they run the full suite, with no note. Never run untrusted forks' pipelines in your project.
An MR can't mark its own failures flaky: known_flaky is read from the MR's base commit.

Part 3: flaky-test re-runs for Go, Jest, Maven and Gradle

When a failure's verdict is unknown, siftwise can re-run just that test. If it passes, the failure is marked flaky (high confidence), so nobody wastes an hour chasing it. In v0.1 this only worked for pytest. Now each failing test gets its own runner:

Language	What siftwise runs	You need
Python	`pytest <node IDs>`	nothing extra
Go	`go test -count=1 -run` with the failing test names, per package	Go; `gotestsum` optional
Jest	`jest --runTestsByPath <file> -t` with the full test name	`jest-junit` with `addFileAttribute: "true"`
Java (Maven)	`mvn -Dtest=Class#method test` (`./mvnw` if present)	nothing extra
Java (Gradle)	`gradle test --tests Class.method` (`./gradlew` if present)	nothing extra

Tested with JDK 17 and 21, Maven 3.9 and 3.10, and Gradle 8.10 and 9.8; siftwise uses your project's own mvn/gradle (or ./mvnw/./gradlew), so other versions should work too.

Turn it on in the GitHub Action's record step (or SIFTWISE_RERUN_UNKNOWN on GitLab):

- if: always()
  uses: PramodBasavarajMenasi/siftwise/actions/record@v0.2.0
  with:
    server-url: ${{ secrets.SIFTWISE_URL }}
    token: ${{ secrets.SIFTWISE_TOKEN }}
    rerun-unknown: 2      # re-run unknown failures up to twice
    rerun-prefix: npx     # how Jest (or pytest, e.g. "uv run") runs in your job

rerun-prefix only applies to pytest and Jest. Go, Maven and Gradle pick their own tool, so a mixed-language repo doesn't break.

Limits: at most 20 tests within 10 minutes; re-runs never turn a red job green; and for Java and Jest the re-run must name tests the way your CI run did (JUnit 5 @DisplayName doesn't match yet). Maven and Gradle re-runs put your CI run's original reports back afterwards, so nothing downstream gets the wrong files.

Part 4: safer PR comments

While building the GitLab note, I noticed the GitHub PR comment didn't escape everything that comes from the PR. A file named like a Markdown link would show up as a real link in siftwise's comment, and a path like pkgs/@acme/... would @mention someone.

Now everything from the PR (test IDs, file paths, variant names) is shown as code, on both GitHub and GitLab. Nothing to configure: upgrade the Action pin to @v0.2.0.

Part 5: dogfooding, siftwise on its own CI

siftwise now runs on its own CI in shadow mode: the full suite still runs every time, and siftwise records what it would have selected and whether that would have caught each failure.

It needs no hosted server: a siftwise server starts inside each CI job, and the history lives in the GitHub Actions cache between runs.

My earlier numbers (2,049 of 2,050 failures caught on rdflib, for example) came from replaying other projects' old CI history. These will be live numbers from real changes, and I'll publish them with v0.3.

What testing on the real thing taught me

Mocks aren't enough. Every GitLab API call was mocked in my tests, and they all passed. On real gitlab.com, the pipeline's own job token got 404 Project Not Found downloading main's artifacts, so the example no longer depends on that.
Re-running a test can delete other reports. A Gradle --tests re-run removed the results of every class it didn't re-run. siftwise now backs up and restores them around every Maven and Gradle re-run.
Every MCP client is a little different: where it starts the server, which environment variables it passes, how it reads JSON schemas. Codex turned optional parameters written as anyOf [type, null] into plain strings, so every schema got simpler. Testing in the real clients still found three bugs the automated tests had missed.

Troubleshooting

You see	Fix
The agent can't start `siftwise`	use the full path from `uv tool dir --bin` in the config
A tool says "not inside a git repository"	open the repo folder itself in the client, or set `SIFTWISE_REPO_PATH`
`select_tests` always says "full suite"	no history yet: record a run on `main` first
Codex: "model not supported"	pick another model with `/model`
Gemini: siftwise says the token is missing	put `SIFTWISE_API_TOKEN` in the server's `env` block
GitLab: no MR note, `HTTP 403` in the log	the token lacks the permissions in Step 2.2
GitLab: full suite on every MR	protected variables don't reach MR branches (see Step 2.3), or the clone is shallow

Known limits

MCP is tested on Windows only; Windsurf and Cline are untested.
GitLab is tested on gitlab.com Free; Self-Managed and Premium are untested.
Java and Jest re-runs need matching test names (no JUnit 5 @DisplayName yet).
Full details are in the CHANGELOG.

What's next: v0.3, smarter selection

Test prioritization: run the tests most likely to fail first.
Time-budget mode: "the best tests that fit in 5 minutes".
A real JS/TS import graph for Jest.
Test selection for Java, not just re-runs.
Live recall numbers from dogfooding.

Try it, break it, tell me

This is still one person building in the open. If you try siftwise with GitLab or with your coding agent, tell me how it went in the comments, especially on macOS or Linux, or with Windsurf or Cline. If it picks the wrong tests, please open an issue: misses are the most useful bug reports. And a ⭐ really helps a solo project get seen. 👇

I built siftwise: run only the tests your change needs (open source, self-hosted)

Pramod Basavaraj Menasi — Wed, 21 Oct 2026 07:25:08 +0000

Update: v0.2 is out: GitLab CI, 5 coding agents tested, and flaky-test re-runs for Go, Jest and Java. Read the v0.2 guide →
Update: v0.3 is out: riskiest tests first, time budgets, and Jest and Java selection. Read the v0.3 guide →

Your coding agent just changed one file. CI is now running all 18,000 tests. Again.

Coding agents now write code faster than CI can test it. Every pull request, whether a person or an agent wrote it, still runs the entire test suite. Anthropic reported a 25x increase in CI jobs over six months as agents sped up writing code (Agentic coding is straining CI), and Linear wrote that AI coding has made CI a bottleneck.

That got me thinking: what if CI only ran the tests a change could actually break?

So over the last six weeks I built siftwise, an open-source, self-hosted tool that does exactly that. v0.1.0 is out on PyPI and GHCR.

pip install siftwise

This post is the whole guide: what it does, how to run it on your laptop, how to connect your own repo, your coding agent, and your CI. By the end you'll have everything you need.

The idea in one sentence

siftwise looks at your diff, works out which tests could be affected, runs only those, and falls back to the full suite whenever it isn't sure, because missing a real failure is worse than running a few extra tests.

Test impact analysis isn't a new idea; there are good commercial tools for it. What I wanted was something open source, self-hosted, and built for agent-driven CI.

Why you might care

⚡ Faster feedback: a one-file change can run just the tests that touch that file, not the whole suite.
🛡️ Safe by default: unsure → full suite. Server down → full suite. It never blocks CI (unless you opt into fail-on).
🔍 Every test has a reason: no black box. You see why each test was picked.
🧪 "Is this my fault?": every failure gets a verdict: new regression, already broken on main, flaky, or unknown.
🤖 Agent-ready: an MCP server, so a coding agent can pick tests and triage failures itself.
🏠 Your data stays yours: runs on your own machine or server, on your own Postgres. No SaaS, no account, no test instrumentation.
📏 Proves itself: shadow mode runs the full suite on a sample of PRs and tells you what selection would have missed.

How it works

select: diffs your branch, builds a Python or Go import graph locally, and asks the server which tests the change can affect.
run: you run only those tests (or the full suite if siftwise isn't confident).
record: JUnit results are uploaded to the server.
rollup: per-test history is updated (failures, durations, flakiness, broken-on-main).
triage: every failure gets a verdict.
shadow compare (optional): on sampled runs, the full suite runs anyway and siftwise checks what it would have missed.
In CI, a sticky PR comment shows it all.

Selection signals

Path mapping (test_cart.py ↔ cart.py)
Import graphs for Python and Go (go list -deps -test)
Failure history: tests that failed when these files changed before
Declared rules in .siftwise.toml, for dependencies no import graph can see

Every selected test comes with a reason, like imports changed module rdflib/term.py.

Supported languages

Language	How tests are selected	Command siftwise gives you
Python (pytest)	`foo.py` → `test_foo.py`; test files importing a changed module (incl. parent packages, conftests); changed test files	`pytest` node IDs or test files
Go	the changed package and every package whose tests import it	`go test ./pkg`, or `go test -run` with the selected test names
JS/TS (Jest)	`foo.ts` → `foo.test.`, `foo.spec.`, `__tests__/foo.*` (needs file paths in the JUnit)	`jest <files>`
Java (Maven/Gradle)	recorded with full history (for triage); selection always runs the full suite (for now)	none: you run your usual `mvn test` or `./gradlew test`

Safe fallbacks

Unknown diff, a changed build/CI/dependency file, no history yet, or a changed file that no signal covers: full suite. If the server is down, CI just runs everything.

Failure triage

new_regression  high   tests.test_cart::test_total
    tests/test_cart.py tests changed src/cart.py
pre_existing    high   tests.test_db::test_query
    failing on main on ubuntu-py3.12 since 4f2a9c1 (3 main runs)
flaky           medium tests.test_net::test_fetch
    flaky 1 time in 90 days (1 unrelated fail-then-pass)
siftwise: 3 failing test(s): 1 new regression, 1 pre existing, 1 flaky, 0 unknown

Nobody wastes an hour debugging a test that was already broken on main.

Before you start

You need:

Docker (with Docker Compose) for the server
git
uv (or pip / pipx) for the CLI; Python 3.11+
Go users: go on your PATH, plus gotestsum for JUnit output (go install gotest.tools/gotestsum@latest)

All commands are bash. On Windows, use Git Bash or WSL.

Step 1: Try it in 5 minutes (quickstart)

Start the server (Postgres included):

git clone https://github.com/PramodBasavarajMenasi/siftwise.git
cd siftwise
[ -f .env ] || echo "SIFTWISE_API_TOKEN=$(openssl rand -hex 32)" > .env
docker compose up -d --wait             # pulls the image, waits until healthy
curl -fsS http://localhost:8000/healthz # {"status":"ok"}

The token in .env is a shared secret you choose; the server and every client use it.

Install the CLI and give it the token:

uv tool install siftwise                # or: pip install siftwise
export SIFTWISE_API_TOKEN=$(sed -n 's/^SIFTWISE_API_TOKEN=//p' .env)
siftwise --version                      # 0.1.0

siftwise: command not found? Run uv tool update-shell once and open a new terminal.

Record a run, change a file, and select:

demo="$(mktemp -d)/shop" && cp -r examples/quickstart "$demo" && cd "$demo"
git init -q && git add -A && git commit -qm "A shop"
uv run pytest -q --junitxml=junit.xml
siftwise record junit.xml --repo demo/shop --branch main --main

echo "# reviewed the bulk discount" >> shop/cart.py && git commit -qam "Touch the cart"
siftwise select --repo demo/shop --base HEAD~1 > selected.sh
cat selected.sh                         # pytest tests/test_cart.py
uv run sh selected.sh

siftwise: python imports: 1 test file(s) import changed modules
siftwise: selective (2 of 4 known tests): 2 of 4 known tests selected

You changed cart.py, and siftwise ran only the tests that import it. 🎉

Step 2: Use it on your own project (locally)

Keep the server running. Now point siftwise at your repo.

2.1 Connect the CLI to the server. Two environment variables are the whole connection. First print your token (in the siftwise folder):

sed -n 's/^SIFTWISE_API_TOKEN=//p' .env

Then, in your project:

cd ~/code/my-app
export SIFTWISE_SERVER=http://localhost:8000     # the default; change it for a remote server
export SIFTWISE_API_TOKEN=<paste the token>

Put both lines in your ~/.bashrc (or ~/.zshrc) so you don't have to repeat them. Never commit the token.

2.2 Record a first full run on main. This gives siftwise history to work with. Pick a repo name (owner/name) and use the same one in every command.

git checkout main

# Python
pytest --junitxml=junit.xml

# Go
gotestsum --junitfile junit.xml ./...

# Jest: use a JUnit reporter that writes file paths (e.g. jest-junit with addFileAttribute="true")

siftwise record junit.xml --repo me/my-app --branch main --main

Want history from the start? If your GitHub Actions runs already upload JUnit XML as artifacts, scripts/backfill.py in the siftwise repo can import past runs (see the README's "Seed history from past CI runs").

2.3 Make a change and select.

git checkout -b fix-discount
# ...edit src/my_app/pricing.py...
git commit -am "Fix discount rounding"

siftwise select --repo me/my-app --base main > selected.sh
cat selected.sh            # only the tests your change can affect
sh selected.sh             # in a uv project: uv run sh selected.sh

--json prints every selected test with the reason it was picked.
An empty selected.sh means no tests are affected.
If you touched something siftwise can't reason about (say pyproject.toml or a CI file), selected.sh runs the full suite. That's by design.
Exit code 3 means "full suite needed, but I don't know your command": just run your normal test command.

2.4 A test failed? Ask whether it's your fault.

Run the failing tests with a JUnit report, then triage it:

pytest tests/test_pricing.py --junitxml=junit.xml    # the tests from selected.sh
siftwise triage junit.xml --repo me/my-app --base main

Add --json for the full evidence.

2.5 Keep feeding it. Record your branch runs too (with --base, so triage can link later failures to this change):

siftwise record junit.xml --repo me/my-app --branch fix-discount --base main

The more runs it has, the better its history: flakiness, "already broken on main", and co-failures all come from recorded runs. In practice, CI does this for you (Step 4).

2.6 Add rules for what imports can't see. Drop a .siftwise.toml in your repo root:

always_run = ["tests/smoke/**", "tests.test_health::*"]  # test file paths or test IDs
generated = ["build/reports/**"]          # files your test runs write (never "changed code")
known_flaky = ["tests/test_net.py"]       # triage: unlinked failures are flaky

[[depends]]                               # dependencies no import graph can see
tests = "test/cli/**"                     # test file paths, or <Go package dir>/<TestName>
on = ["**/*.go", "!**/*_test.go"]         # ! excludes

Globs: ** spans directories; * and ? stay within one folder. Go test IDs contain the import path, so match them with **::TestName.

2.7 Stop the server when you're done: docker compose down in the siftwise folder (down -v also deletes its data).

Step 3: Connect your coding agent (MCP)

siftwise mcp is a standard stdio MCP server with three tools:

Tool	The agent calls it	Returns
`select_tests`	before running tests after a change	the mode, the command to run, a reason per test
`triage_failure`	when tests fail, before fixing them	a verdict per failure, with evidence
`explain`	when a selection or verdict surprises it	one test's recent history and why it would (not) be selected

It includes uncommitted work (the diff runs to your working tree, untracked files too), and it fails open: if the server is down, it tells the agent to run the full suite.

3.1 Install:

uv tool install 'siftwise[mcp]'
uv tool update-shell        # once, if ~/.local/bin isn't on your PATH; then restart the agent
uv tool dir --bin           # prints the folder, if you need the full path below

3.2 Add it to your agent. Open the agent inside your repository: siftwise reads the repo name from your origin remote and the base branch from origin/HEAD (else main). Override with SIFTWISE_REPO / SIFTWISE_BASE_REF if they don't match the name you recorded with.

Claude Code (tested ✅):

claude mcp add siftwise --env SIFTWISE_SERVER=http://localhost:8000 \
    --env SIFTWISE_API_TOKEN=your-token -- siftwise mcp

or a project .mcp.json, which reads the token from your environment so it never lands in git:

{
  "mcpServers": {
    "siftwise": {
      "command": "siftwise",
      "args": ["mcp"],
      "env": {
        "SIFTWISE_SERVER": "http://localhost:8000",
        "SIFTWISE_API_TOKEN": "${SIFTWISE_API_TOKEN}"
      }
    }
  }
}

Codex (~/.codex/config.toml):

[mcp_servers.siftwise]
command = "siftwise"
args = ["mcp"]
env = { SIFTWISE_SERVER = "http://localhost:8000", SIFTWISE_API_TOKEN = "your-token" }

Cursor (~/.cursor/mcp.json for every project, or .cursor/mcp.json in one): the same JSON as above, with the token value written in.

Honest note: siftwise is a standard stdio MCP server, so any MCP client should be able to run it. So far I've tested it with Claude Code. The Codex and Cursor configs follow each tool's MCP format but haven't been tested yet. If you try them, please tell me (or open an issue)!

Agent can't find siftwise? (common on Windows: the agent doesn't see your shell's PATH) Use the full path from uv tool dir --bin as the command. In JSON, double the backslashes: "C:\\Users\\you\\.local\\bin\\siftwise.exe". In TOML, use a single-quoted string.

3.3 Tell the agent when to use it, in CLAUDE.md or AGENTS.md:

Before running tests, call the siftwise `select_tests` tool and run the command it returns.
When tests fail, call `triage_failure` before fixing anything: fix `new_regression` failures,
re-run `flaky` ones, and leave `pre_existing` ones (already failing on main).

What that looks like in practice (an example flow):

You: "Fix the discount rounding bug."
The agent edits pricing.py, calls select_tests → gets pytest tests/test_pricing.py tests/test_cart.py instead of the whole suite.
A test fails → it calls triage_failure → new_regression, linked to the file it just changed → it fixes its own bug.
A different failure is pre_existing (already broken on main) → the agent leaves it alone instead of "fixing" something unrelated.

Less waiting, fewer wrong fixes, and a reason for every decision.

The MCP server only reads. History comes from runs you (or CI) record, so it gets smarter once Step 4 is in place.

Step 4: Connect it to your CI (GitHub Actions)

4.1 Host the server where CI can reach it. GitHub's runners can't see your localhost. The quickstart compose file deliberately binds the API to 127.0.0.1, so on a server you put a TLS reverse proxy in front (the token travels in a request header).

On any Linux VM with Docker:

git clone https://github.com/PramodBasavarajMenasi/siftwise.git && cd siftwise
echo "SIFTWISE_API_TOKEN=$(openssl rand -hex 32)" > .env
docker compose up -d --wait

Point a DNS record (e.g. siftwise.your-domain.com) at the VM, open ports 80 and 443, install Caddy, and use this Caddyfile (Caddy gets the HTTPS certificate for you):

siftwise.your-domain.com {
    reverse_proxy localhost:8000
}

sudo systemctl reload caddy
curl -fsS https://siftwise.your-domain.com/healthz   # {"status":"ok"}

Already have Postgres? Run just the image: docker run -d -p 8000:8000 -e SIFTWISE_API_TOKEN=... -e SIFTWISE_DATABASE_URL=postgresql+psycopg://user:password@db-host:5432/siftwise ghcr.io/pramodbasavarajmenasi/siftwise:0.1.0. It applies database migrations on start.

4.2 Add two repo secrets (Settings → Secrets and variables → Actions):

Secret	Value
`SIFTWISE_URL`	`https://siftwise.your-domain.com`
`SIFTWISE_TOKEN`	the server's `SIFTWISE_API_TOKEN` (from its `.env`)

4.3 Add the workflow. The push: main trigger matters: main-branch runs are the history siftwise learns from.

name: CI
on:
  push:
    branches: [main]      # records main history
  pull_request:           # NEVER pull_request_target: it exposes secrets to fork code
permissions:
  contents: read
  actions: read           # the run's start time
  pull-requests: write    # the PR comment
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0  # full history, so the diff works
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt   # your install step

      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@v0.1.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

      - name: Tests
        env:
          MODE: ${{ steps.siftwise.outputs.mode }}
          COMMAND: ${{ steps.siftwise.outputs.command }}
          JUNIT_DIR: ${{ steps.siftwise.outputs.junit-dir }}
        run: |
          if [ "$MODE" = selective ]; then
            # each pytest command writes its own report to JUNIT_DIR
            if [ -n "$COMMAND" ]; then eval "$COMMAND"; else echo "No tests affected."; fi
          else
            pytest --junitxml="$JUNIT_DIR/full.xml"   # your normal full suite
          fi

      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@v0.1.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

A few rules that save you debugging time:

Pass the command through env: as above, never as ${{ }} inside run:.
record uploads `/junit*.xml** plus everything in junit-dir. Name your report files junit*.xml or set its junit` input.
Go: write JUnit with gotestsum --junitfile. Jest: use a JUnit reporter with file paths.
Java: siftwise gives no command, so run mvn test or ./gradlew test in the full-suite branch. Their reports are named TEST-*.xml, which the default glob misses, so set junit: "**/target/surefire-reports/TEST-*.xml" (Maven) or junit: "**/build/test-results/test/TEST-*.xml" (Gradle) on record.
Pin @v0.1.0 (or a commit SHA) for reproducible CI.
Using a test matrix? Give each leg's record a variant, let only one leg comment (comment: ${{ strategy.job-index == 0 }}), and add one actions/rollup job after all legs. Full example in the README.

4.4 Open a PR. You'll get a sticky comment: what ran, what was skipped and why, and a verdict for every failure.

4.5 Optional extras:

Gate on real regressions: set fail-on: new_regression,unknown on record (single job) or rollup (matrix), give the test step continue-on-error: true, and make that job a required check. With fail-on set, siftwise fails closed: if verdicts are unavailable, the step fails too.
Re-run unknown failures to catch flakes (pytest): rerun-unknown: 2 and rerun-prefix: uv run on record.
Shadow mode (recommended at first): add shadow: 10% to select. On 10% of runs the full suite runs anyway, and the PR comment says what selection would have caught or missed. Check your recall any time:

siftwise report --repo me/my-app --days 30 --by week

My suggested rollout

Run the server and record main for a few days (just the push: main trigger).
Turn on PRs with shadow: always or shadow: 10%, and watch the recall.
Happy with the numbers? Let selection run, and add fail-on if you want gating.

Security and limits (read this before you host it)

One token, full access. SIFTWISE_API_TOKEN can read and write every repo on that server. No per-repo or read-only tokens yet. Use a separate server for repos that shouldn't see each other's history.
Fork PRs get no secrets, so they run the full suite with no comment. That's expected. Keep another required check for them.
Protect .github/workflows/ with CODEOWNERS or a ruleset: a PR that edits workflows can change any gate.
What's stored: test IDs, outcomes, durations, failure messages (truncated to 4 KB), changed paths, branch names, SHAs. Treat the database like your CI logs. PR comments never show raw failure messages.
Always full suite: merge-queue, scheduled and manual runs (no diff base). The Action is tested on Linux runners only.

Troubleshooting

You see	Fix
`siftwise: command not found`	`uv tool update-shell`, then open a new terminal
Every endpoint returns 503	the server has no `SIFTWISE_API_TOKEN`; check its `.env`
`select` fails in CI about a shallow clone	add `fetch-depth: 0` to `actions/checkout`
No PR comment	check `pull-requests: write` permission and the two secrets; fork PRs never get one
Always the full suite	no history yet (record `main` first), or the diff touched a build/CI file; run `select --json` to see the reason
Server restarts with `Can't locate revision ...`	you went back to an older image; use the newer one (with `--build` from a checkout: rebuild after pulling)
Agent can't find `siftwise`	use the full path from `uv tool dir --bin` in its MCP config

Does it actually work?

I didn't want to just claim it works, so I replayed real CI history from open-source projects, with no peeking at the future:

Repo	Failures caught	Test time skipped
rdflib (Python, ~18,000 tests)	2,049 of 2,050; 140 of 141 in selective runs (the one miss: a flaky test)	10.4% on the 10 most recent PRs (from lint-only PRs); 0.6% over 19 main commits
substrate (Go, ~9,000 tests)	2,043 of 2,044; 38 of 39 in selective runs, with two declared rules (26 of 38 without)	5.1% on PRs, 13.9% over 40 main commits
Cirq (Python monorepo, 9 packages)	not measurable (main never failed)	37.9% over 132 merged PRs
kubo (Go, ~4,100 tests, older week-2 code)	19 of 19 (those runs fell back to the full suite)	30.7% over 20 main commits; 0% on recent PRs (all dependency/CI bumps)

Triage: on substrate, new_regression verdicts held up on 319 of 319 labelled failures, and only 5 of 406 labelled regressions were wrongly called flaky (4 likely label noise). On rdflib, 0 regressions were called flaky.

Speed: /select takes about 0.4 s on 18,000 tests.

Where it falls short: savings follow package boundaries. In codebases where everything imports everything (like rdflib's core), safe selection still runs ~99% of the suite. siftwise just runs it, rather than guess. These are replays of history, not live production numbers; shadow mode is how you measure it on your own repo.

All the numbers and methodology: full write-up · eval results.

What's next

v0.2: prove it and widen its reach

Live dogfooding: run siftwise on its own CI and publish live shadow-mode recall numbers.
More runners for rerun-unknown: Go, Jest and Java (currently pytest only).
Other MCP clients: test and document Cursor and Codex.
GitLab CI support, beyond GitHub Actions.

Try it, break it, tell me

This is v0.1.0 (alpha), built by one person in six weeks, and fully open source (Apache-2.0). The fastest way to make it better is you running it on a real repo.

⭐ Star it if the idea clicks: it genuinely helps a solo project get seen.
🐛 Open an issue if it picks the wrong tests. Misses are the most valuable bug reports.
🛠️ Contribute: anything on the v0.2 list (re-run runners, MCP client testing, GitLab CI) is a great place to start. See CONTRIBUTING.md.

Point it at your repo and drop your numbers in the comments. I want to know how much CI time it saves you. 👇

DEV Community: Pramod Basavaraj Menasi

siftwise v0.3: riskiest tests first, time budgets, Jest and Java selection, and live dogfooding numbers

What's new in v0.3

Planned vs shipped

Upgrading from 0.2

Part 1: risk order, riskiest tests first

Does it help? The numbers

The feature that made the numbers worse

Part 2: time budget

Part 3: Jest selection

Step 3.1: let siftwise know each test's file

Step 3.2: the workflow

Part 4: Java selection, Maven and Gradle

Step 4.1: the workflow

The numbers

Part 5: dogfooding, siftwise on its own CI

How it works

The numbers (26 Oct to 9 Nov 2026)

What dogfooding taught me

Part 6: v0.2 follow-ups and fixes

What building v0.3 taught me

Community

Troubleshooting

Known limits

What's next: v0.4

Links

Try it, break it, tell me

siftwise v0.2: run it on GitLab CI and with 5 coding agents (full setup guide)

What's new in v0.2

Before you start

Part 1: connect your coding agent (MCP)

Step 1.1: install the MCP extra

Step 1.2: add it to your client

Step 1.3: tell your agent when to use it

Step 1.4: try it

How siftwise finds your repo

Part 2: GitLab CI

Step 2.1: make your server reachable from GitLab

Step 2.2: create a GitLab token for the comment

Step 2.3: add three CI/CD variables

Step 2.4: add the template to .gitlab-ci.yml

Step 2.5: open a merge request

What keeps it safe

Part 3: flaky-test re-runs for Go, Jest, Maven and Gradle

Part 4: safer PR comments

Part 5: dogfooding, siftwise on its own CI

What testing on the real thing taught me

Troubleshooting

Known limits

What's next: v0.3, smarter selection

Links

Try it, break it, tell me

I built siftwise: run only the tests your change needs (open source, self-hosted)

The idea in one sentence

Why you might care

How it works

Selection signals

Supported languages

Safe fallbacks

Failure triage

Before you start

Step 1: Try it in 5 minutes (quickstart)

Step 2: Use it on your own project (locally)

Step 3: Connect your coding agent (MCP)

Step 4: Connect it to your CI (GitHub Actions)

My suggested rollout

Security and limits (read this before you host it)

Troubleshooting

Does it actually work?

What's next

Links

Try it, break it, tell me

Step 2.4: add the template to `.gitlab-ci.yml`