siftwise v0.3: riskiest tests first, time budgets, Jest and Java selection, and live dogfooding numbers

siftwise is an open-source, self-hosted tool that runs only the tests your change can affect, and tells you whether a failing test is your fault, already broken on main, or just flaky.

New here? Post 1 explains the idea and sets up the server and the GitHub Action. Post 2 adds GitLab CI and five coding agents over MCP.

Today v0.3.0 is out. The theme: smarter selection.

pip install --upgrade siftwise   # 0.3.0

What's new in v0.3

• 🎯 Risk order: the tests most likely to fail run first, so a failure shows up early.
• ⏱️ Time budget: "the riskiest tests that fit in 5 minutes", for fast feedback.
• 🟨 Jest selection: a JS/TS import graph, with no Node.js needed.
• ☕ Java selection for Maven and Gradle, including multi-module builds.
• 🐶 Dogfooding: siftwise now runs on its own CI on every PR and push, with live numbers.
• 🐛 Follow-ups and fixes from testing v0.2 on real repos, three of which made siftwise silently miss tests.
• 🙌 First outside contributors: two of them, the same morning I opened my first good first issue.

Like the earlier posts, this one is the whole guide, with the real numbers, including the ones that didn't go my way.

---

Planned vs shipped

At the end of post 2 I promised five things for v0.3. Here's what happened:

Promised in v0.2	Shipped in v0.3
Run the tests most likely to fail first	✅ Risk order (#42), plus import depth on top (#48)
Time-budget mode	✅ --budget 5m everywhere: CLI, Action, GitLab, MCP (#43)
A real JS/TS import graph for Jest	✅ (#44), plus computed imports like import.meta.glob (#51)
Test selection for Java	✅ Maven and Gradle (#45), plus module build-file scoping, one command per build and a first Gradle replay (#54)
Live recall numbers from dogfooding	⚠️ Live time-saved numbers, yes. Live recall, not yet: none of siftwise's own runs has failed so far, so there's nothing to catch or miss. More in Part 5.

Not planned, but shipped too:

• the v0.2 follow-ups (#46): Jest without addFileAttribute, JUnit 5 @DisplayName re-runs, Go workspaces and nested modules, and the GitLab Tests tab;
• go list no longer edits go.mod (#52);
• six bug fixes, three of which made siftwise silently miss tests (Part 6);
• new evals: time to first failure, recall within a budget, and a Gradle replay.

---

Upgrading from 0.2

v0.3 has two database migrations (0011 and 0012; both only add columns). The server runs them on start, so with Docker it's the usual:

cd siftwise && git pull                       # the compose file now names the 0.3.0 image
docker compose up -d --wait                   # pulls 0.3.0, migrates, restarts
uv tool upgrade siftwise                      # the CLI; or: pip install --upgrade siftwise

Then move your Action pins to @v0.3.0, and the GitLab template to v0.3.0 with its new integrity hash.

Update the server and the clients together. A new CLI talking to a 0.2 server still works: it sends the request again without the new fields and gets 0.2's selection. The exception is a time budget, which can't be dropped silently: the Action, GitLab and MCP fall back to the full suite with a warning, and siftwise select --budget exits with an error. An old CLI works with a new server unchanged.

---

Part 1: risk order, riskiest tests first

In v0.2, siftwise picked the right tests but ran them in path order. If the one that fails is last, you still wait for everything before it.

Now every selected test gets a score, a rank and a reason, and the command runs the riskiest first. The score is a fixed, documented formula, with no machine learning:

score = link + 3 × recency + 2 × failure rate   (recency and rate discounted if the test is flaky)
        + 2 if the test is new

link     = 6 if the change reaches the test directly, 3 through other files, 0 by history only
recency  = halves every 7 days since the test last failed

So a test whose file imports the code you changed, and that failed last week, goes to the front. Ties go to the faster test.

Nothing to configure: it's on by default. siftwise select --json shows each test's score, rank and rank_reason; the PR comment and GitLab note name the three riskiest tests. To keep the old order: --order default, the Action's order: default, SIFTWISE_ORDER: default on GitLab, or order: "default" over MCP.

The order never changes which tests run, only when. How much it's respected depends on the runner:

Runner	What the order guarantees
pytest	arguments run in the order given (unless a plugin like pytest-xdist or pytest-randomly reorders them)
Go	packages run one after another, riskiest first
Jest	a hint: Jest runs files in parallel and may reorder them
Maven	modules run in Maven's reactor (dependency) order

Does it help? The numbers

I replayed real CI history, evaluating each run using only the history from before it. The number is how much of a selective run passes before the first failure shows, as a median (lower is better):

Repo	Default order	Risk order
RDFLib/rdflib (Python), 17 runs	37.3% of the time	1.4%
agent-substrate/substrate (Go), 19 runs	42.0%	25.6%

On rdflib, the failure typically shows in the first 1.4% of the run instead of after a third of it. Recall didn't change: the same tests ran, in a different order.

I also tried a "cost-aware" order (risk per second of estimated test time). It wasn't clearly better on both repos and was worse by test count, so it didn't ship.

The feature that made the numbers worse

v0.3 also adds import depth: a test file that imports the changed code itself counts as a direct link (6), above one that reaches it through other files (3).

My first version did this for every language, and substrate got worse: the median went from 23.3% to 34.4% (by test count), with the first failure coming later in 6 runs and earlier in only 1.

Going failure by failure showed why. Go's graph is per package, and in Go most dependents import the changed package directly: 83% of the import-linked packages were at depth 1. So "direct" lifted almost every package above the e2e suites that a [[depends]] rule selects, and those were the ones failing.

The fix: Go packages aren't promoted by depth; Python, Jest and Java test files are. substrate is back to exactly its earlier numbers. I chose that rule because a test file's imports say much more precisely what a test uses than a package's do, not because it scored best on two repos. Honestly, import depth didn't move rdflib's medians either (only the mean improved a little), so I'm not claiming a speed-up from it yet.

---

Part 2: time budget

Sometimes even the selection is too big for the feedback you want right now. Set a budget, and siftwise runs the riskiest tests whose estimated time fits:

siftwise select --repo acme/shop --base origin/main --budget 5m   # 90s, 5m, 1h30m

Where	Setting
GitHub Action	budget: 5m (outputs budget-limited, budget-left-out)
GitLab	SIFTWISE_BUDGET: 5m
MCP select_tests	budget argument
API	budget_seconds

It's off by default. Two things matter before you use it:

• A budget is an estimate, not a cap. siftwise uses each test's mean duration from history, so a run can take longer. Keep a real CI timeout if you need a hard limit.
• A budget-limited run is not a complete check, and siftwise says so plainly: the PR comment shows "budget-limited (estimated 4m 50s of a 5m target): N tests left out". Run the full selection before merging.

In shadow mode, a failure the budget left out is counted as a miss with the cause budget, so you can measure what a budget costs you on your own repo.

How much a budget catches, on the same replays: runs whose failure is still caught at a budget of 10%, 25% and 50% of each run's time:

Repo	10%	25%	50%	No budget
rdflib	22 of 31	24 of 31	28 of 31	30 of 31
substrate	34 of 72	43 of 72	54 of 72	71 of 72

On rdflib, a 10% budget still catches most failures. On substrate, it catches under half. So a budget is great for a quick first signal, and not a replacement for the full selection.

---

Part 3: Jest selection

Until now, siftwise matched Jest tests only by file name (foo.ts → foo.test.ts), so a change most tests reach through imports ran the whole suite. Now siftwise reads your JS/TS imports and also runs every test file that imports the changed file, at any depth.

What it understands, without running Node:

• relative imports, index files and TypeScript's .js → .ts;
• tsconfig/jsconfig paths and baseUrl (with extends);
• npm, yarn, pnpm and lerna workspace packages;
• a JSON Jest moduleNameMapper;
• JSON and other files a test imports;
• computed imports with a fixed folder: import(`./locales/${lang}.json`), require('./plugins/' + name), Vite's import.meta.glob and webpack's require.context count as imports of every file they could load (#51).

When it can't see, it plays safe: a test that reaches an import it can't bound (like require(somePath)) runs for every JS change, and changes to tsconfig, Jest, Babel or SWC configs, root manifests or lockfiles run the full suite.

New test files run first, by path, once siftwise can tell where Jest expects them. On jestjs/jest (2,272 JS/TS files), the graph builds in about 2 seconds.

Step 3.1: let siftwise know each test's file

siftwise learns which file each test lives in from the JUnit report. Add jest-junit, ideally with addFileAttribute:

npm install --save-dev jest-junit

{
  "jest": {
    "reporters": ["default", ["jest-junit", { "addFileAttribute": "true" }]]
  }
}

New in v0.3: without addFileAttribute, record searches the checkout for the one test file whose string literals match the suite's test names, and adds it to the report. That works for most files, but titles built from variables can't be matched, so setting the attribute is still the reliable option.

Step 3.2: the workflow

      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci

      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@v0.3.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

      - name: Tests
        env:
          MODE: ${{ steps.siftwise.outputs.mode }}
          COMMAND: ${{ steps.siftwise.outputs.command }}
          JEST_JUNIT_OUTPUT_DIR: ${{ steps.siftwise.outputs.junit-dir }}
        run: |
          export PATH="$PWD/node_modules/.bin:$PATH"   # the command starts with `jest`
          if [ "$MODE" = selective ]; then
            if [ -n "$COMMAND" ]; then eval "$COMMAND"; else echo "No tests affected."; fi
          else
            jest   # your normal full suite
          fi

      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@v0.3.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

JEST_JUNIT_OUTPUT_DIR puts the report where record looks for it. As always, pass the command through env:, never as ${{ }} inside run:.

An honest gap: there's no recall number for Jest yet. Replays need old CI runs that uploaded JUnit with file paths, and no public Jest repo I found does that. I validated the graph by hand on jestjs/jest. If you run siftwise in shadow mode on a JS repo, I'd love your numbers.

---

Part 4: Java selection, Maven and Gradle

In v0.2, Java tests were recorded and re-run but always ran in full. Now siftwise reads your build without running Maven, Gradle or a JVM:

• the modules from pom.xml or settings.gradle[.kts];
• a static graph of which test classes use which classes (imports, same-package names, fully-qualified names, Class.forName strings), across modules;
• name matches: changing Foo.java selects FooTest, FooTests, TestFoo and FooIT;
• a module's src/test/resources runs that module's tests.

One command per build where that's safe:

./mvnw -B -pl core,api -am test -Dtest=CartTest,MoneyTest            # Maven
./gradlew :core:test --tests CartTest :api:test --tests MoneyTest   # Gradle

Build files are scoped too (#54): a change to a module's own pom.xml or build.gradle[.kts] runs the tests of that module and of every module that depends on it. The root build file, settings.gradle, version catalogs, wrappers, parent and BOM POMs still run the full suite, because they can affect everything.

Where it can't see, the full suite runs: Kotlin, Groovy or Scala sources, custom source folders, code-generating annotation processors, or a build it can't read. Reflection (Spring scanning, ServiceLoader) needs a [[depends]] rule.

Step 4.1: the workflow

The only difference from Python is where the reports live, so point record at them:

      - uses: actions/setup-java@v6
        with:
          distribution: temurin
          java-version: 21

      - id: siftwise
        uses: PramodBasavarajMenasi/siftwise/actions/select@v0.3.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}

      - name: Tests
        env:
          MODE: ${{ steps.siftwise.outputs.mode }}
          COMMAND: ${{ steps.siftwise.outputs.command }}
        run: |
          if [ "$MODE" = selective ]; then
            if [ -n "$COMMAND" ]; then eval "$COMMAND"; else echo "No tests affected."; fi
          else
            ./mvnw -B test   # or: ./gradlew test
          fi

      - if: always()
        uses: PramodBasavarajMenasi/siftwise/actions/record@v0.3.0
        with:
          server-url: ${{ secrets.SIFTWISE_URL }}
          token: ${{ secrets.SIFTWISE_TOKEN }}
          junit: "**/target/surefire-reports/TEST-*.xml **/build/test-results/test/*.xml"

The numbers

• dropwizard/dropwizard (Maven, 42 modules), 148 PR runs: 74 of 74 failures caught. In the 22 selective runs, siftwise skipped a median 90.6% of the estimated test time (26% to 93%).
• The catch: most of dropwizard's PRs are dependency bumps to its BOM, which correctly run everything. So over all 148 runs, the time saved is much smaller: 11.9%. Module build-file scoping moved 5 more runs from full to selective. A repo with more code changes and fewer bumps would save more.
• naver/fixture-monkey (Gradle): 3 of 3 failures caught. That's a small history, so treat it as a first check, not proof.

The dropwizard replay also found a real bug: Surefire writes a class-level result with an empty test name for a JUnit 5 @Nested class, and siftwise rejected the whole upload for it. Every Windows leg of dropwizard's CI history (781 uploads) was rejected before the fix.

---

Part 5: dogfooding, siftwise on its own CI

This was the last promise from post 2, and the one I care most about. Replays of other projects' history are useful, but they're replays. Dogfooding is siftwise running live on real changes, as they happen.

How it works

Every siftwise PR and push to main runs a Dogfood workflow in shadow mode:

1. A siftwise server starts inside the CI job, on the runner's own Postgres, with a random per-job token. There's no hosted server and nothing to pay for.
2. History comes from the GitHub Actions cache. The job restores the database from main's latest cache entry, and saves a new one at the end.
3. siftwise stores what it would have selected, then the full suite runs anyway.
4. It compares: for every failure, would the selection have caught it? How much time would it have saved?
5. Each run uploads a small summary file. scripts/dogfood_report.py adds them all up.

It gates nothing: no fail-on, no PR comment, and every siftwise step fails open. So it can never block my own CI.

The safety part: a PR's cache entry is scoped to that PR, so a PR (including one from a fork) can read main's history but can never write into it. Only main's own runs build the history. For the numbers, I also leave out the fork PRs' runs, because code from the fork wrote their summaries; each of those changes still counts once, as its push to main.

The numbers (26 Oct to 9 Nov 2026)

Workflow runs	36 (17 PR runs, 19 pushes to main)
Shadow runs counted	32 (the first 2 had no history yet; the 2 fork runs are left out)
Would have been selective	9 of 32
Tests selected in those 9	a median 39% of the known tests
Time saved	14.7% of 23.0 minutes over the 32 runs
Failures	none: every run's suite passed, so recall can't be measured yet
Weekly time saved	21%, 7% and 34%

Read these as early numbers. A merged PR usually counts twice (once as its PR run, once as its push to main), so the 9 selective runs are really about 6 distinct changes.

What dogfooding taught me

• Why 23 of 32 ran in full: 20 changed a build or config file (the Dockerfile and docker-compose.yml in 6 runs each, workflows, and my test fixtures' own build.gradle.kts, tsconfig.json and jest.config.ts), and 3 changed a file no test reaches (like glama.json).
• The fixture finding is the most useful one. siftwise's tests include tiny sample Gradle and TypeScript projects. Their build files are test data, but siftwise treats them like real build files and runs everything. Scoping them safely is the first item for v0.4. I'd never have found this from replays.
• Each run now records the risk rank of every failure, so once a dogfood run does fail, the report shows not just "caught or missed" but how early risk order would have surfaced it.

You can do the same on your repo: the Action's shadow input (always, or a sample like 10%) runs the full suite and reports what siftwise would have caught or missed, before you trust it with anything. Details: Shadow mode in the README.

---

Part 6: v0.2 follow-ups and fixes

Things I found while testing v0.2 on real projects (#46):

• Jest without addFileAttribute: selection and re-runs now work without it (Part 3).
• JUnit 5 @DisplayName and parameterized test names map back to their methods, so Gradle and phrased-Maven re-runs work.
• Go workspaces (go.work) and nested modules work for selection, commands and re-runs.
• The GitLab template now declares its JUnit reports, which the pipeline's Tests tab reads.

Fixes, three of which (🔴) made siftwise silently miss tests:

• 🔴 Nested Go modules: a nested module's dependents were missed, and its tests ran from the root, where Go can't build them. Now they run as go test -C <module> ./pkg.
• 🔴 Missed JS edges: import('./a/' + x), import.meta.glob and require.context created no edges, so changing a file they load didn't select the tests that load it.
• go list could edit go.mod when GOFLAGS had -mod=mod. It now runs read-only (#52). On Go 1.15 or older, set GOFLAGS=-mod=readonly yourself; making siftwise do it is an open good first issue (#67).
• Java uploads rejected for a @Nested class's result (the dropwizard bug above).
• 🔴 Same-named Java test classes in two modules: a change to one module's test resources ran only the other module's copy.
• A deleted Jest test file was still named in the jest command, which then failed with "No tests found".

---

What building v0.3 taught me

• Measure the features you're sure about. Import depth was the "obviously better" change, and it made one repo worse. Without the replay, I'd have shipped it.
• Your dependencies change under you. On release day, a new Typer version started requiring pydantic, which broke the check that keeps the light CLI install small. siftwise's code didn't change; the light install grew from 9 MB to 18 MB, and the check now verifies that siftwise itself never imports server code.
• good first issue works. I labelled a one-line bug, and by the same morning two people had sent fixes. One fixed it; the other added the test that stops it from coming back. Both got merged.

Community

• Thanks to @bfs2021 (#62) and @mursalhayat1 (#63), siftwise's first outside contributors.
• siftwise's MCP server is listed in the HOL Guard MCP server catalog (opt-in, after a review of its code and disclosures) and on Glama.

Troubleshooting

You see	Fix
Jest tests always run in full	add addFileAttribute: "true" to jest-junit (Step 3.1)
jest: command not found	put node_modules/.bin on PATH, as in Step 3.2
Java tests never selected	record didn't find the reports: set its junit input (Step 4.1)
Every Java PR runs the full suite	it changed a root or parent build file, which runs everything by design
--budget fails with a server error	the server is still on 0.2: upgrade it
The order looks wrong in Jest	Jest reorders files across its workers; the order is a hint there

Known limits

• Jest: no recall number yet; jest.config.js can't be read without Node (tsconfig and relative paths still resolve); a custom testMatch isn't read.
• Java: Kotlin, Groovy and Scala tests fall back to the full suite; Maven runs modules in reactor order, not risk order.
• Budget: an estimate, not a cap.
• Dogfooding: 32 runs and no failures yet, so no live recall.
• MCP is tested on Windows only; GitLab on gitlab.com Free only.
• Everything else: the CHANGELOG.

What's next: v0.4

• Safe scoping for build files inside test fixtures (the dogfooding finding).
• -mod=readonly for older Go versions.
• More recall numbers: a Jest replay, more Gradle history, and the first live recall from dogfooding.
• More good first issues. If you'd like to contribute, start here.

Links

• 💻 GitHub: https://github.com/PramodBasavarajMenasi/siftwise
• 📦 PyPI: https://pypi.org/project/siftwise/
• 📝 Release notes: https://github.com/PramodBasavarajMenasi/siftwise/releases/tag/v0.3.0
• 📊 All eval numbers and method: https://github.com/PramodBasavarajMenasi/siftwise/blob/main/docs/eval-results.md
• 📖 Post 1 (server and GitHub Action): https://dev.to/pm_22/i-built-siftwise-run-only-the-tests-your-change-needs-open-source-self-hosted-51jf
• 📖 Post 2 (GitLab CI and coding agents): https://dev.to/pm_22/siftwise-v02-run-it-on-gitlab-ci-and-with-5-coding-agents-full-setup-guide-o2f

Try it, break it, tell me

If you have a Jest or Java repo, try siftwise in shadow mode and tell me what it would have caught or missed: that's exactly the data v0.3 is missing. Misses make the best bug reports. And a ⭐ helps a small project get seen. 👇