DEV Community

Cover image for siftwise v0.2: run it on GitLab CI and with 5 coding agents (full setup guide)
Pramod Basavaraj Menasi
Pramod Basavaraj Menasi

Posted on

siftwise v0.2: run it on GitLab CI and with 5 coding agents (full setup guide)

#opensource #devops #testing #gitlab

A week ago I launched siftwise: an open-source, self-hosted tool that runs only the tests your change can affect, and tells you whether a failing test is your fault, already broken on main, or just flaky.

If you missed it, the first post explains the idea and how to set up the server and the GitHub Action.

Today v0.2.0 is out. The theme: use it in more places.

pip install --upgrade siftwise   # 0.2.0
Enter fullscreen mode Exit fullscreen mode

What's new in v0.2

  • 🦊 GitLab CI: the same test selection, failure verdicts and sticky comment as on GitHub, now on merge requests.
  • 🤖 5 coding agents tested: Claude Code, GitHub Copilot, Cursor, Codex CLI and Gemini CLI can ask siftwise which tests to run and why a test failed.
  • 🔁 Flaky-test re-runs for Go, Jest, Maven and Gradle, not just pytest.
  • 🔒 Safer PR comments: a file name in a PR can no longer sneak a link or an @mention into siftwise's comment.
  • 🐶 Dogfooding: siftwise now runs on its own CI, collecting live numbers for v0.3.

Like the first post, this one is the whole guide: by the end you can use every new feature.

Before you start

You need a siftwise server. If you don't have one yet, Step 1 of the first post gets one running locally in 5 minutes with Docker:

git clone https://github.com/PramodBasavarajMenasi/siftwise.git
cd siftwise
[ -f .env ] || echo "SIFTWISE_API_TOKEN=$(openssl rand -hex 32)" > .env
docker compose up -d --wait
curl -fsS http://localhost:8000/healthz   # {"status":"ok"}
Enter fullscreen mode Exit fullscreen mode

Already on 0.1.0? Upgrading is a drop-in, with no database migration:

cd siftwise && git pull                       # the compose file now names the 0.2.0 image
docker compose up -d --wait                   # pulls 0.2.0 and restarts the server
uv tool upgrade siftwise                      # the CLI; or: pip install --upgrade siftwise
Enter fullscreen mode Exit fullscreen mode

All commands are bash; on Windows, use Git Bash or WSL.

Part 1: connect your coding agent (MCP)

siftwise ships an MCP server with three tools your agent can call:

Tool The agent calls it It gets back
select_tests before running tests the test command, with a reason for each test
triage_failure when tests fail, before fixing a verdict per failure: new regression, pre-existing, flaky or unknown
explain when something looks odd one test's history and why it was (not) selected

What it looks like in practice:

  1. You: "Fix the bulk discount bug."
  2. The agent edits cart.py, calls select_tests, and runs pytest tests/test_cart.py instead of the whole suite.
  3. A test fails. It calls triage_failure and gets new_regression, linked to the file it just changed, so it fixes its own bug.
  4. A failure that's pre_existing (already broken on main) gets left alone instead of "fixed".

In v0.1 I had only tested this with Claude Code. For v0.2 I tested it by hand on Windows in five clients, and all three tools worked in each:

Client Config file How siftwise finds your repo
Claude Code claude mcp add (or .mcp.json) the project folder Claude Code tells it
GitHub Copilot (VS Code, agent mode) .vscode/mcp.json the open workspace folder
Cursor ~/.cursor/mcp.json or .cursor/mcp.json asks Cursor which folder is open, nothing to configure
Codex CLI ~/.codex/config.toml the cwd you set
Gemini CLI ~/.gemini/settings.json the cwd you set

Step 1.1: install the MCP extra 

uv tool install 'siftwise[mcp]'   # or: pip install 'siftwise[mcp]'
uv tool update-shell              # once, if `siftwise` isn't found; then restart the agent
uv tool dir --bin                 # prints the folder, if you need the full path below
Enter fullscreen mode Exit fullscreen mode

You need two values for every client: your server address (http://localhost:8000 locally) and the API token from the server's .env.

Windows tip: agents don't always see your shell's PATH. If a client can't start siftwise, use the full path from uv tool dir --bin, e.g. "C:\\Users\\you\\.local\\bin\\siftwise.exe" in JSON, or 'C:\Users\you\.local\bin\siftwise.exe' (single quotes) in TOML.

Step 1.2: add it to your client

Claude Code (one command, run inside your repo):

claude mcp add --env SIFTWISE_SERVER=http://localhost:8000 \
    --env SIFTWISE_API_TOKEN=your-token --transport stdio siftwise -- siftwise mcp
Enter fullscreen mode Exit fullscreen mode

Check with /mcp in a session.

GitHub Copilot in VS Code: create .vscode/mcp.json in your repo. VS Code asks for the token on first start and stores it securely, so this file is safe to commit:

{
  "inputs": [
    { "type": "promptString", "id": "siftwise-token", "description": "siftwise API token", "password": true }
  ],
  "servers": {
    "siftwise": {
      "type": "stdio",
      "command": "siftwise",
      "args": ["mcp"],
      "env": {
        "SIFTWISE_SERVER": "http://localhost:8000",
        "SIFTWISE_API_TOKEN": "${input:siftwise-token}"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Click Start above "siftwise" in the file, then use Copilot Chat in Agent mode.

Cursor: ~/.cursor/mcp.json (all projects). No repo path needed: siftwise asks Cursor which folder you have open.

{
  "mcpServers": {
    "siftwise": {
      "type": "stdio",
      "command": "siftwise",
      "args": ["mcp"],
      "env": {
        "SIFTWISE_SERVER": "http://localhost:8000",
        "SIFTWISE_API_TOKEN": "your-token"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Check in Cursor's settings (search "MCP"): siftwise should show a green dot and 3 tools.

Codex CLI: ~/.codex/config.toml. Set cwd to your repo:

[mcp_servers.siftwise]
command = "siftwise"
args = ["mcp"]
cwd = "/path/to/your/repo"   # Windows: 'C:\Users\you\code\repo'

[mcp_servers.siftwise.env]
SIFTWISE_SERVER = "http://localhost:8000"
SIFTWISE_API_TOKEN = "your-token"
Enter fullscreen mode Exit fullscreen mode

Check with /mcp. If Codex says the model "isn't supported with a ChatGPT account", pick another one with /model; that's Codex's account check, not siftwise.

Gemini CLI: ~/.gemini/settings.json. The token must be in env, because Gemini CLI strips inherited variables with TOKEN in their name:

{
  "mcpServers": {
    "siftwise": {
      "command": "siftwise",
      "args": ["mcp"],
      "cwd": "/path/to/your/repo",
      "env": {
        "SIFTWISE_SERVER": "http://localhost:8000",
        "SIFTWISE_API_TOKEN": "your-token"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Check with /mcp.

Keep tokens out of git. The Cursor, Codex and Gemini files above live in your home folder, outside any repo. Don't put a real token into a config file inside your project.

Windsurf and Cline configs are in docs/mcp-clients.md, but I haven't tested those yet. I've also only tested on Windows so far; reports from macOS and Linux are very welcome.

Step 1.3: tell your agent when to use it

Add this to your CLAUDE.md, AGENTS.md or the client's rules file:

Before running tests, call the siftwise `select_tests` tool and run the command it returns.
When tests fail, call `triage_failure` before fixing anything: fix `new_regression` failures,
re-run `flaky` ones, and leave `pre_existing` ones (already failing on main).
Enter fullscreen mode Exit fullscreen mode

Step 1.4: try it

Make a change in your repo and ask the agent:

Ask Expect
"Which tests should I run for my change? Use siftwise." a test command with a reason per test
"The tests failed. Ask siftwise before fixing anything." a verdict per failing test
"Why does siftwise select <a test>?" the test's history and the link to your change

Your repo needs some recorded history first (a recorded run on main; see Step 2 of the first post), otherwise siftwise safely says "run the full suite".

How siftwise finds your repo

Clients start MCP servers in different folders, so siftwise uses the first of these that works:

  1. a repo_path the agent passes;
  2. SIFTWISE_REPO_PATH in the config;
  3. the project folder Claude Code sets;
  4. the folders the client has open (MCP roots);
  5. the folder the server started in.

If none is a git repo, every tool says so and how to fix it, and select_tests falls back to "run the full suite".

Part 2: GitLab CI

GitLab merge requests now get what GitHub PRs get. Here's the note from a real test MR on gitlab.com, where I broke a discount calculation on purpose:

siftwise note on a GitLab merge request: selective, 2 of 4 tests, test_bulk_discount as a new regression with high confidence

  • Selective: 2 of 4 tests ran.
  • The verdict: the failing test is a new regression, high confidence, linked to the changed file.
  • One sticky note, updated on every push, not a new comment each time.
  • The gate: fail-on failed the pipeline, as configured.

Step 2.1: make your server reachable from GitLab

GitLab's runners can't see your localhost. Run the server somewhere they can reach, behind HTTPS. The first post's Step 4.1 shows a VM with Docker and Caddy. The token travels in a request header, so use https://; siftwise warns if you don't.

Step 2.2: create a GitLab token for the comment

siftwise posts its note through GitLab's API, so it needs a token. Give it the least that works:

  • GitLab.com Free: a fine-grained personal access token, limited to this one project, with Work Item create, read and update (MR comments are part of it), Merge Request read, User read (so siftwise only edits its own note), and an expiry date. Ideally it belongs to a dedicated bot user with the Reporter role.
  • Premium, Ultimate or Self-Managed: a project access token with the Reporter role and the api scope.

GitLab's built-in CI_JOB_TOKEN can't write MR comments, which is why a separate token is needed.

Step 2.3: add three CI/CD variables

In your project: Settings → CI/CD → Variables → Add variable:

Key Value Visibility
SIFTWISE_SERVER https://siftwise.your-domain.com visible is fine
SIFTWISE_TOKEN your server's SIFTWISE_API_TOKEN Masked and hidden
SIFTWISE_GITLAB_TOKEN the token from Step 2.2 Masked and hidden

Protected or not? Protected variables only reach protected branches, so most MR pipelines wouldn't get the tokens (siftwise then runs the full suite and posts nothing). Unprotected (masked and hidden) works for every MR, but anyone who can push a branch could read them in their pipeline. If everyone with Developer access may use siftwise, go unprotected, and run a separate siftwise server per team or trust boundary. The README's protected or unprotected section has the details.

Step 2.4: add the template to .gitlab-ci.yml

Include the template by its release tag and a hash, so the code that runs with your tokens can't change under you. Get the hash once:

curl -sL https://raw.githubusercontent.com/PramodBasavarajMenasi/siftwise/v0.2.0/gitlab/siftwise.gitlab-ci.yml \
  | openssl dgst -sha256 -binary | openssl base64 -A
Enter fullscreen mode Exit fullscreen mode

Then:

include:
  - remote: https://raw.githubusercontent.com/PramodBasavarajMenasi/siftwise/v0.2.0/gitlab/siftwise.gitlab-ci.yml
    integrity: sha256-PASTE-THE-HASH

workflow:  # merge request pipelines for open MRs, branch pipelines otherwise
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

siftwise-select:
  extends: .siftwise-select

test:
  extends: .siftwise-record
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - |
      if [ "$SIFTWISE_MODE" = selective ]; then
        if [ -s "$SIFTWISE_COMMAND_FILE" ]; then sh "$SIFTWISE_COMMAND_FILE"; else echo "No tests affected."; fi
      else
        pytest --junitxml=.siftwise-ci/junit/full.xml   # your full suite
      fi
  artifacts:
    reports:
      junit: .siftwise-ci/junit/*.xml

siftwise-rollup:
  extends: .siftwise-rollup
  variables:
    SIFTWISE_FAIL_ON: new_regression,unknown   # optional: fail the pipeline on real regressions
Enter fullscreen mode Exit fullscreen mode

What each job does:

  • siftwise-select picks the tests and passes the result to your test job.
  • Your test job runs them, and its after_script uploads the results.
  • siftwise-rollup gives each failure a verdict, updates the MR note and applies fail-on.

GIT_DEPTH: "0" (full history) is already set by the template, since GitLab clones only 20 commits by default.

Step 2.5: open a merge request

  1. Push .gitlab-ci.yml to your default branch first. That pipeline records main's history.
  2. Make a change on a branch and open an MR.
  3. The MR shows the siftwise note, and .siftwise-ci/ in the job artifacts has the full reports.

Using parallel: matrix:? Give each leg a SIFTWISE_VARIANT; the README's GitLab section shows it.

Just want to try it first? examples/gitlab is a tiny project that runs its own siftwise server inside the pipeline, so you need no server at all. Push it to a new private gitlab.com project, add SIFTWISE_GITLAB_TOKEN, and open an MR that breaks shop/cart.py. That's exactly the test behind the screenshot above.

What keeps it safe

  • Tokens only come from environment variables, never command arguments. They're redacted from every log line, file and note, and tests check that.
  • siftwise only edits a note its own token's user wrote, so nobody can hijack it with a fake one.
  • Test names come from the MR's code, so they're shown as code: they can't add links, @mentions or GitLab quick actions.
  • Fork MRs get no tokens: they run the full suite, with no note. Never run untrusted forks' pipelines in your project.
  • An MR can't mark its own failures flaky: known_flaky is read from the MR's base commit.

Part 3: flaky-test re-runs for Go, Jest, Maven and Gradle

When a failure's verdict is unknown, siftwise can re-run just that test. If it passes, the failure is marked flaky (high confidence), so nobody wastes an hour chasing it. In v0.1 this only worked for pytest. Now each failing test gets its own runner:

Language What siftwise runs You need
Python pytest <node IDs> nothing extra
Go go test -count=1 -run with the failing test names, per package Go; gotestsum optional
Jest jest --runTestsByPath <file> -t with the full test name jest-junit with addFileAttribute: "true"
Java (Maven) mvn -Dtest=Class#method test (./mvnw if present) nothing extra
Java (Gradle) gradle test --tests Class.method (./gradlew if present) nothing extra

Tested with JDK 17 and 21, Maven 3.9 and 3.10, and Gradle 8.10 and 9.8; siftwise uses your project's own mvn/gradle (or ./mvnw/./gradlew), so other versions should work too.

Turn it on in the GitHub Action's record step (or SIFTWISE_RERUN_UNKNOWN on GitLab):

- if: always()
  uses: PramodBasavarajMenasi/siftwise/actions/record@v0.2.0
  with:
    server-url: ${{ secrets.SIFTWISE_URL }}
    token: ${{ secrets.SIFTWISE_TOKEN }}
    rerun-unknown: 2      # re-run unknown failures up to twice
    rerun-prefix: npx     # how Jest (or pytest, e.g. "uv run") runs in your job
Enter fullscreen mode Exit fullscreen mode

rerun-prefix only applies to pytest and Jest. Go, Maven and Gradle pick their own tool, so a mixed-language repo doesn't break.

Limits: at most 20 tests within 10 minutes; re-runs never turn a red job green; and for Java and Jest the re-run must name tests the way your CI run did (JUnit 5 @DisplayName doesn't match yet). Maven and Gradle re-runs put your CI run's original reports back afterwards, so nothing downstream gets the wrong files.

Part 4: safer PR comments

While building the GitLab note, I noticed the GitHub PR comment didn't escape everything that comes from the PR. A file named like a Markdown link would show up as a real link in siftwise's comment, and a path like pkgs/@acme/... would @mention someone.

Now everything from the PR (test IDs, file paths, variant names) is shown as code, on both GitHub and GitLab. Nothing to configure: upgrade the Action pin to @v0.2.0.

Part 5: dogfooding, siftwise on its own CI

siftwise now runs on its own CI in shadow mode: the full suite still runs every time, and siftwise records what it would have selected and whether that would have caught each failure.

It needs no hosted server: a siftwise server starts inside each CI job, and the history lives in the GitHub Actions cache between runs.

My earlier numbers (2,049 of 2,050 failures caught on rdflib, for example) came from replaying other projects' old CI history. These will be live numbers from real changes, and I'll publish them with v0.3.

What testing on the real thing taught me

  • Mocks aren't enough. Every GitLab API call was mocked in my tests, and they all passed. On real gitlab.com, the pipeline's own job token got 404 Project Not Found downloading main's artifacts, so the example no longer depends on that.
  • Re-running a test can delete other reports. A Gradle --tests re-run removed the results of every class it didn't re-run. siftwise now backs up and restores them around every Maven and Gradle re-run.
  • Every MCP client is a little different: where it starts the server, which environment variables it passes, how it reads JSON schemas. Codex turned optional parameters written as anyOf [type, null] into plain strings, so every schema got simpler. Testing in the real clients still found three bugs the automated tests had missed.

Troubleshooting

You see Fix
The agent can't start siftwise use the full path from uv tool dir --bin in the config
A tool says "not inside a git repository" open the repo folder itself in the client, or set SIFTWISE_REPO_PATH
select_tests always says "full suite" no history yet: record a run on main first
Codex: "model not supported" pick another model with /model
Gemini: siftwise says the token is missing put SIFTWISE_API_TOKEN in the server's env block
GitLab: no MR note, HTTP 403 in the log the token lacks the permissions in Step 2.2
GitLab: full suite on every MR protected variables don't reach MR branches (see Step 2.3), or the clone is shallow

Known limits

  • MCP is tested on Windows only; Windsurf and Cline are untested.
  • GitLab is tested on gitlab.com Free; Self-Managed and Premium are untested.
  • Java and Jest re-runs need matching test names (no JUnit 5 @DisplayName yet).
  • Full details are in the CHANGELOG.

What's next: v0.3, smarter selection

  • Test prioritization: run the tests most likely to fail first.
  • Time-budget mode: "the best tests that fit in 5 minutes".
  • A real JS/TS import graph for Jest.
  • Test selection for Java, not just re-runs.
  • Live recall numbers from dogfooding.

Links

Try it, break it, tell me

This is still one person building in the open. If you try siftwise with GitLab or with your coding agent, tell me how it went in the comments, especially on macOS or Linux, or with Windsurf or Cline. If it picks the wrong tests, please open an issue: misses are the most useful bug reports. And a ⭐ really helps a solo project get seen. 👇

Top comments (0)