Building a test management tool where AI agents are first-class citizens, not an afterthought

Miguel Angel Alonso Pérez — Wed, 28 Oct 2026 14:27:20 +0000

TestPulse 1.0.0 just shipped. It's an open-source (AGPLv3) test management platform — projects, versioned test cases, suites, runs, reporting, a Jira connector — but the part worth writing about is how it treats AI agents.

The problem with "API access" as an AI story

Most test management tools have a REST API. Some have a decent one. But "has an API" and "is designed for an AI agent to operate it safely" are different bars. An agent acting on your test suite needs:

Its own identity, distinct from a shared service account, so actions are attributable
Permissions scoped the same way a human's are — not "all access because it has an API key"
A way to retry a failed call without double-creating data
Output it can actually parse and act on — keys, not opaque internal IDs

None of that is exotic, but almost nothing ships with it by default. So I built it in from the start.

What's actually in the MCP server

TestPulse exposes an MCP server at /mcp, inside the same binary as the REST API — not a separate service, not a plugin. It's authenticated exactly the way REST is: a Bearer token, scoped to one project and one role, or unscoped and acting at its creator's current permissions, resolved fresh on every call.

Fourteen tools, deliberately coarse-grained — not a CRUD mirror of every REST endpoint, and not a handful of do-everything workflow mega-tools:

upsert_test_case, upsert_step_group, upsert_test_suite, update_suite_membership,
bulk_add_cases, create_test_run, record_result, update_run_item_status,
search, get_details, get_run_progress, list_run_items,
list_unit_test_results, list_projects

Deletes are REST-only, deliberately — no delete tool exists.

Three of those tools (upsert_test_case's create path, bulk_add_cases, create_test_run) accept an optional idempotencyKey. If a call times out and you retry with the same key, the server returns the original result instead of creating a duplicate — keys are scoped per caller and tool, expire after 24 hours, and only a successful call gets cached, so retrying a failed (validation/permission) call is always safe.

The two-call demo

The scenario I kept coming back to while designing this: "Read requirements.md and create a test suite in TestPulse."

An agent connected via MCP does this in two calls:

upsert_test_suite — create the suite
bulk_add_cases — populate it, atomically. One malformed case fails the whole batch; nothing gets half-created.

That atomicity matters more than it sounds like it should — the alternative (loop over a create-case call per case) leaves you debugging a partially-populated suite when call #14 of 20 fails on a schema issue.

Everything else, briefly

Core test management: projects, versioned test cases (every edit creates a new version, not an overwrite), suites, runs with an open/closed lifecycle, JUnit XML import with native plugins for pytest, Maven, Gradle, Jest, Go, .NET, GoogleTest, Jasmine, Mocha, and Vitest.
Identity: local auth with sessions, scoped API tokens, SSO/OIDC, project-scoped RBAC.
Reporting: pass-rate trends, flaky-case detection.
Migration: structural importers from TestRail, Qase, Zephyr Scale, Kiwi TCMS, Xray, and Tuskr.
Deployment: one Go binary, Postgres, multi-arch distroless images, docker compose up to evaluate, a Helm chart for production.

Try it

git clone https://github.com/Barayo/TestPulse
cd TestPulse
docker compose -f deploy/compose/docker-compose.yml up --build -d

That gets you a running instance with a seeded Sample Project — every MCP tool above has real data to operate on immediately, no setup required.

Repo: https://github.com/Barayo/TestPulse
Docs: https://github.com/Barayo/TestPulse/blob/main/docs/mcp-server.md