The blind spot in AI-driven test automation: the agent can't see your existing tests

Karthik Kalaiyarasu — Mon, 27 Jul 2026 21:43:12 +0000

TestAtlas is a zero-config CLI that maps your .NET test automation into one queryable SQLite file — no build, no network, no AI.

Ask a coding agent to "add a test for the checkout discount flow" and watch what happens. It writes a brand-new step for "the cart has two items" — even though three nearly identical steps already exist. It creates a fresh CheckoutApiClient next to the CartApiClient that already does the job. It drops the new page object in the wrong project. The test passes. The suite gets a little worse.

This isn't the model being careless. It's the model being blind.

The blind spot

An AI agent working in your repository sees files — the ones it opens, and whatever fragments it can pull into its context window. What it cannot see is the shape of your test automation: which step definitions already exist, which are bound and which are dead, which page objects wrap which screens, which API client owns which endpoint, and which project anything is supposed to live in.

Humans have the same blind spot; we just hit it more slowly. Anyone who has joined a mature test-automation suite knows the feeling of "surely we already have a step for this" — followed by twenty minutes of grepping to find out.

For an agent the cost is worse, because it acts on that blindness immediately: it duplicates, reinvents, and misplaces at machine speed. The suite doesn't fail — it bloats. Duplicate steps, orphaned helpers, and drifting conventions accumulate until the automation is harder to change than the product it tests.

The missing piece isn't a smarter model. It's a map.

What TestAtlas does

TestAtlas reads your solution once and writes a single SQLite file — codemap.db — that captures the whole structure of your test automation:

Projects and their dependency edges
Gherkin features, scenarios, and steps
Step definitions and their bindings — bound, unbound, or ambiguous
Page objects, API clients, helpers, and test classes
The call and usage edges that connect them all

That file is the map. Everything else — search, impact analysis, the HTML report, the dependency graph, the MCP server — is just a different question asked against it.

No build. No network. No AI.

The three things TestAtlas deliberately does not do are the whole point.

No build. It's a syntax-only pass over your source. It never asks Roslyn for a full compilation or a semantic model, so a solution whose NuGet packages have never been restored still maps cleanly. Gherkin comes straight from the .feature files; step definitions come from the [Given]/[When]/[Then] attributes in your code.

No network. Everything runs locally against your files. Nothing is uploaded.

No AI. The map is produced by static analysis, so it is deterministic and reproducible — the same input yields the same map, every time. That is precisely what makes it useful to an AI agent: the map is the fixed, trustworthy ground truth the model can stand on, rather than one more probabilistic guess.

How it works

TestAtlas parses the solution structure, the .feature files, and the C# source, then resolves the relationships between them and writes the result into a documented, versioned SQLite schema. Because the schema is a public contract, anything downstream — the report, the graph, the MCP server, or your own SQL — reads from a stable shape.

Being honest about scope: because the analysis is syntactic, it captures what's declared in source — attributes, class shapes, references, .feature bindings. It targets Reqnroll/SpecFlow-style BDD suites and deliberately skips generated code-behind. It's a map of your source, not a runtime trace — and for the "does this already exist / what does this touch" questions, source is exactly the right altitude.

What you can ask it

Once the map exists, four questions get easy:

Search — FTS5 full-text search over step definitions and scenarios: does a step for this already exist?
Impact — the blast radius of a change: which scenarios are affected if you touch a class, method, step, or endpoint.
Report — a self-contained HTML drill-down of the whole map.
Map — a self-contained project dependency graph.

Install it as a .NET global tool and point it at a solution:

testatlas index path/to/YourSolution.sln
testatlas search "add a product to the cart" --steps
testatlas impact --class CartApiClient

The answers are concrete. Ask about the existing step, and you get the exact binding:

> testatlas search "the shop API is available" --steps
step definitions matching 'the shop API is available': 1
  [Given] the shop API is available  (CatalogApiSteps, CatalogApiSteps.cs:19)

> testatlas impact --class ProductsApiClient
Impact of class ProductsApiClient (api_client)
Affected: 2 step definition(s) · 3 scenario(s) · 2 feature(s)

Catalog API  (SampleShop.Tests.Api/Features/Catalog.feature)
  Add a product to the cart   via "the shop API is available"
  List available products     via "a request for the product list is made", "the shop API is available"
Identity API  (SampleShop.Tests.Api/Features/Identity.feature)
  Sign in and read profile    via "the shop API is available"

Giving the map to an AI agent (MCP)

This is where the blind spot actually closes. TestAtlas ships an MCP server — testatlas-mcp — that serves the map to any MCP-aware client (Claude, GitHub Copilot agent mode, and others) over stdio. Instead of stuffing source files into the model's context and hoping, the agent asks a precise question and gets an exact, structured answer straight from the .db.

Ask it what breaks if you change CartApiClient, and it answers with the actual scenarios at risk — three scenarios across the Catalog and Identity features. Now the agent knows what already exists before it writes a line — so it reuses the step, extends the right client, and puts the code where it belongs.

Try it

TestAtlas is open source under MIT, and both tools are on NuGet as .NET global tools:

dotnet tool install --global TestAtlas.Cli
dotnet tool install --global TestAtlas.Mcp

The repository includes a real sample solution — an eight-project shop with API tests, UI tests, and an end-to-end journey — so you can generate the report and dependency map on something concrete before pointing it at your own code.

GitHub: https://github.com/Karzone/TestAtlas

If your test-automation suite has grown past the point where any one person holds it in their head — or you want to give your coding agent a fighting chance — give it a map.

DEV Community: Karthik Kalaiyarasu