Building ScenarIQ: An API Coverage Engine with AST-Based Static Analysis

By the ScenarIQ engineering team

---

Every QA team we've worked on eventually hits the same question from a manager, an auditor, or a worried release owner:

"Which of our API endpoints actually have tests?"

And every team we've seen answers it the same way: someone opens the backend repo in one window, the automation repo in another, and starts building a spreadsheet. Two weeks later the spreadsheet exists. Two sprints later it's wrong.

We built ScenarIQ because we got tired of doing that audit by hand. This post is the long version of how it works: the five-stage pipeline, the parser design decisions, the AI hallucination that forced us to build a citation verifier, and the self-audit that took our own coverage recall from 87.2% to 100%.

---

Why we built it

The pain is structural, not organizational.

Most mid-size teams keep their API tests in a separate repository from the backend. The backend repository contains the Spring Boot controllers and business logic, while the automation repository contains the TestNG + REST Assured test suites.

This creates a fundamental visibility problem:

Which backend APIs are actually covered by our automation?

Answering that question manually requires several steps. First, someone needs to understand the backend and identify all the controllers, routes, HTTP methods, parameters, and endpoints. Then they need to inspect the automation repository and determine which tests and HTTP calls correspond to those endpoints.

The mapping between the two repositories usually exists nowhere. It lives in spreadsheets, documentation, or simply in the heads of the engineers who wrote the tests.

This creates several recurring problems:

1. Manual cross-repository audits

   Someone has to inspect both repositories, identify the available endpoints, find the corresponding API calls in the automation code, and create a coverage matrix manually.

   This process is slow, error-prone, and difficult to maintain as the codebase changes.

2. Stale coverage information

   An endpoint is added, an API changes, or an automation test is removed, but the coverage document is not updated.

   Over time, the coverage matrix becomes outdated and can no longer be trusted as an accurate representation of the system.

3. Tribal knowledge

   Teams often rely on individual engineers knowing which tests cover which APIs.

   "The order APIs were covered by the tests written last quarter."

   But what happens when the engineer changes teams or leaves the organization?

   The knowledge disappears with them.

4. Endpoint coverage is not scenario coverage

   Finding a test for an endpoint does not necessarily mean that the endpoint is adequately tested.

   For example, an endpoint might have a happy-path test while completely missing:

*   Validation failures

*   Authentication and authorization scenarios

*   Not-found cases

*   Conflict scenarios

*   Boundary conditions

*   Negative cases

*   Business-rule validations


Therefore, knowing that an endpoint is **covered** is only the first step. Teams also need to understand **what the tests actually validate**.

1. Missing scenarios are difficult to identify

   Even after mapping APIs to tests, teams still need to determine what important scenarios are missing.

   This requires understanding both the backend implementation and the existing automation, which makes the process even more difficult to perform manually.

The information needed to answer these questions already exists in the source code.

The backend repository tells us:

What APIs exist?

The automation repository tells us:

What does the test suite actually call?

The challenge is connecting these two sources and then going one step further:

What is covered, what is actually validated, and what important scenarios are still missing?

That's the problem we wanted to solve with ScenarIQ.

---

Existing solutions and their limits

We looked closely at the existing testing and coverage landscape before writing a line of code.

Line-coverage tools measure a different dimension

Tools such as JaCoCo for Java and Istanbul for JavaScript are extremely useful for measuring which lines, branches, or instructions were executed by tests.

But they don't directly answer the API-level question we were trying to solve.

When API tests live in a separate automation repository and communicate with a deployed backend over HTTP, traditional code coverage does not directly tell us:

Of the N endpoints this service exposes, how many does the automation suite actually exercise?

For example, a backend might have high line coverage while important APIs remain completely untested.

Conversely, the automation repository might have excellent code coverage while the team still has limited visibility into which backend APIs are actually being validated.

ScenarIQ focuses on this missing layer:

API and scenario coverage across the backend and automation repositories.

API platforms don't necessarily understand your automation code

API management, observability, and specification-based tools provide valuable visibility into APIs, traffic, contracts, and OpenAPI definitions.

However, they don't necessarily understand what exists inside an automation repository or which automated tests exercise those APIs.

The product vision

ScenarIQ answers one question — which endpoints are covered by tests? — and answers it with evidence. Every verdict comes with a file:line citation you can click through and verify yourself. No confidence scores, no "probably covered." Covered means: here is the test file, here is the line, here is the resolved URL that matches this endpoint.

Three outputs fall out of that:

• Covered endpoints — with the exact tests that cover them

• Uncovered endpoints — the actionable gap list

• Orphan tests — tests hitting endpoints that no longer exist in the backend (dead weight that still costs CI minutes and maintenance)

And critically: zero execution. ScenarIQ never runs your tests, never needs a build, never touches your CI. It reads source code only.

Architecture overview

ScenarIQ is a Spring Boot 3.4 backend with a React 19 dashboard. Repos are cloned via JGit; Java source is parsed with JavaParser. The analysis is a five-stage pipeline:

1. AST endpoint inventory. Parse the backend's controllers into a complete endpoint list. JavaParser handles Spring Boot (@RestController, @RequestMapping composition, method-level mappings, path variables); a route parser handles Laravel.

2. Dataflow call resolution. Parse the automation repo and resolve every HTTP call's method and path — through variables, constants, static initializers, System.getProperty defaults, helper/wrapper methods, and class inheritance.

3. Service binding. Match test calls to the right backend service by base-URL keyword (from the URL template or the variable name). One automation repo often tests many services; calls have to land against the right endpoint inventory.

4. Strict matching. Deterministic endpoint-to-call matching. Every verdict carries file:line evidence.

1. Optional AI layer. Judges scenario quality — are the right cases tested? — and never decides coverage.

Stages 1–4 are fully deterministic. Run the same scan twice, get the same answer twice. That property turned out to matter more than we initially appreciated — more on that in the AI section.

The stage boundaries are also the extensibility seams. Stages 3 and 4 operate on normalized inventories — endpoint templates on one side, resolved calls on the other — and don't care which parser produced them. Adding a framework means adding a parser (Stage 1) or a resolver profile (Stage 2), not touching matching. That's how Laravel support landed alongside Spring Boot without forking the pipeline, and it's the shape future framework support will take.

Parser and resolver design

Stage 1 is the easy half. Spring annotations are declarative; composing a class-level @RequestMapping("/api/v1/orders") with a method-level @PostMapping("/{id}/approve") into POST /api/v1/orders/{id}/approve is mechanical AST work.

Stage 2 is where the engineering lives, because real automation code never writes URLs as literals. A representative (simplified) example of what we actually see:

// Simplified — representative of real enterprise automation code
public class ApiTestBase {
    protected static final String BASE_URI;

    static {
        BASE_URI = System.getProperty("service.base.uri",
                       "https://staging.internal/order-service");
    }

    protected Response executePost(String path, Object body) {
        return RestAssured.given()
                .baseUri(BASE_URI)
                .body(body)
                .post(path);
    }
}

public class ApproveOrderTest extends ApiTestBase {
    @Test
    public void approveOrder_Test() {
        String endpoint = "/orders/" + orderId + "/approve";
        Response response = executePost(endpoint, payload);
        // ...
    }
}

The actual HTTP call here is POST {BASE_URI}/orders/{id}/approve — but to see that, the resolver has to:

• Follow static initializers to find what BASE_URI is

• Understand System.getProperty defaults — the second argument is the value that matters for path analysis

• Recognize wrapper sinks — executePost() isn't a REST Assured call itself; it's a method whose parameter flows into one. We call these wrapper sinks: the method is a "sink" for a path argument

• Walk inheritance — executePost() is defined on the superclass, not the test class

• Track parameters transitively — the path travels through a local variable (endpoint), into a method parameter (path), into the .post() call

Each of those is a distinct resolver capability, and each one we skipped initially cost us real false negatives (see the self-audit section — it cost us exactly 6 endpoints, 12.8% of recall).

The resolver builds these capabilities as composable passes over the AST: constant folding for string concatenation, a symbol table per class hierarchy for field resolution, and a paramRefs mechanism that lets a value flow through any number of intermediate variables and method parameters before reaching an HTTP sink.

Coverage reconciliation: strict matching

Once we have an endpoint inventory and a resolved call inventory, matching sounds trivial. It isn't, quite — /orders/{id}/approve in the backend has to match /orders/8842/approve and /orders/{orderId}/approve and /orders/" + id + "/approve from the test side.

We made one decision early and never regretted it: matching is strict and deterministic. A call matches an endpoint only when the HTTP method matches and the path structurally matches (literal segments equal, parameter segments aligned). No fuzzy scoring, no "78% similar."

The consequence: our coverage 100% precision on our audited reference repository — when ScenarIQ says an endpoint is covered, it is. The trade-off is that anything we can't resolve becomes an explicit unresolved call, surfaced in the report, rather than a guess. We think that's the right trade for a tool whose whole value proposition is trustworthiness. A coverage report that's occasionally optimistically wrong is worse than no report.

Orphan detection falls out of the same reconciliation for free: a fully resolved test call that matches no endpoint in the inventory means the test targets an endpoint that no longer exists. Those tests still burn CI time and mislead anyone reading the suite — and they're invisible from inside the automation repo, where nothing about them looks wrong.

Beyond binary endpoint coverage, the engine derives scenario coverage: for each endpoint, the set of cases that should exist (success, validation errors, auth failures, not-found) versus the cases the matched tests actually exercise. An endpoint with one happy-path test and an endpoint with a full negative-case suite are both "covered" in the binary sense; scenario coverage is what separates them. On our reference repo pair the deterministic scenario number is 65.8% (258/392) — more on the second, AI-expanded scenario number below.

The AI insight engine

Strict matching answers whether an endpoint is covered. It can't answer how well. An endpoint with one happy-path test and an endpoint with tests for validation errors, auth failures, and boundary cases both show up as "covered."

That quality judgment is genuinely fuzzy, which makes it a good fit for an LLM — with one hard rule that is the core of our positioning:

Deterministic static analysis decides whether an endpoint is covered. AI only judges how well it's covered. Coverage verdicts are never hallucinated because AI never makes them.

The AI layer (Precision + AI scan mode) receives the deterministic engine's output — matched endpoints with their citing tests — plus minimal grounded code snippets, and reports on scenario quality: missing negative cases, absent status-code coverage, suggested additional tests. It never adds or removes an endpoint from the covered list. The division of labor is architectural, not a prompt instruction.

The AI also expands the scenario universe itself: cases the deterministic engine can't enumerate — edge cases and negative paths implied by what an endpoint actually does. That's why we report two scenario numbers on purpose: the deterministic 65.8%, stable and reproducible, is the one to trend over time; the AI-inclusive 48.8% (333/682), with its nearly doubled denominator, is the honest view of true coverage debt. Showing only the flattering number would be marketing; showing both is measurement.

Two other AI-layer behaviors follow the same trust-first design. Risk scoring is blended: a deterministic rubric (coverage, quality, HTTP method) sets the baseline tier, and the LLM may adjust by at most ±1 tier — the baseline is fully reproducible, and the AI can nudge but never overturn. And prompt grounding is automatic: the AI is shown the repo's own conventions — dominant base-URL variable, sample paths, argument order — extracted from the resolved calls, so generated test suggestions match the team's actual style, including required annotations like @Owner. (Before grounding, generated suggestions missed @Owner on 5 of 5 tests for a repo that requires it on every test method.)

Citation verification: the hallucination incident

Early in building the AI layer, a report cited EvaluateOrderValidationTest.java as evidence for a quality finding. The analysis was plausible. The file did not exist. The model had invented a filename that looked exactly like the repo's naming convention — which is precisely what makes LLM hallucination dangerous in developer tooling: the fabrications are plausible.

One fake citation in a report poisons trust in every real one. So we adopted a design principle — a claim we can't verify is a claim we won't show — and built a 4-gate citation verifier that every AI claim must pass before persistence:

• Gate 1 — verbatim source match. Any code the AI quotes must exist verbatim in the analyzed source. Paraphrased or "reconstructed from memory" snippets are rejected.

• Gate 2 — status/assertion match. If the AI claims a test asserts a particular status code or condition, that assertion must actually match what's in the cited test.

• Gate 3 — file existence. Every cited file must exist in the analysis bundle. This gate alone would have caught the original incident.

• Gate 4 — automation-side grounding. Suggested test code must be grounded in automation-repo classes only. This catches a subtle failure: the AI suggesting tests that import backend-only DTOs the automation repo can't see.

Any claim failing any gate is discarded before it's stored. Zero hallucinated file citations are now possible by construction — not because the prompt says "please don't hallucinate," but because unverifiable claims never reach the database. The gates are protected by 14 dedicated regression tests.

Security design

Coverage analysis means cloning customer source code, so the security posture had to be boring and conservative:

• GitHub tokens are encrypted at rest (AES)

• Service-level tokens are supported — different repos can authenticate with different GitHub accounts, matching how enterprises actually partition access

• Code is cloned for analysis, results are stored; source code is never sent to any third party for the deterministic scan

• AI scans send only the minimal grounded snippets needed for scenario analysis — never the whole repo

• JWT secrets fail fast on placeholder values in prod, so a misconfigured deployment refuses to start rather than running with a known secret

The self-audit: 87.2% → 100%

In July 2026 we ran ScenarIQ against a repo pair we could fully ground-truth by hand: a production enterprise microservice with 80 endpoints and a real enterprise automation repo. Then we audited every discrepancy, fixed the engine, and re-ran.

Metric	Before	After
Coverage recall (tested endpoints detected)	87.2% (41/47)	100% (47/47)
False negatives	6 (all wrapper-sink pattern)	0
Coverage precision (no false "covered")	100%	100%
Endpoint discovery recall	100%	100% (80/80)
Endpoint discovery precision	97.5% (2 phantoms)	100% (0 phantoms)
Unresolved calls	65	27 (−58%)
Deterministic scenario coverage	57.9% (209/361)	65.8% (258/392)
AI-inclusive scenario coverage	not available	48.8% (333/682) — new
AI citation gates	2	4
Hallucinated file citations	1 found	0 — structurally impossible
Regression tests for engine accuracy	0	38
Overall engine audit score	7.8/10	9.0/10

The headline bug: all 6 false negatives shared one pattern. Tests called endpoints through a wrapper method (executePost()) inherited from a base class. Our resolver saw the test, saw the wrapper call, and lost the thread. Six endpoints with perfectly good tests were reported as uncovered.

The fix required three resolver upgrades working together:

1. Superclass-chain sink lookup — when a called method isn't defined in the current class, walk the inheritance chain until we find it and check whether it's a wrapper sink

2. Wrapper base-URI field binding — read the wrapper's internal base-URI field (including its static initializer and System.getProperty default) so resolved paths bind to the right service

3. Transitive parameter tracking — follow the path value through local variables and method parameters into the sink

Recall: 87.2% → 100%. The same audit killed 2 phantom endpoints (parser artifacts on the discovery side — and in one case, we found our own hand-built ground truth was wrong, which was its own lesson about why manual audits fail), cut unresolved calls by 58%, and left behind 38 regression tests pinning the engine's accuracy so none of these bugs can quietly return.

The 27 calls still unresolved after the audit are genuinely dynamic — URLs assembled from runtime data no static analyzer can know. We flag them explicitly in the report. We do not guess.

[SCREENSHOT: before/after audit comparison view showing recall, unresolved calls, and audit score deltas]

Scalability considerations

A scan clones two repos and parses them fully, so the practical costs are network and parse time. What keeps a full precision scan at 2–5 minutes on a real production repo pair:

• Clone caching. Repos are cloned once and reused across scans; subsequent scans fetch rather than re-clone.

• Stale-clone detection by remote URL. If a project's configured repo URL changes (repo moved, fork swapped), the cached clone is detected as stale by comparing remote URLs and is re-cloned rather than silently analyzed against the wrong code.

• Batched queries. Persisting an 80-endpoint, several-hundred-call analysis is done with batched database writes rather than row-at-a-time chatter.

Credits for a scan are refunded automatically if it fails or is cancelled — an accuracy-adjacent decision: we don't charge for answers we didn't deliver.

Lessons learned

Ground-truth your own tool, publicly. The before/after audit was uncomfortable — publishing that we shipped with 87.2% recall and a hallucinated citation is not classic marketing. But every fix in that table exists because we measured ourselves against a hand-verified baseline. And our ground truth itself had errors, which is the whole thesis: humans can't maintain this mapping by hand.

Real code is the test suite. Every resolver capability — static initializers, getProperty defaults, wrapper sinks, inheritance — came from patterns in real enterprise automation code, not from patterns we imagined. Synthetic test repos would have told us the engine was finished long before it was.

Determinism is a feature you can't retrofit. Because coverage verdicts are fully deterministic, we could write 38 exact-value regression tests, publish exact numbers, and let users re-verify any verdict. None of that works if a probabilistic component sits in the verdict path.

Constrain the AI structurally, not rhetorically. Prompt instructions reduce hallucination; verification gates eliminate the ones that matter. The 4-gate verifier plus the coverage/quality division of labor means the worst-case AI failure is a discarded claim, never a wrong coverage verdict.

Roadmap

Today ScenarIQ supports Spring Boot (Java) and Laravel (PHP) backends, with TestNG + REST Assured automation repos. More frameworks are coming — the pipeline stages are framework-agnostic; parsers and resolvers are the pluggable parts.

Scan modes today: Quick Scan (free on the Pro plan), Precision Scan (1 credit, the deterministic engine), and Precision + AI (2 credits, adds AI scenario analysis). The deterministic engine is the default — the legacy regex engine has been retired as default.

If your backend and your API tests live in different repos and nobody can say with evidence which endpoints are covered, that's exactly the question we built this to answer.

Try it at scenariq.co — free tier, no credit card.