DEV Community

Cover image for Your React Tests Execute the UI. But Do They Verify It?
Saptarshi Niyogi
Saptarshi Niyogi

Posted on AI-assisted

Your React Tests Execute the UI. But Do They Verify It?

#react #testing #opensource #typescript

Your React Tests Execute the UI. But Do They Verify It?

Passing tests and high code coverage can still leave important UI behavior unproven.

That sounds counterintuitive at first.

If the test passes, the component rendered, the button was clicked, and the relevant code executed, what exactly is missing?

The answer is often the behavioral oracle: the assertion that proves the expected observable outcome actually happened.

That distinction is what led me to build an open-source project called ui-behavior-coverage.

It explores a simple question:

Did the test explicitly verify the UI behavior it exercised?

A test can exercise behavior without verifying it

Consider a simple React component:

export interface SaveButtonProps {
  disabled?: boolean;
  onSave(): void;
}

export function SaveButton({
  disabled,
  onSave,
}: SaveButtonProps) {
  return (
    <button disabled={disabled} onClick={onSave}>
      Save
    </button>
  );
}
Enter fullscreen mode Exit fullscreen mode

Now imagine this test:

it('handles a disabled button', async () => {
  const onSave = vi.fn();

  render(
    <SaveButton
      disabled
      onSave={onSave}
    />,
  );

  await user.click(
    screen.getByRole('button', {
      name: 'Save',
    }),
  );
});
Enter fullscreen mode Exit fullscreen mode

The test:

  • renders the component;
  • provides disabled;
  • finds the button;
  • attempts the interaction;
  • executes the relevant test path.

But what behavior did it actually prove?

The important contract here is:

When the button is disabled, clicking it should not invoke onSave.

The test never checks that.

A stronger test adds the missing oracle:

it('does not save when disabled', async () => {
  const onSave = vi.fn();

  render(
    <SaveButton
      disabled
      onSave={onSave}
    />,
  );

  await user.click(
    screen.getByRole('button', {
      name: 'Save',
    }),
  );

  expect(onSave).not.toHaveBeenCalled();
});
Enter fullscreen mode Exit fullscreen mode

Both tests interact with almost the same implementation.

But only the second test explicitly verifies the expected behavior.

That difference is easy for ordinary execution coverage to miss.

Code coverage is answering a different question

Traditional code coverage is extremely valuable.

Line coverage asks:

Did this line execute?

Branch coverage asks:

Were both sides of this conditional traversed?

Function coverage asks:

Was this function called?

Those are useful signals.

But they do not necessarily tell us:

Did the test prove the behavior that mattered?

That led me to separate three concepts.

Code Coverage 

Did the implementation execute?
Enter fullscreen mode Exit fullscreen mode

Behavior Reach 

Did the test actually exercise the UI behavior?
Enter fullscreen mode Exit fullscreen mode

Behavior Verification 

Did the test explicitly prove the expected outcome?
Enter fullscreen mode Exit fullscreen mode

A test can therefore reach a behavior without verifying it.

That is the gap ui-behavior-coverage, or UBC, is designed to make visible.

DISCOVERED, EXERCISED, and VERIFIED

UBC classifies supported behavioral contracts using three states.

DISCOVERED

The analyzer found a behavior in the component, but the test did not establish evidence that it reached that behavior.

EXERCISED

The test reached or interacted with the behavior, but no matching verification was found.

VERIFIED

The behavior was reached and the test contains explicit matching evidence for the expected result.

For the disabled-button example:

Weak test

Behavior:
disabled=true -> click suppression

Reached:     yes
Verified:    no

Status: EXERCISED
Enter fullscreen mode Exit fullscreen mode

After adding:

expect(onSave).not.toHaveBeenCalled();
Enter fullscreen mode Exit fullscreen mode

the same behavioral contract becomes:

Reached:     yes
Verified:    yes

Status: VERIFIED
Enter fullscreen mode Exit fullscreen mode

Weak oracles are surprisingly easy to write

The problem is not limited to disabled buttons.

Consider a checkbox.

A test might contain:

await user.click(checkbox);

expect(onChange).toHaveBeenCalled();
Enter fullscreen mode Exit fullscreen mode

This proves that some callback happened.

But if the intended behavior is:

unchecked -> checked
Enter fullscreen mode Exit fullscreen mode

then the assertion does not necessarily prove the expected transition.

A stronger callback assertion might be:

expect(onChange).toHaveBeenCalledWith(
  expect.objectContaining({
    target: expect.objectContaining({
      checked: true,
    }),
  }),
);
Enter fullscreen mode Exit fullscreen mode

Or the observable evidence could be rendered DOM state:

expect(checkbox).toBeChecked();
Enter fullscreen mode Exit fullscreen mode

The same principle applies to many common UI behaviors.

For example:

expect(button).toBeDisabled();

expect(input).toHaveValue('expected value');

expect(dialog).toBeVisible();

expect(element).toHaveAttribute(
  'aria-expanded',
  'true',
);
Enter fullscreen mode Exit fullscreen mode

The important question is not simply:

Did the test interact with the component?

It is:

What observable behavior did the test actually prove?

Why this matters even more with AI-generated tests

AI coding assistants can produce tests extremely quickly.

They can:

  • scaffold test files;
  • identify components;
  • create mocks;
  • use Testing Library;
  • click controls;
  • generate assertions;
  • increase traditional code coverage.

That productivity is useful.

But it also creates an interesting test-quality problem.

A generated test can look sophisticated while still containing weak behavioral evidence.

For example, an AI-generated test may:

✓ compile
✓ render the component
✓ click the right control
✓ call the expected mock
✓ pass in Jest or Vitest
✓ increase line coverage
Enter fullscreen mode Exit fullscreen mode

while still failing to verify the actual UI outcome.

UBC does not try to detect whether a test was written by a human or an LLM.

That distinction is not important for the analyzer.

Instead, it evaluates the resulting test evidence.

This creates a potentially useful separation of responsibilities:

AI coding assistant
        ↓
generates tests
        ↓
Jest / Vitest
        ↓
does the test pass?
        ↓
Code coverage
        ↓
did implementation execute?
        ↓
UI Behavior Coverage
        ↓
did the test verify the behavior?
Enter fullscreen mode Exit fullscreen mode

The origin of the test does not matter.

The quality of the evidence does.

What UBC currently analyzes

ui-behavior-coverage is a static analyzer for React component tests.

The current stable release is:

0.1.0
Enter fullscreen mode Exit fullscreen mode

Install it with:

npm install -D ui-behavior-coverage
Enter fullscreen mode Exit fullscreen mode

Then scan a project:

npx ui-behavior-coverage scan .
Enter fullscreen mode Exit fullscreen mode

Or generate JSON:

npx ui-behavior-coverage scan . --json
Enter fullscreen mode Exit fullscreen mode

You can also analyze a single component/test pair:

npx ui-behavior-coverage analyze \
  --component src/SaveButton.tsx \
  --test src/SaveButton.test.tsx
Enter fullscreen mode Exit fullscreen mode

UBC does not execute the application.

It performs conservative static analysis over supported React production and test patterns.

Material UI is currently a first-class semantic provider

The current implementation includes first-class Material UI semantics.

Supported areas include a conservative subset of:

  • native button disabled-event suppression;
  • MUI Button disabled/loading behavior;
  • rendered disabled state;
  • controlled Checkbox behavior;
  • controlled Switch behavior;
  • standalone Radio behavior;
  • controlled TextField callback/value evidence;
  • native-mode MUI Select;
  • selected MUI input value state;
  • Slider public value state;
  • Dialog, Popover, Menu, and Modal visibility;
  • selected public aria-* state;
  • React Admin / React Hook Form style bindings;
  • simple local wrappers;
  • styled() wrappers;
  • barrel exports;
  • TypeScript path aliases;
  • Testing Library rerender() state evidence;
  • statically safe local render helpers.

The word conservative is important.

If the analyzer cannot trace a behavior safely, it should leave that case unclassified instead of inventing evidence.

Precision is more important than producing a large number of findings.

Observable behavior versus implementation detail

Real React code often introduces local event handlers.

For example:

function handleClick() {
  onOpenChange?.(true);
}

<Button
  disabled={disabled}
  onClick={handleClick}
/>
Enter fullscreen mode Exit fullscreen mode

A naive analyzer might treat handleClick itself as a public behavioral obligation.

But that function is an implementation detail.

From the component consumer's perspective, the meaningful observable contract is represented by the public API, such as:

onOpenChange
Enter fullscreen mode Exit fullscreen mode

This distinction matters.

If a static analyzer exposes internal implementation handlers as testing obligations, it can generate false or misleading contracts that users cannot reasonably verify from the public component surface.

UBC therefore applies precision-oriented rules to suppress supported internal-handler cases.

The same issue appears with callback payloads.

Suppose a framework event is transformed before being passed to a public callback.

The analyzer should not automatically assume that the public consumer receives the original framework event.

Behavioral verification has to follow the actual observable API.

Real repositories reveal problems synthetic tests don't

Synthetic fixtures are necessary for unit tests.

But they are not enough for validating an analyzer like this.

Real React applications contain:

  • nested wrappers;
  • barrel exports;
  • aliases;
  • TypeScript path mappings;
  • render helpers;
  • dynamic test IDs;
  • rerenders;
  • local handlers;
  • form abstractions;
  • different testing conventions.

For that reason, I have also been validating UBC against pinned scopes from public open-source React repositories.

One Phase-B evaluation used a pinned snapshot of Cytoscape Web.

That evaluation exposed four analyzer limitations involving:

  1. internal implementation handlers;
  2. render-state evidence hidden behind local helpers;
  3. Testing Library rerender();
  4. exact DOM-property assertions associated with dynamic production data-testid values.

The important part of the process was the order:

run analyzer
    ↓
manually adjudicate findings
    ↓
freeze expected classifications
    ↓
write regression tests
    ↓
change analyzer
    ↓
rerun validation
Enter fullscreen mode Exit fullscreen mode

The corrected analyzer produced the following result for that pinned evaluation scope:

Consumer-facing contracts: 12
Reached contracts:          8
Verified contracts:         1

Behavior Reach:             66.7%
Behavior Verification:       8.3%
Verification Gap:           58.4 percentage points
Enter fullscreen mode Exit fullscreen mode

Those numbers are not an accuracy score.

They describe one pinned application snapshot and the currently supported semantic surface.

The useful result was that real-world validation exposed concrete precision problems that synthetic tests had not.

Behavior Reach and Behavior Verification

UBC reports two aggregate metrics.

Behavior Reach

The percentage of discovered supported behavioral contracts that tests actually reach.

Behavior Verification

The percentage of discovered supported behavioral contracts with explicit matching verification.

This allows a third metric:

Verification Gap =
Behavior Reach - Behavior Verification
Enter fullscreen mode Exit fullscreen mode

For example:

Behavior Reach:          78%
Behavior Verification:   51%
Verification Gap:        27 percentage points
Enter fullscreen mode Exit fullscreen mode

That 27-point gap represents supported behavior that tests are reaching without equivalent explicit verification.

It does not mean the application is wrong.

It does not even necessarily mean the tests are bad.

It means:

The analyzer found stronger evidence of behavioral execution than behavioral proof.

That is worth reviewing.

Where UBC fits in the testing stack

I don't see behavioral verification coverage as a replacement for existing tools.

They answer different questions.

Jest / Vitest
    ↓
Does the test pass?

Istanbul / V8 coverage
    ↓
Did the implementation execute?

Testing Library
    ↓
How does the test interact with observable UI?

UI Behavior Coverage
    ↓
Did the test explicitly verify
the behavior it exercised?

Mutation testing
    ↓
Can deliberate implementation changes
survive the test suite?
Enter fullscreen mode Exit fullscreen mode

A mature testing strategy can use several of these signals together.

Why not just use mutation testing?

Mutation testing is one of the strongest techniques available for evaluating test-suite quality.

A mutation tool deliberately changes the implementation:

disabled = true
Enter fullscreen mode Exit fullscreen mode

might conceptually become:

disabled = false
Enter fullscreen mode Exit fullscreen mode

and then checks whether the test suite detects the change.

That is powerful.

Behavioral verification coverage is addressing a different problem.

UBC attempts to answer:

Does this test contain explicit evidence for this supported UI behavior?

without repeatedly modifying and executing the application.

The approaches can complement each other.

Mutation testing measures whether tests detect implementation changes.

Behavioral verification analysis inspects whether expected observable outcomes are explicitly represented in the tests.

Why static analysis?

Static analysis has useful properties for this problem.

It can be:

  • fast;
  • deterministic;
  • local;
  • CI-friendly;
  • independent of browser execution;
  • inspectable;
  • reproducible.

But it also has limitations.

A static analyzer cannot safely understand every possible React architecture.

UBC intentionally does not claim support for arbitrary:

  • hooks;
  • contexts;
  • effects;
  • state machines;
  • browser layout;
  • portals;
  • computed CSS;
  • animation timing;
  • arbitrary custom form abstractions.

Unsupported behavior should remain unsupported until there is a sufficiently precise rule.

I would rather have:

no classification
Enter fullscreen mode Exit fullscreen mode

than a confident-looking but misleading one.

JSON output is versioned

For automation, UBC exposes versioned JSON.

Example:

{
  "schemaVersion": "1",
  "toolVersion": "0.1.0",
  "reportType": "project",
  "summary": {
    "discovered": 9,
    "exercised": 2,
    "verified": 1,
    "behaviorReach": 22.2,
    "behaviorVerification": 11.1,
    "verificationGap": 11.1
  },
  "report": {}
}
Enter fullscreen mode Exit fullscreen mode

The intent is to make the analyzer usable from scripts and CI systems without depending entirely on human-readable CLI output.

What I want to build next

The current 0.1.0 release establishes the first stable semantic surface.

The next phase is less about adding a huge number of UI-library rules and more about making behavioral verification useful in normal development workflows.

Some areas I am considering include:

Baseline support

A mature project may already contain many verification gaps.

Requiring teams to fix every historical gap before adopting the tool would create unnecessary friction.

A more useful workflow would be:

record existing baseline
        ↓
allow existing gaps
        ↓
fail only when new gaps are introduced
Enter fullscreen mode Exit fullscreen mode

CI policies

For example:

ubc scan . --fail-on-new-gaps
Enter fullscreen mode Exit fullscreen mode

or configurable verification thresholds.

GitHub integration

PR annotations could surface behavioral verification gaps directly during code review.

SARIF output

This could allow UBC findings to participate in existing static-analysis and code-scanning workflows.

Better explanations

If a behavior is classified as EXERCISED, the tool should eventually be able to explain:

Production behavior:
disabled=true -> callback suppression

Test evidence:
disabled=true rendered
button interaction detected

Missing evidence:
no matching suppression assertion found
Enter fullscreen mode Exit fullscreen mode

Coding-agent feedback loops

One especially interesting direction is using UBC as an independent evaluator for AI-generated tests:

AI generates test
      ↓
test passes
      ↓
UBC finds weak verification
      ↓
AI strengthens test
      ↓
UBC rechecks
Enter fullscreen mode Exit fullscreen mode

That is still future work, but I think it is an interesting use case for behavioral test analysis.

Try it

The package is available on npm:

npm install -D ui-behavior-coverage
Enter fullscreen mode Exit fullscreen mode

Scan a React project:

npx ui-behavior-coverage scan .
Enter fullscreen mode Exit fullscreen mode

GitHub:

https://github.com/sapniyogi/ui-behavior-coverage

npm:

https://www.npmjs.com/package/ui-behavior-coverage

The project is open source under the MIT license.

I am especially interested in failure cases

At this stage, feedback that exposes analyzer limitations is more useful than simply adding more component patterns.

If you try UBC, I would particularly like to see examples where:

  • a meaningful behavior was missed;
  • implementation detail was incorrectly surfaced as a contract;
  • a strong assertion was classified as only EXERCISED;
  • a wrapper pattern could not be resolved;
  • a behavior was classified VERIFIED when it should not have been.

Those cases help improve the precision model.

One final question

Modern frontend testing has become very good at answering:

Did the test run successfully?

Coverage tooling is very good at answering:

How much implementation code did the test execute?

As AI makes it cheaper to generate more code, more tests, and more assertions, I think another question becomes increasingly important:

What did those tests actually prove?

That is the question I am exploring with UI Behavior Coverage.

Top comments (0)