This is the third field test in my Scarab Diagnostic Suite series.
The first field test showed SDS converting a known Open WebUI provider/config issue into a bounded repair lane.
The second found a retrieval truth boundary in Open WebUI before the issue text was used as functional proof.
The third field test moved into a different kind of repo entirely.
Not an AI application.
Not a RAG pipeline.
Not a provider adapter.
This time the target was Vite.
Specifically: Vite issue #21979, where a plugin transform hook filter behaved differently in dev and build.
That shift matters.
This tested whether Scarab Diagnostic Suite could generalize beyond AI-app runtime bugs into framework/build-system contracts.
The issue
The reported bug was small on the surface.
A Vite plugin defines a transform hook with an id filter:
ts filter: { id: { include: '*/.(js)', }, }
In dev mode, the filter is applied and the transform reaches JavaScript files.
In build mode, the same filter shape does not apply.
The reporter noted that:
text */.js
works in both dev and build, but:
text */.(js)
does not.
The expectation was simple:
Even if a particular glob syntax is unusual, Vite’s hook filter should behave consistently between dev and build.
That is the important part.
This was not just:
“a glob pattern is weird.”
The real contract was:
The same plugin hook filter should mean the same thing in dev and build.
That is exactly the kind of consistency boundary SDS should be able to see.
What SDS found
SDS surfaced the failure as a framework/build lifecycle hotspot.
The key diagnostic shape was:
text framework_runtime_mode_filter_contract_consistency
Plain English:
A filter contract that should behave consistently across runtime modes was producing different results depending on whether Vite was running in dev or build.
That is a different surface from the Open WebUI tests.
In Open WebUI #25038, the unstable boundary was retrieval truth moving through a Memory/RAG source-context path.
In Vite #21979, the unstable boundary was framework behavior moving through two execution modes:
dev mode,
build mode.
Same plugin contract.
Different outcome.
That is runtime-mode contract drift.
Why this matters
Framework users do not usually think in terms of internal execution-mode contract divergence.
They think:
“I wrote a plugin hook.”
“It works in dev.”
“Why does build behave differently?”
That is why these bugs matter.
Dev/build divergence creates a trust problem.
A developer can test something locally, see it work during dev, and only discover during build that the contract was not actually stable.
The outward symptom is small:
“my plugin did not run during build.”
The deeper issue is:
“the system is interpreting the same declared filter through different semantic paths.”
Scarab’s job is to find that deeper boundary.
The repair lane
The repair was bounded to the build transform path.
The local change made the build wrapper apply Vite’s transform hook filter before invoking the plugin handler, rather than allowing lower-level build filtering behavior to decide that path first.
The goal was not to redesign hook filters.
The goal was not to change all glob semantics.
The goal was not to patch Rolldown broadly.
The goal was one narrow contract repair:
For the reported filter shape, build should apply the transform consistently with dev.
That was the repair slice.
Red/green proof
The focused regression test proved the reported behavior.
Before repair, the test failed because the transform hook did not reach:
text /src/main.js
during build.
The failing shape was:
text expected false to be true
After repair, the same focused test passed.
The validation package was broader than a single unit test:
focused regression passed,
eslint passed on touched files,
oxfmt passed,
Vite build-bundle passed,
Vite build-types passed,
Vite typecheck passed.
That mattered because framework/build-system patches need more than “one test went green.”
They need confidence that the change does not disturb the surrounding build pipeline.
SDS after repair
After the repair, SDS reran against the target.
The specific runtime-mode filter contract finding cleared.
The broader repo was not magically clean, and that is fine.
SDS is not supposed to pretend the entire repo becomes perfect after one bug repair.
The important part is that the selected diagnostic surface changed in the expected direction:
the focused contract finding cleared,
the hotspot quieted enough for Stepwise to move onward,
and the repair remained bounded to the issue-shaped failure.
That is the SDS loop working again:
diagnose the surface,
repair only the selected lane,
prove the behavior,
rerun diagnostics,
confirm the targeted finding changed.
What this field test proved
Field Test #003 proved something new:
SDS is not only useful for AI-app bugs.
It can detect framework/build-system contract drift.
This is a different class of failure than the Open WebUI tests.
The Open WebUI examples were about runtime data and response-shape boundaries.
The Vite example was about execution-mode consistency inside a build tool.
That matters because it suggests the underlying SDS model is not just memorizing one kind of bug.
It is beginning to reason about repo truth across different software surfaces.
In this case, the truth was:
A plugin hook filter should mean the same thing in dev and build.
When that truth broke, SDS found the surface.
Internal classification
Field Test #003
Project: Vite
Issue: #21979
Surface: framework runtime-mode filter contract
Contract class: dev/build hook-filter consistency
Mode: blind diagnostic surface detection followed by focused red/green repair
Result: local repair successful; targeted SDS finding cleared; focused regression passed
Primary lesson: SDS can detect and guide repair of framework/build lifecycle contract drift
Why this matters to Scarab
This field test broadened the product story.
Scarab Diagnostic Suite is not a bug-fix bot.
It is not tied to one repo.
It is not limited to AI application middleware.
It is a diagnostic supervision layer for AI-assisted development.
That means it needs to understand different kinds of repo truth:
provider config truth,
retrieval context truth,
external API metadata truth,
runtime-mode behavior truth,
build artifact truth.
Vite #21979 added a new one:
execution-mode contract truth.
Same input.
Same plugin contract.
Different runtime mode.
Different behavior.
That is a real diagnostic surface.
And SDS found it.
The larger pattern
Across the first three field tests, the pattern is becoming clearer.
The value is not that an AI agent can write code quickly.
The value is that SDS makes the repair smaller before the AI agent starts writing.
That is the difference between unmanaged AI coding and diagnostic-led repair.
Unmanaged AI often starts with a symptom and wanders.
SDS starts with a boundary.
In this field test, the boundary was not:
“Vite is broken.”
It was:
“Vite’s transform hook filter contract diverges between dev and build.”
That is smaller.
That is testable.
That is repairable.
That is the point of Scarab Diagnostic Suite.
Top comments (0)