155 tests passed. Every real request failed.

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I build GigLegal alone. It is a contracts tool for freelancers, running on Next.js 15 with Prisma, and it carries 155 Playwright end to end spec files.

One feature matters for this story. The wizard uses a model to help a freelancer assemble a contract, and the design decision that shapes everything is what the model is forbidden to do. It never writes legal text. Not one sentence. Every clause in a finished contract comes out of a finite library that a human wrote and a lawyer can review, so the model's entire job is selection and filling: pick the right template off the shelf, flip the correct toggles, propose numbers into slots that already have bounds on them.

That constraint is the whole product. It is also the reason this bug was possible, because "choose from a fixed vocabulary" turns into a structured output request carrying a very large schema, and my first version asked for the wizard's sixty odd inputs in a single call.

Bug Fix or Performance Improvement

Every real request failed. Every test passed.

Not most requests. All of them, one hundred percent, while 155 spec files stayed green and told me the feature was fine.

The reason sits in my own provider seam, which I wrote, and which I then stopped reading:

export function resolveAiProviderKind(): AiProviderKind {
  const explicit = process.env.AI_PROVIDER;
  if (explicit && KNOWN_PROVIDERS.includes(explicit)) return explicit;
  if (process.env.NODE_ENV === 'test') return 'mock';
  return 'disabled';
}

That third line is the bug. Not a typo in it, nothing wrong with how it is written. The bug is what it implies about everything downstream.

My tests never touched the real service. They touched a mock, and a mock is not a copy of an API. It is a written record of what you assume the API does, which means my suite was reading my assumptions back to me and calling them results. It did that 155 times without once disagreeing.

Three walls were sitting behind that mock, stacked one in front of the other, so each only became visible after I cleared the one ahead of it.

Wall one, a shape the service rejects outright

The request specified a structured output configuration that the API refuses at validation time, before any inference happens at all. No degraded answer. No partial result. It threw out the shape of the question.

Clearing that revealed the second wall, which had been waiting there the entire time.

Wall two, 66 optional fields against a ceiling of 24

My schema carried 66 optional fields. The service allowed 24.

You cannot reason your way to that number from the outside. It is not a property of the problem, it is a limit somebody at a vendor picked, and the only way to discover it is to walk into it. My mock had no opinion on field counts for the very good reason that I had no opinion on field counts when I sat down and wrote the mock.

Wall three, the step that never answered

Flattening the schema got requests accepted, and most steps began working immediately. One did not. The policies step, twenty fields of interlocking constraints, never came back inside sixty seconds. Never once. My client was configured with exactly that much patience:

timeout: 60_000,
maxRetries: 1,

There is a comfortable explanation available here, and it is that twenty interlocking policy fields are genuinely hard to reason about and the model simply needs longer than a minute. That explanation is plausible. It is also the kind of story that points you straight at raising the timeout and calling it solved.

So I tested it.

I sent a request with one field down the same code path. One field, trivial content, nothing in it worth thinking about.

It was just as slow.

That result kills the difficulty explanation on the spot. If a one field request and a twenty field request burn the same wall clock time, then the cost is not in the reasoning at all, it is in forcing the output to satisfy the shape, and the schema itself was the expense. I would never have found that by staring at the policies step, because the policies step looks exactly like something that ought to be slow.

That single control request is the reason this got fixed instead of worked around. Without it I raise the timeout to ninety seconds, watch it fail, and raise it again.

Code

Three changes. None of them touched the model or the prompt. All three changed the shape of the conversation.

Per step conversations. One call used to carry the entire wizard vocabulary. Now each step gets its own call and sees only its own fields.

// Before: one request, every step's vocabulary, 66 optional fields.
// After:  one request per step. Each schema small and closed.

That bought two things at once. Every request dropped under the field ceiling, and latency fell by five to ten times, because each call now constrains something small instead of something enormous.

Enums derived instead of copied. This one matters more than it sounds:

// DERIVED. A hand-copied six-key list here forbade the model from ever
// emitting the seventh preset the prompt told it about.
import { ALL_PRESET_KEYS } from '@/lib/rules/presets';
const PRESET_KEYS = ALL_PRESET_KEYS;

That comment marks a second bug, found later, and it belongs to the same family as the first one. Somebody copies a literal out of a source of truth. The source of truth moves. The copy does not. Deriving the list made that entire class of failure impossible rather than merely unlikely.

The heaviest step packed into one field instead of twenty. The policies step still needs twenty values. It no longer asks the model to satisfy twenty separately constrained fields, it asks for one and unpacks it on the server.

Range caps deliberately stayed out of the schema. The engine's lintRuleSet is the only range authority in the system, and duplicating those bounds into a schema would have manufactured exactly the divergence that the preset key bug already demonstrates.

My Improvements

	Before	After
Real request success rate	0%	working
Policies step	never returned in 60s	5 to 10 seconds
Optional fields per request	66, ceiling 24	under the ceiling
Latency per step		5 to 10 times faster
Test suite	155 green, 100% wrong	155 green, and honest

No model change. No prompt engineering. The whole fix was engineering against the real constraints of a real service, and none of it was available to me while I still believed my own mock.

There is a thing I would tell someone before they sit down to write a mock, and it is this. A mock is a hypothesis about a service you do not control. Mine made three claims: my response format was fine, field counts did not matter, and latency tracked difficulty. All three were false. A green suite agreed with all three, enthusiastically, 155 times, for weeks.

The test that catches this is not a unit test. It is one real call, against the real service, with the real schema, run once. I have one now.

Best Use of Google AI

GigLegal now runs on Gemini 3.7 Flash.

I want to be precise about the order things happened in, because the order is the interesting part. The redesign above came first. The migration came after. Nothing in that fix was a workaround for a vendor, so when I moved providers the architecture came with me unchanged, and that turned out to be the real test of whether it was a good architecture or just a patch that happened to work.

What I ask a model to do here is deliberately, almost insultingly small.

It cannot write a contract. It cannot write a clause. It cannot write a sentence, or half of one, or a single adjective that ends up in front of a freelancer's client. Every word in a finished GigLegal contract came out of a library a human wrote and a lawyer can sit down and read.

So what is left for the model? Selection. Filling. Look at what this person described about their job, pick the right template off a finite shelf, flip the toggles that match, propose numbers into slots that already have bounds. That is the whole surface.

Three rules hold that surface in place, and none of them moved during any of this:

Anything the model proposes that moves money needs a human tap before it applies. The model can add to a list. It cannot shorten one. The same server side caps that bind a freelancer bind the model too, enforced server side, checked by the same lintRuleSet that checks a human.

It behaves like an assistant. The words stay the platform's.

That is an unfashionable way to use a frontier model in 2026. Most of the interesting work right now points the other way, toward more autonomy and wider surface, and I understand why. But legal text is a domain where a fluent wrong answer is worse than no answer, because a fluent wrong answer is the one that gets signed. So the model gets the part of the job where being wrong is recoverable, and a human keeps the part where it is not.

Here is the part I did not expect. Constraining the model that hard is what made the whole thing fast.

A model that writes prose has to be given room. A model that picks from a closed enum does not, and a small closed schema is cheap to satisfy in a way an enormous open one is not. That is the same finding as the one field control request, arriving from the other direction. The expense was never the thinking. It was the shape.

So the safety architecture and the performance architecture turned out to be the same architecture. I did not design it that way. I found out.

---