Test Your Vibe-Coded Feature Across Every Layer Before You Give It a Production Budget

The request is simple: a user clicks "Summarize," the UI calls /api/summarize, the route calls an AI provider, and the result lands in Postgres. In the vibe-coded version of this feature, every one of those handoffs works in the demo and fails in a different way under load. The UI doesn't know the provider can return a 429. The route doesn't know the UI retries. Nobody owns the timeout.

This post is about the step I now insert before any of that code gets a production API key: a cross-layer failure harness that runs the whole vertical slice against a free model lane, so the seams break where it's cheap instead of where it's metered.

Why a free lane, and where MonkeyCode fits

Testing an AI feature end to end has an awkward economics problem. You want hundreds of runs against the real request path — including the slow, the malformed, and the rate-limited responses — but you don't want to pay production-token prices to learn that your error envelope is inconsistent.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access on its free server option as the development lane for exactly this: it gives you a real HTTP endpoint with real latency and real failure modes, without attaching a billing meter to your test suite. That is the entire role it plays here. Everything below works with any OpenAI-compatible endpoint, and the production provider is a different configuration of the same contract — that's the point.

Do not treat the free lane as representative of production model quality, quota, or availability. Treat it as a sparring partner for your plumbing.

The seam that actually matters

The vibe-coded failure mode I see most often is that the provider SDK is imported in four different files, each with its own idea of what an error looks like. The fix is one contract behind one route:

// server/ai/provider.ts — the ONLY file allowed to talk to a model
export interface SummarizeRequest {
  text: string;
  maxTokens?: number;
}

export type SummarizeResult =
  | { ok: true; summary: string; model: string; latencyMs: number }
  | { ok: false; code: "RATE_LIMITED" | "TIMEOUT" | "UPSTREAM_ERROR" | "BAD_RESPONSE"; retryable: boolean };

export interface Summarizer {
  summarize(req: SummarizeRequest): Promise<SummarizeResult>;
}

Note what the contract does: it collapses every upstream failure into four named codes with a retryable flag. The UI never sees an axios error, a provider-specific JSON blob, or a stack trace. It sees a shape it can switch on.

The free-lane implementation is one configuration of that interface:

// server/ai/openaiCompatible.ts
export function makeOpenAICompatibleSummarizer(cfg: {
  baseUrl: string; apiKey: string; model: string; timeoutMs: number;
}): Summarizer {
  return {
    async summarize({ text, maxTokens = 512 }) {
      const started = Date.now();
      const controller = new AbortController();
      const timer = setTimeout(() => controller.abort(), cfg.timeoutMs);
      try {
        const res = await fetch(`${cfg.baseUrl}/chat/completions`, {
          method: "POST",
          signal: controller.signal,
          headers: {
            "content-type": "application/json",
            authorization: `Bearer ${cfg.apiKey}`,
          },
          body: JSON.stringify({
            model: cfg.model,
            max_tokens: maxTokens,
            messages: [{ role: "user", content: `Summarize:\n\n${text}` }],
          }),
        });
        if (res.status === 429) return { ok: false, code: "RATE_LIMITED", retryable: true };
        if (!res.ok) return { ok: false, code: "UPSTREAM_ERROR", retryable: res.status >= 500 };
        const body = await res.json();
        const summary = body?.choices?.[0]?.message?.content;
        if (typeof summary !== "string") return { ok: false, code: "BAD_RESPONSE", retryable: false };
        return { ok: true, summary, model: cfg.model, latencyMs: Date.now() - started };
      } catch (e) {
        if (e instanceof Error && e.name === "AbortError")
          return { ok: false, code: "TIMEOUT", retryable: true };
        return { ok: false, code: "UPSTREAM_ERROR", retryable: true };
      } finally {
        clearTimeout(timer);
      }
    },
  };
}

Dev lane wiring points at the free server; production wiring points at your paid provider. Same interface, same route, same UI. The difference is two environment variables, not a rewrite.

The harness: fail every layer on purpose

Now the part the prototype skipped. I keep a script that drives the real route — not the provider function in isolation — through the failure states the contract promises:

// scripts/harness.ts — run with: tsx scripts/harness.ts
const cases = [
  { name: "happy path",        text: "A normal paragraph...",       expect: "summary" },
  { name: "empty input",       text: "",                            expect: "4xx-from-our-route" },
  { name: "oversized input",   text: "x".repeat(200_000),           expect: "4xx-from-our-route" },
  { name: "burst of 20",       text: "parallel", concurrency: 20,   expect: "RATE_LIMITED-or-success, never 500-opaque" },
  { name: "slow upstream",     text: "normal", timeoutMs: 1,        expect: "TIMEOUT, retryable: true" },
] as const;

for (const c of cases) {
  const results = await driveRoute("/api/summarize", c);
  assertContractShape(results);       // every response matches SummarizeResult
  assertNoLeakedUpstreamBody(results); // no provider JSON or stack in the payload
  assertIdempotentRetry(c, results);   // retrying a retryable failure must not double-write
}

Three assertions here earn their keep:

1. Contract shape on the failure path. Anyone can return the right shape on success. The harness only cares that RATE_LIMITED from the provider arrives at the UI as the same envelope every time.
2. No leaked upstream bodies. The first time I ran this against a real endpoint, the provider's error HTML was being forwarded verbatim to the browser. The free lane caught it because free lanes tend to have more interesting error bodies, not fewer.
3. Idempotent retry. The UI retries retryable: true results. If your route writes a row before calling the model, a retry after a timeout writes two rows. This is the bug that only shows up under real latency — another reason to test against a real server rather than a mock.

What I log before productionizing

During harness runs, the route records latencyMs, the failure code, and the model identifier from the contract result. That gives me two things for free: a p95 latency baseline from the dev lane (a floor, not a ceiling — production providers differ), and proof that the failure-code distribution is observable in the UI's error handling. If the dashboard only ever shows UPSTREAM_ERROR, the contract exists on paper only.

Decision table: when the free lane is enough

Question	Free model lane	Paid production provider
Validate error envelope end to end	✅	overkill
Test retry/idempotency under real latency	✅	also fine
Rate-limit behavior under burst	✅ (limits hit sooner — a feature)	expensive
Evaluate summary quality for your domain	❌	required
Capacity planning / real p95	❌	required
Anything with user data	❌ never	with your data policy

Limitations and who should skip this

• The free lane tells you nothing about output quality, throughput guarantees, or long-term availability. Promoting based on "the free tests passed" is the same vibe-coding mistake one level up.
• If your feature's risk is in the prompt (hallucination, tone, domain accuracy), a plumbing harness is the wrong tool — you need an eval set against the production model.
• If your provider's SDK gives you a richer contract than OpenAI-compatible HTTP (streaming, tool calling), abstract at that level instead; don't flatten your contract to fit a test lane.
• Never send real user content through any free tier. Seed the harness with synthetic text.

Checklist

• [ ] One file owns all provider contact; everything else imports the interface
• [ ] Every upstream failure maps to a named, retryable-flagged code
• [ ] Harness drives the real route, including 429, timeout, and malformed-body cases
• [ ] Retry path proven not to double-write
• [ ] Latency and failure codes logged per request
• [ ] Production promotion is an env-var change, verified by re-running the same harness

If you want to try this without a bill attached, MonkeyCode's free model access on its free server is a reasonable dev lane to point the harness at — then swap the base URL when the seams hold.

Which layer handoff is least stable in your stack right now — and what's the actual response code or failure state you last saw leak across it? I'd rather debug your concrete case than argue in the abstract.